Add score func for NodeResourcesFit plugin

[yuzhiquan]

What type of PR is this?

/kind feature

What this PR does / why we need it:

A single scoring plugin for node resources

Which issue(s) this PR fixes:

Refs #98746

KEP kubernetes/enhancements#2458 kubernetes/enhancements#2461

Special notes for your reviewer:

Does this PR introduce a user-facing change?

A new score extension for NodeResourcesFit plugin that merges the functionality of NodeResourcesLeastAllocated,NodeResourcesMostAllocated,RequestedToCapacityRatio plugins, which are marked as deprecated as of v1beta2. In v1beta1, the three plugins can still be used in v1beta1 but not at the same time with the score extension of NodeResourcesFit

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/sig scheduling
/triage accepted
/priority important-longterm
/assign @ahg-g

[yuzhiquan]

I did make generated_files, this action auto remove k8s.io/kubernetes/pkg/scheduler/apis/config/v1beta1/zz_generated.conversion.go:109: Convert_config_NodeResourcesFitArgs_To_v1beta1_NodeResourcesFitArgs function, and ci failed for this, now i was confused how to fix this? @ahg-g @Huang-Wei

[ahg-g]

I just took a quick look at the API, will take a look at the full PR later today.

As for the error, you need to update the type in staging (https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kube-scheduler/config/v1beta1/types.go)

Note that we are working on v1beta2 (#99597), but for now you can update the v1beta1 type just so we make sure that the PR successfully passes the tests.

[fejta-bot]

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

[ahg-g]

@liggitt @alculquicondor we address all the comments. We updated the conflict logic to apply to the new score plugin only. Please take a look, I really think this is the safe way to go, it is backward compatible and protects users from surprises. But if you still strongly disagree with this approach then we can change it. Let us know and based on that we will update the release notes on the description.

[ahg-g]

@liggitt friendly ping.

[liggitt]

sorry for the delay, was digging into working default 1.21 configs to assure myself this wasn't breaking compatibility

git checkout upstream/release-1.21
make kube-scheduler
kube-scheduler --master=http://localhost:8080 --write-config-to 1.21_default.yaml
kube-scheduler --master=http://localhost:8080 --config 1.21_default.yaml

passing configs with 1.21 scheduler:
- 1.21_default_fixed.yaml (explicitly disabled then enabled plugins to avoid warnings about duplicates)
- 1.21_default_fixed_no_pluginConfig.yaml

failing configs with 1.21 scheduler:
- 1.21_default.yaml
  - `couldn't create scheduler using provider "DefaultProvider": initializing profiles: one queue sort plugin required for profile with scheduler name "default-scheduler"`
- 1.21_default_fixed_score_nodefit.yaml (try to enable NodeResourcesFit as a score plugin)
  - `couldn't create scheduler using provider "DefaultProvider": initializing profiles: creating profile for scheduler name default-scheduler: plugin "NodeResourcesFit" does not extend ScorePlugin plugin`

---

tested with kube-scheduler built from this PR

passing configs with scheduler built from PR:
- 1.21_default_fixed.yaml (explicitly disabled then enabled plugins to avoid warnings about duplicates)
- 1.21_default_fixed_no_pluginConfig.yaml

failing configs with scheduler built from PR:
- 1.21_default.yaml
  - `profiles[0].plugins.queueSort.Enabled: Invalid value: 2: only one queue sort plugin can be enabled`
- 1.21_default_fixed_score_nodefit.yaml (try to enable NodeResourcesFit as a score plugin)
  - `profiles[0].plugins.score.enabled[8]: Invalid value: "NodeResourcesFit": was conflict with ["NodeResourcesLeastAllocated"] in version "kubescheduler.config.k8s.io/v1beta1" (KubeSchedulerConfiguration is version "kubescheduler.config.k8s.io/v1beta1")`

The 1.21 error loading a config that tried to enable NodeResourcesFit as a score plugin was what I was wanting to be sure of.

I do think the plugin conflict logic is hard to follow and makes it easier to break compatibility in the future by registering a conflict that used to be allowed.

---

1.21_default_fixed_no_pluginConfig.yaml.txt
1.21_default_fixed_score_nodefit.yaml.txt
1.21_default_fixed.yaml.txt
1.21_default.yaml.txt

[liggitt]

/approve
for API changes

/hold for scheduler lgtm and approve if needed

@ahg-g I'd like to see these followups:

• ensure the default config output by --write-config-to is usable without manual fixup
• add a test to capture the working default config from current and previous releases to make sure that continues to load/validate successfully

[ahg-g]

Thanks Jordan for the detailed review and due diligence! I will create issues for the followup items.

[ahg-g]

/release-note "A new score extension for NodeResourcesFit plugin that merges the functionality of NodeResourcesLeastAllocated,NodeResourcesMostAllocated,RequestedToCapacityRatio plugins, which are marked as deprecated as of v1beta2. In v1beta1, the three plugins can still be used in v1beta1 but not at the same time with the score extension of NodeResourcesFit"

[ahg-g]

/hold

until release note is updated.

[ahg-g]

/hold cancel

[ahg-g]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, liggitt, yuzhiquan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cmd/kube-scheduler/OWNERS [ahg-g,liggitt]
• pkg/scheduler/OWNERS [ahg-g,liggitt]
• staging/src/k8s.io/kube-scheduler/config/OWNERS [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[yuzhiquan]

Thanks for help and coauthor commit from @ahg-g , and detailed review comment by all of the reviewers, especially @liggitt and @alculquicondor .

[ahg-g]

Created #103309 and #103308