A single scoring plugin for node resources

[ahg-g]

We have four score plugins that implement different strategies for preferred resource allocation. Those plugins should not be enabled together, only one.

I propose we deprecate those and combine them under one Score plugin, the same one used for filtering (NodeResourcesFit), and add a FitStrategy parameter to NodeResourcesFit that allows users to select which exact scoring strategy to run.

I also suggest that we go ahead and implement four other strategies: LeastAllocatable, MostAllocatable, BestFit and WorstFit. The first two are basically a graduation of https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/noderesources, the latter two are common scheduling placement strategies that k8s currently don't support. BestFit for example is important to achieve higher utilization in clusters that have different VM shapes.

In the future, I think we can explore adding FitStrategy to the pod spec, and make it a workload parameter rather than a static scheduler configuration. This can be useful when we have a mix of serving and batch workloads running on the same cluster. Although scheduling profiles may be an alternative for this, albeit a less flexible one.

Previous discussion on this topic: #93547

/sig scheduling

[k8s-ci-robot]

@ahg-g: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ahg-g]

/cc @Huang-Wei @damemi @alculquicondor

[Huang-Wei]

ref: kubernetes/enhancements#2144

[yuzhiquan]

It's likely already has an sample plugin in scheduler-plugins/#36, it;s reimplement for that plugin?

[alculquicondor]

SGTM.

Should we hold v1beta2 component config to next release to account for this?

#95308 is not merged yet.

[ahg-g]

Should we hold v1beta2 component config to next release to account for this?

I would hold it, any downsides to delaying it?

[alculquicondor]

not really, we are not under pressure of API deprecation like REST APIs are

[ahg-g]

ref: kubernetes/enhancements#2144

Ah, I missed this one. The above proposal tries to support every single combination, which I think is an overkill.

We can make reasonable assumptions that we bake into each strategy to avoid the over-proliferation of options. For example, LeastAllocatable is always associated with LeastAllocated as a second scoring key (as in nodes with the same least allocatable are differentiated by least allocated)

[alculquicondor]

Should we have a KEP for this?

[ahg-g]

Yes, I was thinking we could target 1.22

[ahg-g]

I can put a KEP together in an hour, do we have time to review and merge before the deadline tomorrow? I don't think this needs PRR if we focus this cycle on the config API change to support existing plugins through NodeResourcesFit, wdyt?

[ahg-g]

this will help us deprecate the existing plugins in v1beta2 in 1.22 if all goes to plan and the community is happy with the new api.

[alculquicondor]

Fine with me to do a review last minute, but PRR is a hard requirement now.

[ahg-g]

ok, lets leave it to 1.22 then..

[damemi]

In the future, I think we can explore adding FitStrategy to the pod spec, and make it a workload parameter rather than a static scheduler configuration.

+1 this as a longer-term goal, I like exposing scheduler config options in the pod spec because it gives users better control over their app placement.

[ahg-g]

I opened kubernetes/enhancements#2458

[alculquicondor]

So that't the placeholder for the 1.22 work, right?

[ahg-g]

I will send a PR in 30min, if it gets reviewed by tomorrow, then good, if not then we will delay to 1.22.

[ahg-g]

I sent out kubernetes/enhancements#2461

@Huang-Wei @alculquicondor @damemi timing is too tight, so please feel free to push back... we may not get a PRR too, so...

[wangyanghack]

sry,I don't understand what is difference in (LeastAllocatable and BestFit) or (MostAllocatable and WorstFit)?
Can you explain this for me?Thanks very much

[yuzhiquan]

/assign

[ahg-g]

Here is the plan:

1. Add Score extension point for Fit plugin. This needs to be added to both v1beta1 and v1beta2 (Add score func for NodeResourcesFit plugin #101822)
2. Add validation in v1beta1 such that Fit Score plugin is not used with LeastAllocated, MostAllocated or RequestedToCapacityRatio.
3. Deprecate leastAllocated, mostAllocated or requestedToCapacityRatio in v1beta2
4. One of the following two:
   • Make algorithmProvider.Registry versioned (pass in the component version to NewRegistry). In v1beta2, Fit replaces LeastAllocated score plugin.
   • Remove algorithmProvider flag (it has been marked as deprecated since 1.17), and move the set of default plugins to be set in the CC defaulting logic

[ahg-g]

I created #102151 to remove algorithmProvider pkg.

[chendave]

/cc

[ahg-g]

/close

This is now done.

[k8s-ci-robot]

@ahg-g: Closing this issue.

In response to this:

/close

This is now done.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.