Support kube scheduler ComponentConfig

[damemi]

I know right now we support policy config through the operator, but don't know what our status is on componentconfig. So making this issue to track that and look into it

[damemi]

/cc @ravisantoshgudimetla any idea if we should be working on this? I'm not certain on the status of CC but I know upstream was trying to get it to GA

[soltysh]

/cc @ingvagabund

[damemi]

There was some discussion in our channel about whether we'll ever actually need to support CC. I asked in sig-scheduling and it seems that yes, someday policy config will be deprecated but CC is still a long way from GA. So, this is something we should keep in mind for our operator but more of a long term goal

[damemi]

Upstream tracking issue to deprecate policy in favor of componentconfig: kubernetes/kubernetes#87526

[damemi]

As of 1.19, the v1alpha1 scheduler config library is removed, which is what our operator depends on. We'll need to update our scheduler to use ComponentConfig by then

Currently, we observe the scheduler config by giving the scheduler a configmap that has the policy in it (see

cluster-kube-scheduler-operator/pkg/operator/configobservation/scheduler/observe_scheduler.go

Line 17 in 82b5c99

func ObserveSchedulerConfig(genericListers configobserver.Listers, recorder events.Recorder, existingConfig map[string]interface{}) (map[string]interface{}, []error) {

)

In v1alpha2, the AlgorithmSource field is deprecated, meaning we can no longer simply pass a configmap name that holds a Policy file. Instead, we'll need to pass that observed config directly as a SchedulerProfile. Note that this means documentation should be updated to reflect that the scheduler will now be configured by Plugin (filter/score/etc) componentconfig and NOT policy (predicates/priorities)

For reference, currently when you try to run a >1.19 kube-scheduler image with our operator you get the following error right away:

no kind "KubeSchedulerConfiguration" is registered for version "kubescheduler.config.k8s.io/v1alpha1" in scheme "k8s.io/kubernetes/pkg/scheduler/apis/config/scheme/scheme.go:30"

Updating our KubeSchedulerConfiguration to v1alpha2 (#245) fixes it, but trying to pass a custom config results in this error:

strict decoder error for {"algorithmSource":{"policy":{"configMap":{"name":"policy-configmap","namespace":"openshift-kube-scheduler"}}},"apiVersion":"kubescheduler.config.k8s.io/v1alpha2","clientConnection":{"kubeconfig":"/etc/kubernetes/static-pod-resources/configmaps/scheduler-kubeconfig/kubeconfig"},"kind":"KubeSchedulerConfiguration","leaderElection":{"leaderElect":true,"resourceLock":"configmaps","resourceNamespace":"openshift-kube-scheduler"}}
: v1alpha2.KubeSchedulerConfiguration.ReadObject: found unknown field: algorithmSource, error found in #10 byte of ...|thmSource":{"policy"|..., bigger context ...|{"algorithmSource":{"policy":{"configMap":{"name":"policy-configmap"|...

Which shows the need to switch from our current algorithmSource config reconciliation to schedulerProfiles

cc @ingvagabund ^ what we talked about

[damemi]

/priority important-soon
/kind bug

[damemi]

To update this, the legacy Policy config is still available under kubescheduler.config.k8s.io/v1, but the flags associated with consuming it are still being deprecated. kubernetes/kubernetes#91420 promotes v1alpha2 componentconfig to v1beta1, and #250 adds a metric to measure how many users are actually using custom configs through the operator.

[damemi]

PR transitioning us to the new componentconfig: #255

[damemi]

Should also note: Policy API is scheduled to be removed for 1.23: kubernetes/kubernetes#92143

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[ingvagabund]

/remove-lifecycle stale

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[damemi]

/close
with the implementation of https://github.com/openshift/enhancements/blob/master/enhancements/scheduling/scheduler-profiles.md (#307) we can close this

[openshift-ci-robot]

@damemi: Closing this issue.

In response to this:

/close
with the implementation of https://github.com/openshift/enhancements/blob/master/enhancements/scheduling/scheduler-profiles.md (#307) we can close this

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mmgaggle]

@damemi Can we revisit this? The enhancement proposal (#307) hasn't been updated since Jan 2019. Is there another way to do this?

[damemi]

@mmgaggle the enhancement should definitely be updated to reflect its current state (it is now implemented). Thank you for pointing this out

[mmgaggle]

Excellent! I'm experimenting with this right now, but I'm having trouble figuring out how to configure a plugin. It looks like you can adjust the weights via policy, and can enable / disable plugins. What I'm interested in specifically is the ability to configure / use PodTopologySpread along the lines of this

https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/#cluster-level-default-constraints

It's not clear to me if this can be done, and if it can, how I might do that. Any ideas?

[damemi]

@mmgaggle I'm very sorry that this thread slipped off my radar, to answer your question though the cluster-level default constraints are currently not possible with the default kube-scheduler in OCP. If you wish to configure settings that require a custom KubeSchedulerConfiguration in OpenShift, right now your only option is to run a secondary scheduler in the cluster.