Skip to content

Latest commit

 

History

History
320 lines (225 loc) · 12.7 KB

component-config.md

File metadata and controls

320 lines (225 loc) · 12.7 KB
Raw

ComponentConfig Controller Runtime Support

Author: @christopherhein

Last Updated on: 03/02/2020

Table of Contents

Summary

Currently controllers that use controller-runtime need to configure the ctrl.Manager by using flags or hardcoding values into the initialization methods. Core Kubernetes has started to move away from using flags as a mechanism for configuring components and standardized on ComponentConfig or Versioned Component Configuration Files. This proposal is to bring ComponentConfig to controller-runtime to allow controller authors to make go types backed by apimachinery to unmarshal and configure the ctrl.Manager{} reducing the flags and allowing code based tools to easily configure controllers instead of requiring them to mutate CLI args.

Motivation

This change is important because:

  • it will help make it easier for controllers to be configured by other machine processes
  • it will reduce the required flags required to start a controller
  • allow for configuration types which aren't natively supported by flags
  • allow using and upgrading older configurations avoiding breaking changes in flags

Links to Open Issues

Goals

  • Provide an interface for pulling configuration data out of exposed ComponentConfig types (see below for implementation)
  • Provide a new ctrl.NewFromComponentConfig() function for initializing a manager
  • Provide an embeddable ControllerManagerConfiguration type for easily authoring ComponentConfig types
  • Provide an DefaultControllerConfig to make the switching easier for clients

Non-Goals/Future Work

  • kubebuilder implementation and design in another PR
  • Changing the default controller-runtime implementation
  • Dynamically reloading ComponentConfig object
  • Providing flags interface and overrides

Proposal

The ctrl.Manager SHOULD support loading configurations from ComponentConfig like objects. An interface for that object with getters for the specific configuration parameters is created to bridge existing patterns.

Without breaking the current ctrl.NewManager which uses an exported ctrl.Options{} the manager.go can expose a new func, NewFromComponentConfig() this would be able to loop through the getters to populate an internal ctrl.Options{} and pass that into New().

//pkg/manager/manager.go

// ManagerConfiguration defines what the ComponentConfig object for ControllerRuntime needs to support
type ManagerConfiguration interface {
	GetSyncPeriod() *time.Duration

	GetLeaderElection() bool
	GetLeaderElectionNamespace() string
	GetLeaderElectionID() string

	GetLeaseDuration() *time.Duration
	GetRenewDeadline() *time.Duration
	GetRetryPeriod() *time.Duration

	GetNamespace() string
	GetMetricsBindAddress() string
	GetHealthProbeBindAddress() string

	GetReadinessEndpointName() string
	GetLivenessEndpointName() string

	GetPort() int
	GetHost() string

	GetCertDir() string
}

func NewFromComponentConfig(config *rest.Config, scheme *runtime.Scheme, filename string, managerconfig ManagerConfiguration) (Manager, error) {
	codecs := serializer.NewCodecFactory(scheme)
    if err := decodeComponentConfigFileInto(codecs, filename, managerconfig); err != nil {

	}
	options := Options{}

	if scheme != nil {
		options.Scheme = scheme
	}

	// Loop through getters
	if managerconfig.GetLeaderElection() {
		options.LeaderElection = managerconfig.GetLeaderElection()
	}
	// ...

	return New(config, options)
}

ComponentConfig Load Order

ComponentConfig Load Order

Embeddable ComponentConfig Type

To make this easier for Controller authors controller-runtime can expose a set of config.ControllerConfiguration type that can be embedded similar to the way that k8s.io/apimachinery/pkg/apis/meta/v1 works for TypeMeta and ObjectMeta these could live in pkg/api/config/v1alpha1/types.go. See the DefaultComponentConfig type below for and example implementation.

// pkg/api/config/v1alpha1/types.go
package v1alpha1

import (
	"time"

	configv1alpha1 "k8s.io/component-base/config/v1alpha1"
)

// ControllerManagerConfiguration defines the embedded RuntimeConfiguration for controller-runtime clients.
type ControllerManagerConfiguration struct {
	Namespace string `json:"namespace,omitempty"`

	SyncPeriod *time.Duration `json:"syncPeriod,omitempty"`

	LeaderElection configv1alpha1.LeaderElectionConfiguration `json:"leaderElection,omitempty"`

	MetricsBindAddress string `json:"metricsBindAddress,omitempty"`

	Health ControllerManagerConfigurationHealth `json:"health,omitempty"`

	Port *int   `json:"port,omitempty"`
	Host string `json:"host,omitempty"`

	CertDir string `json:"certDir,omitempty"`
}

// ControllerManagerConfigurationHealth defines the health configs
type ControllerManagerConfigurationHealth struct {
	HealthProbeBindAddress string `json:"healthProbeBindAddress,omitempty"`

	ReadinessEndpointName string `json:"readinessEndpointName,omitempty"`
	LivenessEndpointName  string `json:"livenessEndpointName,omitempty"`
}

Default ComponentConfig Type

To enable controller-runtime to have a default ComponentConfig struct which can be used instead of requiring each controller or extension to build its own ComponentConfig type, we can create a DefaultControllerConfiguration type which can exist in pkg/api/config/v1alpha1/types.go. This will allow the controller authors to use this before needing to implement their own type with additional configs.

// pkg/api/config/v1alpha1/types.go
package v1alpha1

import (
	"time"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	configv1alpha1 "sigs.k8s.io/controller-runtime/pkg/apis/config/v1alpha1"
)

// DefaultControllerManagerConfiguration is the Schema for the DefaultControllerManagerConfigurations API
type DefaultControllerManagerConfiguration struct {
	metav1.TypeMeta `json:",inline"`

	Spec   configv1alpha1.ControllerManagerConfiguration   `json:"spec,omitempty"`
}

This would allow a controller author to use this struct with any config that supports the json/yaml structure. For example a controller author could define their Kind as FoobarControllerConfiguration and have it defined as the following.

# config.yaml
apiVersion: config.somedomain.io/v1alpha1
kind: FoobarControllerManagerConfiguration
spec:
  port: 9443
  metricsBindAddress: ":8080"
  leaderElection:
    leaderElect: false

Given the following config and DefaultControllerManagerConfiguration we'd be able to initialize the controller using the following.

mgr, err := ctrl.NewManagerFromComponentConfig(ctrl.GetConfigOrDie(), scheme, configname, &defaultv1alpha1.DefaultControllerManagerConfiguration{})
if err != nil {
	// ...
}

The above example uses configname which is the name of the file to load the configuration from and uses scheme to get the specific serializer, eg serializer.NewCodecFactory(scheme). This will allow the configuration to be unmarshalled into the runtime.Object type and passed into the ctrl.NewManagerFromComponentConfig() as a ManagerConfiguration interface.

Using Flags w/ ComponentConfig

Since this design still requires setting up the initial ComponentConfig type and passing in a pointer to ctrl.NewFromComponentConfig() if you want to allow for the use of flags, your controller can use any of the different flagging interfaces. eg flag, pflag, flagnum and set values on the ComponentConfig type prior to passing the pointer into the ctrl.NewFromComponentConfig(), example below.

leaderElect := true

config := &defaultv1alpha1.DefaultControllerManagerConfiguration{
	Spec: configv1alpha1.ControllerManagerConfiguration{
		LeaderElection: configv1alpha1.LeaderElectionConfiguration{
			LeaderElect: &leaderElect,
		},
	},
}
mgr, err := ctrl.NewManagerFromComponentConfig(ctrl.GetConfigOrDie(), scheme, configname, config)
if err != nil {
	// ...
}

Kubebuilder Scaffolding Example

Within expanded in a separate design (link once created) this will allow controller authors to generate a type that implements the ManagerConfiguration interface. The following is a sample of what this looks like:

package config

import (
  "time"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	configv1alpha1 "sigs.k8s.io/controller-runtime/pkg/apis/config/v1alpha1"
)

type ControllerNameConfigurationSpec struct {
	configv1alpha1.ControllerManagerConfiguration `json:",inline"`
}

type ControllerNameConfiguration struct {
  metav1.TypeMeta

  Spec ControllerNameConfigurationSpec `json:"spec"`
}

Usage of this custom ComponentConfig type would require then changing the ctrl.NewFromComponentConfig() to use the new struct vs the DefaultControllerManagerConfiguration.

User Stories

Controller Author with controller-runtime and default type

  • Mount ConfigMap
  • Initialize ctrl.Manager with NewFromComponentConfig with config name and DefaultControllerManagerConfiguration type
  • Build custom controller as usual

Controller Author with controller-runtime and custom type

  • Implement ComponentConfig type
  • Embed ControllerManagerConfiguration type
  • Mount ConfigMap
  • Initialize ctrl.Manager with NewFromComponentConfig with config name and ComponentConfig type
  • Build custom controller as usual

Controller Author with kubebuilder (tbd proposal for kubebuilder)

  • Initialize kubebuilder project using --component-config-name=XYZConfiguration
  • Build custom controller as usual

Controller User without modifications to config

Provided that the controller provides manifests

  • Apply the controller to the cluster
  • Deploy custom resources

Controller User with modifications to config

  • Following from previous example without changes
  • Create a new ConfigMap for changes
  • Modify the controller-runtime pod to use the new ConfigMap
  • Apply the controller to the cluster
  • Deploy custom resources

Risks and Mitigations

  • Given that this isn't changing the core Manager initialization for controller-runtime it's fairly low risk

Alternatives

  • NewFromComponentConfig() could load the object from disk based on the file name and hydrate the ComponentConfig type.

Implementation History

  • 02/19/2020: Proposed idea in an issue or community meeting
  • 02/24/2020: Proposal submitted to controller-runtime
  • 03/02/2020: Updated with default DefaultControllerManagerConfiguration
  • 03/04/2020: Updated with embeddable RuntimeConfig
  • 03/10/2020: Updated embeddable name to ControllerManagerConfiguration