Adds Progressing status to BackupBucketsReady seed condition (#6587)

* Adds configuration for the BackupBucketsCheck controller

* Adds Progressing status to BackupBucketsReady condition

* Adds integration test

* Addresses review comments

* Adds ability to create test client with field selector support

* Deflake integration test

* Addresses review comments

charts/gardener/controlplane/charts/runtime/templates/controller-manager/configmap-componentconfig.yaml

@@ -96,6 +96,15 @@ data:
 {{ toYaml .Values.global.controller.config.controllers.seedExtensionsCheck.conditionThresholds | indent 8 }}
         {{- end }}
       {{- end }}
+      {{- if .Values.global.controller.config.controllers.seedBackupBucketsCheck }}
+      seedBackupBucketsCheck:
+        concurrentSyncs: {{ required ".Values.global.controller.config.controllers.seedBackupBucketsCheck.concurrentSyncs is required" .Values.global.controller.config.controllers.seedBackupBucketsCheck.concurrentSyncs }}
+        syncPeriod: {{ required ".Values.global.controller.config.controllers.seedBackupBucketsCheck.syncPeriod is required" .Values.global.controller.config.controllers.seedBackupBucketsCheck.syncPeriod }}
+        {{- if .Values.global.controller.config.controllers.seedBackupBucketsCheck.conditionThresholds }}
+        conditionThresholds:
+{{ toYaml .Values.global.controller.config.controllers.seedBackupBucketsCheck.conditionThresholds | indent 8 }}
+        {{- end }}
+      {{- end }}
       {{- if .Values.global.controller.config.controllers.event }}
       event:
         {{- if .Values.global.controller.config.controllers.event.concurrentSyncs }}

charts/gardener/controlplane/values.yaml

@@ -395,6 +395,12 @@ global:
           conditionThresholds:
           - type: ExtensionsCheck
             duration: 1m
+        seedBackupBucketsCheck:
+          concurrentSyncs: 5
+          syncPeriod: 30s
+          conditionThresholds:
+          - type: BackupBucketsReady
+            duration: 1m
         shootMaintenance:
           concurrentSyncs: 5
           enableShootControlPlaneRestarter: true

docs/concepts/controller-manager.md

@@ -89,7 +89,7 @@ controllers:
 ```
 
 The Project controller takes the shown `config` and creates a `ResourceQuota` with the name `gardener` in the project namespace.
-If a `ResourceQuota` resource with the name `gardener` already exists, the controller will only update fields in `spec.hard` which are **unavailable** at that time. 
+If a `ResourceQuota` resource with the name `gardener` already exists, the controller will only update fields in `spec.hard` which are **unavailable** at that time.
 Labels and annotations on the `ResourceQuota` `config` get merged with the respective fields on existing `ResourceQuota`s.
 An optional `projectSelector` narrows down the amount of projects that are equipped with the given `config`.
 If multiple configs match for a project, then only the first match in the list is applied to the project namespace.
@@ -186,12 +186,12 @@ The "main" reconciler takes care about this replication:
 |:-------:|:---------:|:-----:|
 | Secret | garden | gardener.cloud/role |
 
-#### "Backup Bucket" Reconciler
+#### "Backup Buckets Check" Reconciler
 
 Every time a `BackupBucket` object is created or updated, the referenced `Seed` object is enqueued for reconciliation.
 It's the reconciler's task to check the `status` subresource of all existing `BackupBuckets` that belong to this seed.
 If at least one `BackupBucket` has `.status.lastError`, the seed condition `BackupBucketsReady` will turn `false` and
-consequently the seed is considered as `NotReady`. Once the `BackupBucket` is healthy again, the seed will be re-queued
+consequently the seed is considered as `NotReady`. If the `SeedBackupBucketsCheckControllerConfiguration`, which is part of `gardener-controller-manager`s `ControllerManagerControllerConfiguration`, contains a `conditionThreshold` for the `BackupBucketsReady`, the condition will instead first be set to `progressing` and eventually to `false` once the `conditionThreshold` expires, see [the example config file](../../example/20-componentconfig-gardener-controller-manager.yaml) for details. Once the `BackupBucket` is healthy again, the seed will be re-queued
 and the condition will turn `true`.
 
 #### "Lifecycle" Reconciler
@@ -205,7 +205,7 @@ In case a `Lease` is not renewed for the configured amount in `config.controller
 1. The reconciler assumes that the Gardenlet stopped operating and updates the `GardenletReady` condition to `Unknown`.
 2. Additionally, conditions and constraints of all `Shoot` resources scheduled on the affected seed are set to `Unknown` as well
 because a striking Gardenlet won't be able to maintain these conditions any more.
-3. If the gardenlet's client certificate has expired (identified based on the `.status.clientCertificateExpirationTimestamp` field in the `Seed` resource) and if it is managed by a `ManagedSeed` then this will be triggered for a reconciliation. This will trigger the bootstrapping process again and allows gardenlets to obtain a fresh client certificate. 
+3. If the gardenlet's client certificate has expired (identified based on the `.status.clientCertificateExpirationTimestamp` field in the `Seed` resource) and if it is managed by a `ManagedSeed` then this will be triggered for a reconciliation. This will trigger the bootstrapping process again and allows gardenlets to obtain a fresh client certificate.
 
 ### ControllerRegistration Controller
 
@@ -241,5 +241,5 @@ On startup the gardenlet uses a `kubeconfig` with a [bootstrap token](https://ku
 
 The controller in `gardener-controller-manager` checks whether the `CertificateSigningRequest` has the expected organisation, common name and usages which the gardenlet would request.
 
-It only auto-approves the CSR if the client making the request is allowed to "create" the 
+It only auto-approves the CSR if the client making the request is allowed to "create" the
 `certificatesigningrequests/seedclient` subresource. Clients with the `system:bootstrappers` group are bound to the `gardener.cloud:system:seed-bootstrapper` `ClusterRole`, hence, they have such privileges. As the bootstrap kubeconfig for the gardenlet contains a bootstrap token which is authenticated as being part of the [`systems:bootstrappers` group](../../charts/gardener/controlplane/charts/application/templates/clusterrolebinding-seed-bootstrapper.yaml), its created CSR gets auto-approved.

example/20-componentconfig-gardener-controller-manager.yaml

@@ -21,6 +21,12 @@ controllers:
     conditionThresholds:
       - type: ExtensionsReady
         duration: 1m
+  seedBackupBucketsCheck:
+    concurrentSyncs: 5
+    syncPeriod: 30s
+    conditionThresholds:
+      - type: BackupBucketsReady
+        duration: 1m
   shootMaintenance:
     concurrentSyncs: 5
   # enableShootControlPlaneRestarter: true

pkg/controllermanager/apis/config/types.go

@@ -71,6 +71,8 @@ type ControllerManagerControllerConfiguration struct {
 	Seed *SeedControllerConfiguration
 	// SeedExtensionsCheck defines the configuration of the SeedExtensionsCheck controller.
 	SeedExtensionsCheck *SeedExtensionsCheckControllerConfiguration
+	// SeedBackupBucketsCheck defines the configuration of the SeedBackupBucketsCheck controller.
+	SeedBackupBucketsCheck *SeedBackupBucketsCheckControllerConfiguration
 	// ShootMaintenance defines the configuration of the ShootMaintenance controller.
 	ShootMaintenance ShootMaintenanceControllerConfiguration
 	// ShootQuota defines the configuration of the ShootQuota controller.
@@ -217,6 +219,19 @@ type SeedExtensionsCheckControllerConfiguration struct {
 	ConditionThresholds []ConditionThreshold
 }
 
+// SeedBackupBucketsCheckControllerConfiguration defines the configuration of the
+// SeedBackupBucketsCheck controller.
+type SeedBackupBucketsCheckControllerConfiguration struct {
+	// ConcurrentSyncs is the number of workers used for the controller to work on
+	// events.
+	ConcurrentSyncs *int
+	// SyncPeriod is the duration how often the existing resources are reconciled (how
+	// often the health check of BackupBuckets is performed).
+	SyncPeriod *metav1.Duration
+	// ConditionThresholds defines the condition threshold per condition type.
+	ConditionThresholds []ConditionThreshold
+}
+
 // ShootMaintenanceControllerConfiguration defines the configuration of the
 // ShootMaintenance controller.
 type ShootMaintenanceControllerConfiguration struct {

pkg/controllermanager/apis/config/v1alpha1/defaults.go

@@ -141,6 +141,10 @@ func SetDefaults_ControllerManagerConfiguration(obj *ControllerManagerConfigurat
 		obj.Controllers.SeedExtensionsCheck = &SeedExtensionsCheckControllerConfiguration{}
 	}
 
+	if obj.Controllers.SeedBackupBucketsCheck == nil {
+		obj.Controllers.SeedBackupBucketsCheck = &SeedBackupBucketsCheckControllerConfiguration{}
+	}
+
 	if obj.Controllers.ShootMaintenance.ConcurrentSyncs == nil {
 		v := DefaultControllerConcurrentSyncs
 		obj.Controllers.ShootMaintenance.ConcurrentSyncs = &v
@@ -301,3 +305,14 @@ func SetDefaults_SeedExtensionsCheckControllerConfiguration(obj *SeedExtensionsC
 		obj.SyncPeriod = &v
 	}
 }
+
+// SetDefaults_SeedBackupBucketsCheckControllerConfiguration sets defaults for the given SeedBackupBucketsCheckControllerConfiguration.
+func SetDefaults_SeedBackupBucketsCheckControllerConfiguration(obj *SeedBackupBucketsCheckControllerConfiguration) {
+	if obj.ConcurrentSyncs == nil {
+		v := DefaultControllerConcurrentSyncs
+		obj.ConcurrentSyncs = &v
+	}
+	if obj.SyncPeriod == nil {
+		obj.SyncPeriod = &metav1.Duration{Duration: 30 * time.Second}
+	}
+}

pkg/controllermanager/apis/config/v1alpha1/defaults_test.go

@@ -86,6 +86,8 @@ var _ = Describe("Defaults", func() {
 
 			Expect(obj.Controllers.SeedExtensionsCheck).NotTo(BeNil())
 
+			Expect(obj.Controllers.SeedBackupBucketsCheck).NotTo(BeNil())
+
 			Expect(obj.Controllers.ShootMaintenance.ConcurrentSyncs).NotTo(BeNil())
 			Expect(obj.Controllers.ShootMaintenance.ConcurrentSyncs).To(PointTo(Equal(5)))
 
@@ -226,6 +228,16 @@ var _ = Describe("Defaults", func() {
 			Expect(obj.SyncPeriod).To(PointTo(Equal(metav1.Duration{Duration: 30 * time.Second})))
 		})
 	})
+
+	Describe("#SetDefaults_SeedBackupBucketsCheckControllerConfiguration", func() {
+		It("should correctly default the SeedBackupBucketsCheck Controller configuration", func() {
+			obj := &SeedBackupBucketsCheckControllerConfiguration{}
+
+			SetDefaults_SeedBackupBucketsCheckControllerConfiguration(obj)
+			Expect(obj.ConcurrentSyncs).To(PointTo(Equal(5)))
+			Expect(obj.SyncPeriod).To(PointTo(Equal(metav1.Duration{Duration: 30 * time.Second})))
+		})
+	})
 })
 
 var _ = Describe("Constants", func() {

pkg/controllermanager/apis/config/v1alpha1/types.go

@@ -85,6 +85,9 @@ type ControllerManagerControllerConfiguration struct {
 	// SeedExtensionsCheck defines the configuration of the SeedExtensionsCheck controller.
 	// +optional
 	SeedExtensionsCheck *SeedExtensionsCheckControllerConfiguration `json:"seedExtensionsCheck,omitempty"`
+	// SeedBackupBucketsCheck defines the configuration of the SeedBackupBucketsCheck controller.
+	// +optional
+	SeedBackupBucketsCheck *SeedBackupBucketsCheckControllerConfiguration `json:"seedBackupBucketsCheck,omitempty"`
 	// ShootMaintenance defines the configuration of the ShootMaintenance controller.
 	ShootMaintenance ShootMaintenanceControllerConfiguration `json:"shootMaintenance"`
 	// ShootQuota defines the configuration of the ShootQuota controller.
@@ -259,6 +262,22 @@ type SeedExtensionsCheckControllerConfiguration struct {
 	ConditionThresholds []ConditionThreshold `json:"conditionThresholds,omitempty"`
 }
 
+// SeedBackupBucketsCheckControllerConfiguration defines the configuration of the SeedBackupBucketsCheck
+// controller.
+type SeedBackupBucketsCheckControllerConfiguration struct {
+	// ConcurrentSyncs is the number of workers used for the controller to work on
+	// events.
+	// +optional
+	ConcurrentSyncs *int `json:"concurrentSyncs,omitempty"`
+	// SyncPeriod is the duration how often the existing resources are reconciled (how
+	// often the health check of BackupBuckets is performed).
+	// +optional
+	SyncPeriod *metav1.Duration `json:"syncPeriod,omitempty"`
+	// ConditionThresholds defines the condition threshold per condition type.
+	// +optional
+	ConditionThresholds []ConditionThreshold `json:"conditionThresholds,omitempty"`
+}
+
 // ShootMaintenanceControllerConfiguration defines the configuration of the
 // ShootMaintenance controller.
 type ShootMaintenanceControllerConfiguration struct {