Quota for user facing resources

[timuthy]

What would you like to be added:
Gardener should have configurable, project scoped quotas for resources that end users (specifically project members) are permitted to create:

- apiGroups:
  - ""
  resources:
  - secrets
  - configmaps
  - serviceaccounts

- apiGroups:
  - garden.sapcloud.io
  - core.gardener.cloud
  resources:
  - shoots
  - secretbindings
  - quotas
  - plants

- apiGroups:
  - settings.gardener.cloud
  resources:
  - openidconnectpresets

- apiGroups:
  - garden.sapcloud.io
  - core.gardener.cloud
  resources:
  - projects

The list above contains all possible resources, although some of them might be negligible or aren't critical for the system stability (tbd).

A quota could look like:

apiVersion: core.gardener.cloud/v1alpha1
kind: ResourceQuota
metadata:
  name: standard-quota
  namespace: garden-project
spec:
- apiGroups:
  - garden.sapcloud.io
  - core.gardener.cloud
  resources:
  - shoots
  - secretbindings
  - quotas
  - plants
  limits:
    maxObjects: 200
    maxSizePerObject: 1m 

Unfortunately, k8s native ResourceQuotas don't cover all necessary API objects and uses cases:
https://kubernetes.io/docs/tasks/administer-cluster/quota-api-object/

Why is this needed:
Quotas are one countermeasure to prevent the Gardener- and Kube-Apiserver, it's backing etcds as well as involving controllers from being flooded.

/cc @petersutter @ThormaehlenFred

[swilen-iwanow]

I also think, that it is reasonable gardener admins to be able to specify a quota for creating projects. Otherwise the project API could be abused.

[rfranzke]

It might be interesting to look into whether ResourceQuotas are possible to be reused for object counts of our Gardener-managed resources (https://kubernetes.io/docs/concepts/policy/resource-quotas/#object-count-quota only states "custom resources"). Is it possible?

For the object size there is kubernetes/kubernetes#83261. We could revendor k8s.io/apiserver to v0.16.7 to get this 3MB limit: https://github.com/kubernetes/apiserver/blob/v0.16.7/pkg/server/config.go#L298-L311 (cc @ialidzhikov).

[timuthy]

Thanks for the suggestion @rfranzke. I gave it a shoot and tested ResourceQuota but apparently it doesn't work with core.gardener.cloud resources out of the box. We need some further investigation if it'd work by adjusting the Gardener Apiserver.

[petersutter]

with #2032 also roles and rolebindings need to be considered

[rfranzke]

Thanks @timuthy for trying it out. Indeed, it won't work with custom API servers. How do we proceed here now? Shall we go ahead with your initial proposal and implement our own resource?

[timuthy]

I tried out ResourceQuota a second time because I saw this PR kubernetes/kubernetes#72384 and partly managed to get the right quota count for shoots:

Unfortunately, it didn't work reliably in my local setup, e.g. outdated resource count or no admission denial after the quota exhaustion. I'll take some time to find out more and hope that this is still the way to go since it's a K8s standard feature.

[rfranzke]

/remove lifecycle/stale
/assign @timuthy
/in-progress

[timuthy]

We can use the ResourceQuota to restrict the number of resources. But for Gardener managed resources, i.e. API group core.gardener.cloud we need to implement our own resource quota admission plugin for the Gardener API server analogue to the one from the Kube API server.

[rfranzke]

Thanks for the update @timuthy. Do you have more background? Why? Last time we talked about it you mentioned there was a problem with the resource quota controller that collects the current number of shoots, for example. Do you now also have observed issues in the admission plugin (even if the controller would compute the values correctly)?

[timuthy]

The resource quota controller only updates the current or actual resource usage of Pods and Services as explained by the design doc and implemented here.

It's the admission plugin's task to check if any incoming object potentially violates the hard resource count as well as to update the ResourceQuota status. But the admission plugin is never invoked for API groups offered via the Gardener extension API server.