Without the list and watch permissions on secrets a log error is shown

[paologallinaharbur]

dev-newrelic… │ E0429 13:52:53.496860      12 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:default:dev-newrelic-infra-operators" cannot list resource "secrets" in API group "" at the cluster scope

kubernetes-sigs/controller-runtime#1156

[invidian]

maybe we can disable the cache then? with ClientDisableCacheFor https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/manager. Not ideal though..

[paologallinaharbur]

I tested with:

Client: client.New(mgr.GetConfig(), client.Options{Scheme: mgr.GetScheme(), Mapper: mgr.GetRESTMapper()})

Instead of mgr.GetClient() and seems to be working fine

This was the possible workaround mentioned in the issue

[invidian]

Client: client.New(mgr.GetConfig(), client.Options{Scheme: mgr.GetScheme(), Mapper: mgr.GetRESTMapper()})

Perhaps we would need a separate client for secrets then, something like uncachedClient or something.

See also kubernetes-sigs/controller-runtime#244.

EDIT: it seems it will be possible with new release: kubernetes-sigs/controller-runtime@cd065bf.

[paologallinaharbur]

Docs: https://github.com/kubernetes-sigs/controller-runtime/blob/4aecab5d395c264b1a3d2df7cfed21489168d4be/designs/use-selectors-at-cache.md#L114-L115

[paologallinaharbur]

Moreover resourceName cannot be used when creating the secret. So the rbac for creating should embrace every secret. Not a big issue I believe

[paologallinaharbur]

So far I got it working with the following permissions: @roobre

rules:
  - apiGroups: [""]
    resources:
      - "secrets"
    verbs: ["get", "update", "patch"]
    resourceNames: [{{ template "newrelic-infra-operator.fullname" . }}-config]
  - apiGroups: [""]
    resources:
      - "secrets"
    verbs: ["create"]
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["clusterrolebindings"]
    verbs: ["list", "watch", "get"]

I am using the different client:

Client: client.New(mgr.GetConfig(), client.Options{Scheme: mgr.GetScheme(), Mapper: mgr.GetRESTMapper()})

With the new method to reduce the size of the cache was still showing the same behaviour, not sure why

[roobre]

That ruleset is looking great to me!

[invidian]

Shouldn't it be more like the following:

rules:
  # To create license secret for infrastructure agent sidecar.
  - apiGroups: [""]
    resources:
      - "secrets"
    verbs: ["get", "update", "patch", "create"]
    resourceNames: [{{ template "newrelic-infra-operator.fullname" . }}-config]
  # To enable ClusterRoleBinding resources caching.
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["clusterrolebindings"]
    verbs: ["list", "watch", "get"]
  # To allow granting extra service accounts access to node metrics.
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["clusterrolebindings"]
    verbs: ["update", "patch"]
    resourceNames: [{{ template "newrelic-infra-operator.fullname" . }}-config]

[invidian]

So, what do we do with this issue? We can close it with proposed RBAC rules are added to the Helm chart?

[roobre]

I think @paologallinaharbur commented that create for secrets cannot be restricted to resourceNames, is this still the case?

[invidian]

I think @paologallinaharbur commented that create for secrets cannot be restricted to resourceNames, is this still the case?

You're right: kubernetes/kubernetes#80295 (comment).

This is working as designed. Authorization is based on the request path, not the content of the request body. That means resourceNames is not usable with the create verb. See https://kubernetes.io/docs/reference/access-authn-authz/rbac/

[paologallinaharbur]

yes that is why I had a separate create, I would say we can close this once with Helm chart is enforcing these (or any working) permissions

[invidian]

yes that is why I had a separate create, I would say we can close this once with Helm chart is enforcing these (or any working) permissions

We should have a comment for that I think. It may not be obvious for newcomers.

[invidian]

Documented in newrelic/helm-charts@7dfd2e6.