Make the grafana operator able to create grafana service accounts and tokens from yaml

[MickeHedlund]

Is your feature request related to a problem? Please describe.
In cases where the grafana instances and grafana operator was removed from the cluster and had to be set up again, we had to manually create all the service accounts and tokens again and update the token everywhere it was being used. This is because today the service accounts can only be created in the grafana gui or by calling the http api endpoint. But that generates a new token automatically and has to be done as a manual step after deployment.

Describe the solution you'd like
I would like to be able to include a yaml file with Kind: GrafanaServiceAccount or similar, where I could specify the name of the service account, scope, name of token and set the token to what I want it to be, if not set it could create a token automatically when token name is set. The operator reads the objects and creates the service accounts in relevant grafana. Could use selectorLabels just like dashboards to get to the correct grafana instance etc.

Describe alternatives you've considered
I have concidered writing tools that uses the http api to create them all automatically for us, but that still results in a new auto generated token that has to be replaced everywhere its used. And has to be done as an extra step after deployment.

[HVBE]

Thanks, @MickeHedlund this sounds like a great suggestion, however we'd like to have a design document around this prior to implementation. Our main concerns here are:

1. The ephemerality of the default configuration of our Grafana instance, means that after a pod restart the serviceaccount would no longer be present (the token would change due to reprovisioning). This means that enabling ServiceAccounts for the Grafana instance would add an extra requirement to provision PersistentVolumes to host the sqlite database for the instance, or any other persistence, such as an external postgres.
2. How to share the serviceaccount token, there might be a few choices, however the default we'd go for is a k8s Secret
3. Security-wise, we'd like to ensure the provisioned service account can't be attached to a Grafana instance that it doesn't belong to, this would require some logic to ensure cross-namespace SA sharing doesn't, by default, extend itself, outside of where it's supposed to be created
4. How to handle token expiration when using seconds to live, does the operator recreate the SA with a new token? etc.

[jkroepke]

I would like to share an alternatives which I recently discuss in grafana Slack

Instead setup and maintain grafana service accounts, use the existing Kubernetes Service Account architecture could be are more archivable goal to reach. Pushing this would be a great solution for all non-operator deployments, too.

This concept works already great together with AWS IRSA; GCP Workload Identity and Azure Workload Identity

The token of Kubernetes service account has a builtin rotation and is JWT/OIDC compliant. The Kubernetes API provides a JWKS endpoint which contains public key for the service account signature.

Grafana nativly support auth.jwt, but some features are missing on context for kubernetes, ref: grafana/grafana#81794

By using Kubernetes Service Accounts, some points can be resolved:

1. The ephemerality of the default configuration of our Grafana instance, means that after a pod restart the serviceaccount would no longer be present (the token would change due to reprovisioning). This means that enabling ServiceAccounts for the Grafana instance would add an extra requirement to provision PersistentVolumes to host the sqlite database for the instance, or any other persistence, such as an external postgres.

auth.jwt can be staticly configured and survives restarts.

---

2. How to share the serviceaccount token, there might be a few choices, however the default we'd go for is a k8s Secret

The kubernetes service account token is always present inside the container. There is no need to sync secrets. No additional logic mandatory.

---

3. Security-wise, we'd like to ensure the provisioned service account can't be attached to a Grafana instance that it doesn't belong to, this would require some logic to ensure cross-namespace SA sharing doesn't, by default, extend itself, outside of where it's supposed to be created

This can be problematic. auth.jwt supports expect_claims which can be used to restrict service accounts, e.g.:

expect_claims='{"sub": "system:serviceaccount:grafana-operator:grafana-operator"}'

but wildcards wont work:

expect_claims='{"sub": "system:serviceaccount:namespace:*"}'

---

4. How to handle token expiration when using seconds to live, does the operator recreate the SA with a new token? etc.

service account token rotation is a builtin feature.

---

The good thing is that service account tokens are not longer stored inside secrets. It's really hard to steal such short-living tokens.

[danlz]

Kubernetes service accounts are nice, but if you use external Grafana (ie. Amazon Managed Grafana) you have to use Grafana service accounts.