[Bug] secureJsonData don't work in 5.8.1

[smuda]

Describe the bug
After upgrading to operator 5.8.1 from 5.8.0, grafana cannot fetch metrics from thanos anymore.

Version
Affected: 5.8.1
Not affected: 5.8.0.

To Reproduce

• Install operator 5.8.1.
• Create an sa grafana-instance-sa-token and give it access to thanos.
• Install grafana:

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana-instance
  labels:
    dashboards: grafana
spec:
  serviceAccount:
    metadata:
      annotations:
        serviceaccounts.openshift.io/oauth-redirectreference.primary: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"grafana-instance-route"}}'
  route:
    spec:
      host: {{ .Values.grafana.hostname }}
      port:
        targetPort: https
      tls:
        termination: reencrypt
      to:
        kind: Service
        name: grafana-instance-service
        weight: 100
      wildcardPolicy: None
  deployment:
    spec:
      template:
        spec:
          volumes:
            - name: grafana-tls
              secret:
                secretName: grafana-tls
            - name: grafana-proxy
              secret:
                secretName: grafana-proxy
            - name: ocp-injected-certs
              configMap:
                name: ocp-injected-certs
          containers:
            - args:
                - '-provider=openshift'
                - '-pass-basic-auth=false'
                - '-https-address=:9091'
                - '-http-address='
                - '-email-domain=*'
                - '-upstream=http://localhost:3000'
                - '-openshift-sar={"resource": "namespaces", "verb": "get"}'
                - '-openshift-delegate-urls={"/": {"resource": "namespaces", "verb": "get"}}'
                - '-tls-cert=/etc/tls/private/tls.crt'
                - '-tls-key=/etc/tls/private/tls.key'
                - '-client-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token'
                - '-cookie-secret-file=/etc/proxy/secrets/session_secret'
                - '-openshift-service-account=grafana-instance-sa'
                - '-openshift-ca=/etc/pki/tls/cert.pem'
                - '-openshift-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
                - '-openshift-ca=/etc/proxy/certs/ca-bundle.crt'
                - '-skip-auth-regex=^/metrics'
              image: 'quay.io/openshift/origin-oauth-proxy'
              name: grafana-proxy
              ports:
                - containerPort: 9091
                  name: https
              resources:
                requests:
                  cpu: 10m
                  memory: 100Mi
                limits:
                  memory: 100Mi
              volumeMounts:
                - mountPath: /etc/tls/private
                  name: grafana-tls
                  readOnly: false
                - mountPath: /etc/proxy/secrets
                  name: grafana-proxy
                  readOnly: false
                - mountPath: /etc/proxy/certs
                  name: ocp-injected-certs
                  readOnly: false
  service:
    metadata:
      annotations:
        service.beta.openshift.io/serving-cert-secret-name: grafana-tls
    spec:
      ports:
        - name: https
          port: 9091
          protocol: TCP
          targetPort: https
  client:
    preferIngress: false
  config:
    log:
      mode: "console"
    auth.anonymous:
      enabled: "True"
    auth:
      disable_login_form: "False"
      disable_signout_menu: "True"
    auth.basic:
      enabled: "True"
    auth.proxy:
      enabled: "True"
      enable_login_token: "True"
      header_property: "username"
      header_name: "X-Forwarded-User"
    security:
      admin_user: "admin"
      {{- if .Values.grafana.secretPassword }}
      admin_password: {{ .Values.grafana.secretPassword | quote }}
      {{- else }}
      admin_password: {{ randAlphaNum 24 | quote }}
      {{- end }}
    users:
      auto_assign_org_role: Editor

Add a datasource for thanos:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: {{ .Release.Name }}-thanos
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  datasource:
    access: proxy
    basicAuth: false
    editable: true
    isDefault: true
    jsonData:
      httpHeaderName1: 'Authorization'
      timeInterval: 5s
      tlsSkipVerify: true
    secureJsonData:
      httpHeaderValue1: 'Bearer ${token}'
    name: Prometheus
    orgId: 1
    type: prometheus
    url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'
  valuesFrom:
    - targetPath: "secureJsonData.httpHeaderValue1"
      valueFrom:
        secretKeyRef:
          name: grafana-instance-sa-token
          key: token

Expected behavior
Grafana should be able to fetch metrics from thanos.

Suspect component/Location where the bug might be occurring
unknown

Screenshots
If applicable, add screenshots to help explain your problem.

Runtime (please complete the following information):
OKD 4.15.0-0.okd-2024-03-10-010116
grafana-operator 4.8.1
grafana 9.2.10

[smuda]

The error returned from thanos is 403 and a web page for authentication. When I redirect the traffic to a netcat the Authorization: Bearer header is missing so the error from thanos is reasonable. :-)

Curling the grafana /api/datasources does not show anything suspicious.

[smuda]

I'm going to QCon for the next week but I'll try to answer anything.

[pb82]

@theSuess getDatasourceContent returns now a models.Datasource from here: https://github.com/grafana/grafana-openapi-client-go/blob/main/models/data_source.go#L19

This type does not have a secureJsonData field, but we're still relying on it. Is secureJsonData no longer used / is there an alternative for it?

Or should we deserialize into to type here?