Use Cloudflare for Ingress

[junoriosity]

I want to install kubeapps on my Kubernetes cluster. For this, I first install Pinniper conierge and supervisor like this, although I do some modifications to the certs by using Cloudflare (the following is written in Terraform code, but the YAML files should be readily seen and whenever something appears like count = var.is_main_cluster ? 1 : 0, this is only means that this resource is only deployed to the main cluster):

resource "kubernetes_secret" "cloudflare-secret" {
  count = var.is_main_cluster ? 1 : 0

  depends_on = [ kubectl_manifest.pinniped_supervisor ]

  metadata {
    name = "cloudflare-tls"
    namespace = "pinniped-supervisor"
  }

  data = {
    "tls.crt" = file("${var.cloudflare_cert_path}/origin-ca.crt")
    "tls.key" = file("${var.cloudflare_cert_path}/origin-ca.pk")
  }

  type = "kubernetes.io/tls"
}

resource "kubectl_manifest" "federation-domain" {
  count = var.is_main_cluster ? 1 : 0
    yaml_body = <<YAML
apiVersion: config.supervisor.pinniped.dev/v1alpha1
kind: FederationDomain
metadata:
  name: demo-federation-domain
  namespace: pinniped-supervisor
spec:
  # You can choose an arbitrary path for the issuer URL.
  issuer: "https://supervisor.my-domain.com/issuer"
  tls:
    secretName: cloudflare-tls
YAML
}


resource "kubectl_manifest" "auth0_client_credentials" {
  count = var.is_main_cluster ? 1 : 0
    yaml_body = <<YAML
apiVersion: v1
kind: Secret
metadata:
  namespace: pinniped-supervisor
  name: auth0-client-credentials
type: secrets.pinniped.dev/oidc-client
stringData:
  # This should be the app's "Client ID"
  clientID: ${var.auth0_client_id}
  # This should be the app's "Client secret"
  clientSecret: ${var.auth0_client_secret}
YAML
}

resource "kubectl_manifest" "oidc_provider" {
  depends_on = [ kubectl_manifest.auth0_client_credentials ]

  count = var.is_main_cluster ? 1 : 0
    yaml_body = <<YAML
apiVersion: idp.supervisor.pinniped.dev/v1alpha1
kind: OIDCIdentityProvider
metadata:
  namespace: pinniped-supervisor
  name: auth0
spec:
  # This should be the app's "Auth0 Domain".
  issuer: ${var.auth0_domain}/
  authorizationConfig:
    additionalScopes: [groups, email, offline_access]
  claims:
    username: email
    groups: groups
  client:
    secretName: auth0-client-credentials
YAML
}


data "kubectl_file_documents" "pinniped_concierge_crds" {
    content = file("${path.module}/yaml_files/pinniped/install-pinniped-concierge-crds.yaml")
}

resource "kubectl_manifest" "pinniped_concierge_crds" {
    count = length(data.kubectl_file_documents.pinniped_concierge_crds.documents)
    yaml_body = element(data.kubectl_file_documents.pinniped_concierge_crds.documents, count.index)
}

data "kubectl_file_documents" "pinniped_concierge_resources" {
    content = file("${path.module}/yaml_files/pinniped/install-pinniped-concierge-resources.yaml")
}

resource "kubectl_manifest" "pinniped_concierge_resources" {
    depends_on = [ kubectl_manifest.pinniped_concierge_crds ]
    count = length(data.kubectl_file_documents.pinniped_concierge_resources.documents)
    yaml_body = element(data.kubectl_file_documents.pinniped_concierge_resources.documents, count.index)
}

resource "kubectl_manifest" "jwt_authenticator" {
  depends_on = [ kubectl_manifest.pinniped_concierge_resources ]
  
  yaml_body = <<YAML
apiVersion: authentication.concierge.pinniped.dev/v1alpha1
kind: JWTAuthenticator
metadata:
  name: supervisor-jwt-authenticator
spec:
  # This should be the issuer URL that was declared in the FederationDomain.
  issuer: "https://supervisor.my-domain.com/issuer"
  # Replace this with your own unique value.
  audience: ${var.context}-${random_password.audience_append.result}
YAML
}

Now, this works like a charm. But the next thing is to install kubeapps with the helm charts and the following values:

useHelm3: true
ingress:
  enabled: true
  hostname: kubeapps.my-domain.com
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"

authProxy:
  additionalFlags:
    - --oidc-issuer-url=https://my-domain.eu.auth0.com/ # Example: https://antoniogamez.eu.auth0.com/  # mind the last "/" character
  clientID: my-client-id
  clientSecret: my-client-secret
  cookieSecret: bm90LWdvb2Qtc2VjcmV0Cg==
  enabled: true
  provider: oidc

pinnipedProxy:
  enabled: true

clusters:
  - name: cluster1
    apiServiceURL: https://... # impersonation proxy URL
    certificateAuthorityData: ... #  impersonation proxy CA
    pinnipedConfig:
      enabled: true
  - name:cluster2
    apiServiceURL: https://... # impersonation proxy URL
    certificateAuthorityData: ... #  impersonation proxy CA
    pinnipedConfig:
      enabled: true

The things come up, apart from the auth proxy, which says

[2022/08/01 11:11:23] [main.go:60] ERROR: Failed to initialise OAuth2 Proxy: error intiailising provider: could not create provider data: error building OIDC ProviderVerifier: invalid provider verifier options: missing required setting: issuer-url

Do you know how to fix that? Could it be related to Pinniped supervisor?

[castelblanque]

Hi @junoriosity , interesting setup that you are trying, adding Cloudflare and Terraform into the mix 😃

With major release of Kubeapps chart v8.0.0 there were some renames done.
At first sight I would say that the issue might be related to the property authProxy.additionalFlags not existing anymore, but renamed to authProxy.extraFlags. See note in Readme file here.

Please double check that you have all parameters valid by checking the official README.md file.

Hope this helps!

[junoriosity]

Hi @castelblanque,

many thanks for your fast input. I did that and now the domain comes up. When I click on

I see some redirection wheel, and then I am back at the aforementioned Page, so I cannot login.

These are the logs for the auth-proxy

...
192.168.2.3:34874 - badae09224db1af29af8e676f439810f - test@test.com [2022/08/01 14:09:43] kubeapps.my-domain.com GET / "/config.json" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" 304 0 0.024
192.168.1.2:42838 - 54574a4a76fa1a24edb3b46df6848933 - - [2022/08/01 14:09:45] kubeapps.my-domain.com GET - "/oauth2/start" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" 302 318 0.000
192.168.2.3:34874 - 9f5815b3602734af2deff28c83cc197e - test@test.com [2022/08/01 14:09:46] [AuthSuccess] Authenticated via OAuth2: Session{email:test@test.com user: PreferredUsername: token:true id_token:true created:2022-08-01 14:09:46.456170085 +0000 UTC m=+1215.361068689 expires:2022-08-02 14:09:46.343183348 +0000 UTC m=+87615.248081961}
192.168.2.3:34874 - 9f5815b3602734af2deff28c83cc197e - - [2022/08/01 14:09:46] kubeapps.my-domain.com GET - "/oauth2/callback?code=IJmCZUi2VoUUF46AmLFHAysg2hbRNBXg4C8naSwpA5cyA&state=TavkgOfbVozcBzw8EWsb7W4ryH3Ey7YakGpgBSzAKBg%3A%2F" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" 302 24 0.367
192.168.1.2:42838 - 42b472b8c1996b0c02bb3d1b4849dbf9 - test@test.com [2022/08/01 14:10:12] [AuthSuccess] Authenticated via OAuth2: Session{email:test@test.com user: PreferredUsername: token:true id_token:true created:2022-08-01 14:10:12.106140007 +0000 UTC m=+1241.011038610 expires:2022-08-02 14:10:11.999050759 +0000 UTC m=+87640.903949372}
192.168.1.2:42838 - 42b472b8c1996b0c02bb3d1b4849dbf9 - - [2022/08/01 14:10:11] kubeapps.my-domain.com GET - "/oauth2/callback?code=-8MBwsRkCDy77DBZ9sHwGrRj_8YXubPDC3kPo-WSHdpj-&state=JOo6K_lG6SFhP_phE1TPdVKwRA9YIKVUe3vPTP3tCXI%3A%2F" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" 302 24 0.229
192.168.2.3:34874 - 27fbf72be060fc25ab0ebce47a1ddc26 - test@test.com [2022/08/01 14:10:12] kubeapps.my-domain.com POST / "/apis/kubeappsapis.plugins.resources.v1alpha1.ResourcesService/CheckNamespaceExists" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" 200 0 0.118
192.168.2.3:34874 - 47d5b63a9442e11561a082cd750bd0f1 - test@test.com [2022/08/01 14:11:00] kubeapps.my-domain.com POST / "/apis/kubeappsapis.plugins.resources.v1alpha1.ResourcesService/CheckNamespaceExists" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" 200 0 0.982

and these are the logs for the pinniped-proxy:

 INFO  pinniped_proxy > Listening on http://0.0.0.0:3333
 ERROR pinniped_proxy::service > Unauthorized by pinniped: authentication failed

Caused by:
    authentication failed
 INFO  pinniped_proxy::service > GET https://34.141.242.24/api/v1/namespaces/default 401 Unauthorized

It occurs to me as if it is struggling the get the details right with pinniped, but perhaps I am wrong here.

Do you know how to fix that?

[junoriosity]

Hi @castelblanque @antgamdia @absoludity

do you have an idea, how I could fix that? I am happy to provide you with more information, if you consider it to be helpful. 😀

[castelblanque]

Hi @junoriosity, please follow the document for debugging failures in OIDC. Specially if you see a valid token back from OAuth2-proxy, try to place a call directly to K8s APIs with it, just to see what happens when Kubeapps is not in the picture.

Then please check the logs of both the K8s API server and the Pinniped Concierge to get the full view.
For Pinniped, something like:

kubectl logs deployment/pinniped-concierge -n pinniped-concierge -f

If the concierge deployment has multiple replicas, you might want to scale it to 1 and replicate the issue to avoid checking logs for all pods.

[junoriosity]

Hi @castelblanque, many thanks for getting back to me.

I tried to follow the document you provided, but in the debug mode I do not find the errors as outlined.

This is how it looks:

Furthermore, these are the logs of pinniped-concierge

...

{"level":"info","timestamp":"2022-08-08T08:25:49.524667Z","logger":"jwtcachefiller-controller","caller":"go.pinniped.dev/internal/controller/authenticator/jwtcachefiller/jwtcachefiller.go:118$jwtcachefiller.(*controller).Sync","message":"actual jwt authenticator and desired jwt authenticator are the same","jwtAuthenticator":{"name":"supervisor-jwt-authenticator"},"issuer":"https://supervisor.my-domain.com/issuer"}
{"level":"info","timestamp":"2022-08-08T08:27:46.536724Z","caller":"k8s.io/utils@v0.0.0-20220210201930-3a6ce19ff2f9/trace/trace.go:205$trace.(*Trace).logTrace","message":"Trace[1782062799]: \"create\" kind:TokenCredentialRequest (08-Aug-2022 08:27:46.536) (total time: 0ms):\nTrace[1782062799]: ---\"failure\" failureType:token authentication,msg:no such authenticator 0ms (08:27:46.536)\nTrace[1782062799]: [91.373µs] [91.373µs] END\n"}
{"level":"error","timestamp":"2022-08-08T08:28:46.739996Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"kube-cert-agent-controller: { } failed with: could not find a healthy kube-controller-manager pod (0 candidates)\n"}
{"level":"error","timestamp":"2022-08-08T08:28:48.324218Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"kube-cert-agent-controller: { } failed with: could not find a healthy kube-controller-manager pod (0 candidates)\n"}
{"level":"info","timestamp":"2022-08-08T08:28:49.525341Z","logger":"jwtcachefiller-controller","caller":"go.pinniped.dev/internal/controller/authenticator/jwtcachefiller/jwtcachefiller.go:118$jwtcachefiller.(*controller).Sync","message":"actual jwt authenticator and desired jwt authenticator are the same","jwtAuthenticator":{"name":"supervisor-jwt-authenticator"},"issuer":"https://supervisor.my-domain.com/issuer"}
{"level":"info","timestamp":"2022-08-08T08:31:17.173745Z","caller":"k8s.io/utils@v0.0.0-20220210201930-3a6ce19ff2f9/trace/trace.go:205$trace.(*Trace).logTrace","message":"Trace[1553534547]: \"create\" kind:TokenCredentialRequest (08-Aug-2022 08:31:17.173) (total time: 0ms):\nTrace[1553534547]: ---\"failure\" failureType:token authentication,msg:no such authenticator 0ms (08:31:17.173)\nTrace[1553534547]: [47.191µs] [47.191µs] END\n"}
{"level":"error","timestamp":"2022-08-08T08:31:48.325674Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"kube-cert-agent-controller: { } failed with: could not find a healthy kube-controller-manager pod (0 candidates)\n"}
{"level":"info","timestamp":"2022-08-08T08:31:49.618872Z","logger":"jwtcachefiller-controller","caller":"go.pinniped.dev/internal/controller/authenticator/jwtcachefiller/jwtcachefiller.go:118$jwtcachefiller.(*controller).Sync","message":"actual jwt authenticator and desired jwt authenticator are the same","jwtAuthenticator":{"name":"supervisor-jwt-authenticator"},"issuer":"https://supervisor.my-domain.com/issuer"}
{"level":"error","timestamp":"2022-08-08T08:34:48.325978Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"kube-cert-agent-controller: { } failed with: could not find a healthy kube-controller-manager pod (0 candidates)\n"}
{"level":"info","timestamp":"2022-08-08T08:34:49.619604Z","logger":"jwtcachefiller-controller","caller":"go.pinniped.dev/internal/controller/authenticator/jwtcachefiller/jwtcachefiller.go:118$jwtcachefiller.(*controller).Sync","message":"actual jwt authenticator and desired jwt authenticator are the same","jwtAuthenticator":{"name":"supervisor-jwt-authenticator"},"issuer":"https://supervisor.my-domain.com/issuer"}

Do you have some ideas, what else I could do?

[castelblanque]

{"level":"info","timestamp":"2022-08-08T08:31:17.173745Z","caller":"k8s.io/utils@v0.0.0-20220210201930-3a6ce19ff2f9/trace/trace.go:205$trace.(*Trace).logTrace","message":"Trace[1553534547]: "create" kind:TokenCredentialRequest (08-Aug-2022 08:31:17.173) (total time: 0ms):\nTrace[1553534547]: ---"failure" failureType:token authentication,msg:no such authenticator 0ms (08:31:17.173)\nTrace[1553534547]: [47.191µs] [47.191µs] END\n"}
{"level":"error","timestamp":"2022-08-08T08:31:48.325674Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"kube-cert-agent-controller: { } failed with: could not find a healthy kube-controller-manager pod (0 candidates)\n"}

It is hard to tell just by looking at the logs, but it could be a problem with the correct setup of Pinniped itself, unrelated to Kubeapps? See this Pinniped issue with the same no such authenticator error.
Please check that the JWTAuthenticator is valid. One source of problems is the missing CA data. I see in your Terraform script that it does not have the certificateAuthorityData data, so Pinniped might not be trusting your OIDC provider.

Another area to look at is the Pinniped Impersonation proxy. In the clusters: section of Kubeapps values, you need to set the values (URL, CA) from the impersonation proxy, not the actual K8s API server data. This usually applies when using a managed Kubernetes cluster.

[junoriosity]

Hi @cfryanr @antgamdia @castelblanque ,

do you need some more logs or anything here or is that sufficient? 😀

[cfryanr]

Sorry, I'm not familiar enough with Kubeapps. One thing that comes to mind is I see that you're using email as the username claim, so you'll have to make sure that the Kubeapps auth proxy is sending enough params on the authorize request in order to get the email claim in the resulting ID tokens. Not sure how to debug that though. Hopefully the Kubeapps team can offer a suggestion.

[junoriosity]

Hi @cfryanr, still many thanks for all your support so far. 😀

@antgamdia @castelblanque @absoludity What are your thoughts on the current situation? Do you have any idea what I could do to make it work? For instance, the approach to use two JWTAuthenticator sounds quite appealing, yet it does not work.

Hence, if you have some ideas, suggestions or other questions to make it work, that would be more than welcome. 😀

[junoriosity]

Hi @antgamdia @castelblanque @absoludity do you have some ideas how I could resolve that issue? :)

[antgamdia]

@junoriosity , most of the team has been on PTO 🌴, so we haven't had a chance to follow the discussion yet (thanks Ryan for your help here!!).
Please, ping me on slack, so that I can catch up with the issue more quickly :)