Failed to query provider Client sent an HTTP request to an HTTPS server with SAML Enterprise App Auth using Dex implementation

[wlawton]

Hi. I've followed these instructions to integrate ArgoCD with Azure AD for SSO purposes. When I click the 'Login via Saml' button on the ArgoCD UI login page, I get the response Failed to query provider "https://argocd.dev.xxxx.com/api/dex": 400 Bad Request: Client sent an HTTP request to an HTTPS server (i've masked the hostname). ArgoCD is deployed into a Kubernetes cluster thats running in AWS cloud. The UI ingress is facilitated by nginx-application-controller which creates an internal AWS NLB. SSL connections are terminated at the NLB so connections to ArgoCD are made using HTTP i.e. the ArgoCD ingress resource is configured with the following:

  nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
  nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
  external-dns.alpha.kubernetes.io/hostname: argocd.dev.xxxx.com

Ingress spec looks like:

spec:
rules:
- host: argocd.dev.xxxx.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: argocd-server
port:
number: 80

Some potentially significant configuration from the nginx-ingress-controller:

annotations:
  service.beta.kubernetes.io/aws-load-balancer-type: nlb
  service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
  service.beta.kubernetes.io/aws-load-balancer-internal: "true"
  
# Enable ports on load-balancer
enableHttp: false
enableHttps: true
  
# Force the traffic between the load balancer and the ingress proxy to the http port
# TLS termination on load balancer
targetPorts:
  http: http
  https: http

The argocd-dex-server logs indicate its listening for an https connection:

{"level":"info","msg":"listening (telemetry) on 0.0.0.0:5558","time":"2023-06-19T12:44:17Z"}
{"level":"info","msg":"listening (https) on 0.0.0.0:5556","time":"2023-06-19T12:44:17Z"}
{"level":"info","msg":"listening (grpc) on 0.0.0.0:5557","time":"2023-06-19T12:44:17Z"}

But the dex-server container spec is configured as follows, so bit of a contradiction between (https) above and - name: http below.

containers:
- name: dex-server
image: '.dkr.ecr.af-south-1.amazonaws.com/argocd-dex-server:v2.35.3'
command:
- /shared/argocd-dex
- rundex
ports:
- name: http
containerPort: 5556
protocol: TCP
- name: grpc
containerPort: 5557
protocol: TCP
- name: metrics
containerPort: 5558
protocol: TCP

argocd-server logs the following when the SSO button is pushed:

time="2023-06-19T12:45:18Z" level=info msg="Initializing OIDC provider (issuer: https://argocd.dev.xxxx.com/api/dex)"
time="2023-06-19T12:45:34Z" level=info msg="finished streaming call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = no session information" grpc.code=Unauthenticated grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2023-06-19T12:45:34Z" grpc.time_ms=5.163 span.kind=server system=grpc
time="2023-06-19T12:45:34Z" level=info msg="finished streaming call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = no session information" grpc.code=Unauthenticated grpc.method=WatchResourceTree grpc.service=application.ApplicationService grpc.start_time="2023-06-19T12:45:34Z" grpc.time_ms=6.314 span.kind=server system=grpc

[wlawton]

I think i've found the solution for this. A little dig into the argocd code turned up this line in argocd_dex.go:

command.Flags().BoolVar(&disableTLS, "disable-tls", env.ParseBoolFromEnv("ARGOCD_DEX_SERVER_DISABLE_TLS", false), "Disable TLS on the HTTP endpoint")

And sure enough, configuring that environment variable in the argocd values.yaml resolved the issue i was having i.e.

dex:
  env:
    - name: ARGOCD_DEX_SERVER_DISABLE_TLS
      value: "true"

And if i'd found this documentation earlier it would also have been a big help: https://argo-cd.readthedocs.io/en/stable/operator-manual/tls/#configuring-inbound-tls-for-argocd-dex-server

[heechankim]

Just in case anyone is currently experiencing the same issue.

If you want to set the ARGOCD_DEX_SERVER_DISABLE_TLS environment variable to true,
you can set it via the argocd-cmd-params-cm ConfigMap.

Helm Values

configs:
  params:
    server.insecure: true
    dexserver.disable.tls: true

[sjoukedv]

For reference; I am also seeing this error when the connectors[].config.caData does not have valid base64 input (as indicated by the dex logs)

  dex.config: |
                connectors:
                  - type: saml
                    id: aws_sso
                    name: AWS SSO
                    config:
                      caData: <invalid-b64-string>