Possible regression in client credential validation/handling?

[adriankostrubiak-tomtom]

When using the more recent buildkit docker images to build a container via buildctl-daemonless.sh with a remote buildkitd server, connections have begun failing where they previously did not with errors like:

error: failed to dial "tcp://buildkit.my-server.com:443" . make sure buildkitd is running: ClientConn's authority from transport creds "buildkit.my-server.com" and dial option "buildkit.my-server.com:443" don't match

---

We use the buildctl client, coupled with a custom cacert file, to perform in-cloud builds of local content with a hosted buildkitd server.

A sample command of how we run such a build is:

docker run \
    -it \
    --rm \
    --privileged \
    -v "$(pwd):/tmp/work" \
    -v /tmp/build/certificates/:/tmp/crt \
    -v /tmp/build/buildkitcache/:/var/lib/buildkit \
    -v /tmp/build/out/:/tmp/out \
    -v ~/.docker:/root/.docker \
    --entrypoint buildctl-daemonless.sh \
    moby/buildkit:v0.9.3 \
        --addr tcp://buildkit.my-server:443 \
        --tlscacert /tmp/crt/certificate.crt \
        build \
        --frontend  dockerfile.v0 \
        --opt source=docker.io/docker/dockerfile:1.3.0 \
        --opt "filename=Dockerfile" \
        --local context=/tmp/work \
        --local dockerfile=/tmp/work/build \
        --output type=docker,name=my-app,dest=/tmp/out/my-app.tar 

The above command succeeds, as expected. Specifically, that is with

moby/buildkit:v0.9.3 d6c89b7085b106301645ddcc77cf64eb7b705ab507b72d52d130ac33f1300417

When we change the docker image being to the following,

moby/buildkit:master  05d46af706739de88ab39733bb66753aedbd8d44178c7ae521111876828fe340

The build fails as noted earlier.

The request flow here is

| client, with custom cacert | --> | ingress, tls terminiation | --> | builkitd tcp server |

The client is supplying a cacert that is valid for the certificate presented by the ingress.

Finally, I'm not sure when this stopped working, but certainly as of today, Dec 8, 2021

[tonistiigi]

cc @everpeace (via #1567)

[everpeace]

Hi, thanks for pinging. I looked into the issue.

I found google.golang.org/grpc@v1.42.0 (updated in #2481) introduced better handling for :authority pseudo-header implemented in grpc/grpc-go#4817. This requires WithAuthority dial option value must match with the server name of transport creds when specified.

The current implementation always uses url.Host ("host:port") of --addr as an authority pseudo-header value. This is the root cause.

I posted #2518 for the fix.