Remote containers >= 0.238.2 breaks proxy usage

[sveint]

• VSCode Version: 1.69.2
• Local OS Version: Ubuntu 22.04
• Remote OS Version: RHEL 8.6, Docker version 20.10.7
• Remote Extension/Connection Type: SSH + Containers

Steps to Reproduce:

1. Create a devcontainer.json with a Dockerfile including RUN steps depending on internet access.
2. Use Reopen in container on a machine that only allows internet via proxy.
3. Building the image times out on RUN step relying on internet access.

Upgrading the remote containers extension completely broke our development workflow, as our development server has a mandatory proxy for internet access. After debugging for a bit, I think I've narrowed it down to usage of docker buildx build rather than the normal docker build. In both cases we could use Buildkit for building, so I'm not sure why this change is necessary. Using docker build with Buildkit enabled works, as Docker uses the system-wide Docker proxy configuration that is already in place, even with Buildkit.

I've googled extensively and from what I can see there is currently no workaround possible for making Buildkit pick up a systemwide proxy configuration via docker buildx build. One can add parameters or a config to Buildkit on the docker buildx create command, but that is not used in the vscode workflow as far as I can see. I don't think we can configure that one either in devcontainer.json.

As far as I understand, remote container extension is now completely broken for all users that has to use a proxy for internet access. Downgrading to 0.234.0 fixes the issue, but I'm concerned how long that version will be supported.

Is it a hard requirement to switch to docker buildx build rather than DOCKER_BUILDKIT=1 docker build or can it be reverted until a workaround is in place?

[sveint]

@chrmarti Hi, sorry for bumping this, but do you have any workaround for us on this one? Our workflow is still broken after the extension got upgraded and I'm concerned how long we can stay on 0.234.0.

[chrmarti]

We are tracking this on the CLI side as devcontainers/cli#185. No ETA yet, sorry.

[sveint]

We are tracking this on the CLI side as devcontainers/cli#185. No ETA yet, sorry.

@chrmarti Thanks for your reply. Apologies if I'm misunderstanding, but to me it seems like that issue is unrelated (?). My issue is with how Buildkit itself is invoked, not using nodejs to download external resources. I.e. it's the RUN step building a Dockerfile that is failing.

I suspect the devcontainers repo referenced will also encounter the same problem when building an image from scratch (even with a fix for the issue you referenced applied), but I didn't dig too deep into the code so I might be mistaken.

That said, I see there's a fix coming in devcontainers/cli, so I'm very happy if I'm wrong.

[eitsupi]

FYI, I use Devcontainer on a daily basis behind a proxy and have no proxy issues.
I am just doing the proxy configuration to the Docker daemon and containers as is commonly done.
(The problems I have been facing lately due to corporate proxies etc. are #6961 and #7150)

[sveint]

FYI, I use Devcontainer on a daily basis behind a proxy and have no proxy issues. I am just doing the proxy configuration to the Docker daemon and containers as is commonly done. (The problems I have been facing lately due to corporate proxies etc. are #6961 and #7150)

@eitsupi Thank you, that's promising. What Docker version are you using? Also, what proxy configuration have you done?

[Jondeen]

@chrmarti Thanks for your reply. Apologies if I'm misunderstanding, but to me it seems like that issue is unrelated (?). My issue is with how Buildkit itself is invoked, not using nodejs to download external resources. I.e. it's the RUN step building a Dockerfile that is failing.

I suspect the devcontainers repo referenced will also encounter the same problem when building an image from scratch (even with a fix for the issue you referenced applied), but I didn't dig too deep into the code so I might be mistaken.

That said, I see there's a fix coming in devcontainers/cli, so I'm very happy if I'm wrong.

To elaborate on what @sveint said:
The problem at hand here is not the VSCode container plugin being unable to communicate through its Node API, but the lack of integration of system wide proxy definitions from buildx. While docker build pulls in the proxy definitions in ~/.docker/config.json, docker xbuild has no such inherent environment inheritance. Additionally, there is an issue with userns not being correctly forwarded (which matters particularly with rootless setups) in buildx embedded in Docker <v20.10.13.

One solution is to setup a dedicated proxy-aware buildx builder to be used for VSCode image building, with something akin to:

proxy="proxy-host:proxy-port"
docker buildx create --use \
    --platform linux/amd64 \
    --platform linux/amd64/v2 \
    --platform linux/amd64/v3 \
    --platform linux/amd64/v4 \
    --platform linux/386 \
    --driver-opt env.HTTP_PROXY="$proxy" \
    --driver-opt env.HTTPS_PROXY="$proxy" \
    --driver-opt env.http_proxy="$proxy" \
    --driver-opt env.https_proxy="$proxy"

*Catch-all capitalisation, not including proxy-ignore statements

As mentioned requires Docker version >20.10.13 (which additionally corrects buildx handling of rootless setups, through correct forwarding of the --userns argument). Alternatively, buildx can be bumped in the user environment by:
id=$(docker create docker/buildx-bin /) && docker cp $id:/buildx ~/.docker/cli-plugins/docker-buildx && docker rm -v $id

Fixing this on the VSCode side would require there to be an option of disabling docker buildx and keep the old docker build-way of doing things.

I hope this clarifies things.

[eitsupi]

@sveint Only daemon settings that can be edited with the sudo systemctl edit docker command and ~/.docker/config.json.
I am using Docker CE on Ubuntu and the version is the same as yours.

This is the first time I have heard of buildx ignoring ~/.docker/config.json I rather use docker buildx bake as an alternative of docker compose build because compose V2 ignores ~/.docker/config.json. (docker/compose#8797 (comment))

[Jondeen]

@eitsupi Could you confirm that this is a setup using rootless docker?

[eitsupi]

@Jondeen Sorry, I have never used rootless mode, so if you mean that it is a problem specific to rootless mode, that makes sense for me.

[Jondeen]

No worries - there are several overlapping issues here I think...

A minimal example to reproduce these errors, under a squid proxy protecting 80 and 443 (proxy envs defined in .docker/config.json), using rootless Docker 20.10.7 (so ping doesn't work on buildx docker/buildx#887 ):

$ cat Dockerfile
from debian

workdir /tmp

$ docker build --progress=plain -f Dockerfile .
#1 [internal] load build definition from Dockerfile
#1 sha256:5d311af47efeb6fed9117273fd8bbd9a0005e2ca72ee33805b6f6fac211429d0
#1 transferring dockerfile: 39B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:18ccd329ad6597ad6a5eb6ed648ca034346a0557dfa47860ed926261878d294d
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/debian:latest
#3 sha256:e0024f5d107f06a67d6d91876c0f23e77ddb32c643d8736d0f211a49a879a246
#3 DONE 0.5s

#4 [1/2] FROM docker.io/library/debian@sha256:e538a2f0566efc44db21503277c7312a142f4d0dedc5d2886932b92626104bff
#4 sha256:e2e11577d97225630ca5382350a991ba08214f7ce3aba07f77a4cc749436a47a
#4 DONE 0.0s

#5 [2/2] WORKDIR /tmp
#5 sha256:a27af2e874ddd56865c768d966deeeb4059d71949e08ef861f017a448d1eeee3
#5 CACHED

#6 exporting to image
#6 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#6 exporting layers done
#6 writing image sha256:f884aaa3ad8ca802e33379c31c6e2105ac1801e91125addca3c2328f242c0568 done
#6 DONE 0.0s

$ docker buildx build --load --progress=plain -f Dockerfile .
#1 [internal] booting buildkit
#1 sha256:6bbb03885fff96b42d928be3f4ab55b6e0fb232f74bdeab79e2eb489db6220a8
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 1.3s done
#1 creating container buildx_buildkit_gifted_chatelet0
#1 creating container buildx_buildkit_gifted_chatelet0 0.5s done
#1 ERROR: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: write sysctl key net.ipv4.ping_group_range: write /proc/sys/net/ipv4/ping_group_range: invalid argument: unknown
------
 > [internal] booting buildkit:
------
error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: write sysctl key net.ipv4.ping_group_range: write /proc/sys/net/ipv4/ping_group_range: invalid argument: unknown

Upgrading buildx:

$ id=$(docker create docker/buildx-bin /) && docker cp $id:/buildx ~/.docker/cli-plugins/docker-buildx && docker rm -v $id
$ docker buildx build --load --progress=plain -f Dockerfile .
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 63B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/debian:latest
<HANGS>

Adding proxies:

$ proxy="proxy-host:proxy-port"
$ docker buildx create --use \
    --platform linux/amd64 \
    --platform linux/amd64/v2 \
    --platform linux/amd64/v3 \
    --platform linux/amd64/v4 \
    --platform linux/386 \
    --driver-opt env.HTTP_PROXY="$proxy" \
    --driver-opt env.HTTPS_PROXY="$proxy" \
    --driver-opt env.http_proxy="$proxy" \
    --driver-opt env.https_proxy="$proxy"

$ docker buildx build --load --progress=plain -f Dockerfile .
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 1.3s done
#1 creating container buildx_buildkit_zen_grothendieck0
#1 creating container buildx_buildkit_zen_grothendieck0 0.9s done
#1 DONE 2.2s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load build definition from Dockerfile
#3 transferring dockerfile: 63B done
#3 DONE 0.0s

#4 [internal] load metadata for docker.io/library/debian:latest
#4 DONE 1.6s

#5 [1/2] FROM docker.io/library/debian@sha256:e538a2f0566efc44db21503277c7312a142f4d0dedc5d2886932b92626104bff
#5 resolve docker.io/library/debian@sha256:e538a2f0566efc44db21503277c7312a142f4d0dedc5d2886932b92626104bff done
#5 sha256:f606d8928ed378229f2460b94b504cca239fb906efc57acbdf9340bd298d5ddf 2.10MB / 55.05MB 0.2s
#5 sha256:f606d8928ed378229f2460b94b504cca239fb906efc57acbdf9340bd298d5ddf 32.51MB / 55.05MB 0.3s
#5 sha256:f606d8928ed378229f2460b94b504cca239fb906efc57acbdf9340bd298d5ddf 55.05MB / 55.05MB 0.5s done
#5 extracting sha256:f606d8928ed378229f2460b94b504cca239fb906efc57acbdf9340bd298d5ddf
#5 extracting sha256:f606d8928ed378229f2460b94b504cca239fb906efc57acbdf9340bd298d5ddf 2.3s done
#5 DONE 2.8s

#6 [2/2] WORKDIR /tmp
#6 DONE 0.3s

#7 exporting to oci image format
#7 exporting layers 0.0s done
#7 exporting manifest sha256:412442369bc130751c32db55eefd7e50f551011c9ab3c9d6b44e6b87a15d94fe done
#7 exporting config sha256:737bf9b02fda35d8af9c0bd4d977d8b02db38af408f6308ec491d45732a2b267 done
#7 sending tarball
#7 sending tarball 0.4s done
#7 DONE 0.5s

#8 importing to docker
#8 DONE 0.0s

[Jondeen]

To give a brief update,

By using the host docker builder instance instead of a docker-container builder (called default on my system), and the updated buildx, everything works smoothly, and there is no need to create a new builder:

$ docker buildx ls
NAME/NODE         DRIVER/ENDPOINT                    STATUS  BUILDKIT PLATFORMS
silly_rosalind    docker-container
  silly_rosalind0 unix:///run/user/xxxxx/docker.sock stopped
default *         docker
  default         default                            running 20.10.7  linux/amd64, linux/386

[sveint]

As Jondeen mentioned, this is fixed with latest buildx. Seems like the issue was fixed in this PR: docker/buildx#959 . When Googling initially, all the resources I found said that one had to do this manually by passing --build-args to buildx, but it seems like that information is out of date.

Closing issue as there's nothing for vscode to fix here. Thanks a lot for all the assistance.