Docker v26.0.2 fails to create containers on gitpod (Error: open /proc/sys/net/ipv6/conf/eth0/disable_ipv6: read-only file system)

[joeabbey]

Description

Error: open /proc/sys/net/ipv6/conf/eth0/disable_ipv6: read-only file system

Reproduce

Even with the attempted workaround:

docker run --sysctl=net.ipv6.conf.all.disable_ipv6=1 ubuntu

Output:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: failed to add interface vethc71242a to sandbox: error setting interface "vethc71242a" IPv6 to <nil>: failed to configure ipv6: failed to disable IPv6 forwarding for container's interface eth0: open /proc/sys/net/ipv6/conf/eth0/disable_ipv6: read-only file system: unknown.
ERRO[0005] error waiting for container: context canceled 

Expected behavior

No response

docker version

Client: Docker Engine - Community
 Version:           26.0.2
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        3c863ff
 Built:             Thu Apr 18 16:27:07 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.2
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       7cef0d9
  Built:            Thu Apr 18 16:27:07 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 gitpod:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfcb
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    26.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 2
 Server Version: 26.0.2
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: gitpod io.containerd.runc.v2 runc
 Default Runtime: gitpod
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.9-0-gccaecfcb
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.75-060175-generic
 Operating System: Ubuntu 22.04.4 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.79GiB
 Name: dakotaxyz-dakota-2w017zk6yxn
 ID: 29bd3cc6-8cd0-43ce-a49c-d76bc53fd5e3
 Docker Root Dir: /workspace/.docker-root
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

This is running within a gitpod, and so there may be another layer in the middle.

[thaJeztah]

cc @robmry @akerouanton

[thaJeztah]

Oh, also 👋 @joeabbey ! (just now spotted who opened this ticket) hope you're doing well!

[joeabbey]

I'm almost certain there's additional isolation at play preventing the container from being able to disable the IPV6 forwarding... so My best guess is that the fix will be on @gitpod-io's side to allow this procfs path.

[robmry]

It's not clear exactly what's happening in gitpod's environment ... the daemon must have most of CAP_NET_ADMIN in order to set up networks (at least the default bridge) and iptables, which would normally mean /proc/sys/net is writable. In this case there's no write access to at-least /proc/sys/net/ipv6/conf/eth0/disable_ipv6 but the file exists.

In any case, we need to provide an escape hatch ...

• The daemon shouldn't try to write the file if it doesn't need to change.
• If the daemon needs to disable IPv6 but can't - error, but offer an override (either a daemon flag or an env var).
• If the daemon needs to enable IPv6 but can't - just log.
  • Could error - the user configured IPv6 on a network and we can't set it up. So, the user should either not configure IPv6, enable it on the docker host by default, or open up write access to the config. But, the existing code only logs if there's no disable_ipv6 file (IPv6 disabled in the kernel) - it's probably safest to stick with that, missing-IPv6 is broken, but fairly obviously so.
• In a near-future release, once we've made it easy to enable IPv6 without having to configure IPAM addresses etc, remove the override - at that point, the user must explicitly enable IPv6 if the daemon can't disable it on an interface.

[goncalossilva]

There is a similar problem when using docker under Jip-Hop/jailmaker. The report here is the same as in the OP. I can also reproduce this. I've downgraded to 26.0.1 for now.

[robmry]

There is a similar problem when using docker under Jip-Hop/jailmaker. The report here is the same as in the OP. I can also reproduce this. I've downgraded to 26.0.1 for now.

Hi @goncalossilva - thank you for the links ... the plan I outlined should give you a way to work around the problem.

I've also tried to repro with 'jailmaker' in order to try to understand the issue a bit better. But, no luck. Probably because I don't have a NAS, so had to persuade jailmaker to run on an Ubuntu VM.

For me - with jailmaker's deprecated docker_compatibility option, or with the template docker config, /proc/sys/net ends up mounted rw in the jail, and docker works normally.

The docker_compatibility option uses systemd-nspawn --capabilities=all. The template uses a private network, so systemd-nspawn itself adds CAP_NET_ADMIN, which is what makes /proc/sys/net read-write.

So - how are you running and configuring jailmaker? If you run mount inside the jail, does it show /proc/sys/net mounted as ro or rw? (When creating the jail, jailmaker prints the systemd command it's using - if you have that, it'd be interesting to see.)

[goncalossilva]

Hey, thanks for the thorough response! I'll try my best to help. :)

I'm running under the old docker_compatibility option. The whole configuration looks like this:

startup=1
docker_compatible=1
seccomp=0
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
systemd_nspawn_user_args=--bind='/mnt/rpool/some-mount-point:/mnt/some-mount-point'
systemd_run_default_args=--property=KillMode=mixed
    --property=Type=notify
    --property=RestartForceExitStatus=133
    --property=SuccessExitStatus=133
    --property=Delegate=yes
    --property=TasksMax=infinity
    --collect
    --setenv=SYSTEMD_NSPAWN_LOCK=0
systemd_nspawn_default_args=--keep-unit
    --quiet
    --boot
    --bind-ro=/sys/module
    --inaccessible=/sys/module/apparmor

seccomp was added later, it doesn't seem to make a difference.

The final command appears to be:

systemd-run --property=KillMode=mixed --property=Type=notify --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=Delegate=yes --property=TasksMax=infinity --collect --setenv=SYSTEMD_NSPAWN_LOCK=0 --unit=jlmkr-docker --working-directory=./jails/docker '--description=My nspawn jail docker [created with jailmaker]' --setenv=SYSTEMD_SECCOMP=0 -- systemd-nspawn --keep-unit --quiet --boot --bind-ro=/sys/module --inaccessible=/sys/module/apparmor --machine=docker --directory=rootfs --capability=all --bind=/mnt/rpool/some-mount-point:/mnt/some-mount-point

Despite including --capability=all, I hit the same exact error as OP. Can I provide more information?

[goncalossilva]

If you run mount inside the jail, does it show /proc/sys/net mounted as ro or rw?

On 26.0.1, it is mounted as read-only:

root@docker:~# mount | grep /proc/sys
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)

Yet, it works. Should I update to 26.0.2 (or 26.1.0) to try and see if it's any different?

[robmry]

Thank you, that's interesting - I ended up with this, which would explain the difference ...

proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)

(In older versions, we'd only log if we couldn't disable IPv6 on an interface. So, if the rest of CAP_NET_ADMIN was working and docker could create its network/iptables, we wouldn't have noticed the issue with a read-only /proc/sys/net.)

Is /proc/sys/net read-write on the host where you're running the jail?

[goncalossilva]

Yes, it appears to be read-only on TrueNAS SCALE:

# ls -la /proc/sys/net
total 0
dr-xr-xr-x 1 root root 0 Apr 26 01:27 .
dr-xr-xr-x 1 root root 0 Apr 26 01:27 ..
dr-xr-xr-x 1 root root 0 Apr 26 01:55 bridge
dr-xr-xr-x 1 root root 0 Apr 26 19:29 core
dr-xr-xr-x 1 root root 0 Apr 26 01:55 ipv4
dr-xr-xr-x 1 root root 0 Apr 26 19:29 ipv6
dr-xr-xr-x 1 root root 0 Apr 26 19:29 mptcp
dr-xr-xr-x 1 root root 0 Apr 26 19:29 netfilter
-rw-r--r-- 1 root root 0 Apr 26 19:29 nf_conntrack_max
dr-xr-xr-x 1 root root 0 Apr 26 19:29 unix

[robmry]

Ahh - right! Thank you, I think that explains things.

[jaedmunt]

There is a similar problem when using docker under Jip-Hop/jailmaker. The report here is the same as in the OP. I can also reproduce this. I've downgraded to 26.0.1 for now.

@goncalossilva How can I downgrade Docker version? We had no issues with our workspace last night, and then today we build it and get the error:

0.412 runc run failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: failed to add interface vethd92accd to sandbox: error setting interface "vethd92accd" IPv6 to <nil>: failed to configure ipv6: failed to disable IPv6 forwarding for container's interface eth0: open /proc/sys/net/ipv6/conf/eth0/disable_ipv6: read-only file system. 

when a dockerfile attempts RUN pip install...

Any guidance on using a downgraded version of docker would be welcome. We'll have to switch to another platform soon if this conflict can't be fixed or if we don't have a workaround.

[robmry]

Hi @jaedmunt - there are instructions for various supported Linux distros at https://docs.docker.com/engine/install/ ... the page for each platform has a "Specific version" tab in section "Install Docker Engine".

[goncalossilva]

@jaedmunt if you're using jailmaker and are on Debian bookwork, run this from the jail:

apt-get install docker-ce=5:26.0.1-1~debian.12~bookworm docker-ce-cli=5:26.0.1-1~debian.12~bookworm

Also, make sure to disable unattended upgrades for the moment (dpkg-reconfigure unattended-upgrades) if you have them enabled. Otherwise, things will break every few hours.

[jaedmunt]

Thank you, both, much appreciated!