Rootless Docker Install: Failed to start docker.service #47653

HanGuo97 · 2024-03-30T02:36:54Z

HanGuo97
Mar 30, 2024

Hi, I'm trying to install Rootless Docker on a shared cluster (to work with Slurm). However, I repeatedly ran into Failed to start docker.service issue. This persists after a fresh installation of Docker (after uninstalling Docker). Any help would be very valuable!

$ dockerd-rootless-setuptool.sh install
[WARNING] File already exists, skipping: /afs/.../.config/systemd/user/docker.service
[INFO] starting systemd service docker.service
+ systemctl --user start docker.service
Failed to start docker.service: Unit docker.service not found.
+ set +x
[ERROR] Failed to start docker.service. Run `journalctl -n 20 --no-pager --user --unit docker.service` to show the error log.
[ERROR] Before retrying installation, you might need to uninstall the current setup: `/usr/bin/dockerd-rootless-setuptool.sh uninstall -f ; /usr/bin/rootlesskit rm -rf /afs/.../.local/share/docker`
No journal files were opened due to insufficient permissions.

AkihiroSuda · 2024-03-30T20:12:17Z

AkihiroSuda
Mar 30, 2024
Maintainer

Please try running dockerd-rootless.sh directly to get the failure logs

0 replies

HanGuo97 · 2024-03-31T01:37:41Z

HanGuo97
Mar 31, 2024
Author

Thanks! I have attached the logs below. What do you think the problem is?

INFO[2024-03-30T21:35:26.953915368-04:00] Starting up
WARN[2024-03-30T21:35:26.954018027-04:00] Running in rootless mode. This mode has feature limitations.
INFO[2024-03-30T21:35:26.954022823-04:00] Running with RootlessKit integration
INFO[2024-03-30T21:35:26.959594745-04:00] containerd not running, starting managed containerd
INFO[2024-03-30T21:35:26.961526603-04:00] started new containerd process                address=/run/user/26489/docker/containerd/containerd.sock module=libcontainerd pid=266709
INFO[2024-03-30T21:35:27.000547812-04:00] starting containerd                           revision=ae07eda36dd25f8a1b98dfbf587313b99c0190bb version=1.6.28
INFO[2024-03-30T21:35:27.042593585-04:00] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
INFO[2024-03-30T21:35:27.048479851-04:00] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2024-03-30T21:35:27.048613355-04:00] loading plugin "io.containerd.snapshotter.v1.btrfs"...  type=io.containerd.snapshotter.v1
INFO[2024-03-30T21:35:27.049239516-04:00] skip loading plugin "io.containerd.snapshotter.v1.btrfs"...  error="path /afs/home/.local/share/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (afs) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2024-03-30T21:35:27.049308744-04:00] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
WARN[2024-03-30T21:35:27.049346407-04:00] failed to load plugin io.containerd.snapshotter.v1.devmapper  error="devmapper not configured"
INFO[2024-03-30T21:35:27.049379955-04:00] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2024-03-30T21:35:27.049495822-04:00] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2024-03-30T21:35:27.139853080-04:00] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2024-03-30T21:35:27.140485966-04:00] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  error="path /afs/home/.local/share/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2024-03-30T21:35:27.140547603-04:00] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2024-03-30T21:35:27.140639198-04:00] could not use snapshotter devmapper in metadata plugin  error="devmapper not configured"
INFO[2024-03-30T21:35:27.140671584-04:00] metadata content store policy set             policy=shared
INFO[2024-03-30T21:35:27.143463801-04:00] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2024-03-30T21:35:27.143540474-04:00] loading plugin "io.containerd.event.v1.exchange"...  type=io.containerd.event.v1
INFO[2024-03-30T21:35:27.143593767-04:00] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2024-03-30T21:35:27.143673841-04:00] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2024-03-30T21:35:27.143860165-04:00] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2024-03-30T21:35:27.144000115-04:00] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
INFO[2024-03-30T21:35:27.144104836-04:00] loading plugin "io.containerd.warning.v1.deprecations"...  type=io.containerd.warning.v1
INFO[2024-03-30T21:35:27.144145653-04:00] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2024-03-30T21:35:27.144847195-04:00] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2024-03-30T21:35:27.144938494-04:00] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2024-03-30T21:35:27.144979762-04:00] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2024-03-30T21:35:27.145021150-04:00] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2024-03-30T21:35:27.145063462-04:00] loading plugin "io.containerd.service.v1.introspection-service"...  type=io.containerd.service.v1
INFO[2024-03-30T21:35:27.145103173-04:00] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2024-03-30T21:35:27.145139741-04:00] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2024-03-30T21:35:27.145177265-04:00] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2024-03-30T21:35:27.145214841-04:00] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2024-03-30T21:35:27.145273012-04:00] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2024-03-30T21:35:27.145316126-04:00] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2024-03-30T21:35:27.145373745-04:00] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2024-03-30T21:35:27.145410163-04:00] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2024-03-30T21:35:27.145445174-04:00] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2024-03-30T21:35:27.145485566-04:00] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2024-03-30T21:35:27.145521858-04:00] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2024-03-30T21:35:27.145558902-04:00] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2024-03-30T21:35:27.145597893-04:00] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2024-03-30T21:35:27.145635972-04:00] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2024-03-30T21:35:27.145676124-04:00] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2024-03-30T21:35:27.145711320-04:00] loading plugin "io.containerd.tracing.processor.v1.otlp"...  type=io.containerd.tracing.processor.v1
INFO[2024-03-30T21:35:27.145753275-04:00] skip loading plugin "io.containerd.tracing.processor.v1.otlp"...  error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
INFO[2024-03-30T21:35:27.145801787-04:00] loading plugin "io.containerd.internal.v1.tracing"...  type=io.containerd.internal.v1
ERRO[2024-03-30T21:35:27.145884275-04:00] failed to initialize a tracing processor "otlp"  error="no OpenTelemetry endpoint: skip plugin"
INFO[2024-03-30T21:35:27.146003060-04:00] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2024-03-30T21:35:27.146106705-04:00] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2024-03-30T21:35:27.146465676-04:00] serving...                                    address=/run/user/26489/docker/containerd/containerd-debug.sock
INFO[2024-03-30T21:35:27.146575721-04:00] serving...                                    address=/run/user/26489/docker/containerd/containerd.sock.ttrpc
INFO[2024-03-30T21:35:27.146671734-04:00] serving...                                    address=/run/user/26489/docker/containerd/containerd.sock
INFO[2024-03-30T21:35:27.146726840-04:00] containerd successfully booted in 0.147479s
ERRO[2024-03-30T21:35:28.153024476-04:00] failed to mount overlay: invalid argument     storage-driver=overlay2
ERRO[2024-03-30T21:35:28.153347426-04:00] exec: "fuse-overlayfs": executable file not found in $PATH  storage-driver=fuse-overlayfs
ERRO[2024-03-30T21:35:28.153612726-04:00] No zfs dataset found for root                 backingFS=unknown root=/afs/home/.local/share/docker storage-driver=zfs
INFO[2024-03-30T21:35:28.154321188-04:00] Loading containers: start.
WARN[2024-03-30T21:35:28.163100435-04:00] Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: could not insert 'br_netfilter': Operation not permitted
insmod /lib/modules/5.4.0-174-generic/kernel/net/bridge/br_netfilter.ko
, error: exit status 1
INFO[2024-03-30T21:35:28.163200206-04:00] skipping firewalld management for rootless mode
INFO[2024-03-30T21:35:28.527292858-04:00] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[2024-03-30T21:35:28.714044294-04:00] Loading containers: done.
WARN[2024-03-30T21:35:28.729525415-04:00] WARNING: No cpu cfs quota support
WARN[2024-03-30T21:35:28.729583120-04:00] WARNING: No cpu cfs period support
WARN[2024-03-30T21:35:28.729600814-04:00] WARNING: No cpu shares support
WARN[2024-03-30T21:35:28.729614840-04:00] WARNING: No cpuset support
WARN[2024-03-30T21:35:28.729627712-04:00] WARNING: No io.weight support
WARN[2024-03-30T21:35:28.729643996-04:00] WARNING: No io.weight (per device) support
WARN[2024-03-30T21:35:28.729659126-04:00] WARNING: No io.max (rbps) support
WARN[2024-03-30T21:35:28.729674708-04:00] WARNING: No io.max (wbps) support
WARN[2024-03-30T21:35:28.729691572-04:00] WARNING: No io.max (riops) support
WARN[2024-03-30T21:35:28.729707475-04:00] WARNING: No io.max (wiops) support
WARN[2024-03-30T21:35:28.729723788-04:00] WARNING: bridge-nf-call-iptables is disabled
WARN[2024-03-30T21:35:28.729740499-04:00] WARNING: bridge-nf-call-ip6tables is disabled
INFO[2024-03-30T21:35:28.729788331-04:00] Docker daemon                                 commit=8b79278 containerd-snapshotter=false storage-driver=vfs version=26.0.0
INFO[2024-03-30T21:35:28.729986769-04:00] Daemon has completed initialization
INFO[2024-03-30T21:35:29.491115425-04:00] API listen on /run/user/26489/docker.sock

1 reply

AkihiroSuda Mar 31, 2024
Maintainer

This seems an issue in dockerd-rootless-setuptool.sh.

The daemon itself seems working, so docker -H unix:///run/user/26489/docker.sock run ... should work.

AkihiroSuda · 2024-03-31T06:41:25Z

AkihiroSuda
Mar 31, 2024
Maintainer

/afs/.../.config/systemd/user/docker.service

This path seems unusual.
Please try symlink it to ~/.config/systemd/user/docker.service

1 reply

HanGuo97 Mar 31, 2024
Author

This path seems unusual.

Ah, the path is essentially home directory in AFS (I used ... just to remove a long string of user names etc, sorry for the confusion).

This seems an issue in dockerd-rootless-setuptool.sh.
The daemon itself seems working, so docker -H unix:///run/user/26489/docker.sock run ... should work.

Could you elaborate on the differences between dockerd-rootless-setuptool.sh. and dockerd-rootless.sh? What would I miss if I just do the latter even though the former throws an error? For context, I'm primarily interested in using rootless Docker with slurm.

AkihiroSuda · 2024-03-31T16:09:21Z

AkihiroSuda
Mar 31, 2024
Maintainer

AFS

This is unlikely to work.
You'll have to mount an ext4 or XFS under ~/.local/share/docker.

differences between dockerd-rootless-setuptool.sh. and dockerd-rootless.sh

The former sets up systemd for running the latter.

0 replies

HanGuo97 · 2024-03-31T16:55:29Z

HanGuo97
Mar 31, 2024
Author

Sorry for the confusion. In my case, the home directory ~ is in AFS (hence ~/.local/share/docker is /afs/.../.local/share/docker).

Are you suggesting that AFS is the problem? (And that I should install rootless Docker in non-AFS directory?)

2 replies

AkihiroSuda Mar 31, 2024
Maintainer

AFS is the problem

The systemctl issue below might be irrelevant to AFS, but it is unlikely to work anyway when ~/.local/share/docker is a "strange" filesystem

+ systemctl --user start docker.service
Failed to start docker.service: Unit docker.service not found.

HanGuo97 Mar 31, 2024
Author

Understood, thanks again! I will try installing this in a non-AFS directory.

HanGuo97 · 2024-03-31T18:20:37Z

HanGuo97
Mar 31, 2024
Author

Hmm, I tried HOME=/scratch/.../docker dockerd-rootless-setuptool.sh install (/scratch/ is non-AFS) but the problem didn't go away.

1 reply

AkihiroSuda Mar 31, 2024
Maintainer

Please try if systemctl --user start works with non-Docker service. If it does not work, it is an issue of systemd setup.

HanGuo97 · 2024-04-04T21:51:03Z

HanGuo97
Apr 4, 2024
Author

Hi, sorry for the belated reply. I have tried to start a different service, and few other fixes. In the end, it seems like placing docker.service in /etc/systemd/user (via sudo) instead of ~/.config/systemd/user/docker.service would work.

However, re-running the dockerd-rootless-setuptool.sh again ran into another problem:

[INFO] Creating [path removed for brevity]
[INFO] starting systemd service docker.service
+ systemctl --user start docker.service
Job for docker.service failed because the control process exited with error code.
See "systemctl --user status docker.service" and "journalctl --user -xe" for details.
+ set +x
[ERROR] Failed to start docker.service. Run `journalctl -n 20 --no-pager --user --unit docker.service` to show the error log.
[ERROR] Before retrying installation, you might need to uninstall the current setup: `/usr/bin/dockerd-rootless-setuptool.sh uninstall -f ; /usr/bin/rootlesskit rm -rf /afs/csail.mit.edu/u/h/hanguo/.local/share/docker`
No journal files were opened due to insufficient permissions.

Following the suggestion, I ran the journalctl command,

$ journalctl -n 20 --no-pager --user --unit docker.service
Hint: You are currently not seeing messages from the system.
      Users in groups 'adm', 'systemd-journal' can see all messages.
      Pass -q to turn off this notice.
No journal files were opened due to insufficient permissions.

Adding sudo,

$ sudo journalctl -n 20 --no-pager --user --unit docker.service
No journal files were found.

If this is helpful, here's the status output

$ systemctl --user --no-pager --full status docker.service
● docker.service - Docker Application Container Engine (Rootless)
     Loaded: loaded (/etc/xdg/systemd/user/docker.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2024-04-04 17:43:09 EDT; 7min ago
       Docs: https://docs.docker.com/go/rootless/
    Process: 1705080 ExecStart=/usr/bin/dockerd-rootless.sh (code=exited, status=1/FAILURE)
   Main PID: 1705080 (code=exited, status=1/FAILURE)
        CPU: 379ms

Do you have any insights regarding how I should proceed from that? Thanks again for the help!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rootless Docker Install: Failed to start docker.service #47653

{{title}}

Replies: 7 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Rootless Docker Install: Failed to start docker.service #47653

HanGuo97 Mar 30, 2024

Replies: 7 comments · 5 replies

AkihiroSuda Mar 30, 2024 Maintainer

HanGuo97 Mar 31, 2024 Author

AkihiroSuda Mar 31, 2024 Maintainer

AkihiroSuda Mar 31, 2024 Maintainer

HanGuo97 Mar 31, 2024 Author

AkihiroSuda Mar 31, 2024 Maintainer

HanGuo97 Mar 31, 2024 Author

AkihiroSuda Mar 31, 2024 Maintainer

HanGuo97 Mar 31, 2024 Author

HanGuo97 Mar 31, 2024 Author

AkihiroSuda Mar 31, 2024 Maintainer

HanGuo97 Apr 4, 2024 Author

HanGuo97
Mar 30, 2024

Replies: 7 comments 5 replies

AkihiroSuda
Mar 30, 2024
Maintainer

HanGuo97
Mar 31, 2024
Author

AkihiroSuda Mar 31, 2024
Maintainer

AkihiroSuda
Mar 31, 2024
Maintainer

HanGuo97 Mar 31, 2024
Author

AkihiroSuda
Mar 31, 2024
Maintainer

HanGuo97
Mar 31, 2024
Author

AkihiroSuda Mar 31, 2024
Maintainer

HanGuo97 Mar 31, 2024
Author

HanGuo97
Mar 31, 2024
Author

AkihiroSuda Mar 31, 2024
Maintainer

HanGuo97
Apr 4, 2024
Author