failed in the docker-env-setup experiment

[jokerwenxiao]

I referred to docs/docker-env-setup. md to deploy nydus, but when I reached the step 8, the following exception occurred:

root@nydus:~/nydus# docker run -d --restart=always -p 5000:5000 registry
Unable to find image 'registry:latest' locally
8a60daaa55ab: Download complete
386cdae4ba70: Download complete
ff1857193a0b: Download complete
docker: failed to prepare extraction snapshot "extract-983914117-nQvy sha256:cc2447e1835a40530975ab80bb1f872fbab0f2a0faecf2ab16fbbb89b3589438": connection error: desc = "transport: Error while dialing dial unix:///run/containerd-nydus/containerd-nydus-grpc.sock: timeout": unavailable.


root@nydus:~/nydus# systemctl status nydus-snapshotter
× nydus-snapshotter.service - nydus snapshotter
     Loaded: loaded (/etc/systemd/system/nydus-snapshotter.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2023-11-13 09:05:20 CST; 1h 14min ago
   Main PID: 1314 (code=exited, status=1/FAILURE)
        CPU: 20ms

11月 13 09:05:19 nydus systemd[1]: nydus-snapshotter.service: Main process exited, code=exited, status=1/FAILURE
11月 13 09:05:19 nydus systemd[1]: nydus-snapshotter.service: Failed with result 'exit-code'.
11月 13 09:05:20 nydus systemd[1]: nydus-snapshotter.service: Scheduled restart job, restart counter is at 6.
11月 13 09:05:20 nydus systemd[1]: Stopped nydus snapshotter.
11月 13 09:05:20 nydus systemd[1]: nydus-snapshotter.service: Start request repeated too quickly.
11月 13 09:05:20 nydus systemd[1]: nydus-snapshotter.service: Failed with result 'exit-code'.
11月 13 09:05:20 nydus systemd[1]: Failed to start nydus snapshotter.

version info:

root@nydus:~/nydus# nydusd --version
Version:        v2.2.4
Git Commit:     1c9c819942ce6fb0b1ebf178df0b3966021ae6bb
Build Time:     2023-11-02T11:32:06.442899984Z
Profile:        release
Rustc:          rustc 1.66.1 (90743e729 2023-01-10)

root@nydus:~/nydus# containerd-nydus-grpc --version
Version:     v0.13.3
Revision:    0dfc6a45217592e3ac7071634cd8e82ecb458eba
Go version:  go1.19.6
Build time:  2023-10-19T06:12:58
root@nydus:~/nydus# uname -r
6.2.0-36-generic
root@nydus:~/nydus# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

docker info:

root@nydus:~/nydus# docker info
Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 24.0.7
 Storage Driver: nydus
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.0-36-generic
 Operating System: Ubuntu 22.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.781GiB
 Name: nydus
 ID: 5debbfdd-e095-4023-a4fb-cff9ffecb3f0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

config:

root@nydus:~/nydus# cat /etc/docker/daemon.json
{
  "features": {
    "containerd-snapshotter": true
  },
  "storage-driver": "nydus"
}

root@nydus:~/nydus# cat /etc/nydus/config.toml
version = 1
# Snapshotter's own home directory where it stores and creates necessary resources
root = "/var/lib/containerd-nydus"
# The snapshotter's GRPC server socket, containerd will connect to plugin on this socket
address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
daemon_mode = "dedicated"
# Whether snapshotter should try to clean up resources when it is closed
cleanup_on_close = false

[system]
# Snapshotter's debug and trace HTTP server interface
enable = true
# Unix domain socket path where system controller is listening on
address = "/run/containerd-nydus/system.sock"

[system.debug]
# Snapshotter can profile the CPU utilization of each nydusd daemon when it is being started.
# This option specifies the profile duration when nydusd is downloading and uncomproessing data.
daemon_cpu_profile_duration_secs = 5
# Enable by assigning an address, empty indicates pprof server is disabled
pprof_address = ""

[daemon]
# Specify a configuration file for nydusd
nydusd_config = "/etc/nydus/nydusd-config.fusedev.json"
nydusd_path = "/usr/local/bin/nydusd"
nydusimage_path = "/usr/local/bin/nydus-image"
# fusedev or fscache
fs_driver = "fusedev"
# How to process when daemon dies: "none", "restart" or "failover"
recover_policy = "restart"
# Nydusd worker thread number to handle FUSE or fscache requests, [0-1024].
# Setting to 0 will use the default configuration of nydusd.
threads_number = 4
# Log rotation size for nydusd, in unit MB(megabytes)
log_rotation_size = 100

[cgroup]
# Whether to use separate cgroup for nydusd.
enable = true
# The memory limit for nydusd cgroup, which contains all nydusd processes.
# Percentage is supported as well, please ensure it is end with "%".
# The default unit is bytes. Acceptable values include "209715200", "200MiB", "200Mi" and "10%".
memory_limit = ""

[log]
# Print logs to stdout rather than logging files
log_to_stdout = false
# Snapshotter's log level
level = "info"
log_rotation_compress = true
log_rotation_local_time = true
# Max number of days to retain logs
log_rotation_max_age = 7
log_rotation_max_backups = 5
# In unit MB(megabytes)
log_rotation_max_size = 100

[metrics]
# Enable by assigning an address, empty indicates metrics server is disabled
address = ":9110"

[remote]
convert_vpc_registry = false

[remote.mirrors_config]
# Snapshotter will overwrite daemon's mirrors configuration
# if the values loaded from this driectory are not null before starting a daemon.
# Set to "" or an empty directory to disable it.
#dir = "/etc/nydus/certs.d"

[remote.auth]
# Fetch the private registry auth by listening to K8s API server
enable_kubeconfig_keychain = false
# synchronize `kubernetes.io/dockerconfigjson` secret from kubernetes API server with specified kubeconfig (default `$KUBECONFIG` or `~/.kube/config`)
kubeconfig_path = ""
# Fetch the private registry auth as CRI image service proxy
enable_cri_keychain = false
# the target image service when using image proxy
#image_service_address = "/run/containerd/containerd.sock"

[snapshot]
# Let containerd use nydus-overlayfs mount helper
enable_nydus_overlayfs = false
# Insert Kata Virtual Volume option to `Mount.Options`
enable_kata_volume = false
# Whether to remove resources when a snapshot is removed
sync_remove = false

[cache_manager]
# Disable or enable recyclebin
disable = false
# How long to keep deleted files in recyclebin
gc_period = "24h"
# Directory to host cached files
cache_dir = ""

[image]
public_key_file = ""
validate_signature = false

# The configuraions for features that are not production ready
[experimental]
# Whether to enable stargz support
enable_stargz = false
# Whether to enable referrers support
# The option enables trying to fetch the Nydus image associated with the OCI image and run it.
# Also see https://github.com/opencontainers/distribution-spec/blob/main/spec.md#listing-referrers
enable_referrer_detect = false
[experimental.tarfs]
# Whether to enable nydus tarfs mode. Tarfs is supported by:
# - The EROFS filesystem driver since Linux 6.4
# - Nydus Image Service release v2.3
enable_tarfs = false
# Mount rafs on host by loopdev and EROFS
mount_tarfs_on_host = false
# Only enable nydus tarfs mode for images with `tarfs hint` label when true
tarfs_hint = false
# Maximum of concurrence to converting OCIv1 images to tarfs, 0 means default
max_concurrent_proc = 0
# Mode to export tarfs images:
# - "none" or "": do not export tarfs
# - "layer_verity_only": only generate disk verity information for a layer blob
# - "image_verity_only": only generate disk verity information for all blobs of an image
# - "layer_block": generate a raw block disk image with tarfs for a layer
# - "image_block": generate a raw block disk image with tarfs for an image
# - "layer_block_with_verity": generate a raw block disk image with tarfs for a layer with dm-verity info
# - "image_block_with_verity": generate a raw block disk image with tarfs for an image with dm-verity info
export_mode = ""
root@nydus:~/nydus# cat /etc/nydus/nydusd-config.json
{
  "device": {
    "backend": {
      "type": "registry",
      "config": {
        "timeout": 5,
        "connect_timeout": 5,
        "retry_limit": 2
      }
    },
    "cache": {
      "type": "blobcache"
    }
  },
  "mode": "direct",
  "digest_validate": false,
  "iostats_files": false,
  "enable_xattr": true,
  "fs_prefetch": {
    "enable": true,
    "threads_count": 8,
    "merging_size": 1048576,
    "prefetch_all": true
  }
}


root@nydus:~/nydus# cat /etc/containerd/config.toml
disabled_plugins = ["cri"]

#root = "/var/lib/containerd"
#state = "/run/containerd"
#subreaper = true
#oom_score = 0

#[grpc]
#  address = "/run/containerd/containerd.sock"
#  uid = 0
#  gid = 0

#[debug]
#  address = "/run/containerd/debug.sock"
#  uid = 0
#  gid = 0
#  level = "info"


# Plug nydus snapshotter into containerd
[proxy_plugins]
  [proxy_plugins.nydus]
    type = "snapshot"
    address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

how to solve this problem？

[imeoer]

It seems the nydus snapshotter process has exited exceptionally, could you paste the snapshotter logs?

[adamqqqplay]

@jokerwenxiao Have you completed step 5?
If completed, try restarting the containerd and dockerd services.

sudo systemctl restart containerd
sudo systemctl restart docker
sudo systemctl restart nydus-snapshotter

[jokerwenxiao]

@imeoer It looks like the configuration file cannot be found.

root@nydus:/var/log# /usr/local/bin/containerd-nydus-grpc --config /etc/nydus/config.toml
INFO[2023-11-13T11:03:06.544513712+08:00] Start nydus-snapshotter. Version: v0.13.3, PID: 104187, FsDriver: fusedev, DaemonMode: dedicated
INFO[2023-11-13T11:03:06.545149122+08:00] parsed cgroup config: cgroup.Config{MemoryLimitInBytes:-1}
INFO[2023-11-13T11:03:06.545180181+08:00] cgroup mode: unified
INFO[2023-11-13T11:03:06.545233042+08:00] root subtree controllers: [cpuset cpu io memory pids]
INFO[2023-11-13T11:03:06.545514117+08:00] create cgroup (v2) successful, controllers: [cpuset cpu io memory pids]
FATA[2023-11-13T11:03:06.545564042+08:00] failed to start nydus-snapshotter             error="failed to initialize snapshotter: load daemon configuration: read FUSE configuration file /etc/nydus/nydusd-config.fusedev.json: open /etc/nydus/nydusd-config.fusedev.json: no such file or directory"

[jokerwenxiao]

@jokerwenxiao Have you completed step 5? If completed, try restarting the containerd and dockerd services.

sudo systemctl restart containerd
sudo systemctl restart docker
sudo systemctl restart nydus-snapshotter

now, the containerd conf is :

root@nydus:/var/log# cat /etc/containerd/config.toml

#disabled_plugins = ["cri"]
disabled_plugins = []


#root = "/var/lib/containerd"
#state = "/run/containerd"
#subreaper = true
#oom_score = 0

#[grpc]
#  address = "/run/containerd/containerd.sock"
#  uid = 0
#  gid = 0

#[debug]
#  address = "/run/containerd/debug.sock"
#  uid = 0
#  gid = 0
#  level = "info"


version = 2

# Plug nydus snapshotter into containerd
[proxy_plugins]
  [proxy_plugins.nydus]
    type = "snapshot"
    address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

but nydus-snapshotter service can not running

[jokerwenxiao]

change step 2 command wget -O /etc/nydus/nydusd-config.json https://raw.githubusercontent.com/containerd/nydus-snapshotter/"$TAG"/misc/snapshotter/nydusd-config.fusedev.json to wget -O /etc/nydus/nydusd-config.fusedev.json https://raw.githubusercontent.com/containerd/nydus-snapshotter/"$TAG"/misc/snapshotter/nydusd-config.fusedev.json can solve it.

but there is a new problem:

root@nydus:/var/log# docker run -d --restart=always -p 5000:5000 registry
Unable to find image 'registry:latest' locally
8a60daaa55ab: Download complete
386cdae4ba70: Download complete
ff1857193a0b: Download complete
87a69098c0a9: Download complete
96526aa774ef: Download complete
afc17120a9f7: Download complete
e5ac04f3acf5: Download complete
834bccaa730c: Download complete
docker: Error response from daemon: failed to mount : mkdir /var/lib/docker/rootfs/nydus/8739f226034652af794388cdd231cc57b7648880426b0daaf70cd387d2db23c6: no such file or directory.

[jokerwenxiao]

change step 2 command wget -O /etc/nydus/nydusd-config.json https://raw.githubusercontent.com/containerd/nydus-snapshotter/"$TAG"/misc/snapshotter/nydusd-config.fusedev.json to wget -O /etc/nydus/nydusd-config.fusedev.json https://raw.githubusercontent.com/containerd/nydus-snapshotter/"$TAG"/misc/snapshotter/nydusd-config.fusedev.json can solve it.

but there is a new problem:

root@nydus:/var/log# docker run -d --restart=always -p 5000:5000 registry
Unable to find image 'registry:latest' locally
8a60daaa55ab: Download complete
386cdae4ba70: Download complete
ff1857193a0b: Download complete
87a69098c0a9: Download complete
96526aa774ef: Download complete
afc17120a9f7: Download complete
e5ac04f3acf5: Download complete
834bccaa730c: Download complete
docker: Error response from daemon: failed to mount : mkdir /var/lib/docker/rootfs/nydus/8739f226034652af794388cdd231cc57b7648880426b0daaf70cd387d2db23c6: no such file or directory.

exec mkdir -p /var/lib/docker/rootfs/nydus can solve it.

[adamqqqplay]

change step 2 command wget -O /etc/nydus/nydusd-config.json https://raw.githubusercontent.com/containerd/nydus-snapshotter/"$TAG"/misc/snapshotter/nydusd-config.fusedev.json to wget -O /etc/nydus/nydusd-config.fusedev.json https://raw.githubusercontent.com/containerd/nydus-snapshotter/"$TAG"/misc/snapshotter/nydusd-config.fusedev.json can solve it.
but there is a new problem:

root@nydus:/var/log# docker run -d --restart=always -p 5000:5000 registry
Unable to find image 'registry:latest' locally
8a60daaa55ab: Download complete
386cdae4ba70: Download complete
ff1857193a0b: Download complete
87a69098c0a9: Download complete
96526aa774ef: Download complete
afc17120a9f7: Download complete
e5ac04f3acf5: Download complete
834bccaa730c: Download complete
docker: Error response from daemon: failed to mount : mkdir /var/lib/docker/rootfs/nydus/8739f226034652af794388cdd231cc57b7648880426b0daaf70cd387d2db23c6: no such file or directory.

exec mkdir -p /var/lib/docker/rootfs/nydus can solve it.

@jokerwenxiao Good job! Can you submit a PR in docs/docker-env-setup.md to update your solution? Thanks in advance.