Getting "error while resolving symlinks in base directory" when using subpath

[reynhartono]

Description

Getting the following error when using the subpath feature on a CIFS volume with configured uid and gid.

Error response from daemon: error while resolving symlinks in base directory "/var/lib/docker/volumes/volume_name/_data": lstat /var/lib/docker/volumes/volume_name/_data: stale NFS file handle

Reproduce

1. Create a named volume with uid and gid

volumes:
  a_cifs_volume:
    driver: local
    driver_opts:
      type: cifs
      o: username=username,password=password,uid=1000,gid=1000
      device: //hostname/share

services:
  busybox:
    image: busybox
    container_name: busybox
    volumes:
      - type: volume 
        source: a_cifs_volume
        target: /subpath
        volume:
          subpath: subpath

2. Run docker compose up -d

The following docker-compose worked (without using subpath):

volumes:
  a_cifs_volume:
    driver: local
    driver_opts:
      type: cifs
      o: username=username,password=password,uid=1000,gid=1000
      device: //hostname/share/subpath

services:
  busybox:
    image: busybox
    container_name: busybox
    volumes:
      - type: volume 
        source: a_cifs_volume
        target: /subpath

Note: I'm using SMB share from Windows 10 machine.

Expected behavior

No response

docker version

Client: Docker Engine - Community
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:17:48 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:17:48 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1
    Path:     /home/username/.docker/cli-plugins/docker-compose

Server:
 Containers: 12
  Running: 7
  Paused: 0
  Stopped: 5
 Images: 12
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-101-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 7.752GiB
 Name: 
 ID: 
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

[vvoland]

I can't reproduce it on my side with a smb server hosted by smbd 4.18.9.

Are you sure this is a Docker specific issue, and not an issue with your specific NFS server configuration?

[reynhartono]

Thanks for reviewing this @vvoland. I just found that this only happened when I'm setting the gid and uid option on a SMB share (not NFS). I just updated the steps to reproduce.

[vvoland]

Yeah sorry I meant smb but the error message poisoned my brain's cache and wrote NFS instead 😅
The gid and uid part also works for me fine.

Wondering why the error mentions NFS, when you're using SMB though.

[reynhartono]

Maybe it's specific to Windows SMB Share. But it's kinda strange. I can make it work without using subpath. But once the subpath option is used, it's not working anymore.

[vvoland]

Anyway, with SMB I don't think you really need the subpath volume opt, you can just specify the directory directly in the share address like //hostname/share/subpath

[reynhartono]

I know, as I mentioned in the post, it's working fine by specifying the full path in the named volume (and that's my workaround for now). But, since I have multiple directories to be mounted on different containers, I think using this subpath feature would be more efficient.