Last version `5:25.0.4-1~raspbian.12~bookworm` seems broken on raspberry with armv6l (`Illegal instruction`) #47543

RouxAntoine · 2024-03-08T21:41:28Z

RouxAntoine
Mar 8, 2024

Description

After upgrading my raspbian distribution I get segmentation fault with docker command.

Cf : following section for details and available for more deep dive. Why not helping to made a pull request 🙂

Reproduce

With version 5:25.0.4-1~raspbian.12~bookworm I get an error illegal instruction with segfault:

pi@rasp2:~ $ VERSION='5:25.0.4-1~raspbian.12~bookworm'; sudo -E apt-get install docker-ce=$VERSION
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libevent-2.1-7 libfuse2 libntfs-3g883 libpcre2-posix2 vcdbg
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  docker-ce-cli
Suggested packages:
  cgroupfs-mount | cgroup-lite
The following packages will be upgraded:
  docker-ce docker-ce-cli
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/27.2 MB of archives.
After this operation, 19.5 kB disk space will be freed.
Do you want to continue? [Y/n] Y
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
Reading changelogs... Done
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = "en_US.UTF-8",
        LC_CTYPE = "en_US.UTF-8",
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_GB.UTF-8").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
(Reading database ... 50219 files and directories currently installed.)
Preparing to unpack .../docker-ce-cli_5%3a25.0.4-1~raspbian.12~bookworm_armhf.deb ...
Unpacking docker-ce-cli (5:25.0.4-1~raspbian.12~bookworm) over (5:25.0.3-1~raspbian.12~bookworm) ...
Preparing to unpack .../docker-ce_5%3a25.0.4-1~raspbian.12~bookworm_armhf.deb ...
Unpacking docker-ce (5:25.0.4-1~raspbian.12~bookworm) over (5:25.0.3-1~raspbian.12~bookworm) ...
Setting up docker-ce-cli (5:25.0.4-1~raspbian.12~bookworm) ...
Setting up docker-ce (5:25.0.4-1~raspbian.12~bookworm) ...
Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 145.
Processing triggers for man-db (2.11.2-2) ...
pi@rasp2:~ $ docker ps
Illegal instruction

Expected behavior

With version 5:25.0.3-1~raspbian.12~bookworm docker ps command is ok :

pi@rasp2:~ $ VERSION='5:25.0.3-1~raspbian.12~bookworm'; sudo -E apt-get install docker-ce=$VERSION docker-ce-cli=$VERSION
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
docker-ce is already the newest version (5:25.0.3-1~raspbian.12~bookworm).
The following packages were automatically installed and are no longer required:
  libevent-2.1-7 libfuse2 libntfs-3g883 libpcre2-posix2 vcdbg
Use 'sudo apt autoremove' to remove them.
The following packages will be DOWNGRADED:
  docker-ce-cli
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
Need to get 12.3 MB of archives.
After this operation, 2048 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 https://download.docker.com/linux/raspbian bookworm/stable armhf docker-ce-cli armhf 5:25.0.3-1~raspbian.12~bookworm [12.3 MB]
Fetched 12.3 MB in 12s (1041 kB/s)
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = "en_US.UTF-8",
        LC_CTYPE = "en_US.UTF-8",
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_GB.UTF-8").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
dpkg: warning: downgrading docker-ce-cli from 5:25.0.4-1~raspbian.12~bookworm to 5:25.0.3-1~raspbian.12~bookworm
(Reading database ... 50219 files and directories currently installed.)
Preparing to unpack .../docker-ce-cli_5%3a25.0.3-1~raspbian.12~bookworm_armhf.deb ...
Unpacking docker-ce-cli (5:25.0.3-1~raspbian.12~bookworm) over (5:25.0.4-1~raspbian.12~bookworm) ...
Setting up docker-ce-cli (5:25.0.3-1~raspbian.12~bookworm) ...
Processing triggers for man-db (2.11.2-2) ...
pi@rasp2:~ $ docker ps
CONTAINER ID   IMAGE                              COMMAND                  CREATED        STATUS          PORTS                               NAMES

docker version

information contained in `/proc/cpuinfo`:


pi@rasp2:~ $ cat /proc/cpuinfo 
processor       : 0
model name      : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS        : 697.95
Features        : half thumb fastmult vfp edsp java tls 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xb76
CPU revision    : 7

Hardware        : BCM2835
Revision        : 0010
Serial          : 00000000e4bbe03e
Model           : Raspberry Pi Model B Plus Rev 1.2



### docker info

```bash
There is no docker info because docker command line segfault

Additional Info

Execution with strace:

pi@rasp2:~ $ strace docker ps
execve("/usr/bin/docker", ["docker", "ps"], 0xbec84564 /* 26 vars */) = 0
brk(NULL)                               = 0x225f000
access("/etc/ld.so.preload", R_OK)      = 0
openat(AT_FDCWD, "/etc/ld.so.preload", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=54, ...}) = 0
mmap2(NULL, 54, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) = 0xb6f04000
close(3)                                = 0
readlink("/proc/self/exe", "/usr/bin/docker", 4096) = 15
openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\250\3\0\0004\0\0\0"..., 512) = 512
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=9512, ...}) = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6f02000
mmap2(NULL, 139308, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6edf000
mmap2(0xb6ee0000, 73772, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0xb6ee0000
munmap(0xb6edf000, 4096)                = 0
munmap(0xb6ef3000, 57388)               = 0
mprotect(0xb6ee2000, 61440, PROT_NONE)  = 0
mmap2(0xb6ef1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0xb6ef1000
close(3)                                = 0
munmap(0xb6f04000, 54)                  = 0
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=29999, ...}) = 0
mmap2(NULL, 29999, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb6efa000
close(3)                                = 0
openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\20\346\1\0004\0\0\0"..., 512) = 512
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=1508068, ...}) = 0
mmap2(NULL, 1541036, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6d67000
mmap2(0xb6ed3000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16c000) = 0xb6ed3000
mmap2(0xb6ed6000, 37804, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb6ed6000
close(3)                                = 0
set_tls(0xb6f03080)                     = 0
set_tid_address(0xb6f02be8)             = 3937
set_robust_list(0xb6f02bec, 12)         = 0
rseq(0xb6f03060, 0x20, 0, 0xe7f5def3)   = 0
mprotect(0xb6ed3000, 8192, PROT_READ)   = 0
mprotect(0xb6ef1000, 4096, PROT_READ)   = 0
mprotect(0x136b000, 9019392, PROT_READ) = 0
mprotect(0xb6f2c000, 4096, PROT_READ)   = 0
ugetrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
munmap(0xb6efa000, 29999)               = 0
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPC, si_addr=0x5fd9d0} ---
+++ killed by SIGILL +++
Illegal instruction

piersfinlayson · 2024-03-09T10:20:56Z

piersfinlayson
Mar 9, 2024

I also hit the same problem using Raspberry Pi Zero Ws and Raspberry Pi 1Bs, with 5:25.0.4-1~~raspbian.10~~buster upgraded from 5:25.0.3-1~~raspbian.10~~buster 9which was working fine). Presumably a problem with the latest armhf build.

0 replies

dangersnake · 2024-03-09T17:07:37Z

dangersnake
Mar 9, 2024

Just a note for anyone stuck on this issue.
I'm on rpi zero w and was able to rollback related packages to get things working again

uname

6.1.0-rpi8-rpi-v6 #1 Raspbian 1:6.1.73-1+rpt1 (2024-01-25) armv6l GNU/Linux

lsb_release

Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm

I used this to rollback

apt install docker-ce=5:25.0.3-1~raspbian.12~bookworm docker-ce-cli=5:25.0.3-1~raspbian.12~bookworm docker-ce-rootless-extras=5:25.0.3-1~raspbian.12~bookworm

docker --version

Docker version 25.0.3, build 4debf41

3 replies

ve3bok Mar 12, 2024

This rollback worked for me. Thanks dangersnake!

clmcavaney Mar 14, 2024

Brilliant - that worked for me too. Thanks @dangersnake

clmcavaney Mar 25, 2024

I see that version 26.x now fixes the problem

neersighted · 2024-03-11T12:43:24Z

neersighted
Mar 11, 2024
Maintainer

As noted, this is because of some changes in the Go compiler packaging that have lead to our packaging pipeline building with the wrong compiler options. I'm working to ensure this won't happen again by correctly controlling the build/performing it from a more "stable" environment, and you should likely see things working in the next patch.

No promises, but I'll also consider a packaging-only rebuild with the compilation fixes if that's not too much additional effort.

(Also, I converted this to a discussion as this is really a Docker Inc. packaging issue, and not a bug or anything fixable in moby/moby.)

0 replies

LewisSpring · 2024-03-11T15:35:14Z

LewisSpring
Mar 11, 2024

Just a note for anyone stuck on this issue. I'm on rpi zero w and was able to rollback related packages to get things working again

uname
6.1.0-rpi8-rpi-v6 #1 Raspbian 1:6.1.73-1+rpt1 (2024-01-25) armv6l GNU/Linux
lsb_release
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm
I used this to rollback
apt install docker-ce=5:25.0.3-1~raspbian.12~bookworm docker-ce-cli=5:25.0.3-1~raspbian.12~bookworm docker-ce-rootless-extras=5:25.0.3-1~raspbian.12~bookworm
docker --version
Docker version 25.0.3, build 4debf41

Many thanks for this!
Do you know which versions of compose and buildx I can install in the same fashion? Have not been able to find it just yet

1 reply

LewisSpring Mar 11, 2024

Got it.
docker-buildx-plugin=0.11.1-1~raspbian.12~bookworm
docker-compose-plugin=2.24.6-1~raspbian.12~bookworm

neersighted · 2024-03-19T23:55:26Z

neersighted
Mar 19, 2024
Maintainer

Version 25.0.5 is published to the repositories now, and should consist of valid ARMv6 binaries.

Note that Raspbian has had a high maintenance burden over time, and is only required for BCM2835 SoCs. As such we are critically reviewing whether or not it is worth the added complexity to support, or if Raspbian users on Raspberry Pi 2+ should be directed to use the Debian packages, and Raspbian users on BCM2835 (Raspberry Pi "1," Raspberry Pi Zero "1") should instead rely on the static ARMv5 binaries.

0 replies

RouxAntoine · 2024-03-20T07:30:49Z

RouxAntoine
Mar 20, 2024
Author

Thanks for fix.

Version 25.0.5 work properly with no more segfault

4 replies

neersighted Mar 20, 2024
Maintainer

Your screenshot quite clearly shows 25.0.3, not 25.0.5. Also, the failures here are a CPU (illegal) instruction fault, and not a segmentation fault (which is an invalid memory address access).

Nonetheless, the binaries are ARMv6 and have been verified on real BCM2835 hardware; I expect that when you do update, you will find they work.

RouxAntoine Mar 20, 2024
Author

Yes I also find that strange but apt said I have version 5:25.0.5-1~raspbian.12~bookworm installed and the docker cli work well.
So I just supposed that docker version is not directly linked to package version. Does it is ?

ps : Indeed I always got an (illegal) instruction fault but yes it work.

neersighted Mar 20, 2024
Maintainer

docker version should reflect the package version; I'm not sure why you're seeing what you are, as my system shows the correct versions:

Client: Docker Engine - Community
 Version:           25.0.5
 API version:       1.44
 Go version:        go1.21.8
 Git commit:        5dc9bcc
 Built:             Tue Mar 19 15:05:32 2024
 OS/Arch:           linux/arm
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.5
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       e63daec
  Built:            Tue Mar 19 15:05:32 2024
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Client: Docker Engine - Community
Version:    25.0.5
Context:    default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
  Version:  v0.13.1
  Path:     /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
  Version:  v2.25.0
  Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: 25.0.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
seccomp
 Profile: builtin
cgroupns
Kernel Version: 6.1.21+
Operating System: Raspbian GNU/Linux 11 (bullseye)
OSType: linux
Architecture: armv6l
CPUs: 1
Total Memory: 429.3MiB
Name: raspberrypi
ID: c586a7d7-7ea0-470b-9852-6fb5d6b86776
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

RouxAntoine Mar 20, 2024
Author

Everything is good. apt show <paquet> show last available version and not the last installed one.

thanks for your time

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Last version `5:25.0.4-1~raspbian.12~bookworm` seems broken on raspberry with armv6l (`Illegal instruction`) #47543

{{title}}

Replies: 6 comments 8 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Last version 5:25.0.4-1~raspbian.12~bookworm seems broken on raspberry with armv6l (Illegal instruction) #47543

Description

Reproduce

Expected behavior

docker version

Additional Info

Replies: 6 comments · 8 replies

neersighted Mar 11, 2024 Maintainer

neersighted Mar 19, 2024 Maintainer

RouxAntoine Mar 20, 2024 Author

neersighted Mar 20, 2024 Maintainer

RouxAntoine Mar 20, 2024 Author

neersighted Mar 20, 2024 Maintainer

RouxAntoine Mar 20, 2024 Author

Last version `5:25.0.4-1~raspbian.12~bookworm` seems broken on raspberry with armv6l (`Illegal instruction`) #47543

Replies: 6 comments 8 replies

neersighted
Mar 11, 2024
Maintainer

neersighted
Mar 19, 2024
Maintainer

RouxAntoine
Mar 20, 2024
Author

neersighted Mar 20, 2024
Maintainer

RouxAntoine Mar 20, 2024
Author

neersighted Mar 20, 2024
Maintainer

RouxAntoine Mar 20, 2024
Author