Docker images based on Ubuntu 22.04 cannot be started Because of EPERM in pthread_create on Debian 10

[TomBenjamins]

Hello,

I have a strange problem.
We have a Debian 10 server on which we run docker images via docker compose.
The images that have been built by paketo buildpacker used by spring boot 2.7.18 (which uses a jdk based on Ubuntu 22.04)
cannot be started, they give the following error:

Setting Active Processor Count to 4
Calculating JVM memory based on 23461124K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx22731761K -XX:MaxMetaspaceSize=217362K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 23461124K, Thread Count: 250, Loaded Class Count: 35962, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 137 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=4 -XX:MaxDirectMemorySize=10M -Xmx22731761K -XX:MaxMetaspaceSize=217362K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
**[0.003s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.**
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log

This seems to suggest that there is a memory problem, but when we set

security-opt 
  - seccomp=unconfined

in docker-compose.yml

The docker image is working again.

2024-01-26 11:30:40,404 DEBUG org.springframework.boot.StartupInfoLogger [main] Running with Spring Boot v2.7.18, Spring v5.3.31

So it seems to be caused by seccomp combined with docker instead of the new buildpacks.

This is what we tried so far but did not fix it (based on searches on the error message above):

• updated docker to version 24.0.7
• Updated libseccomp2 to latest available version in buster-backport repository.
• also setting a custom seccomp yaml (with pthread_create added, perhaps the syntax is incorrect) did not seem to make a difference in the command line

pthread_create.json

Before I make a bug report about this, I wanted to ask here what options do I have to see what is going on here?
Are there other things we could try?

this is the docker server info:

Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 57
  Running: 56
  Paused: 0
  Stopped: 1
 Images: 222
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dd1e886e55dd695541fdcd67420c2888645a495
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 4.19.0-25-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 47.17GiB
 Name: XXXXXXXXXXXX
 ID: PCMI:3I4G:33LS:5QE6:VB3W:4A2L:CJIG:3R23:HZXD:AUQV:MKVZ:F2O7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/16, Size: 24

XXXX@XXXXX:/home/XXXXXXXXXX$ dpkg -l | grep libseccomp
ii  libseccomp2:amd64                    2.5.1-1~bpo10+1               amd64        high level interface to Linux seccomp filter

Any tips for troubleshooting or fixing this would be greatly appreciated.

Tom

[thomasbach-dev]

I'm bit by this as well. https://branislavjenco.github.io/docker-jemalloc-clone3/ discusses the issue quiet nicely. The problem is rather an outdated glibc on the host system.
It's not obviously possible to simply disable seccomp all together in dockerd. (I tried passing "seccomp-profile": "unconfined" in daemon.json, but get an error.)
So, passing seccomp=unconfined to the container is probably the only solution here.

[neersighted]

You can actually set the default seccomp profile to unconfined as shown in the docs example.

That would probably be the easiest thing to do here.

[TomBenjamins]

For people that experience the same problem, the following fixed it for me:

Check whether the runc instance used by docker is actually the version delivered with docker+ docker compose. In my case there was a symlink in /usr/local/bin pointing to an old version of runc. After removing this symlink it starts working again.