docker logs crash due to out of memory

[rty813]

Description

When I use the docker logs command, Docker exits abnormally due to an out-of-memory (OOM) issue.
Although I have only around 1.5GB of available memory, the log file is only about 1MB in size.

My daemon.json:

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "1m",
    "max-file": "5",
    "compress": "false",
    "mode": "non-blocking"
  }
}

paritial dockerd log:

Jun 14 10:15:11 r13 dockerd[23797]: runtime: out of memory: cannot allocate 1937014784-byte block (19464192 in use)
Jun 14 10:15:11 r13 dockerd[23797]: fatal error: out of memory
Jun 14 10:15:11 r13 dockerd[23797]: runtime stack:
Jun 14 10:15:11 r13 dockerd[23797]: runtime.throw(0x569784e6, 0xd)
Jun 14 10:15:11 r13 dockerd[23797]: #011/usr/local/go/src/runtime/panic.go:774 +0x60
Jun 14 10:15:11 r13 dockerd[23797]: runtime.largeAlloc(0x73746473, 0x35330101, 0x3)
Jun 14 10:15:11 r13 dockerd[23797]: #011/usr/local/go/src/runtime/malloc.go:1140 +0x130
Jun 14 10:15:11 r13 dockerd[23797]: runtime.mallocgc.func1()
Jun 14 10:15:11 r13 dockerd[23797]: #011/usr/local/go/src/runtime/malloc.go:1033 +0x38
Jun 14 10:15:11 r13 dockerd[23797]: runtime.systemstack(0x0)
Jun 14 10:15:11 r13 dockerd[23797]: #011/usr/local/go/src/runtime/asm_arm.s:354 +0x90
Jun 14 10:15:11 r13 dockerd[23797]: runtime.mstart()
Jun 14 10:15:11 r13 dockerd[23797]: #011/usr/local/go/src/runtime/proc.go:1146
Jun 14 10:15:11 r13 dockerd[23797]: goroutine 708 [running]:
Jun 14 10:15:11 r13 dockerd[23797]: runtime.systemstack_switch()
Jun 14 10:15:11 r13 dockerd[23797]: #011/usr/local/go/src/runtime/asm_arm.s:298 +0x4 fp=0x34c8abe4 sp=0x34c8abe0 pc=0x550e60f0
Jun 14 10:15:11 r13 dockerd[23797]: runtime.mallocgc(0x73746473, 0x573eb2f0, 0x1, 0x38)
Jun 14 10:15:11 r13 dockerd[23797]: #011/usr/local/go/src/runtime/malloc.go:1032 +0x904 fp=0x34c8ac48 sp=0x34c8abe4 pc=0x5508d41c
Jun 14 10:15:11 r13 dockerd[23797]: runtime.makeslice(0x573eb2f0, 0x73746473, 0x73746473, 0x4)
Jun 14 10:15:11 r13 dockerd[23797]: #011/usr/local/go/src/runtime/slice.go:49 +0x6c fp=0x34c8ac5c sp=0x34c8ac48 pc=0x550ce8a0
Jun 14 10:15:11 r13 dockerd[23797]: github.com/docker/docker/daemon/logger/local.(*decoder).Decode(0x352073c0, 0x34c8ad24, 0x2, 0x1)
Jun 14 10:15:11 r13 dockerd[23797]: #011/go/src/github.com/docker/docker/daemon/logger/local/read.go:139 +0x30c fp=0x34c8acc0 sp=0x34c8ac5c pc=0x555a16b8
Jun 14 10:15:11 r13 dockerd[23797]: github.com/docker/docker/daemon/logger/loggerutils.tailFiles(0x35173600, 0x2, 0x2, 0x35432990, 0x57750fb8, 0x352073c0, 0x5770767c, 0x0, 0x0, 0x0, ...)
Jun 14 10:15:11 r13 dockerd[23797]: #011/go/src/github.com/docker/docker/daemon/logger/loggerutils/logfile.go:590 +0x368 fp=0x34c8ae00 sp=0x34c8acc0 pc=0x555988fc
Jun 14 10:15:11 r13 dockerd[23797]: github.com/docker/docker/daemon/logger/loggerutils.(*LogFile).ReadLogs(0x35520280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Jun 14 10:15:11 r13 dockerd[23797]: #011/go/src/github.com/docker/docker/daemon/logger/loggerutils/logfile.go:419 +0x66c fp=0x34c8af58 sp=0x34c8ae00 pc=0x555977f0
Jun 14 10:15:11 r13 dockerd[23797]: github.com/docker/docker/daemon/logger/local.(*driver).readLogs(0x3545dae0, 0x35432990, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Jun 14 10:15:11 r13 dockerd[23797]: #011/go/src/github.com/docker/docker/daemon/logger/local/read.go:31 +0xb0 fp=0x34c8afb4 sp=0x34c8af58 pc=0x555a0acc
Jun 14 10:15:11 r13 dockerd[23797]: runtime.goexit()
Jun 14 10:15:11 r13 dockerd[23797]: #011/usr/local/go/src/runtime/asm_arm.s:868 +0x4 fp=0x34c8afb4 sp=0x34c8afb4 pc=0x550e7fe0
Jun 14 10:15:11 r13 dockerd[23797]: created by github.com/docker/docker/daemon/logger/local.(*driver).ReadLogs
Jun 14 10:15:11 r13 dockerd[23797]: #011/go/src/github.com/docker/docker/daemon/logger/local/read.go:20 +0x10c
Jun 14 10:15:11 r13 dockerd[23797]: goroutine 1 [chan receive, 4 minutes]:
Jun 14 10:15:11 r13 dockerd[23797]: main.(*DaemonCli).start(0x34edd800, 0x348a76e0, 0x0, 0x0)
Jun 14 10:15:11 r13 dockerd[23797]: #011/go/src/github.com/docker/docker/cmd/dockerd/daemon.go:249 +0x968
Jun 14 10:15:11 r13 dockerd[23797]: main.runDaemon(...)
Jun 14 10:15:11 r13 dockerd[23797]: #011/go/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:13
Jun 14 10:15:11 r13 dockerd[23797]: main.newDaemonCommand.func1(0x348ca6e0, 0x34edd7c0, 0x0, 0x3, 0x0, 0x0)
Jun 14 10:15:11 r13 dockerd[23797]: #011/go/src/github.com/docker/docker/cmd/dockerd/docker.go:34 +0x70
Jun 14 10:15:11 r13 dockerd[23797]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0x348ca6e0, 0x34894108, 0x3, 0x3, 0x348ca6e0, 0x34894108)
Jun 14 10:15:11 r13 dockerd[23797]: #011/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:850 +0x350
Jun 14 10:15:11 r13 dockerd[23797]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x348ca6e0, 0x0, 0x0, 0x10)
Jun 14 10:15:11 r13 dockerd[23797]: #011/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:958 +0x28c

Reproduce

docker logs -f xxxx

Expected behavior

should print all stdout log

docker version

Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:57:47 2021
 OS/Arch:           linux/arm
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:55:49 2021
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  compose: Docker Compose (Docker Inc., v2.18.1)

Server:
 Containers: 10
  Running: 7
  Paused: 0
  Stopped: 3
 Images: 10
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: local
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.1.15
 Operating System: Ubuntu 16.04.7 LTS
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 1.963GiB
 Name: r13
 ID: KBAR:J5KE:76PK:2KS5:WWSZ:DDY2:Z65D:MVHG:Q364:AYEG:NMUO:BBJA
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

[corhere]

Duplicate of #42125, fixed by #43043 (backported to v20.10 in #43165)

[rty813]

Duplicate of #42125, fixed by #43043 (backported to v20.10 in #43165)

@corhere I have updated docker to newest version 20.10.7 supported by my current system, but the problem still exists. So what do I need to do to avoid this problem?

root@190 ~# sudo apt-get install docker-ce
Reading package lists... Done
Building dependency tree       
Reading state information... Done
docker-ce is already the newest version (5:20.10.7~3-0~ubuntu-xenial).
The following packages were automatically installed and are no longer required:
  golang-1.6-go golang-1.6-src golang-src
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 14 not upgraded.

[corhere]

Use the json-file log driver or upgrade to v20.10.24 or newer. You are running Ubuntu Xenial, which is out of long term support. Maybe you could get newer versions of Docker packages from Canonical through paid support with their extended security maintenance program, though you would have to reach out to them to confirm.

[rty813]

Use the json-file log driver or upgrade to v20.10.24 or newer. You are running Ubuntu Xenial, which is out of long term support. Maybe you could get newer versions of Docker packages from Canonical through paid support with their extended security maintenance program, though you would have to reach out to them to confirm.

thanks!