Workers error with "EOF" when upgrading to 7.11.0 on ubuntu focal

[jfharden]

Summary

When upgrading to either 7.10 or 7.11 (from 7.8.3) the workers start, but any job (including resource checks) that runs on them errors with in the task view in the UI:

run check: start process: backend error: Exit status: 500, message: {"Type":"","Message":"EOF","Handle":"","ProcessID":"","Binary":""}

The logs on the worker show

{"timestamp":"2023-11-14T07:12:17.964626613Z","level":"error","source":"guardian","message":"guardian.api.garden-server.run.failed","data":{"error":"EOF","handle":"c922086f-cb28-4df1-51ed-f2d01a7ab421","session":"3.1.63"}}

A more extensive section of the logs

{"timestamp":"2023-11-14T07:12:07.641197441Z","level":"info","source":"guardian","message":"guardian.create.start","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84"}}
{"timestamp":"2023-11-14T07:12:07.641329082Z","level":"info","source":"guardian","message":"guardian.create.network-depot-setup-bindmounts.start","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84.2"}}
{"timestamp":"2023-11-14T07:12:07.641634352Z","level":"info","source":"guardian","message":"guardian.create.network-depot-setup-bindmounts.finished","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84.2"}}
{"timestamp":"2023-11-14T07:12:07.641666614Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.start","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84.3"}}
{"timestamp":"2023-11-14T07:12:07.645021328Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.depot-create.started","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84.3.1"}}
{"timestamp":"2023-11-14T07:12:07.645481039Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.depot-create.finished","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84.3.1"}}
{"timestamp":"2023-11-14T07:12:07.645793285Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.creating","data":{"bundle":"/opt/concourse/worker/depot/b39b0419-e641-4aac-761e-63ee5cd00ac8","bundlePath":"/opt/concourse/worker/depot/b39b0419-e641-4aac-761e-63ee5cd00ac8","handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","id":"b39b0419-e641-4aac-761e-63ee5cd00ac8","logPath":"/opt/concourse/worker/depot/b39b0419-e641-4aac-761e-63ee5cd00ac8/create.log","pidFilePath":"/opt/concourse/worker/depot/b39b0419-e641-4aac-761e-63ee5cd00ac8/pidfile","runc":"goci.RuncBinary{Path:\"/var/gdn/assets/linux/bin/runc\", Root:\"/run/runc\"}","session":"84.3.2"}}
{"timestamp":"2023-11-14T07:12:07.796180983Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.completing","data":{"bundle":"/opt/concourse/worker/depot/b39b0419-e641-4aac-761e-63ee5cd00ac8","handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84.3.2"}}
{"timestamp":"2023-11-14T07:12:07.796950124Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.finished","data":{"bundle":"/opt/concourse/worker/depot/b39b0419-e641-4aac-761e-63ee5cd00ac8","handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84.3.2"}}
{"timestamp":"2023-11-14T07:12:07.797284056Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.finished","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84.3"}}
{"timestamp":"2023-11-14T07:12:07.797226602Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.watch.watching","data":{"bundle":"/opt/concourse/worker/depot/b39b0419-e641-4aac-761e-63ee5cd00ac8","handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84.3.2.1"}}
{"timestamp":"2023-11-14T07:12:07.821796469Z","level":"info","source":"guardian","message":"guardian.create.network.started","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84.7","spec":""}}
{"timestamp":"2023-11-14T07:12:07.821863513Z","level":"info","source":"guardian","message":"guardian.create.network.config-create","data":{"config":{"ContainerHandle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","HostIntf":"w5rbhjfd7t56-0","ContainerIntf":"w5rbhjfd7t56-1","IPTablePrefix":"w--","IPTableInstance":"5rbhjfd7t56","BridgeName":"wbrdg-0afe0004","BridgeIP":"10.254.0.5","ContainerIP":"10.254.0.6","ExternalIP":"10.1.10.173","Subnet":{"IP":"10.254.0.4","Mask":"/////A=="},"Mtu":1500,"PluginNameservers":null,"OperatorNameservers":["169.254.169.253"],"AdditionalNameservers":[],"AdditionalHostEntries":null,"PluginSearchDomains":null},"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84.7","spec":""}}
{"timestamp":"2023-11-14T07:12:07.967002239Z","level":"info","source":"guardian","message":"guardian.create.network.finished","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84.7","spec":""}}
{"timestamp":"2023-11-14T07:12:07.967531500Z","level":"info","source":"guardian","message":"guardian.create.created","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84"}}
{"timestamp":"2023-11-14T07:12:07.967811784Z","level":"info","source":"guardian","message":"guardian.api.garden-server.create.created","data":{"request":{"Handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","GraceTime":0,"RootFSPath":"raw:///opt/concourse/worker/volumes/live/c3775264-9d82-4fb2-4a9f-83ac52fcdfe0/volume","BindMounts":[{"src_path":"/opt/concourse/worker/volumes/live/12bbf7c6-57a8-4e5d-7404-e17fc024fe5f/volume","dst_path":"/scratch","mode":1}],"Network":"","Privileged":false,"Limits":{"bandwidth_limits":{},"cpu_limits":{},"disk_limits":{},"memory_limits":{},"pid_limits":{}}},"session":"3.1.54"}}
{"timestamp":"2023-11-14T07:12:08.009307722Z","level":"info","source":"guardian","message":"guardian.api.garden-server.get-properties.got-properties","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"3.1.55"}}
{"timestamp":"2023-11-14T07:12:08.014780285Z","level":"info","source":"guardian","message":"guardian.run.started","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","path":"/opt/resource/check","session":"85"}}
{"timestamp":"2023-11-14T07:12:08.015520080Z","level":"info","source":"guardian","message":"guardian.run.exec.start","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","path":"/opt/resource/check","session":"85.1"}}
{"timestamp":"2023-11-14T07:12:08.018095034Z","level":"info","source":"guardian","message":"guardian.run.exec.exec-with-bndl.start","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","path":"/opt/resource/check","session":"85.1.2"}}
{"timestamp":"2023-11-14T07:12:08.034798383Z","level":"info","source":"guardian","message":"guardian.run.exec.exec-with-bndl.execrunner.start","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","id":"1a2938e3-55d6-4a9f-6a03-ad3976665426","path":"/opt/resource/check","session":"85.1.2.2"}}
{"timestamp":"2023-11-14T07:12:08.038683963Z","level":"error","source":"guardian","message":"guardian.api.garden-server.run.failed","data":{"error":"EOF","handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"3.1.56"}}
{"timestamp":"2023-11-14T07:12:08.038481065Z","level":"info","source":"guardian","message":"guardian.run.exec.exec-with-bndl.execrunner.done","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","id":"1a2938e3-55d6-4a9f-6a03-ad3976665426","path":"/opt/resource/check","session":"85.1.2.2"}}
{"timestamp":"2023-11-14T07:12:08.038623715Z","level":"info","source":"guardian","message":"guardian.run.exec.exec-with-bndl.finished","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","path":"/opt/resource/check","session":"85.1.2"}}
{"timestamp":"2023-11-14T07:12:08.038650655Z","level":"info","source":"guardian","message":"guardian.run.exec.finished","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","path":"/opt/resource/check","session":"85.1"}}
{"timestamp":"2023-11-14T07:12:08.038664517Z","level":"info","source":"guardian","message":"guardian.run.finished","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","path":"/opt/resource/check","session":"85"}}
{"timestamp":"2023-11-14T07:12:13.526555511Z","level":"info","source":"guardian","message":"guardian.destroy.start","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87"}}
{"timestamp":"2023-11-14T07:12:13.526886525Z","level":"info","source":"guardian","message":"guardian.destroy.destroy.started","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87.1"}}
{"timestamp":"2023-11-14T07:12:13.526918325Z","level":"info","source":"guardian","message":"guardian.destroy.destroy.delete.started","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87.1.1"}}
{"timestamp":"2023-11-14T07:12:13.533444877Z","level":"info","source":"guardian","message":"guardian.destroy.destroy.delete.state","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87.1.1","state":{"Pid":1988,"Status":"running"}}}
{"timestamp":"2023-11-14T07:12:13.533488858Z","level":"info","source":"guardian","message":"guardian.destroy.destroy.delete.runc-delete.started","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87.1.1.2"}}
{"timestamp":"2023-11-14T07:12:13.641518954Z","level":"info","source":"guardian","message":"guardian.destroy.destroy.delete.runc-delete.finished","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87.1.1.2"}}
{"timestamp":"2023-11-14T07:12:13.641588015Z","level":"info","source":"guardian","message":"guardian.destroy.destroy.delete.finished","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87.1.1"}}
{"timestamp":"2023-11-14T07:12:13.641603258Z","level":"info","source":"guardian","message":"guardian.destroy.destroy.finished","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87.1"}}
{"timestamp":"2023-11-14T07:12:13.656440524Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.watch.done","data":{"bundle":"/opt/concourse/worker/depot/b39b0419-e641-4aac-761e-63ee5cd00ac8","handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"84.3.2.1"}}
{"timestamp":"2023-11-14T07:12:13.711707320Z","level":"info","source":"guardian","message":"guardian.destroy.network-depot-destroy.start","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87.2"}}
{"timestamp":"2023-11-14T07:12:13.711851916Z","level":"info","source":"guardian","message":"guardian.destroy.network-depot-destroy.finished","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87.2"}}
{"timestamp":"2023-11-14T07:12:13.711880867Z","level":"info","source":"guardian","message":"guardian.destroy.remove-bundle.destroy.started","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87.4.1"}}
{"timestamp":"2023-11-14T07:12:13.712077708Z","level":"info","source":"guardian","message":"guardian.destroy.remove-bundle.destroy.finished","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87.4.1"}}
{"timestamp":"2023-11-14T07:12:13.712106056Z","level":"info","source":"guardian","message":"guardian.destroy.remove-bundle.destroy.started","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87.4.2"}}
{"timestamp":"2023-11-14T07:12:13.712126039Z","level":"info","source":"guardian","message":"guardian.destroy.remove-bundle.destroy.finished","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87.4.2"}}
{"timestamp":"2023-11-14T07:12:13.712137529Z","level":"info","source":"guardian","message":"guardian.destroy.finished","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"87"}}
{"timestamp":"2023-11-14T07:12:13.712149284Z","level":"info","source":"guardian","message":"guardian.api.garden-server.destroy.destroyed","data":{"handle":"b39b0419-e641-4aac-761e-63ee5cd00ac8","session":"3.1.58"}}

Steps to reproduce

These are the only steps I can be 100% sure work

1. Have concourse 7.8.3 running
2. Upgrade to concourse 7.11.0 by:
   1. landing then terminating all worker instances
   2. Upgrade the web node to 7.11
   3. Launch a worker on 7.11

Expected results

The worker runs jobs as expected

Actual results

All jobs and resource checks fail with error

run check: start process: backend error: Exit status: 500, message: {"Type":"","Message":"EOF","Handle":"","ProcessID":"","Binary":""}

Additional context

Installed on ubuntu Focal 20.04

Triaging info

• Concourse version: 7.8.3
• Browser (if applicable): N/A
• Server OS: Ubuntu 20.04
• Did this used to work? Previously we tried to upgrade to 7.9 which failed with the bundled iptables not having the required shared libraries in ubuntu 20.04 (attempts by concourse to run the bundled iptables command just failed and the workers would not start at all). I tried to 7.10 and 7.11 today and in both cases recevied the errors in this PR

[xtremerui]

Hi is it possible to use containerd runtime? We know there is garden related issues for ubuntu 20.04 and 22.04.

[jfharden]

We have upgraded to containerd, and then later also to ubuntu 22.04, and have since successfully upgraded to concourse 7.11.0

Sadly I didn't try an upgrade when just switching to containerd.

It might be good to update the prerequisites in the manual perhaps?