update containerd binary to v1.6.1 #42942
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thaJeztah
force-pushed
the
containerd_binary_1.6
branch
from
ce86779
to
7faf13b
Compare
thaJeztah
changed the title
thaJeztah
force-pushed
the
containerd_binary_1.6
branch
3 times, most recently
from
53a9eea
to
8958328
Compare
samuelkarp
approved these changes
Nov 3, 2021
thaJeztah
force-pushed
the
containerd_binary_1.6
branch
from
8958328
to
aeb3bb5
Compare
thaJeztah
changed the title
thaJeztah
commented
Nov 18, 2021
thaJeztah
force-pushed
the
containerd_binary_1.6
branch
4 times, most recently
from
e7bdede
to
038ae6a
Compare
thaJeztah
changed the title
thaJeztah
changed the title
thaJeztah
force-pushed
the
containerd_binary_1.6
branch
from
038ae6a
to
958440a
Compare
|
Argh.. I never understand PowerShell; Guess it's just back to concatenating stuff |
thaJeztah
force-pushed
the
containerd_binary_1.6
branch
from
958440a
to
3a30d85
Compare
$location='https://github.com/containerd/containerd/releases/download/v'+$Env:CONTAINERD_VERSION+'/containerd-'+$Env:CONTAINERD_VERSION+'-windows-amd64.tar.gz';Could instead be: $location = 'https://github.com/containerd/containerd/releases/download/v{0}/containerd-{0}-windows-amd64.tar.gz' -f $env:CONTAINERD_VERSION;Demo: PS C:\> $location = 'https://github.com/containerd/containerd/releases/download/v{0}/containerd-{0}-windows-amd64.tar.gz' -f $env:CONTAINERD_VERSION
PS C:\> write-host $location
https://github.com/containerd/containerd/releases/download/v1.6.0-beta.4/containerd-1.6.0-beta.4-windows-amd64.tar.gz |
|
Thanks! So I tried putting the $location="https://github.com/containerd/containerd/releases/download/v${Env:CONTAINERD_VERSION}/containerd-${Env:CONTAINERD_VERSION}-windows-amd64.tar.gz"Which (looking at https://docs.microsoft.com/en-us/powershell/scripting/learn/deep-dives/everything-about-string-substitutions?view=powershell-7.2) seemed like it was supported, but not sure why it then tries to execute the string. Guess the devil is in the detail ... somewhere. Not worth spending too much time on for now, I guess, so I just reverted it to what it was |
|
Maybe it expects it to be used without |
|
Failing tests:
This test looks like a possible regression, although it was known to be racy before (see #11966) This could be a regression; although I can find various mentions of this test, some of which indicate it may have been racy https://github.com/moby/moby/search?p=1&q=TestRunMutableNetworkFiles&type=issues This could be a regression (don't see many mentions of that test failing, and online from a long time ago) https://github.com/moby/moby/search?q=TestDaemonRestartSaveContainerExitCode&type=issues Failure in Windows 2022 / containerd (flaky test); #42739 Windows RS5 (flaky test?) was flaky before: #36801 |
|
New run of CI: On amd64: this is starting to look like a regression: On Windows RS5: known flaky test #38521 Giving it another run |
|
Running |
|
@tianon thanks for trying! So 🤔 differences in our CI situation would be;
For easier finding; this is the test that's failing; moby/integration-cli/docker_cli_daemon_test.go Line 2556 in 0f1d65b And it looks pretty straightforward (run a container that uses a non-existing binary as command); moby/integration-cli/docker_cli_daemon_test.go Lines 2568 to 2575 in 0f1d65b |
|
oh, and of course the CI uses the frozen images, so the version of the busybox image is Lines 91 to 92 in 0f1d65b |
|
Looks like the server is using 20.10.10-rc1 -- any idea what version of Maybe we can try running that specific integration test locally to see if we can reproduce outside CI too (or making a test commit that limits CI to just that test and see if that passes 😂 🙈). |
thaJeztah
force-pushed
the
containerd_binary_1.6
branch
from
3a30d85
to
c6000f1
Compare
thaJeztah
changed the title
|
I'm able to reproduce the issue when running inside the dev container ( docker pull busybox
docker run --name error-values --init=false busybox toto
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "toto": executable file not found in $PATH: unknown.
ERRO[0004] error waiting for container: context canceled
docker inspect -f '{{.State.ExitCode}}' error-values
137daemon logs related to the above: Compared with daemon logs on containerd v1.5.8 |
|
I cleaned up logs for both the 1.6.0 and 1.5.8 situation (replacing timestamps, container ID's etc.) and did a compare between the two: containerd-1.5.8.log.txt Diff between those: diff --git a/containerd-1.5.8.log b/containerd-1.6.0.log
index 73f052e5e6..ca156f0821 100644
--- a/containerd-1.5.8.log
+++ b/containerd-1.6.0.log
@@ -1,4 +1,3 @@
-DEBU[xxxx] Calling GET /_ping
DEBU[xxxx] Calling POST /v1.30/containers/create?name=error-values
DEBU[xxxx] form data: {"AttachStderr":true,"AttachStdin":false,"AttachStdout":true,"Cmd":["toto"],"Domainname":"","Entrypoint":null,"Env":[],"HostConfig":{"AutoRemove":false,"Binds":null,"BlkioDeviceReadBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceWriteIOps":null,"BlkioWeight":0,"BlkioWeightDevice":null,"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","ConsoleSize":[0,0],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","DeviceCgroupRules":null,"Devices":[],"DiskQuota":0,"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IOMaximumBandwidth":0,"IOMaximumIOps":0,"Init":false,"IpcMode":"","Isolation":"","KernelMemory":0,"Links":null,"LogConfig":{"Config":{},"Type":""},"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NanoCpus":0,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":null,"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"busybox","Labels":{},"NetworkingConfig":{"EndpointsConfig":{}},"OnBuild":null,"OpenStdin":false,"StdinOnce":false,"Tty":false,"User":"","Volumes":{},"WorkingDir":""}
DEBU[xxxx] container mounted via layerStore: &{/var/lib/docker/overlay2/yyyy/merged 0x344f560 0x344f560} container=yyyy
@@ -17,16 +16,24 @@ DEBU[xxxx] EnableService yyyy START
DEBU[xxxx] EnableService yyyy DONE
DEBU[xxxx] bundle dir created bundle=/var/run/docker/containerd/yyyy module=libcontainerd namespace=moby root=/var/lib/docker/overlay2/yyyy/merged
DEBU[xxxx] event published ns=moby topic=/containers/create type=containerd.events.ContainerCreate
-time="xxxx" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/yyyy pid=659
+time="xxxx" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
+time="xxxx" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
+time="xxxx" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
+time="xxxx" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/yyyy pid=944 runtime=io.containerd.runc.v2
DEBU[xxxx] garbage collected d=1.xxxxms
DEBU[xxxx] sandbox set key processing took 62.xxxxms for container yyyy
DEBU[xxxx] failed to delete task error="rpc error: code = NotFound desc = container not created: not found" id=yyyy
INFO[xxxx] shim disconnected id=yyyy
WARN[xxxx] cleaning up after shim disconnected id=yyyy namespace=moby
INFO[xxxx] cleaning up dead shim
-WARN[xxxx] cleanup warnings time="xxxx" level=info msg="starting signal loop" namespace=moby pid=xxxx
+WARN[xxxx] cleanup warnings time="xxxx" level=info msg="starting signal loop" namespace=moby pid=xxxx runtime=io.containerd.runc.v2
+DEBU[xxxx] event published ns=moby topic=/tasks/exit type=containerd.events.TaskExit
+DEBU[xxxx] event published ns=moby topic=/tasks/delete type=containerd.events.TaskDelete
ERRO[xxxx] copy shim log error="read /proc/self/fd/13: file already closed"
+DEBU[xxxx] event module=libcontainerd namespace=moby topic=/tasks/exit
+DEBU[xxxx] event module=libcontainerd namespace=moby topic=/tasks/delete
ERRO[xxxx] stream copy error: reading from a closed fifo
+INFO[xxxx] ignoring event container=yyyy module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
ERRO[xxxx] stream copy error: reading from a closed fifo
DEBU[xxxx] attach: stdout: end
DEBU[xxxx] attach: stderr: end
@@ -39,3 +46,6 @@ DEBU[xxxx] Releasing addresses for endpoint error-values's interface on network
DEBU[xxxx] ReleaseAddress(LocalDefault/172.18.0.0/16, 172.18.0.2)
DEBU[xxxx] Released address PoolID:LocalDefault/172.18.0.0/16, Address:172.18.0.2 Sequence:App: ipam/default/data, ID: LocalDefault/172.18.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65532, Sequence: (0xe0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:3
ERRO[xxxx] yyyy cleanup: failed to delete container from containerd: no such container
+WARN[xxxx] error locating sandbox id yyyy: sandbox yyyy not found
+DEBU[xxxx] Failed to unmount yyyy overlay: /var/lib/docker/overlay2/yyyy/merged - no such file or directory storage-driver=overlay2
+ERRO[xxxx] yyyy cleanup: failed to delete container from containerd: no such container |
thaJeztah
force-pushed
the
containerd_binary_1.6
branch
from
c6000f1
to
5318a69
Compare
|
containerd/containerd#6465 was merged, which should address the regression, so I pushed a temporary commit to update the Linux version to current main; [WIP] update containerd binary to v1.6.0-rc.1-4-g731518417This contains a fix for the regression we saw on Linux; containerd/containerd@v1.6.0-rc.1...v1.6.0-rc.1-4-g731518417 |
thaJeztah
force-pushed
the
containerd_binary_1.6
branch
from
5318a69
to
afe4bc5
Compare
thaJeztah
changed the title
|
containerd 1.6.0 already released yesterday which it solve a lot of bugs with experimental checkpoint feature. |
thaJeztah
force-pushed
the
containerd_binary_1.6
branch
from
afe4bc5
to
12214a4
Compare
thaJeztah
changed the title
|
Could you rebase and update to v1.6.1? |
|
Given @AkihiroSuda showing me what I was doing wrong in containerd/containerd#6659 (comment) (thanks again! ❤️), I'm +1 on updating to 1.6.1 also. |
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
thaJeztah
force-pushed
the
containerd_binary_1.6
branch
from
12214a4
to
06abe8d
Compare
thaJeztah
changed the title
|
Updated to v1.6.1 |
tianon
approved these changes
Mar 12, 2022
|
@cpuguy83 @tonistiigi @AkihiroSuda we all ok with going to v1.6 for the next release? |
AkihiroSuda
approved these changes
Mar 12, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
- What I did
- How I did it
- How to verify it
- Description for the changelog
- A picture of a cute animal (not mandatory but encouraged)