Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[20.10 backport] update containerd binary v1.5.9, runc v1.0.3, and some script changes #43151

Merged
merged 25 commits into from Feb 10, 2022
Merged

[20.10 backport] update containerd binary v1.5.9, runc v1.0.3, and some script changes #43151

merged 25 commits into from Feb 10, 2022

Conversation

thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Jan 13, 2022 •
edited

Per discussion on #43138 #43138 (comment);

Can we backport this to 20.10?

containerd v1.4 EOL is very soon (February 3, 2022) https://github.com/containerd/containerd/blob/main/RELEASES.md#support-horizon

This backports the containerd binary changes to the 20.10 branch for testing, first roling back / reverting the 20.10-only commits, then cherry-picking the updates from master (to preserve history of those commits)

details below:

# https://github.com/moby/moby/pull/43024
git revert -s -S d47de2a4c7f9638ab4db17ad8c4d149e771954b9

# https://github.com/moby/moby/pull/42901
git revert -s -S 129a2000cf752e0afbe935d9e258f916becf8367
git revert -s -S 6835d15f5523063f0a04a86d4810a637c6010d62

# https://github.com/moby/moby/pull/42695
git revert -s -S e8fb8f7acd461db932bce5d7752c33fc9d75020d

# https://github.com/moby/moby/pull/42657
git revert -s -S 067918a8c3018580c86e6c6e6326f68add162876

# https://github.com/moby/moby/pull/42637
git revert -s -S 793340a33a33088f9d5f76031a59062e7e290eba

# https://github.com/moby/moby/pull/42398
git revert -s -S 56541eca9a9ec79fefb750605285d3d88899fb00

# https://github.com/moby/moby/pull/42372
git revert -s -S 01f734cb4f432760fc94a0fb6f64207628ca0662

# https://github.com/moby/moby/pull/42149
git cherry-pick -s -S -x 9b2f55bc1ca1544e407082bbfc626c4bf1745d7c

# https://github.com/moby/moby/pull/42383
git cherry-pick -s -S -x 22c0291333281e2b5d3f39b3c7bf5a31db829522

# https://github.com/moby/moby/pull/42399
git cherry-pick -s -S -x 8e3186fc8f05569200129ee5fd055eb6f25bba20

# https://github.com/moby/moby/pull/42636
git cherry-pick -s -S -x 5ae2af41eebab4e54c730d2b987bd6889dde432c

# https://github.com/moby/moby/pull/42656
git cherry-pick -s -S -x cf1328cd46987b07285fdb9f60b1b7da631f672d

# https://github.com/moby/moby/pull/42697
git cherry-pick -s -S -x 4a07b89e9a23f61de06a9390e49a3e16283f3672

# https://github.com/moby/moby/pull/42776
git cherry-pick -s -S -x b585c64e2b01f924fc358fe059871baa469bb460

# https://github.com/moby/moby/pull/42674 - see if this works
git cherry-pick -s -S -x 3c7c18a4992138ccd6e1d2a63b5ca44327f663bd
git cherry-pick -s -S -x 3cec4b8cd42179e67dcfdda2adce03516c6a5cd1
# conflict in Dockerfile.windows not yet having CONTAINERD_VERSION in it (see below)
git cherry-pick -s -S -x a7a7c732c0dc02ee5b5515f4ca868ef50cafa4a1
# conflict because hack/dockerfile/install/tomll.installer is not in this branch
git cherry-pick -s -S -x 14ff070469d8a1e1f1ea61df9593281f3147e238
# conflict in Dockerfile.windows not yet having CONTAINERD_VERSION in it
git cherry-pick -s -S -x 1b8db067856387b4cc80bd5813993475382993b6

# https://github.com/moby/moby/pull/42902
# conflict in Dockerfile.windows not yet having CONTAINERD_VERSION in it (see below)
git cherry-pick -s -S -x b746a2bf9b2a86a1072e7be719f1207a64a576fb
git cherry-pick -s -S -x fa4a9702be2a2707e7d7345272d58ff9e381f393

# https://github.com/moby/moby/pull/43025
git cherry-pick -s -S -x aef782f34844e70c79481cbecd35b01c9bb25ffa

# https://github.com/moby/moby/pull/43062
git cherry-pick -s -S -x 53397ac5391c49ebd534d06cc57b6a2a12a739d3

# https://github.com/moby/moby/pull/43138
# conflict in Dockerfile.windows not yet having CONTAINERD_VERSION in it (see below)
git cherry-pick -s -S -x df3ea5da03fcff55a47277a8190967e91b7363fa

Conflicts in a7a7c732c0dc02ee5b5515f4ca868ef50cafa4a1, and cherry-picks after that due to Dockerfile.windows not yet having CONTAINERD_VERSION in it in the 20.10 branch;

diff --cc Dockerfile.windows
index 00de2b0941,d56fce719a..0000000000
--- a/Dockerfile.windows
+++ b/Dockerfile.windows
@@@ -165,8 -165,9 +165,14 @@@ FROM microsoft/windowsservercor
  # Use PowerShell as the default shell
  SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

++<<<<<<< HEAD
 +ARG GO_VERSION=1.16.12
 +ARG GOTESTSUM_COMMIT=v0.5.3
++=======
+ ARG GO_VERSION=1.17.0
+ ARG CONTAINERD_VERSION=1.5.5
+ ARG GOTESTSUM_VERSION=v0.5.3
++>>>>>>> a7a7c732c0 (Dockerfile: use version for some utilities instead of commit-sha)

  # Environment variable notes:
  #  - GO_VERSION must be consistent with 'Dockerfile' used by Linux.

@thaJeztah thaJeztah added this to the 20.10.13 milestone Jan 13, 2022
@thaJeztah
Copy link
Member Author

Looks like we still have some test for the binary commits; need to check

=== RUN   TestInfoBinaryCommits
    info_linux_test.go:30: assertion failed:  (info.InitCommit.Expected string) != de40ad0 (info.InitCommit.ID string)
--- FAIL: TestInfoBinaryCommits (0.02s)

thaJeztah and others added 25 commits January 20, 2022 09:24
@thaJeztah
Revert "[20.10] update containerd binary to v1.4.12"
13de46f 
This reverts commit d47de2a.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Revert "[20.10] update containerd binary to v1.4.11"
79fd9c1 
This reverts commit 129a200.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Revert "[20.10] update containerd binary to v1.4.10"
4e838e5 
This reverts commit 6835d15.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Revert "[20.10] update containerd binary to v1.4.9"
eb2acf2 
This reverts commit e8fb8f7.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Revert "[20.10] update containerd binary v1.4.8"
9e36f77 
This reverts commit 067918a.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Revert "[20.10] update containerd binary to v1.4.7"
de656f9 
This reverts commit 793340a.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Revert "[20.10] update containerd binary to v1.4.6"
b097d29 
This reverts commit 56541ec.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Revert "[20.10] update containerd binary to v1.4.5"
1dd3775 
This reverts commit 01f734c.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
update containerd binary to v1.5.0
f036a34 
Welcome to the v1.5.0 release of containerd!

The sixth major release of containerd includes many stability improvements
and code organization changes to make contribution easier and make future
features cleaner to develop. This includes bringing CRI development into the
main containerd repository and switching to Go modules. This release also
brings support for the Node Resource Interface (NRI).

Highlights
--------------------------------------------------------------------------------

*Project Organization*

- Merge containerd/cri codebase into containerd/containerd
- Move to Go modules
- Remove selinux build tag
- Add json log format output option for daemon log

*Snapshots*

- Add configurable overlayfs path
- Separate overlay implementation from plugin
- Native snapshotter configuration and plugin separation
- Devmapper snapshotter configuration and plugin separation
- AUFS snapshotter configuration and plugin separation
- ZFS snapshotter configuration and plugin separation
- Pass custom snapshot labels when creating snapshot
- Add platform check for snapshotter support when unpacking
- Handle loopback mounts
- Support userxattr mount option for overlay in user namespace
- ZFS snapshotter implementation of usage

*Distribution*

- Improve registry response errors
- Improve image pull performance over HTTP 1.1
- Registry configuration package
- Add support for layers compressed with zstd
- Allow arm64 to fallback to arm (v8, v7, v6, v5)

*Runtime*

- Add annotations to containerd task update API
- Add logging binary support when terminal is true
- Runtime support on FreeBSD

*Windows*

- Implement windowsDiff.Compare to allow outputting OCI images
- Optimize WCOW snapshotter to commit writable layers as read-only parent layers
- Optimize LCOW snapshotter use of scratch layers

*CRI*

- Add NRI injection points cri#1552
- Add support for registry host directory configuration
- Update privileged containers to use current capabilities instead of known capabilities
- Add pod annotations to CNI call
- Enable ocicrypt by default
- Support PID NamespaceMode_TARGET

Impactful Client Updates
--------------------------------------------------------------------------------

This release has changes which may affect projects which import containerd.

*Switch to Go modules*

containerd and all containerd sub-repositories are now using Go modules. This
should help make importing easier for handling transitive dependencies. As of
this release, containerd still does not guarantee client library compatibility
for 1.x versions, although best effort is made to minimize impact from changes
to exported Go packages.

*CRI plugin moved to main repository*

With the CRI plugin moving into the main repository, imports under github.com/containerd/cri/
can now be found github.com/containerd/containerd/pkg/cri/.
There are no changes required for end users of CRI.

*Library changes*

oci

The WithAllCapabilities has been removed and replaced with WithAllCurrentCapabilities
and WithAllKnownCapabilities. WithAllKnownCapabilities has similar
functionality to the previous WithAllCapabilities with added support for newer
capabilities. WithAllCurrentCapabilities can be used to give privileged
containers the same set of permissions as the calling process, preventing errors
when privileged containers attempt to get more permissions than given to the
caller.

*Configuration changes*

New registry.config_path for CRI plugin

registry.config_path specifies a directory to look for registry hosts
configuration. When resolving an image name during pull operations, the CRI
plugin will look in the <registry.config_path>/<image hostname>/ directory
for host configuration. An optional hosts.toml file in that directory may be
used to configure which hosts will be used for the pull operation as well
host-specific configurations. Updates under that directory do not require
restarting the containerd daemon.

Enable registry.config_path in the containerd configuration file.

    [plugins."io.containerd.grpc.v1.cri".registry]
       config_path = "/etc/containerd/certs.d"
    Configure registry hosts, such as /etc/containerd/certs.d/docker.io/hosts.toml
    for any image under the docker.io namespace (any image on Docker Hub).

    server = "https://registry-1.docker.io"

    [host."https://public-mirror.example.com"]
      capabilities = ["pull"]
    [host."https://docker-mirror.internal"]
      capabilities = ["pull", "resolve"]
      ca = "docker-mirror.crt"

If no hosts.toml configuration exists in the host directory, it will fallback
to check certificate files based on Docker's certificate file
pattern (".crt" files for CA certificates and ".cert"/".key" files for client
certificates).

*Deprecation of registry.mirrors and registry.configs in CRI plugin*

Mirroring and TLS can now be configured using the new registry.config_path
option. Existing configurations may be migrated to new host directory
configuration. These fields are only deprecated with no planned removal,
however, these configurations cannot be used while registry.config_path is
defined.

*Version 1 schema is deprecated*

Version 2 of the containerd configuration toml is recommended format and the
default. Starting this version, a deprecation warning will be logged when
version 1 is used.

To check version, see the version value in the containerd toml configuration.

    version=2

FreeBSD Runtime Support (Experimental)
--------------------------------------------------------------------------------

This release includes changes that allow containerd to run on FreeBSD with a
compatible runtime, such as runj. This
support should be considered experimental and currently there are no official
binary releases for FreeBSD. The runtimes used by containerd are maintained
separately and have their own stability guarantees. The containerd project
strives to be compatible with any runtime which aims to implement containerd's
shim API and OCI runtime specification.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 9b2f55b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
update containerd binary to v1.5.1
23f23c9 
full diff: containerd/containerd@v1.5.0...v1.5.1

Notable Updates

- Update runc to rc94
- Fix registry mirror authorization logic in CRI plugin
- Fix regression in cri-cni-release to include cri tools

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 22c0291)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
update containerd binary to v1.5.2
5f09d5c 
full diff: containerd/containerd@v1.5.1...v1.5.2

The second patch release for containerd 1.5 is a security release to update
runc for CVE-2021-30465

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 8e3186f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Update containerd binary to v1.5.3
1cd13dc 
full diff: containerd/containerd@v1.5.2...v1.5.3

Welcome to the v1.5.3 release of containerd!

The third patch release for containerd 1.5 updates runc to 1.0.0 and contains
various other fixes.

Notable Updates

- Update runc binary to 1.0.0
- Send pod UID to CNI plugins as K8S_POD_UID
- Fix invalid validation error checking
- Fix error on image pull resume
- Fix User Agent sent to registry authentication server
- Fix symlink resolution for disk mounts on Windows

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 5ae2af4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
update containerd binary v1.4.8
3021146 
Update to containerd 1.4.8 to address [CVE-2021-32760][1].

[1]: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-32760

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit cf1328c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
update containerd binary to v1.5.5
fd32c70 
Welcome to the v1.5.5 release of containerd!

The fifth patch release for containerd 1.5 updates runc to 1.0.1 and contains
other minor updates.

Notable Updates

- Update runc binary to 1.0.1
- Update pull logic to try next mirror on non-404 response
- Update pull authorization logic on redirect

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 4a07b89)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
info: remove "expected" check for tini version
fb45fe6 
These checks were added when we required a specific version of containerd
and runc (different versions were known to be incompatible). I don't think
we had a similar requirement for tini, so this check was redundant. Let's
remove the check altogether.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b585c64)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
remove deprecated "nokmem" build-tag for runc
0f925d5 
This build-tag was removed in opencontainers/runc@52390d6,
which is part of runc v1.0.0-rc94 and up, so no longer relevant.

the kmem options are now always disabled in runc.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 3c7c18a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Dockerfile: remove GOPROXY override (was for go < 1.14)
2a18825 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 3cec4b8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Dockerfile: use version for some utilities instead of commit-sha
b35a170 
The golangci-lint, gotestsum, shfmt, and vndr utilities should generally
be ok to be pinned by version instead of a specific sha. Also rename
the corresponding env-vars / build-args accordingly:

- GOLANGCI_LINT_COMMIT -> GOLANGCI_LINT_VERSION
- GOTESTSUM_COMMIT -> GOTESTSUM_VERSION
- SHFMT_COMMIT -> SHFMT_VERSION
- VNDR_COMMIT -> VNDR_VERSION
- CONTAINERD_COMMIT -> CONTAINERD_VERSION
- RUNC_COMMIT -> RUNC_VERSION
- ROOTLESS_COMMIT -> ROOTLESS_VERSION

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a7a7c73)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Dockerfile: use "go install" to install utilities
2716336 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 14ff070)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Dockerfile: update gotestsum to v1.7.0
0f37f29 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 1b8db06)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Update containerd binary to v1.5.6
3700adb 
- Install apparmor parser for arm64 and update seccomp to 2.5.1
- Update runc binary to 1.0.2
- Update hcsshim to v0.8.21 to fix layer issue on Windows Server 2019
- Add support for 'clone3' syscall to fix issue with certain images when seccomp is enabled
- Add image config labels in CRI container creation
- Fix panic in metadata content writer on copy error

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b746a2b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Update containerd binary to v1.5.7
3fd0b8d 
The seventh patch release for containerd 1.5 is a security release to fix CVE-2021-41103.

Notable Updates:

- Fix insufficiently restricted permissions on container root and plugin directories
  GHSA-c2h3-6mxw-7mvq

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit fa4a970)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
update containerd binary to v1.5.8
bd42e17 
The eighth patch release for containerd 1.5 contains a mitigation for CVE-2021-41190
as well as several fixes and updates.

Notable Updates

* Handle ambiguous OCI manifest parsing
* Filter selinux xattr for image volumes in CRI plugin
* Use DeactiveLayer to unlock layers that cannot be renamed in Windows snapshotter
* Fix pull failure on unexpected EOF
* Close task IO before waiting on delete
* Log a warning for ignored invalid image labels rather than erroring
* Update pull to handle of non-https urls in descriptors

See the changelog for complete list of changes

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit aef782f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@AkihiroSuda @thaJeztah
update runc binary to v1.0.3
3e5eea4 
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit 53397ac)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
update containerd binary to v1.5.9
829f071 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit df3ea5d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah thaJeztah marked this pull request as ready for review January 20, 2022 10:17
@thaJeztah
Copy link
Member Author

whoop; CI is green now. I'm moving this out of draft, but we can discuss further

cpuguy83
cpuguy83 approved these changes Feb 10, 2022
View reviewed changes
Copy link
Member

@cpuguy83 cpuguy83 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah thaJeztah merged commit 9edb938 into moby:20.10 Feb 10, 2022
@thaJeztah thaJeztah deleted the 20.10_backport_containerd_15 branch February 10, 2022 19:36
@AkihiroSuda
Copy link
Member

containerd v1.4 reached EOL: https://github.com/containerd/containerd/pull/6614/files

Can we have Docker 20.10.13 with containerd v1.5 rpms/debs soon?

@thaJeztah
Copy link
Member Author

Yes, working on that; I'll push some final packages of containerd.io 1.4, and doing test-builds of 20.10.13 (and containerd.io 1.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cpuguy83 cpuguy83 cpuguy83 approved these changes

@AkihiroSuda AkihiroSuda AkihiroSuda approved these changes

@tianon tianon Awaiting requested review from tianon tianon is a code owner

Assignees
No one assigned
Labels
area/packaging area/runtime impact/changelog status/2-code-review
Projects
None yet
Milestone
20.10.13
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@thaJeztah @AkihiroSuda @cpuguy83