Update containerd to v1.6.10

[dalbani]

containerd has recently be released at version 1.6.9: see https://github.com/containerd/containerd/releases/tag/v1.6.9.
Apart from the good practice of using the latest release, the reason why I'm raising this up is that v1.6.9 contains a fix related to the native snapshotter. Which happens to be the snapshotter to use apparently when running on ZFS, as per the conversation at the bottom of #3980.
See canonical/microk8s#3499 for more details about the issue with the native snapshotter, which I raised for Microk8s but very probably applies to k3s as well (I haven't tested in practice).

[dalbani]

By the way, I don't know if it's already documented somewhere, but it would be interesting to know more about the process that leads to a tag in Git for the containerd "fork" at https://github.com/k3s-io/containerd/tags.

[brandond]

It's manual, and involves rebasing our patches to add registry rewrites and a couple other things.

[kurokobo]

I expect containerd to be updated to 1.6.9 in the near future.
I'm facing another issue with containerd v1.6.8-k3s1 where I am unable to pull certain images that already fixed in 1.6.9.

• Discussion and Issue:
  • `ctr image pull` fail to extract layer due to `operation not permitted` error containerd/containerd#7083
  • Fail to extract layer due to operation not permitted error containerd/containerd#7093
• PR for 1.6:
  • [release/1.6] cherry-pick: make xattr EPERM non-fatal in createTarFile containerd/containerd#7447

$ sudo $(which k3s) crictl version
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  v1.6.8-k3s1
RuntimeApiVersion:  v1

$ sudo $(which k3s) crictl pull quay.io/pulp/galaxy:4.6.2
E1115 15:40:14.338598 2355381 remote_image.go:242] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"quay.io/pulp/galaxy:4.6.2\": failed to extract layer sha256:db4e88ec5915e6d2ca0aaab9198f8dc4e2eb40285da4d9e751f853cef1315964: operation not permitted: unknown" image="quay.io/pulp/galaxy:4.6.2"
FATA[0010] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "quay.io/pulp/galaxy:4.6.2": failed to extract layer sha256:db4e88ec5915e6d2ca0aaab9198f8dc4e2eb40285da4d9e751f853cef1315964: operation not permitted: unknown 

$ sudo $(which k3s) crictl pull pulp/pulp-galaxy-ng:latest
E1115 15:40:57.810207 2355556 remote_image.go:242] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"docker.io/pulp/pulp-galaxy-ng:latest\": failed to extract layer sha256:a3b8450d21451a6808b1f3ca7390772b12554abfb223273830ce6ca1e19b1fb1: operation not permitted: unknown" image="pulp/pulp-galaxy-ng:latest"
FATA[0010] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/pulp/pulp-galaxy-ng:latest": failed to extract layer sha256:a3b8450d21451a6808b1f3ca7390772b12554abfb223273830ce6ca1e19b1fb1: operation not permitted: unknown 

[dalbani]

I can confirm that the upgrade to containerd v1.6.9 fixed the issue in MicroK8s.
I'm still a bit surprised that the problem is discovered just "now", and that the QA process of neither containerd, MicroK8s nor K3s discovered it earlier on.

[brandond]

I haven't seen any reports of this issue in the wild, other than the sparse conversation here. I don't think anyone really uses the native snapshotter unless they have absolutely no other choice, as its performance is (as far as I know) pretty poor. We certainly don't test it on any regular basis.

[bguzman-3pillar]

Validated on commit a07bb55

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Cluster Configuration:

1 server

Testing Steps

1. Install k3s using commit curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=a07bb555ba052516a666b9d1b11ec581f9c7b5f8 INSTALL_K3S_EXEC="server" sh -
2. Run kubectl get node -A -o wide to check the version (Look for containerd://1.6.10-k3s1)

Validation Results:

• Containerd 1.6.10 is displayed.

$ kubectl get node -A -o wide
NAME               STATUS   ROLES                  AGE     VERSION                INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
ip-172-31-43-134   Ready    control-plane,master   2m20s   v1.25.4+k3s-a07bb555   172.31.43.134   <none>        Ubuntu 22.04.1 LTS   5.15.0-1019-aws   containerd://1.6.10-k3s1

Additional context / logs:

[dalbani]

For the record, related commit: 6462a31.