kubelet and containerd in endless loop for CreateContainer with unexpected media type octet-stream

[thomasmey]

Description

I think the private container registry had some intermittent errors and did return wrong data for some layers.

The problem I think is that kubelet or containerd is in a state were it assumes that a given OCI image is already downloaded locally, but then some parsing of this layer fails and the loop starts again.

Steps to reproduce the issue

1. Not sure, probably container registry needs to return media type octet-stream for some layer/manifest instead of correct media type.

Describe the results you received and expected

kubelet and or containerd should detect the wrong layer/manifest state and delete the incomplete/erroneous download and pull again from container registry.

What version of containerd are you using?

1.7.10

Any other relevant information

Kubelet 1.28.7-gke.1026000
Containerd 1.7.10
Linux 6.1.58+

In the pod events we see thousands of events like:
"Container image "xyz" already present on machine" Pulled.
This seems to originate from here:
https://github.com/kubernetes/kubernetes/blob/v1.28.7/pkg/kubelet/images/image_manager.go#L138

So kubelet thinks that image is already present and successful pulled, but then containerd fails with:
"failed to create containerd container: error unpacking image: unexpected media type application/octet-stream for sha:xyz: not found: CreateContainerError"

The containerd fails probably here:
https://github.com/containerd/containerd/blob/v1.7.10/images/image.go#L238

Looks like something with the local manifest bookkeeping is not correct, which hinders container creation.

Maybe containerd can detect that image layer/manifest is read from local cache and remove erroneous cached local image layers.

Show configuration if it is related to CRI plugin.

No response

[thomasmey]

I have no idea what I'm doing! But maybe something in this direction?

diff --git a/pkg/cri/server/container_create.go b/pkg/cri/server/container_create.go
index a3c97f506..8bca51734 100644
--- a/pkg/cri/server/container_create.go
+++ b/pkg/cri/server/container_create.go
@@ -33,6 +33,7 @@ import (
 
 	"github.com/containerd/containerd"
 	"github.com/containerd/containerd/containers"
+	"github.com/containerd/containerd/images"
 	"github.com/containerd/containerd/log"
 	"github.com/containerd/containerd/oci"
 	criconfig "github.com/containerd/containerd/pkg/cri/config"
@@ -103,6 +104,13 @@ func (c *criService) CreateContainer(ctx context.Context, r *runtime.CreateConta
 		return nil, fmt.Errorf("failed to get image from containerd %q: %w", image.ID, err)
 	}
 
+	// validate manifest early and delete image from store if something is not right
+	if _, err := images.Manifest(ctx, containerdImage.ContentStore(), containerdImage.Target(), containerdImage.Platform()); err != nil {
+		//c.imageStore.Delete(image.ID)
+		containerdImage.ContentStore().Delete(ctx, containerdImage.Metadata().Target.Digest)
+		return nil, fmt.Errorf("failed to validate manifest: %w", err)
+	}
+
 	start := time.Now()
 	// Run container using the same runtime with sandbox.
 	sandboxInfo, err := sandbox.Container.Info(ctx)

[phooq]

I thought Manifest is inspected before image gets pulled, I am not sure about this tho.
So is your case like the image was pulled successfully (manifest is inspected and was deemed fine), but later CreateContainer found that the manifest has unknown media type?

Also, is the private registry randomly return different media types? I feel it is unlikely, but seems this is the case based on your description

[thomasmey]

Hi, yes something like this, as I said our private container registry had some intermittent errors.
If I understand the code correctly in the CreateContainer path, the manifest media type comes from the metadata store, which is a bbolt db.
Maybe the wrong media typed ended up in this metadata store as it only seems to check if the media type is not empty when storing:
https://github.com/containerd/containerd/blob/v1.7.10/metadata/images.go#L304

But let me also follow/understand the "pull image" CRI code path that is triggered by k8s and get a better understanding of how this unrecoverable state could have happened.

Anyway containerd should recover from this error state and not keep stuck in an endless loop in k8s.

In the end I had to throw away the node in k8s to recover from this error state. But I want to have a root cause fix for this error state.