[Help] test the performance of containerd and moby in kubernetes

[xrmzju]

We have used moby in our production env for a long time, and recently we decided to explorer the possibility to replace docker(moby) with containerd for it's simplicity, and we have done some tests between containerd and moby.
The first test we do is pod_start_up_latency. Theoretically, containerd should perform better in latency, and that is what we have got from a lot of blog or papers( like the offical GA POST) .
test env: 2 vm with same flavor. vm1 start kubelet with docker 18.09.6, vm2 start kubelet with--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock。both vm1 and vm2 use containerd 1.2.6(894b81a)
test process: start 50 pods with same predownload images, and check the pod_start_latency metrics for kubelet.
result: i run the test 5 times. moby performs 10% better in pod_start_latency than containerd

[xrmzju]

is there any clue or some config i need change? or anybody who have done comparison test between containerd and docker can you share your method ?

[mikebrow]

are you also running docker containers in that host while running containerd independently? FYI kubelet still has dockershim built into it even if it is configured to load CRI services via external container runtime. How are you "preloading" images into containerd? Docker and containerd do not share container image caches. v1.2 of containerd shipped years ago that original comparison was against the latest docker at the time. The latest version of containerd is v1.4.

[xrmzju]

are you also running docker containers in that host while running containerd independently? FYI kubelet still has dockershim built into it even if it is configured to load CRI services via external container runtime. How are you "preloading" images into containerd? Docker and containerd do not share container image caches. v1.2 of containerd shipped years ago that original comparison was against the latest docker at the time. The latest version of containerd is v1.4.

i have stopped the docker. preloading means i've use ctr command line to prepull the test image and pause image.

[ v1.2 of containerd shipped years ago that original comparison was against the latest docker at the time. The latest version of containerd is v1.4.]

the containerd GA blog from kubernetes posted in 2018...

i'll close the debug mode in containerd and try the latest containerd version and see the results

[mikebrow]

when you used ctr which namespace did you load into?

[mikebrow]

if you are using kubelet over cri to load images that will map to k8s.io if you didn't set namespace it's going to be filtered out

sudo ctr ns ls
NAME    LABELS 
default        
density        
k8s.io         
moby           
stress         
test  

[xrmzju]

in k8s.io namespaces.

[mikebrow]

there's an external dockershim we are using for ci for cri-tools ... running container runtimes at least through crictl is more likely to give some decent results.. but you need to make sure to configure the crictl container runtime sock for each test as crictl has a "sequence" now for automatically trying various default container runtime socks.

There is a big difference between 50 pods each with one busy box container and 50 pods with 5-10 containers making up an actual application.

I suppose without specifying what you are testing, and how you are testing it.. it's sort of hard to discern why there are differences.

With that old version of containerd vs newer version of docker/docker shim in kublet you are also comparing old golang vs newer golang. Some container images will work better on docker's graph driver others will work better on containerd, also depends on which FS you are using. Containerd shim v2 is much better than the old v1 shim. So many changes over the last two years.

[xrmzju]

i use the same containerd binary which docker 18.09 uses, and my kubelet version is v14.4， my test case is simple and clear: 50 pods each with on container and check the pod_start_up_latency metrics from kubelet. both docker and containerd use overlay fs. and i tried with the latest containerd binary yesterday, there is no better results, even with containerd shim v2.
i suppose this is a common user case: a docker runtime user wants to change there runtime to containerd by directly use the containerd binary they have in there machine. and i think my test case is what a k8s end user cared most, instead of results from cri-tools or crictl.

[fuweid]

Hi, @xrmzju I am curious about the result. What is pod_start_up_latency metrics? would you mind to show the code in kubelet?

https://github.com/kubernetes/kubernetes/blob/release-1.14/pkg/kubelet/kubelet.go#L1519-L1522

I am not sure that I find the right code in the link. If it is true, the latency is not collected from containerd or docker directly, which means that it depends on kubelet sync mechanism(get event from APIServer, 1s List All container PLEG and blabla). When you try to get the performance result, you should directly get the latency from the containerd, not from kubelet. crictl maybe good tool to test it .

And one more thing is that if you use docker, the chain of call will be kubelet(docker-shim) -> docker -> containerd. And if you use cri-containerd plugin with kubelet, the call will be kubelet -> cri-containerd. I think from chain of call perspective, cri-containerd reduces one hop which can improve performance. However, in declared state, kubelet just maintains and force actual state to declared state. It doesn't show us the performance from CRI directly even if CRI implementation does better. :)

Hope it can help.

[xrmzju]

Hi, @xrmzju I am curious about the result. What is pod_start_up_latency metrics? would you mind to show the code in kubelet?

https://github.com/kubernetes/kubernetes/blob/release-1.14/pkg/kubelet/kubelet.go#L1519-L1522

I am not sure that I find the right code in the link. If it is true, the latency is not collected from containerd or docker directly, which means that it depends on kubelet sync mechanism(get event from APIServer, 1s List All container PLEG and blabla). When you try to get the performance result, you should directly get the latency from the containerd, not from kubelet. crictl maybe good tool to test it .

And one more thing is that if you use docker, the chain of call will be kubelet(docker-shim) -> docker -> containerd. And if you use cri-containerd plugin with kubelet, the call will be kubelet -> cri-containerd. I think from chain of call perspective, cri-containerd reduces one hop which can improve performance. However, in declared state, kubelet just maintains and force actual state to declared state. It doesn't show us the performance from CRI directly even if CRI implementation does better. :)

Hope it can help.

oh i see... yes u r right, the pod_start_up_latency depends mainly on the kubelet sync mechanism, i'll try the crictl tool