Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not set devices cgroup when updating containers to avoid eBPF programs leak #2876

Closed
wants to merge 1 commit into from

Conversation

borgerli
Copy link

Do not set devices cgroup when updating containers to fix #2366 for cgroup v2

We're running k8s 1.20 with containerd + runc + cgroup v2, and encountered the issue of "failed to call BPF_PROG_ATTACH (BPF_CGROUP_DEVICE, BPF_F_ALLOW_MULTI): argument list too long" when updating containers.

Steps to reproduce with runc

  1. Run a container with id test
  2. Try to update the memory 64 times and you'll get error(there are 64 bpf programs attached ):
root@VM-16-3-centos ~]# for i in {1..64}; do runc update --memory=10240000 test;done 
WARN[0000] Setting back cgroup configs failed due to error: failed to call BPF_PROG_ATTACH (BPF_CGROUP_DEVICE, BPF_F_ALLOW_MULTI): argument list too long, your state.json and actual configs might be inconsistent. 
ERRO[0000] failed to call BPF_PROG_ATTACH (BPF_CGROUP_DEVICE, BPF_F_ALLOW_MULTI): argument list too long

[root@VM-16-3-centos ~]# bpftool cgroup list /sys/fs/cgroup/user.slice/user-0.slice/test | wc -l
65

…rams leak

Signed-off-by: Li Bo <borgerli@tencent.com>
@cyphar
Copy link
Member

cyphar commented Mar 28, 2021

Uh, this means that if you try to change the set of cgroup limits runc update will ignore them (this is almost certainly why the tests are failing). You'd need to only do this if the new configuration doesn't change the cgroup config.

(Though I think a better solution would be to fix device updates -- see #2366 -- rather than patching around the issue.)

@borgerli
Copy link
Author

@cyphar Does runc have plan to support updating device cgroup in future? For now I see that there are no such options to update devices setting. Thanks.

@cyphar
Copy link
Member

cyphar commented Apr 3, 2021

You can update the devices cgroup AFAIK -- the devices cgroup is part of the "resources" object in the JSON configuration. There is another devices list but that's for the creation of the container, not the cgroup configuration.

@cyphar
Copy link
Member

cyphar commented May 25, 2021

No longer needed now that #2951 was merged.

@cyphar cyphar closed this May 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cgroup: devices updates appear to be broken
3 participants