Use Control Group v2

[agners]

Update: Second attempt.

To make the device permission update work, the following changes are required

• Use Control Group v2 buildroot#3
• Support Device CGroup rules update through runc os-agent#92 (along with a version bump in HAOS, pending)

[agners]

Docker 20.10 is supporting cgroupv2 by default, and with 20.10.6 its also marked as stable: moby/moby#41916. Several distros like Fedora use it by default, Ubutnu 20.10 and Debian 11 (bullseye) are scheduled to make use of it.

While cgroupv2 is a API break compared to v1, its mostly abstracted by systemd/Docker. However, it would break two features currently:

• Currently cgroupv2 doesn't support limiting real-time scheduling (basically rendering Support realtime scheduling for add-ons supervisor#2593 useless). It seems that the detection for real-time scheduling support works well and the Supervisor won't try to set a scheduling limit with cgroupv2. The realtime flag on addons still influences priority limits and memory lock limit through rlimit.
• The new dynamic device support through cgroup in OS Agent (see CGroup os-agent#15) won't work anymore. cgroupv2 uses eBPF filters via ioctl on the cgroup directory to control device access (see https://github.com/containerd/cgroups/blob/master/v2/devicefilter.go#L44). It is possible to set such filters from the OS Agent as well, however it would need messing with the existing filters, which we cannot easily recreate (the default rule already has filters by default. We could recreate that default, but if filters are set at creation time/Docker changes the defaults OS Agent would need updates as well...)

[pvizeli]

The benefit on the OS-Agent is, that we can import the modules from containerd or eBPF and don't need to rewrite the code.

The question is, do we want to wait for this until OS-7 and have a bit more time to make it perfect working? So we can focus on cgroup1 to make that work. We need anyway the cgroup1 for supervisor debian support, which means I have my focus now to cgroup1 and don't can add cgroup2 support before OS-7 over all place (Supervisor itself also generates only cgroup1 rules)

[agners]

I did open an issue about updating Device CGroup in Docker: moby/moby#42255

With CGroup v2 this seems that there is currently a limitation of runc.

[github-actions]

There hasn't been any activity on this pull request recently. This pull request has been automatically marked as stale because of that and will be closed if no further activity occurs within 7 days.
Thank you for your contributions.

[jens-maus]

Please note, that disabling realtime scheduling will have a negative effect on addons using that functionality like the "RaspberryMatic CCU" addon.

[pvizeli]

@agners the supervisor startup script have also option for RT

[agners]

Please note, that disabling realtime scheduling will have a negative effect on addons using that functionality like the "RaspberryMatic CCU" addon.

This is not disabling realtime scheduling, it's just disables (group) allocation of CPU bandwith for realtime scheduling.

In other words, with this disabled, the add-on is no longer restricted how much CPU can be used for realtime scheduling, but it still should be able to do so.

Unfortunately, realtime scheduling isn't available in cgroup v2. I wasn't aware of that back when we added realtime scheduling with #1236. In hindsight, we probably should have waited for this until it becomes available in cgroup v2 as well. Moving to cgroup v2 seems more important than having realtime bandwidth allocation limits to me.

[agners]

@agners the supervisor startup script have also option for RT

Actually did not see any problems, it seems we also have an auto-detection based on a path existing: https://github.com/home-assistant/supervisor/blob/2022.08.2/supervisor/docker/manager.py#L81.

But yes, not setting the ENV_SUPERVISOR_CPU_RT environment is the right thing to do, thanks for mentioning!

[agners]

@jens-maus from a quick test the add-on seems to work. But I don't have a complete network here, so might be worth testing. The latest development build currently on the dev channel uses CGroups v2 (and no RT group scheduling support).