K0s assumes a systemd cgroup layout by default

[twz123]

Is your feature request related to a problem? Please describe.

K0s hard-codes the typical systemd cgroup structure when launching kubelet and containerd. Those defaults are not always a good choice, and changing them via k0s configuration is only possible by using some deprecated kubelet flags.

See:

• cgroups inheritance when using k0s in docker #4234

Describe the solution you would like

This requires some brainstorming. A first step would be to remove the obstacles in the configuration part, so that worker profiles can be used to override the settings in the usual way. Not sure about the rest. Would there be some handy config switches that could be added to k0s to make this easier? Would these config switches be a real improvement over using worker profiles (if they're fixed)? Regarding defaults, one approach might be to define cgroups relative to the cgroup of the k0s main process. Also, AFAIR correctly, both kubelet and containerd have two different cgroup modes (cgroupfs and systemd), which mostly define the cgroup naming scheme (.slice suffixes and such). Should k0s set those values as well?

Describe alternatives you've considered

Not sure, as already mentioned, needs brainstorming. At the very least, doing nothing is not a solution, as it makes k0s unnecessarily difficult to use for certain use cases.

Additional context

In issue #205, it was originally mentioned that k0s should have some sort of systemd detection to choose cgroup settings accordingly, but this was implemented by always using systemd defaults.