Use systemd as cgroup driver

[rfranzke]

How to categorize this issue?

/area open-source
/kind enhancement

What would you like to be added:
With #5255 we have switched the cgroup driver from cgroupfs to systemd for shoots >= 1.23. However, this change was reverted with #5324.

Let's look into that the consequences of this change are, how the container runtime needs to be configured and how to properly roll this out.

Why is this needed:
Following upstream recommendations.

Work Items

• Switch default kubelet configuration to systemd for all k8s versions >= 1.24 as was done in https://github.com/gardener/gardener/pull/5255/files#diff-fc4776dd3fd3277fcbaac11c0bcfe15e6bd797038eb1cc601444de5ca4c3bc68 but later reverted
  • We don't care about docker configuration as that's not a possible configuration in Gardener for k8s >= 1.23
• Introduce a property/helper function to figure out what the cgroupfs driver configuration is
• Switch g/g containerd configuration to respect the selected cgroupfs driver

• Adapt containerd bootstrapping in OS extensions to use systemd as cgroupfs driver
  • Pass in an additional parameter similar to isContainerDEnabled that indicates which cgroupfs driver is used.
  • Gardenlinux
  • Flatcar
  • Ubuntu
  • SUSE Chost

Open Questions

• How do we prevent running with systemd as cgroupfs driver when the OS extensions have not been updated to respect this in their bootstrapping logic?
• Should we have a killswitch e.g. in cloud-config/kubelet reconfiguration that throws and error when there's a mismatch between kubelet config and containerruntime?

[vpnachev]

Thinking again on this topic, Gardener can configure both kubelet and containerd (similarly to how it control the pause image) to use the wanted cgroup driver, so from this point of view the OS extensions should not do anything. What do you think?

[voelzmo]

Thinking again on this topic, Gardener can configure both kubelet and containerd (similarly to how it control the pause image) to use the wanted cgroup driver, so from this point of view the OS extensions should not do anything. What do you think?

I can see how gardener modifies the containerd configuration file to inject the pause image and was thinking about something similar for the cgroup driver setting as well. This would, however, mean that we bootstrap a node first with the containerd configuration file from the OS extensions (flatcar example here, and later on change the cgroup driver, when the cloud-config is applied. My understanding was that changing the cgroup driver for a running node was something we would want to avoid? Or do you think in this scenario this would be okay?

[voelzmo]

@vpnachev do you have some thoughts on the above discussion? It may be that we have some time to pick this up in the near future.

[gardener-ci-robot]

The Gardener project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close

/lifecycle stale

[rfranzke]

/remove-lifecycle stale
cc @MrBatschner

[voelzmo]

I understand that this is only a prerequisite for enabling cgroupsv2, but please be aware that additional things need to be done before switching the cgroup:

• ensure that we're using at least containerd 1.5.12 or 1.6.3 to get the fix for incorrectly detected OOMKill events, or otherwise VPA cannot act on OOMKill events.
  • Done in gardenlinux since 576.12
  • other supported OSes?

And a few interesting things from https://blog.kintone.io/entry/2022/03/08/170206

For example (taken from the above link) regarding uber-go/automaxprocs:

If you're using a version until v1.4.0, you need to patch it with a cgroup v2 PR or set GOMAXPROCS manually for a while.

And regarding the JDK version

We need to use JDK 15 or later to run Java applications properly in the cgroup v2 environment. Let me describe it briefly.

JDK has built-in support for the container environment from version 8u131. JDK 10 introduced the +UseContainerSupport option and enabled it by default. With this option, JDK inspects the cgroup filesystem and reads the CPU and memory quotas for its use. The CPU quota information is available on Runtime.availableProcessors() (see Docker blog for details). The memory quota affects its heap memory usage. This mechanism should be adjusted for cgroup v2, and the fix became available with JDK 15.

[gardener-ci-robot]

The Gardener project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close

/lifecycle stale

[ialidzhikov]

/remove-lifecycle stale
/lifecycle frozen

[MrBatschner]

/assign @danielfoehrKn

[rfranzke]

@danielfoehrKn @MrBatschner Could you check the PR description and tell whether this is still up-to-date/the plan you are pursuing?

[rfranzke]

ping @danielfoehrKn @MrBatschner

[MrBatschner]

@danielfoehrKn @MrBatschner Could you check the PR description and tell whether this is still up-to-date/the plan you are pursuing?

Sorry for the delay in answering. The PR linked above is meant to configure containerd and kubelet on Garden Linux to use systemd as cgroup driver if cgroup v2/unified hierarchy is detected (as Garden Linux 934 came with cgroup v2 support). However, this is just for Garden Linux. The CoreOS/FlatCar extension contains similar code and eventually, the Ubuntu and cHost extensions would require such code too - especially since the systemd announced to drop cgroup v1 support in a future release. Instead of having quite similar code to deal with the cgroup versions in each operating system extension separately, we should explore if this can be moved into the OS config part of Gardener itself - and that is the direction @danielfoehrKn and myself were looking into so far. Thus, this issue in my opinion is still relevant.