Minimal kernel version plan: probably 4.18/4.19+

[pacoxu]

Summary

checked means EOF

• 3.10, Current System Validator check
• 3.16, Namespaced IP Local Reserved Ports (safe-sysctl)
• 3.18, containerd suggestion when using btrfs.(containerd, btrfs)
• 4.x, containerd suggested min kernel version (containerd, core code and snapshotters)
• 4.1, reuse min supported kernel version (kube-proxy)
• 4.5, namespaced net.ipv4.tcp_keepalive_time (safe-sysctl)
• 4.9 LTS, EOF in Jan 2023. (linux, LTS)
• ~~4.11 EOF~
  • calico flex can reduce CPU usage by setting iptablesRefreshInterval (calico,cpu)
  • net.ipv4.ip_unprivileged_port_start (safe-sysctl)
• 4.14 LTS EOF in Jan 2024 (linux, LTS)
• 4.19 / 4.18 RHEL 🌟🌟🌟
  • cilium bump in v1.4 for eBPF supports(cilium,ebpf)
  • RSS problem with docker(fixed in 4.19.77 or 5.2.19+, 5.3.4+) (docker)
  • 4.19 LTS EOF Dec 2024. (linux, LTS)
• 5.4 LTS EOF Dec 2025
• 5.8+, Cgroup V2 suggestions (cgroup v2) 🌟🌟
  • Move cgroup v1 support into maintenance mode Move cgroup v1 support into maintenance mode enhancements#4572
• 5.10 LTS EOF Dec 2026
• 5.12
  • Recursively Read-only (RRO) hostPath mounts (RRO)
  • Support idmapped mounts (UserNamespace)
• 5.15 LTS EOF Oct 2026
• 6.1 LTS Dec 2026

What would you like to be added?

Currently, there is a kernel version check-in system validator that is used by kubeadm. (// Requires 3.10+, or newer)

• This issue aims to track which version should minimum kernel version be raised to. The current status and the plan.

kubernetes/vendor/k8s.io/system-validators/validators/types_unix.go

Line 31 in c9ff286

Versions: []string{`^3\.[1-9][0-9].*$`, `^([4-9]|[1-9][0-9]+)\.([0-9]+)\.([0-9]+).*$`}, // Requires 3.10+, or newer

During the weekly sig-node meeting, there was a discussion about adding a kernel-version-sensitive safe sysctl in kubelet. It was noted that 3.16 is considered to be a very low kernel version and it was suggested that we introduce a minimal kernel version for Kubernetes. Alternatively, we should have a warning for low kernel versions(I think that we should add a warning if the kernel version is less than 3.18 or 4.0).

• Cgroups v2 GA opens up the possibility for new features but sets the minimal bar for dependencies again.
  • Cgroup v2 suggests Linux Kernel version is 5.8 or later. https://kubernetes.io/docs/concepts/architecture/cgroups/

kubernetes/pkg/kubelet/sysctl/safe_sysctls.go

Line 25 in c9ff286

const ipLocalReservedPortsMinNamespacedKernelVersion = "3.16"

Why is this needed?

Some historical issues related and some CNCF projects like cilium: #30706

• Bump minimal kernel version to 4.19 / 4.18 RHEL cilium/cilium#22116
• cluster: ipvs conntrack module vs kernel version #89327

kubernetes/pkg/proxy/ipvs/proxier.go

Line 93 in c9ff286

connReuseMinSupportedKernelVersion = "4.1"

kubernetes/cluster/addons/calico-policy-controller/felixconfigurations-crd.yaml

Lines 320 to 324 in c9ff286

	refresh. Note: the default for this value is lower than the other
	refresh intervals as a workaround for a Linux kernel bug that was
	fixed in kernel version 4.11. If you are using v4.11 or greater
	you may want to set this to, a higher value to reduce Felix CPU
	usage. [Default: 10s]'

https://github.com/containerd/containerd#runtime-requirements

There are specific features used by containerd core code and snapshotters that will require a minimum kernel version on Linux. With the understood caveat of distro kernel versioning, a reasonable starting point for Linux is a minimum 4.x kernel version.

[pacoxu]

/sig node
/priority important-longterm

[pacoxu]

A new use case is #117873 (comment) which is scoped sysctl since v4.5.

[pacoxu]

/triage accepted

[pacoxu]

"KEP-3857: Recursively Read-only (RRO) hostPath mounts"

The "rro" bind mount options is implemented by calling mount_setattr(2)
with MOUNT_ATTR_RDONLY and AT_RECURSIVE.

Requires runc >= 1.1 && kernel >= 5.12.

See https://github.com/kubernetes/enhancements/pull/3858/files#diff-a5e3a174567e889120ab32af5f159c2bcddd459cc36abdf56f3eceb5ad86d6c5R183 for more details.

[pacoxu]

net.ipv4.ip_unprivileged_port_start requires kernel version 4.11 or higher.

• See Use ip_unprivileged_port_start=53 for coredns #105309 (comment).
• net.ipv4.ip_unprivileged_port_start is supported in kubernetes sysctl since v1.22. It has been a safe sysctl since in Kubernetes v1.22. Mark net.ipv4.ip_unprivileged_port_start as a safe sysctl #103298
• containerd supports the option in CRI since containerd v1.6: CRI: Support enable_unprivileged_icmp and enable_unprivileged_ports options containerd/containerd#6170
  and containerd plans to make it default in v2.0 [2.0] Enable enable_unprivileged_icmp and enable_unprivileged_ports by default containerd/containerd#6924
• cri-o has no limitation for it.

xref #56374 and #102612 for more details.

[pacoxu]

https://endoflife.date/linux

Some status from the Linux community :

• 4.9 (LTS) | 6 years ago(11 Dec 2016) | Ended 6 months and 2 weeks ago(07 Jan 2023)

[pacoxu]

Containerd update: containerd/containerd#5890 Support idmapped mounts (kernel 5.12)

[danwinship]

#120895 discusses moving the existing GetKernelVersion API out of pkg/proxy/ipvs (since it's also being used by kubelet); it seems like wherever we move that to could easily become the place where we document the required/preferred/optional kernel versions for different components...

[tzneal]

/cc

[pacoxu]

docker/for-linux#693 (A RSS problem in old kernel with docker. Not sure if it will be with other container runtime)

Then found torvalds/linux@f9c6456 which was included in 4.19 kernel from 4.19.77 (and in newer ones since 5.2.19 and 5.3.4)

[pacoxu]

https://endoflife.date/linux

Key kernel versions:

• ✅ 3.10, Current System Validator check
• 4.x, containerd suggested min kernel version (containerd, core code, and snapshotters)
• 4.9 LTS, EOF in Jan 2023. (Linux, LTS)
• 4.14 LTS EOF in Jan 2024 (Linux, LTS)
• 🎯 4.19 LTS EOF Dec 2024. (Linux, LTS)
• 5.4 LTS EOF Dec 2025
• 5.8+, Cgroup V2 suggestions (cgroup v2) 🌟🌟
• 5.10 LTS EOF Dec 2026
• 5.15 LTS EOF Oct 2026
• 6.1 LTS Dec 2026

According to key points and Linux LTS status, I would suggest to change the suggested minimal kernel version to 4.19 in Kubernetes v1.30 to follow the EOL date of Linux. Any inputs?

[danwinship]

The RHEL 8 kernel is 4.18 plus backports of way more than what's in 4.19...

Presumably the current minimum kernel is 3.10 because that's the nominal version of the RHEL 7 kernel, so having the new minimum version be the nominal version of the RHEL 8 kernel seems like it makes sense...

[pacoxu]

Low-level ARM specific problems (example: Instant::now() was 70x slower on ARM before this Pull Request to Rust: rust-lang/rust#88652)

[pacoxu]

The cgroup v1 deprecation in 1.30 was discussed in recent sig node meeting.

• BTW, Cgroup v2 suggests Linux Kernel version is 5.8 or later. https://kubernetes.io/docs/concepts/architecture/cgroups/.

[pacoxu]

#118996 (kubernetes/release#3076) will land in Kubernetes 1.29, containing a switch to debian:bookworm. This means we now require glibc 2.36 (compared to 2.31 on bullseye) for non static binaries like the kubelet.

kubernetes/release#3246 is about glibc.

[kannon92]

@pacoxu we have #124060 where we are exploring requiring tmpfs noswap option (where tmpfs mounts are not allowed to use swap memory).

[haircommander]

a limitation about this approach is often downstream kernels backport needed features so would technically be compliant. I think it's worth defining features we need and maybe guess which kernels support, but it's more robust to attempt to use the feature and smartly not use it if it's not available

[danwinship]

it's more robust to attempt to use the feature and smartly not use it if it's not available

That can work for features, but bugfixes are often impossible to autodetect without essentially doing an e2e test.

When we added pkg/util/kernel I had a vague idea that maybe there could eventually be an API like utilkernel.Require(utilkernel.IPVSConnReuseModeFixed), where the argument is an array of regexps and minimum versions so you can say "if the kernel version matches /.*\.el9\..*/ then the version must be greater than 5.14.0-322 or else if it matches /.*/ then the version must be greater than 6.1".