Kubernetes v1.35: Transitioning from cgroup v1 to v2

The recent shift to cgroup v2 in Kubernetes, marked by the deprecation of cgroup v1 in Kubernetes v1.35, is a significant change for administrators and developers alike. As of this version, Kubernetes has set failCgroupV1 to true by default, meaning that kubelet will not start on a node using cgroup v1 unless explicitly configured otherwise. This move aims to leverage the more streamlined and consistent resource management capabilities offered by cgroup v2.

Why This Matters

For those maintaining production codebases in Kubernetes, understanding the implications of this shift is crucial. Cgroup v2 provides a single unified hierarchy and a more consistent interface, which enhances resource isolation and management. This is particularly important for applications that require precise resource allocation, such as those running in containerized environments where multiple applications share the same resources.

The deprecation of cgroup v1 means that if you’re running an older version of Kubernetes, you must migrate your Linux nodes to cgroup v2 before upgrading to v1.35 or later. Failing to do so will result in kubelet startup failures under default configurations.

Steps for Migrating to cgroup v2

Here are some practical steps you can take to migrate to cgroup v2:

• Ensure your Kubernetes version is at least v1.25, as this is when support for cgroup v2 became stable.
• Verify that your Linux kernel is version 5.8 or later; 5.9 is recommended if you plan to use memory QoS features.
• Update your container runtime to a version that supports cgroup v2, such as containerd v1.4 or later, or CRI-O v1.20 or later.
• Configure the kubelet and container runtime to use the correct cgroup driver. If you're using kubeadm, the recommended driver is systemd.
• For kubeadm-managed clusters, be prepared for stricter checks during kubeadm init, join, and upgrade commands when cgroup v1 is detected.
• Set failCgroupV1: false temporarily in the kubelet configuration if you need to run cgroup v1 nodes during the transition, but plan for its removal according to the Kubernetes deprecation policy.

Here's a sample configuration snippet for enabling tiered memory protection with Memory QoS:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  MemoryQoS: true
memoryReservationPolicy: TieredReservation
memoryThrottlingFactor: 0.9

Limitations and Considerations

While cgroup v2 offers many advantages, there are some limitations and considerations:

• Active_file memory is not considered available memory, which can cause the kubelet to report memory pressure for I/O-intensive workloads. This requires careful configuration of memory requests and limits.
• Memory QoS remains in alpha; enabling it in production should be approached with caution and thorough testing.
• Ensure that your system and software are compatible with cgroup v2, especially if they directly interact with the cgroup filesystem. Software such as cAdvisor, Java, Node.js, and automaxprocs may require updates to work correctly with cgroup v2.
• Older kernels might experience issues like the memory.high livelock when Memory QoS is enabled. The kubelet will log a warning if it detects a potentially affected kernel.

Practical Steps for Kubernetes Administrators

Administrators should take the following actions to ensure a smooth transition:

• Audit current Kubernetes node configurations to identify any that still use cgroup v1. This will help you plan your migration strategy and prioritize nodes that need upgrading.
• Plan kernel and container runtime upgrades where necessary to support cgroup v2. This involves ensuring that your kernel is up to date and that your container runtime is compatible with cgroup v2.
• Test configurations in a staging environment before deploying them to production, particularly if you are enabling new features like Memory QoS. This helps catch any potential issues early and allows for adjustments before impacting production workloads.
• Review and update any monitoring tools or scripts that interact with cgroup metrics, as the transition from v1 to v2 can affect these. For instance, Pressure Stall Information (PSI) reporting requires cgroup v2 and a kernel configured with CONFIG_PSI=y.
• Check the compatibility of any third-party tools or libraries you use that might rely on cgroup v1 specifics, such as cpu.shares, and adjust them to accommodate cgroup v2's cpu.weight.

By following these steps, you can mitigate risks associated with the transition and take full advantage of the improvements that cgroup v2 offers. In my experience, thorough testing and careful planning are crucial in such transitions to prevent unforeseen issues in production environments.

Container-aware OOM Handling and Rootless Support

With cgroup v2, Kubernetes enhances container-aware Out-Of-Memory (OOM) handling. By default, singleProcessOOMKill is set to false, meaning an OOM event will kill all processes in a container as a group. This behavior, scoped to container cgroups rather than the entire Pod, prevents partial container failures and aligns with the resource isolation goals of cgroup v2. However, if you need the cgroup v1 behavior, where the kernel may kill one process at a time, you can set singleProcessOOMKill: true.

Additionally, cgroup v2 supports rootless containers more securely through systemd delegation. This is separate from Kubernetes's Pod user namespaces and allows for safer delegation of controllers to less privileged users, aligning with modern security practices. In cgroup v1, delegating controllers to less privileged containers could pose security risks. However, in cgroup v2, this delegation is officially supported, reducing the potential for privilege escalation.

eBPF and Cgroup BPF Integration

Another significant change with cgroup v2 is the implementation of device access controls using eBPF (extended Berkeley Packet Filter) instead of traditional interface files. This shift allows for more flexible and efficient management of device access within containers. For instance, Cilium, a popular networking solution for Kubernetes, utilizes BPF programs for socket-based load balancing, with its default cgroup root being /run/cilium/cgroupv2. This integration with eBPF reflects the ongoing evolution of Kubernetes towards more sophisticated and secure resource management mechanisms.

Pressure Stall Information (PSI) and Monitoring Enhancements

Pressure Stall Information (PSI) is another feature that benefits from cgroup v2. PSI reports CPU, memory, and I/O contention at the node, Pod, and container levels, providing a granular view of resource pressure. This feature requires Linux 4.20 or later with CONFIG_PSI=y enabled, and PSI data is exposed through the kubelet's Summary API and /metrics/cadvisor. This enhancement allows administrators to monitor resource usage more effectively and take preemptive action to alleviate resource bottlenecks.

In-place Resource Updates

In-place resource updates, which became stable in Kubernetes v1.35, are further enhanced in v1.36 with the default enablement of in-place vertical scaling for Pod-level resources. This feature, which requires cgroup v2 for accurate enforcement, allows for dynamic adjustment of resource limits and requests without Pod restarts. The kubelet coordinates these changes to ensure that increases create headroom before container limits grow, and decreases constrain containers before shrinking the Pod-level boundary. This capability offers more flexibility in managing resource allocations dynamically, reducing disruptions to running workloads.

By embracing these new features and capabilities, Kubernetes users can achieve more efficient resource management and isolation, paving the way for more robust and scalable containerized applications. However, it is essential to remain vigilant about the compatibility and stability of these features, particularly those in alpha or beta stages, to ensure they meet the specific needs and constraints of your production environments.