Kubernetes v1.37 Adds Scheduler Preemption for In-Place Pod Resize

Kubernetes v1.37 introduces scheduler preemption for in-place Pod resize, currently in Alpha. This feature allows the Kubernetes scheduler to actively free up capacity on a fully-utilized node by preempting lower-priority workloads, enabling the pending in-place resizes of critical, higher-priority applications to succeed. The preemption mechanism is activated through the InPlacePodVerticalScalingSchedulerPreemption feature gate, providing a dynamic solution for maintaining critical workloads without disruptive restarts.

Understanding the Deferred Resize Challenge

In Kubernetes, resource allocation has traditionally been a static process, determined at the time of a Pod's initial scheduling. With the in-place Pod resize feature, which reached General Availability in v1.35, developers and operators could dynamically adjust CPU and memory allocations for running containers, avoiding the need for disruptive restarts. However, this introduced a unique challenge: if a running Pod requested resources beyond what the node could currently allocate, the request was marked as Deferred. This meant that the Pod would wait indefinitely for resources to free up on that node, potentially leading to service disruptions if critical applications could not scale as needed.

Before this update, administrators faced several limited options to manage deferred resize requests:

• Manually evict lower-priority Pods to free up resources.
• Rely on a cluster autoscaler to provision a larger node, which could be disruptive as it might involve rescheduling the Pod.
• Implement custom autoscaling solutions that could dynamically adjust node resources or trigger preemption.

These methods often contradicted the core benefit of in-place resizing: avoiding restarts. This operational dilemma forced a choice between maintaining buffer capacity for high-priority workloads or risking blocked resize requests due to resource constraints.

Why Scheduler Preemption Matters

In a production Kubernetes environment, efficient resource utilization is crucial. A common strategy is to bin-pack unused capacity on nodes with lower-priority workloads, such as batch jobs or background tasks. However, without scheduler preemption for in-place resizing, this could lead to operational challenges. Lower-priority workloads might consume all available headroom on a node, blocking higher-priority applications from scaling when needed. This feature allows administrators to confidently maximize resource utilization across their clusters without risking the performance of critical workloads.

With scheduler preemption for in-place Pod resize, the Kubernetes scheduler can automatically preempt lower-priority Pods to clear headroom for high-priority workloads, ensuring that critical services remain responsive and reliable. This not only enhances cluster utilization but also aligns with the goal of cost efficiency by leveraging available resources effectively.

How Scheduler Preemption Works

The scheduler preemption feature integrates directly into the core scheduling cycle, allowing the Kubernetes scheduler to actively manage resource allocation on fully-utilized nodes. It preempts lower-priority workloads to free up capacity for higher-priority Pods needing in-place resizes. Here's how it functions:

Centralized Scheduler Tracking

The kube-scheduler now monitors Pods with a deferred resize status, keeping them in active scheduling evaluations to trigger preemption. This ensures that the scheduler continuously tracks these Pods until the resize is successfully completed. Normally, Pods with a defined spec.nodeName are considered placed and bypass the scheduler, but under this feature gate, the scheduler intercepts these Pods to facilitate preemption.

Single-Node Preemption Boundary

Preemption for in-place resizing is localized to the Pod's assigned node. The scheduler identifies eligible lower-priority Pods for eviction on the same node, freeing up local capacity. If even after evicting these Pods, the node cannot accommodate the resize, the request remains deferred. This localized approach ensures that the scheduler does not need to evaluate all nodes in the cluster, focusing only on the node where the resize is required.

Resource Reservation Safety

To prevent double allocation and scheduling races, the scheduler treats resources requested for a resize as already consumed. This allows the Kubelet to actuate the resize once the preemption occurs. By reserving the requested resources in advance, the scheduler prevents other Pods from claiming them during the preemption process.

Separation of Concerns

With this feature, the Kubelet delegates preemption decisions entirely to the scheduler. This separation ensures that resize-related preemption logic is centrally orchestrated, respecting global priorities and Pod disruption budgets. The Kubelet's critical Pod admission handler, which can perform local evictions for initial Pod admissions, does not engage in preemption for in-place resizes, leaving this responsibility to the scheduler.

Rolling This Out on Your Cluster

To leverage scheduler preemption for in-place Pod resize, ensure your cluster runs Kubernetes v1.37 or later. The feature gate InPlacePodVerticalScalingSchedulerPreemption must be enabled across all control plane components and the kubelet. Here’s a step-by-step guide to test this feature:

1. Create a kind cluster with the feature gate enabled by configuring it in a file named kind-config.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  InPlacePodVerticalScalingSchedulerPreemption: true

Create the cluster using this configuration:

kind create cluster --config kind-config.yaml --image kindest/node:v1.37.0

Ensure the node image corresponds to version 1.37 or later.

2. Create PriorityClasses and deploy Pods:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "High priority workload"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: low-priority
value: 1000
globalDefault: false
description: "Low priority workload"
---
apiVersion: v1
kind: Pod
metadata:
  name: low-priority-pod
spec:
  priorityClassName: low-priority
  containers:
  - name: worker
    image: nginx
    resources:
      requests:
        cpu: "3"
        memory: "500Mi"
      limits:
        cpu: "3"
        memory: "500Mi"
---
apiVersion: v1
kind: Pod
metadata:
  name: high-priority-pod
spec:
  priorityClassName: high-priority
  containers:
  - name: app
    image: nginx
    resources:
      requests:
        cpu: "4"
        memory: "1Gi"
      limits:
        cpu: "4"
        memory: "1Gi"

Apply this manifest:

kubectl apply -f preemption-demo.yaml

3. Patch the high-priority Pod to increase its CPU request:

kubectl patch pod high-priority-pod --subresource resize --patch \
  '{"spec":{"containers":[{"name":"app", "resources":{"requests":{"cpu":"6"}, "limits":{"cpu":"6"}}}]}}'

4. Inspect the preemption event on the low-priority Pod to verify that preemption occurred.

kubectl get events --field-selector involvedObject.name=low-priority-pod

Limitations to Consider

While scheduler preemption for in-place Pod resize offers significant benefits, there are limitations. The preemption is localized to a single node, meaning that if a node cannot accommodate a resize even after preempting all lower-priority workloads, the request remains deferred. This feature also requires enabling a feature gate, which means it is not yet part of the stable Kubernetes release and may undergo changes.

Additionally, administrators may need to carefully configure node-level preemption policies to balance between automatic preemption and other resource management strategies. This feature is best suited for clusters where high-priority workloads need guaranteed resource availability without the risk of disruption.

For more insights into Kubernetes updates, you can check our previous posts on enhancements in Kubernetes v1.37 and other related topics.