Kubernetes v1.37: Securing Storage with New Bind Mount Options

Kubernetes v1.37 introduces crucial developments in container storage security, specifically bind mount options and emptyDir permission modes. These features equip application developers and security engineers with the ability to implement strict security policies directly in Kubernetes. This post will explore what these changes mean for your projects and how you can use them to make your workloads more secure.

What Changed in Kubernetes v1.37?

The new features in Kubernetes v1.37 focus on enhancing security by allowing for finer control over how storage volumes are mounted and accessed. Bind mount options now include flags such as noexec, nosuid, and nodev, which help prevent the execution of arbitrary binaries and restrict special device interpretation. Additionally, the emptyDir volume type now supports setting permission modes, including the sticky bit, to control file deletion rights.

Bind Mount Flags

The introduction of bind mount flags like noexec, nosuid, and nodev marks a major step forward. Each serves a distinct purpose:

• noexec: Prevents the execution of binaries within the mounted filesystem. This is crucial for mitigating risks associated with malicious code execution if a container is compromised.
• nosuid: Disallows the use of set-user-identifier or set-group-identifier bits, which helps prevent privilege escalation attacks within the container environment.
• nodev: Ensures that character and block special devices are not interpreted on the filesystem, further tightening security.

emptyDir Volume Permissions

The emptyDir volume type, often used for temporary storage, now supports permission modes. Previously, these volumes defaulted to a mode of 0777, allowing any process to read, write, and delete files within the volume. With the addition of permission modes, including the sticky bit (01777), there's now much-needed control over file operations within shared directories.

Why These Changes Matter

In containerized applications, security is paramount. Traditionally, Kubernetes volumes lacked the necessary security features to prevent unauthorized file execution and modifications across shared directories. This change is important because it lets developers enforce security policies directly in Kubernetes without using complex workarounds such as modifying container images or using init containers to set permissions.

For example, by setting the noexec flag, you ensure that even if a malicious payload is downloaded into a writable volume, it cannot be executed. Similarly, setting the sticky bit on an emptyDir prevents unauthorized deletion of files, which is particularly useful in multi-container pods where shared scratch space is common.

Real-World Use Cases

These features are particularly beneficial in scenarios where security is a top priority:

• Preventing Privilege Escalation on Writable Mounts: By configuring volumes with nosuid and noexec, developers can ensure that even if an application is compromised, the potential for privilege escalation is minimized.
• Securing Shared Scratch Space in Multi-Container Pods: Setting the sticky bit (01777) on emptyDir volumes allows multiple containers to share a workspace without the risk of one container deleting another's files.
• Enforcing Principle of Least Privilege for Application Data: By setting a mode such as 0750, developers can restrict access to specific users and groups, ensuring that only authorized processes can interact with the data.

Implementing Storage Security in Your Kubernetes Environment

To implement these new storage security features, you must first enable the Alpha feature gates VolumeBindMountOptions and EmptyDirVolumeMode on your API server and kubelet. Once enabled, you can start defining your storage policies in your Pod manifests.

Example: Enforcing Bind Mount Options

apiVersion: v1
kind: Pod
metadata:
  name: hardened-bindmount-pod
  namespace: default
spec:
  os:
    name: linux
  containers:
    - name: hardened-app
      image: alpine:latest
      command: ["sleep", "3600"]
      securityContext:
        readOnlyRootFilesystem: true
      volumeMounts:
        - name: temp-storage
          mountPath: /tmp
          bindMountOptions:
            - noexec
            - nosuid
  volumes:
    - name: temp-storage
      emptyDir: {}

This YAML manifest demonstrates how to use bind mount options to prevent the execution of binaries and the use of set-user-identifier bits on a mounted emptyDir volume.

Example: Setting EmptyDir Volume Permissions

apiVersion: v1
kind: Pod
metadata:
  name: hardened-emptydir-pod
  namespace: default
spec:
  os:
    name: linux
  containers:
    - name: app-container
      image: alpine:latest
      command: ["sleep", "3600"]
      volumeMounts:
        - name: shared-tmp
          mountPath: /tmp
  volumes:
    - name: shared-tmp
      emptyDir:
        mode: 01777

Here, the sticky bit is set on the emptyDir volume to mimic traditional Unix /tmp directory behavior, ensuring files cannot be deleted by non-owners.

Verifying the Features

Once implemented, you can verify these security features by attempting to execute blocked actions. For instance, try creating and executing a script in a volume mounted with noexec. You should see a permission denied error, confirming that execution is disabled.

Verifying noexec

To verify that the noexec flag is working, you can perform the following steps:

1. Exec into the pod using kubectl exec -it hardened-bindmount-pod -- sh.
2. Create an executable script on the mounted volume:

cd /tmp
echo '#!/bin/sh' > test.sh
echo 'echo "Executing untrusted code..."' >> test.sh
chmod +x test.sh

3. Attempt to run the script using ./test.sh.
4. Observe the expected result: sh: ./test.sh: Permission denied. This indicates that the noexec flag is effectively preventing execution.

Verifying the Sticky Bit

To verify the sticky bit functionality, follow these steps:

1. Exec into the pod using kubectl exec -it hardened-emptydir-pod -- sh.
2. Verify directory permissions on /tmp using ls -ld /tmp. The output should show drwxrwxrwt, indicating the sticky bit is set.
3. Create a file as a guest user:

su -s /bin/sh -c "touch /tmp/guest_file" guest

4. Attempt to delete the file as another user, e.g., nobody:

su -s /bin/sh -c "rm /tmp/guest_file" nobody

5. Observe the expected result: rm: can't remove '/tmp/guest_file': Operation not permitted. This confirms that the sticky bit is effectively preventing unauthorized deletions.

Limitations and Considerations

While these features significantly enhance security, they come with some limitations. They are currently in Alpha, so you need to enable the feature gates explicitly. Additionally, these features are Linux-specific and do not apply to Windows nodes. Also, the use of fsGroup in a Pod's security context may override the emptyDir mode settings.

Version skew can be an issue; while the API server might accept these fields, they may be ignored if the kubelet isn't updated. Ensure that your cluster's nodes support the necessary runtime features to avoid deployment issues.