How to mitigate Kubernetes Insecure Workload Configuration

Problem

Dockerfile, the magic piece of text that defines how to build a container image, is nowadays more often written by developers. When the developer writing it is less security-minded, or trained on topics as: Linux Kernel and containers isolation in Kubernetes and how to reduce Kubernetes attack surface, chances are pretty big your software will make it to production with a security vulnerability called in Kubernetes security literature: container escape or container breakout or insecure workload configuration. A malicious user could then exploit this vulnerability in order to gain access to the host Kernel, in Kubernetes also called worker node. From there the attack surface is pretty large.

Even if you’re using Java Image Builder (Jib), a developer friendly container tool from Google, to containerize your Java application without a Dockerfile, as I’ve seen in a project I’ve consulted over the summer, the person who introduced JIB in the codebase of the project, forgot to use more advance configurations like the container user.

Containers under the hood sketches

Container image using the root user to run PID 1 - if that process is compromised, the attacker has root in the container, and any mis-configurations become much easier to exploit.

Solution

To mitigate container escapes, build containers running as non-root, meaning there will be an extra step to get root access from user access, to be able to break out of the container.

Below Dockerfile defines how to invoke useradd with the -u (–uid) option Linux command to create a user with name subway and UID 1000. Base image in Dockerfile is IBM released Semeru Runtimes based on OpenJ9 Java Virtual Machine (JVM).

Example is for Java 17 on Linux (Ubuntu Jammy Jellyfish)

FROM ibm-semeru-runtimes:open-17.0.4.1_1-jre-jammy

RUN useradd -u 1000 subway

USER 1000
...

[Listing 1 - Dockerfile definition snippet for a base Java service image]

Kubelet in k8s version 1.21 requires USER uid instead of USER username to avoid Error: container has runAsNonRoot and image has non-numeric user (subway), cannot verify user is non-root

Configure a security context for pod or container in your Kubernetes workload definition, and prevent privilege escalation.

  template:
    spec:
      containers:
        name: payments
        securityContext:
          allowPrivilegeEscalation: false
          runAsNonRoot: true

[Listing 2 - Snippet from Kubernetes Deployment workload resource definition of Java payments container]

Verify that container is using the defined non-root user to run PID 1, by acquiring a shell to the running container.

In example at hand, pod has only one container payments

When creating a new user, the default behavior of the useradd command is to create a group with the same name as the username, and same GID as UID.

Reference Shelf

Above was just the tip of the iceberg. If you want to learn more on Kubernetes application security best practices and exploits, you might find useful:

• 11 Ways (Not) to Get Hacked
• How to stop running containers as root
• How to Stop Container Escape and Prevent Privilege Escalation
• Open Web Application Security Project: 2022 Top 10 Kubernetes Security Vulnerabilities
• Snyk 10 Kubernetes Security Context settings you should understand
• Configure a Security Context for a Pod or Container