How to Secure Deployments in Kubernetes?

This blog was originally published by ARMO here.

Written by Leonid Sandler, CTO & Co-founder, ARMO.

Security is crucial ‌for containerized applications that run on a shared infrastructure. With more and more organizations moving their container workloads to Kubernetes, K8s has become the go-to platform for container orchestration. And with this trend comes a growing number of ‌threats and new ways of attack that necessitate strengthening all layers of security.

In Kubernetes, there are two aspects to security: cluster security and application security. We have already covered cluster security in a separate post. In this post, we’ll explore how to secure ‌Kubernetes deployments and applications in general.

Brush Up on the Basics

Just to quickly recap the basics here: A pod is the logical atomic unit that runs one or more containers in the cluster; it is wrapped by other resources, such as ReplicaSet, Deployment, StatefulSets, etc. There are various ways to improve the security posture of applications running in Kubernetes.

In a Kubernetes deployment, the template section contains the pod specs, which define the workload this deployment has to run. Several security-relevant sections are highlighted in bold in the template below:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx
spec:
 selector:
   matchLabels:
     app: nginx
 template:
   metadata:
     labels:
       app: nginx
   spec:
     serviceAccountName: nginx-sa
     securityContext:
       runAsUser: 1000
       runAsGroup: 3000
       fsGroup: 2000
     containers:
     - name: nginx
       image: my-private-registry.io/nginx:1.34
       resources:
         limits:
           memory: "128Mi"
           cpu: "500m"
       ports:
       - containerPort: 80
       securityContext:
         allowPrivilegeEscalation: false
         seccompProfile:
           type: RuntimeDefault
         capabilities:
           add: ["NET_ADMIN", "SYS_TIME"]
         seLinuxOptions:
           level: "s0:c123,c456"

Now, let’s take a closer look at these sections in the pod specs.

Service Account

When a process inside the container communicates to the API server, you should use the service account to authenticate. If you don’t define a service account for the pod, the default one will be used. Creating an application-specific service account with minimal privileges required to perform the function is recommended. If you choose to give roles to the default service account, those privileges will be available to every pod that doesn’t define a service account in the specs. This can inadvertently allow over-permissions to other applications and is therefore not recommended. In Kubernetes version 1.6 and above, you can opt-out of automounting API tokens for a service account in the container by setting

automountServiceAccountToken: false. 
For example:

apiVersion: v1
kind: ServiceAccount
metadata:
 name: nginx-sa
automountServiceAccountToken: false

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
 resources: ["pods"]
 verbs: ["get", "watch", "list"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 name: read-pods
 namespace: <strong>default</strong>
subjects:
- kind: ServiceAccount
 name: nginx-sa # "name" is <strong>case </strong>sensitive
 namespace: <strong>default</strong>
roleRef:
 kind: Role
 name: pod-reader
 apiGroup: rbac.authorization.k8s.io
---

Security Contexts

Security contexts define privileges and access-control settings in the pod and containers. Here’s a list of a few important ones:

• seLinuxOptions (Security-Enhanced Linux): Applies mechanisms that provide more granular access and control policies
• runAsUser and runAsGroup: Specific UserID or GroupID (UID and GID) to run the entry point of the container process; if unspecified, each default to the user-specified in image metadata (neither work in Windows containers).
• privileged: Runs a container in privileged mode, with false default; same as root (with all the capabilities) on the host
• runAsNonRoot: Container has to run as a non-root user (if the UID is 0 when Kubelet validates at ‌‌runtime, the containers will fail to start).
• capabilities: To add or drop capabilities when running containers; the container runtime grants capabilities, which are the default.
• procMount: Specifies the proc mount type for the containers and defaults to DefaultProcMount; this uses container runtime defaults for read-only and masked paths.
• AppArmor: Similar to SELinux, the capabilities of individual programs can be restricted via profiles.
• seccompProfile: The secomp options to use by the container; filter the system calls of the process
• readOnlyRootFilesystem: Mounts the root filesystem in the container as readOnly; default is false
• AllowPrivilegeEscalation: Dictates if a process can gain more privileges than its parent process; if the container runs as Privileged or with CAP_SYS_ADMIN capability, it is always true. This field must be explicitly set to false, because its default behavior may be changed in PSP.

Images

Source images are often taken from various public repositories; developers put their application code on top of these base images. You can also deploy OOTB applications directly from popular public registries.

There are three things to bear in mind about images, which we’ll discuss below.

Source of the Image

Make sure you're sourcing the images from a trusted registry. An attacker can place a malicious image in a public registry, which in turn can lead to problems such as data leaks or the attacker gaining access to your cluster. A number of public images have been found to be infected by crypto miners as well.

As an organization, you can create golden images of a base image and share them with developers, who can then safely use them from their internal repositories.

Distroless and Container-Optimized Images

These images are secure and optimized to run in containers, providing a reduced surface area for a potential attack. They contain only your application and dependent libraries, while package managers, shells, and programs typically available on Linux OS are removed.

Continuous Vulnerability Scanning

It’s highly recommended to implement continuous image scanning to detect vulnerabilities, malware, and other security threats (for example, unauthorized connections to untrusted networks) in‌ your container images. There are a number of available‌ security solutions, including Kubescape.

Pod Security Admission

You might have heard of PodSecurityPolicies, which is now being deprecated and will be removed in Kubernetes 1.25. Pod Security Admission (PSA) which will replace it, handles security and other security-related requirements. It defines different isolation levels for pods, such as privileged, baseline, and restricted. PSA is currently in beta as of 1.23.

These levels are described in detail in Kubernetes’ Pod Security Standards, but here below is a summary from Kubernetes’ own documentation:

Pod Security admission works with a built-in Pod Security admission controller; you need to enable this in the cluster using --feature-gates=”...,PodSecurity=true” or by using the pod admission webhook. It’s applied at the namespace level, with the following labels:

In the namespace, use the following annotations to enable Pod Security admission:

# MODE can be one of enforce, audit or warn
# LEVEL can be one of privileged, baseline or restricted
# VERSION must be valid Kubernetes minor version or latest.
pod-<strong>security</strong>.kubernetes.io/<<strong>MODE</strong>>: <<strong>LEVEL</strong>>
pod-<strong>security</strong>.kubernetes.io/<<strong>MODE</strong>>-version: <VERSION>

Using Secrets

If you have sensitive information (like credentials, tokens, encryption keys, and certificates) available in the application, use Kubernetes Secrets. You can create Secrets with the literal values or the files and then mount them in the pod. Do not store such information in the container images and Git repositories.

When using Secrets, it’s also best not to use environment variables to project the credentials into the container—use files instead.

Keep in mind that the Secrets are base64-encoded values. They’re not encrypted, so access to security objects must be restricted, and you should enable encryption at write in the API server.

Conclusion

Kubernetes provides various ways to improve your organization’s security posture. Developers need to consider these constructs to make their applications safer.

To recap:

1. Use service accounts per application and bind the service account with minimal roles and privilege requirements to achieve your objectives.
2. Don’t automount the service account token if it’s not required in your application.
3. Use security contexts to implement various techniques, such as preventing containers from running as the root user in privilege mode, using SELinux or AppArmor profiles, and more.
4. Make sure the source of your container images is trustworthy, and if possible, store them in private registries.
5. Try to use container-optimized images that reduce surface area to minimize threats.
6. Deploy a continuous vulnerability scanning solution, not only in CI/CD but in the cluster as well, that can monitor and take action in real-time.
7. Use a Pod Security admission profile and model to provide different isolation levels to your workload.
8. Use Secrets to store sensitive information, and apply least-privilege RBAC to restrict user/SA secret access.

This can all appear overwhelming to application developers. Tools like Kubescape can help with risk analysis, enforcement of security standards in ‌CI/CD, easy-to-understand RBAC visualizations, automated vulnerability scanning, and much more. Kubescape assists developers in achieving secure deployments of their applications.