How to Secure Kubernetes Pods Post-PSPs Deprecation

This blog was originally published by ARMO here.

Written by Amir Kaushansky, VP Product, ARMO.

Kubernetes pods are the basic building blocks of Kubernetes. It's managing one or more tightly coupled application containers allowing them to share resources and networks. Pods are hosted on nodes, which are either physical or virtual machines.

When defining a Pod we need to think not only about how much CPU or memory we want to assign to it but also about what would be the interaction between it and the underlying infrastructure. For example, we can allow a Pod to access all devices on the host, which might be a potential threat if someone gets inside of it.

That's why it's very important to secure pods to prevent that from happening.

In this blog, we will review ways to secure Pods, especially in light of the decision to deprecate PSPs in V1.25

Kubernetes Security

By default, nothing is protected in Kubernetes. This means we can create workloads capable of accessing the underlying infrastructure, posing a potential threat. There are several takes on this problem. We can limit access to clusters by introducing RBAC. When creating Pods, we can specify what capabilities they will have by setting permissions with a Pod security context. But this doesn’t cover everything.

Cluster admins want to enforce developers to create workloads with the lowest privileges possible. If developers happen to be aware that running a container in a Pod as root is a bad idea, those admins have an easy job. But this is often not the case. Many times, they either don't know all the potential threats or forget about a proper security configuration. People make mistakes—this is normal.

Pod Security Policies

To prevent issues from happening, the Kubernetes community introduced a built-in mechanism called a Pod Security Policy (PSP). On a cluster level, we can create a list of rules that new Pods need to meet to be accepted

Here is a simple example of PSP definition:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: simple-policy
spec:
  privileged: false
  runAsUser:
    rule: MustRunAsNonRoot
  readOnlyRootFilesystem: true

With that, three rules are being enforced:

• Disallow a container to run in a privileged mode
• Disallow processes to run in a container as a root user
• Disable making writes to a container file system.

These are not the only policies that we can choose from. There are plenty of them! All of them are listed in the official Kubernetes documentation.

Unfortunately, there are issues with PSPs themselves. The main problem is that they’re not intuitive to use. All policies need to be bonded both to the users (who create pods directly) and service accounts (like ReplicaSet). This needs to be remembered all the time, but it's easy to forget. Moreover, a PSP is applied to pods only. It can't be applied to a deployment, which means that nothing will prevent it from creating it. The resulting Pod may still not be created, but it could be hard to audit why it failed.

Finally, introducing a PSP to an already-running cluster is very challenging. All policies need to be created before enforcing a validation; and, in a huge system, it can be very tricky to figure out which permissions are required by all workloads. Even more of a headache is that PSPs do not apply to already-running pods, so you might only find a potential misconfiguration later on.

All of these things made the Kubernetes team decide to deprecate PSPs and remove them starting from v1.25 That's why you need to use other ways and tools to reach the same goal.

Pod Security Standards

After deprecating PSP, Kubernetes developers decided to take a different approach to security. Instead of plenty of fine-grained policies that can be combined, there are three standards from which we can choose: privileged, baseline, and restricted.

Privileged

This is an unrestricted policy without any rules. Users or service accounts can create any pod with special privileges. This is a good option when there is no need to be concerned about security at all (e.g., for local development). Also, you can use it for special applications that require modifying something on a host. You should limit the use of these policies to an absolute minimum—only when there is no other way—and treat them with extra caution.

Baseline

This provides a basic set of rules, as it prevents you from sharing a host namespace or running harmful Linux capabilities. It's a middle ground between the previous and next, highly secured, policy. It should be applied when you are concerned about potential threats, but not yet ready for full protection. You can use these, for example, for less exposed environments (like test environments). A list of permissions that this policy includes can be found in the official documentation.

Restricted

This contains all rules from the previous policy and adds, even more, to provide for best practices in pod hardening, for example, restricting you from running applications as a root. It's targeted to be applied to all critical applications that are running in production systems. Similar to the previous policy, all details about it can be found in the official documentation.

These are the only options we have to choose from, but they cover most real-life scenarios.

Pod Security Admission

Whenever a modifying kubectl command is run, a Kubernetes API server first needs to validate if it can be applied. This mechanism is called an admission controller, and it's checked after successful authorization. By default, each Kubernetes cluster has a couple of them turned on. They fulfill tasks such as removing all workloads in the case of a namespace deletion.

The one used to enforce policies defined in the PSS is called Pod Security Admission (PSA).

Enabling it is very easy. The only thing you need to do is add the pod-security.kubernetes.io/<mode>=<standard> label to a namespace. The<mode> defines what action will be applied if a policy is violated:

• Enforce - A pod will not be created if it violates a policy.
• Audit - A pod will be created, but an entry will be added to an audit log.
• Warn - A pod will be created, but a warning will be printed in the console.

The best way to start working with PSA is not to use the restricted policy with the enforce label at the beginning. Probably many already-existing pods would violate this standard, so it's better to start with only logging and warning users. This allows time for you to transition to more secure workload definitions.

Let's assume there is already a production namespace we want to apply:

kubectl label --overwrite ns production \ pod-security.kubernetes.io/enforce=baseline \ pod-security.kubernetes.io/warn=restricted \ pod-security.kubernetes.io/audit=restricted

For the privileged policy, we would use the privileged value.

Now, if we try to install any application that violates the restricted policy (here, it's installed with the popular tool Helm):

helm install postgres bitnami/postgresql -n production --version 11.1.28 --wait

The output we get is:

W0509 06:53:32.784714    5472 warnings.go:70<em>]</em> would violate PodSecurity 
"restricted:latest": allowPrivilegeEscalation != false (container "postgresql" must 
set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities 
(container "postgresql" must set securityContext.capabilities.drop=["ALL"]), 
runAsNonRoot != true (pod or container "postgresql" must set 
securityContext.runAsNonRoot=true), seccompProfile (pod or container "postgresql" 
must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

This is the first downside of using PSA. It's still not a widespread mechanism, since it's a beta, and even popular Helm Charts don’t yet comply with it out of the box.

There is yet another problem, this time inherited from PSP. Standards are applied only for pods, and if we create them via deployment (which is done in most cases), PSA won't prevent it from happening. Pods won't be created, but there will be no immediate feedback.

Finally, PSP does not allow for the creation of custom policies. In some cases, it’s OK to use the restricted standard, apart from maybe two or three rules. In such a situation, there is no real alternative in PSA, apart from changing a standard to being less restrictive.

Alternatives

Pod Security Admission is a Kubernetes built-in response to PSP deprecation. There are, however, other options. Some solutions, like Open Policy Agent (OPA) and Keyverno, allow for writing policies and enforcing them via custom admission controllers.

OPA's greatest strength is its flexibility, but this is also its greatest weakness. Using Rego language, we can declare any policy we want. However, it requires learning a new language or maintaining policies, and this complexity might not be desirable in some cases.

A middle-ground solution would be to use a scanning solution, which added to a CI/CD pipeline would quickly find potential security vulnerabilities.