Kubernetes Security Best Practices

Written by the CSA Serverless Working Group

Kubernetes is an open-source container orchestration engine for automating deployment, scaling, and management of containerized applications. A Kubernetes cluster consists of worker nodes/pods that host applications. The Kubernetes control plane manages the pod network in the cluster. We cover Kubernetes and container security best practices in the four sections below:

Part One: The Basics

TLS everywhere

TLS should be enabled for every component that supports it to prevent traffic sniffing and to authenticate the identity of both sides of a connection.

Run a service mesh

A service mesh is a web of encrypted persistent connections made between high-performance “sidecar” Envoy proxy servers. A service mesh adds traffic management, monitoring, and policy - all without microservice changes.

Use Network Policies

By default, Kubernetes networking allows all pod-to-pod traffic; this can be restricted using a Network Policy.

Use Open Policy Agent (OPA)

With OPA you can enforce custom policies on Kubernetes objects without recompiling or reconfiguring the Kubernetes API server.

Logging and Monitoring

You need this to detect anomalies on the application and infrastructure levels. If there are any attacks or anomalies – high usage or potential compromises – logging and monitoring will detect them. Application performance monitoring will detect other breaches such as DDoS attacks.

Consider using a bastion host

This is a special-purpose computer on a network specifically designed and configured to withstand attacks. Access to masters should only be done through bastion hosts, which can be hardened and can monitor users who have privileged access to Kubernetes.

Private networking

Both Kubernetes master and worker nodes need to be deployed on private subnets to ensure secure connectivity with corporate networks, prevent direct reachability from the Internet, and reduce the overall attack surface.

Use Linux security features

The Linux kernel has several security extensions (SELinux) that can be configured to provide the least privilege to applications. The Linux kernel has many overlapping security extensions (capabilities, SELinux, AppArmor, seccomp-bpf) that can be configured to provide less privilege to applications.

Cluster node images

When you are building in a Kubernetes cluster, you will be using a Linux image. It needs to be CIS benchmarked to make sure all the Linux security controls are in place. Not doing the operating system hardening process can open up the infrastructure to software vulnerabilities. It is also important to consider the security of the entire software supply chain to ensure that the software chain of custody is being followed according to security best practices. Binary Authorization, tags, attestations, etc. are examples of how this can be implemented.

Separate and firewall your etcd cluster

etcd stores information on state and secrets and is a critical Kubernetes component - it should be protected differently from the rest of your cluster.

Rotate encryption keys

A security best practice is to regularly rotate encryption keys and certificates to limit the “blast radius” of a key compromise.

Part Two: Container Security

Use PodSecurityPolicies

Policies are a vital but often overlooked piece of security that simultaneously function as a validating and mutating controller. PodSecurityPolicy is an optional admission controller enabled by default through the API; thus, policies can deploy without the PSP admission plugin enabled.

Statically analyze YAML

Where PodSecurityPolicies deny access to the API server, static analysis should be used in the development workflow to model an organization’s compliance requirements or risk appetite.

Run containers as a non-root user

Containers that run as the root frequently have far more permissions than their workload requires, which, in case of compromise, could help an attacker further their attack.

Part Three: Automation Security

Image scanning

All the deployments should be controlled through an automated CI/CD environment. At a high level, everything in Kubernetes is deployed as a container image. When someone creates an application, it becomes a container image and gets added to the container registry. The container image scanner needs to be part of the CI/CD pipeline. When someone creates a container image, it needs to be continuously scanned for vulnerabilities. You can do image whitelisting through an admission controller in Kubernetes. If your application uses certain images, those need to be approved.

Secret management

Your clusters also need to be integrated via secret management systems. This ensures application pods will automatically receive required passwords and secrets at runtime based on the AppRoles attached to the pods.

Code analysis

Code scanning and static code analysis are also integral parts of automation security. When working on any application code in Kubernetes, you should scan the source code to ensure it does not have any vulnerabilities or any hard-coded anomalies.

Third-party vulnerability scanning

Vulnerability assessments are a core requirement for organizations. This is especially necessary when using upstream components which could be vulnerable. Third-party vulnerability scanning (BlackDuck, Tenable, etc.), when conducted continuously, can help organizations stay ahead of threat agents. These standard vulnerability scanning tools scan open-source libraries for known vulnerabilities published in various vulnerability databases (such as National Vulnerability Database). Once the vulnerabilities are identified, the remediation of these vulnerabilities will depend on how these might impact an organization’s operations, business, regulatory requirements, or reputation. Example: PCI DSS 3.2.1 requires that known vulnerabilities of high/critical type should be remediated within 30 days.

DevSecOps (CI/CD)

Security should be built into the entire DevSecOps process. The Agile process that feeds DevSecOps must also be secure and security user stories must be in the backlog. Embed security throughout the software lifecycle to identify vulnerabilities earlier, perform faster fixes, and therefore reduce cost. Use continuous monitoring to ensure devices, tools, accounts, etc., are continuously discovered and validated.

Part Four: Identity and Access Management

Enable RBAC with least privilege

Role-based access control (RBAC) provides fine-grained policy management for user access to resources, such as access to namespaces. Centralizing authentication and authorization across an organization (aka Single Sign On) helps onboarding, offboarding, and consistent permissions for users.

Use third-party auth for API server

To secure and manage access to Kubernetes APIs, administrators need to create detailed authentication and authorization policies and implement advanced, full featured screening technologies.