Kubernetes Security Best Practices
Written by the CSA Serverless Working Group
Kubernetes is an open-source container orchestration engine for automating deployment, scaling, and management of containerized applications. A Kubernetes cluster consists of worker nodes/pods that host applications. The Kubernetes control plane manages the pod network in the cluster. We cover Kubernetes and container security best practices in the four sections below:
Part One: The Basics
TLS should be enabled for every component that supports it to prevent traffic sniffing and to authenticate the identity of both sides of a connection.
Run a service mesh
A service mesh is a web of encrypted persistent connections made between high-performance “sidecar” Envoy proxy servers. A service mesh adds traffic management, monitoring, and policy - all without microservice changes.
Use Network Policies
By default, Kubernetes networking allows all pod-to-pod traffic; this can be restricted using a Network Policy.
Use Open Policy Agent (OPA)
With OPA you can enforce custom policies on Kubernetes objects without recompiling or reconfiguring the Kubernetes API server.
Logging and Monitoring
You need this to detect anomalies on the application and infrastructure levels. If there are any attacks or anomalies – high usage or potential compromises – logging and monitoring will detect them. Application performance monitoring will detect other breaches such as DDoS attacks.
Consider using a bastion host
This is a special-purpose computer on a network specifically designed and configured to withstand attacks. Access to masters should only be done through bastion hosts, which can be hardened and can monitor users who have privileged access to Kubernetes.
Both Kubernetes master and worker nodes need to be deployed on private subnets to ensure secure connectivity with corporate networks, prevent direct reachability from the Internet, and reduce the overall attack surface.
Use Linux security features
The Linux kernel has several security extensions (SELinux) that can be configured to provide the least privilege to applications. The Linux kernel has many overlapping security extensions (capabilities, SELinux, AppArmor, seccomp-bpf) that can be configured to provide less privilege to applications.
Cluster node images
When you are building in a Kubernetes cluster, you will be using a Linux image. It needs to be CIS benchmarked to make sure all the Linux security controls are in place. Not doing the operating system hardening process can open up the infrastructure to software vulnerabilities. It is also important to consider the security of the entire software supply chain to ensure that the software chain of custody is being followed according to security best practices. Binary Authorization, tags, attestations, etc. are examples of how this can be implemented.
Separate and firewall your etcd cluster
etcd stores information on state and secrets and is a critical Kubernetes component - it should be protected differently from the rest of your cluster.
Rotate encryption keys
A security best practice is to regularly rotate encryption keys and certificates to limit the “blast radius” of a key compromise.
Part Two: Container Security
Policies are a vital but often overlooked piece of security that simultaneously function as a validating and mutating controller. PodSecurityPolicy is an optional admission controller enabled by default through the API; thus, policies can deploy without the PSP admission plugin enabled.
Statically analyze YAML
Where PodSecurityPolicies deny access to the API server, static analysis should be used in the development workflow to model an organization’s compliance requirements or risk appetite.
Run containers as a non-root user
Containers that run as the root frequently have far more permissions than their workload requires, which, in case of compromise, could help an attacker further their attack.
Part Three: Automation Security
All the deployments should be controlled through an automated CI/CD environment. At a high level, everything in Kubernetes is deployed as a container image. When someone creates an application, it becomes a container image and gets added to the container registry. The container image scanner needs to be part of the CI/CD pipeline. When someone creates a container image, it needs to be continuously scanned for vulnerabilities. You can do image whitelisting through an admission controller in Kubernetes. If your application uses certain images, those need to be approved.
Your clusters also need to be integrated via secret management systems. This ensures application pods will automatically receive required passwords and secrets at runtime based on the AppRoles attached to the pods.
Code scanning and static code analysis are also integral parts of automation security. When working on any application code in Kubernetes, you should scan the source code to ensure it does not have any vulnerabilities or any hard-coded anomalies.
Third-party vulnerability scanning
Vulnerability assessments are a core requirement for organizations. This is especially necessary when using upstream components which could be vulnerable. Third-party vulnerability scanning (BlackDuck, Tenable, etc.), when conducted continuously, can help organizations stay ahead of threat agents. These standard vulnerability scanning tools scan open-source libraries for known vulnerabilities published in various vulnerability databases (such as National Vulnerability Database). Once the vulnerabilities are identified, the remediation of these vulnerabilities will depend on how these might impact an organization’s operations, business, regulatory requirements, or reputation. Example: PCI DSS 3.2.1 requires that known vulnerabilities of high/critical type should be remediated within 30 days.
Security should be built into the entire DevSecOps process. The Agile process that feeds DevSecOps must also be secure and security user stories must be in the backlog. Embed security throughout the software lifecycle to identify vulnerabilities earlier, perform faster fixes, and therefore reduce cost. Use continuous monitoring to ensure devices, tools, accounts, etc., are continuously discovered and validated.
Part Four: Identity and Access Management
Enable RBAC with least privilege
Role-based access control (RBAC) provides fine-grained policy management for user access to resources, such as access to namespaces. Centralizing authentication and authorization across an organization (aka Single Sign On) helps onboarding, offboarding, and consistent permissions for users.
Use third-party auth for API server
To secure and manage access to Kubernetes APIs, administrators need to create detailed authentication and authorization policies and implement advanced, full featured screening technologies.
Sign up to receive CSA's latest blogs
This list receives 1-2 emails a month.