Lessons Learned from Scanning Over 10,000 Kubernetes Clusters

This blog was originally published by ARMO here.

Written by Jonathan Kaftzan, VP Marketing & Business Development, ARMO.

With Kubernetes adoption continuing to rise, we've seen multiple studies add to the growing body of research for enterprise K8s deployments this past year. Companies leveraging managed services and packaged platforms drive much of the continued growth in adoption. An annual study conducted by the Cloud Native Computing Foundation (CNCF) found that 96% of organizations surveyed are either using or evaluating K8s currently. 5.6 million developers use K8s worldwide, accounting for 31% of all backend developers across the globe.

Survey respondents using K8s cite improved scalability and availability and shorter deployment times as the main reason for leveraging the technology. Among those users, weekly and daily release cadences are commonplace. While the number of organizations leveraging K8s will only continue to grow, there are rising concerns about security.

According to Gartner, through 2025, more than 99% of cloud breaches will have a root cause of customer misconfigurations or mistakes. This matches up with the findings from Gartner’s 2021 Hype Cycle for Cloud Security report which shows that 59% of security incidents with known causes occurred from a detected misconfiguration. When asked which risks concerned cloud leaders the most for their containers and K8s environments, most respondents identified misconfigurations as their number 1 concern.

K8s users and organizations fight misconfiguration and vulnerabilities across the SDLC. Kubescape is a Kubernetes open-source security platform providing a multi-cloud K8s single pane of glass, including risk analysis, security compliance, RBAC visualizer, and image vulnerabilities scanning.

Kubescape scans K8s clusters, YAML files, HELM charts, worker nodes and APIs servers, detecting misconfigurations according to multiple frameworks (such as the NSA-CISA, MITRE ATT&CK, and more), software vulnerabilities, and RBAC (role-based-access-control) violations across the SDLC, calculates risk score instantly and shows risk trends over time.

Since we launched Kubescape in August 2021, it has scanned more than 10,000 K8s clusters. We aggregated the data and did some analysis to highlight essential stats on the state of K8s security, risk, and compliance.

Here are the main conclusions:

Keep your Kubernetes risk scores below 30

Our scans included implementations across multiple regions, including North America, Europe, the Middle East, Africa, and the Asia Pacific. Our users range from DevOps and DevSecOps engineers to security engineers, architects, and risk analysts.

So what have we learned?

We assessed our findings in four frameworks, including ArmoBest, MITRE ATT&CK, NSA-CISA Hardening guidance, and DevOps Best Practices.

For each assessment, 100 represents the highest risk and 0 the lowest. A best practice is for organizations to keep their risk scores below 30, with 60+ putting them in the worst 5%. A risk score below 10 puts an organization in the best 10%.

The Top 5 Security misconfiguration

Our scans found that 100% of clusters contained at least one misconfiguration, while 65% had at least one high severity misconfiguration. 50% of clusters had 14 or more failed security controls. Here is a look at the top 5 misconfigurations found during our scans:

• Run privileged containers - Containers with access to run privileged might create unintended access to host information and should generally be configured to run as a non-root user.

• Cluster admin binding - Users with the cluster-admin role can perform any action on any resource, leaving a considerable vulnerability across the implementation.

• Missing Resource policies - 63% of clusters had workloads without proper resource limitations. Resource policies may control access to security-sensitive aspects of a pod's specification. Failing to configure resource policies properly can expose your system and lead to costly breaches.

• Immutable container filesystem - Cyber attackers that gain execution within a container can create files, download scripts, and modify applications. Leveraging an immutable container configuration can prevent damage but may also introduce complications in application function if not properly managed.

• Ingress and Egress blocked - 63% of clusters had workloads outside the cluster without the proper ingress blocked. It's critical to use ingress and egress filtering to protect network access points.

Top and common Vulnerabilities

63% of the containers we scanned had one or more vulnerabilities, while 46% had one or more critical vulnerabilities. 53% had one or more RCE (Remote Code Execution) vulnerabilities. Here's a look at the top 5 vulnerabilities that we found.

In all-

• 63% of clusters had workloads without proper resource limitations
• 37% of clusters had applications with credentials in configuration files
• 23% of clusters had applications running with dangerous Linux capabilities
• 35% of clusters had workloads running with insecure capabilities
• 100% of clusters had misconfiguration

Misconfiguration still plagues K8s implementations

Even as organizations level up their cloud-native development and operations competencies, we still found misconfigurations in 100% of the scanned clusters. This demonstrates the importance of having the right security frameworks and tools in place to bolster the security posture of K8s implementations.