Exploiting and detecting CVE-2021-25735: Kubernetes validating admission webhook bypass

This blog was originally published by Sysdig here

Written by Stefano Chierici, Sysdig

The CVE-2021-25735 medium-level vulnerability has been found in Kubernetes kube-apiserver that could bypass a Validating Admission Webhook and allow unauthorised node updates.

The kube-apiserver affected are:

• kube-apiserver v1.20.0 to v1.20.5
• kube-apiserver v1.19.0 to v1.19.9
• kube-apiserver <= v1.18.17

You are only affected by this vulnerability if both of the following conditions are valid:

1. You are using Validating Admission Webhook for nodes.
2. You are using Validating Admission Webhook with old values.

By exploiting the vulnerability, adversaries could bypass the Validating Admission Webhook checks and allow update actions on Kubernetes nodes.

In this article, you’ll understand how CVE-2021-25735 works, what parts of Kubernetes are affected, and how to exploit and mitigate it.

Preliminary

Before we start exploring the CVE-2021-25735 itself, let’s take a quick look at what admission controllers are.

Admission controllers are special controllers that can intercept Kubernetes API requests, and modify or reject them based on custom logic.

You can learn more in our “Kubernetes admission controllers in 5 minutes” article.

Admission Webhooks are HTTP callbacks that Admission Controllers call when there is an API request. There are two types of Admission Webhooks in Kubernetes:

• Mutating Admission Webhooks
• Validating Admission Webhooks

For this specific CVE-2021-25735, we’ll focus our attention on the Validating Admission controller.

The Validation Admission Controller receives resource requests after passing authentication and authorization, but before admitting those resources into the cluster. Then, the admission controller validates the request, returning a fail or successful response to the API Requests. Only after receiving a successful response, the resource will be admitted to the cluster.

The CVE-2021-25735 issue

The CVE-2021-25735 kube-apiserver vulnerability could allow node updates bypassing a Validating Admission Webhook. The kube-apiserver affected are:

• kube-apiserver v1.20.0 to v1.20.5
• kube-apiserver v1.19.0 to v1.19.9
• kube-apiserver <= v1.18.17

The vulnerability can be exploited when an update action on node resources is performed and an admission controller is in place and configured to validate the action. The information that is provided to the admission controller might not be correct, thus, leading the admission controller to accept requests that should be blocked.

In this case, kube-apiserver passes request.object and request.oldObject to Validating Admission Webhook. The request.object contains new node configs and request.oldObject contains old configs.

As you can see from the code changes, the overwrite of a part of oldNode.spec with node.spec causes this issue and it has been removed in the fixed version. In particular, Node.ObjectMeta overwrites the old values which may be used for admission decisions. Since the overwriting happens before the admission evaluation, editing the parameters used for the decision may cause unauthorised change in the cluster nodes.

Exploiting CVE-2021-25735

Let’s go now through how to exploit the CVE-2021-25735. To replicate the issue you need to have one of the vulnerable kube-apiserver versions deployed in your environment and a validating admission controller working with nodes update actions into your cluster.

In the repo you can find an example of a nodejs server admission controller webhook on nodes changes.

In this scenario the custom label changeAllowed is set in advance to enable or disable the opportunity to apply changes to specific cluster nodes. The admission controller uses this custom label value to make the admission decision on the submitted change. If the label is set to false, the admission controller refuses the change otherwise it returns a success validation.

When the node label changeAllowed is set to false we get the following error from the admission controller, since we are not allowed to perform any changes on the node:

Let’s try now to change the changeAllowed: "true" label, and add a new one in the node. As we can see from the screen below, this time the update action has been accepted by the Admission controller and the new label is added without any blocks.

By performing the combined action to change the changeAllowed label to true and adding a new one, the vulnerability has been triggered and the new value has been overwritten by the admission controller.

In this way the admission controller validation has been completely bypassed, permitting any kind of changes on the cluster nodes without any validation.

The impact of CVE-2021-25735

According to the CVSS system, CVE-2021-25735 scores 6.5 as medium severity.

As we have seen, the vulnerability has a low attack complexity and a high impact in terms of integrity and confidentiality.

In the circumstances described before, the vulnerability permits to completely bypass the controls in place in the validating Admission Controller over nodes updates.

Adversaries might be able to perform unwanted and unauthorised changes on nodes settings without getting stopped by the admission controller.

Mitigating CVE-2021-25735

If you’re impacted by this CVE, the issue is fixed in the following versions:

• kube-apiserver v1.21.0
• kube-apiserver v1.20.6
• kube-apiserver v1.19.10
• kube-apiserver v1.18.18

If you cannot patch your systems immediately, detecting exploitation attempts is important to preventing or stopping an attack. Even though you might have already upgraded your system and containers affected by the vulnerability, it is still necessary to detect any exploitation attempts in your environment.

If you cannot patch your systems immediately, detecting exploitation attempts is important to preventing or stopping an attack.

Many runtime security solutions are able to read events from the Kubernetes audit log.

In a Kubernetes cluster, you can configure kube-apiserver with an audit policy to enable the audit of events on nodes:

Once the Kubernetes Auditing is enabled, you can set up rules in your runtime security solution to detect changes in values used for the admission decision.

You will need to tailor those rules to your specific environment. In our previous example, we'll need to track the editing and setting of the custom node label ChangeAllowed to true, which would mean a bypass of the admission controller.

Even though you might have already upgraded your system and containers affected by the vulnerability, it is still necessary to detect any exploitation attempts in your environment.