Securing Non-Human Identities: Lessons from the Cloudflare Breach

Originally published by Oasis Security.

Written by Roey Rozi, Director of Solutions Architecture, Oasis Security.

Cloudflare disclosed on February 2nd that it had been breached by a suspected nation-state attacker. This breach exploited multiple unrotated and exposed secrets. The chain of events began with the Okta breach in October 2023, during which the attacker gained administrative access to Cloudflare’s Okta system. Although the Cloudflare team attempted to rotate all relevant credentials within Okta, they inadvertently missed one access token and three service accounts, mistakenly believing they were unused. Subsequently, the attacker utilized these four non-human identities to gain access to Cloudflare’s Confluence, Jira, and Bitbucket systems. The breach was eventually detected by a detection system, prompting the initiation of a thorough investigation.

It is noteworthy that the Cloudflare team was aware of the Okta breach in October, yet they couldn’t prevent the subsequent breach. Despite the awareness and the recognized need to rotate all exposed credentials, timely action was impossible to execute quickly enough and precisely due the inherent operational complexity of the task, even by an experienced team like the one at Cloudflare. Consequently, the attacker capitalized on the initial Okta access to gain further credentials, facilitating lateral movement.

CloudFlare Breach Timeline

In the wake of the breach, Cloudflare’s team was faced with a huge challenge that requires an incredible effort to solve: rotate all their production secrets, analyze all testing and development environments, and return data center hardware back to the vendor for analysis. A process that took them until January to complete, while developers were still working on hardening systems. As it often happens, the challenge of responding to risks is usually much greater than implementing best practices that prevent them to begin with.

‍

The Challenge of Secret Rotation

Rotating secrets is inherently difficult:

• They outnumber human identities by a factor 10-50x. In the CloudFlare case, they had to rotate more than 5000 of them!
• They are everywhere in the environment, making it hard to maintain an accurate and complete inventory of all identities and secrets.
• Rotating an identity without knowing what system depend on it may lead to infrastructure disruption

The lack of relevant management tools leaves most organizations struggling to perform regular rotations, especially during security incidents. Furthermore, non-human identities lack multifactor authentication (MFA) and often possess privileged access, making them prime targets for attackers seeking to execute supply chain attacks, perform lateral movement, and maintain persistence.

CloudFlare Unrotated NHI

How to Secure Non-Human Identities

The best approach for an organization to eliminate the security risk exposure from NHIs is to efficiently manage them throughout their lifecycle. This entails implementing several key best practices:

• Ensure that a non-human identity is dedicated to a single process or application.
• Rightsize the NHI privileges for its operation. No more, no less.
• Periodically rotate the identities' secrets to mitigate the risk of unauthorized access.
• Decommission stale identities that are no longer in use.

‍When an organization achieves this ideal state, an identity based attack becomes practically impossible. For example, after the Okta breach, this organization could trigger a wide and through rotation, thus eliminating the risk. Reaching this ideal state requires a combination of security policies and great tooling that enable the organization to follow said policies efficiently.