Five Lessons Learned From Okta’s Support Site Breach

Originally published by Valence.

Written by Adrian Sanabria.

On September 29th, 2023, security vendor 1Password discovered unauthorized activity in their Okta tenant. An employee unexpectedly received an email that they had requested a report listing Okta administrators. A 1Password employee had recently uploaded a HTTP Archive (a HAR file), which is a browser session logging format that is typically used for troubleshooting, to Okta’s support portal. After the Okta logs didn’t indicate that their support team accessed the HAR before the breach occurred, 1Password suspected the employee’s account was compromised or that malware on an employee’s laptop was responsible.

On October 2nd, 2023, security vendor BeyondTrust discovered and stopped an attack against them. This occurred only minutes after sharing a HAR file with Okta’s support staff. This wasn’t a coincidence. The attackers found ways to bypass BeyondTrust’s admin policies by performing API actions, leveraging the fact that non-human identities typically don’t have the same level of restrictions and security controls. BeyondTrust suspected Okta had been compromised and told them as much.

On October 18th, 2023, security vendor Cloudflare discovered attacks on their systems and traced them back to Okta. Again, a Cloudflare employee had recently created an Okta support ticket that included a session token. And like the others, Cloudflare notified Okta of a possible breach of their systems.

While 1Password had been assured there was no unauthorized access on Okta’s side and turned the focus of their investigation to their own employee that had uploaded the HAR file, BeyondTrust and Cloudflare insisted that Okta was the source of the compromise.

On October 20th, 2023, Okta publicly announced that they had discovered unauthorized access in their support case management system. They note that all impacted customers were notified. Without more information from Okta, however, we don’t have the full picture, and don’t know how many other customers were affected. It’s quite possible the list is longer than just 1Password, BeyondTrust, and Cloudflare.

There is a pattern here: identity and security vendors. Okta, 1Password, BeyondTrust, and Cloudflare are all security vendors with products that protect some aspect of enterprise identities: authentication, passwords, and/or authorization. They also hold very high privileged access to their customer environments and in some cases, hold the “keys to the kingdom”, making them a very lucrative target for attackers.

Lesson 1: Protect Your Session Tokens

When troubleshooting issues with SaaS applications, it’s difficult to understand what’s happening on a customer system without access to the full session information. As such, it is common for support to request HAR files during the troubleshooting process.

A HAR file is simply a recording of a browser session. As it records everything, it is common for it to include sensitive data, like credentials and auth tokens. While Okta’s documentation for producing HAR files includes a warning to remove sensitive data before sharing it with them, it doesn’t appear that this advice was commonly followed, or enforced.

Recommendations:

1. Build security defenses that don’t trust post MFA session tokens as the only line of defense - as mentioned in our recent blog, attackers are stealing these tokens to bypass MFA which is commonly used as the main security strategy.
2. Employees should be aware that HAR files may contain sensitive data, and should make efforts to sanitize them before sharing with external parties. Tools for automating this process, like HAR Sanitizer, are readily available.
3. Vendors commonly requisition HAR files from customers should assume mistakes will be made, warnings ignored, and/or steps skipped. As such, they should also make an effort to scrub sensitive data from HAR files upon receipt of these files.

Lesson 2: Monitor for Unusual Behavior

At least three of the vendors targeted by Okta’s attackers succeeded in catching the attacks against them quickly. BeyondTrust detected the attack using its own Identity Security Insights tool, as an attacker created new accounts within the Okta tenant. A 1Password employee received an email saying they generated a report on administrative accounts (that they did not generate) and quickly reported the anomaly. Cloudflare is less specific about how they detected the attack, but the trend here is clear - their detection controls worked as expected.

Recommendations:

1. Detective controls should include notifications when new accounts are created, on any significant changes to administrative access/authorization, and any strange behavior coming from accounts with admin-level privileges.
2. Ensure you’re collecting all the forensic data necessary to determine the cause and source of an attack. In recent breaches, we commonly see that logs are either not enabled, or don’t go back more than a few weeks, creating a blind spot for investigations.
3. Employees should be trained to spot and report suspicious behavior - people are a detection control also!

Lesson 3: SaaS Vendors Are Common Targets

A troubling trend we’ve noticed is that attackers are recognizing the value of scale that comes from targeting the trusted third-party vendors. As most companies have adopted a shared responsibility model in the move to cloud and SaaS, the cloud and SaaS vendors become attractive targets. Especially SaaS applications that are trusted with high privileged access. Why compromise one target, when compromising part of the supply chain will get you access to thousands of potential targets?

Security vendors are targets, and clearly, they’re also customers of other security vendors. 1Password, BeyondTrust, and Cloudflare were all customers of Okta, and it’s entirely possible that Okta is/was a customer of each of them. Security vendors gain the same benefits of efficiency and scale from using SaaS and third parties as anyone else.

Recommendations:

1. Assume some of your vendors will be breached (not just security vendors) and plan accordingly. Determine the potential impact of each compromise and ensure you have compensating controls in place.
2. Determine what steps you can take to minimize the impact of a vendor or supply chain compromise. In this scenario, all three of the vendors affected by Okta’s breach applauded the principle of least privilege or zero trust principles and controls to minimize the impact of the breach on their systems. The result? All three of these vendors state their own compromises were limited such that their own customers weren’t impacted.
3. Monitor your SaaS security posture - SaaS vendors leave it to the customer to leverage their security controls. Understand and maximize the native SaaS security configurations vendors offer. SaaS Security Posture Management (SSPM) tools can make SaaS security knowledge accessible, manage SaaS risks, and detect configuration drifts.

Lesson 4: Zero Trust Principles Work

1Password, BeyondTrust, and Cloudflare each mention that their application of Zero Trust principles helped either slow or mostly contain each of their respective attacks.

• BeyondTrust: “Custom policy controls blocked the attacker's initial activity…”
• 1Password: “This activity was confined to our Okta instance…”
• Cloudflare: “Cloudflare’s Zero Trust architecture protects our production environment, which helped prevent any impact to our customers.”

Recommendations:

1. Study how attackers pivot between cloud systems, identity systems, and SaaS applications - don’t underestimate the power of non-human identities such as API keys and OAuth tokens.
2. Engineer detections to alert of potential attempts to pivot between systems - including with SaaS-to-SaaS activities.
3. Put controls in place to limit an attacker’s ability to pivot
4. Separate administrative or superuser privileges from user accounts - this makes it possible to put tighter controls on accounts with administrator privileges without impacting productivity for normal user accounts.

Lesson 5: MFA isn’t a binary control

For many years, security experts have pushed for organizations to enable MFA wherever possible. In 2023, it’s clear that this advice needs to get more specific. Attackers have successfully compromised and evaded lower level multi-factor authentication options, like one-time codes sent to phones via SMS. Cloudflare specifically calls out higher-quality MFA controls as an advantage here (referring to the use of security keys):

“Cloudflare’s use of hard keys for multi-factor authentication stopped this attack”

However, MFA can’t be used everywhere. BeyondTrust notes that, while their MFA controls prevented the attacker from accessing their Okta instance via the Web-based interface, the attackers were able to perform malicious actions via Okta’s API. Since APIs are intended to be used programmatically (e.g. by a script, or application) MFA can’t be used, as there’s typically no human to respond to prompts for additional authentication factors.

Recommendations:

1. Ensure MFA is enabled and enforced for ALL interactive user accounts.
2. Use higher quality MFA whenever possible.
3. Apply extra monitoring to non-human identities that cannot be protected by MFA.

Conclusion

Detecting, preventing, and responding to attacks in SaaS requires additional visibility and preparation. Proper governance will reduce the amount of opportunities for attackers to exploit. Applying Zero Trust principles will minimize an attacker's ability to move laterally between SaaS applications, clouds, and other environments. One of the most important things we can do is study attacks when they happen and search for lessons to be learned from them.