Not Just Code Vulnerabilities: The Overlooked Cause of Software Supply Chain Attacks

Originally published by Astrix.

According to Gartner: “Software supply chain attacks have added a new dimension to software security problems because the software delivery pipelines and the tools used to build and deploy software are the new attack vectors.”

While the software supply chain has been a huge catalyst for vulnerabilities, and consequently attacks, there is a new type of supply chain attacks that has proliferated in the last couple of years – taking advantage of the third-party tools and services that are connected to development environments such as AWS, GCP, Azure and Git platforms.

While DevOps and other AppSec stakeholders turn to classic AppSec solutions to secure their code and engineering environments, which are vital for securing the app’s code, CI/CD environments and sometimes runtime, there is a crucial missing piece - ungoverned third-party and internal non-human access via keys and tokens. But what does that actually mean, and why should you care?

1. Unmonitored internal access keys and tokens: To develop an app and make it work, R&D teams regularly generate ‘secrets’ - keys and tokens, as part of their day to day job. These keys and tokens allow access to resources, code and infrastructure - aka your most valuable intellectual property, and the vitality of your app. Safekeeping these keys is a challenge since they are scattered across different secret managers (vaults), and regularly accessed by R&D teams that often unintentionally expose them (for debugging, for example).

While AppSec solutions with shift left methodologies secure code vulnerabilities & dependencies, CI/CD processes and runtime environments, security teams lack visibility into the one thing that allows access to your entire operation - the actual access keys - where they are, whether they are exposed, what permissions they have etc. And secret scanners or vaults don’t cut it- vaults only store your secrets without any essential context like secret risk severity or usage insight, and secret scanners only find exposed secrets, again without any context or ability to prioritize risk.

A stolen key can allow attackers to access your app’s code, alter it, steal it, render your app unusable, cause costly downtime and possible reputation damage.

2. Third party non-human access: IT, DevOps, developers, and even security teams are increasingly authorizing new third-party tools and services that access core engineering environments like GitHub, GitLab, AWS and Big Query to streamline development efforts and increase agility. With the growing trend of bottom up software adoption and freemium cloud services, many of these connections are done by developers without any security governance.

These create shadow connections between sensitive engineering systems to external third-party applications and processes, which are done via API keys, service accounts, webhooks, OAuth tokens, or SSH keys. These keys and tokens are often created with a high level of permissions and sometimes with unlimited, permanent access. Many of these are never revoked even after users are finished with the connected service.

An example of this kind of shadow integration would be a developer using a new cloud-based CI/CD tool, such as the increasingly popular CircleCI, which relies on API access to the GitHub source code repository. And the result of such ungoverned permissive access?

As you may know, CircleCI was breached earlier in 2023. In this breach, attackers gained access to CircleCI’s customers’ keys, which means they could push code to deployment (!). The results of this attack could have ranged from pushing malicious code to production and stealing customers data, to rendering the app unusable. This is a software supply chain attack through non-human access.

Recent High profile attacks against engineering environments through non-human access:

1. CircleCI breach (January 2023): In this attack, engineering employees’ computer, which was compromised by malware that bypassed their antivirus solution. The compromised machine allowed the threat actors to access and steal session tokens. Stolen session tokens give threat actors the same access as the account owner, even when the accounts are protected with two factor authentication.
2. Slack GitHub Repositories (January 2023): In this incident, threat actors gained access to Slack’s externally hosted GitHub repositories via a “limited” number of stolen Slack employee tokens. From there, they were able to download private code repositories.
3. GitHub Personal Access Token (December 2022): At this event, repositories from GitHub’s atom, desktop, and other deprecated GitHub-owned organizations were cloned by a compromised Personal Access Token (PAT) associated with a machine account. The malicious actor then used the PATs to read these repositories, which contained sensitive information.

Findings in real engineering environments:

1. Dev teams create around 20-30 new personal access tokens, and SSH keys in GitHub organizations every week.
2. In a typical GitHub environment approximately 1 of 4 tokens (PAT and SSH keys) is not in use and can be safely removed without impacting the business.
3. 1 in 5 users in a Snowflake production environment is in fact a service account.