Secure Software Supply Chain: Why Every Link Matters

This blog was originally published by Sysdig here.

Written by Álvaro Iradier, Sysdig.

The new threats in software development are not only related to the specific company itself. The whole software supply chain is a target for attackers and it is really important to make sure that we put all our effort into securing each link because if one fails, everything will be affected.

Supply chain activities include each step of the transformation of raw materials, components, and resources into a completed product, and its delivery to the end customer.

Each step could be a complex process itself and cause a security incident.

What is a software supply chain

The software supply chain is similar to other activities or industries. Some resources are consumed, then transformed, through a series of steps and processes, and finally supplied as a product or service to a customer.

In software, the raw materials are common libraries, code, hardware, and tools that transform code into a final deliverable. This deliverable can be deployed as either a user-facing application, a service (starting over with the same supply chain loop), or another package artifact that is included as a dependency, part of a different product.

The chain can be long and quite complex, so let's try an extremely simplified example:

In order to produce the final "web application" for our customer, we need to transform (compile) a source code and consume information from third-party services. The source code itself depends on external libraries, which are produced from another code, etc.

Software supply chain attack happens when some malicious element is introduced in this chain.

A successful attack in any link of the supply can propagate the compromised code or component downstream, completely unnoticed, and cause mayhem across different stages.

In fact, many of these attacks focus on compromising a software vendor by injecting some malware or vulnerability at an intermediate step in order to exploit the final customers with fatal consequences.

As your company or product is just a piece in the overall software supply chain, security measures related to the supply chain can be applied on three different points:

1. Inputs: Library and package dependencies, third-party tools, software, services, or any artifact you are consuming, either public or from a private vendor.
2. Outputs: Guarantee the integrity of your deliverables and provide ways to verify the components downstream.
3. Processes and infrastructure: Protect your network, identities, credentials, signature keys, repositories, and processes.

We will explain some of the common supply chain attacks, with recent examples, and provide tips and practices that you can apply to mitigate the risks.

What is a supply chain attack

We already gave a definition of the Software supply chain with a simplified example, talking about code, libraries, and services. But let's dig a bit deeper and put the focus on the source code:

Zooming in on any link on the chain will reveal additional details:

In the example, your source code will live in a private git repository, which could be part of your infrastructure or SaaS provided by a vendor, as well as compiler tools, base container image registries, etc.

Some dependencies are hosted in public repositories, like Docker Hub or Quay.io, and could be compromised. Also, we are publishing our application as a container image in a public repository too.

Some of the components in the chain (blue) are under your umbrella, like your private source code git repository, the application code itself, and the final binary or container image that is produced. But many other components or services (in green) are public services and resources, or provided by other companies, totally out of your control.

So a software supply chain attack can target you directly, or it can target any upstream element (like external dependencies or provided services), so you become a victim, either by directly suffering the attack, or by becoming a supplier of compromised resources.

Examples of supply chain attacks

Trends show that supply chain attacks are increasing at an exponential rate of 4-5x per year, with several thousand last year, the most common being related to dependency confusion or typosquatting, followed by malicious source code injection.

Fortunately, not every attack has a big enough impact to appear in the newspaper, but let's analyze some of the most relevant and recent ones. Many other examples of different types of supply chain attacks are also collected by the CNCF in their Catalog of Supply Chain Compromises.

CodeCov

Leaked credentials in a Docker image from CodeCov allowed the attackers to modify a bash script. This script was downloaded and executed by customers, resulting in leaked customer credentials and attackers getting access to their git repositories.

Multiple customers like Monday.com, Hashicorp, Twilio, or Confluent were affected.

Solarwinds

The attackers infiltrated Solarwinds' network and managed to inject malicious software into their build process. This malware (related to APT29, Nobelium) was bundled as part of Orion (a Network Management System) product updates. As part of the build process, the artifact was digitally signed and then downloaded by hundreds of customers.

Once the malware was running in the customer networks, the attackers spied and stole their information.

This incident also shows how a supply chain attack can propagate downstream and impact multiple customers: as part of the compromised Orion software, a TLS certificate private key for a mail server in Mimecast (a cloud cybersecurity services company) was leaked. This allowed attackers to perform a man-in-the-middle attack on the mail server and access customer emails.

We might never know the real impact of the Solarwinds attack, as the compromised software was running for several weeks or months inside networks of potentially thousands of customers.

And it is not yet finished: the attackers behind Solarwinds have been recently attempting to replicate similar attacks but to different parts of the supply chain, like resellers and other technology service providers that customize, deploy, and manage cloud services and other technologies on behalf of their customers.

Kaseya

It’s quite similar to the Solarwinds attack, but in this case, the attackers exploited a zero-day vulnerability in Kaseya systems. Once they took control of the systems, they executed remote commands in the customers' systems using VSA, a remote management and monitoring tool.

Apple Xcode and XcodeGhost

In this attack, a trojanized version of the legitimate Xcode project "TabBarInteraction" was published in a public repository. Developers using the fake version of the project were inadvertently executing a script on each project build. The script opened a connection to a C2 (command and control) server.

We can find another example where repackaged versions of Xcode, with malicious code, were uploaded to Chinese file hosting services. Developers downloading the compromised version from these mirrors ended up with a modified object file. The object file is linked in the final executable when creating iOS applications. At least two well-known applications reached the official Apple AppStore, successfully passing Apple certification and code review.

NPM package ua-parser-js

On Oct. 22, 2021, developers of a very common NPM package, ua-parser-js, discovered that some attackers uploaded a compromised version of the package containing malware for Linux and Windows, and were capable of stealing data (at least passwords and cookies from the browser).

This ua-parser-js library is part of the supply chain for a large list of software and companies, including Facebook, for some of its user-facing applications in millions of computers.

Unicode and code compilers - Invisible vulnerabilities

Some researchers at the University of Cambridge claim the discovery of a novel supply-chain attack targeting source code in the subtleties of text-encoding standards, such as Unicode. The way Unicode works, it is possible to add special characters to the comments or some parts of the code that make the "logical encoding" of the code work in a different order from the way it is displayed. This makes it possible to hide malicious code inside code that looks correct to humans, bypassing review procedures.

But before panicking, please note the issue might not be that new and invisible, and it can be prevented by establishing trust in the code contributors and doing proper reviews.

The Huawei Ban

Finally, we include this example not as a known software supply chain attack, but as a demonstration of the importance of trusting providers in every single link of the chain. Even the hardware supply chain can be a target. For years, the U.S. government warned the world and American companies about security threats in Chinese network equipment:

Based on available classified and unclassified information, Huawei and ZTE cannot be trusted to be free of foreign state influence and thus pose a security threat to the United States and to our systems.

The warning seems to be based on the lack of information and the inability to satisfy some security concerns, and not on real evidence of any existing backdoor in the systems.

So we might never get real evidence, but the lack of confidence was so severe that Huawei was banned in May 2019 from doing business with any organization that operates in the United States.

Supply chain taxonomy (and controversy)

Not everybody agrees that an attack targeting any stage of the software supply chain falls into the category of "supply chain attack". The European Union Agency for Cybersecurity (ENISA) in the Threat Landscape for Supply Chain Attacks report "proposes a taxonomy to characterize supply chain attacks and structure their subsequent analysis":

But on the same report, they add:

"If no customer is attacked, or no supplier attacked, then it is probably not a supply chain attack."

Explicitly, they mention some examples that don't meet the requirements, like:

• Some name-squatting or brand-hijacking packages were uploaded to public repositories because they did not compromise existing packages or the software repositories.
• Discovered vulnerabilities or unintentional errors that were not used in known attacks, because they don't target a supplier or a customer.
• Public-facing vulnerable versions of software that allowed exfiltration of customer data, but in this case the supplier was not compromised (the attack was targeted as French IT providers) and the vulnerabilities were not intentional nor specifically crafted for this attack.

So for some incidents, there is not even a global consensus whether it can be classified as a supply chain attack or not.

Securing the software supply chain

Attacks to the software supply chain are broadening the attack surface of companies, as their security does not only depend on internal procedures. Now, they also need to ensure that third parties (including software, hardware, services, etc.) are not a gateway to attackers who can affect them.

Like traditional security, it is impossible to secure everything, especially as new kinds of software supply chain attacks are being discovered continuously. But let's explain where you need to focus on each of the layers to be as protected as possible.

Inputs of your software development

As a consumer of software artifacts, hardware, and services from multiple providers, ensure that your providers apply a high level of security.

The software supply chain is becoming so critical that in May 2021, the president of the United States issued an Executive Order targeted to improve the nation's cybersecurity. As a result of the order, the NIST produced the Recommended Minimum Standards for Vendor or Developer Verification (Testing) of Software Under Executive Order (EO) 14028 guide, which recommends minimum standards for verification by software vendors or developers.

Because it is impossible to achieve and guarantee 100 percent security, please also pay attention to any risks or incidents impacting your providers, and be ready to take quick corrective action in case a compromised component is either detected or reported.

Software development in your company

With regard to code and development processes, the mentioned Recommended Minimum Standards for Vendor or Developer Verification (Testing) of Software Under Executive Order (EO) 14028 includes the following set of techniques as the minimal safety requirements for software development life cycle. Make sure you don't miss any of them:

• Apply threat modeling to identify key or potentially overlooked testing targets.
• Automated testing.
• Code-based (static) analysis, using a code-scanner, and review for hard-coded secrets.
• Dynamic analysis, with built-in checks and protections, black-box and fuzzy testing, web-app scanner, etc.
• Apply similar checks to included software (third-party dependencies).
• Fix critical bugs as soon as possible.

With Infrastructure as Code (IaC), threats in the software supply chain now potentially target not just software and applications, but the underlying infrastructure too. The same software verification principles should be applied to the deployment and management of infrastructure as code, shifting left security.

But security must be pervasive in all development stages and the whole company culture and practices. Securing the code and the development process is clearly not enough, as outlined by multiple organization researches and guides:

• The Defending Against Software Supply Chain Attacks guide from Cybersecurity and Infrastructure Security Agency considers that the Software Supply Chain Lifecycle has six phases where "software is at risk of malicious or inadvertent introduction of vulnerabilities": Design, Development and production, Distribution, Acquisition and deployment, Maintenance, Disposal
• The Cloud Native Computing Foundation (CNCF) keeps a living, community maintained document: the Software Supply Chain Security Paper. This paper aims to contribute to the community with a series of recommended practices, tooling options, and design considerations that can reduce the likelihood and overall impact of a successful supply chain attack.
• The CNCF, Linux Foundation, VMware, Intel, Google, and others are also working on SLSA - Supply-chain Levels for Software Artifacts, a security framework, and a common language for increasing levels of software security and supply chain integrity for anyone working with the software. Each level provides an increasing degree of confidence, a way to say that software hasn’t been tampered with and can be securely traced back to its source.

Kubernetes and containers are so common nowadays that NSA/CISA also released a Kubernetes Hardening Guidance, highlighting "Supply chain risks" as one of three sources of compromises, and proposing the following hardening measures and mitigations:

• Scan containers and Pods for vulnerabilities or misconfigurations.
• Run containers and Pods with the least privileges possible.
• Use network separation to control the amount of damage a compromise can cause.
• Use firewalls to limit unneeded network connectivity and encryption to protect confidentiality.
• Use strong authentication and authorization to limit user and administrator access, as well as to limit the attack surface.
• Use log auditing so that administrators can monitor activity and be alerted to potential malicious activity.
• Periodically review all Kubernetes settings and use vulnerability scans to help ensure risks are appropriately accounted for and security patches are applied.

Your software is the input to other companies or users

Everything you apply when choosing your best providers with a high level of confidence will apply to your company when acting as a supplier for other companies or for end-users. Even if you correctly implement security into every step and process of your software supply chain life cycle, you still need to manifest it and add specific measures to the delivered artifacts.

• Provide evidence of regulatory compliance and security certifications within your organization that applies to the software supply chain.
• Add a Software Bill of Materials (SBOM) as a way to track dependencies and third-party sources of compromise in your software.
• Include digital signatures to prevent tampering and verify the source of your artifacts.
• Use safe distribution channels, encrypted communications, and trusted hosting or storage infrastructure.

Conclusion

Software supply chain attacks are not really something recent. Ken Thompson, after receiving the Turing Award with Dennis Ritchie in 1984, wrote this speech about the importance of trust in the [code] providers. He shows how a trojanized version of the compiler binary produces modified versions of the "login" Unix command with a backdoor.

"To what extent should one trust a statement that a program is free of Trojan horses? Perhaps it is more important to trust the people who wrote the software."

But the ever-growing complexity of software, infrastructure, and dependencies, and the rate of growth of attacks targeting the supply chain, are making industries and organizations more and more concerned about its security.

At this point, you already noticed that the software supply chain is a very complex network of interconnected and heterogeneous pieces, from code and libraries to hardware components. So the approach for securing every single piece and link is very different and cannot be covered as a whole. But, whether you are a supplier or a consumer (or both), you need to put the focus on securing your processes and establishing strong connections with trusted and verified providers. Because even if you apply the best security practices and put all efforts in your own code, infrastructure, etc., you still depend on third-party components completely outside of your control. And the security of the software supply chain depends on every individual link.