From the Trenches: Common-Sense Measures to Prevent Cloud Incidents - Part 1

Written by Omri Segev Moyal & Brenton Morris, Profero - Rapid IR

Introduction

As an incident response team, we see a lot of cloud breaches that could have been prevented. Adequate protection requires in-depth knowledge of the cloud provider and its APIs and ample preparation. In cases when a company faces time constraints, or its engineers have not received up-to-date training after a cloud migration, vulnerabilities open up. Whatever the reason, many cloud attacks can be easily avoided—in the following case studies, we offer advice on how.

This post is by no means a comprehensive guide to creating secure cloud environments. Rather, the examples included illustrate common weaknesses we encounter during our engagements, and steps companies can take to bolster their defenses.

SystemdMiner running in GCP

A healthcare provider contacted us after they received a GCP alert about one of their Linux instances. We carried out a forensic analysis of the instance in question and found a variant of SystemdMiner running on the machine. This miner consists of a series of obfuscated bash scripts and uses a cronjob to achieve persistence, and contains a few other tricks to squeeze as much mining power out of an organization as possible before detection.

SystemdMiner attempts to use SaltStack as a means to spread throughout an organization. If it is executed on a host which acts as a salt master node, it will execute a base64 encoded payload on every node to which the machine has access. It does this by running the salt command line:

Included in this next stage is a snippet of code that uses a list of DNS over HTTPS services to resolve the relay.tor2socks.in domain, a service which this malware uses to communicate with .onion hidden services hosting its C2 servers on the Tor network:

This style of attack is becoming increasingly commonplace as more organizations have begun to take advantage of infrastructure as code—but without, unfortunately, utilizing the greatest strengths of this way of working:

• The ability to block all changes to an organization’s infrastructure until the changes are reviewed and approved by multiple parties.
• The removal of the need to grant engineers working on the infrastructure access to the underlying resources.

This attack serves as a great example of why protecting your deployment and management resources is important. A single engineer should never have the ability to make changes to infrastructure—this special access should instead be delegated to a secure pipeline where changes need to go through a review process and be approved by multiple parties before execution. Additionally, heightened care needs to be taken when defending these pipelines and their critical assets such as salt master nodes, terraform state files, code repositories storing the automation templates, etc.

AWS VM Export Abused to Breach Hashicorp Vault

Often when moving from traditional IT infrastructure to a cloud environment, some seemingly minor details get overlooked. This can lead to a domino effect of severe consequences in the case of a breach. In one example, after a large AWS breach, we conducted an IR investigation for our client to figure out how the attacker gained access to so many access tokens within the organization. The only correlation among these tokens was that they were all stored in a HashiCorp Vault instance which was running in EC2.

When examining the CloudTrail logs, we found a log showing a call to the CreateInstanceExportTask API. This was used to create a snapshot of the HashiCorp Vault instance, which the attacker then downloaded from an S3 bucket. From there, many other services were compromised using the secrets contained within this instance.

Although this was not the initial point of entry used to obtain access to the organization, this esc2alation of privileges was made possible because the client had not adequately restricted calls to these APIs—a small miss that resulted in a cascade of larger issues.

When setting up AWS and other cloud-based infrastructure, it is essential to consider how an attacker could access data contained within compute and database instances. We also recommend placing high-value VMs such as Vault instances or sensitive data in their own restricted AWS account, and to establish a system to monitor any authentications happening inside these accounts.

EC2 GPU Miners

One common method used to monetize AWS account access in recent years is to spin up GPU enabled EC2 instances, which mine on behalf of the attacker—on the victims’ dime. Tesla fell victim to this kind of attack in 2018.

Although these attacks are not the most effective way to turn illegitimate access into income, they are usually the result of wider-scale automated attacks and not carried out in a manual, hands-on-keyboard approach.

These attacks show why it is important to have billing alerts enabled, so that you will receive an alert when an account’s usage is above normal. We also recommend tailoring your CloudTrail alerts to your organizations’ usage. For example, if you don’t use GPU enabled EC2 instances or higher-tier CPU instances, you should be alerted if one is spun up in your account, prompting your company to investigate accordingly.

MongoDB Compromise

During another incident, an extortionist attacker made claims that they had a copy of the client’s production MongoDB database, containing sensitive customer information.

The database in question was housed in a private subnet, accessible only via a few backend API instances—none of which showed any signs of compromise. We analyzed the HTTP request logs and also found no signs of compromise–so how did the attacker gain access to this data?

After investigating the CloudTrail logs, we discovered the attacker had simply restored a snapshot of the DocumentDB / MongoDB clusters using a public subnet and a security group that allowed connections from 0.0.0.0/0. Then, they connected to the database and pulled down all the data before destroying the cluster they had created. This was possible because the attacker had access to an AWS access key which could create new clusters using these snapshots, in addition to the fact that the victim organization had not set up CloudTrail alarms for such actions.

Mitigating the risk of this kind of attack requires adhering to the principles of least privilege and actively monitoring for anything out of the ordinary. Alerts that we would recommend creating to prevent this example include:

• Any new security group policies allowing connections from anywhere, or depending on your use case, any new security group policies allowing connections from any non RFC 1918
• Any new data resources being created (RDS instances, DocumentDB clusters, ElastiCache, etc.) as these resources are seldomly created. In cases when they are created more than usual—in staging/dev environments or during blue/green deployments—simply add the appropriate exceptions in your alerting.

Timely responses to these kinds of alerts can be the difference between stopping the exfiltration of private data or picking up the pieces in the aftermath of an incident.

Read Part 2 of this blog to learn about methods of looking for offensive attackers in cloud environments that we have found to be useful in the past.

About the Authors

Omri Segev Moyal is a highly successful entrepreneur, nationally recognized speaker and as of recently, a Forbes 30 under 30 achiever. He is also the Co-founder of Profero. Omri’s passion for empowering others through public speaking, stems from the many years of experience building successful companies of his own. He is dedicated to sharing his knowledge and proven skills with others in hopes that he can provide them with simple solutions that can solve complex problems and help to remove any barriers that stand in their way. His intense focus and drive for success has positioned him to be one of the top thought leaders in the entrepreneurial and start-up space.

Brenton Morris is the Sr Incident Responder at Profero. Brenton leads Incident Response engagements on a daily basis. From cloud sophisticated attackers to ransomware events. Brenton has a unique set of combined security research and devop experience, allowing him to resolve many cyber-attacks while fully understanding the impact on production systems.