How Attackers Can Exploit GCP’s Multicloud Workload Solution

Originally published by Ermetic.

Written by Liv Matan.

Introduction

When integrating with other workloads, it is never a good idea to send secrets over the network and hard code them. Massive breaches leveraging credentials stolen from GitHub, local machines and other sources have already shown us the impact of not choosing the secure way to carry out an authenticated/authorized operation on the internet.

In cloud workload integrations, many organizations and individuals need to access data from outside the cloud environment. Google Cloud Platform (GCP) has a tool called Workload Identity Federation (WIF) that lets users access the Google Cloud customer's data from outside the cloud using token exchange operations. This bypasses the need to store the service account key insecurely.

TL;DR

• Workload Identity Federation is GCP’s primary method of granting on-premise, external or multicloud workloads access to Google Cloud resources without storing a long-lasting service account key in the target workload.
• Long-lasting keys are a common attack vector, and using Workload Identity Federation to avoid them is strongly preferable.
• In this post, we provide a deep dive into the inner workings of Workload Identity Federation.
• Workload Identity Federation has risks of its own; we highlight possible misconfigurations and explain how to avoid them.

Why Workload Identity Federation?

Let’s say you want to give an external entity access to your resources on GCP. So, for example, you want to enable a given workload to access certain Google Cloud storage buckets.

If it were running on GCP’s Compute Engine, the workload would have a service account associated with it via the Instance Metadata Service (IMDS), which would grant it certain permissions.

However, if the workload is not running on GCP but, rather, on another cloud (AWS, Azure) or an on-premise server, we will need a way to grant it GCP entitlements.

The Dangers of Service Account Keys

The easiest way to grant entitlements to external resources is with service account keys. You can create service accounts with certain role assignments, create long-lasting keys for them, and use them from any workload.

We have written before about the dangers of using long-lasting service account keys; suffice it to say that credential leakage of long-lasting credentials is one of the most common compromise vectors for cloud environments.

With identity federation, you can use Identity and Access Management (IAM) to grant external identities IAM roles, including the ability to impersonate service accounts. This approach eliminates the maintenance and security burden associated with service account keys.

Enter Workload Identity Federation

GCP Workload Identity Federation is a service that allows you to securely connect applications running on GCP with your existing identity provider (IdP). It enables you to use Google Cloud IAM roles to access cloud resources without managing any credentials.

Workload Identity Federation is designed to be used by applications, while its close cousin, Workforce Identity Federation is meant for use by human users.

Workload Identity Pools are used to organize and manage external identities. They are associated with Workload identity pool providers which manage the external identities and grant them access to different GCP Service Accounts.

There are three primary authentication flows for WIF: a generic SAML authentication flow, an OIDC flow (also used for authentication with Azure) and a special flow for AWS. We’ll focus on the latter two in this blog.

All of them share the same basic principles:

• An Identity Pool is created in GCP and associated with certain GCP IAM principals, usually a service account or a principal set.
• One or more Identity Providers are created.
• The providers provide tokens:
  • OIDC providers provide JSON Web Tokens (JWT) that meet certain attributes.
  • In the AWS flow, a presigned GetCallerIdentity request is passed.
• The tokens are verified (verification method varies by provider type).
• A GCP token to the service account is given in exchange for the OpenID Connect (OIDC) token.

OIDC Flow

We won’t do a full deep dive into OIDC authentication flows in this blog, but the internet is full of blogs, comics, and tutorials on the matter. Suffice it to say for now that the protocol enables the exchange of a JWT token from an identity provider with, in our case, a GCP access token.

OIDC flow setup varies slightly depending on the provider, however, the broad strokes are very similar.

We won’t repeat the entire setup flow, for those interested, the official GCP documentation explains it well.

This is essentially an implementation of the OIDC flow.

After setup, the Azure workload is set up with its own token, accessed via the Azure VM Instance Metadata Service (IMDS):

curl \
"http://169.254.169.254/metadata/identity/oauth2/token?resource=APP_ID_URI&api-version=2018-02-01" \
-H "Metadata: true" | jq -r .access_token

There are several ways to move forward here, one of them is to create a credentials JSON file:

{
"type": "external_account",
"audience": "//iam.googleapis.com/projects/222222251267/locations/global/workloadIdentityPools/livs-pool/providers
/livnoamazureoidc",
"subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
"token_url": "https://sts.googleapis.com/v1/token",
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts
/[email protected]:generateAccessToken",
"credential_source": {
"url": "http://169.254.169.254/metadata/identity/oauth2/token?resource=api
://2f345678-2345-678a-2345-2356789021a2&api-version=2018-02-01",
"headers": {
"Metadata": "True"
},
"format": {
"type": "json",
"subject_token_field_name": "access_token"
}
}
}

Then, the environment variable GOOGLE_APPLICATION_CREDENTIALS can be configured to point to the JSON file, which together with defining the GOOGLE_CLOUD_PROJECT environment variable, can be used to make the CLI or SDKs access GCP APIs with these credentials, for example:

import google.auth
import os
from google.cloud import storage

os.environ['GOOGLE_APPLICATION_CREDENTIALS']="creds.json"
os.environ['GOOGLE_CLOUD_PROJECT']="research-325443"

client=storage.Client()
buckets = client.list_buckets()
for bucket in buckets:
print(bucket.name)

Behind the scenes, this initiates the OIDC flow with the Workload Identity Federation pool and provider, exchanging the Azure JWT token for a GCP access token for the configured service account.

AWS Flow

The AWS flow is quite similar, with one important difference. On the user’s side, configuration is very similar - a JSON credential file is written and used by setting an environment variable. However, by looking at the configuration file, we begin to understand the subtle but substantial differences in the background.

{
"type": "external_account",
"audience": "//iam.googleapis.com/projects/222222251267/locations/global/workloadIdentityPools
/livs-pool/providers/my-provider",
"subject_token_type": "urn:ietf:params:aws:token-type:aws4_request",
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts
/[email protected]:generateAccessToken",
"token_url": "https://sts.googleapis.com/v1/token",
"credential_source": {
"environment_id": "aws1",
"region_url": "http://169.254.169.254/latest/meta-data/placement/availability-zone",
"url": "http://169.254.169.254/latest/meta-data/iam/security-credentials",
"regional_cred_verification_url": "https://sts.{region}.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15"
}
}

Note the parameter:

"regional_cred_verification_url": "https://sts.{region}.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15"

It is not an OIDC flow. Instead, the user presigns an sts:GetCallerIdentity request to an sts endpoint.

We can verify that is exactly the case by examining the traffic with a proxy.

Later, GCP takes our signed request and passes it to AWS STS (Security Token Service) and the response is used to verify the identity of the requester. This is quite different from the OIDC flow, because instead of a signed JWT token verifying its own authenticity, contact with AWS is required to authenticate.

Looking at the logs in AWS Cloudtrail, we can actually see the request retransmitted by GCP:

{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AAAAAA2SDBAWQJKAQ2SNM:i-03sa134b2672bsdqq",
"arn": "arn:aws:sts::123456111111:assumed-role/assumed-role/i-03sa134b2672bsdqq",
"accountId": "123456111111",
"accessKeyId": "AAQDSQ2BAADWR2BA1WAQ",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AAAAAA2SDBAWQJKAQ2SNM",
"arn": "arn:aws:iam::123456111111:role/assumed-role",
"accountId": "123456111111",
"userName": "assumed-role"
},
"webIdFederationData": {},
"attributes": {
"creationDate": "2023-01-23T09:38:24Z",
"mfaAuthenticated": "false"
},
"ec2RoleDelivery": "1.0"
}
},
"eventTime": "2023-01-23T09:43:27Z",
"eventSource": "sts.amazonaws.com",
"eventName": "GetCallerIdentity",
"awsRegion": "eu-central-1",
"sourceIPAddress": "107.178.196.97",
"userAgent": "Google-Identity-Federation",
"requestParameters": null,
"responseElements": null,
"requestID": "e0a2bsa1-bda1-4abf-9abc-ee9a2168fba6",
"eventID": "572abd73-5ab9-a8a6-8a8b-18af63s3d525",
"readOnly": true,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "123456111111",
"eventCategory": "Management",
"tlsDetails": {
"tlsVersion": "TLSv1.2",
"cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
"clientProvidedHostHeader": "sts.eu-central-1.amazonaws.com"
}
}

Note the Google-owned source IP "sourceIPAddress":"107.178.196.97", and the user agent "userAgent":"Google-Identity-Federation".

Misconfigurations

Although WIF is intended to prevent the risks of long-term credential storage, it can introduce other risks and misconfigurations if used incorrectly.

Vector 1 - Dangerous defaults V1

When first creating a workload identity pool, we need to attach the external identities as members to a specific service account.

We were about to click save and grant access to our service account, although something was not quite right.

If this default is chosen when granting access to a service account for our workload identity pool, it will grant access to all identities in the pool.

The first problem with this configuration is that any identity that can authenticate with a provider can access the workload identity pool’s attached service account, regardless of the environment it comes from (and its associated provider).Therefore, the attack surface for an attacker to laterally move to a GCP workload is large by default.

Let’s review a possible example. Suppose a victim has 3 external identities configured in his pool - AWS, Azure and Github actions and an attached service account configured to allow access to all identities in the pool. Any attacker that compromises any one of those external identities can federate themselves to the service account (even if it was only meant for external identities coming from, say, the AWS production account).

Users can misconfigure their workload identity by attaching it to a highly privileged service account, allowing privilege escalation from outside of Google’s cloud environment.
Below is a demonstration of a misconfigured service account that has iam.roles.update permission attached to it and is also attached to a workload identity pool.

Vector 2 - Dangerous defaults V2

AWS Provider

Let’s rewind a bit. Before attaching a service account to the workload identity pool, we created a workload identity pool and a workload identity provider.

After setting up the provider, we clicked “continue” and were about to lock it in and save our super secure provider.

Zooming in on the small print, we see that Google provides another, deeper layer of restriction any user can utilize to limit privileges to his identity pool. Despite that, by default, all identities belonging to providers in the pool we created can authenticate, and the default condition attributes attached to our provider are empty. That means any role under the AWS account ID we attached is allowed! This includes read-only roles, third-party roles, support roles, auditor roles, etc.

OIDC Provider

A decoded JWT OpenID token consists of keys and values that contain information about an authenticated user. For example, our Azure application JWT:

There are also other keys that may be included in an OpenID Connect token, depending on the implementation and the requested scope.

As a GCP user, it is important to validate those parameters with attribute conditions when creating a new OIDC provider. Just like the AWS provider, these conditions are not set by default, and when creating a new OIDC provider, these parameters are not validated.

It means that any identity that matches the Issuer URL set in the GCP provider is able to federate itself and get a Google service account token. We can give an Azure OIDC example, and infer from the following configuration screenshot that any Azure managed identity under the tenant id of 0a9d5a36-caa3-4aaf-9aa3-5aa1d6a96af5 and the application id efa2caf8-9a2a-4a7a-8a7a-1a5a3a1a02ac can successfully federate itself to the GCP service account attached:

Note that GCP warns users about the misconfiguration mentioned above:

Also note that we did not mention the “Default audience” property since it is currently not working in accordance with our tests and GCP users have to specify the allowed audiences for the feature to work. For example in Azure: the application id.

Takeaways

• Users should adhere to the least privilege model, be careful and take a long look at the permissions attached to the service account to prevent opening a new door for attackers with overly permissive access.
• Enforce custom attribute mapping and conditions to prevent arbitrary roles accessing your GCP workloads. In the OIDC example above, you can create a condition to check for a specific identity under the value of the sub parameter (in the JWT).
• Use app role assignments to restrict which identities can obtain an access token for your application.

Vector 3 - Provider Privilege Escalation

Suppose the user has correctly set his identity pool and restricted it to a specific role. Another vector users should consider is attackers getting their hands on the Update Workload Identity Pool Provider permission - iam.workloadIdentityPoolProviders.update and using it as a backdoor to an existing identity pool.

An example of a possible attack flow:

• A random GCP victim has a workload identity pool attached to a highly privileged service account in their environment.
• This workload identity pool is also attached to an external identity provider (e.g. AWS and an account ID).
• Attackers with the following permission to update the pool - iam.workloadIdentityPoolProviders.update, can update the victim's pool and attach their own AWS account ID to later on federate themselves into the highly-privileged service account attached to the victim’s pool they updated.
• Bonus - Attackers can use the following permission to create a new provider for the identity pool instead of the current one created by the victim: iam.googleapis.com/workloadIdentityPoolProviders.create

Some of you might have been wondering… can an attacker create a new identity pool and provider, attach a highly privileged service account to it and use it? Fortunately, to do so, attackers need the iam.serviceAccounts.setIamPolicy. Therefore, workload identity pool creation is useless in this specific scenario.

Last Takeaways

• Manage your workload identity roles and permissions carefully.
• Beware of these two specific permissions:

iam.workloadIdentityPoolProviders.update

iam.workloadIdentityPoolProviders.create