Does Cloud Log Management Shield You From Threats? CloudTrail vs CloudWatch

Originally published by Sysdig.

Written by Brett Wolmarans, Sysdig.

What is different about cloud log management versus on-premises? The answer may seem simple, but several elements such as CloudTrail vs. CloudWatch come into play.

In this article, we will cover some of the most important differences, and then dig deeper into a specific example of AWS CloudTrail vs. CloudWatch.

What are cloud logs?

Let’s say you run an Apache web server (remember to check if it is updated so as not to be vulnerable to the new cve-2022-42889) in your cloud provider, AWS for example. When it fails a web request, this is logged to /var/log/httpd/error_log on the machine. This log entry would happen even if Apache was deployed in a brick-and-mortar environment.

Therefore, we are dealing with something that is not a cloud log.

On the other hand, cloud logs come directly from your cloud services. What’s only found in the cloud are what we assume to be true cloud logs.

To define cloud logs, here is this example that shows an Identity and Access Management (IAM) user, called 'Brett,' using the AWS CLI to call the Amazon EC2 StartInstances action. It uses the ec2-start-instances command for the instance name 'i-ebeaf9e2' because its previous state was 'stopped.'

{"Records": [{
    "eventVersion": "1.0",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "EX_PRINCIPAL_ID",
        "arn": "arn:aws:iam::123456789012:user/Brett",
        "accessKeyId": "EXAMPLE_KEY_ID",
        "accountId": "123456789012",
        "userName": "Brett"
    },
    "eventTime": "2014-03-06T21:22:54Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "StartInstances",
    "awsRegion": "us-east-2",
    "sourceIPAddress": "<External-IP>",
    "userAgent": "ec2-api-tools 1.6.12.2",
        "instanceId": "i-ebeaf9e2",
        "currentState": {
            "code": 0,
            "name": "pending"
        },
        "previousState": {
            "code": 80,
            "name": "stopped"
        }
    }]}}
}]}

The current state is 'pending' as a result of the action.

How should you manage your logs?

At a high level, you have to decide how you are going to implement these three processes:

• Creating logs
• Transmitting logs
• Storing logs (including rotating and deleting logs)

Best practices

AWS log management best practices is a big topic. For example, if you follow the Payment Card Industry (PCI) standard, you have to store logs for at least one year. We don’t have room to cover best practices here, but the folks over at Loggly have a nice list, and AWS maintains a list specifically for security events as well.

Some best practices are:

• If you send logs remotely, use TCP and encryption.
• For AWS CloudTrail, if you want logs longer than 90 days, you have to create a Trail.
• In a microservices environment, it is not feasible or practical to SSH into containers to see what is happening, so logs become our eyes. Collect all the data you can because the more data you capture, the more visibility you have.

If you want to learn more about best practices, dig deeper with 26 AWS Security Best Practices.

New challenges inside cloud log management

The cloud brings new challenges with it:

• Multi-cloud: It means dealing with different logging specific functionality, and types of logs.
• Cloud logs can be massive: Even a medium-sized cloud environment may generate millions of log events per day. Strategies to control costs related to sheer logging volume are one consideration here. Another solution is to add runtime detection to the cloud detection and response, and try to avoid storing a large lake of data and speed up the alerting process.
• Security concerns: Logs are a very valuable target for attackers, either to steal critical information or to find ways to hide their tracks by deleting them.

With all this in mind, what solutions can we apply to cover the different threats we may face?

Public cloud logging services compared

The major public cloud providers have a very important audit log capability: to log all the changes made in their cloud. Think of this as a log for every change on every device in your entire datacenter. Without this, you would have no paper trail and no record of changes made.

There are logs for virtual machines, security, databases, and many other services.

All the major cloud providers offer a logging agent for their virtual machines. For example, AWS offers the CloudWatch agent which can be injected into an EC2 instance using several techniques, including Userdata at provisioning time

A sample of the logs available in Google Cloud, Microsoft Azure, and AWS is shown in the following table:

To write something useful about all of these would make this blog too long, so instead, let’s take a look at two major logs in one of the clouds. In this case, we will compare and contrast AWS CloudTrail vs. CloudWatch.

AWS CloudTrail vs. CloudWatch

What is AWS CloudTrail?

AWS CloudTrail is an AWS service that helps you enable operational and risk auditing, governance, and compliance of your AWS account.

Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS software development kit (SDK) and application programming interfaces (API).

CloudTrail is enabled by default on your AWS account. You can easily view recent events in the CloudTrail console by going to Event history.

For an ongoing record of activity and events in your AWS account, create a trail. Otherwise, they will be deleted from the default S3 bucket after 90 days. Therefore, it is important to be quick in detecting and responding to a vulnerability or abuse of our accounts and that the logs used to detect the source of the problem do not disappear.

What is Amazon CloudWatch?

Amazon CloudWatch is a performance monitoring service, but it can do a few more tricks than that. CloudWatch also includes the capability to detect anomalous behavior, visualize logs, and set alarms. However, keep in mind that very little of this extended functionality is available out-of-the-box. You may need to write code in AWS Lambda and configure triggers in Amazon EventBridge to get this working.

Source

By default, AWS CloudWatch exists to monitor the performance of your AWS resources and applications. AWS CloudWatch graphs metrics over time, and is not out-of-box a way to get security alerts. If you want alerts, you have to create your own, based on metrics. And if you want to create an alert for a discrete event, you have to generate a non-zero metric and key off of that.

What kinds of threats can be identified in advance in the cloud logs?

Depending on the type of event that needs to be detected, we will use one type of cloud log management service or another. For this example, let’s configure Cloudtrail to find a simple event: making an S3 bucket public. You can be forgiven if you might think that public s3 bucket misconfiguration is no longer relevant.

In fact, this cloud mistake is still very relevant. For example, there is a GitHub project created just to track widely known public s3 bucket incidents, as well as a website with a search engine to find exposed cloud storages.

Let’s continue with our example. First, we make the bucket public, which thankfully includes a dire warning and a flow that makes you click twice to avoid doing this by mistake:

Within a few minutes, this event shows up in CloudTrail:

"requestParameters": {
        "publicAccessBlock": "",
        "bucketName": "brett-public-test-bucket",
        "PublicAccessBlockConfiguration": {
            "xmlns": "http://s3.amazonaws.com/doc/2006-03-01/",
            "RestrictPublicBuckets": false,
            "BlockPublicPolicy": false,
            "BlockPublicAcls": false,
            "IgnorePublicAcls": false
        },

Here, we see “BlockPublicPolicy” and the other parameters being set to “false” that together make this bucket public on the Internet.

In this scenario, CloudWatch could be useful to detect massive bucket usage. But not so useful for a bucket changing from private to public.

Finding the event in CloudWatch vs. Cloudtrail

We found a few ways to find this event in CloudWatch, but there is no easy button.

There are some complex ways to integrate CloudTrail with CloudWatch, including henry and s3-public-alerts. However, these projects have not been updated in some time, and we could not successfully get them to work.

We followed this set of steps from AWS exactly, and we were able to see our bucket alert, but only in AWS Config. Also, the blog post is a little bit out of date and AWS services have been updated since then, so maybe that’s why we had some challenges.

If the scenario we have is completely different, our main cloud detection and response tool could be CloudWatch.

Let's say, for example, that we have been affected by an attacker that has taken control of our EC2 machines and has started cryptomining. The way we have to detect the attack is based on the percentage of CPU used, as it will have increased dramatically. For this, cloudtrail is not much help but CloudWatch gives us valuable information.

Other services within AWS also use them, such as GuardDuty.

Therefore, depending on the threat we are facing, we must be clear about which service is the most suitable for detecting it.

Useful tools

During the writing of this article, we experimented with several useful tools focused around what we initially thought would be a very simple event: finding out “who, what, when, and where” somebody changed an S3 bucket from private to public.

Our results were mixed, but it helps prove our point that cloud logs are not something to take lightly.

You really have to understand their functionality and how the different parts work together. Finally, you have to understand their limitations.

Other examples to be detected by this cloud log service could be:

• Exploitation of vulnerable lambdas
• Cloud Lateral movement by the attackers
• Changes in DNS configuration
• Networking misconfiguration with Security Groups

Conclusions

Logging in the cloud needs to be top of mind. If done incorrectly, it can lead to confusion and even disaster.

Cloud log management can be quite different than in the traditional datacenter. For example, there is no single audit log for the entire datacenter like there is in the cloud. There are several surprising limitations.

The visibility you have within the cloud depends entirely on how you manage logs, and should be a priority. Each cloud provider offers many types of logs, but at the top of the list is the audit log. This is also a key place to look for security issues. Another key piece is the strategy you take when storing them, since a closer option to realtime helps in the early cloud detection and response of malicious events.

For example, when we compare AWS CloudTrail vs. CloudWatch, we find that although these are sometimes both described as logging functions, they are very different. And while they can be made to work together, it often requires a lot of additional configuration.