Proactive Threat Hunting with Anomaly Detection in the Cloud

Originally published by Uptycs.

Written by Laura Kenner.

As our systems increasingly shift towards the cloud, security strategies must evolve to protect these novel environments. Traditional security approaches may not apply or function optimally in the cloud's unique landscape.

Craig Chamberlain, our Director of Algorithmic Threat Detection at Uptycs, recently delved into this conundrum during his enlightening presentation at the 2023 SANS Cyber Solutions Fest. He emphasized the crucial role of anomaly detection in the cloud.

Proactive threat hunting in the cloud

Many cloud resources, like instances and container workloads, can be safeguarded with monitoring and security measures akin to those used for on-premises systems. But the realm of pure cloud services, which includes everything from message queues to logging, security, and code execution services, demands a different approach.

Traditional endpoint agents or network security monitoring tools can't always provide the coverage needed, creating a blind spot in our defense. This is where the value of API logs truly shines. These logs offer a comprehensive audit trail, capturing every transaction within a cloud service, whether it's initiated by humans or automated systems.

The real power, however, lies not in the logs themselves but in our ability to detect anomalies within their data. By meticulously analyzing these logs, we can spot inconsistencies and potential threats, making anomaly detection a vital tool in our cloud security strategy.

What is anomaly detection?

Anomaly detection is a crucial component of your cloud security strategy, and this holds especially true in the context of cloud threat hunting. As Craig Chamberlain points out, anomaly detection is about finding data that deviates from what is considered 'normal'. It’s the process of identifying unexpected events or outliers in data sets that do not conform to the expected patterns. In the realm of cybersecurity, these anomalies can often signal malicious activity or system faults that require immediate attention.

Finding a needle in the needle stack

Chamberlain highlights the sheer volume of data generated in a cloud environment, making it difficult to manually parse for potential threats. The need for anomaly detection becomes clear in this scenario. With CloudTrail logs in AWS, for example, it's normal to see millions, or even billions, of transactions every week. Manual inspection of such colossal amounts of data is practically impossible, highlighting the necessity of automated anomaly detection. It acts as an effective filter to identify potentially malicious or otherwise interesting events from the 'noise'.

“In many cases, the difference between completely innocent and normal user activity day-to-day and threat actor activity that's taking place via credentialed access or compromised credentials, is often a matter of nuance. Sometimes the nuance is small enough that it's hard to see and it goes unnoticed.”

- Craig Chamberlain, Director of Algorithmic Threat Detection at Uptycs

Real-world example

To bring the concept into the real world, Chamberlain presents a case involving an unusual combination of method and role. He discusses an event where a role was found executing the 'describe instances' command - a benign command that lists virtual servers but had never been associated with that particular role. This case underscores how anomaly detection can help unearth unusual combinations, potentially pointing to suspicious activities.

Where in the world? - Detect geographic anomalies

Chamberlain also delves into the detection of geographical anomalies. Here, the focus is on an example where 'assume role' events, a common action in AWS, were detected originating from an unexpected geographical source, in this case, China. While the 'assume role' action is typical in AWS operations, its occurrence from an unusual geographical location was flagged as an anomaly.

Discover new threats in the wild

Anomaly detection is a proactive method for uncovering new and emerging threats. Chamberlain discusses an example where a 'get federation token' method, which was newly discovered to be potentially used as a persistence mechanism by attackers, was identified as an anomaly since it was being used by an account that had not used it before. This illustrates the potential of anomaly detection in identifying threats even when specific threat intelligence may not be available yet.

Overcome false positives

False positives, a common challenge in cybersecurity, are also a part of anomaly detection. In his talk, Chamberlain explains how the detection of rare functions can result in false positives. However, he also suggests that utilizing 'new functions', looking for new combinations of methods and roles, can drastically improve the signal-to-noise ratio, helping security teams focus on more actionable insights.

Leverage machine learning

Machine learning can significantly aid in anomaly detection. We’re working on using techniques like clustering and algorithms like K-means to identify additional interesting things that can be hard to find. Machine learning algorithms can sift through vast amounts of data to learn what's normal and then alert on the abnormal, acting as a powerful ally in the fight against cyber threats.

Dive Deeper: Check out these additional resources

Traditional security approaches fall short when dealing with the complexities and unique challenges of the cloud environment. It's clear from Craig Chamberlain's presentation that anomaly detection, driven by a deep dive into API logs, provides an effective solution for identifying potential threats and ensuring the security of our cloud systems.

There's much more to learn about anomaly detection in the cloud. For a more in-depth understanding, explore Craig’s previous blog How Anomaly Detection Advances Threat Hunting & Detection—Especially in the Cloud for a comprehensive overview of the subject. It's time to take control of your cloud security and let the data tell the story.