Four Reasons for Alert Fatigue and How to Make It Stop

This blog was originally published by LogicHub here.

Written by Ryan Thomas, VP of Product Management, LogicHub.

• Alert (or alarm) fatigue is the phenomenon of becoming desensitized (and thus ignoring or failing to respond appropriately) to signals meant to warn us about emergencies.
• IT security operations professionals are especially prone to this fatigue due to systems that are overloaded with data and can't accurately triage alerts.
• False positives, human error and the rising cost of data storage all contribute to alert fatigue. The best solution right now? Intelligent automation.

1. (Unintentional) Bad Habits

Habits are powerful. At their best, they can be a positive force for good. Making the bed, a meditation practice, a daily walk after dinner — they're the building blocks of a healthy life. But habituation — a psychological process our bodies naturally go through, which is meant to help reduce stress and prevent sensory overload, can be catastrophic if applied in the wrong circumstance.

Habituation is your mind's way of eliminating your awareness of unpleasant or distressing signals when they're happening incessantly — and seem to have no bearing on immediate consequences. Your brain realizes that the noise doesn’t affect you, so you tune it out. Technically, you don’t stop hearing it, but it seems that way because you stop paying attention to it.

This is the root cause of alert (or alarm) fatigue: the phenomenon of becoming desensitized to alerts. This fatigue leads to us ignoring or failing to respond appropriately to sounds, messages or other signals meant to warn us of an impending danger.

When the urgency of alerts is diminished, critical alarms lose the significance they deserve

Alert fatigue is a widespread common phenomenon among IT security operations professionals. Why does it happen, and how can we make it stop?

2. SIEM Technology Is Outdated

Let's face it: SIEM (Security Information and Event Management) technology was game-changing when it was new – some 20 years ago. But this first-generation approach to security operations is insufficient to manage the massive amounts of data flowing through today's business ecosystems.

Until recently, SIEMs weren't designed to handle the terabytes (and teraflops) of data most enterprises generate now. With the rise of SIEM in the cloud, handling this amount of data is possible, but typically with an egregious cost associated. Most teams however, lack the speed and nuance to do so without overwhelming the team as a whole. It is nearly impossible to separate the "signal from the noise,” so even skilled analysts have a difficult time “hearing” the real ones. Who could when they are getting scores of them within the span of a day?

The harsh reality is that a significant percentage of alerts are overlooked. When security teams are only able to investigate a fraction of alerts, many turn out to be “false positives,” and chances are some of the true threats go unattended. Whether these alerts are triaged by people or automation, the problem is unavoidable. If many events require no real response, the alerts lose their urgency. So human alert fatigue increases exponentially.

SIEM vendors charge the equivalent of airport prices for storage fees. To offset costs, some security operations teams will only upload certain data to their SIEM rather than everything that passes through their systems. The result is that critical threats may go undetected. But what if you could do exponentially more analysis with less labor? The cost savings could enable more storage and more data processing.

3. Systems Overload

Today's security operations center (SOC) teams use dozens of applications and tools. In many cases, the tools are not integrated, so an analyst spends a tremendous amount of time toggling between various systems in an attempt to synthesize the information they need before they can even respond to a threat. Stating the obvious, even with the use of SIEM as the aggregation point, this model is inefficient.

4. We’re Only Human, After All

Security is a 24/7, 365-day-a-year job. Unfortunately, we humans can't survive without sleep and occasional breaks (even though some of us try). Plus, many small to medium-sized businesses are unable to staff their SOCs around the clock. But even dozens of analysts working nonstop couldn't review the massive amounts of data most organizations need to handle.

And when resources are tight, most of us need our people to do what they do best: the high-level work that only humans can do.

Pro Tip: In the midst of a skilled labor shortage, drowning your team in alerts and tedium is not the best way to retain the people who make an organization successful.

Let the Machines Handle It

While SIEM systems are simple and rules based, a modern approach to detection and response is driven by intelligent decision automation. Alerts should be instantly triaged by bots that follow playbooks created by security experts. Think of these bots as your analysts' always-on assistants who never sleep, never get tired, and operate at machine speeds.

As the bots encounter the unique circumstances of your business processes and systems, they should progressively learn, evolve, and respond within your organization. But ultimately, human decision making takes precedence.

About the Author

As a 25-year information security veteran, Ryan brings a wealth of customer empathy to his role as head of product management at LogicHub. With early experience in security consulting, Ryan taught classes in white hat hacking, cryptography, and intrusion detection, and has secured the networks of major telcos, banks, health care providers. As head of solution development at ArcSight, Ryan shifted his focus to product management and brought numerous solutions to the SIEM market including regulatory compliance offerings and a first of its kind identity monitoring solution. Prior to LogicHub, Ryan served as VP of Product Management at a cloud infrastructure security provider. Ryan earned a B.S. in Physics from the University of Georgia.