One Day Is a Lifetime in Container Years

By Jon King, Security Technologist and Principal Engineer, Intel Security

Securing virtual assets that appear and disappear.

The average life span of a container is short and getting shorter. While some organizations use containers as replacements for virtual machines, many are using them increasingly for elastic compute resources, with life spans measured in hours or even minutes. Containers allow an organization to treat the individual servers providing a service as disposable units, to be shut down or spun up on a whim when traffic or behavior dictates. Since the value of an individual container is low, and startup time is short, a company can be far more aggressive about its scaling policies, allowing the container service to scale both up and down faster. Since new containers can be spun up on the order of seconds or sub seconds instead of minutes, they also allow an organization to scale down further than would previously have provided sufficient available overhead to manage traffic spikes. Finally, if a service is advanced enough to have automated monitoring and self-healing, a minuscule perturbation in container behavior might be sufficient to cause the misbehaving instance to be destroyed and a new container started in its place. At container speeds, behavior and traffic monitoring happens too quickly for humans to process and react. By the time an event is triaged, assigned, and investigated, the container will be gone. Security and retention policies need to be set correctly from the time the container is spawned. Is this workload allowed to run in this location? Are rules set up to manage the arbitration between security policies and SLAs? The volume of events from containers also overwhelms human capabilities. Automation and machine learning are essential to collect this data, filter it, and augment the human security professionals who are doing triage. Identifying suspicious traffic or unexpected container behavior through pattern recognition, correlation, and historical comparisons are essential jobs that machines are very good at. Perhaps the biggest issue with container life spans is the potential lack of information available for investigations. If you have a container breach, the container is probably gone when you need it for forensic details. It’s like the scene of a crime being deleted before the detectives arrive. The good news is that if you collect information from a container while it is running, you have a wealth of information available to you. Memory dumps can be captured and analyzed for traces of a malware infection or exfiltration function. And stopped containers can be saved for later analysis. Done well, this is like going back in time to a crime scene, able to examine every detail—not just the faint traces the criminal left behind. Of course, saving this type of data is counter to many of the container benefits of ephemerality, and could quickly consume a huge amount of storage, so once again automation and machine learning are crucial to help decide what artifacts to retain. As the latest form of resource virtualization, containers enable a new and growing set of security opportunities and threats. Actively involving the security team in container architecture discussions will make sure you are using them to best advantage.