Pipeline Sprawl in DevOps: It’s a Thing

Originally published by Dazz.

Written by Tomer Schwartz, Co-founder & CTO, Dazz.

CI/CD pipeline sprawl is happening faster than you can rein it in

Companies are developing software in the cloud in a big way. Under the umbrella of digital transformation, and driven by customer expectations and competitive pressure, they’re building more software than ever — applications to run their business more efficiently, analyze the reams of data they’re collecting, and connect more intimately with their customers.

Technology teams have built formidable cloud-based DevOps stacks to transition from old-style development to fast-moving continuous integration/continuous deployment processes. They’ve secured those pipelines with best-in-class developer-first tools that help them “shift left,” addressing code flaws and infrastructure misconfigurations earlier in the dev cycle when they're easier to fix.

But are they addressing cloud security issues across all of their development pipelines? Using a Remediation Cloud to perform a cloud security remediation assessment across many of our customers, we found the answer to be a resounding no. In fact, in most assessments, we found CI/CD pipeline sprawl to be the cause of 50-80% of cloud vulnerabilities. The controls are available, they’re just not being implemented correctly in the right pipelines!

CI/CD pipeline sprawl is a thing, and it’s happening faster than we can rein it in. In just a few years, our customers’ cloud development efforts have mushroomed from a handful of projects to thousands today. And as their (often highly-distributed) development teams grow to keep up with the growth in new development, new engineers often don’t adhere to scanning protocols, and many aren’t even aware of the development security tools available to them.

What tipped us off to shadow pipelines

We performed a read-only integration with companies’ CI/CD pipeline tools, then integrated with detection tools to find (and de-duplicate) cloud security alerts across development and production. Usually, we found far more issues in production as in development, indicating a gap in early detection. Why was that?

The culprit: misconfigurations in development security tools

Automatically tracing these security issues back to their root causes, we saw two scenarios: issues that teams found and fixed in development that never made it to production, and issues they found in production that were never found earlier… because no one had end to end visibility to those pipelines.

The pipeline either looked like this, if the development security tool was properly configured:

‍

Or this, if it wasn’t:

‍

As you might imagine, finding and remediating these issues in development is far more efficient than waiting until they’re in production, where, of course, they present actual risk versus simply theoretical risk. Adding to the complexity, they also throw off duplicate alerts across many images AND are more time-consuming to troubleshoot and fix when they’re in production.

What securing development does

Once they brought the newly-discovered shadow pipelines into the fold, our customers cut the number of security alerts in production by 40%, and their mean time to remediate across the board by as much as 90%.

How we monitor them going forward

Tech stacks in fast-moving development pipelines are not static, and they never will be. Developers and DevOps engineers keep building new ones and refactoring the old ones. Security and development teams can stay ahead of changes — whether new authors, projects, cloud resources, or detection tools — by continuously discovering and automatically remediating the gaps.