CISOs, Sewers, and Unsolvable Problems

Originally published by CXO REvolutionaries.

Written by Daniel Ballmer, Senior Transformation Analyst, Zscaler.

CISOs and sewer engineers face similar challenges, and it goes beyond keeping the contents of their systems moving fast and smooth. There is a mathematical exercise that has many names, but I learned it as the New York City Sewer Problem. In short, it states that finding the optimal configuration of the NYC sewer system is a problem modern computing cannot solve in a human lifetime. By this, I mean we are unable to calculate every possible configuration of pipe diameter, pumping station location, etc., to determine the perfect layout.

Solving this problem only becomes more complex as the sewer system expands into new areas. This is similarly true for business enterprise technology. Both systems started off serving a small number of people and were expanded, ad hoc, as more users demanded their service. The original designers of the NYC sewer system are no longer with the city to provide guidance. As a CISO, you are likely also dealing with complex and sparsely documented systems designed by someone who moved on long ago.

“If it ain’t broke…”

When taking over business enterprise security (or the sewer system of a metropolis) your first impulse may be to leave everything in place. As the old saying goes, “If it ain’t broke, don’t fix it.” Yet, leaders cannot sit idly by until the wheels inevitably come off of the bus. Knowing they must act, CISOs often adopt new security tools to layer over existing ones. This tactic increases tech bloat and often introduces friction into workflows.

This expansion of security controls continues as organizations age. Every CISO leaves the original security structure in place while making new additions to address current threats. Likewise, wastewater engineers don’t have a holistic view of the NYC sewer system when they are asked to accommodate a new suburb. They simply want to get the job done in a timely and functional manner. Sewer systems and business enterprise security expand onward and outward, with no higher authority ensuring optimal growth.

This pattern results in a functional but inefficient system. While citizens may not care how quickly their wastewater reaches a treatment plant, businesses thrive on performing more efficiently than competitors.

Analysis requires visibility

It is extremely difficult to analyze a system without being able to see its inner workings. A toilet in a new housing development is flushed, the wastewater reaches the primary treatment plant 30 seconds later. The next day, the same flush takes 38 seconds to reach primary treatment, then 42 seconds. What is going on? Is there a leak somewhere in the pipes? Is there a blockage? Is the delay due to more houses being added to the wastewater system? While it may not concern those at the transmission point, a sewer engineer would want to understand the cause of the problem and fix it. Without scoping the pipes, or checking their status through other methods, it is impossible to know.

The same holds true for user experience. A CISO receives a complaint that users are experiencing slow connections after the latest round of security patches. The patch is critical, so rolling it back is not an option. Is the new security patch somehow interfering with SaaS connectivity? Is it interacting with something in the security stack and causing the slowdown? Is failing hardware a possible culprit?

Without visibility, you are going to have to play a long (and potentially expensive) game of testing components, one-by-one, until the situation improves. While you do this, the productivity of the workforce is creeping along at a suboptimal pace.

Using the right tools

Modern computing might struggle to find the optimal design of the NYC sewer system, but it’s quite capable of finding a good one. Once a problem’s answer is determined to be mathematically out of our grasp, it’s time to focus on close-to-perfect solutions. This quest eventually brought researchers to genetic algorithms, an optimization metaheuristic based on the idea of natural selection. Basically, genetic algorithms take known solutions to a problem and attempt to breed better ones by combining them. This two-minute video does a good job of quickly summing up the concept.

Engineers using genetic algorithms can find highly-fit solutions to complex problems like finding the best sewer layout. For example, it may not be possible to find the perfect configuration of the NYC sewer system, but we may discover a layout that is 99.3% optimal. We do this by quantifying several known layouts that work well, combining them, and testing their “children” for improvement. Repeat this process tens of thousands of times and you will arrive at a sewer layout that is pretty darn good. CISOs can use similar tools to optimize the security and performance of their enterprise.

Figure 1: Process for finding highly-fit solutions using genetic algorithms

Wastewater engineers have used genetic algorithms for decades, and the heuristic has become popular in machine learning and AI. CISOs have access to similar problem-solving tools and capabilities right now. Imagine you are a new CISO who is facing their own NYC sewer-sized problem, namely:

• Reduce IT costs and complexity
• Ensure a fast, seamless, user experience
• Significantly reduce cyber risk

Any one of these tasks could be a multi-year project. Yet, you are expected to accomplish all three tasks in a year or less. Like the wastewater engineer, your system documentation is slipshod, your visibility into the environment is spotty, and the original designers are long gone. You’ve inherited a bunch of subsystems that represent years of ad hoc expansion, but how they work together is unclear. How will you establish visibility into your environment and assert control?

SASE delivers

A secure access service edge (SASE) framework provides all the tools necessary for attaining visibility and control over your environment. It delivers a suite of integrated capabilities that can replace the hodge-podge of random tools and security layers businesses accumulate over time.

Figure 2: The complimentary security technologies in a SASE platform

A SASE platform will generally include the following integrated technologies:

• Secure web gateway (SWG)
• Zero trust network access (ZTNA)
• Cloud access security broker (CASB)
• Firewall-as-a-service (FWaaS)
• Software-defined wide area network (SD-WAN)

It may also include other security controls like inline decryption, data loss prevention (DLP), browser isolation, and deception technologies. By adopting a cloud-based SASE platform you leave behind many of the security problems associated with older, vulnerable network infrastructure. The platform manages securing and establishing connections between users, apps, devices, cloud workloads, and the internet. In doing so, it gives CISOs and security leaders a centralized overview of the state of their enterprise.

Consider the benefits of users using a SASE app connector to access applications (instead of authenticating users to the entire domain). What might you learn from seeing how individual users interact with specific applications? First, you could determine whether the security measures governing the connection are effective. Then, you can see how efficiently communication is occurring. Furthermore, automated AI tools could aggregate this kind of information from across the enterprise and offer suggestions for improving performance and security.

To quickly recap how SASE can help you achieve the three goals set forth by your hypothetical employer:

• Reduce IT costs and complexity: You replace an unwieldy security stack and vulnerable legacy network architecture with a single, cloud-based, integrated platform. By adopting cloud solutions where possible and coordinating communications through a SASE, antiquated security controls and devices can be retired.
• Ensure a fast, seamless, user experience: Devices use secure app connectors to route traffic to apps, cloud services, and the internet. Telemetry from the app connectors can be used to intelligently manage and optimize connections regardless of user location.
• Significantly reduce cyber risk: The integrated capabilities ZTNA, SWG, CASB, and in-line decryption greatly improve your security posture. Should an attacker manage to compromise an endpoint (eg. through stolen credentials), they cannot move laterally or discover other enterprise resources. Access is limited to the few resources available to the compromised account, and activity is monitored for suspicious behavior. Unexpected access and behavior can result in a re-authentication request, connection termination, or other measures.

This description of SASE barely scratches the surface of the advanced capabilities it provides. For example, AI-assisted tools can uncover your external attack surface, determine internal and external risk factors, and recommend fixes. You can generate automated risk reports that map your security controls to well-known frameworks like MITRE ATT&CK and NIST CSF. This can be useful for negotiating cyber insurance costs or filing SEC reports.

Settling for greatness

The old saying “don’t let perfect be the enemy of good” should be a guiding principle of CISOs and wastewater engineers. In both cases, advanced mathematics and technology delivers incredible solutions for problems plaguing the industry. While the perfect solution for enterprise security (and city sewer systems) may be beyond our reach, innovative approaches can deliver near-perfection. Making strides towards improving your security is a journey that steadily increases the distance between your organization and significant business risks.