How to Establish a Culture of Secure DevOps

This blog was originally published by Sysdig here.

Written by Chris Kranz, Sysdig.

We’re constantly told to “Shift Left” and that Secure DevOps is the only way to have confidence in your cloud native applications. But speaking to end-users and industry colleagues, it’s clear that there are some major challenges in adopting Secure DevOps. If we read our history books, we know that DevOps wasn’t successfully adopted by buying tools, and a true cultural movement towards DevOps wasn’t established by having a small dedicated team of DevOps specialists. I’m sure many of us have read the Phoenix Project and the follow on Unicorn Project (if you haven’t, highly recommended, almost essential reading!).

So what are some of the key takeaways from the first part of this cultural transformation that can help us with adopting a culture of Secure DevOps? Two of the major challenges that we see and hear about in the industry and from our customers:

• Lack of education & training (both on security for DevOps teams, and on DevOps for security teams).
• Lack of involvement with security.

Short of mandating a full cultural shift, what are some simple things we can do to help start moving the needle on these? I personally think there’s two very powerful things you can do. Actually, it’s one thing but in two different directions.

Things you’ll need before conducting this exercise:

• A security team or teams.
• A DevOps or DevOps adjacent (CloudOps / InfraOps / etc.) team or teams.
• Commitment to invest a little “free” time (more on this later).

Foreign Exchange Students

When I was in school, we had a foreign exchange program. I spent a week in France with a French family, speaking nothing but French. I was pretty terrible at the language, so I’m surprised I survived, but it was a great experience to live and breathe a different culture, language, and way of life.

To many, security is a totally different language, and (for various very valid reasons) it’s often a very different culture than the fast paced DevOps and Cloud world. So what if we take the foreign exchange student approach?

First, get some of your DevOps engineers, CloudOps folks, Platform architects, yes even application developers (for simplicity, I’m just going to refer to this group as DevOps teams going forward), and assign them to work in one of the security teams for one day a month. Let them get involved in security teams’ day-to-day operations, especially if there’s an incident or a major project going on.

Next, get some of your security folks (architects, engineers, SOC analysts, CSIRT, etc.) and assign them to work in one of the DevOps teams. If your DevOps teams work in sprints, think about assigning someone from security through an entire sprint (including the pre and post ceremonies) – This increases the “free” time I mentioned earlier, but seeing a full sprint through has advantages of understanding the whole context of activities and processes.

I find that a good approach here is to pair your foreign exchange student up directly with a specific individual, get them to shadow their typical tasks and working day. If you rotate this across the teams, you’ll probably end up with a few folks being foreign exchange students at any point in time (depending on your company size and shape). By creating these pairings, you establish more of a personal relationship rather than a generic, “this is the team” and people flounder in the corner looking lost.

There are a handful of golden rules to make this effective:

• Leave your current job behind. As an exchange student, I couldn’t speak English or talk to my school friends, I was fully embedded with my French family. So this should be similar. Dismiss yourself from any existing projects, being on-call, turn mobile notifications off, etc.
• Ask questions! This is super important. If you don’t understand something, ask. The best way for me to survive my French immersion was to ask questions and work things out. Quickly, I found out that “croissant” is the same in French. Who knew? The best way to learn is to ask questions; question those TLAs and don’t be afraid to ask why. If you are the foreign exchange host, don’t assume any level of knowledge. Explain the lingo and acronyms, as well as pet names for systems and solutions.
• Shed any arrogance, there’s no shame in learning and there’s nothing to prove by knowing more than someone who doesn’t know anything about what you know! There’s no score being kept. It’s important to understand that this process will make your life better in the future, so it’s crucial to embrace this and share as much knowledge as possible! Yes, it can be frustrating (I have a toddler that’s recently learned the question “why?”, trust me I know frustration levels never before understood), but this is important in the learning process.
• Don’t try to look for immediate metric improvements or ROI models. It can be tempting to say, “well I’m giving up this time, so I want to see a correlated improvement in the next few months to show this is worthwhile.” These things can take awhile to show their true benefits and cultural changes are often hard to measure. Try to focus on the gradual benefits over a longer period of time, but this could be things like amount of rework, security blocking deployments, vulnerabilities discovered, or other security-based metrics in your release cycles.

But Why?

“I get you Chris, but we’re fighting fires here and we don’t have the time to spare anyone for an hour a month, let alone a day a month!”

Parkinson’s Law explains: “Work expands to fill the time available for its completion.” But that’s also oversimplifying it. A percentage of your work is undoubtedly filled with rework caused by lack of knowledge or understanding of someone else’s work. Security said no, so we need to go back and change something. DevOps said yes, so security needs to make some changes to accommodate.

As a side note, although we’re specifically talking about security here, I personally think this is a really useful exercise to do across the business. Go spend time with legal, procurement, HR, etc. You’ll be surprised what improvements you can make by doing this!

One day per month, based on a 40 hour work week, is less than 30 minutes per day. That’s definitely procrastination time you lose already, or the time lost in context switching through distractions (potentially even caused by the rework itself), multi-tasking, and my favourite, “have you got 2 minutes for a quick question?” (I carefully selected this reference as it’s clear that sometimes you should interrupt someone). I wouldn’t be surprised to find there’s a psychological impact of seeing two people already working together and chatting that makes “just a quick question” interruption less likely than a solo worker. I can almost guarantee you won’t lose much productivity time by giving up one day per month to this exercise, and the potential gains are hard to argue.

Benefits

There are some core benefits of this simple exercise that are difficult to ignore or overlook…

Education

Without engaging in training programs, online courses, or other educational methods, your DevOps teams will get to learn relevant information about security, and security about DevOps. It will be contextually relevant to your business, which is often a hard thing to do just learning from generic training material. And let's be honest, it’s going to be pretty cheap too (once you get over the misconception of lost productivity).

Additionally on education, I think it’s widely recognised and proven that you are more likely to remember something if you continue to learn that thing. So doing a single training course will have less of a benefit compared to an internal foreign exchange programme where you continue to reinforce and learn. That’s not to say a training course isn’t worth it, but you need to make sure you adopt and continue what you learn, which is easier using the suggested method.

Culture

Nothing destroys a great company culture like siloed teams. By breaking down these traditional internal firewalls between teams, you get to spread a culture of support, teamwork, and collaboration across those boundaries. An additional benefit is summed up in Conway’s Law. The TL;DR is that organisations that design systems are constrained to produce designs, which are copies of the communication structures in these organisations.

What does that mean? If the security team isn’t part of your application design team or communication structure, security won’t be part of your application design!

Establishing these bonds with security is a great way to get them included in the DevOps culture, and it helps create habits of security within the organisation. This means that security isn’t an afterthought, it’s there as part of the design phase, build phase, implementation, acceptance process, and so on. It simply becomes an immovable part of the process. Dare I say, it’s fundamental to the “shift left” movement!

Engagement

Hopefully, we know that DevOps isn’t about getting a handful individuals to know “DevOps,” but is rather about building cross-functional teams that include the relevant expertise. Secure DevOps is exactly the same. Your DevOps teams need security stakeholders. That could be folks from security, or it could be your foreign exchange students that took an interest in security and became specialists. It could even be a combination!

By engaging with security during the foreign exchange programme, you are engaging them in everything you do. This means you get rapid feedback: “Yeah, publishing a payment processing system without TLS is probably a bad idea,” “We generally don’t encourage Telnet anymore,” “Maybe that database should have a password and the management interface not exposed to the internet,” etc. But it also means that security gets better visibility in what the business is trying to do.

I bet that when you start this programme, you’ll find that many people are surprised at what others are doing or working on.

Friendships

Maybe this is a bit fluffy and some of the harder management types will scoff at this, but I love my job because of the people I work with. Sure, I get to work with a great product in a growing industry, but what makes me smile every day is that I get to work with awesome people.

Happiness at work is a really complex topic, so I’m not going to go into all the details here, but happy workers are productive, more loyal, produce better work, and really help with staff retention!

Business goals aside, it’s just nice to have a happy workplace! There’s enough terrible things going on in the world that should mean everyone is going to benefit from making a few new friends that help them smile. What can be so bad about that? As a manager, my job is simply (!) to make sure my team is happy, and sometimes that means breaking down silos and getting people to talk to each other!

Automation

Maybe this isn’t directly obvious, but this is something that will also result from this cultural change.

DevOps is very centric around automation, so involving security can open up ideas for automating security. This could be simply automating security decisions, or it could be key security designs. As you get more advanced, it could be audits and even automated responses. Automation of security is a really important goal to move towards. Automation allows you to focus on the more value-add activities, and going back to the time investment, it’s one of the areas for biggest time saving improvements.

By spending time with DevOps, security can make useful suggestions on embedded security controls, best practices, and more directly in the existing automation workflows. On the other hand, DevOps spending time with security can show automation tools and techniques that could help improve their workflows.

Takeaways

Establish a foreign exchange programme right now! If you’re a manager, then get your teams together and start working out the schedule. Talk to the team about the benefits and encourage collaboration However, don’t mandate it or force people into it. If you have folks on your team that don’t want to do this, leave them off the rotation to begin with and focus on the members that want to give this a try. Soon enough, everyone will realise the benefits and improvements being made.

If you’re an employee, speak to your manager about wanting to do this yourself. What have they got to lose, one day a month? That’s a risk worth taking, and you can be the guinea pig. Make sure to write up your experiences and the benefits you’ve gained, even if it’s anecdotal or subjective. Many of the benefits are difficult to show with real data, and they can take time to establish and prove. Stick with it!

Finally, don’t make this a one-off effort. Make this a continuous programme, get your exchange students (or yourself) to do this regularly. This will help both with the repetition of learning, and also in helping establish those cross-functional bonds and new relationships.

But at the very least, please do this one thing: invite security into your stand-ups, scrum ceremonies, sprint planning and product demos. This is very low effort but will immediately boost engagement with the security teams.