What the DevOps movement has done for -- and to -- developers

It shouldn't come as a surprise that the evolution of -- and trends within -- the DevOps movement fascinates Gene Kim. What practices and ideas manifest within productive DevOps initiatives and shops?

Kim is a researcher, author and the founder of IT Revolution, an IT publishing and research company based in Portland, Ore. With the release of his book, The Unicorn Project, Kim joined the Test & Release podcast to talk DevOps, the five ideals covered in the novel, developer productivity -- and its dark side -- and other IT trends.

Practices and principles for DevOps initiatives

Developers express frustration in organizations that are rigidly bureaucratic, Kim explains. In fact, it's a central theme of The Unicorn Project, in which he articulates what stops developers from being productive, and what they require to get work done.

"Developers have to be productive in order for the organization to achieve its goal," he said. DevOps organizations should look for the behaviors and mindsets to achieve such productivity, and Kim has ideas about how and where to find them.

The Unicorn Project centers around Maxine, a fictitious senior lead developer and architect, who is exiled to an undesirable project as punishment for contributing to a payroll system outage. She must try to thrive in a bureaucratic, slow-moving system. As the protagonist works through her challenges, Kim lays out five ideals -- conditions that IT should follow to avoid common mistakes and achieve success. Those five ideals, which Kim discusses in our interview, are:

1. locality and simplicity;
2. focus, flow and joy;
3. improvement of daily work;
4. psychological safety; and
5. customer focus.

At the employee level, Kim emphasizes the importance of the tools and systems developers use for their everyday work.

"If you look at Google, Amazon, Netflix, Microsoft -- they put their best engineers on that bottom level, [i.e.,] on dev productivity," he said. Organizations' ability to be productive, in Kim's opinion, is not necessarily thanks to their vast resources. In this podcast, he asserts that, regardless of how many engineers a company employs, success is a matter of how much investment there is toward developer productivity.

Another DevOps -- or rather DevSecOps -- priority that Kim affirms in this podcast is the need to integrate security objectives into every team member's daily work in every department from product leadership to engineering, development, QA, ops or infrastructure. Share the responsibility for security tasks under the DevOps movement, he said; don't silo it with security specialists.

Facing up to developer burnout

Shared responsibilities in the DevOps approach, while inclusive, does put a burden on the team. Kim addresses how ever-increasing and broadening workloads can lead to developer burnout. The notions of a 10x engineer or full-stack developer are actually counterproductive, he argues. Both are IT professionals supposedly proficient at every step and with every technology in the SDLC, but this type of worker is near-impossible to find, and the workload is often unsustainable.

"To get developers productive, we need them just to focus -- to have their best energies spent on solving the business problem," he said. "The goal of infrastructure and security is really to enable those developers to work securely, safely and quickly with fast feedback."

That, of course, raises the question of how to even strike the right balance for developers' workloads and whose responsibility it is to push back when the equilibrium is unfair. Developers must communicate when overworked, Kim said, particularly when pressured to do tasks that are outside their expertise.

Evolution of the DevOps movement

Kim, whose company IT Revolution organizes the DevOps Enterprise Summit conferences, reflected on how differently the industry sees the DevOps movement in 2019 compared to 2013.

"I think [in 2013] people associated DevOps with the unicorns," he said. "It was very easy for people to say, 'DevOps is for them, not me.'" Now, he said, DevOps is ubiquitous, as organizations try to incorporate principles and practices, with numerous case studies from which to learn.

Kim discusses consolidation among DevOps vendors, and why he thinks it's a good thing for DevOps engineers. And he reveals why he believes data will fuel the next wave of innovation within the DevOps movement.

Editor's note: Kim spoke with site editor David Carty and assistant site editor Ryan Black. The transcript has been lightly edited for clarity and brevity.

DevOps is a mindset. But it's one that requires an actual course correction in terms of organizational structure and collaboration. So for the forward-thinking person, maybe a person leaving the DevOps Enterprise Summit and trying not just to inspire their team members, but to get them to make discernible changes, what advice would you give them? How can they bring about change in a practical manner?

Gene Kim: Yeah, I think one of the things that I do admire most about the DevOps enterprise community is that the people who are figuring out how to elevate the status of practice in their technology organizations, in large, complex organizations, are often surrounded by very powerful entrenched, conservative organizations that really don't want to change the way they're working. And I think over the last six years of the DevOps Enterprise Summit, the level of the resistance has been conveyed so well. I think what I admire so much about the technology leaders that we're presenting, [is that they] are the ones who are succeeding. They're getting promoted, and the stories they tell, for me, are just so admirable. And there's the level of courageousness it takes and fortitude to keep fighting the good fight. And yet, to me, it's beyond question that these people are being appreciated by their organizations; they are being given more responsibility.

The advice I would give is that -- I think we've done so much work around the State of DevOps Report, and that study shows that DevOps principles and patterns are great for IT performance [and] organizational performance. And I think the experience, of course, highlighted at the DevOps Enterprise Summit, [it] really intends to showcase the success stories so that we can learn what behaviors to model, what mindsets to adopt, so that hopefully we can replicate those amazing outcomes.

Is there anything new among these common traits and success stories that you've seen lately that that's been a little surprising to you?

Kim: We spent the last couple of years [and] a portion of the programming to keep increasing the amount that we're giving to technology leaders presenting with their business counterparts. We're not looking for the business counterpart who is there just to be in close proximity to the person. We're looking for the person who is going to say, 'Thank goodness for my technology counterpart, because without that person, none of my goals, dreams and aspirations could have come true.' We're looking for that business counterpart who is going to brag about all their exploits and achievements that were done in conjunction with a technology leader.

Over the years, I think the level of seniority [of] that business person keeps rising, just as the levels of seniority among the technology leader speakers keeps rising. To me, that's just an indication of just how important and how relevant these technology leaders are within their organizations. I think that's just a great trend to see and just another indication that right is on the side of the people who are pushing DevOps initiatives within these organizations.

You've got probably a lot of friends among the different software companies out there, Gene, but I did want to ask you about tools that pertain to the DevOps space: We've seen a lot of consolidation in the last year -- CloudBees and SmartBear come to mind immediately [as] some of the more aggressive companies out there. How do you think this type of consolidation will ultimately impact users? What are some of the potential positives and negatives you see coming out of that?

Kim: Yeah, I think positive is -- I think my personal interpretation is that it just shows that the space is vibrant, and there's a ton of great innovation going on, and that these acquisitions are happening with a premium multiple being associated with it. I can think of other eras where the nature of acquisitions, where they were all fire sales and that doesn't feel good to anyone. I'm just delighted to see this level of interest in the software segment.

I think the other thing that's really interesting to me is the fact that it is happening across all the user communities, whether it's developers, designers, QA, operations, security. So, I think what it shows me is that DevOps really touches everyone involved in [the] technology value stream, and we're all benefiting from it. So it's not like QA is disappearing, or ops and infrastructure is disappearing. Instead, it really is these principles and patterns being applied to everyone that does technology work. And to me, that's very, very exciting.

Sure. But if you're a smaller vendor in that space, do you look at these sorts of acquisitions as putting your customer base at risk? Are those smaller vendors more at risk of being swept up by the competition of the emerging giants there?

Or maybe crowded out?

Kim: Possibly. I guess I kind of view it as two extremes. One, it's a very competitive space and, holy cow, to survive and thrive in the space, it actually takes a lot of skill these days. It's not like 20 years ago, where maybe there wasn't much competition. Yeah, but I would take that any day, over the opposite where we're building products that no one cares about and whatever acquisitions are happening are being done at bargain-basement prices. So I think we all want to be where the innovation is happening, and where the party's at.

Gene, I'm sure mileage will vary depending on the organization and compliance needs, that sort of thing, but what sorts of security processes do you see as mandatory for more mature DevOps shops? Do you recommend something like DevSecOps, rugged DevOps or another organized sort of approach like that?

Kim: Oh, absolutely. In fact, to your question: What DevOps security practices are required? All of them! I don't care what you call it; I have friends in all of those communities, however we segment ourselves, but essentially what they're all saying is that we need to integrate information security objectives into everyone else's daily work. And that applies whether we're a product leader, engineering, development, QA, operations or infrastructure. Increasingly, it's not security work in its own silo working within, through the GRC tools. Really, it's how you get that expertise embedded into every other domain of engineering.

I think the surface area keeps increasing, the amount of data keeps increasing. So, [with] the work that needs to be done, [security's] relevance keeps going up. And, so, thank goodness we can all help carry the responsibilities and share the security responsibilities. So, I actually think it's just a great time to be in security these days.

Switching gears to The Unicorn Project, and congratulations on the book. I know it's coming out soon. But I did want to ask you; you try to present some common IT struggles in a narrative form, just as you did with The Phoenix Project. So I want to briefly read from the book description here to set the scene for our listeners: [paraphrased] Maxine, a senior lead developer and architect, tries to survive in what feels like a heartless and uncaring bureaucracy and to work within a system where no one can get anything done without endless committees, paperwork and approvals. Sounds a little bit like a dystopia, but I'm sure there's some real-world inspiration to this character and this situation. Can you tell me what are some of the top frustrations you hear from developers, particularly the ones that are in this sort of bureaucratic system that you describe?

Kim: Yeah, in fact, I think this dystopia is really what the DevOps enterprise community is trying to overthrow, right? In other words, it is a rebellion trying to overthrow this kind of powerful, uncaring order. And so The Unicorn Project really is The Phoenix Project retold from a developer perspective.

And if I were to zoom way back, right, why are these problems important? [It's] because there all these invisible structures that are required to make developers productive. And developers have to be productive in order for the organization to achieve its goal. [Which is] impeded by technical debt. It's impeded by a culture of fear. And it's impeded by architectures. [It is] very difficult for individual teams to get anything done by themselves. Instead, they have to work with 20, 30, 40 different other teams to actually get into production. Or, even working on [a] feature, they might have to work with five or 10 different teams that all have to communicate, coordinate, synchronize, marshal, prioritize together.

I think one of the things that I just find dazzling these days is that, just like in the DevOps movement, the problem that we were trying to solve was how to get the features and code into where it needs to go, which is in production, where customers can actually start getting value from it. There's other kind of parallel universes -- orthogonal universe, where there's the same problem with data where we have data trapped inside of these data warehouses, and in order to get data to the teams that actually need it, that might take six, nine months to actually do a simple SQL change. And even that's very error prone, where suddenly you have people's addresses showing up in the ZIP code field. Just all these things, I think, conspired to make it very, very difficult for developers to actually get what needs to get done, done.

Right, sort of in that same sympathetic mindset in the book, you seem to focus a bit more on the mindset and well-being of IT professionals, at least from what we've seen in the book description so far. So, how would you recommend managers and executives promote a culture that is both healthy for IT workers and developers, and encourages creativity without taking a hit to productivity?

Kim: That's a great question. So yeah, in The Phoenix Project, we use the constructs of the three ways and the four types of work. And so in The Unicorn Project, we're using the notion of the five ideals.

So the first ideal is locality and simplicity -- that's really architectural concerns. So how do you really organize teams and code so that we can get things done by touching only one thing instead of having to touch everything?

The second ideal is really a side effect of that. It's when we have a little challenge, the simplicity we can work with [is] focus, flow and even joy.

The third ideal is the improvement of daily work. So this [really has] to come from leadership, that we prioritize improvement of daily work even over the daily work itself.

The fourth ideal is psychological safety. How do you create conditions so that members on teams and members within an organization feel like they can say what they really think, without fear of being castigated or blamed -- that they can make mistakes, they can take risks?

And then the fifth ideal is customer focus. [I] really think that the construct there is having an unflinching view of which things are core versus context. So 'core' are things that really create competitive advantage, that customers want, that customers are willing to pay for. Whereas 'context' is everything else; it might be mission critical, but customers actually don't value [it]. In other words, having a world class payroll system is something that customers are not willing to pay a premium for, that makes it context. So it's important, but not necessary -- but the sum of all that context conspires to starve us of our ability to fund core.

So those are the things that I think are as relevant to engineers and engineering leaders as they are for the top levels of management.

Hi, Gene, it's Ryan. I was wondering, do you see any sort of connection between the phenomenon that is developer burnout and also the increased expectations in the industry for developers in DevOps to perform tasks related to security, QA, etc.? Essentially duties that maybe someone would think is traditionally outside their wheelhouse, because, of course, [developers] also have all their coding and programming responsibilities [still].

Kim: Yes, this is such a great concept. I think that the whole notion of the 10x engineer or the full-stack engineer is absolutely at the core of it. In my experience, that is the opposite of what I want to do. In fact, for the last 20 years I've self-identified primarily as an ops person, despite getting my master's [degree] in compiler design and high-speed networking, I've really identified as ops just because I think that was where the most fun was happening; that's where the saves are made. And I think it was also because for the vast majority of the last couple decades, someone needed to protect customers from developers.

But these days -- three years ago, I started self-identifying as a developer, and I think it's because of learning Clojure, the functional programming language. And I've never had so much fun programming. But the side effect is that I no longer want to work on anything outside of my application. I just want to work within my pure little bubble. I don't want to connect it to anything. I don't want to have to deal with secrets or security. I don't want to deal with infrastructure or YAML configuration files. And, for me, what my experience with that suggests is that, really, we don't want this 10x engineer who's forced to work with everything up and down the entire application stack, and [with every] database and environment, and make files, configurations and SSL certificates. Really, to get developers productive, we need them just to focus -- to have their best energies spent on solving the business problem. And everything else we should do through platforms.

So as a security or an infrastructure person, we want our expertise in the platform, whether it's in [container management technology] Kubernetes, or a platform as a service, or whether as a security person we want it in libraries or things that we can pull from. Because that [all] enables developers to work with focus and achieve a state of flow, where they're actually having fun with conditions of fast feedback. And Dr. [Mihaly] Csikszentmihalyi he wrote this amazing book called Flow, on what it takes to have joy in our work. He says these are the conditions where we can actually have joy in our work. I think that all this is a very long way of saying, 'We don't want developers to be 10x engineers. We want them to be focused on solving a business problem. And the goal of infrastructure and security is really to enable those developers to work securely, safely and quickly with fast feedback.' Does that resonate with you, Ryan?

That does make sense. The one question I would have as a follow up though is: You talked a bit about how that better balance could be found, but I [am] wondering who, in your opinion, is the impetus on within an organization to find and strike that better balance? Is it on the developers themselves to advocate to get to a place where they can have that flow, or is it on the organization themselves? Or, their parent organization, I should say.

Kim: Oh, man, that's a great question. I think it's everybody. [For] the person doing the work, the person whose daily work is to do development, I think it's having the fussiness … to say, 'Hey look, I don't want to be having to stand up my own CI server. I don't want to have to Google [some] Stack Overflow security settings. I don't want to have to learn how to do secrets management all by myself.' Really, that [all] should be done by someone who spent decades doing that. And that person should be … not doing work through the ticketing system, we want that person's expertise to be embedded in platforms.

Similarly, anyone in the infrastructure and security domains, we want our work -- we should feel our job is to take the best knowledge we have and make it accessible to everyone in the organization, whether you're present or not, because it should be in the code and the platform. And then certainly this has implications for how we're organized, how teams organize [and for] the platform and architectures we create. So that typically comes from the leaders in the company. I think it takes all of those constituencies; certainly having just leadership on board and no one else wanting to do that, that doesn't work. Nor does having all the people doing the work [taking a stand], but not having leadership on board. I think [that's] really one of the goals of The Unicorn Project, [to] really show in high fidelity, what do the problems look like and feel like, and to go through the DevOps enterprise journey of taking some of the best patterns that we've seen in that community and just showing how it unfolds and transforms how everybody works within a technology organization. In fact, not just technology, but throughout the entire business.

You brought up the book and I did actually have another question about it. I saw a recent write-up of a DevOpsDays talk you did, and it said that in the talk you talked about there's a 'spectrum of decisions' that, 'enable greatness or create underperformance.' Does that sound familiar?

Kim: Yeah, I think this is the Westrum organizational typology model about culture -- of cultures of fear, cultures of bureaucracy and cultures of, they're called generative cultures. Is that what you're referring to?

I believe so. I was wondering if you could explain maybe a bit more [about] what exactly that spectrum is, and what exactly it enables/creates?

Kim: Yeah, for sure. So it's called the Westrum model [in] State of DevOps report that showed up in 2015, and it's one of my favorites. So, Dr. Ron Westrum, what he discovered almost 20 years ago when he was studying healthcare organizations was that, [with] the organizations with the best patient outcomes and the fewest number of accidents, it was highly correlated with the culture of the organization. They had this beautiful model of three kind of archetypal categories.

The first one he called pathological organizations. These were the ones with the worst healthcare outcomes; the information was hidden, messengers of bad news were shot, bridging between teams was discouraged. So in this world, [those teams were] maybe patient intake, nursing, surgery, pharmacy, outpatient care. And new ideas were crushed. In the middle, they had what's called bureaucratic cultures: [what's] considered merciful cultures or just cultures. And then at the highest level is what he called generative cultures. These were the ones with the best patient outcomes. These are organizations where information is actively sought, messengers are trained to tell bad news. In our world, this would be like training every engineer to lead blameless post-mortems, so we can create an accurate chronology of what happened [and] better prevent these problems from happening in the future. We encourage responsibilities to be shared. Just as we were talking about before, info sec is [not just] info sec's job, just like operations and availability is not just ops' job -- it's everybody's job. And when failures happen, it causes a genuine sense of inquiry.

What we found in the State of DevOps report was this notion of culture, as measured by the Westrum model, was one of the top predictors of performance. I just find it interesting that this all embodies the notion of psychological safety. I think it gives us a language to talk about.

One more thing. I think it would be a mistake to say that this is just a job of leaders to set the cultural tone. Psychological safety is also a function of our peers, our people's histories and backgrounds, our moods are heightened, so psychological safety is so fragile. There's this wonderful study that has been done by Google … that just shows how important psychological safety is; it was actually one of the top determinants of team effectiveness and effectiveness of managers.

Psychological safety, that was one of the ideals you mentioned earlier, right?

Yes, it was ideal number four: psychological safety.

I did want to ask you about those ideals as well. Because I know you say that the ideals are pretty central to The Unicorn Project, and in previous books I think you've also said that you focused on principles and practices. So how exactly would you differentiate those two?

Kim: The three ways [in The Phoenix Project] are a set of principles from which you can derive all of the patterns that we see put in practice. I think the five ideals are just -- I don't know if they're orthogonal. I can't say exactly, with precision, the relationship to each other.

Just to tell a story: In January, [I] had just gotten done with the first draft, and I read through it, after almost a year and a half of full-time work on it. My feeling was, 'Oh, my gosh, is this just a pile of words that say nothing?' [Laughs] [I spent] a couple of weeks of soul searching, and I think what came out of it [were] these five things, that I just think are so important, that really frame what goes wrong in organizations. And it actually shows us the direction of what we want. So, sorry, I can't give you a more precise answer than that. But I think they are really sort of five things that represent these peak experiences that we have and the conditions we need around us in order to make that happen. And also [the conditions], which I assert, are also I think what we need for our organizations to actually win in the marketplace.

I did also want to ask you about kind of like this existential question of feature velocity versus quality. That seems to be the dichotomy that everyone always tends to think about, one versus the other. From your conversations with people in the industry, how do you see DevOps high adopters -- how do they strike this balance between technical debt and feature velocity?

Kim: So here's the way I frame it in my head: When I talk about The Unicorn Project [and] the one thing that it talks about [is] these invisible structures that enable developers to get their work done. And when I say 'developers,' I mean really everybody.

So my mental model is, it's like these three things, like a triangle. At the very top you have features. Everyone can see [the features], and everyone values that because we can see the app or you can see our feature or the app on our phone, and so anyone can get funding for that. But underneath it is this middle level of all these architectures that allow us to build the features or the systems that can get us the data we need -- so these are the back-end APIs and so forth. But even underneath that are all the systems and tools that developers use as part of the daily work, and those are almost invisible, and almost no one values those. These are even valued less than the architectures.

What I find amazing is that: If you look at Google, Amazon, Netflix, Microsoft -- they put their best engineers on that bottom level, on dev productivity. So they spend somewhere between 3 to 5% of all their developers on dev productivity. It's not just junior engineers, it's their absolute best engineers. In Google's case, I think that's 1,500 developers working on developer productivity. Microsoft has like 3,000 to 5,000 developers working on dev productivity. Contrast that to most organizations where they give those tasks to the summer interns, or people who aren't good enough to work on features. And I think this shows why the Facebooks, Amazons, Netflixes, Googles, Microsofts are so much more productive -- they can generate so many more features. [It's] because they put their best people on making developers more productive.

A quick question about that, though, is that reflective of the fact that those organizations, like Amazon and Microsoft, have the most resources and can afford to do that? Or, is the fact that they're successful because they do that?

Kim: I would assert it's the second, that in any organization you're going to see that, regardless of how many engineers they have, [success is a matter of] how much are they investing in making developers productive. And to do that, these are the non-functional requirements; these are things that are not features. Satya Nadella, the CEO of Microsoft, [has] said, 'If any engineer has to choose between working on a feature or working on dev productivity, always choose dev productivity.' I think that what he's saying is that it's like the opposite of technical debt; now you're playing the game where you have compounded interest working in your favor. He's playing the forever game; the more you work on becoming more productive, the more you can get done.

There [was] a great presentation at DevOps Enterprise London from a genealogy company. [The organization has] 500 people, and it's just an amazing story. Not in the large, like at Google, but in the small. It's just amazing to see how the same things that you see in play at Google and Microsoft are happening in smaller software companies. I just think the evidence is very decisive that these things are as relevant, regardless of whether you have 60,000 engineers, like Microsoft, or 50 -- or even five -- engineers.

In the more than six years that have passed between the debut of The Phoenix Project and now the launch of The Unicorn Project, what are the more significant ways in which you've seen the DevOps movement evolve? And what do you think is next for the DevOps movement?

Kim: For one thing, when The Phoenix Project came out in 2013, I think then people associated DevOps with the unicorns -- the Facebooks, Amazons, Netflixes, Googles Microsofts -- and it was very easy for people to say, 'DevOps is for them, not me.' I think certainly something that's changed in the six years is that increasingly DevOps is an important agenda item for every software company. The Marc Andreessen quote that, 'Software is eating the world,', the Jeffrey Immelt quote from GE saying, 'Every company is a software company.' I think that [reality] is increasingly accepted by almost everyone inside of most modern organizations -- and by modern organization, I mean, organizations that are still around -- they might be hundreds of years old. Now the question really becomes: How do we incorporate those principles and practices into organizations? And I just love that the DevOps enterprise community I think has done such a great part in helping build that body of evidence to show [that] not only can it be done, but here's how it's being done. I think there's over 300-plus case studies [now] that have been recorded that [are] available for everyone to download and watch, show organizations across every industry vertical, government agencies, not-for-profits, showing what they're doing, why they're doing it and what the outcomes were.

I think the second thing that's changed is really the elevation of data. There's this great book by Risto Siilasmaa, he's the chairman of Nokia who wrote this amazing book called Transforming Nokia. [As] the chairman of the board, he was there during the declining years -- their exit of the phone business and their five-fold increase in market cap [since he became chairman in mid-2012], as they have started to take market share away from the top vendors in the carrier market in the course of switching companies. Anyway, so in this talk he says everyone at Nokia, needs to understand five things. One is you have to have an intuition of what machine learning is, and what problems can and can't be solved. Two is they have to be able to see the business problems in front of them and if they see opportunities where machine learning can help with achievement and solving those problems, they should have the resources available to do that. Three, he was talking about how critical data is to the company. He even makes this point that says, 'Our customers, which are large telco carriers, they don't want to give us their data, and it is now our job to show us how it's actually good for them.' And it's so valuable to the company that maybe they need to even pay their customers for that data, because it is that important for the future of the company. Then he makes this kind of incredible claim that says, 'We have to identify [for] the company what the most strategic sources of data are,' so that they can make decisive steps to build that data or even acquire that data. I think, for me, it was just a breathtaking view of how visionary business leaders view the importance of data.

That's certainly something I've tried to put into The Unicorn Project as really kind of showing how data enables innovation, and it certainly introduces a whole bunch of challenges in terms of: How do we get that data? How do we keep teams responsible, accountable for cleaning it and making it available for other teams? I think that is much more at the forefront [now] than it was, say, six years ago.

Well, Gene, this has been really insightful. Thank you so much.

Kim: Oh, man. David and Ryan, thank you so much.