6 lessons learned from 'The Phoenix Project'

1. DevOps is a broad approach to work in general, not just specific to IT

The term DevOps is a portmanteau of the words development and operations, which are two departments that have opposite goals but must work together seamlessly to succeed. Development is where work starts, where the product is created, and operations is where the product is managed and applied. If the departments do not understand each other's goals, they will stifle each other, and the whole system will suffer.

Gene Kim

The words themselves are "IT words," so to speak, but they represent a dynamic present in all types of organizations and any system with an input and output. Put simply, development is creative chaos, and operations is order. A balance is necessary for success.

The book consistently compares IT work to manufacturing work because the same basic truth about work is in both. It's also why DevOps can be considered a philosophy of work.

"Currently, DevOps is more like a philosophical movement, not yet a precise collection of practices, descriptive or prescriptive." -- Gene Kim, author of The Phoenix Project

While this is true to a degree now, it was truer when the book was published in 2013. DevOps is still a broad philosophy, but now, sets of common practices are developing as more organizations adopt DevOps. It would be more accurate to say that DevOps is both a philosophy and a collection of practices that have sprung from that philosophy, as illustrated by this next quote.

"DevOps and its resulting technical, architectural and cultural practices represent a convergence of many philosophical and management movements (including): lean, Theory of Constraints, Toyota production system, resilience engineering, learning organizations, safety culture, human factors, high-trust management cultures, servant leadership, organizational change management and Agile methods." -- Gene Kim

2. For DevOps to succeed, systems thinking and global thinking are key

The Phoenix Project mentions systems thinking and global thinking as part of The Three Ways of DevOps. For the company to function, everyone should have an idea of the larger goal that the departments are working together to solve -- both business and IT goals included.

Bill often consults a DevOps guru named Erik, a Yoda-like figure with a background in manufacturing. He constantly compares the work Bill does in IT to the work that takes place in a factory -- something is created, managed and shipped, all under a time-constraint. The simple lesson he teaches Bill boils down to the fact that the different parts of the manufacturing line need to work together and not get bogged down in or blinded by their own process.

"If you think IT Operations has nothing to learn from Plant Operations, you're wrong. Dead wrong." -- Erik Reid, board candidate in The Phoenix Project

Erik worked his way up from the factory floor to owning the factory and, as a result, has both a bottom-up perspective and a top-down perspective of the way a factory works. He shows us -- and Bill -- that DevOps is a way of thinking about work that includes the macro and the micro. Each department, both in the factory and IT, must understand the big picture and how its small-picture, departmental goals fit into it.

"Until you gain a better understanding of what work is, any conversation we have about controlling work will be totally lost on you. It would be like talking about acrobatics to someone who doesn't believe in gravity yet." -- Erik Reid

By understanding the system and their place in it, departments can view problems as systemic instead of blaming each other. Departments can then adjust their respective departmental goals and processes in pursuit of a common goal, instead of each fixating on their individual goals with no context.

Having small, cross-functional teams own the entire process is one way to encourage systems thinking. Groups are small enough to keep feedback loops tight and stay organized and cross-functional so that they understand the goals of other departments and how their work affects other departments. This way, they can maximize efficiency by decreasing work in progress and limiting batch sizes. At no point is the product completely out of their control.

3. DevOps is about adapting to, managing and thriving on change

"It's crazy what programmers, and even managers like me, have to learn every couple of years. Sometimes it's a totally new database technology, a new programming or project management method, or a new technology delivery model, like cloud computing." -- Chris Allers, VP of application development in The Phoenix Project

The Phoenix Project starts with Bill being bombarded with unplanned work and having no reliable process for completing it. By the end of the book, he has positioned the company to handle unplanned work so that it provides value instead of obstructing planned work. Continuous experimentation and chaos engineering also can result in more unplanned work, which Bill and Parts Unlimited are able to handle, and even welcome, by the end of the book because they plan for unplanned work.

One example of this is the implementation of Project Chaos Monkey, which is a continuous and random penetration testing (pen testing) project run by John Pesche, chief information security officer (CISO) at Parts Unlimited. By building sturdier processes, the company can continuously attack its own networks for the sake of improving security and know that, even if this causes an outage, it won't be catastrophic. With a strong foundation, it can afford to take the risks necessary for change and improvement.

4. For a company to change its way of working, its culture must change, not just its processes

The unique thing about The Phoenix Project is that it is a novel, not just a textbook. Some chapters are dense with terminology, but others focus on the relationships between the characters and Bill's inner monologues. This style of writing illustrates the relationship between hard and soft skills and how the technical side and the human side of work interact. Sometimes, the reason a technical solution does not work is because of a personal problem and vice versa.

As Bill goes about reforming the processes at Parts Unlimited, he finds that a huge part of implementing the necessary changes is getting his co-workers to trust one another. This enables characters in different departments to give each other the benefit of the doubt and not blame each other on a personal level for the problems they encounter. Trust is necessary for systems thinking.

"If Ops goes to lunch with Ops and Dev goes to lunch with Dev, a low level of efficiency is a certainty." -- Unknown

Trust is often built during activities that, at first glance, may seem unproductive. In one chapter, Bill takes a character he was previously at odds with out for drinks, where they can relate on a personal level and realize that their goals are ultimately the same. In another chapter, all the characters gather to share their life stories and talk about some of their vulnerabilities as people. These moments, though seemingly trivial, build trust so that they can rely on each other when disaster strikes.

"I've been trying to keep John's people in line for years. They never follow our process, and it always causes problems." -- Patty McKee, director of IT service support in The Phoenix Project

John is one character that goes through a dramatic change over the course of the book. He starts by being intentionally excluded from meetings and ends up taking initiative and reforming the processes Patty refers to in this quote. It's true that the processes Patty is talking about are clunky, and many people don't use them. But, because John is always excluded, he has no interest in helping to improve them at first. When he starts getting invited to meetings, he starts to participate in reforming those processes instead of ignoring them and causing problems.

"Wow, boss. You don't hang out at the watercooler much, do you?" -- Patty McKee

5. In the DevOps methodology, failures are a learning opportunity

The first half of The Phoenix Project can feel like an endless barrage of disasters. In the second chapter, for example, Bill is tasked with fixing a payroll outage that, unchecked, would result in many employees not being paid on time. When he goes to investigate it, he and his colleagues don't know where to start.

This is partially because the company rarely uses the clunky change management processes in place. Most employees view it as a waste of time, so instead, they call on engineers individually to fix their problems without documenting the change. Because so many small failures and changes go undocumented, when a big one -- like a payroll outage -- comes along, there is no timeline to work with and no way to work backward and find the source of the issue. It turns a serious problem into a catastrophic one.

"Don't give me that bullshit about 'throwing the pig over the wall.' We invited your people to our architecture and planning meetings, but I can count on one hand the number of times you guys actually showed up. We routinely have had to wait days or even weeks to get anything we need from you guys." -- Chris Allers

Having proper change management processes -- that people actually use and understand the importance of using -- enables the company to learn from its failures in the future. Organizations can afford to fail more often because preemptive safety measures have been put in place.

On the development side of things, continuous delivery (CD) employs the same principle. Code is kept in an always deployable state, and over time, many iterations are tested and delivered so that the organization can fail fast and learn more quickly from these low-stakes failures. Embracing failure and having processes in place to quickly learn what doesn't work give organizations a head start in learning what does.

However, failing for the sake of it is still counterproductive. One principle of The Three Ways of DevOps is to ensure that work is done right the first time to minimize upstream work. Choosing to delay a release that a developer feels is likely to fail is the lesser of two evils. A failure is a valuable learning opportunity but only when it exposes new information. Knowingly deploying substandard code teaches you nothing -- you already knew it would fail -- and wastes time sending it back upstream to be fixed. The time spent fixing the problem is only valuable when something new is learned in the process.

6. IT is more than a department in the company; it is a central competency that supports every department in the company

This may be the most important lesson -- that having a functional IT department is about as valuable to the modern enterprise as having access to electricity. Every department uses services that the IT department provides and maintains. In the book, when things go bad, the business side of the company tends to treat every "tech problem" as the IT department's fault. What they take for granted is that every department uses tech and relies on IT to keep it running. Business goals and IT should line up and often don't when the departments don't understand each other's work.

"My official title doesn't have 'operations' anywhere in it, but it's the part of my job that I love most. When a company is as big as we are, with so many business processes, so many managers and workers, almost everything is complex. As smart as Steve is, even he needs help making sure the company strategy and goals are realistic and making an objective assessment of what we're actually capable of." -- Dick Landry, chief financial officer (CFO) in The Phoenix Project

As Bill and his colleagues learn throughout the book, it is important for every department in the company to understand the work necessary to uphold that standard. One of the main challenges the company faces and eventually surmounts is realizing they all play a part in maintaining and improving that infrastructure, even if it doesn't directly align with a business goal.

"That's some pretty nice, soaring rhetoric, Steve. Very moving. But you know what your problem is? You guys in the business are punch drunk on projects, taking on new work that doesn't have a prayer of succeeding. Why? Because you have no idea what capacity you actually have. You're like the guy who is always writing checks that bounce, because you don't know how much money you have and never bother opening your mail." -- Erik Reid

This rift in understanding the importance of IT is especially clear in the first chapter when Chief Executive Officer (CEO) Steve Masters is trying to persuade Bill to take the promotion to VP of IT operations. Steve says to Bill that he needs someone reliable, up to any challenge, which appeals to Bill's sense of duty and desire to solve problems -- traits that are common in IT workers. However, once Bill accepts the promotion, Steve says what he really means -- that IT is like keeping the toilets running. He should never have to think about it; they should just work. While there is some truth to Steve's analogy, that task is extremely difficult when left up to a single, siloed IT department, like the one Parts Unlimited has at the beginning of the book.

"Great, he sees me as a glorified janitor." -- Bill Palmer

It's this dynamic that can cause IT workers to be unsatisfied in their jobs as Bill is in the book's first half. Erik, Bill's mentor, points out at the end of the book that companies without comprehensive IT departments will find themselves in deep trouble, as technology becomes more pervasive in everyday work environments and the speed at which it evolves only accelerates. The bottom line is that business goals and IT goals are ultimately the same, even if they seem at odds.