New Agile 2 development aims to plug gaps, complement DevOps

Developers bristling under Agile directives may find helpful suggestions in a new Agile 2 statement from a 15-member international team with expertise in product development, program management, systems engineering and business.

The Agile 2 initiative, released on Oct. 19, aimed to tackle important issues the group claims current Agile development practices fail to address, including big data, distributed systems and leadership at scale. It also attempts to offer alternatives to one-size-fits-all Agile principles, such as face-to-face conversation, that may not be optimal with remote teams, which have become common in the wake of the COVID-19 pandemic. 

"What we're saying is: Don't use our stuff as a Bible, like you did with Agile. Just treat it as a set of ideas. Use other ideas, too. Be open-minded. But don't lock yourself into today's 'Agile' ideas. Think more broadly," said Cliff Berg, the Agile and DevOps consultant who initiated the Agile 2 project and assembled the global team.

Berg expressed hopes that Agile 2 would empower people who are disenfranchised in Agile development environments, where managers often impose practices without input from the developers they will affect.

Berg's long career in technology started with electronic systems design and compiler work. He later co-founded and served as CTO of Digital Focus, which built technology platforms for companies such as FedEx, McKesson and large banks. Berg more recently founded Agile Griffin, a boutique firm that provides Agile coaches and focuses on merging Agile and DevOps. He also teaches an online course on DevOps testing and contributes to several open source projects.

In this recent interview with SearchAppArchitecture, Berg discussed the impetus for Agile 2 and how its principles relate to popular development approaches such as DevOps and microservices.

What was the impetus for Agile 2?

Cliff Berg

Cliff Berg: A large part of the community feels that Agile has some correct core ideas but doesn't work right the way it's generally implemented or expressed. I felt that a lot of processes were too heavyweight, too top-down and too document-centric. The Agile movement definitely pointed to some real dysfunction. There are terrible problems among the Agile community, which has kind of splintered and turned into a turf war.

Agile 2 was done because frankly I, and others in the group, feel that Agile is severely at risk. I'm a DevOps trainer and do a lot of DevOps consulting. I fear that organizations today often feel they have a choice between Agile and DevOps, and they're increasingly going to just choose DevOps. We're trying to show you need both, and here's how you synthesize those ideas. Here's a better model for Agile that's more compatible with DevOps.

Agile 2 is a rich set of content. We wanted to create a thoughtful piece of work. There's a lot in it. I think there are 43 principles. It's not something you can put on a bumper sticker. That's why we didn't call it a manifesto. We didn't want it to be an emotional battle cry. We don't feel we have all the answers. We think Agile is great. We're not trying to tear it down. But we think some things about it are wrong and need to be fixed to help make it work better.

What are the main gaps in the original Agile Manifesto that Agile 2 attempts to address?

Berg: We feel that the idea of leadership has been extremely poorly addressed by the Agile community. Leadership is something that we address front and center in Agile 2. Leadership is a very complex, multi-dimensional subject, and there are many different forms and aspects of leadership that apply in different situations. We tried to identify those facets, and we reached out to other work in that field. There are thousands of books about leadership.

Besides leadership, there are issues pertaining to collaboration. The Agile community has culturally evolved into a kind of one-size-fits-all collaboration model where the favored mode is everybody in person, talking in a room. We feel that is an excellent model for some situations and some people, but that's a tiny slice of what effective collaboration looks like. Effective collaboration often takes many forms. It depends on the complexity of the subject, the nature of what you're collaborating about, the personalities of the people and the amount of time you have. Collaboration is something that needs to be done thoughtfully.

There's also the issue of people being able to focus. The Agile community is very evangelistic in general about people being near each other and talking within earshot. But, in my experience -- and this is borne out by comments on programmer websites -- programmers hate that. They don't like sitting near everybody, where they can't concentrate.

Another thing that was completely unaddressed by the Agile Manifesto had to do with data and information. It's not just about building working software. It's a step in creating data and using information effectively.

How would you characterize Agile 2 in relation to DevOps?

Berg: One of the basic ideas of Agile was, instead of creating documents and passing them along, let's think about what the customer actually wants and build that. If it were a healthy community, Agile would have evolved into DevOps, but unfortunately, it got stuck. There were various interest groups that seized on Agile to monetize it and kind of froze it in its tracks. Agile turned into different methodologies and frameworks. People got certified and wanted to insist everybody use those methods.

DevOps techniques developed at organizations like Amazon, Google and Netflix because they were trying to solve real problems from an engineering perspective: 'How do we deploy new code changes many times a day, at enormous scale, and have great confidence that it actually will work?' That's agility, and that's DevOps.

The techniques you hear about today for DevOps are really continuous delivery techniques. The first person I'm aware of who connected the dots and started talking about it was Jez Humble. His book, Continuous Delivery, came out in 2010. He uses the word Agile more than 40 times in that book. He went to Agile conferences, and his talks were well received. But it didn't get back to the Agile community at large at all. You never heard Agile coaches talking about it.

So, DevOps became its own movement. That's why today, if you talk to DevOps people, they generally don't know much about Agile. And if you talk to Agile coaches, 90% of them know nothing about DevOps. That's a severe dysfunction.

In what ways could Agile 2 help to make DevOps less separate and distinct from Agile?

Berg: Agile 2 is not about practices, so you're not going to see DevOps practices in it. It's principles. But many of those principles speak to ideas like, for example, the need to focus on a product instead of a team. DevOps is about the product. Agile's obsessive focus with the team really is not quite right. We need to be thinking about the product, too. Agile 2 emphasizes the product. It also emphasizes the individual, not just the team, and systems thinking.

We emphasize in Agile 2 that you cannot have an intellectual silo, whether you're on the business side, the coaching side, the technical notation side or the product design side. People need to take an interest in all aspects of a product, not just the business or the technical aspects. Today you often have a dysfunction where Agile coaches don't take an interest in how the product works or how it's built.

That's why I created DevOps training for Agile coaches. I have to teach them some programming, because they generally don't have a programming background. They don't need to be experts, but they need to take an interest in it. Otherwise, they cannot participate in conversations that the teams are having. 

Which Agile 2 principle is perhaps most important for DevOps?

Berg: The big one is leadership. I see lots of product teams acting independently, with a complete lack of leadership pulling them together. Organizations that try to be Agile often set up Scrum teams, and the development managers in charge don't know how to manage them because it's not part of their experience.

Say there are 30 teams for a product, which is a very common size for a large company. You might have a development manager in charge of those teams but not providing enough coordination, or not taking the steps needed to make sure those teams work together as a unit and deliver the right thing in a timely manner. It's just assumed that they will somehow self-organize. So, it's like 30 cats all doing different things. They coordinate somewhat, but nowhere near enough for what they need to do. 

How does Agile 2 apply to CI/CD [continuous integration/continuous delivery]?

Berg: CI comes from extreme programming. It originates from a time when your product was all-in-one runtime, where you had one big piece of code that was all compiled and linked together and you'd unit test on all the methods in that code. But today, integration is different. Today, integration is multiple runtimes. It's hundreds of microservices, and they're not all on the same runtime. So the CI methods don't work for that. The integration needs to include deployments to test environments to do integration tests on multiple runtimes in real time. There's concurrency on a whole different paradigm in terms of testing. And Scrum teams generally don't have a clue of how to do that.

We have some principles related to continuous improvement, and one of them is integrate early and often. That's a nod to what's in the DevOps world. The basic idea is, if you have a group of people or several groups of people, they need to frequently check that their stuff works together, instead of waiting for the end for some integration phase.

Why should people take the Agile 2 initiative more seriously than past attempts to improve on the Agile Manifesto?

Berg: The group of 15 people that I put together has broad expertise. We have people from product design, program management, systems engineering, DevOps, human resources and business. It's a very diverse group, not like the original [Agile] group that was pretty homogeneous in many ways.

We have people who are fairly accomplished, like Kurt Cagle, who has written [22] books in IT. Navneet Nair worked at Google in product design. Everyone in the group has a lot of real-world experience and is a thoughtful person. The main criteria were that they had to have demonstrated independent thinking through their writing or something that I've seen, and they must not have a lot invested in the status quo. We wanted fresh thinking.

If you look at the Agile Manifesto and some of these other efforts to create updates to it, there are statements like 'We believe this' or 'We value that.' Our work provides all of the thinking and narrative behind each idea. We feel that it's a much more thoughtful and nuanced piece of work addressing complex issues. It cannot be summarized in a few sentences.

Is there a commercial aspect to Agile 2?

Berg: There's no commercial intent. We have a book coming out at the end of February or in early March by Wiley. Six of us are working on that. That will attempt to explain and make the case for this at a high level. We've talked about developing training for it, but we don't know if we're going to do that.