Adaptation in project management through agile

Adapting over Conforming

A traditional project manager focuses on following the plan with minimal changes, whereas an agile leader focuses on adapting successfully to inevitable changes. 

Traditional managers view the plan as the goal, whereas agile leaders view customer value as the goal. If you doubt the former, just look at the defini­tion of "success" from the Standish Group, who has published success (and failure) rates of software projects over a long period of time. Success, per the Standish Group is "the project is completed on time and on budget, with all the features and functions originally specified."1 This is not a value-based definition but a constraint-based one. Using this definition, then, managers focus on following the plan with minimal changes. Colleague Rob Austin would classify this as a dysfunctional measurement (Austin 1996)—one that leads to the opposite behavior of what was intended.

When customer value and quality are the goals, then a plan becomes a means to achieve those goals, not the goal itself. The constraints embedded in those plans are still important; they still guide the project; we still want to understand variations from the plans, but—and this is a big but—plans are not sacrosanct; they are meant to be flexible; they are meant to be guides, not straightjackets.

1 Standish Group. Chaos Reports (http://www.standishgroup.com/chaos_resources/chronicles.php). 

Figure 4-1 
Adaptive Principle Statements
Both traditional and agile leaders plan, and both spend a fair amount of time planning. But they view plans in radically different ways. They both believe in plans as baselines, but traditional managers are constantly trying to "correct" actual results to that baseline. In the PMBOK2, for example, the relevant activity is described as "corrective action" to guide the team back to the plan. In agile project management, we use "adaptive action" to describe which course of action to take (and one of those actions may be to correct to the plan).
The agile principles documents—the Agile Manifesto and the Declara­tion of Interdependence—contain five principle statements about adapta­tion, as shown in Figure 4-1.

These principles could be summarized as follows:

• We expect change (uncertainty) and respond accordingly rather than follow outdated plans.
• We adapt our processes and practices as necessary.

The ability to respond to change drives competitive advantage. Think of the possibilities (not the problems) of being able to release a new version of a product weekly. Think of the competitive advantage of being able to pack­

2 The Project Management Institute's Project Management Body of Knowledge, known as the PMBOK.

age features so customers feel they have software specifically customized for them (and the cost to maintain the software remains low).

Teams must adapt, but they can't lose track of the ultimate goals of the project. Teams should constantly evaluate progress, whether adapting or anticipating, by asking these four questions:

• Is value, in the form of a releasable product, being delivered?
• Is the quality goal of building a reliable, adaptable product being met?
• Is the project progressing satisfactorily within acceptable con­straints?
• Is the team adapting effectively to changes imposed by management, customers, or technology?

The dictionary defines change as: "To cause to be different, to give a com­pletely different form or appearance to." It defines adapt as: "To make suit­able to or fit for a specific use or situation." Changing and adapting are not the same and the difference between them is important. There is no goal inherent in change—as the quip says, "stuff happens." Adaptation, on the other hand, is directed towards a goal (suitability). Change is mindless; adaptation is mindful.

Adaptation can be considered a mindful response to change. 

The Science of Adaptation

Former Visa International CEO Dee Hock (1999) coined the word "chaordic" to describe both the world around us and his approach to man­aging a far-flung enterprise—balanced on the precipice between chaos and order. Our sense of the world dictates management style. If the world is per­ceived as static, then production-style management practices will dominate. If the world is perceived as dynamic, however, then exploration-style man­agement practices will come to the fore. Of course, it's not that simple— there is always a blend of static and dynamic, which means that managers must always perform a delicate balancing act.

In the last two decades a vanguard of scientists and managers have artic­ulated a profound shift in their view about how organisms and organizations evolve, respond to change, and manage their growth. Scientists' findings about the tipping points of chemical reactions and the "swarm" behavior of ants have given organizational researchers insights into what makes success­ful companies and successful managers. Practitioners have studied how innovative groups work most effectively.

As quantum physics changed our notions of predictability and Darwin changed our perspective on evolution, complex adaptive systems (CAS) the­ory has reshaped scientific and management thinking. In an era of rapid change, we need better ways of making sense of the world around us. Just as biologists now study ecosystems as well as species, executives and managers need to understand the global economic and political ecosystems in which their companies compete.

"A complex adaptive system, be it biologic or economic below, is an 
ensemble of independent agents 

• Who interact to create an ecosystem
• Whose interaction is defined by the exchange of information
• Whose individual actions are based on some system of internal rules
• Who self-organize in nonlinear ways to produce emergent results
• Who exhibit characteristics of both order and chaos
• Who evolve over time" (Highsmith 2000).

For an agile project, the ensemble includes core team members, customers, suppliers, executives, and other participants who interact with each other in various ways. It is these interactions, and the tacit and explicit information exchanges that occur within them, that project management practices need to facilitate.

The individual agent's actions are driven by a set of internal rules—the core ideology and values of APM, for example. Both scientific and manage­ment researchers have shown that a simple set of rules can generate complex behaviors and outcomes, whether in ant colonies or project teams. Complex rules, on the other hand, often become bureaucratic. How these rules are formulated has a significant impact on how the complex system operates.

Newtonian approaches predict results. CAS approaches create emer­gent results. "Emergence is a property of complex adaptive systems that cre­ates some greater property of the whole (system behavior) from the interaction of the parts (self-organizing agent behavior). Emergent results cannot be predicted in the normal sense of cause and effect relationships, but they can be anticipated by creating patterns that have previously pro­duced similar results" (Highsmith 2000). Creativity and innovation are the emergent results of well functioning agile teams.

An adaptive development process has a different character from an optimizing one. Optimizing reflects a basic prescriptive Plan-Design-Build lifecycle. Adapting reflects an organic, evolutionary Envision-Explore-Adapt lifecycle. An adaptive approach begins not with a single solution, but with multiple potential solutions (experiments). It explores and selects the best by applying a series of fitness tests (actual product features or simula­tions subjected to acceptance tests) and then adapting to feedback. When uncertainty is low, adaptive approaches run the risk of higher costs. When uncertainty is high, optimizing approaches run the risk of settling too early on a particular solution and stifling innovation. The salient point is that these two fundamental approaches to development are very different, and they require different processes, different management approaches, and dif­ferent measurements of success.

Newtonian versus quantum, predictability versus flexibility, optimiza­tion versus adaptation, efficiency versus innovation—all these dichotomies reflect a fundamentally different way of making sense about the world and how to manage effectively within it. Because of high iteration costs, the tra­ditional perspective was predictive and change averse, and deterministic processes arose to support this traditional viewpoint. But our viewpoint needs to change. Executives, project leaders, and development teams must embrace a different view of the new product development world, one that not only recognizes change in the business world, but also understands the power of driving down iteration costs to enable experimentation and emer­gent processes. Understanding these differences and how they affect prod­uct development is key to understanding APM.

Exploring

Agility is the ability to both create and respond to change in order to profit in a turbulent business environment (from Chapter 1). The ability to respond to change is good. The ability to create change for competitors is even better. When you create change you are on the competitive offensive. When you respond to competitors' changes you are on the defensive. When you can respond to change at any point in the development lifecycle, even late, then you have a distinct advantage.

Adaptation needs to exceed the rate of market changes.

But change is hard. Although agile values tell us that responding to change is more important than following a plan, and that embracing rather than resisting change leads to better products, working in a high-change environ­ment can be nerve-wracking for team members. Exploration is difficult; it raises anxiety, trepidation, and sometimes even a little fear. Agile project leaders need to encourage and inspire team members to work through the difficulties of a high-change environment. Remaining calm themselves, encouraging experimentation, learning through both successes and mis­takes, and helping team members understand the vision are all part of this encouragement. Good leaders create a safe environment in which people can voice outlandish ideas, some of which turn out not to be so outlandish after all. External encouragement and inspiration help teams build internal motivation.

Great explorations flow from inspirational leaders. Cook, Magellan, Shackleton, and Columbus were inspirational leaders with vision. They per­severed in the face of monumental obstacles, not the least of which was fear of the unknown. Magellan, after years of dealing with the entrenched Span­ish bureaucracy trying to scuttle his plans, launched his five-ship fleet on October 3, 1519. On September 6, 1522, the Victoria, last of the ships, sailed into port without Magellan, who had died in the Philippines after completing the most treacherous part of the journey. The expedition estab­lished a route around Cape Horn and sailed across the vast Pacific Ocean for the first time (Joyner 1992).

Great explorers articulate goals that inspire people—goals that get peo­ple excited such that they inspire themselves. These goals or visions serve as a unifying focal point of effort, galvanizing people and creating an esprit de corps among the team. Inspirational goals need to be energizing, com­pelling, clear, and feasible, but just barely. Inspirational goals tap into a team's passion.

Encouraging leaders also know the difference between good goals and bad ones. We all know of egocentric managers who point to some mountain and say, "Let's get up there, team," when everyone else is thinking, "Who is he kidding? There's not a snowball's chance in the hot place that we can carry that off." "Bad BHAGs [Big Hairy Audacious Goals], it turns out, are set with bravado; good BHAGs are set with understanding," says Jim Collins (2001). Inspirational leaders know that setting a vision for the product is a team effort, one based on analysis, understanding, and realistic risk assess­ment, combined with a sprinkle of adventure.

Innovative product development teams are led, not managed. They allow their leaders to be inspirational. They internalize the leader's encour­agement. Great new products, outstanding enhancements to existing prod­ucts, and creative new business initiatives are driven by passion and inspiration. Project managers who focus on network diagrams, cost budgets, and resource histograms are dooming their teams to mediocrity.3, 4

Leaders help articulate the goals; teams internalize them and motivate themselves. This internal motivation enables exploration. We don't arrive at something new, better, and different without trial and error, launching off in multiple new directions to find the one that seems promising. Magellan and his ships spent 38 days covering the 334 miles of the straits that bear his name. In the vast expanse of islands and peninsulas, they explored many dead ends before finding the correct passages (Kelley 2001).

Magellan's ship Victoria sailed nearly 1,000 miles, back and forth—up estuaries that dead-ended and back out—time and time again. Magellan (his crew, actually) was the first to circumnavigate the globe. But Magellan would probably have driven a production-style project manager or execu­tive a little crazy, because he surely didn't follow a plan. But then, any detailed plan would have been foolish—no one even knew whether ships could get around Cape Horn; none had found the way when Magellan launched. No one knew how large the Pacific Ocean was, and even the best guestimates turned out to be thousands of miles short. His vision never changed, but his "execution" changed every day based on new information.  Teams need a shared purpose and goal, but they also need encourage­ment to adapt—to experiment, explore, make mistakes, regroup, and forge ahead again.

3 This sentence should not be interpreted as saying these things are unimportant, because properly used, each can be useful to the project leader. It is when they become the focal point that trouble ensues. 4 Ken Delcol observes,

Responding to Change

We expect change (uncertainty) and respond accordingly rather than follow outdated plans. 

This statement reflects the agile viewpoint characterized further by

• Envision-Explore versus Plan-Do
• Adapting versus anticipating

In Artful Making, Harvard Business School professor and colleague Rob Austin and his coauthor Lee Devin (2003) discuss a $125 million IT project disaster in which the company refused to improvise and change from the detailed plan set down prior to the project's start. "'Plan the work and work the plan' was their implicit mantra," they write. "And it led them directly to a costly and destructive course of action.…We'd all like to believe that this kind of problem is rare in business. It's not."

Every project has knowns and unknowns, certainties and uncertainties, and therefore every project has to balance planning and adapting. Balancing is require

d because projects also run the gamut from production-style ones in which uncertainty is low, to exploration-style ones in which uncertainty is high. Exploration-style projects, similar to development of the Sketchbook Pro© product introduced in Chapter 1, require a process that emphasizes envisioning and then exploring into that vision rather than detailed planning and relatively strict execution of tasks. It's not that one is right and the other wrong, but that each style is more or less applicable to a particular project type.

Another factor that impacts project management style is the cost of an iteration; that is, the cost of experimenting. Even if the need for innovation is great, high iteration costs may dictate a process with greater anticipatory work. Low-cost iterations, like those mentioned earlier, enable an adaptive style of development in which plans, architectures, and designs evolve con­currently with the actual product.

Product, Process, People

Faced with major redirection in release 2.0 of their product, the Sketchbook Pro© team introduced in Chapter 1 delivered the revised product in 42 days. As I quip in workshops, "I know teams who would have complained for 42 days with comments such as: "They don't know what they want. They are always changing their minds." Adaptability has three components—prod­uct, process, and people. You need to have a gung-ho agile team with the right attitude about change. You need processes and practices that allow the team to adapt to circumstances. And you need high quality code with auto­mated tests. You can have pristine code and a non-agile team and change will be difficult. All three are required to have an agile, adaptable environ­ment.

The barrier to agility in many software organizations is their failure to deal with the technical debt in legacy code. The failure is understandable because the solution can be costly and time consuming. However, failure to address this significant barrier keeps many organizations from realizing their agile potential. It took years for legacy code to degenerate; it will take signif­icant time to revitalize the code. It requires a systematic investment in refac­toring and automated testing—over several release cycles—to begin to solve the problems of years of neglect.

Barriers or Opportunities.

One of the constant excuses, complaints, or rationalizations about some agile practices are "They would take too much time," or "They would cost too much." This has been said about short iterations, frequent database updates, continuous integration, automated testing, and a host of other agile practices. All too often companies succumb to what colleague Israel Gat calls the "new toy" syndrome—placing all their emphasis on new develop­ment and ignoring legacy code. Things like messy old code then become excuses and barriers to change. Some activities certainly are cost-prohibi­tive, but many of these are artificial barriers that people voice. Experienced agilists turn these barriers into opportunities. They ask, "What would be the benefit if we could do this?"

Working with a large company—and a very large program team (multi­ple projects, one integrated product suite) of something over 500 people— several years ago we wanted them to do a complete multi-project code integration at the end of every couple of iterations. The reply was, "We can't do that, it would take multiple people and several weeks of time out of development." This was from a group who had experienced severe prob­lems in prior releases when they integrated products very late in the release cycle. Our response was, "What would the benefit be if you could do the integration quickly and at low cost?" and, "You don't have a choice; if you want to be agile you must integrate across the entire product suite early and often." Grumbling, they managed the first integration with significant effort, but with far less time than they anticipated. By the time 3–4 integra­tions had occurred, they had figured out how to do them in a few days with limited personnel. The benefits from frequent integration were significant, eliminating many problems that previously would have lingered, unfound, until close to release date.

Most, but not all, of the time perceived barriers to change (it costs too much) really point out inefficiencies—opportunities to streamline the process and enhance the organization's ability to adapt. Agile development demands short-cycle iterations. Doing short-cycle iterations demands find­ing ways to do repetitive things quickly and inexpensively. Doing things quickly and inexpensively enables teams to respond to changes in ways they never anticipated previously. Doing things quickly and inexpensively fosters innovation because it encourages teams to experiment. These innovations ripple out into other parts of the organization. Lowering the cost of change enables companies to rethink their business models.

Reliable, Not Repeatable

Note that the word "repeatable" isn't in the agile lexicon. Implementing repeatable processes has been the goal of many companies, but in product development, repeatability turns out to be the wrong goal; in fact, it turns out to be an extremely counterproductive goal. Repeatable means doing the same thing in the same way to product the same results. Reliable means meeting targets regardless of the impediments thrown in your way—it means constantly adapting to meet a goal.

Repeatable processes reduce variability through measurement and con­stant process correction. The term originated in manufacturing, where results were well defined and repeatability meant that if a process had con­sistent inputs, then defined outputs would be produced. Repeatable means that the conversion of inputs to outputs can be replicated with little varia­tion. It implies that no new information can be generated during the process because we have to know all the information in advance to predict the out­put results accurately. Repeatable processes are not effective for product development projects because precise results are rarely predictable, inputs vary considerably from project to project, and the input-to-output conver­sions themselves are highly variable.

Reliable processes focus on outputs, not inputs. Using a reliable process, team members figure out ways to consistently achieve a given goal even though the inputs vary dramatically. Because of the input variations, the team may not use the same processes or practices from one project, or even one iteration, to the next. Reliability is results driven. Repeatability is input driven. The irony is that if every project process was somehow made repeatable, the project would be extremely unstable because of input and transformation variations. Even those organizations that purport to have repeatable processes are often successful not because of those processes, but because of the adaptability of the people who are using those processes.

At best, a repeatable process can deliver only what was specified in the beginning. A reliable, emergent process can actually deliver a better result than anyone ever conceived in the beginning. An emergent process can produce what you wish you had thought about at the start if only you had been smart and prescient enough.

Herein lies a definitional issue with project scope. With production-style projects, those amenable to repeatable processes, scope is considered to be the defined requirements. But in product development, requirements evolve and change over the life of the project, so "scope" can never be precisely defined in the beginning. Therefore, the correct scope to consider for agile projects isn't defined requirements but the articulated product vision—a releasable product. Product managers may be worried about specific requirements, but executives are concerned about the product as a whole— Does it meet the vision of the customer? When management asks the ever-popular question, "Did the project meet its scope, schedule, and cost targets?" The answer should be evaluated according to the vision, value, and totality of the capabilities delivered. That is, the evaluation of success can be encapsulated in the question "Do we have a releasable product?" not on whether the set of specific features defined at the start of the project was produced.

Agile Project Management is both reliable and predictable: It can deliver products that meet the customer's evolving needs within the bound­ary constraints better than any other process for a given level of uncertainty. Why does this happen? Not because some project manager specified detailed tasks and micromanaged them, but because an agile leader estab­lished an environment in which people wanted to excel and meet their goals.Although APM is reliable, it is not infallible, and it cannot eliminate the vagaries of uncertainty, nor the surprises encountered while exploring. APM can shift the odds toward success. If executives expect projects to deliver on the product vision, within the constraints of specified time and cost, every time, without fail, then they should be running an assembly line, not devel­oping products.

Reflection and Retrospective

Adapting requires both a certain mindset and a set of skills. If we are to be adaptable, then we must be willing to seriously and critically evaluate our performance as individual contributors and as teams. Effective teams cover four key subject areas in their retrospectives: product from both the cus­tomer's perspective and a technical quality perspective; process, as in how well the processes and practices being used by the team are working; team, as in how well the group is working as a high-performance team; and project, as in how the project is progressing according to plan. Feedback in each of these areas—at the end of each iteration and at the end of the project—leads to adaptations that improve performance. The "how to" of retrospectives and reflection is covered in Chapter 10, "The Adapt and Close Phases." Principles to Practices
We adapt our processes and practices as necessary. 
 

Ultimately, what people do, how they behave, is what creates great products. Principles and practices are guides; they help identify and reinforce certain behaviors. Although principles guide agile teams, specific practices are necessary to actually accomplish work. A process structure and specific practices form a minimal, flexible framework for self-organizing teams. In an agile project there must be both anticipatory and adaptive processes and practices. Release planning uses known information to "anticipate" the future. Refac­toring uses information found later in the project to "adapt" code. Ron Jef­fries once said, "I have more confidence in my ability to adapt than in my ability to plan." Agilists do anticipate, but they always try to understand the limits of anticipation and try to err on the side of less of it.

Final Thoughts

Developing great products requires exploration, not tracking against a plan. Exploring and adapting are two behavioral traits required to innovate— having the courage to explore into the unknown and having the humility to recognize mistakes and adapt to the situation. Magellan had a vision, a goal, and some general ideas about sailing from Spain down the coast of South America, avoiding Portuguese ships if at all possible, exploring dead end after dead end to finding a way around Cape Horn, then tracking across the Pacific to once-again known territory in the Southeast Asia archipelagoes. Great new products come from similarly audacious goals and rough plans that often include large gaps in which "miracles happen," much like the mir­acle of finding the Straits of Magellan.