How the Agile paradigm saved app development

To shift or not to shift

The software industry was for a long time dominated by a paradigm of industrial views and beliefs. This was in fact a copy-paste of old manufacturing routines and theories. An essential element in this landscape of knowledge, views and practices was the Taylorist1 conviction that 'workers' can't be trusted to intelligently, autonomously and creatively perform their work. They are expected to only carry out pre-defined, executable tasks. Their work must be prepared, designed and planned by more senior staff. And then still, hierarchical supervisors must vigilantly oversee the execution of these carefully prepared tasks. Quality is assured by admitting the good and rejecting the bad batches of outputs. Monetary rewards are used to stimulate desired behavior. Unwanted behavior is punished. The old 'carrots and sticks' strategies.

The serious flaws of the old paradigm in software development are known and well documented. In particular, the Chaos reports of the Standish Group [Standish, 2011; Standish, 2013] have over and over revealed the low success rates of traditional software development. Many shortcomings and errors resulting from the application of the industrial paradigm in software development are well beyond reasonable levels of tolerance. The unfortunate response seems to have been to lower expectations. It became accepted that only 10-20% of software projects were successful. The definition of 'success' in the industrial paradigm is made up of the combination of on-time, within budget and including all scope.

Although these criteria for success can be disputed, it is the paradigm's promise. It became accepted that quality is low, and that over 50% of features of traditionally delivered software applications are never used [Standish, 2002; Standish, 2013]. Although it is not widely and consciously admitted, the industrial paradigm did put the software industry in a serious crisis. Many tried to overcome this crisis by fortifying the industrial approach. More plans were created, more phases scheduled, more designs made, more work was done upfront, hoping that the actual work would be executed more effectively. The exhaustiveness of the upfront work was increased. The core idea remained that the 'workers' needed to be directed, but with even more detailed instructions. Supervision was increased and intensified. As the success rates did not increase, the industrial paradigm assumes that the instructions are not clear and detailed enough.

Yet, little improved. Many flaws, defects and low quality remained and had to be tolerated. It took some time, but inevitably new ideas and insights started forming upon observing the significant anomalies of the industrial paradigm.

The seeds of a new world view were already sown in the 1990's. But it was in 2001 that these resulted in the formal naming of 'Agile', a turning point in the history of software development. A new paradigm was born, in the realm of the software industry but in the meantime expanding to other domains of society. It is a paradigm that thrives upon heuristics and creativity, a paradigm that thrives upon the (restored) respect for the creative nature of the work and the intelligence of the 'workers'.

The software industry has good reasons to keep moving to the new paradigm; the existing flaws are significant, widely known and the presence of software in society grows exponentially, making it a critical aspect of our modern world. However, by definition, a shift to a new paradigm takes time. And the old paradigm seems to have deep roots and a considerable half-life time. An industrial approach to software development continues to be taught and promoted as the most appropriate one.

Many say that Agile is too radical and they, therefore, propagate a gradual introduction of Agile practices within the existing, traditional frames. However, there is reason to be very skeptical about such gradual evolution, a slow progression from the old to the new paradigm, from waterfall to Agile.

The chances are high that a gradual evolution will never go beyond the surface, will not do more than just scratch that surface. New names will be installed, new terms and new practices will be imposed, but the fundamental thinking and behavior of people and organizations remain the same. Essential flaws remain untouched; especially the disrespect for people that leads to the continued treatment of creative, intelligent people as mindless 'workers', as 'resources'.

The preservation of the traditional foundations will keep existing data, metrics and standards in place, and the new paradigm will be measured against those old standards. Different paradigms by their nature however consist of fundamentally different concepts and ideas, generally mutually exclusive. No meaningful comparison between the industrial and the Agile paradigm is possible. It requires the honesty to accept the serious flaws of the old ways. It requires leadership, vision, entrepreneurship and persistence to embrace the new ways, thereby abandoning the old thinking.

There is overwhelming evidence that the old paradigm doesn't work. Much of the evidence on Agile used to be anecdotal, personal or relatively minor. The Chaos report of 2011 by the Standish Group [Standish, 2011] marked a turning point, holding clear research results for the first time that were confirmed in all later Chaos reports. Extensive research was done in comparing traditional projects with projects that used Agile methods. The report shows that an Agile approach results in a much higher yield, even against the old expectations that software must be delivered on time, on budget and with all the promised scope. The report shows that the Agile projects were three times as successful, and there were three times fewer failed Agile projects compared to traditional projects. For large projects however, the changes in success rates were less outspoken, which is likely more about starting with the wrong expectations in the large, i.e. the combination of time+budget+scope. Against the right expectations, with a focus on active customer collaboration and frequent delivery of value, the new paradigm would be performing even better, with vertical slices of value, frequently delivered, to overcome the volume problem.

Yet, Agile is a choice, not a must. It is one way to improve the software industry. Research shows it is more successful.

The distinct rules of Scrum help in getting a grip on the new paradigm. The small set of prescriptions allows immediate action and results in a more fruitful absorption of the new paradigm. Scrum is a tangible way to adopt the Agile paradigm. Using Scrum, people do develop new ways of working; through discovery, experimentation-based learning and collaboration. They enter a new state of being, a state of agility; a state of constant change, flux, evolution and adaptation. This process helps their organizations transform towards such a state of agility, freeing up time, people and energy for being innovative (again).

Nevertheless, despite its practicality, experience shows that adopting Scrum often represents a giant leap. This may be because of the uncertainty induced by letting go of old certainties, even if those old certainties have proven not to be very reliable. It may be the time that it takes to make a substantial shift. It may be the determination and hard work that is required. Over and over again it is shown that Scrum is simple, not easy.

The second edition of Scrum -- A Pocket Guide is available from Van Haren Publishing.