Why human instinct causes mistakes in project management

Software projects start with rational decisions, but many devolve into irrational decisions and unnecessary risk-taking. If we understand how these mistakes in project management happen and the potential ramifications, can we make better software?

Andrew Brown, principal technical consultant at SQS, a managed services provider and consultancy based in Cologne, Germany, discussed this descent into mistakes in project management at the EuroSTAR Conference in November 2018. His presentation, titled "How do I think? -- Je ne sais pas!" included research that explains why organizations make mistakes in project management, and he provided suggestions to help recognize irrationality before it strikes.

Brown developed an independent line of research into understanding why humans make mistakes in project management that lead to software defects. After his presentation, Brown discussed over email how rational and irrational thinking affects software projects.

Will you explain what you mean by rational and irrational behaviors in projects?

Andrew Brown: Rational choice theory is a framework for understanding how individuals behave. It assumes that an individual has preferences amongst the available, alternative choices. A rational individual is assumed to make use of available information, probabilities of outcomes and costs and benefits to determine his or her preference. In addition, a rational individual is expected to behave consistently in selecting the best course of action to maximize value or utility.

The expected utility hypothesis predicts an individual's preferred choice where outcomes are uncertain. This theory takes into account that individuals are generally risk-averse.

On projects that are overdue, you will often see decisions taken that are irrational, in the economic sense of the word. For example, many of us have been on overdue projects where a large amount of potentially valuable functionality, in the form of code already written, is simply abandoned during an attempt to get the project back onto track and meet -- or minimize the overshoot of -- an important deadline. When this occurs, the project -- or, rather, the project decision-makers -- are making decisions that are not maximizing utility.

In a similar manner, many of us have been on overdue projects where risky choices, rather than safe choices, have been selected because, although the risky choices are unpleasant, they do at least offer the possibility of completing the project on time. When this occurs, the risk appetite of the project has switched from risk-averse into risk-seeking.

You cited the work of Daniel Kahneman [professor of psychology and public affairs emeritus at Princeton University] in your presentation. What role does his research play in talking about how rational a team is?

Brown: Pretty central. Daniel Kahneman and [cognitive and mathematical psychologist] Amos Tversky developed prospect theory, which is central to understanding the circumstances under which we will switch our risk appetite from risk-averse into risk-seeking behavior. In addition, many aspects of their research are relevant to our behavior.

For example, the anchoring effect plays a major part in causing the initial project estimate to be an underestimation, which, in turn, leads to the conditions that cause the switch in risk appetite. In addition, their work around heuristics, such as availability and representativeness, and their work around overconfidence and choices has shed a great deal of light onto many of the problems that we currently struggle with in software development.

You also talk extensively about Kahneman's prospect theory. Can you give us the short version?

Brown: Prospect theory describes how people choose between alternative outcomes that are not definite but are probabilistic. It predicts that people make decisions based on the potential value of the changes, rather than the final outcome. Also, it states that people evaluate the changes using heuristics.

Prospect theory breaks the decision process down into two stages: an editing phase and an evaluation phase. In the editing phase, the outcomes of the decision are ordered and simplified. In the subsequent evaluation phase, the value of each alternative choice is evaluated.

The theory produced a value function, which represented the value that an individual would assign to a given loss or gain.

• There is a discontinuity in the curve at the origin, with the slope for losses being steeper than the slope for gains. This difference in slope indicates that individuals feel the pain of loss far greater than they feel the pleasure of a gain.
• The curve in the gains part of the graph is concave and steepest at the origin. This indicates that individuals are risk-averse in situations of gain.
• The curve in the losses part of the grass is convex and steepest at the origin. This indicates that individuals are risk-seeking in situations of loss.

You also mention evolutionary psychology and how risk-seeking behavior occurs. Why do we take certain high risks in project decisions? When do we do so?

A good indication that a project is at risk of switching into irrational decision-making is when the project status switches from green to red.

Brown: To understand why we behave in the way that we do, we need to understand the circumstances under which that particular behavior would have given our ancestors an evolutionary advantage.

If we look at the foraging behavior of animals, we can devise experiments to observe that many species will switch their risk appetite in response to food availability. For example, during periods of ample food supply, animals can be observed to adopt a risk-averse strategy. During such periods, adopting a risk-averse strategy maximizes the animal's chances of survival.

Conversely, under conditions of food scarcity, animals adopt a risk-seeking strategy. Although [this] strategy does carry attendant risk, it also carries a possibility of survival. A risk-averse strategy during periods of food scarcity will result in death.

This risk-switching strategy under conditions of gains and losses is the most credible explanation for why projects have an increased tendency to take high-risk decisions when they become overdue or behind schedule.

What can project teams do to recognize that their behavior and decisions are becoming irrational? How can they halt the process?

Brown: A good indication that a project is at risk of switching into irrational decision-making is when the project status switches from green to red. At this point, project members will perceive that the situation has switched from one of gains to losses, which, in turn, will trigger a tendency to switch from risk-averse and utility maximization behavior into risk-seeking behavior that abandons utility.