Defect tracking: Lean principles for getting the right data at the right time

Is the data you collect adding value?

Few would argue with the well known software engineering principle that the more defects we find early, the higher the quality of software we can develop. But does this principle hold true in all organizations under all project conditions?

A few years ago I was working with a client who wanted to increase their software team's agility, but they wanted to make sure that any changes they made did not jeopardize their CMMI Level 3 compliance. To achieve this goal, we started by building flow diagrams of the processes the software team was currently using. 

We intentionally did not look at their documented CMMI Level 3 processes, but rather built these flow diagrams based on what people told us they actually did to get their software done. We annotated each flow diagram with any process assets (e.g. procedures, templates, guidelines) that people told us they used. Any process asset that existed in their CMMI Level 3 process repository that did not end up on a flow diagram became a candidate for elimination or streamlining. 

But before eliminating or streamlining an asset, we asked a few questions: If no one was using something in the repository, why was it there? Were we wasting time teaching people about certain process assets or steps in a process that added no value? Did we believe if people did follow a certain process step that they weren't following that it would help them get their job done more effectively? 

Let me give you a specific example. This organization had a defect tracking system that required a great deal of data to be collected about each defect entered into the system (e.g. phase found, phase injected, type of defect, reason for defect, what phase the defect should have been found in). The process also required periodic analysis of the collected data. When we built the flow diagram for the defect tracking system that showed what people were actually doing, we found out that everyone was entering all the data because the defect tracking tool required it, but no one was going back and analyzing or using that data. 

On further investigation, we determined that the reason all the data about each defect was being required was because someone wrongly thought that the CMMI required it. We also discovered that because the process was so onerous it had actually discouraged people from entering real defects found. After we streamlined the process eliminating much of the data required and even eliminating the need to report certain types of defects we found the people started using the system more consistently. This activity led to improved software quality, and ended up strengthening the organization's CMMI compliance because the people were following their defined process more consistently. 

Three questions to help you decide when to track defects

This story leads to a few questions. How did we ensure we were not jeopardizing the organization's CMMI compliance when we streamlined the defect reporting system? Why would reducing the information that needed to be entered concerning a defect and reducing the types of defects reported improve quality? Doesn't this fly in the face of the well known software engineering principle that the more defects you find early, the higher the quality of your software?

To understand the answers to these questions, let me first share three questions each organization should ask about defects that will help you decide when to start tracking your defects. 

Question 1: What are the two or three most common types of defects that are causing the greatest trouble to your software quality? 

Question 2: When are these defects most often injected into the product?

For each piece of data you are considering to collect for each defect ask:

Question 3: Who is going to use this data, and how will what they are going to do with the data help improve the quality of the software?

The Pareto principle tells us that 80% of the defects in a product are caused by 20% of the problems. I have found through years of consulting that most organizations know the two or three most common types of defects that are causing their biggest problems, and they have a very good idea when these defects are being injected into the product. I have also found that you don't need to collect a great deal of information about a defect to be able to take effective action to counter future similar occurrences and improve the quality of your software. 

Reduce amount of data and focus on the most valuable data

How did we ensure we weren't jeopardizing CMMI compliance? It is a common myth many hold that the CMMI requires certain data to be collected about defects. The CMMI does expect organizations to conduct peer reviews, and it expects organizations to analyze data from peer reviews, such as defect data. But the CMMI does not tell you what specific data about your defects you need to collect, or the types of defects you need to collect. Each organization should determine the answers to these questions based on what makes sense for their organization. Once you know what defects are most important to look for and when, they most often get injected into your product you can easily determine the optimal time in the life cycle to focus on defect tracking. 

For example, if an organization tends to have a lot of trouble with poorly defined or ambiguous requirements that get injected early, then you need to track and take action early to eliminate those defects. But if your problems are primarily in detailed design or coding and requirements tend to be stable, then the payback may not be there for tracking requirements defects early. 

Reducing the amount of data you collect and focusing on the most valuable data can encourage your team to use your defect tracking system more consistently. Contrary to what many of us have been taught, it is less important how many defects you find early and more important that you find the most critical and potentially costly defects right when they are about to be injected into your software. 

By first asking the three questions about defects, your organization can put in place a Lean defect tracking system that will help focus on the most costly problems in order to get the optimum payback for your defect tracking investment. 

For more examples of effective Lean and Agile techniques that can help your organization improve the quality of its software while also maintaining your CMMI compliance refer to the authors book, Integrating CMMI and Agile Development: Case Studies and Proven Techniques for Faster Performance Improvement. 

Also see Quality metrics: A guide to measuring software quality for further resources on quality metrics.