Blog

Measurements can help you to manage and improve software development. Many high-performance organizations use measurements to have insight into the operation, and steer their projects and activities. An important issue when implementing measurements is to decide what should be measured? 

Some things are easy to measure, like time, money and defects. They seem to be ideal candidates to quantify what’s being done in the organization: We all spend time and money on the work that we do, and we find defects. But as William Bruce Cameron (not Albert Einstein) said: “Not everything that can be counted counts, and not everything that counts can be counted.” To manage effectively, you need to measure the right things: Those things that provide insight in the work performed, and that you are able to influence. Is it time, money and defects?

Too many measurements

To give an example: I’ve worked with an organization that started a measurement program. They defined lots of measurements, and with the strong commitment and support of the R&D manager they implemented most of them. Detailed data were collected on all activities performed, for each product, project, activity (design, testing), team and even per document or code module! They wanted to see how much time and money was spent on making something, and on the inspection, rework, testing, maintenance, etc. They also collected data on how many defects were found in inspections and testing, aiming to see the costs of defects. It was huge!

When they tried to use the data there were some problems, however. First, the hour registration system was so complex that many employees didn’t understand it; hence many hours were reported on the wrong activities. Sometimes this was clearly visible (and could be corrected by the manager), but there were also cases where they simply could not determine if the reported hour were correct or not. So the data was not reliable enough.

Also, there was so much data to report, that employees weren’t able to do it. They postponed filling in timesheets and reporting defects, or didn’t do it at all. People complained, and questioned the usefulness of the data that they were putting in. As a result, the data that the managers had was incomplete and also often too late available to take action.

Business need?

It turned out that many of the measurements that they had implemented had no real business value; their managers could no use them to get insight into the work, and make decisions. They measured everything that could be measured, but “not everything that can be counted, counts”.

They learned and changed: Some measurements were stopped, and for others the level of detail was reduced. They also introduced other measurements that provided relevant information (like measuring introduction and detection of defects), which posed some other challenges (“not everything that counts can be counted”), more about this in a future article.

Measure what matters!

Making sure that you measure what is relevant isn’t easy, but very important. Over the years I’ve seen companies measuring things that didn’t matter, simply because it was easy to measure. But the data didn’t help them to manage their operation, and hence the measurements were abandoned. Waste of time, money, and as a result also employees got frustrated.

You can’t manage what you can’t measure. But you must make sure that you only measure what is relevant. Otherwise, it’s like looking for your keys under the lamppost, not because you lost them there but because the light is better which makes it easier to search. Make sure you only measure what you need. And use your measurements, do a proper analysis, discuss the results with those involved, decide, and take action!

Driving quality with measurements

Measurements can help you to improve the quality of your products. My advice is to start with quantifying the value that you deliver to your customers and start measuring it. Ask your customers for feedback and use that to increase your understanding where the value is.

The book What Drives Quality helps you to prevent software problems from happening by building a shared understanding what drives software quality. The chapter “Measure Defects” describes why and how you can collect data from defects, analyze the data, and use it to deliver high-quality products with agile teams.

This blog was posted November 28, 2011 and updated May 1, 2016: Corrected attribution on “Not everything that can be counted counts …” to William Bruce Cameron (thanks to Torkil Myhre for pointing this out), and updated March 29, extended with driving quality with measurements.