R&D Organizations are more and more relying on measurements, but many struggle to implement them. There are the usual technical problems associated with collecting and storing data, and creating usable reports. However, the biggest challenges are often related to using the data to actually make decisions and steer the activities of the organization.
Having data can help you to manage your organization, but using measurements effectively isn’t easy. A tongue-in-cheek look at statistical analysis from a TED Talk by Sebastian Wernicke, where he comes up with a metric for creating “the optimum TEDTalk” based on user ratings.
Statistical Process Management is a way to manage an R&D operation based upon measurements and analysis. Methods and models such as the CMMI (with the Measurement Roadmap) and Lean Six Sigma describe how data can be collected and used to steer the operation. But often, managers have great difficulty understanding the reports and all the figures that are reported to them. Miscommunication, incomplete analysis, and corrective actions that seem to come from nowhere create resistance to the whole idea of measurements. They don’t trust the data, and stop using is. The saying goes that you can lie with data, but the opposite is probably even worse: Correct data that is not communicated and used because it is not understood or trusted. That’s a missed opportunity!
A proven solution is to give the raw data from the measurements to the people who did the work, and have them perform the analysis. Why you may ask? Because they know the story behind the data. For instance, lead-time and budget precision of a project should be discussed with the project manager, while the rate at which defects are detected could be analyzed with the test team. It is much more effective to have the people that will solve the problems (and have created them) to do their own analysis and decide their course of action. Of course, they will report the data, their analysis, the causes they determined, and the actions they took to management. Management is still able to decide if the analysis is sufficient and if the actions that are taken are appropriate and sufficient.
Many organizations that I have seen measure too much. They measure things that have insufficient relevance for the business or the way people do their work (AKA “the process”). And are not doing other measurements, because they are so hard to do. This is similar to looking for lost keys where the light is better, and not where you lost them. To manage effectively, you need to measure the right things. Those things that provide insight in the work performed, and that you are able to influence, and use effectively to take decisions.
Why not report measurement data directly to senior management? There are several potential problems. What if the data is incorrect and the wrong actions are taken as a result? For example, a project is reported as having a lead-time precision of 83%, which is below the goal of 90%. Senior management orders corrective actions based on this data. Later it turns out that some input data was incorrectly reported, the lead-time precision was actually 91%. The incorrect data was found only when the data was analyzed, after the decision was made and after the corrective actions have started. Besides being a waste of money, corrective actions based on incorrect data can seriously hamper the motivation of your professionals, and reduce the trust that they have in the data.
Even if the data is correct, the story behind the data is still unknown. If management draws its own conclusions, they may turn out to be wrong. Managers do not have in-depth knowledge of the operation, and they do not have the time to go into detail. People doing the work have this knowledge, becoming agile and lean by involving them is much more effective! If management orders additional analysis, corrective actions are delayed. Even if management is able to determine the right causes, and the corrective actions are on target, those actions often meet resistance from team members if they are insufficiently understood or if the relationship between the actions and the causes is unclear. In the end, the measurements are not effective, as they do not help the organization to steer the work related to the goals of the organization.
Improve the usage of your data to actually make decisions, by having analysis done by those involved in the actual work. Have them report the root causes underlying the data, and suggest corrective actions. This shortens the measurement – action loop (AKA Plan Do Check Act or PDCA), and helps you to improve the performance of your organization!
(This blog was posted mar 31, 2011, and updated dec 9, 2012: Extended with background on applying measurements more effectively).
