Blog

My 2nd book What Drives Quality is available on Amazon, in my webshop, and in all other major bookstores.

The serie on “What Drives Quality” continues. Previously covering Requirements Quality,  Architecture and Design Quality Coding Quality, and Review and Inspection Quality, this post covers Testing. Understanding what drives quality enables you to take action before problems actually occur, thus saving time and money.

Testing covers many ways of verifying and validating the product. For example it can start with functional testing, followed by system testing, integration testing up to and including release testing. These are just examples, most companies define their own testing stages. There is a lot published on testing, I guess it’s number 1 in Quality Assurance wrt publications, conferences, and professional network. However, QA is not testing, and many of the defects that are found with testing could have been found earlier in a more economical way. But ok, testing is important to quality, so that’s why this blog posting looks at how testing can drive and contribute to better product quality.

Some factors that influence testing quality are:

1. Test Process Adherence – Checks (such as audits or assessments) to see if the baselined processes are being followed and if they are effective and efficient.
2. Test Strategy – The strategy that describes which test phases are performed in the project and which kinds of defects are expected to be found in them.
3. Test Capability – The skill and experience level of the people doing the testing.
4. Test Process Maturity – The quality of the defined and baselined processes, including all supporting material such as training and templates.
5. Test Environment – The quality of the test environment (e.g., tools and supported functionality, stability, performance).
6. Platform Quality – The functionality and stability of the product platform (e.g., operating system, middle ware, support libraries, etc).
7. Requirements Performance – Result of the previous phase (e.g., specifying the products to be developed and the supporting activities such as requirements clarifications, reviews and inspections, tracing).
8. Coding (review) Performance – Result of the previous phase (i.e., all ways to find defects directly in the source code, such as walkthroughs, reviews and formal inspections before the code is submitted to test).
9. Project Management Performance – Definition, planning, tracking, and control of quality in the development projects and the delivered products.
10. Process Management Performance – Defining and baselining the processes to be used for management and technical work. The support in operational usage of the processes for training, instructions, tools and templates and the availability of websites and experienced personnel.

Now, let’s take a look at some of these factors in more detail? I’ve pulled out a couple, to give you my view on how they can drive quality, and to share some of my experiences with test improvement.

How many defects do you expect to find?

Defining a good test strategy and making test plans is crucial to deliver quality products. I see many test plans which  descibe the test phases, activities performed, and the test environment and resources, which is a good foundation. From a quality point of view I want to see what kind of defects (and if possible how many defects) they expect to find with testing, but very few plans contain this. Still I consider this to be very important. First, this information helps you to build a business case for testing. If you expect to find defects, then there might be good reasons to do testing, unless you can find those defect in a cheaper way, like using reviews or by defect prevention. Defining a business case will help to arrange sufficient time and money to do testing. Second, if the number and severity of the defects is expected to be low, then the questions arises how much testing should be done? You might be able to save testing time, of to invest in doing different testing which will find defects before you release the product. Stating expectations, and discussing them will help you to manage product risks and quality.

Be careful however how you measure the cost of defects, In a recent book from Capers Jones on The Economics of Software Quality he explains very clearly how measuring costs of defects per testing phase can give the idea that the less defects you find (and thus the higher the quality of your product), the more expensive testing would become. I agree with him that this “cost of defect” measurement is wrong, and it also doesn;t help you to take better decisions wrt what to test, where and when. The two best approaches to measuring quality and testing that I have used are Cost of Quality (CoQ) and Total Cost of Ownerschip (TCO). Remind me to post my experiences with these measurements, they really work!

Discussing the expectations from testing with developers in an early stage can save a project time and money. I have used a Project Defect Model to plan and track quality. This model supported discussions between developers and testers on the expected quality, and helped them focus the test activities to find defects as early as possible. The benefits are clear: Less disturbances in project, able to deliver on time, and lower maintenance costs. And not to forget, happier customers, and it saves your employees some frustration.

Can you learn from defects and prevent them in the future?

Root Cause Analysis helps to analyse defects that have slipped though testing, and to define and implement improvements to prevent similar defects slipping through in the future. Root cause analysis can also provide insight in the preceding activities, i.e. Requirements Quality,  Architecture and Design Quality and Coding Quality, thus providing opportunities to improve acros the full product development livecycle in a lean way.

The CMMI positions Root Cause Analysis at level 5, in the process area Causal Analysis and Resolution (CAR). I agree that, given the amount of problems that could be investigated in most companies, soe kind of policy is needed to decide which ones would help you to come up with effective actions for prevention of similar defects. I use 3 criteria for whch defects should be investigated:

• Major defects from test or customers
• Significant disturbances (off-track)
• Re-occuring problems

If the problem fits within at least one of the three criteria, then you should do a Root Cause Analyis. If not, save your time and money for other problems. Actually, CMMI V.13 has extended the possibilities to do Root Cause Analysis at lower maturity levels, recognizing the business benefits that they can bring.

What about testing processes?

Mentioning process management performance as factor in testing quality may look surprising. My experience is that testing processes (the combination of all testing stages, test techniques, test environments, etc) can become very complex and are difficult to manage. Taking a process view to testing, by analyzing test data and process performance can help you to continuously improve your testing activities, resulting to higher quality at lower costs.

There has been and still is much debate on the need for processes, with more and more organizations implementing agile. But remember my definition of a process, which is “the way we work around here”. Agile teams also have processes, just look at the Definition of Done. Agile actually has process management build in, by using retrospectives to evaluate and learn, and the planning game to tailor the process towards the needs of the current sprint. Process management isn’t gone with agile development, it’s actually cheaper, and often more effective!

Conclusion

Given that testing often consumes up a large part of the project costs and time, it is important to assure that it contributes towards delivering higher quality. The factors that drive how testing influence quality can help you to get maximum business value out of your testing activities.