It’s time to start orchestrating our testing efforts

In this guest post, Joel Montvelisky explores how increased collaboration to deliver faster with better quality is driving the need to coordinate testing work throughout the delivery cycle.

As he calls it: It’s time to start orchestrating our testing efforts.

Let’s start from the end.  

Chances are you are spending a great deal of time and effort in your manual and automated testing, but are not getting the value you seek. This is because you are not investing in coordinating and presenting your results in a way that is easy to understand by your stakeholders. Many organizations have the same issue.

It is natural based on the current reality

From talks with some colleagues in the Testing Tools Industry, I see that when an organization is working with Test Automation, they are not using one tool, but in many cases, as much as 6 different automation frameworks or tools.

This is both natural and problematic.It is natural because automation is not a single operation or task.  We have automation as part of Unit Tests – and these will depend on the technologies that are used in your code.  There are also scripts you want to run on the API level.  And tests that are there to validate the system from the actual GUI as it is run on Web Browser, Desktop Clients and Mobile Apps.

Each of these will require a different platform, and even within the same level you will end up using different tools that specialize on specific technologies and/or different aspects of the testing objectives. For example when evaluating accessibility aspects and not functional behaviour.

And this is not even taking into account legacy tests, written in your old testing tools, that you want to run until it is time to re-write it on the new tools you are currently using. This means that if you are doing a good job on your automation front, there is no way you are using only two or three tools for this.

It is a matter of evolution

In my opinion it is also because of how the testing profession has been evolving over the last couple of decades.  During the waterfall period we saw the work of the Testing and Development teams as completely isolated.  We worked on different phases of the process, physically divided into separate teams, and so it was only logical that our Automation efforts should evolve and develop independently.

While programmers were focused on Unit Tests and getting coverage metrics out of their nightly builds.  We,as testers, were busy with our marathonic functional regression automation suits, running days on end, and trying to provide some high level “grade” of the stability of the application.

We never thought about coordinating our efforts.  Afterall we operated in different teams and for stakeholders, and trying to work together would have resulted in a Tower of Babel mess of miscommunication and lack of alignment between the needs of our teams and the technologies available to supply them.

The fact that Agile and DevOps came along and told us that we need to work together as a joint team was not strong enough to help us bridge the gaps built over the years, and forged on the different technologies and platforms evolving independently and in parallel.

Times are now calling for a different approach

History is nice, and evolution is fine to understand how we got here.  But we are not a race of passive hikers that do not take active roles in forging our future.  Change comes from the decisions we make, and these decisions are usually motivated by the business needs pushing us forward. The current need dictating our path is the one for speed and quality.  

This is the need pushing to DevOps and Continuous Delivery, it is the same need pushing technology to ML, AI and 5G communications.  And it is the same need that is asking us to stop wasting time and efforts on isolated testing running in silos.

If we need to release our products as fast as possible, our processes need to be fine-tuned and coordinated. For example, by unifying the management and reporting of your unit and integration tests (those executed by your CI platforms) together with the automated mobile certification testing run separately, you can then decide if there is still a need for manual sanity test of your product before pushing it to production, and if this is the case you can even select what areas are the ones that are still risky so that you know whether them more or less in this case.

We cannot waste time in communication disconnects or duplicated work.

It is time to start approaching our isolated testing efforts – regardless if they are manual scripted or exploratory; automated unit, API, functional, etc – in the same way the conductor of a great orchestra approaches the different instruments and players in his or her ensemble. We have to look at how to work together in order to bring a stronger synergy and higher level of value to our organizations.

Coordination starts from the results we get from our tests

Just like in an orchestra, we need to start based on what each of our testing instruments adds to the overall performance of the ensemble.  In this case, what is the best information value they can provide. Once we know what each test can do best, we need to start looking at how we can use them in conjunction, to create the best overall approach.  

This can take a number of ways.

Starting from how can we use them together, for example: generating data via API tests that is then used in integration and functional GUI tests.  You can define groups of cross platform tests that should be run based on different areas that were modified, or files touched as part of the development process. You can compose sets of tests that you run on your production environments, as functional monitors covering different aspects of the system.

But in my experience, the simplest and most value-for-efforts operation that you can do is to generate dashboards and reports. These will look at the application using the inputs generated by all your testing efforts together, instead of forcing your stakeholders to navigate isolated graphs and dashboards, or ,in even worst cases, directly ask engineers about the status of the system based on their testing results. 

Joel Montvelisky is a Chief Solution Architect at PractiTest. Find him on Twitter: @joelmonte

Coordinating test work

Thank you Joel Montvelisky for this guest post (sponsored by Practitest). As you mentioned, isolated testing holds back teams from delivering true value, the time has come to collaborate and align testing activities throughout the organization. 

Collaboration is key to agile. If you want to reach results with agile, developers, testers, and other professionals should look for ways to work together intensively and effectively. Here are my tips for improving collaboration.

Effective agile teams consist of people working intensively together to deliver value. An orchestrated toolset can support collaboration, making it easier for testers and developers to pair up.

In the webinar enhancing quality and testing in agile teams that I did in collaboration with PractiTest l showed how you can improve quality and testing using gamification. Playing games with the Agile Testing Coaching Cards and Agile Quality Coaching Cards makes it possible to explore your current quality and testing practice and reach a consensus on what could be improved.


Gastblogs zijn artikelen van diverse schrijvers, waarin ze schrijven over Agile, Lean en Continue Verbetering. Interesse om een gastblog te publiceren op Neem dan contact met mij op!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.