Test Design Automation
Analyst Coverage: Daniel Howard
Test design automation is the ability to automate the entirety of the testing process starting at an extremely high level. Products in this space are capable of generating test data, test scripts, and test cases automatically, effectively automating the design of the tests themselves.Test design automation is hot because it “shifts left”. That is, it allows automation to be introduced earlier into the development process. In agile environments, in particular, the earlier you introduce testing the better. Secondly, it is the nature of automated processes that they reduce or eliminate manual processes that are both expensive and error-prone. More technically, test design automation introduces automation into the testing process at an earlier stage. This has all of the benefits you would expect from a higher level of automation, not the least of which are potentially enormous gains to efficiency and productivity. These effects are particularly pronounced when it comes to test maintenance, and many products in the space feature capabilities to assist further in this area. Moreover, test design automation allows you to test much more quickly than the alternatives. In an Agile world where development is only getting faster, this is quickly becoming necessary for testing to keep pace.
In addition, with the increase to efficiency afforded by automatic test design, it becomes a practical proposition to procedurally and exhaustively test your applications. This can result in significantly increased test coverage – leading to fewer defects – and to more efficient coverage with fewer tests. The result is a more thoroughly tested application that provides a better experience for the end-user, which in and of itself is a driver for business value.
How does it work?
There is no universal template for test design automation tools, but the majority leverage model-based testing to allow you to create or import a graphical representation (that is, a model) of the system under test. The details and complexity of this model vary by product, but in most cases, it will be a visual flow chart, where each step in the flow chart will be equipped with the necessary apparatus of testing, such as test data, test steps, and so on. Often, the model will also be able to link back to requirements generated either through the product or by a requirements management tool. The product can then generate test cases corresponding to different routes through your model, often in combination with distinct sets of test data. These test cases can then be executed within the product or exported as test scripts to a dedicated test execution framework. If the model is later changed, most products feature change or impact analysis to automatically modify or recreate affected test cases. In addition, many of the products in the space can be configured for continuous testing, so that this entire process can be done automatically.
Manual testing is incapable of keeping pace with modern development. According to the Capgemini World Quality Report, 30% of IT spending is on testing, and this is expected to rise to 40% by 2019. Yet 44% of testing teams do not have time for core testing and, according to PAC, 89% of testing teams do not have sufficient budget to meet user expectations. These figures show that there is a desperate need for testing to be more efficient, both in time and cost. Automated testing will allow you to lower testing costs while increasing test coverage and raising the quality of your releases. For example, one major restaurant chain estimated that test design automation had saved it “2,000 man-hours of testing in the first year”, while a major telecom stated that it was able to use test design automation to achieve 100% test coverage across two applications that were previously stuck at 0% and 80% respectively.
Further research suggests that only 18% of testing teams believe they are delivering what the business wants and needs. The message is clear: testing as it currently stands is not producing a good experience for the end-user. If you want to meet customer expectations, you must invest in automated testing.
“The quantitative benefit from [test design automation] was the time it took to create 1,200 test cases. It originally took 420 hours and with [test design automation] it took 70.”
Healthcare insurance company
“[Test design automation] will help us increase the efficiency of our business analysts
by 10 percent and testers by more than 30 percent over the next three years.”
“The ability to be sure that a new web or mobile update is going to increase user satisfaction is worth millions.”
The key question for users will be how far to shift left. Fully-fledged test design automation products (which go furthest left) are relatively rare, as they represent the cutting edge of test automation. For example, you can choose between products that are essentially tabular (with an embedded object model), offerings that model the system under test and software that starts with requirements modelling. Further, there are other products in the test automation space that either enhance or overlap significantly with more advanced products, usually enabling a very significant amount of automation without automating test design per se.
Several spaces complement test design automation. Most products in this space support the import of requirements from relevant third-party tools, while test data management and service virtualisation products often integrate with test design automation. In addition, test automation itself works with test design automation. In fact, depending on your chosen product, automated test execution tools may be necessary to automate the whole of your testing process. Continuous testing is another natural way to extend test automation, and as such it may also be worth investigating continuous testing products.
What is the bottom line?
Test design automation provides a potentially massive boost to testing speed and efficiency. Properly leveraged, it can also dramatically improve the end-user experience.
Further resources to broaden your knowledge:
Testplant Eggplant AI
This paper is a discussion and technical evaluation of Eggplant AI and related products in the Testplant Digital Automation Intelligence Suite.
Conformiq Creator and Conformiq Transformer
This a technical evaluation of Conformiq’s products for Test Design Automation.
Best-of-breed test design automation
The tools evaluated here certainly automate the generation of test cases, but some of them do so in only a limited fashion while many of them do rather more ...
This is a technical evaluation of Turnkey Solutions' offering.
This is a technical evaluation of Hexawise, which is a modern pairwise testing tool.
CA Agile Requirements Designer
This paper represents a technical evaluation of CA Agile Requirements Designer.
Just what is risk based testing, anyway?
Ask almost any vendor what they mean by risk based testing and they'll give you a different answer.