Analyst Coverage: Daniel Howard
Keysight Technologies manufactures electronics test and measurement equipment. The company was formally founded in 2014, having been spun off from Agilent Technologies, which was itself spun off from Hewlett-Packard at the end of the last century. It is based in Santa Rosa, CA, operates at a global scale, and boasts more than 30,000 customers as well as almost 14,000 employees.
In 2020, Keysight acquired Eggplant, a test automation vendor founded in 2008 to help put users at the centre of software testing. Its flagship product, Eggplant AI, was launched in 2017, and now forms the core of Keysight’s test automation offering as part of the Eggplant DAI (Digital Automation Intelligence) platform.
Last Updated: 26th April 2018
For Eggplant, testing and testing software should, above all else, increase customer satisfaction and improve the experience of the end user. In their words, testing should result in a product that “delights” its users. Unfortunately, this is not often the case. According to Eggplant’s research, the vast majority of companies which employ testing are regularly meeting their testing objectives, yet only a small fraction considers themselves to be meeting customer expectations. Not only that, but on average, almost a third of IT budgets are spent on testing and QA, yet close to 90 percent of QA teams cannot keep up with their workload.
Eggplant addresses both of these issues with its Digital Automation Intelligence Suite, and particularly with Eggplant AI, a product within that suite which uses artificial intelligence, machine learning, and analytics to find defects and maximise coverage of user journeys. Eggplant AI improves efficiency and productivity by providing automation in every area of testing, not just test execution. In addition, to ensure a good end-user experience, everything within Eggplant AI is done from the perspective of an end user. For instance, testing is driven through the user interface (UI), and emphatically not the backend or the application code. Moreover, testing is not restricted to user journeys that appear sensible, taking the more realistic approach that any and every path through the application is a valid one.
Eggplant has a long list of business partners, including resellers such as CGI Group, Pactera, and IT Ecology, that resell the product and have access to the support, services, and technical training needed to help deliver it; service providers, such as NCC Group, Sogeti, Cigniti, and Infosys; and systems integrators, such as IBM, Tata, and Capgemini. Both of the latter groups work with customers to deliver comprehensive test automation solutions.
Between their offices and their partners, Eggplant has well over 400 active customers in upwards of 30 countries, and have served more than 600 customers in total since their inception. These customers come from all manner of sectors, including automotive, aerospace, defence, financial services, healthcare, media, and retail.
Eggplant AI uses model-based testing to automate test case generation. This means that you create a graphical model of the system under test that is then leveraged by the product to automatically generate test cases and related assets. Furthermore, the product uses machine learning and predictive analytics to ensure that all new test cases will either increase test coverage or retest a path that has previously resulted in a defect. In addition, Eggplant AI is able to learn patterns from previous tests that have resulted in errors, and will then repeat these patterns in new test cases until the errors stop occurring. Eggplant AI also comes with comprehensive reporting capabilities, including the run report, which details test execution status; the coverage report, which lists each object in your model along with their test coverage percent and may additionally be displayed graphically as a heat map of your system model; and the test case report, which allows you to search your run report for a specific set of test cases.
Apart from Eggplant AI, there are several other products present in the Digital Automation Intelligence Suite. Eggplant Functional tests from the user perspective through intelligent image and text understanding, API automation, and WebDriver object automation — all within a single test. Eggplant Performance and Eggplant Network together form a performance testing solution capable of testing in a variety of network and performance conditions on demand. Eggplant Automation Cloud allows you to generate and administer a centralised, cloud-based testing lab. Eggplant Manager helps you to manage your test execution process on a large scale, complete with continuous integration.
Eggplant offer FastStart services to shorten implementation times and get teams up and running faster. The company also provides training and certification in a number of Eggplant products, extensive documentation (available in English, Chinese, and Japanese), and a community forum. In addition, all of their partners are given access to the latest tools, training, and other support needed to deliver a full-fledged test automation solution.
Last Updated: 5th February 2020
Mutable Award: Gold 2019
The Eggplant Digital Automation Intelligence suite aims to automate every aspect of your product quality pipeline. In Eggplant AI’s case, AI and machine learning are used to automatically generate maximally covering test cases and actively target defects. In addition, to ensure your testing results in a good end-user experience, everything within Eggplant AI is done primarily from the perspective of an end user. Moreover, testing is not restricted to user journeys that appear sensible, taking the more realistic approach that any and every path through the application is a valid one, thus mimicking the real-world behaviour of your users.
Apart from Eggplant AI, the Eggplant Digital Automation Intelligence suite also contains Eggplant Functional, for test execution, and Eggplant Release Insights, which provides metrics for measuring and analysing release quality, including predictors for upcoming releases. Outside the suite, Eggplant provides performance and load testing as well as various customer monitoring products, including an entire second suite of products – Eggplant Customer Experience Insights – dedicated to monitoring your application once it’s in production. The two suites are fully compatible, and are available together as Eggplant Customer Experience Optimization.
“It’s so important to test the whole product, and testing it through the UI is the best way to do that. By running and looking at an Eggplant script, you can tell what the user was doing, what the business logic was, and what the UX was intended to be... So far, I haven’t found anything that Eggplant can’t do.”
“Other tools focus on testing at the code level, but Eggplant allows us to test our systems like a user.”
“Eggplant provides the best overall mix of capabilities from an end-to-end test perspective.”
Eggplant AI uses model-based testing to graphically model your system under test, as shown in Figure 1. You can create your model manually, import an existing model, or generate your model automatically from Eggplant tests, Gherkin feature files, monitoring data, or other sources. This process is intelligent, and will detect all valid paths implicit in the imported files even if they are not defined explicitly, thus unlocking additional coverage from your existing assets without any additional effort.
Your model can then be leveraged to automatically generate test cases, using AI to maximise the increase in test coverage that each new test case provides. You can create test cases manually by selecting a path through your model, import existing test cases from, say, Selenium, and weight individual parts of your model to make them proportionally more or less likely to be used in your automatically generated tests. Furthermore, the product uses machine learning to identify and understand patterns in your tests. This includes risk patterns, which can be leveraged alongside monitoring information to ensure that the most critical or valuable tests are always executed first, as well as any common patterns in your failing tests, which will be repeated in new tests until they stop resulting in failures, at which point they will gradually (and automatically) be unlearnt.
Your generated test cases will be executable if the corresponding parts of your model have been equipped with code snippets, which can be written manually or generated automatically using Eggplant’s ‘autosnippets’ image capture functionality. In addition, an accelerator allows you to generate your model directly from these autosnippets. Eggplant’s Universal Fusion Engine allows you to execute your scripts against any system, browser or device non-intrusively and without modifying any of your testing assets, and you have several ways of viewing your test results, including a coverage heat map as seen in Figure 2.
Eggplant also provides performance testing via Eggplant Performance, which is highly integrated with Eggplant AI. To wit, any Eggplant AI model can be used to create performance tests automatically. This goes a long way in making performance testing more accessible.
For Eggplant, testing should, above all else, increase customer satisfaction and improve the experience of the end user. In their words, testing should result in a product that “delights” its users. Unfortunately, this is rarely the case. According to Eggplant’s research, the vast majority of companies which employ testing are regularly meeting their testing objectives, yet only a small fraction considers themselves to be meeting customer expectations. Not only that, but on average, almost a third of IT budgets are spent on testing and QA, yet close to 90 percent of QA teams cannot keep up with their workload. In short, current testing is both insufficient and laborious. There is only one plausible solution: automation, and lots of it.
Accordingly, the automation present in Eggplant AI is extremely extensive, to the point that even test design is handled primarily via AI. This automation is suitable for either replacing or augmenting your more manual efforts, but either way it promises to dramatically increase the speed at which you can produce test assets, while at the same time maximising coverage and minimising bias. Moreover, the product provides several highly automated methods for generating your initial model, whether you have a suite of existing test assets or not. This makes it exceedingly fast and easy to get up and running with an Eggplant AI model, and by extension Eggplant AI.
The other products Eggplant offers are also highly valuable. Performance testing is a key component for ensuring a good customer experience – poorly performing apps are rarely well received – and the sophisticated and extensive release and customer monitoring Eggplant provides is highly useful for shifting right, managing change, and ensuring that your applications continuously improve over their entire lifetime. In combination, these products allow Eggplant to deliver a vast swathe of testing capability within a single platform via an intuitive, uniform experience. Furthermore, this intuitiveness allows your existing nontechnical users easy access to your testing process, empowering them to play an active role in it.
The Bottom Line
Eggplant is laser focused on enabling you to provide an exceptional customer experience via highly efficient and automated testing backed up by extensive analytics. Eggplant AI is both a core part of the company’s offering and a leading product in the Test Design Automation space. It should not be overlooked.
Mutable Award: Gold 2019
Eggplant Digital Automation Intelligence (DAI)
Last Updated: 4th March 2022
Eggplant Digital Automation Intelligence (DAI) is a software automation platform that leverages AI and machine learning (ML) to create a digital twin model (as seen in Figure 1) for functional, regression, and user experience testing across any device, browser, operating system, or platform. This typically includes automating all major aspects of your testing within your CI/CD pipeline, stretching from low-level test execution all the way up to high-level test design. In fact, one of Eggplant’s recent goals is to offer complete, end-to-end testing over a variety of systems at enterprise scale, including software and (thanks to integration with Keysight) hardware as well. To offer only one example of this, Eggplant and Keysight are currently being used to emulate and test entire 5G network towers.
Moreover, the testing approach taken is unflinchingly focused on the end user. This means that tests are created with little bias – Eggplant takes the point of view that any plausible path through your application is one a user might take, no matter how complex, and should therefore be considered in your testing – and can be executed just as if a human was piloting them, even operating at human-like speeds when appropriate.
Eggplant DAI is the core platform containing multiple products, including Eggplant Functional, for test execution, and Eggplant Real Customer Insights, for measuring and analysing release quality. Accelerators, such as Eggplant Salesforce Solution, are also available, to help customers quickly deploy Eggplant’s test automation capabilities (for their Salesforce SaaS instance, in this case).
In Eggplant DAI, you build a graphical model of your application, interface or website (your ‘system under test’, or SUT), which you can then use to automatically generate exploratory test cases, test scripts, and code snippets. This build process can be automated in terms of both model and test creation, and AI is used to maximise test coverage. You can also create test cases manually if you so choose, though it is recommended that you do this sparingly (if you need to prove you have certain tests in place for compliance reasons, for example). In addition, you can import existing (Selenium, say) test cases into your Eggplant environment. Risk-based testing is also available – again driven by AI – in order to prioritise particularly critical tests and identify, then repeat, common patterns found within failing tests until they no longer consistently fail.
Test cases generated by Eggplant are, as a rule, fully executable within both Eggplant Functional and directly inside Eggplant DAI. Eggplant DAI’s Universal Fusion Engine means that tests can be executed against any system, non-intrusively and without modifying your testing assets. Performance/load testing is also on offer, and in fact, Eggplant Functional scripts can also be used as performance tests.
Eggplant test scripts can also self-heal based on a set of parameters generated by user responses to failed tests. In other words, if you (explicitly or implicitly) tell Eggplant DAI that it makes no difference that an element on a webpage under test has changed size, Eggplant will recognise that and adjust the relevant tests to ignore an appropriate amount of change in scale. Over time, this should lead to a robust set of tests that will largely maintain themselves in the face of superficial changes to your SUT. However, running this process fully autonomously is not recommended, due to the possibility of false positives. Rather, a review process should be implemented. You can also let your tests locate elements via their underlying HTML/XML attributes directly (“DOM matching”) rather than their displayed properties, which can additionally contribute to test resilience.
You are provided several ways of viewing your test results, notably including a coverage graph and heat map (the latter of which is shown in Figure 2), and the model itself provides tagging and automated screenshotting, so that (for example) when a test fails, you can see what happened at each step without needing to rerun the test. There are also dashboards (see Figure 3) used to display various monitoring results: the dashboard for displaying the methodology behind the decisions Eggplant DAI is making is particularly notable.
Robust integration with Keysight is provided, and automated via the SenseTalk proprietary language developed in-house. Various other integrations are available, for instance, with KeyCloak for security and access management. You can also provide access to Eggplant DAI in the cloud via a series of microservices, or even deploy it as a cloud container in AWS.
Eggplant DAI has many strengths, but its greatest might be the extreme level of automation – and indeed, AI – present in the product. This includes automation all the way up to the design level, encompassing large parts of test creation, test execution, model creation, and test maintenance. It can be used to either replace or augment your more manual efforts, but either way it promises to dramatically increase the speed and ease with which you can produce test assets, while at the same time maximising test coverage and minimising bias. In addition, the automated model creation provided makes it fast and easy to get up and running with an Eggplant digital twin model, and by extension the product itself, minimising onboarding time and costs. The degree of humanlike and end-to-end testing provided by the product is also very impressive.
The Bottom Line
Eggplant DAI is a highly automated and intelligent test automation offering designed above all to facilitate the creation of seamless (testing of) user experiences. In this regard, we rate it very highly indeed.
Eggplant Salesforce Solution
Last Updated: 26th November 2021
Eggplant Digital Automation Intelligence (DAI) is a software automation platform that leverages AI and machine learning (ML) to create a digital twin model for functional, regression, and user experience testing across any device, browser, operating system, or platform. This typically includes automating all major aspects of your testing within your CI/CD pipeline, stretching from low-level test execution all the way up to high-level test design. AI and ML are used to automatically generate maximum test coverage with a testing approach that always focuses on the end user. DAI also contains metrics for measuring and analysing release quality, including predictors of success for upcoming releases.
The Eggplant Salesforce Solution augments Eggplant DAI in order to provide an out-of-the-box solution that is tailor-made for accelerating the testing of Salesforce applications and releases. The goal is to intelligently automate your Salesforce testing, reduce release times, increase test coverage and so on. As with Eggplant DAI, this is largely accomplished through model-based testing built around an AI-driven digital twin, a Universal Fusion Engine for executing your test cases against any device, and robust test and release monitoring capabilities. The difference is that this solution has specific functionality to address common pain points found when testing applications in a Salesforce context. It thereby helps you to effectively test Salesforce SaaS application environments and enable faster delivery of new Salesforce capabilities without compromising on quality.
In Eggplant DAI, you build a graphical model of your application, interface or website (your ‘system under test’, or SUT), which you can then use to automatically generate exploratory test cases, test scripts, and code snippets. This build process can be automated, and AI is used to maximise coverage during test case creation. Risk-based testing is also available – again driven by AI – in order to prioritise particularly critical tests and identify, then repeat, common patterns found within failing tests until they no longer consistently fail.
Test cases generated by Eggplant are, as a rule, fully executable, and the suite’s Universal AI Fusion Engine means that they can be executed against any system, non-intrusively and without modifying your testing assets. Performance testing is also available, and in fact, Eggplant DAI models can be used to automatically create performance tests.
Eggplant Salesforce Solution builds on this foundation in order to create a testing solution tailored to Salesforce in particular. This means that it can examine your Salesforce environment, its metadata, and the applications therein, then use what it finds there to automatically build your Eggplant testing model, and from there the rest of your testing assets. This includes capturing any business rules implemented within or other customisations made to your Salesforce environment, that will in turn be taken into account and incorporated into your model. Third-party offerings available through Salesforce (for instance, via the Salesforce AppExchange) are also compatible, enabling your testing model to be highly encompassing and holistic.
Eggplant DAI’s general advantages – extensive auto-generation of test scripts, AI-driven exploratory testing, and user experience-focused test automation capabilities, for instance – are highly applicable to the efficient testing of Salesforce environments. The Eggplant Salesforce Solution doubles down on these advantages while also introducing specific capabilities to address Salesforce’s unique foibles. For instance, Salesforce’s three mandatory releases per year is a common pain point that Eggplant addresses by marrying its testing model creation and test automation facilities to your Salesforce environment, and thereby providing the speed and automation necessary to allow you to create, update and execute your tests in good time prior to each such release.
For a second example, consider that there are two versions of Salesforce available: Salesforce Classic and Salesforce Lightning. Although the latter provides a more modern UI, both are widely used, and it is not unusual to, for example, switch to Salesforce Lightning and build apps in that while also having some older Salesforce Classic apps still kicking around and providing value (rather a lot of it, in some cases). Unfortunately, this can cause problems for your testing, because now you have two different application UI architectures living in the same Salesforce system. Thus, you will likely end up with two distinct, incompatible sets of test suites for it, neither capable of covering the whole environment and likely featuring a lot of repetition. It should go without saying that this is highly inefficient. The Eggplant Salesforce Solution addresses this issue by allowing you to leverage a common set of tests across both Classic and Lightning instances that can all be created from your singular Eggplant model. In short, it lets you test both Classic and Lightning apps interchangeably, without worrying overmuch about the implementation details of which kind of app each test will need to be run on.
The Bottom Line
Eggplant’s recent focus on enterprise applications stands to benefit greatly from platform-specific offerings, like Eggplant Salesforce Solution, that enable it to apply its AI test automation facilities to highly complex, large-scale enterprise SaaS application platforms, such as Salesforce, as easily and effectively as possible.