Compare the top automated functional testing tools
This roundup of top automated functional testing tools, including Selenium, Micro Focus UFT and TestComplete, will guide you to pick the best choice for software testing needs.
Application testing's variety of technologies and styles make it impossible to pick a single, universal test tool for every use case and software team -- companies need a toolbox. In this article, we highlight the top tools to perform automated functional testing, which turns test execution steps into scripts and sets of variables.
Functional test automation tools cannot design application tests for you, and their insight into what went wrong when a test fails is generally limited. Automated functional testing tools mimic how a user would interact with the software under test via the UI. Within this category, record-and-playback tools track the user interactions with the software under test and create a set of commands based on them, akin to a script. That script can be played back again and again.
Record-and-playback tools often fail when a code change alters the appearance of a UI screen, though the actual functionality is unchanged. Because of this, many of these scripts are stripped down -- the test is recorded, then a human reviews and simplifies the code that hooks to an element of the UI, so the code is more forgiving. For example, a UI button is hooked to a unique identifier so you click it no matter where it is on the screen or what text it has. Keyword-driven testing is often used with record-and-playback tools, to look at the specifics of interactions rather than every aspect for each line in the script.
Overall, when evaluating the automated functional testing tools below, consider the type of application under test -- is it a compiled desktop application, a website, a contained web application, a mobile website or a mobile application? Each setup requires a different approach and tool set. Native mobile applications can be difficult to automate or test from a desktop. Mobile web applications can run on a desktop but may look or behave differently than on the mobile device. Unless it's an all-in-one tool, a web app automation test tool likely won't support mobile applications.
Most, if not all, of the products in this roundup have free trials. Narrow the list down to potential choices, then explore and build a proof of concept before committing to a tool.
All-in-one automated functional testing products
These tool suites are designed for testers with no familiarity with coding or those with some programming expertise.
SmartBear TestComplete. TestComplete meets the needs of Windows-based less-technical testers who don't want to write code. It tests Windows, web and mobile (iOS and Android) applications all in one package.
Based on record-and-playback technology, TestComplete has two views: visual and code. The visual view of TestComplete shows the commands, such as click, wait and type, followed by parameters -- where to click, how long to wait, and what to type. That means a target user or product manager can read the test to ensure that it's what the software should do. Test engineers can dive into code in TestComplete but wrap that code in a function for the visual view. For example, the coded login function can be a single line in the visual view, taking a username and password as inputs.
Teams that want to version control tests along with software code should use Git, Subversion, Mercurial or Microsoft Team Foundation Server. TestComplete application uses its own format for data, and you'll want to check code in from the TestComplete development environment.
TestComplete's paid support includes a training package of videos you can purchase, a monthly two-day remote video training course, in-person training and hourly support. The biggest issues our clients experience with TestComplete are with its support for updated versions of browsers (especially Internet Explorer) and Java applets -- automatic browser updates can sometimes trigger before the SmartBear team has a new version in place.
TestComplete also has expanded into the open source world, adding support for Jenkins and Selenium by way of Java's JUnit. If your team uses those tools and wants to run every application test in one place, TestComplete might be the right choice.
Micro Focus UFT. If the software test automation must also include tracking when the tests ran and on what build and server, you might consider Micro Focus' Unified Functional Testing (UFT). Like TestComplete, UFT uses a visual model, but its model is more like a flowchart than a grid, with the possibility of loops and conditionals -- if statements -- visualized.
Micro Focus UFT, formerly Hewlett Packard Enterprise's QuickTest Professional product, is designed to integrate with Quality Center, Micro Focus' application test coverage tool, as well as Sprinter, its exploratory test coverage tool. Together, these products provide documentation and evidence for an organization with strict compliance regulations, such as an insurance, medical or financial services company. UFT can also use Micro Focus' Business Process Testing to jumpstart test suite builds. The component-based test framework for functional test automation is effective for organizations testing ERP installations and upgrades, for example.
The UFT suite works well for organizations that already rely on Micro Focus tools and use the vendor's service and support.
Tricentis Tosca. The Tosca test automation tool has two primary selling points for enterprises: It's a model-driven testing tool, and it supports dozens of user interfaces.
Tosca models the application under test, including the inputs, to help with test design and execution. That makes maintenance easier. To support a code change, the tester can change the model, generating new tests. This model-based testing is different than the click-click-check model that Telerik, TestComplete, and Ranorex use.
The user interfaces it supports include .NET applications, SAP, Flash, Dreamweaver, Java, Powerbuilder, Visual Basic, WPF and many more. Tosca can drive Excel to create the spreadsheet needed to import into an application for functional tests, and even read from PDF reports. At publication date, Tosca runs only on Windows, and my experience is that it tends to require a relatively new and fast computer at that.
Idera Ranorex Studio. Ranorex is a Windows-based application with a working metaphor similar to TestComplete or Telerik: Each recorded test case has "steps" as a row in a grid, and you "play" it back. Suites consist of a series of test cases. The primary way to work with Ranorex is via record-and-playback and the visual interface.
Ranorex's main strength is an ability to test older Windows applications, as it has tools to uniquely identify elements that new tools may lack. Teams that support a wide variety of applications, but want traditional, straightforward test design, should evaluate Ranorex. Organizations that already use Idera's TestRail by Gurock test case management platform might also be inclined to stick with the same company for test execution automation.
Telerik Test Studio. Teams that use Microsoft Visual Studio integrated development environment to create web or desktop applications might be interested in Telerik Test Studio. This is especially true for technical testers who work with code, or teams that want their programmers to contribute to automated testing.
Telerik's code is the same native programming language as Visual Studio. Drop into the code in Telerik to extend it, and you're in the Visual Studio IDE, writing code in the exact same language as the programmers on the team. That means the programmers and testers can share code libraries. TestComplete's C# and VBScript modules are not true Visual Studio code and have minor syntax differences, which means programmers and testers might not be able to share the same code libraries.
The tight integration between Telerik Test Studio and Visual Studio means teams must use Windows to run the automated functional testing tool, not Linux or Apple.
Eggplant Functional. Teams writing code on Linux, Apple or in a mixed environment will want to look at Eggplant Functional. Eggplant has added AI to test automation to create visual tests, moving in the direction of Tosca's model-based capabilities. These tests use massive amounts of test data about failures to interpret changes and retest after them -- in other words "healing" the test, similar to web-based tools such as Mabl and Testim.
Visualizations for Eggplant create an intuitive view into software flows for non-technical stakeholders, such as project managers, graphic designers and executives, bringing them into the test conversation in a new way. That said, it is possible for testers to get out of their depth with this feature, creating a complex system that is hard to understand as a whole.
Web-based startups: Mabl, Testim, TestProject. You may have noticed a pattern above: OS is a problem for automated functional testing tools. Support for mobile applications introduces another problem. Adding these tools into the continuous integration loop is another challenge.
To address these roadblocks, a crop of vendors has emerged with test tools that run on the cloud. Mabl, Testim and TestProject all record the functional test in a browser or browser extension, store it in the cloud, and can test any public website. Mabl and Testim both use AI to guess when a test fails and offer the aforementioned self-healing, which assists in test maintenance. And, of course, they have free trials.
As of early 2020, these products are starting to gain traction with early adopters. Give them a try.
Stand-alone web and mobile automated testing tools
While the tool suites above are for testers, stand-alone automated test tools are generally used by programmers or testers who code.
Selenium. Teams that are comfortable with code and only need to test web applications through the UI look to Selenium, a free open source project for testing websites and web applications. Its most popular component is WebDriver, a code library for many programming languages that automates the web browser with commands such as open, click and type. Selenium provides a number of components that must be installed to work effectively -- you need the code library, browser and browser driver, which is an executable, plus the code to do something with it. Without all these installed in the right directories and aware of each other, the program calls will generate errors.
The on-ramp to Selenium WebDriver is the Selenium IDE, an in-browser record-and-playback tool that can produce and record scripts, and also convert them to WebDriver code in Java. As a record-and-playback tool, the IDE only runs in FireFox and Chrome browsers, but WebDriver runs in Internet Explorer/Edge, FireFox, Chrome and Safari.
Selenium WebDriver has a large development community and supports various popular programming and scripting languages. Teams that want to write test code in the same language as the production code will find support for Java, C# Ruby, Python, JavaScript (Node.js) and more.
Commercial software products support Selenium. Sauce Labs, for example, provides virtual test environments to simulate any modern browser and the grid to run many application tests at once. TestComplete can run Selenium tests written in Java, and Mabl and Testim.io both use Selenium under the hood.
Appium. Companies that develop mobile applications and mobile websites should look beyond Selenium to Appium, an open source mobile testing tool. Appium uses the same API as Selenium WebDriver, which means the application testing frameworks available for WebDriver also are available for Appium. Sauce Labs provides Appium support as well.
Perfecto Mobile. Although it started out as a virtual lab of real devices for users to test on, Perfecto Mobile has extended its features to support automated testing against that network of real devices, along with boot camps and training options. The company has a proprietary interface to design software tests with little to no coding. It also supports Selenium and other open source frameworks.
If your product will be available on diverse mobile devices -- Android and iOS, tablet and phone -- and you must support both older devices with smaller screen resolutions and newer devices with larger screen resolutions, you might want to look into Perfecto Mobile.
Sauce Labs. Mobile software makers have to manage a library of test devices. If you work with Apple only, there are four iPads, three iPhones and the Apple Watch that fall into the category of newer versions -- you might also want to test for use on older versions. The entire Android ecosystem is much larger.
Sauce Labs offers a large set of emulated or real mobile devices to test on as a service in the cloud. This service works for manual and automated testing. The company offers support and training options, so that any reasonably skilled programmer can set up tests tooling to run in a few hours.
