Nine ways to evaluate automated software testing tools
Evaluate automated software testing tools more thoroughly with these tips for analyzing cost, support, total cost of ownership, usability and more. This tip suggests questions to ask about each attribute of a tool and a vendor's support for it.
Software test automation tools can be very beneficial to a company's software testing organization. The aim of these tools is to increase test coverage while decreasing the cost of the actual testing function, in addition to shortening the amount of time needed to test and requiring less resources. In theory, total "automation" means absolutely no manual intervention is necessary -- such actions include starting and stopping scripts, creating new scripts and maintaining them as the project's needs change. But although "automation" sounds like a great feature for software testing tools to have, there isn't a tool on the market today that is 100%, fully automated.
Software quality assurance professionals can find themselves teetering on a line between the amount of money they want to invest in a software test tool, and the software testing tool's capabilities. The decision often comes down to either an open source software testing tool or a vendor-based software testing application. So how can you be sure you're purchasing the right software test automation tool for the job?
Before you do anything, it is crucial that you put together a strategy prior to searching for a tool. Research and review the needs of your company and what issue(s) the tool will be expected to solve. Make a list of tasks that the tool should be able to perform, then describe how each task will improve its respective business function. Think about how large or small the tool's user base will be and keep their requirements in mind as you align the tool's functions to your company's strategy.
After you have a strategy in place, you'll be ready to choose a tool. Here are nine important factors to consider:
Maintenance Longevity
Ask: How long will the vendor be able to maintain the software testing tool application? Is our company stable enough to continue maintenance if we invest sufficient time, money and/or resources? The application may be offered at a discounted price (or even for free), but there must be enough manpower to follow through and maintain the product.
If it's an open source application and the open source maintenance team loses their way due to lack of resources or other problems, then so might the company's ability to manage the product. But remember, if you maintain the product in-house then you will also need to test it in-house, which can put a strain time and manpower, along with opening a window for bugs. Open source products are great for environments that have long project schedules or where high-level R&D exists. They are not as feasible in environments where all that need to be accomplished are quick, automated tasks.
Updates
Ask: Will the open source or vendor-based application be regularly updated with new versions of the software? What if the old software has a 'bug'? As system environments change (operating systems, plug-ins, browsers, etc.), so does the need for updates. Updates should be conducted on a weekly -- or at the very least, quarterly -- basis. This is an important step to ensuring that the software testing tool's maintenance group receives excellent internal support.
Usability
Ask: Is the layout of the open source or vendor-based application designed in such a way that it takes fewer actions to work on any given function? Would both a novice tester and expert tester find it easy to use? The layout of open source applications can be difficult to manage and sometimes requires an in-depth knowledge of computer programming language such as C+ or VB. And many new open source applications are coming out with layouts that are similar to their vendor-based counterparts. Therefore, you must determine how much time the tool would take to implement and the kind of engineer it is designed for use by. Software testing tools have become more user-friendly as of late, but the user must still be capable of fully understanding the intricacies of the testing strategy and the application that is being tested. And the tool will reach its highest potential if it is used by a "quality-minded" individual (or group of people) who understand that quality is integral to reaching the company's goals.
Compatibility and integration
Ask: Does the open source or vendor-based application have the capability to integrate with other tools, such as a defect-tracking tool? Is the tool compatible in the company's environment?
The tool's functional ability increases if it has a built-in capacity to integrate with other applications and databases. Make sure that the software test automation tool is compatible with the application being used to automate and test, the client operating system that it is running on, Web browsers and other items like change control. Additionally, the tool's compatibility is a key indicator of how accessible it will be when executing scripts. You should find out if the tool can be executed from any machine or if an automation client needs to be installed first, both of which will then determine the price of the license.
Manuals
Ask: Does the product include an operation manual that will help support it throughout use? It is always a good idea to have some sort of reference material to support the tool's end-user, especially when it comes to executing a unique function specific to that tool. If reference documents are not available -- as is the case with some open source applications -- you should consider creating your own. Reference guides that are compact but still indicate the full functionality of the tool are a great asset to give the end-user, especially if the original product manual is cumbersome to the everyday user.
Customer support
Ask: Is adequate customer support made available by the product or vendor? What is their hour of operations? What is their response rate? Does it come at an additional cost?
Many times what you are truly paying for when you buy a testing tool is product marketing, R&D and support. So, keep in mind that if you are paying a lot of money for a tool, it should be top-notch … or in the end, "you will pay for what you get." Find out if the product includes customer support. Some open source applications might claim they have customer support, but read the fine print to find out if support is only offered online or if you can actually speak to a live person on the phone. Another thing to find out is where customer support is physically located. If you have an urgent problem at 7 a.m. and you live on the east coast, don't expect an immediate call back from the vendor's customer service center if it is located on the west coast. Some vendors may even have offshore resources. As you test the product it is important that you also test the customer service offerings by calling and creating service requests.
Online user communities
Ask: Does the application offer an online community for users to share information with other users?
How often have you had a question that someone else knew the answer to? A product's online community is a wonderful resource to utilize when you want to find out if other users have the same problems/like/dislikes about the product as you do. Online communities are an added-value to a tool's capabilities. Not only do they help users share information with each other, but they increase brand loyalty and spur trust among current users and potential users alike. When researching if your tool has such a community, beware of the slew of unofficial collaboration communities and Web sites out there -- for the most accurate information, go straight to the tool's official online community Web page.
"Center of Testing Excellence"
Ask: Who will manage the software test automation tools, internally? Will there be a centralized group that will manage all department software test automation tools or will each department manage the tool by themselves?
The method through which a tool is managed will make or break its success. Maintaining a Center of Testing Excellence makes sense -- especially with software testing automation tools -- because it acts as the central departmental hub where all information about the tool is stored; from management and installation, to knowledge and usage. The Center of Excellence assists each department that uses the software testing tool within the departments. Sometimes smaller groups may have independent testers who float to and from other departments. Larger groups may have permanent testers who work full-time testing in only one department. Regardless, the Center of Testing Excellence is a pillar of support -- and in some circumstances, it can even act as a stand-in for departmental testers.
Sales
Buyer beware: the software automation testing tool you are about to purchase encompasses much more than the fluff presented by your sales representative. "Try before you buy" in order to get a feel for the technical aspects of the tool and understand how it will or won't meet your needs -- these are things that your sales rep most likely never told you about. Make sure that you install the tool in a test environment, use it frequently, and then comprise a list of questions for the vendor and its technical team. Be aware of the multitude "services" a vendor promotes; step back and analyze if it is cost effective and if it meets the business' goals. Sales teams may use the software test automation tool as a springboard for introducing additional, often unnecessary services to their potential clients.
In some cases, it may be better to compare multiple systems at the same time and conduct your own analysis. Each tool may act differently, depending on how your company conducts testing. Don't expect the vendor or independent agencies to make a comparison of the tools for you to save time, nor should you rely on them as experts. As the end-user, you are an expert within your industry and you need to take the time to understand if the application makes sense for your environment. If you speed through this process without any direction or plan, you will be stuck with a tool that meets neither the needs of the company nor the users.
As previously mentioned, it is very important to have a group of end-users involved in the decision process of whether or not to purchase the tool and any additional services. Corporate politics sometimes plays a large part in the decision-making process, and the end users are stuck with a tool that doesn't work for them and the corporate goals. So, don't be fooled by the vendor's sales team. You might end up losing in the end.
Just a note - you may want to take screenshots of the application while you are using the temporary demo, by producing a document that compares each screenshot of functionality. After the short-term license expires, you may need this documentation to help you reflect upon what was positive and negative about the tool during your final decision-making process.
Up-front and indirect costs
"Cost" means not just price of the tool, but also installation time, ramp-up knowledge and other indirect costs. Some of these factors were already indicated above, like training, ramp-up speed, resources, documentation, installation, compatibility and manuals. You should take a step back in order to account for all costs -- especially the indirect ones -- because sometimes the total price of the vendor-based tool may be less expensive than the open source application and vice versa depending on how well the open source product was supported. To this end, indirect costs can cause a major issue and often creep up when you least expect it. For example, I have witnessed situations where applications were purchased but no budget was allocated for their installation or environment, which resulted in having the tool operate in an inadequate, shared environment.
 |
||||
|
|
|||
|
||||
Keep these nine items in mind as a guide while you review your options to purchase software test automation tools. Make sure that you fully understand your company's goals before you begin the process -- it will reduce the risk of choosing a tool that meets only part of the business' needs. Remember, "Quality" is the backbone of the Software Development Life Cycle. If the acquisition of a software test automation tool is rushed and conducted unsuccessful, there's a good chance that it will cause a domino effect and infiltrate other departments. So, take your time. The better you understand what you're buying, the more confident you'll be in your final decision.
About the author: John Scarpino is director of quality assurance and a university instructor in Pittsburgh. You may contact him at [email protected].
