alphaspirit - Fotolia
Why a COTS package requires testing
Buying COTS systems doesn't necessarily mean security. Learn why COTS testing is needed.
Testing a commercial off-the-shelf (COTS) software system? Why would a COTS package need testing? Often, project managers and other stakeholders mistakenly believe that one of the benefits to purchasing COTS software is that there is little, if any, testing needed. This could not be further from the truth.
COTS packages are applications that are sold or licensed by vendors to organizations. This includes common enterprise applications such as Salesforce, Workday and PeopleSoft. The code delivered to each purchasing organization is identical; however, there is usually an administration module through which the application can be configured more closely to match the needs of the buyer. The configurations will usually be done by the vendor or by an integrator hired by the purchasing organization. Some COTS software vendors also make customizations, which involve changes to the base code, to accommodate purchasing organizations. Software as a service products are usually COTS software.
COTS testing requires a different focus from traditional testing approaches. Although no COTS package will be delivered free of bugs, the focus of testing from the purchasing organization's perspective is not on validating the base functionality. Since the COTS software is not developed specifically to meet user-defined requirements, requirements-based testing is not straightforward. In order to plan the testing effectively, test managers and testers need to focus on the areas where changes in the end-to-end workflow are made. The major areas of focus for COTS testing include customizations and configurations, integrations, data and performance.
Testing configurations and customizations
The focus of traditional functional testing when implementing a COTS package is on the customizations and the configurations. Customizations, since they involve changes to the actual code, carry the highest risk. However, configurations are vital, as they are the basis of the workflows. Testers need to understand what parts of the workflow involve configurations versus base code or customized code. Although the integrators sometimes provide this information, often the test team must obtain it from vendor documentation. Business workflows will need to change in order to achieve the same results through COTS software and testers must consider this as they develop their test cases.
Integrations are a critical area of focus when testing a COTS package. Often COTS software packages are large customer relationship management or enterprise resource planning systems and, as such, must be integrated with many legacy systems within the organization. Often, the legacy systems have older architectures and different methods of sending and receiving data. Adding to the complexity, new code is almost always needed to connect to the COTS package. Understanding the types of architectures and testing through the APIs and other methods of data transmission is a new challenge for many testers.
Data testing is extremely important to the end-to-end testing of COTS software. Testers must understand the data dictionary of the new application, since data names and types may not match the existing software. Testers will have to work with the vendor or integrator to understand the data dictionary. In addition, the tester must also understand the extract, transform and load (ETL) mechanisms. This can be especially complicated if there is a data warehouse involved. Since a data migration will likely been needed, the data transformations will need to be thoroughly tested.
ETL testing requires a completely different skill set from that of the manual, front-end tester. The organization purchasing the COTS package will need to contract with resources that have the appropriate skills or developers may fill the role with test lead oversight. Knowledge of SQL and a thorough understanding of how to simulate data interactions using SOAP or XML are required for data testing. An understanding of SOA and the tools used to test Web messages is also quite helpful.
Performance testing is another area requiring a different approach. Many systems, especially Web applications, require a focus on load testing or validating that the application can handle the required number of users simultaneously. However, with large COTS applications that will be used internally within an organization, the focus is on the speed and number of transactions that can be processed, as opposed to the number of users. The test scenarios for this type of performance testing can be huge in number and complexity. Furthermore, the more complex scenarios are also data intensive. This testing not only requires testers with solid technical performance test skills, but also requires a detailed data coordination effort for integration.
From beginning to end, testing the brave new world of COTS software requires a completely different approach focusing on configurations, integrations, data and performance. This new approach offers new challenges and provides opportunities for testers to develop new strategies and skill sets.
