Manual testing vs. automated testing: A decision point

Choosing to implement automation in a test organization can be a somewhat controversial decision. Many automation efforts fail or do not yield the expected return on investment. In this tip, we’ll look at specific key factors that ensure success with your automation initiative.

Concerns

Automated test execution has become an accepted and well-established software testing strategy in the last few years. Automation is not new anymore, and IT organizations understand the strengths and weaknesses, as well as the role of automation in improving time-to-market and test coverage while reducing manual labor cost of testing. Yet, a surprising amount of automation initiatives have failed, and as a result, automation’s popularity as an effective and efficient strategy is being questioned by many IT leaders. Equally surprising is the low level of automation of test execution that is noticeable in most IT organizations, across industry sectors like banking, insurance, healthcare, energy, manufacturing, etc., despite investing heavily in the procurement of automation tools and infrastructure. Lack of success stories and insufficient data on return on investment is leading to concerns about the initial investments required to automate test execution. In fact, very few IT organizations capture basic automation testing metrics needed to assess the maturity of their automation processes and plan future improvements. No wonder most IT organizations still prefer manual testing strategies. So we are left wondering if automation will ever succeed. What variables ensure the success or failure of automation strategies? Will automation testing replace manual testing?

History of automation

To understand the answers to these questions, let us step back in time and understand the evolution and need for test automation. In the late 90’s, many test automation tools which relied heavily on record and playback started becoming popular. However, most of the automation projects failed since record and playback was not a workable option. In early 2000, new versions of automation tools started emerging with the increasing complexity of technology. The volume of testing increased, cost escalated and the pressure to attempt automation came naturally. This resulted in increased investments in automation projects. With technological changes and legacy modernization adding to the maintenance workload, the lack of effective governance to deal with these rapid changes led to more failures than successes for the automation projects. In time, testing organizations started realizing the stringent selection and maintenance procedures that were needed to make automation successful. Even though they lacked a clear understanding of what parameters influenced the outcome of automation projects, this also moved the focus back towards manual testing.

These events provided IT organizations with important lessons, and they realized three key points:

1. Automation is not just a one-time initiative.
2. Failure rates for automation are higher than success rates, and therefore careful planning is essential.
3. Automation skills and tools are expensive. Effective governance is instrumental in making automation successful across the enterprise.

IT organizations have witnessed the revolution in technology advancement in the last decade; technology has become more complex, applications have become more distributed and testing has become an accepted discipline. Organizations learned the following:

1. The cost of testing would escalate and there was an immediate need to improve effectiveness and efficiency of testing processes.
2. Automation was still an underutilized testing strategy.
3. Automation as a concept had matured.
4. The reasons for deciding between manual and automation testing were clear.

All this led to the development of automation competencies in organizations which had the direction, focus and funding to make test automation a success.

Ensuring successful automation

So how can organizations ensure the success of their automation projects? Some of the key factors which need to be taken into consideration to ensure successful automation are listed below:

Enlarge diagram.

Clearly there are several parameters which need to be taken into consideration to ensure the success of automation and reduce the percentage of manual execution. Organizations can lay a strong foundation to ensure the success of automation by focusing on the initiatives listed in the above table. This will help improve the effectiveness and usage of automation.

Role of manual testing

While automation continues to evolve rapidly, it is too early in the technological revolution to replace manual testing completely with automation. In fact, most of the new features, complex validations and business intensive functionalities will continue to be tested manually. The goal of 100% automation is not just ambitious, it is also impractical. Typically 70% automation helps maximize return on investments. Hence, manual testing remains and will continue to dominate in organizations with lower levels of automation maturity and in areas where ROI on automation is not significant.

Conclusion

While the focus on automation continues, manual testing will remain dominant for the next few years. As the urge to utilize automation grows, the urge to uncover the complex paradigm shift necessary to increase automation and reduce manual testing still needs attention. It is clear that a careful and structured approach is necessary to increase the rate of automation. Since multiple variables affect automation, a strong foundation, followed by a focus on excellence in execution, is necessary for succeeding in automation. Very soon, the automation of test execution will become inevitable. An organization’s preparedness will dictate its testing cost structure and future competitiveness.