Manual mainframe testing persists in the age of automation
Mainframe testing is still done manually by many IT shops, despite mainframe modernization and digitization efforts that benefit from automated software testing methods.
A recent study indicates that although most IT organizations recognize software test automation benefits their app development lifecycle, the majority of mainframe testing is done manually, which creates bottlenecks in the implementation of modern digital services.
The bottom line is that mainframe shops that want to add new, modern apps need to adopt test automation and they need to do it quickly or get left behind in a world of potential backlogs and buggy code.
However, while it's true that mainframe shops have been slow to implement automated testing, it's mostly been because they haven't really had to; most mainframe shops are in maintenance mode, said Thomas Murphy, an analyst at Gartner.
"There is a need to clean up crusty old code, but that is less automated 'testing' and more automated analysis like CAST," he said. "In an API/service world, I think there is a decent footprint for service virtualization and API testing and services around this. There are a lot of boutique consulting firms that also do various pieces of test automation."
Yet, Detroit-based mainframe software maker Compuware, which commissioned the study conducted by Vanson Bourne, a market research firm, found that as many as 86% of respondents to its survey said they find it difficult to automate the testing of mainframe code. Only 7% of respondents said they automate the execution of test cases on mainframe code and 75% of respondents said they do not have automated processes that test code at every stage of development.
The survey polled 400 senior IT leaders responsible for application development in organizations with a mainframe and more than 1,000 employees.
Overall, mainframe app developers -- as opposed to those working in distributed environments -- have been slow to automate mainframe testing of code, but demand for new, more complex applications continues to grow to the point where 92% of respondents said their organization's mainframe teams spend much more time testing code than was required in the past. On average, mainframe app development teams spend 51% of their time on testing new mainframe applications, features or functionality, according to the survey.
Shift left
To remedy this, mainframe shops need to "shift left" and bring automated testing, particularly automated unit testing, into the application lifecycle earlier to avoid security risks and improve the quality of their software. But only 24% of organizations reported that they perform both unit and functional mainframe testing on code before it is released into production. Moreover, automation and the shift to Agile and DevOps practices are "crucial" to the effort to both cut the time required to build and improve the quality of mainframe software, said Chris O'Malley, CEO of Compuware.
Yet, 53% of mainframe application development managers said the time required to conduct thorough testing is the biggest barrier to integrating the mainframe into Agile and DevOps.
Mainframes continue to be viewed as the gold standard for data privacy, security and resiliency, though IT pros say there is not enough automated software testing for systems like the IBM system z, pictured here.
Eighty-five percent of respondents said they feel pressure to cut corners in testing that could result in compromised code quality and bugs in production code. Fifty percent said they fear cutting corners could lead to potential security flaws, 38% said they are concerned about disrupting operations and 28% said they are most concerned about the potential negative impact on revenue.
In addition, 82% of respondents said that the paucity of automated test cases could lead to poor customer experiences, and 90% said that automating more test cases could be the single most important factor in their success, with 87% noting that it will help organizations overcome the shortage of skilled mainframe app developers.
Automated mainframe testing tools in short supply
Truth be told, there are fewer tools available to automate the testing of mainframe software and there is not much to be found in the open source market.
And though IBM -- and its financial results after every new mainframe introduction -- might beg to differ, many industry observers, like Gartner's Murphy, view the mainframe as dead.
The mainframe isn't where our headspace is at. We use that new mainframe -- the cloud -- now.
Thomas MurphyAnalyst, Gartner
"The mainframe isn't where our headspace is at," Murphy said. "We use that new mainframe -- the cloud -- now. There isn't sufficient business pressure or mandate. If there were a bunch of recurring issues, if the mainframe was holding us back, then people would address the problem. Probably by shooting the mainframe and moving elsewhere."
Outside of the mainframe industry, companies such as Parasoft, SmartBear and others regularly innovate and deliver new automated testing functionality for developers in distributed, web and mobile environments. For instance, Parasoft earlier this fall introduced Selenic, its AI-powered automated testing tool for Selenium. Selenium is an automated testing suite for web apps that has become a de facto standard for testing user interfaces. Parasoft's Selenic integrates into existing CI/CD pipelines to ease the way for organizations that employ DevOps practices. Selenic's AI capabilities provide recommendations that automate the "self-healing" of any broken Selenium scripts and provide deep code analysis to users.
For its part, Gartner named SmartBear, another prominent test automation provider, as a leader in the 2019 Gartner Magic Quadrant for Software Test Automation. Among the highlights of what the company has done for developers in 2019, the company expanded into CI/CD pipeline integration for native mobile test automation with the acquisition of Bitbar, added new tools for behavior-driven development and introduced testing support for GraphQL.
Dig Deeper on Software testing tools and techniques
