The pros and cons of using generative AI for API testing

Types of API testing

APIs serve as the bedrock of composability, reuse and integration for web applications. They provide a standardized way to exchange data and extend capabilities in a mature ecosystem. As a result, developers need to address multiple complex dimensions for comprehensive API testing. Some of the key areas include the following:

• Backward compatibility and API versioning.
• Data dependencies and contract adherence.
• Test data management and evolution.
• Data generation and programming of complex scenarios.
• Environment parity and configurations.
• Lockstep testing for intertwined integrations.
• Third-party integrations.
• Performance and load testing.

Traditional testing strategies rely on statically generated data sets and response validation. Although these methods are generally effective, conventional testing approaches might not fully account for unexpected functions that could arise from complex API interactions and responses. For example, some APIs might respond with different validations for separate geographic regions. Others might behave differently when chronology is disrupted, like events occurring out of sequence in rapid succession. This variability warrants a closer simulation of factors such as environment, users, performance and throughput to produce more accurate results and identify unanticipated outcomes.

Benefits of generative AI in API testing

With the right tools and strategies, GenAI technologies such as large language models, ChatGPT and applications built over GPT models such as RestGPT offer new approaches for creatively structuring tests.

Context-specific configuration, quality training data and customized testing parameters are key to effective deployment. The following categories outline some of the emerging advantages and possible benefits of GenAI as teams look to find the right blend of manual administration and automation.

Data management

Language models can dynamically generate test data for API contracts, data types and scenarios, widening coverage for both success and error cases. The models are capable of preemptive and ad hoc data generation.

Response and contract validation

Language models can validate responses, and GenAI can flexibly adapt to multiple data formats such as JSON and XML, as well as standards like HTML, REST and SOAP. GenAI can also assist with backward compatibility by handling testing across versions.

Wider scenario coverage

GenAI can generate initial test scenario lists, identify gaps in coverage, augment existing test sets, help cover edge cases and evaluate and expand the test suite as APIs evolve with limited manual intervention.

The technology is also capable of contextually generating data and responses for multiple API versions, different consumer types and other complex scenarios. These capabilities can reduce costs associated with manual intervention, data creation and execution.

Automation

GenAI's data generation and validation capabilities can bring enhanced automation to API testing. It can also execute complex lockstep API orchestrations in interdependent systems that require data synchronization. In dynamic and complex environments, GenAI can act as a source of coherent data to reliably test a complex integration.

Vulnerability spotting

With a comprehensive understanding of API context, application roles, users and permissions, GenAI can help spot data leakage, privilege escalation and other security issues by constructing well-formed requests that can identify a lack of required controls and checks.

Lower cost of evolution

Evolving an API contract with multiple versions and subscriber sprawl could mean testing tens of versions for backward compatibility and validating hundreds of scenarios for errors. Each version or scenario could require data generation and validation, making it both an expensive and protracted process.

GenAI can augment capabilities and bring down costs associated with scenario and data generation and validation. Another possible benefit is that teams could channel energy into responsibilities that add direct value to the organization rather than spend time executing tedious and redundant tasks.

Downsides of generative AI in API testing

Although there are many perceived benefits of generative AI in the API testing space, it's important to understand the limitations of the technology. Some tradeoffs might be immediately apparent. But as a new and developing technology, there could be unrealized implications associated with GenAI integration.

Erosion of institutional knowledge

Teams responsible for implementing test harnesses acquire knowledge of system functionality, functional edge cases and data correlation between different parts of the system. Over-reliance on automated tools erodes institutional knowledge that can be otherwise difficult to build over time. In the absence of this expertise, functional questions might require exhaustive testing or time-consuming and inefficient code reviews.

Variable reliability

Since generative AI responses can be inconsistent, the execution path and outcome for certain test cases can change, resulting in false positives and negatives. Unpredictable outcomes could potentially contribute to a growing lack of confidence in the reliability and effectiveness of existing safety nets.

Specialized skills

Prompt engineering can alter behavior in GenAI tools. But when prompts create impediments and fail to address issues, expert consultation could be necessary. Candidates with specialized skills in prompt engineering can be difficult and expensive to hire, train and retain.

Resource intensive

Generative AI is resource-intensive, especially in test execution scenarios that invoke AI in real time. Revising test data and scenarios prior to execution could offer a more balanced approach in terms of mitigating costs and resource usage.

Compliance issues

Industries governed by strong regulatory and compliance requirements, like banking and finance, require explainability for decision-making throughout the software development lifecycle. Questions surrounding GenAI explainability and transparency make it difficult to consider the possibility of pure generative AI testing without manual oversight. There are also concerns about leaking sensitive or proprietary data inadvertently through third-party hosted models. As a result, these considerations warrant a more circumspect use of generative AI in certain domains, along with clear security policies.

Priyank Gupta is a polyglot technologist who is well versed with the craft of building distributed systems that operate at scale. He is an active open source contributor and speaker who loves to solve a difficult business challenge using technology at scale.