Effective scalability testing helps to evaluate customer experience, protect against major failures and safeguard an organization's reputation. Here's how to set a strategy.
Scalability testing evaluates how your application ecosystem, including network, hardware, CPU and databases, performs under varying numbers of user activities. Understanding that performance is important because it goes directly to UX.
Ticketmaster's systems failures in 2022 during the sale of tickets for Taylor Swift's The Eras Tour highlight the importance of scalability testing. Ticketmaster's systems could not scale to meet the demand of millions of devoted fans eager to access a limited supply of concert tickets.
Scalability failures of that sort can anger potential customers -- fans filed suit -- and even harm sales. Let's look at the dos and don'ts of effective scalability testing.
The dos of scalability testing
To develop a strategy that makes good use of scalability testing, there are a few key points to bear in mind.
Understand the purpose
A scalability test demonstrates the effects of projected increases in the use of an application. This looks at increases not only in the number of concurrent users, but also increases in data volume and the number of user requests; plus, it considers network bandwidth and CPU usage.
A scalability test demonstrates the effects of projected increases in the use of an application.
Scalability testing differs from soak and load testing. Soak testing examines the effect of heavy load for long periods of time. It provides information about how the application acts outside of its intended use. Load testing also examines the impact of load on an application, but it focuses on the point at which responses fail. Scalability testing, meanwhile, seeks to understand the effect of changes in numbers of users and other performance attributes.
Plan scalability tests strategically
It is critical to know when to run tests. Test when changes are made to attributes such as network bandwidth, server size and capabilities, changes to the CPU and so on. Especially when planning scalability testing for retail websites, it is critical to anticipate and test for sharp increases in user volume. These increases may be seasonal, such as holiday shopping, or before major events, such as a Taylor Swift tour.
Understand the attributes of scalability testing
Important attributes of scalability testing and application performance include the following:
Response time. Response time is the amount of time elapsed between the request made by the user and the system's response. If the application is scalable, the response time does not decrease as the number of concurrent user requests increases.
Throughput. Throughput is the number of user requests processed in a pre-selected amount of time. An application can be considered scalable if increasing load does not harm throughput.
CPU usage. CPU usage, which is the amount of CPU necessary for processing a given task, should not increase as the number of user transactions, data volumes and other load factors increase. The most scalable applications are those that are optimized to use the least CPU usage per transaction.
Memory usage. Code should be optimized so a transaction can be processed using the least possible amount of memory. This reduces the risk of running out of memory when the number of transactions jumps.
Network usage. This measures the amount of bandwidth consumed per transaction. The goal is to reduce this amount so that increased volume does not require increased bandwidth.
Choose the right test tools
Tool choice is an important part of your overall strategy for measuring an application's performance. Look for tools that can handle multiple types of performance testing, including scalability, load, soak and stress. Consider the type and complexity of your system and your test objectives, as well as the browser, platform and device combinations under test. Apache JMeter, SmartBear LoadUI Pro, Tricentis NeoLoad and Micro Focus LoadRunner are examples of full-spectrum performance testing tools. More specific scalability testing tools include System Stability Tester, FurMark and HeavyLoad.
The don'ts of scalability testing
When devising a strategy for scalability testing, there are things to avoid.
Don't underestimate the importance of scalability requirements analysis
Every performance test must begin with a thorough understanding and analysis of the requirements. Since scalability is so critical to UX, you must get those requirements right. Remember that load can be much higher than expected. Research carefully, as you need to balance the cost of planning for higher-than-expected load with the risk of failures from not planning for these loads.
Don't try to test everything
While test coverage is important, scalability testing is expensive. Testing everything isn't worth it. The important rule of thumb for collecting metrics applies here: Don't measure anything that you are not going to use.
Gerie Owen is a lead quality engineer at ZS. She is a conference presenter, author on technology and testing topics and certified Scrum master.
