Proper storage performance testing: A tutorial

Why storage performance testing is important

Performance testing can help organizations better understand storage systems and upgrade or modify their configuration. It can also help to compare multiple storage products. Benefits of effective performance testing include the following:

• improved overall application performance;
• reduced risks and unplanned downtime;
• more efficient troubleshooting for bottlenecks and other issues;
• reduced overall maintenance overhead;
• more informed decisions about equipment purchases;
• more effective scaling, upgrades or reconfiguration for storage systems; and
• reduced capital and operational storage costs.

Test your storage performance step by step

Despite the different possibilities, there are five basic steps that should be part of any storage performance testing process.

1. Do the preparation necessary to carry out your tests

Performance testing a storage system can be a significant undertaking. As such, it requires proper preparation to avoid wasting time and resources and to minimize disruptions.

Define your testing objectives and evaluation criteria. What determines that your tests have been successful and are complete?

Identify the key players and stakeholders. Who will carry out which tasks? Who needs to be informed of the testing process? What impact could there be on end users?

Identify, acquire and learn about the tools. You'll need these to perform your tests and report on their results.

Establish a system of documentation. Track all relevant information about the test environment, testing process, assumptions made when conducting the tests and any other important details.

Set up a test schedule. Specify when the tests should begin and how long they'll take. Build some flexibility into the schedule in case a test takes longer than expected, additional tests are warranted, or other complications arise.

Mimic your production environment. When applicable and possible, set up a testing environment and testing resources that approximate the production environment as closely as possible, including the test data and workloads. You might need to purge or pre-condition storage devices before running your tests.

2. Identify the system components that should be tested

A storage system is made up of more than just SSDs or HDDs. The system typically includes multiple layers that each have their own complexities and characteristics. Bottlenecks can occur in anything along the way: routers, host bus adapters, storage controllers, application hosts, replication servers or any of the other components.

Whether you're dealing with a SAN, NAS, hyperconverged infrastructure appliance, or other storage configuration, there are many points of possible failure, which can make performance testing complex and difficult. On the other hand, you might be concerned with only one or two components. For example, you may need to add a disk to an array and want to gauge its performance impact. In that case, the scope of your project is more limited. Whatever the scenario, determine exactly which components you'll test.

3. Determine which metrics to capture for each storage component

In most cases, you'll want to capture three metrics where appropriate.

Latency. The average time it takes for a component to complete a single data request. It is the measure of time between issuing a request and receiving a response. The lower the latency, the better the performance.

Throughput. The amount of data that passes through or originates from a component during a specific period. Throughput typically refers to the number of bits per second, such as Kbps, Mbps or Gbps. The higher the throughput, the better the performance.

IOPS. The number of I/O operations that a storage system can process each second. IOPS is concerned only with the number of read and write operations. It does not specify the amount of data included in each operation. Higher IOPS is typically better than lower IOPS, but also consider latency and throughput when you evaluate the data.

Evaluate other storage-specific metrics, such as cache usage, queue depth, I/O splitting, or capacity.

4. Begin the testing process

Do a trial run to ensure your tools work as expected, you've configured them correctly, and you can capture what you need. If you run the tests for a client, have them review the initial results to verify you tested the correct information. You'll probably need to tweak your process, but after you do, you should be ready to run your actual tests. Keep in mind a few important points:

• Perform the tests as long as necessary to ensure that you get the full picture on performance.
• Run the tests multiple times to verify their results and ensure your processes are repeatable, considering any variables that might have changed.
• Make sure your tests are large enough to ensure you get realistic results.
• Intentionally fail components in the storage system to study the impact on performance. For example, you might pull a disk out of an array or take a server offline.
• Monitor each step to ensure you see consistent results throughout the entire storage performance testing process.

Continuously verify that you collected the correct data and you can easily access that information for your final reporting.

5. Generate and disseminate the collected data

Performance testing is not complete until you've collected and aggregated the results and provided them to the right people. The results should be in a format that is readable and that users can quickly understand and act upon. A report that contains meaningful visuals, such as tables, charts and other graphics, can go a long way in providing key players with quick insight into what your performance tests revealed. Your reports should also clearly point out any bottlenecks, anomalies or other issues observed.

Monitor each step to ensure you see consistent results throughout the entire storage performance testing process.

Your reports should include basic information about the testing process itself, such as the components you tested, the types of tests you performed, the tools you used to perform the tests, and who conducted them and when.

After the key stakeholders review the results, they can then address the situation. They might purchase one system over another, upgrade the storage network, add an application server or replace storage devices.

Choose the right tools for testing

The tools for performance tests will depend on the type of storage system, the components within that system, the vendors that built the components and the team's level of experience. Tools should meet specific requirements. To this end, there are several questions to ask about any storage performance testing tool:

• Does the tool make performance testing easier or more difficult?
• Does it support the organization's particular storage systems, type of storage and underlying platforms and infrastructure?
• Does it support the operating environments?
• Does it capture the metric data the organization must track?
• Does it include the features and configuration settings to meet current requirements and those in the foreseeable future?
• Does it add overheard to the storage system being tested or require additional equipment to run?
• Can the organization customize it to meet specific requirements?
• Can it integrate easily with third-party systems and services?
• What are the licensing fees, and what support do they include?

Verify whether the equipment vendor offers storage performance testing or monitoring tools that meet requirements. Some platforms, such as Windows Server, include built-in tools for tracking performance. Check out open-source tools available for testing performance. For example, Flexible I/O is a popular and highly tunable I/O tester that's available on GitHub.