SaaS application testing: Keep performance tests in mind
Performance testing must be a high priority for SaaS application testers. Learn why and pick up tips on how to test a SaaS system's performance.
For providers of Software as a Service (SaaS), performance is critical. Even so, many providers skip SaaS performance testing due to a lack of time, lack of understanding or because the specifications are never determined. As test-team members, we frequently assume development is going to handle performance within their unit tests or as part of the code design. I've found it frequently fails higher up. Let's take a look at the basics of performance tests as a part of SaaS application testing, why performance testing is critical and what type of testing your team can do to find performance flaws before your customers do.
Think about performance early
Amy Reichert
For many SaaS-based applications, performance doesn't get attention from product management until customers start reporting issues. Performance and scalability should be written into design requirements up front, before any services falter and die. It should not take a negative business event for application performance to get attention. Customers deserve better. Software development organizations cannot continue to use their customers to find performance bugs.
SaaS performance testing specifications need to be included in requirements and design planning. The specs must include details such as timings expected from each Web service component, each customer or tenant in the SaaS system and the responding application. It must be broken down to the component level based on the complexity of the workflow provided and the number of participating customers.
For example, I once received performance requirements that read exactly: "System shall respond with exceptional performance." Seriously. What part of the system? Does this cover the whole workflow or just one pathway? What does exceptional mean here? These requirements are too short and too vague, which makes them difficult to deliver on.
Performance testing is a mindset
An effective SaaS application testing plan needs to include performance testing at each level within the SaaS structure because SaaS-based applications deliver Web services, data or access to applications via the Internet. SaaS applications are generally designed to allow multiple customers to utilize a shared database model. If done well, they experience increasing levels of throughput and therefore need to be able to handle demand increases and surges in order to remain reliable for customers or other end users. Basically, there are multiple points of failure for SaaS, and the services need to remain highly responsive and reliable.
Slow services cause dissatisfaction, as well as possible violations of service level agreements (SLAs) between providers and customers. In other words, the company that provides slow services loses business and reputation value at the same time. It's imperative that SaaS performance testing becomes a mindset within cloud software development organizations. It may not be the only important item, or even the most critical, but performance is right there at the top of the priorities list. By making performance a part of the mindset of the development lifecycle, testing it becomes second nature to product designers, developers and testers. Rather than being a separate effort, SaaS performance testing should be assimilated into the general development process.
Performance testing techniques
Performance testing, like security testing, is a specialized field. In order to do it fully and properly, application development organizations may need to enlist professional services or create a team that focuses solely on performance test execution. However, project managers can use the following techniques to add performance testing into the development process and gain positive benefits.
The first thing to consider is the degree of complexity involved in the SaaS system. How many customers or tenants does it serve? Are business agreements increasing? Are current users processing more requests? In order to find the answers, software performance testers need to map out the system into workflows. It's the most common customer workflows that need to be tested first. When assessing complexity, be sure to consider how to test individual components, such as each Web service both consumed and exchanged. Each piece needs to be performance-tested along with the workflow.
It's also up to testers to determine the throughput expected in the workflow. Throughput is the number of transactions produced over time -- or the capacity a system can handle. Test for throughput by loading multiple users to make requests at the same time and then continue performing the same transactions at a steady pace. Continue adding users and transactions until either the system starts to falter or the application starts to lag or show latency. This process should show the upper boundary value the system can handle effectively. This value can become the baseline for comparing future throughput results to see if they improve or worsen.
Simulate a surge of users or transactions to do stress testing. By stressing the system with load tests, the team can determine the application's ability to handle unsteady loads and find the maximum supported levels. From there, the team can move on to endurance testing which tests the system under continuously high load. Endurance testing is a method for detecting memory leaks and identifying at what point performance degradation occurs. It can also show how the system copes under high demand for long periods of time. The nice thing about these techniques is that each can be simulated either manually with multiple users or instances, or by using automated testing tools. The choice of manual testing or automated tests is really up to the team and the complexity of the system they're testing.
SaaS application testing, with a focus on ensuring performance, is imperative to a SaaS provider's success. Poorly performing systems are no longer acceptable and rarely tolerated. Gone are the days when software development companies use customers to test their system performance. Customers are more demanding, have a nearly endless choice of vendors and are usually equipped with contractual SLAs. It's imperative that software development organizations stop skipping the "extra" testing, and include performance testing at all levels within their software development cycle. Product managers, developers and testers must build performance testing into their normal design routine so it becomes nearly second nature.
What questions do you have about software performance management, testing SaaS applications, mobile application project management, or any other software quality topics? Let us know and we'll find you the expert answers you're looking for.
