How to craft a cloud testing strategy

What is the cloud?

Cloud computing is a service model in which organizations consume compute resources on demand, as needed. The delivery of these resources occurs over the internet or via internal networks. The cloud offers many advantages, including lower costs and extensive scalability. With the cloud, organizations can eliminate upfront infrastructure costs and pay only for what they use. Cloud environments can be built quickly.

Cloud architecture is based on the as a service delivery model, in which organizations pay to use computing assets on demand, rather than purchasing and maintaining these assets themselves.

There are three primary types of as a service cloud:

• Infrastructure as a service (IaaS). Hardware resources including servers, network and storage devices are managed by a third-party provider. Users share these resources, but are unaware of the other tenants. IaaS providers also offer management services, though the user cannot make changes to the infrastructure itself.
• Platform as a service (PaaS). The cloud provider manages the complete runtime environment, but not the applications and data. An organization uses PaaS to establish environments that support application development and delivery, without the need to manage the underlying resources.
• Software as a service (SaaS). SaaS is software that the cloud provider hosts, runs and manages. Organizations purchase licenses to use SaaS-based products, and might be able to make customizations, but do not support the software.

A public cloud provides services to everyone. Customers request cloud services or resources and pay per usage. AWS, Microsoft Azure and Google Cloud are public cloud providers, and each offers a range of services.

A private cloud runs in a data center that is owned by an individual company, or is otherwise closed off from other tenants. It is shared only by users who have access to the organization's firewall.

A hybrid cloud is a mix of public and private clouds. Hybrid clouds enable an organization to use the public cloud as needed while maintaining some services within the organization.

Dev, test and production code, or some combination of these three, could exist in the cloud. A cloud testing strategy often refers to evaluating the infrastructure, platforms and software packages provided through the service delivery model. Conversely, cloud services enable software QA teams to perform application testing more quickly, efficiently and effectively.

Testing cloud resources

A cloud test strategy might refer to an assessment of cloud architecture and resources that support an organization's business efforts. Thus, this strategy involves testing a cloud provider's IaaS or PaaS offering to ensure it works as expected.

QA professionals can apply functional tests to cloud-based software. Options include acceptance, system verification and interoperability tests. But, to ensure an app will work on the cloud, do nonfunctional tests. IT professionals such as testers, ops, data scientists and cloud architects all take on responsibility in non-functional cloud testing. Some common IaaS and PaaS nonfunctional tests include scalability testing, disaster recovery testing and data testing.

Cloud scalability testing. Scalability is the ability for a system to use more resources to handle an increased workload, and also decrease resources in use as demand falls. Scalability is a distinct advantage of cloud computing over other methods of application hosting. Virtualization enables systems to increase size and power, taking more capacity from other infrastructure not in use. Cloud resources can scale within minutes -- or even seconds. Thus, scalability is one of the main reasons that organizations choose a cloud strategy for an application.

Scalability testing is a type of nonfunctional test that measures the performance of the application or network as the number of users increases. Organizations test cloud scalability to ensure that a sudden need for more cloud resources can be met successfully.

In cloud computing, scalability can be approached in three ways:

• vertically, adding more power to the existing server;
• horizontally, increasing the number of servers allocated to the instance; or
• diagonally, a combination of both approaches.

Testing horizontal and diagonal scaling requires comprehensive and complex test designs. Whatever the architecture, testing should begin early in development and be executed often as the system matures and load increases. These tests help catch potential scalability issues early and mitigate them. Customize scalability test designs based on the application and its architectures.

There are several criteria to measure in any scalability test approach, including:

• response time
• failure rate
• throughput
• memory
• CPU usage
• network usage

Be sure to test loads that increase well over the highest expected real-world demand, and monitor the level of performance as the load increases.

Disaster recovery testing. One benefit of IaaS and PaaS is the ability to spin up additional environments to host applications when disaster strikes. Cloud disaster recovery helps organizations recover from outages or disruptions caused by power failure, unethical hacking, weather emergencies or other issues. All organizations should establish a disaster recovery plan, and then execute disaster recovery tests to ensure its effectiveness.

A disaster recovery testing approach depends on the organization's cloud strategy. Public cloud customers should work with the provider to develop and test a disaster recovery plan. Most vendors make it easy to back up cloud workloads and spin them up on resources in other regions. The public cloud provider might offer a disaster recovery service. AWS, Microsoft and Google all have cloud offerings to facilitate data recovery.

Hybrid or private cloud adopters might opt for a disaster recovery as a service provider or create their own infrastructure environment and plan. In the cloud, new environments can spin up via virtualization, so they don't require the organization to invest in duplicate infrastructure, unless you need to create a new backup location. Test all data recovery plans not only to ensure that they work technologically, but also to make sure the team members involved execute the plans correctly.

There are several ways to test disaster recovery. Paper reviews validate a plan, but don't offer hands-on experience. Instead, conduct parallel tests, in which the new environment runs along with production. This parallel testing approach illuminates any flaws in the plan or application design, and also provides a practical learning experience for the team. The ability to set up and pay for cloud resources only when needed makes this test approach both practical and efficient.

Cloud data testing. Data poses some unique testing challenges in cloud applications. Testers must understand how data migrates in the cloud and plan to evaluate both statically and dynamically migrated data.

Data privacy and security are critical components of a cloud test strategy, as all data, including nonpublic information, transmits via and is stored in the cloud. QA professionals must validate the security of the transmission channel, as well as the cloud tenant, in the test framework.

Testing cloud-native software

As another cloud testing strategy, QA professionals can assess software that resides in the cloud. SaaS means different things to different companies. Some SaaS products are designed for a large base of customers, such as Salesforce, SAP and Workday. Other SaaS offerings are for an internal audience, which would require a different strategy. Or, a QA professional might test a SaaS vendor's product from the client's perspective, which is yet another approach. Consider the unique challenges of each approach before you design the cloud test plan.

Nonfunctional testing for SaaS products includes many different components. QA professionals must apply performance, load and stress tests. They must also assess availability, reliability and scalability. Security testing is probably the most complex nonfunctional test, as multi-tenancy requires in-depth testing of roles and permissions. Multi-tenancy provides the basis of SaaS architecture, where the same code is deployed to multiple concurrent users. Thus, the test must ensure one customer's product operates in isolation from other tenants, as they all run the same code. Connectivity and API testing are also important elements of SaaS testing.

Many SaaS apps have components that cross multiple types of clouds, including public, private and hybrid. In this scenario, testing takes place on a hybrid cloud infrastructure. This testing approach is valuable when there are many legacy systems and a robust end-to-end test is needed.

But testing isn't solely about technical functionality, as the cloud opens up more business cases to consider. As with most testing, the challenges begin with requirements. SaaS vendors create a product to meet the needs of many customers. The purchasing organization must engage in a gap analysis to determine how to fit its business processes into that software. This process creates a challenge for both business analysts and for testers who must apply the gap analysis to QA. The core product code of the SaaS application is the same for anyone who purchases it, so the cloud testing strategy changes from functional testing to business process testing.

The SaaS provider, or an in-house development team at the purchaser, customizes and configures the product to meet needs, and testers must adapt. Customizations refer to code changes made to the core product, whereas configurations are options selected within the product. QA engineers must test both, but it is especially important to do regression testing around any customizations. Most SaaS products require integration with multiple downstream legacy systems. Full end-to-end systems' testing is a critical component of the cloud testing strategy.

Finally, the SaaS vendor controls the frequency of patches and upgrades and the window of time organizations have to test. The SaaS testing strategy must account for short iterations. Agile methodologies tend to work best in this environment.

While it's a complex endeavor, SaaS testing comes with its advantages. Testers no longer need to include multiversion testing or backward compatibility in their cloud test plans, and there are no infrastructure installations to test, because the SaaS vendor manages software hosting.

Testing via cloud tools

A cloud testing strategy might also refer to the use of cloud resources or tools to augment a QA strategy. Cloud environments are useful for testing web-based software. The organization uploads an application to a cloud environment, where it can perform large and real-world simulations.

Testing with cloud-based tools offers many advantages, especially in the area of environment management. Testers might struggle to get the QA environments they need because of limited availability of infrastructure on premises. Multiple test teams often share environments to avoid consuming too many resources for testing. As a result, nonfunctional testing may occur in the same environment as functional testing. Shared environments delay projects as the test teams wait their turn. Also, testing environments often don't mirror production environments, leading to missed bugs, especially in load testing and performance testing.

Testing via a cloud-based setup can remedy these test environment management issues. It's quick and easy to provision test environments. Anytime availability eliminates the need to share environments among test teams. Cloud test environments can scale to mirror production for more accurate nonfunctional testing. QA environments and the provisioning thereof are owned by the vendor, which ensures standardization. Finally, the as-a-service model reduces cost.

Cloud resources and tools can open up other dev and QA possibilities that might not be available on premises. While limitations vary depending on the organization, some possibilities might include distributed version control for dev or test repositories and resources that enable more comprehensive test automation.