Guide to creating a cloud migration testing strategy

Why is cloud migration testing important?

Cloud migration testing helps IT teams ensure the application continues to perform as it should after moving to the cloud and also ensure that the user experience is better. To do this, they must gauge the app's performance on both sides of the equation: how it ran on-premises and how it works in the cloud.

Metrics from cloud migration tests help identify and quantify any issues rather than relying on guesswork. Key metrics include application start times and response times, performance during peak demand and off-hours, and usability on various platforms, such as desktops, remote connections and mobile devices.

Also, cloud vendors issue updates that can cause performance issues in an application. Use your testing metrics to validate the issues, and work with your cloud provider to find out what happened and what adjustments you have to make to correct the issues.

Finally, cloud migration testing can tell IT teams where to adjust performance or UX to justify keeping the application in the cloud.

Business benefits of cloud migration testing

Besides providing users with a well-performing, modern application, a solid testing program can deliver other advantages to the business, including the following:

• Reduced chance of service disruptions.
• Higher customer satisfaction from smooth transitions.
• Less stress and wear on staff.
• Net cost savings after the upfront expense of the testing itself.

While a reduction in issues is a real benefit of testing, it is often hard to quantify because issues can occur without additional testing. The odds are better with testing, however.

How cloud migration testing differs from traditional app testing

The main difference between testing applications on-premises versus apps that have moved to the cloud is that you have to accommodate the cloud's scalability and additional integrations and dependencies. Some of the integrations might be difficult to identify and understand. Your cloud testing framework might differ from an on-premises framework, and some tools might be different or not allowed, such as for load testing or penetration testing. It's important to check with your cloud vendor about what testing tools are permitted.

Keep in mind, though, that the goals of testing should not change between on-premises and the cloud. Having different goals means you're no longer testing apples to apples, so you should use many of the same steps and tools to get a clear picture of your efforts to migrate your application to the cloud. Changes to the testing framework or methodologies can skew results.

Types of testing for a cloud migration

The goal with cloud migration testing isn't to test every possible feature and function, but you do need a solid cross section of tests and metrics to ensure your application performs as expected. Several types of tests are especially important for a cloud migration:

• Functional validation. This test confirms that the build meets requirements for component and service functionality on both sides of the cloud migration effort. Anything missing tends to jump out quickly.
• Performance. This test spans several metrics to measure how an application performs in real-world conditions, e.g., ability to handle data volumes, capacity loads, and CPU and memory usage. It also includes the customer experience to ensure better or at least consistent performance for users.
• Integration. Your application likely ties into other services and apps to share data in the cloud and on-premises -- and possibly both. IT teams must verify these connections still work in the cloud. This can be tricky if dependent services are in a different cloud or remain on-premises. You also need to understand and document what happens when a service goes down at each location to understand the impact it will have on customers.

Other aspects to test

If the application has a dedicated portal or requires interface tweaks to run on mobile devices, test it on both Android and Apple platforms. Testing every possible device type might not be feasible, so work with your user base to determine which mobile platforms best meet their needs.

Also, test on both cellular and Wi-Fi networks because different data speeds impact an app's behavior. Do this outside of your company's Wi-Fi network. The internal side of your network might access services differently than an external Wi-Fi network, which could cause the application to work differently or not at all.

Finally, test for printability. People do still print, and for some, it's a critical job function. Don't underestimate this potentially challenging technical problem. Printing from a cloud-based application to a local printer can bring security and network challenges that you typically don't have on-premises.

Security considerations for cloud migration testing

You should perform a proper application security audit and make adjustments to user privileges before you begin to move an application from on-premises to the cloud. The last thing you want is to introduce additional changes that muddle or complicate your understanding of how an app works and its dependencies.

The cloud migration testing should include a security validation that confirms the following three key factors:

1. Who has access to the app? Start with the accounts, features and functions that users access. Adding more variables doesn't help or speed up the testing. Users' access should mirror what they had with the app on-premises, no more and no less.
2. How have you ensured the principle of least privilege? As your data moves between different services in the cloud, what steps ensure the principle of least privilege? Cloud vendors offer services and reports to help with this; take the time to learn, document and use them.
3. Do security changes impact the app? Always rerun application testing after you make a security change, as such changes can potentially affect performance. For example, does data encryption at flight and at rest change application response time or add overhead to CPU cycles? This can be a subset of full testing, but you must revalidate that the application still works as baselined and that the security change doesn't take the application offline.

7 common challenges in cloud migration testing -- and best practices to address them

Application testing is always a challenge, and despite all the planning and testing, you will encounter issues. Nevertheless, most problems are solvable. Here's what you're likely to face and ways to address them.

Challenge 1: Prioritizing app testing metrics

An application's UX and security are important to the overall application, but ultimately, the application must work as intended. An application that lacks core functionality, even if it's absolutely secure or wonderfully streamlined for users, does no one any good.

Best practice: Balance design and security with critical functionality

Take the time to address all the factors -- functionality, UX and security -- early in the cloud migration process. They are not separate aspects you can work on at different stages. They're interconnected, and you have to treat them as such.

Challenge 2: Cutting corners just to make the app run

It's critical to make sure the application works properly in the cloud, but don't adopt bad habits to achieve short-term goals. IT teams might reduce or remove security controls to get an app functional in a new environment and forget to restore those safeguards afterward. That may save time initially, but it creates more work later and makes testing more difficult overall.

Best practice: Plan security from the start

Security permission issues often arise and can impede an app's functionality, but resist the knee-jerk reaction to make large-scale changes to get things running right away. Build time into your cloud migration plan to address these problems.

Challenge 3: Leaving the testing entirely up to IT

Cloud migrations are complex efforts that require many disciplines to ensure a migrated app works securely and at scale. IT has a major responsibility and can help lead these efforts, but you can't just forklift and focus on the technical side without a development team's expertise.

Best practice: Form a diverse testing team

Set up a cross-functional team to provide app support and identify possible issues. The team must be nimble, and you need subject matter experts to ensure all aspects are being looked at. Set aside certain items in meetings so you don't overburden everyone with too much information. Some attendees might get bored and tune out technobabble that's outside their expertise.

Challenge 4: Resisting easy updates and scope creep

Enabling encryption for an app that's migrating to the cloud might seem ideal and deceptively simple. But beware: Any change can unknowingly impact many other aspects of the app's operation and increase technical concerns and costs. This kind of scope or feature creep in migration testing can quickly grow to unsustainable levels.

Best practice: Test and vet additional functions

First, try to stick to the original plan for the application's core functionality; the testing phase isn't the time or place for feature expansion. If you must add any minor new features or functions, get everyone's buy-in, and test and vet the changes to understand their broader impacts. Be able to say no to items. It might seem easy in concept but can be quite difficult to say to some people, especially managers.

Challenge 5: Balancing testing cadence against competing priorities

Ideally, you should perform continual security testing and at least weekly testing to validate data and handle error logs. Unfortunately, money, time and shifting priorities of IT and management can overrule those plans. The cloud costs money for every second it's in use. Staff require time and money, and after an app's release, most ops and dev teams move on to other projects.

Best practice: Plan around major updates and push for more

IT always finds ways to fit what is required into what's available. Break down cloud migration testing into what management wants, needs and expects. Set up a testing framework for daily and weekly checks around major changes to the base application or platform. Push for more regularity if more resources are available.

Challenge 6: Narrowing UX testing

Some IT teams rely on a regular pool of users for on-premises testing or a few who are experts in the specific software. However, when you move an application to the cloud, you need a more comprehensive picture of it. That means gathering data about overall UX from multiple perspectives and not just technical ones.

Best practice: Expand user test teams

Coordinate user testing with a wide pool of testers from different experiences and roles. Include people who are cloud-savvy but also novices. Most users don't know or care about the cloud's inner machinery, but they see the application and have certain performance expectations. And, sometimes, they might know it a little better than you do.

Challenge 7: Testing metrics that overlook cloud cost concerns

In an on-premises migration, typically, you own all the pieces the application touches. In a cloud migration, you incur costs for every process step, every service called and every data transfer to a cache or back on-premises. There are integrations and dependencies to other cloud services -- some of which might not be obvious. You might be using cloud-based testing tools. When the bill comes due, it might come as a shock and force you to adjust the app or even reevaluate the cloud migration.

Best practice: Regularly test to reveal cost trends

Cost management isn't a central concern of cloud migration testing, but testing reveals exactly what an app does when it runs in the cloud. Thus, frequent cloud migration testing can help an organization get a better feel for data transfers, usage patterns and key dependencies. That means no surprises when the bill arrives.

Creating a cloud migration testing strategy

When it comes to checklists and step-by-step procedures, people often focus on the things they want and skip over the things they don't like, or they assume following the steps is easy. This approach never ends well, so it's important to understand the impact and challenges of the following steps:

1. Assess. This is an easy one to gloss over by saying it's as simple as taking the infrastructure an application is running on and plopping it in the cloud. Local resources, such as big hard drives and massive CPUs, are fairly cheap in a local data center. That math changes in the cloud because you pay per month for everything, so you have to be clear about what you need -- not what you already have.
2. Map. It takes one simple firewall rule to disable an entire application stack or a single data share that's not configured correctly. It's crucial to remember that everything talks to each other and to know where you need additional resources to ensure the map is accurate.
3. Select. Remember that not everything is meant for the cloud. Be honest about whether an application is not the right fit or is too expensive to migrate. And, when it comes to choosing the right cloud, everything should be on the table. Don't assume you have the right answer going in.
4. Develop. Make a plan, and follow it. It helps keep the team focused and prevents project creep. Celebrate milestones when you reach them, and make sure users and managers are aware of the progress. Keep things transparent.
5. Create. This is the step people often want to skip to, but nothing should be spun up until the first steps are completed. That's because, when you skip to the fun part, you often end up redoing it because you discover you missed something that would have been identified in the planning.
6. Replicate. Don't break what is already working, and don't introduce unplanned changes to the application just because you have a chance since it hasn't gone live yet.
7. Stage. Make sure a wide range of users and devices are involved in testing the application. It's often the casual user who finds a common issue, not the power user who simply works around the issue or doesn't even notice that there's a problem.
8. Check. Technically, this is the easiest step because you have been doing security all along and can say it is already done. If you wait until the end to tighten security, you must reverify that the application still works.

Key risks of cloud migration without testing

Testing is a difficult thing in the best scenario, and in the high-priced world of IT, it often loses out to luck or hopeful ambition. This tends to happen when management sees IT as a cost center and not an asset. From management's standpoint, the cloud is a big box they can't see into or understand, and it's just supposed to work as well as it does delivering music and entertainment to their smartphones.

This lack of understanding is what IT project managers must counteract by communicating to stakeholders the gravity of what could happen without good cloud migration testing.

The benefits and risks can be summarized like this. Successful testing results in the following:

• Confident staff. Seeing a successful test engenders confidence in the tools, training and time frames -- all key elements of a successful migration.
• Cost savings. Investing upfront in testing can save untold dollars on the back end.
• Minimized outages. Planned outages are things users understand, can plan for and work with. It's the unplanned ones that get people upset.

In contrast, poor or no cloud migration testing can lead to the following:

• Outages. Feeling the rage of users complaining to management doesn't fix the issue faster and is a particular risk with unplanned outages.
• Need to call in consultants. When you run into issues you can't immediately solve, management demands fixes ASAP. Hiring consultants becomes the only choice. The consultants know this and raise their prices accordingly.
• Staff burnout. You need people to be at the top of their game for a migration because it isn't over in a day. Minor issues inevitably come up, and getting off on the wrong foot makes it next to impossible to right the ship.

Cloud migration testing is optional, and there is a cost to doing it. The question becomes how much risk the organization is willing to take on by skipping it and whether it can handle the impact if a major outage is the result.

Brian Kirsch, an IT architect and Milwaukee Area Technical College instructor, has been in IT for 30 years and holds multiple certifications.