Why to use staging environments for IT infrastructure testing

What is a staging environment?

A staging environment is an isolated copy of production that serves as the last testing ground before actual production. The more closely a staging environment reflects production, the more reliable the test.

Compared to physical services, virtualization and public cloud have streamlined the creation of a staging environment. With virtualization, for example, admins can clone VMs to create copies of production servers. On the public cloud, the same automation tools that deploy the production environment can deploy an identical environment for staging. The ability for teams to spin up and destroy staging environments on demand ensures changes are tested before deployment to production. However, network isolation between production and staging environments is crucial, as production users should never access the latter.

The scale of the staging environment depends on the change admins will test; only place IT resources in a staging environment if they're associated with the components being tested. If operations staff changes only one database server, for example, a complete copy of all production resources is likely unnecessary. However, to test an OS upgrade on the core infrastructure, consider putting all resources into the staging environment for a complete test. A thorough application dependency map can dictate what to include in staging.

Staging ensures that changes do not break production applications. A good staging environment relies on automation, rather than a group of users, to test applications. Automated testing is essential for full validation that applications will continue to work after a change is made. Consult an application dependency map to determine which applications to test for a particular change.

Use automated testing to validate that a newly created staging environment is functional before making any change, then use the same automated tests after the change. Infrastructure changes should not change application behavior; the results of the post-change tests should mirror those of the pre-change tests.

Operating system patches, and the validation of new monitoring and management tools, are common examples of IT infrastructure testing in staging environments.

Prepare for patches and upgrades

Patching and upgrading OSes is a classic IT operations requirement -- and a common source of unexpected application downtime. To assess any effects on production, perform tests in staging.

Staging ensures that changes do not break production applications.

For example, test the monthly patching cycle in staging before production deployment, and ensure it passes the automated tests. Patch tests should be a routine and relatively quick activity in staging. However, tests for OS version upgrades might be more complex and time-consuming. The staging environment is a great place to work out the upgrade process, from the correct order and timing of steps to the duration and scale of any required service outage. 

Defining this process for a major version upgrade, such as Windows Server 2016 to 2019, might require several attempts. The staging environment -- not production -- is the right place for this experimentation. IT admins can destroy and recreate the staging environment as required to find the right upgrade path, without any effect on production. Testing in staging also reveals pertinent information, such as the timing and effect of the upgrade process, in advance.

Validate new infrastructure tools

Admins can also use a staging environment to perform IT infrastructure testing related to the use of new tools and methodologies. For example, use staging environments to evaluate a new system monitoring product and the effects of its agents, or to assess a new process for server deployment. A proper staging environment is essential to prove how these new tools and practices will work in an IT environment. Positive tests in staging suggest a positive outcome in production.

These evaluations in staging are also an excellent way to demonstrate the value of new systems and processes to both IT and business leaders.