A production readiness checklist for software development

Is the build ready for production?

Despite a development teams' diligent work to build and test an application, it can still be hard to know exactly when a build is ready to enter a production data center. To answer this question, the software must be thoroughly tested. Consider the following elements of a production readiness checklist.

Features and functionality. The application should do all the things it's required to do. A tenet of software testing is to validate the build against any documented requirements that, for example, a software requirements specification (SRS) outlines.

In Agile development, missing functions are common -- and perfectly normal -- as a development team will frequently release new iterations of the software. However, the software still must meet certain basic requirements before an initial deployment. For instance, customers must be able to use the application for its most fundamental purpose(s).

Bugs and bug rates. Find the faults and errors in a release candidate. It's not enough for a build to perform a feature or function -- it must do that work accurately and produce expected outputs in response to known inputs. Testers should look for a "stable" or "quality" build with error rates within acceptable or established limits. However, in certain industries like medical or finance, there may not be an acceptable error rate.

Demonstrable errors, or unacceptably high numbers of errors, can delay a deployment decision until the development team addresses those bugs. High error rates may even call the software's quality into question and trigger a more extensive review of the product's architecture and coding.

Performance. Software is frequently subject to performance requirements or metrics an SRS outlines. Metrics may include the following:

• computing rates, such as transactions per second or response time;
• infrastructure or resource usage, such as application CPU utilization;
• network metrics, such as latency or concurrent user connections; and
• other data points deemed important by the development team.

QA professionals often conduct performance testing later in the SDLC. However, such tests are critical to validate the build against any prevailing metrics before a build can be considered as a release candidate.

In addition, the enterprise must ensure that any instruments -- such as application performance management or app health tools -- are already in place to support the software when it finally reaches production.

User acceptance or experience. User acceptance testing gauges how well actual users like the software and how easily they can use it. Teams will commonly use the UX metric to estimate user satisfaction by factoring application response times, failed requests and other parameters into a calculation or score.

If consumers can't easily use the build, employees, partners or customers may not readily adopt the software -- and the project may fail. Poor acceptance may delay software release until the development team addresses and remediates the user issues which cause the problems.

Security. Security has increasingly become a high priority in testing release candidates from both a safety and compliance standpoint. Security testing often involves identifying and remediating:

• potential vulnerabilities;
• proper authorization and authentication techniques;
• data encryption (both at rest and in flight); and
• resistance to common malicious activity.

Another vital security element to consider as part of predeployment QA is penetration testing, which determines if known tools and techniques can hack a build.

Does your production readiness checklist consult the business side?

Software deployment is a go-to-market decision and as such, those decisions should also include business perspectives on the following considerations.

Support. Whether the software is a new release, a major update or a minor patch, it's important to consider supporting documentation, like knowledge base articles or even a simple change log.

Additionally, the helpdesk staff must be trained to help users understand what's new, how to use new features and how to resolve any known issues or limitations. Introducing a build without support resources in place can blindside both support staff and customers.

Stakeholder comfort. A software project involves numerous stakeholders from all areas of the business -- developers, IT staff, project managers and business executives. It needs signoff from all those leaders.

If management isn't comfortable with deployment, the relevant parties must understand why and the development team should address the problems before advancing.

Is production ready for the build?

There are still more pieces to the puzzle for production deployment than reaching specific build criteria. The operations side must also be ready to receive and introduce the build.

Today's Agile, DevOps and other continuous development paradigms have erased much of the traditional isolation between development and operations. Handoffs are far more frequent and fluid than in years past and development teams are increasingly responsible for directly handling deployment.

While it doesn't specifically matter which group handles the software deployment, it's vital to involve production considerations in any production readiness checklist -- and the development team should have a major role in this discussion on the following areas.

Infrastructure. The quintessential operations issue. The team needs to ensure that any servers, storage, networking and other resources (such as load balancers) are available to meet the software's hardware requirements. Operations staff must evaluate and discuss whether any patches and updates will shift resource requirements to any significant degree that would affect a go/no-go deployment decision.

Where the IT organization is to introduce an entirely new software instance, infrastructure considerations are most acute for new builds or builds intended for concurrent deployment models (such as A/B or blue/green deployment). Operations staff and trained development teams must provision the required resources and ensure they are available to receive the software.

Dependencies. Enterprise applications rarely operate independently and usually rely on an assortment of middleware and back-end applications such as databases. The team needs to understand all the required dependencies for the build and verify that they're met within the existing infrastructure.

If, for example, there are missing or outdated dependences -- such as a build requiring a newer OS version than what's currently in use -- the development team needs to address this concern before it delays the deployment schedule.

Process. Deployments demand a detailed installation plan for the build. It may be as simple as replacing a container instance or powering down a virtual machine and patching files. But enterprise-class software will often involve numerous components which must be configured with IP addresses, port and path designations and other details to support data traffic between components.

For example, an application that requires database access needs the details of that database server to facilitate communication. If teams want to practice complex deployments, they can build in a lab or staging environment before they attempt to install a complex build into production.

SLAs. Service-level agreements (SLAs) increasingly govern enterprise applications. Specifically, SLAs guarantee key attributes of the software's performance and availability to major constituencies, such as departments, partners and customers.

Teams need to also talk about how the IT organization will monitor and enforce SLAs once they introduce the build. In many cases, SLAs are verified through application performance monitoring tools and other software instrumentation.

The deployment plan will typically include the steps to connect and configure those tools to accommodate the new build. Additional discussion might also include any changes to monitoring or reporting.

Compliance. Most businesses are obligated to meet one or more regulatory compliance standards -- such as the Sarbanes-Oxley Act or HIPAA. In turn, the software that an organization introduces will need to support those regulatory and compliance demands. A go/no-go deployment discussion should include an assessment of prevailing compliance requirements -- like data encryption -- and a review of how those requirements are met with the new build.

Support. Trouble happens and rollouts can get rocky, even with everyone's best efforts. A deployment decision should include a support discussion that details problem reporting and ticketing, and escalation. An IT organization should train the helpdesk staff and prepare documentation to handle user problems and issues. Also, the enterprise must identify criteria to monitor and report -- such as UX -- and which relevant parties should understand the fallback and rollback plans.

How to reduce deployment risks

IT organizations and their development teams have a range of strategies in their production readiness checklist that can help mitigate the risks and anxieties of software deployment.

For example, soft launches are common. This approach, where organizations introduce a build for limited use in A/B or blue/green environments, lets proven software products or builds continue functioning concurrently with new builds. Limited users or user groups have access to the new build and can verify the software behavior before systematically cutting over and opening the new build to additional users. If trouble arises with the new build, the original software is still available.

Second, have a fallback plan in place. Such preparation can involve a means of rolling back the new build to restore a previous and proven build. Successful rollback plans depend on carefully considered criteria, such as what situation, circumstances or metrics should trigger a rollback, and when it should take place to further minimize disruptions to users and the business.

Finally, knowledge can make the difference between the success or failure of a software deployment. Knowledgeable developers and IT staff should be on hand to oversee the deployment and assist if problems arise. Teams need to make the investment in documentation and training so users understand the new software and have the instruction necessary to use the software productively.