A guide to testing in DevOps and key strategies, practices

Role of testing in DevOps

In DevOps, the role of a tester -- and testing -- is different than in other software development approaches. Testing is incorporated into every stage of a CI/CD pipeline, which can make traditional software test practices seem almost out of place. Many DevOps pros aren't sure how to incorporate testing, and others don't see testing as important because the DevOps process enables them to quickly fix and redeploy software.

That attitude needs to change.

A CI/CD pipeline allows for so much solid testing in a DevOps environment. Put another way, DevOps teams must deliver working software that can withstand actual production use and testing in production. Testers have a key role to play in how the software gets to that place. DevOps practitioners need to strategize how to test both the CI/CD tool pipeline and the software that uses that infrastructure.

At first, DevOps can feel like a whole new language and workflow for testers, with a steep learning curve to boot. Testers will need to accept the challenge of new approaches to development and delivery to understand and contribute to the DevOps workflow.

As DevOps is about operations and development working more closely together, that by extension includes testing. As teams shift to DevOps, though, they may not think much about testing. Testers may have to carve out their own roles, and it may be different in every IT organization. Testers will need to insert themselves into DevOps workflows. That means looking for opportunities to implement testing not only into development and integration, but also in production.

To get more specific, testers on DevOps teams may find themselves:

• acting as QA consultants;
• crafting testing standards and objectives;
• being test engineers;
• setting up test automation environments and tests within a CI/CD workflow; and
• figuring out with developers what to test and what test techniques to employ.

Types of DevOps testing tools

DevOps requires more tool functionality than just about any other SDLC methodology. After all, DevOps is based on CI/CD, and a code delivery pipeline with automation at every stage introduces steps and processes that require tooling. For instance, a development team needs tools for version control, integration, testing, release management, deployment and production monitoring with orchestration.

Automated feedback is critical to DevOps, which makes infrastructure monitoring, application monitoring and aggregators crucial components in a DevOps testing toolchain.

A DevOps team needs tools capable of functional automation, nonfunctional testing and mobile testing to perform continuous testing. The process also demands tools that can spin up environments, emulate the behavior of applications and run multiple applications on a single OS.

Service virtualization tools emulate application behaviors under test so integration testing can take place without all available components. Containers enable multiple applications to run simultaneously without the need for separate virtual machines. Also, a DevOps team may want tools that can deploy and manage containerized applications.

However, there is no definitive "best" DevOps testing tool. Instead, familiarize yourself with DevOps testing tools for each of these functions.

Role of test automation in DevOps

As testing struggled with speed and efficiency, testers in recent years have increasingly turned to test automation. For our purposes, let's define test automation as the ability to record tests and play them repeatedly during the development and delivery process, with little or no tester intervention. For DevOps teams, test automation represents a potential means for faster, more complete and more consistent testing.

Yet automation doesn't address the fact that someone must still design and record the tests. Further, while automated tests are essential to the smoke testing of new builds, no matter how often you build, you'll rarely run your entire test suite as a part of smoke testing. Many tests won't need to be run more than once or twice.

More importantly, test automation for unit and integration usually looks at the UI controls and responses. In the vast majority of applications, testing is much more than that. Testers typically also look at APIs, end-to-end tests, unit tests and integrations that are routinely run as a part of deployment. DevOps test automation has typically struggled to get under the hood of an application.

Finally, automation can lull teams into a false sense of security. DevOps teams that look at the data from automated testing runs may not accurately represent the project's overall health, quality or readiness. Testers and domain experts must supplement that information by working with the product itself.

None of this means that test automation doesn't have an important role in DevOps testing practices. Automation will improve speed and create reliability in testing, but it's not the all-consuming sea change that many envisioned when test automation first started to proliferate.

For instance, automation can accelerate many aspects of testing -- but not without cost. The upfront and maintenance associated with automated test scripts can be significant, and the results are less than ideal. And, the practice doesn't look at every aspect that human testers do.

That's why there's an important role for domain expertise, even if those people aren't professional testers. Domain experts stress an application in ways that no one on the DevOps team could imagine.

Prerequisites of continuous testing in DevOps

Testers are the champions of quality throughout the DevOps process, because managing software quality requires much more than automated tasks. Within this framework, communication skills are critical to effective DevOps testing. Testers need to share information and collaborate with the entire DevOps team -- including developers, operations, security experts and business personnel.

It takes an all-encompassing cultural shift to achieve continuous testing. All team members must understand and embrace responsibility for quality, but testers -- the champions of quality -- should lead this transformation. Rely not on technical skills to encourage continuous testing in DevOps, but rather on leadership skills -- specifically communication, collaboration and creativity.

DevOps calls for communication up, down and across the company. Testers collaborate with multiple, diverse groups to achieve organizational goals and take a visible role in QA. They also foster alignment with business objectives and goals. Testers must be creative and innovative to resolve problems and assess application quality. These are power skills for testers, and they can be the key to continuous testing in DevOps.

Talk to me. Communication is difficult. What we say and how another person interprets that can be completely different.

The best way to be visible and provide a clear, consistent message to all stakeholders is by talking directly to them. Testers need to lead by example: Put away electronic devices during meetings, and take time for conversation, especially for offshore team members.

Work with me. Have you ever been in a meeting where someone suggests an idea and others build on it? That's the power of collaboration. For testers to lead, we must be visible to all stakeholders -- not just our own teams. Through collaboration, goals align. Scope creep, which can throw off deadlines and expectations, usually results from the misalignment of objectives. To meet the rapid iteration schedules of DevOps and Agile organizations, ensure everyone works under the same scope.

Transparent metrics are a way to increase testers' visibility and align scope. With data coming in from many disparate tracking applications, transparent metrics are difficult to create. However, with a focus on these metrics, you can enable continuous testing.

Create with me. Collaboration can foster creativity and innovation, but it isn't easy to achieve.

Usually, when a team meets to discuss a problem, members look at how they handled similar situations in the past. We wrongly segregate ideas and limit our ability to consider a new approach. Within every group, there will be people with dissimilar interests and expertise. This diversity increases the intersection of ideas, which is where true innovation originates.

Think with me. By understanding the mindsets and biases of team members -- as well as ourselves -- we can begin to foster a cultural shift toward continuous testing in DevOps, where everyone on the team is responsible for quality. Focus on demonstrating exemplary communication, collaboration and creativity to champion quality processes; the technical test process will be the easy part of the transformation.

Benefits of chaos engineering in DevOps

Chaos engineering is the process of looking for weaknesses within an application by continually subjecting it to random behavior, where you essentially sabotage your own application in production to see what it can withstand. Testing prior to production doesn't give insight into the quality of an application.

Chaos engineering also means more than testing an application's quality. It looks at the entire production environment, including data center complications, OS instances, hardware, load balancing, network outages and DNS. Testing in production enables an organization to shift right, which is an important DevOps practice. The technique is one way for testers on DevOps teams to carve out a role for themselves.

Chaos engineering, however, concerns experienced testers. After all, chaos engineering experiments try to make applications fail. For example, this approach prompts a DevOps tester to bring down instances in a cluster, simulate a denial-of-service attack or create a network outage. However, breaking an application like this is also important with traditional testing.

The second concern is the idea of testing in production. For years, testers worked in a staging environment. But with today's cloud-hosted applications, it's possible to replicate the production environment only in production. If testers control their chaos process, then they should expect rapid diagnosis and recovery periods.

Uses of ChatOps in DevOps

ChatOps is an approach to using various communication channels -- including ticket management systems and collaboration management tools such as Slack and Twist -- to establish a framework for a cohesive collaboration environment. This simple name doesn't do justice to a set of technologies that are geared toward rapidly resolving production issues through collaboration.

Consider the typical pattern in an unexpected application outage or performance problem. An alert will go to IT or operations staff, and that leads to a quick fix. If it's a complicated application or network problem, the trouble will need to be escalated, but to whom? The senior IT tech? The security person? A developer? The team approach works best here.

ChatOps is the key to that team approach. The issue is escalated -- not to an individual, but to a defined group. Ideally, one member of the group knows how to diagnose and resolve the problem. But more often, it requires several people working together to understand and resolve the problem. We share our knowledge of the application, the infrastructure and the code to reach a fast and (hopefully) accurate diagnosis and fix.

Second, ChatOps allows for the use of bots or other automated tools to work with servers or the application remotely. These bots can reboot the server, change some configuration settings or diagnose the problem more deeply.

With ChatOps, the whole is greater than the sum of its parts. People with different skill sets can collaborate in real time to resolve problems quickly.

ChatOps tools. Products such as Slack and Microsoft Teams allow users to easily form groups and create communications channels to help facilitate planned or ad hoc group communications. With these group IM tools, teams can quickly communicate, exchange images and files.

For managing incidents and escalations, applications such as Splunk On-Call and PagerDuty provide a routing mechanism for incident management. These apps can determine the priority of a given alert type and make sure the alert reaches those best equipped to resolve it. These tools integrate with IM systems and alerting engines to provide a fully automated approach to receiving, routing, communicating and resolving application issues.

Role of testers in ChatOps. In many cases, testers have relevant knowledge. The problem may be one they've seen during prior testing, and they can quickly search the testing database to determine what the fix was.

Testers are important contributors to a team approach to problem solving. They are highly focused problem solvers adept at taking application behavior and deducing the root cause of that behavior. While testers rarely fix a problem, they are often the first ones to identify a solution.

Astute testers should be on ChatOps teams to contribute application-specific and problem-solving experience. They join with developers, network specialists, systems administrators, security experts and others who hold responsibility for ensuring that an application is developed, tested, delivered and operated as expected.