Spreading the DevOps observability responsibility across IT

The quest for better visibility

In the waterfall software development model, developers built new features while separating software testing. However, monitoring activities and infrastructure operations were outside the purview of development's scope. Developers didn't understand fully the implications of infrastructure dependencies and application semantics. Therefore, apps and services were built with low intrinsic dependability.

In contrast, modern software development and operations teams follow cloud-native and Agile development methodologies. This enables teams to release software without affecting other teams and reach the market quickly, thus enabling higher productivity and better profitability -- but with decreased visibility. Debugging issues in production becomes a nightmare and results in restless nights unless application components become observable.

What is observability? Why do we need it?

Observability refers to a system's capability to identify its internal state by analyzing its outputs. It helps create a deep understanding of the actual health of the system and provides insights on troubleshooting options. Observability provides actionable insight into the root causes of errors within a system and -- more importantly -- why the error has occurred. And IT organizations have several observability platforms from which to choose, including OpenTelemetry, Zipkin and Jaegar.

Observability goes beyond monitoring and provides increased visibility into every layer of the business, which presents the opportunity for more strategic initiatives. It uses logs and monitoring tools to produce actionable insights into the entire infrastructure. Observability systems enable admins to quickly determine what is occurring with each request and to pinpoint the root cause of a given problem as soon as it occurs.

Observability and monitoring are not the same

Observability is often confused with monitoring, but they are two distinctly different concepts. Monitoring provides a comprehensive picture of system behavior and performance, enabling IT teams to evaluate trends and generate reports, as well as get notifications when anything goes wrong. Observability, on the other hand, complements monitoring via a variety of telemetry sources to represent the deeper states of all system components.

Who holds responsibility for observability?

Often, the ownership of monitoring and observability lies with the development team. However, monitoring and observability shouldn't be the exclusive responsibility of a single person or team. This will not only help avoid a single failure, but it also helps IT professionals to understand and improve the system overall.

In many organizations, this role falls under site reliability engineering (SRE) or IT operations, while the other teams collaborate on a centralized team to manage observability. An observability engineer needs to develop a monitoring infrastructure -- including logging, tracing and metrics -- and work closely with the production engineers to provide the best tools to measure systems reliability.

Challenges faced by distributed SRE or observability teams

SREs are embedded in various teams throughout the company in a distributed SRE or observability function. Although this is a strong starting point for observability, it has some drawbacks in the long run:

• effort duplication results in wasting time and effort;
• getting a global perspective across all the teams becomes difficult; and
• monitoring and observability practices begin to vary across teams.

To solve the observability challenges, organizations should communicate the roles and responsibilities to the developers clearly. It is important to ensure that all developers are skilled in observability and monitoring, as this will encourage a culture of data-driven decisions and thus decrease outages.

Issue escalation

The operations teams should be aware of who knows what and how to contact them to fix a problem. In the event of an escalation, they should be able to contact the concerned individual.

Communicate and collaborate about observability

To build trust within an organization, it is imperative that efficient and accurate communication pathways are in place. When departments trust one another to provide correct information, the additional fact-checking step that can stifle productivity is eliminated.

DevOps has transformed systems development lifecycles (SDLC) -- monitoring is no longer about collecting log data, metrics and distributed event trace information. Instead, monitoring is used to improve the system's observability. Monitoring extends to the development segment, which is made possible by people, processes or technologies that operate across the SDLC pipeline.

Collaboration across cross-functional teams, such as development, IT ops and quality assurance (QA), is essential to build a high-quality, dependable system. Communication and feedback between developers and operations teams is critical to the system's observability goals, which help QA produce accurate and insightful monitoring throughout the testing process. This enables the DevOps teams to conduct real-world performance testing on systems and products in an organization.

Proper communication and collaboration boosts teamwork and productivity to a considerable extent. Moreover, these are great ways to ensure that staff feel valued, to boost morale and to keep them engaged. Continuous iteration in response to performance feedback can also help identify potential issues before they affect the end users.