New Relic open sources Pixie observability platform

Aiming for comprehensive, low-friction observability

Pixie enables developers to see all of their applications' metrics, events, logs and traces with a single command-line interface command, said Ishan Mukherjee, co-founder of Pixie Labs and vice president at New Relic. It provides comprehensive observability for developers on its own, without the need for extra instrumentation, setting up dashboards or moving data off the cluster, he added.

As an open source project, developers can run an entirely self-hosted version of Pixie without third-party dependencies.

The assets New Relic contributed to open source include Vizier, the tool's data collection and query engine, which runs on a Kubernetes cluster where it collects data and stores it locally to execute queries for Pixie clients. It also contributed Pixie Cloud, Pixie Docs and the Pixie website.

Ishan Mukherjee

Making Pixie ubiquitous

In addition, New Relic is dedicating the majority of the Pixie engineering team to continue its work on the open source project, said Bill Staples, president and chief product officer at New Relic in a blog post.

"It allows us to get Pixie in the hands of engineers across the world -- similar to Meteor for Kubernetes itself," Asgar said.

Asgar and Mukherjee envisioned building a SaaS product in the observability space, which was traditionally called application performance monitoring (APM), where vendors like New Relic, Splunk, AppDynamics, Datadog, as well as legacy vendors such as IBM and hyperscalers with CloudWatch (AWS) and Stackdriver (Google), play. However, as they realized traditional APM was merging with metrics, dashboarding and logs, they knew Pixie had to be all those things and more.

"Pixie Labs' technology provides observability/visibility/security for Kubernetes environments," said Stephen Elliot, an analyst at IDC. "It uses telemetry data collected entirely inside Kubernetes. Aimed at developers and IT operations teams that require Kubernetes visibility, it's an increasingly important capability and set of data for these teams. For the community, it adds more fuel to the fire and provides a robust growth opportunity for Kubernetes in general."

It allows us to get Pixie in the hands of engineers across the world -- similar to Meteor for Kubernetes itself.

The AWS connection

Meanwhile, in an AWS blog post by Colin Bookman, an independent software vendor senior solutions architect at AWS, and Mark Carter, general manager of observability services at AWS, wrote that they are "particularly enthusiastic" about the programmability of the Pixie platform. They also mentioned Pixie's use of Extended Berkeley Packet Filter (eBPF) to provide rich, automatic visibility of application events.

"Pixie makes observability easily accessible to developers," the AWS post said. "At AWS, we share that vision, to provide every developer access to high-quality observability data with minimal effort." As such, AWS is partnering with New Relic on the CNCF project and has appointed Jaana Dogan, an AWS principal engineer, to sit on the Pixie board.

Bookman and Carter cited Pixie's auto-instrumentation, programmatic data access and Kubernetes-native edge compute capabilities as key differentiators for the project.

"Pixie runs entirely inside Kubernetes as a distributed machine data system, meaning you don't need to transfer any data outside the cluster," the AWS post said. "Pixie's architecture gives you a secure, cost-effective and scalable way to access unlimited data, deploy [AI/machine learning] models at source and set up streaming telemetry pipelines."

Zain Asgar

Pixie is standardizing its observability offerings with the CNCF OpenTelemetry project standards, Asgar said. The new partnership with AWS is an expansion of a recent collaboration between AWS and New Relic on the AWS Distro for OpenTelemetry.

"Might this signal a change in the AWS approach to open source?" asked Eric Newcomer, CTO of WSO2, which offers a variety of open source software for API management and other tasks. "They have been criticized in the past for not giving back to the community. I will be very interested to see whether this means they will be more actively contributing to CNCF projects."

AWS' move is pragmatic, however. "Pure-play observability vendors such as New Relic, along with Pixie added through acquisition, are in high demand among cloud partners like AWS clamoring for technology to fill out their operational lifecycle management requirements," said Charlotte Dunlap, an analyst at GlobalData in Santa Cruz, Calif.