Embattled New Relic cuts Kubernetes observability overhead

New Relic boosts Kubernetes observability scale for large customers

One bright spot for New Relic in its earnings report this month is an increase in customer accounts worth more than $100,000, even as its overall customer count fell. Such customers numbered 862 in the fiscal first quarter of 2021, compared with 1,099 in the fiscal fourth quarter of 2022, according to the company's annual financial report. New Relic officials also said such large-scale customers are looking to manage fewer, larger Kubernetes clusters as production container deployments grow, which this week's product updates reflect.

"When you are a large streaming company or telco and you're moving your workloads from VM clusters to Kubernetes clusters, for [network] efficiency, you can't have that many small clusters," said Ishan Mukherjee, group vice president of product at New Relic. "One thing that was prohibiting them from using APM and observability tools in general is the memory footprint and the overhead of the agents, and we have significantly reduced that so that [these customers] can use it at 1,000-node scale."

The New Relic platform now offers two methods of data collection within Kubernetes clusters: via Extended Berkeley Packet Filter (eBPF) instrumented with New Relic's Pixie open source tool, or through a more traditional software agent. The eBPF approach has gained favor in large-scale shops for observability and security in recent years because of its lower resource overhead, and will be New Relic's focus in the future.

However, many New Relic customers also still use its agents, which it has made more efficient with this week's release, Mukherjee said.

"We're reducing the memory utilization [on Kubernetes nodes] and using a more advanced [data] sampling technique," Mukherjee said. "With Kubernetes clusters specifically, the volume of inter-entity transaction chatter is 10 to 100 times higher than in VM clusters, so if you use the same [data] capture and sampling approaches, you will end up having [exponentially] more data collected."

Ishan Mukherjee

With this new release, the New Relic agent samples data at a lower rate and excludes low-value chatter data to mitigate this problem, Mukherjee said. To set the stage for its shift to eBPF, this week's release also deepens the integration between Pixie, which New Relic acquired in late 2020 and donated to open source last year, and the New Relic platform. Last year, the platform began to support real-time views of eBPF data in running clusters in its topology dashboard and will now ingest historical data for comparisons beyond the real-time window of six hours.

Pixie isn't without its own memory overhead, roughly 512 MB per node. Customers can reduce data retention time and use data compression features to keep memory usage manageable, according to New Relic officials.

Security add-on will increase pricing pressure

New Relic also rolled out its first security monitoring feature this week, adding vulnerability data to its APM dashboard within the New Relic platform. New Relic is the latest among many APM and observability vendors to diversify into the security realm -- Splunk, Sysdig, Sumo Logic, Elastic, Dynatrace, AppDynamics and Datadog, among others, have already connected security and observability data within their product lines as DevSecOps becomes a priority for platform engineers and SREs.

At least initially, New Relic isn't looking to create its own general-purpose security tools, Mukherjee said.

We'll focus on the 'left' in 'shift left' and send security signals to developers in pre-production environments.

"Our user base is primarily developers, and we're going to be developer-focused, harvesting vulnerability data from our instrumentation and working with the Laceworks and Snyks of the world," he said. "We'll focus on the 'left' in 'shift left' and send security signals to developers in preproduction environments."

New Relic Vulnerability Management is in beta, and the vendor is piloting other security-related feature ideas such as security information and event management (SIEM) tools; database posture management and bespoke integrations with specific applications such as SAP, Office 365 and popular databases.

For now, Vulnerability Management is free to customers of Data Plus, a pricing tier with a 90-day retention period launched April 14 and priced at $0.50 per gigabyte. Data Plus combines features such as high-performance querying and FedRAMP and HIPAA compliance support that were previously priced separately. The vendor estimates the combined Data Plus is 56% cheaper than buying separate features, butthere aren't specific plans to adjust data ingestion pricing to for lower-access tiers of archival observability data similar to those made by competitors such as Sumo Logic and Splunk, according to Mukherjee.

Thurai predicted New Relic will need to make further pricing concessions as observability data volumes grow, a tough decision while it faces financial worries. Without it, the company risks losing further customers to free and open source tools.

"Pricing seems to be a sore pain point for many large digital enterprises," Thurai said. "Given the choice and variety that is available in the market ... if New Relic doesn't reduce pricing fast enough, they will either lose customers to smaller players or to open source."

Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.