EBay made Kubernetes monitoring more flexible for developers and consistent for ops through modifications that are now part of Elastic's Beats software.
EBay's Kubernetes monitoring improvements to Elasticsearch are now available to other IT shops as part of the core open source product.
EBay modified Beats to make Kubernetes monitoring more cohesive, with software agents that collect data from endpoints in infrastructures monitored with the Elasticsearch, Logstash and Kibana stack. In late 2017, eBay donated its Beats contribution to the ElasticSearch open source community, and it was added to the main Beats production in late February 2018, according to executives at Elastic Inc., the corporate originators and maintainers of the Elastic Stack (formerly called the ELK stack) and Beats, based in Mountain View, Calif.
The impetus for eBay's Beats project was a desire to move away from restrictive internal software frameworks and to give its developers more freedom to choose their own application stacks. But at eBay's scale -- the company* generates 1.2 PB per day of log data, and that number grows organically by 50% each year -- the company needed a programmatic way to manage Kubernetes monitoring data, regardless of application format.
"We needed a way to collect logs and metrics from multiple clusters in a very generic fashion," said Vijay Samuel, an eBay senior software engineer who leads efforts to centralize Kubernetes monitoring, as the retailer deploys the container orchestration platform in all its data centers. "We're trying to get away from developers having to instrument their logs and metrics with our internal APIs and start to collect data without requiring application owners to redeploy code."
Beats is an open source utility Elastic acquired in 2015 with Packetbeat. EBay was familiar with Elasticsearch for Kubernetes monitoring from previous IT projects, and it also favored Beats because the tool can collect both metrics and log monitoring data through one lightweight software agent.
EBay's customizations for Beats correlate log and metric data with Kubernetes metadata, such as Node and Pod location and labels. Beats agents on the Kubernetes cluster can see when new apps are being deployed and automatically send Kubernetes monitoring data to a central IT monitoring tool eBay built, called Sherlock.
"The Sherlock platform is where all our data comes in, and we generate reports out of it," Samuel said. "These reports help developers to understand how their application behaves."
Diagram of eBay Kubernetes monitoring with Elasticsearch Beats
EBay Elasticsearch plans include self-service, QoS
Centralized logging and monitoring is our vision ... we just expose the APIs users need to know to send data in and consume the data.
Vijay Samuelsenior software engineer, eBay
Developers can visualize and analyze Sherlock's central pool of data using a variety of tools, such as Kibana and Grafana.* As eBay expands Kubernetes deployments to encompass its whole data center infrastructure, Sherlock will also provide a self-service interface for developers to access the data collected from their applications through Beats.
"Having centralized logging and monitoring is our vision, but some groups use their own stacks," Samuel said. "We just expose the APIs users need to know to send data in and consume the data."
Updates to the eBay Beats modules are in the works to lighten the data-collection load on the company's networks. Quality-of-service (QoS) features will prioritize metrics and events over log data collection performance,* while sampling and throttling for all data types will cut down on network saturation during data collection for apps whose every transaction doesn't need to be stored and cataloged.
QoS will require balance between flexibility and manageability, eBay engineers said, and the implementation details aren't worked out yet.
"The way we do QoS isn't like regular networking, and some companies will do it differently than others," said Andy Santosa, eBay's director of engineering platforms. "Creating something that can be used generically in different use cases is something that we want to work on carefully with the community."
EBay's Beats donation joins other donations from the company to open source communities for Kubernetes management, which also included the Tess Master utility in 2017.
