Portworx brings storage solution to LogDNA's Kubernetes need
LogDNA wanted to reduce its cloud storage costs and bring Kubernetes-based software to on-premises servers. Portworx provided a Kubernetes-native solution and ongoing support.
Once you get a lumber yard worth of logs rolling, it's hard to get them to stop.
LogDNA, a logging-as-a-service vendor, has more than 4,000 clients, ranging from solo developers to enterprise businesses, such as IBM, who use its platform to manage and archive application logs.
Rising public cloud storage prices, however, forced the company to bring the service under its roof and control. What started as a logistical nightmare for one LogDNA engineer eventually became a managed offering with the help of Pure Storage's Portworx platform.
Built upon Kubernetes, LogDNA offers clients an all-in-one log management and archive service by allowing them to set usage quotas, exclusion rules and data visualization.
The company originally used several public cloud providers for their storage needs, as the flexibility and seemingly infinite space offered the company room to grow, said LogDNA co-founder and CTO Lee Liu.
Unfortunately, this growth came with the sticker shock of ingress and egress fees, as well as complexity related to varying tiers of archive storage. Faced with mounting costs, but trying to avoid passing costs onto customers, LogDNA decided to bring its storage to a private environment.
"Logs don't really stop, so there's always a huge deluge of logs," Liu said. "We manage petabytes and petabytes of logs. … Public cloud is super easy to get started, but, man. Before you know it, you're paying almost $1.5 to $2 million a month on public cloud. It gets expensive really, really fast."
On the chopping block and file
LogDNA eventually decided to make use of Equinix Metal, which helped slash its storage costs by half, according to Liu. This shift let LogDNA take advantage of bare-metal servers with NVMe and SSD drives at Equinix data centers to maintain performance, while also meeting regulation compliance for clients.
There's not a whole lot of expertise in this area, in my opinion. It was really refreshing [to talk to] a whole team that knows this stuff.
Lee LiuCo-founder and CTO, LogDNA
This shift to localized racks did require LogDNA to make some adjustments.
"The tricky thing for us was how we mimic the setup we had on public cloud but with our own bare-metal storage nodes," he said. "When you have to roll your own Kubernetes, it's challenging when you have a bunch of plain servers with local disks. … There's no easy way to use the space on local disks."
LogDNA originally made use of a storage provisioner developed in-house, but it encountered problems.
"It worked well for a year or so before it became so overwhelming," Liu said. "The one engineer we assigned it to was inundated with bug fixes and other problems."
Liu found Pure Storage's Kubernetes-native Portworx as a solution to LogDNA's dilemma.
"With Portworx, … we could take the disks that were on each individual server, create a pool off of them and then be able to use that storage very much like we did on public cloud," Liu said. "Portworx lets us effectively automate that second piece, which is take storage from local servers and be able to add it to a pool."
More importantly, it helped save the sanity of the engineer by placing an entire team of support behind the provisioner.
"To be honest, if I had more money back then and I could dedicate a team to a provisioner, I probably wouldn't need Portworx," Liu said. "But Portworx is on call 24/7 for us. We can hit them on Slack and they can look at something immediately. I don't believe I could replicate that support element."
He added the support team's expertise is "refreshing" to speak with, considering the niche Kubernetes provisioning needs his company faces.
"You're like three levels deep in the rabbit hole here," he said. "There's not a whole lot of expertise in this area, in my opinion. It was really refreshing [to talk to] a whole team that knows this stuff. The support really sold me on this."
The transition to Portworx did require some finesse, such as modifications to the Kubernetes scheduler and minor changes to how Portworx's software worked with LogDNA's particular needs. Still, Liu said the Portworx team solved most of those issues with minimal headaches.
"A lot of Portworx was designed for the pooled storage use case, whereas we need more local storage," Liu said. "In general, a lot of it almost worked out of the box as is."
Sticking points
Scott Sinclair, a senior analyst at Enterprise Strategy Group, a division of TechTarget, echoed Liu's comments that Portworx is among the few companies that can boast of having knowledge and support for Kubernetes storage environments.
"The trick is that not every vendor has the same level of expertise in-house when it comes to production Kubernetes and container-based application environments, especially for support," Sinclair said. "I expect we will get there in a few years. But for now, it is a nice differentiator for Portworx. I am not saying they are the only ones that have that expertise in-house, but they are one of the few that recognize its value and prioritize it."
The cloud storage cost pitfall LogDNA encountered isn't uncommon for many startups, he noted, especially because many startup companies are likely right to avoid hardware purchases, which may not matter in a few years.
"Early on, a company may not have a large IT staff or a data center location, so starting on public cloud services is the smart move," Sinclair said. "The cost of cloud, however, scales quickly and can become a major cost of revenue for these firms at scale. Moving it back on premises to a data center or to another partner can help offer some cost benefits and help provide economies of scale as the business continues to grow."
