Log analytics tools gain application performance support skills
Log analytics tools are boosting real-time capabilities to monitor application performance, and users are in the market for tools that best slice, dice and store log data.
In IT, logs are a fact of life, but not a boring one. Major log analytics tools are stretching to handle new challenges and entertain new use cases.
All log analytics tools deal with unstructured data. Their focus is on reading all the data concerned with IT operations and systems logs, without a particular model or database setup. Some tools, such as Splunk, also incorporate security data. Some vendors' log analytics products are evolving to behave similar to application performance monitoring (APM) tools, according to Charley Rich, research director at Gartner.
"[Log analytics tools] are missing a number of components, like diagnostics, tracking and depth in capturing metrics, but they are starting to move in that direction," Rich said. Tools now handle near-real-time data streaming from logs. However, analytics platforms still typically lack an understanding of topology -- the dependencies and relationships that reveal sources of application performance problems. Topology is still the domain of APM tools.
As IT organizations demand new approaches to management methods and tools, makers of log analytics tools are responding by integrating AI into traditional tool sets, referred to as AIOps. AIOps platforms unite automation with analytics and machine learning enacted upon the big data generated by logs and other information sources.
Get into the data, not the vendors
IT organizations cannot focus on log analytics capabilities without also dealing with the realities of disruption in the vendors that sell them, and for users, the ideal scenario is integration.
"Having been burned many times by vendors that were acquired, went out of business or later raised prices or removed services or features, I've been looking at ways to stitch together cloud offerings that offer more flexibility down the road," said Alan Majer, CEO of Good Robot, a Toronto-based IoT consultancy. Majer's log analytics tool selection criteria revolve around API integrations with other products, rather than attractive GUIs and dashboards. He hopes to find vendors that don't, in his words, "try to place themselves at the center of the universe."
Majer selected Loggly and Dell Boomi for the task. Loggly, he said, offered a flexible and affordable log analytics platform, with numerous ways to capture, view and analyze data. Boomi, although much more expensive, appealed to Good Robot because of its ability to stitch together multiple APIs and services in the cloud, Majer explained.
Both logging and analytics platforms are waking up to the reality that they are part of a system.
Alan MajerCEO, Good Robot
IT organizations want more ways to interpret log data, whether to identify patterns or catch problems proactively. "There are many ways we have to look at our data. ... We want to store that log data somewhere so we can later look at it," said Vaibhav Puranik, vice president of engineering, big data and platform at GumGum, a computer vision company. After an attempt to manage and scale log analytics on its own with the open source Elastic Stack (originally called ELK Stack for its components: Elasticsearch, Logstash and Kibana), GumGum evaluated Sumo Logic and Splunk. At the time, Splunk's cloud offering didn't meet the company's standards, and Sumo Logic's cloud-first approach fit the scale it needed. (The evaluation was three years ago, and Splunk has since released a redesigned cloud version of its product.)
Sumo Logic takes all the log data, Puranik said, and helps the team figure out what went wrong when a system behaves in an unacceptable manner. It had more complex alerting functionality than Splunk during the evaluation period, he said. And engineers can search through log data as needed.
Echoing Gartner's Rich, Puranik said Sumo Logic also presented a surprise. "One interesting thing that has come up, which was unexpected for a log management system, is that, with Sumo [Logic], we can have two things handled by one system; it does not completely displace monitoring, but it does handle a lot of our monitoring use cases," he said. For example, Sumo Logic is prebuilt to digest data from GumGum's AWS operations and provide graphs and alerts on it.
"Both logging and analytics platforms are waking up to the reality that they are part of a system," Majer said, and the vendors need to work with IT professionals who demand that level of integration with other tools and the deployment environment. "APIs, Docker containers, easily scalable resources and cloud-friendly formats, like JSON, all help to put the customer [the IT admin] in control, allowing them to weave a flexible and distributed cloud-based system across many different services or vendors," Majer said.
While Splunk, Sumo Logic and other log analytics tools move further into data analysis, especially in the AIOps arena, APM vendors are adding log analytics capabilities to their portfolios, Rich said. AIOps in log analytics and other IT management tools should reduce false alarms and do more to find the causes of problems.
