log analytics
What is log analytics?
Log analytics is the assessment of a recorded set of information from one or more events, captured from a computer, network, application operating system (OS) or other IT ecosystem component. An organization can use log analytics to uncover patterns in user behaviors, identify problems, audit security activities or ensure compliance with established rules, and plan for capacity or IT infrastructure changes.
An Event is an identifiable or significant occurrence within hardware or software, and the information about it is recorded in a log. A user or computer system can generate an event. For example, a server hardware failure is an event.
Organizations rely on tools, such as Windows Event Viewer for Windows OS or the application SolarWinds Security Event Manager, to access, view and analyze logs.
Log analytics software collects and parses error logs to help an organization diagnose an issue: what caused the problem, where it is located and how serious it is. Log analytics can also aid a user to determine trends in an application's or system's operation. Log analytics tools aggregate logs from disparate data sources, compiling a view of the widespread operation of an IT ecosystem.
How log analytics works
Log analytics software collects logs from events, such as application installation, security breaches, and system setup and startup operational information. An example security event is a system login attempt. An example operational event is when an application opens successfully. Setup events focus on the control of domains, such as where a log is stored after a disk configuration. System events focus on components such as the central processing unit (CPU) and storage.
A log entry includes such information as the date and time the event occurred, the computer the event occurred on, an identification of the user, the category of the event -- such as setup or security -- and the program that initiated the event.
Log analytics occurs by organizing data via pattern recognition, classification and tagging, correlation analysis, and artificial ignorance. Pattern recognition compares incoming events with past events to determine which new occurrences will be significant. Classification and tagging puts events into ordered classes and assigns a keyword to each event to describe it in a standardized way. Correlation analysis can sort logs by warning events and then alert administrators to a widespread system error if a critical warning appears in multiple logs. Artificial ignorance, a machine learning program, discards log entries that occur regularly. It helps reduce noise and find uncommon events. Artificial ignorance is well suited to a system that operates consistently with a low number of issues.
Each system that generates log messages writes them in a way that is specific to itself, so log analytics software must pull everything under cohesive terminology. For example, one application logs a moderate software failure event as a warning, while another application labels the same event as an error.
Log analytics tool features and products
A log analytics tool performs log aggregation and gives users a query language to glean insights from the collected information. Log analytics tools can also automatically process logs for insights into specific events, or perform deeper analysis to extract meaningful conclusions or make predictions about the pattern of events taking place over time.
These tools typically tier events by level of urgency. For example, Windows Event Viewer uses information, warning, error and critical urgency levels. Information is the least severe log entry, typically for successful events. Warnings give attention to potential issues, but do not indicate that something needs to be fixed. Error-level events occur when an application starts to fail unexpectedly. A critical error happens when a program is forced to stop and can no longer run properly without further attention.
Log analytics tools commonly offer graphical user interface dashboards that display the most relevant and critical information gleaned from log input. A dashboard might include a total count of events, alerts, log search queries, graphs, and filters for security or change management. Graphs can show statistics on disk space, CPU status and event categories. Some dashboards are customizable.
Log analytics tools commonly include search functionality, which helps users find logged events. For example, if a log analytics tool uses classification and tagging, then the user can quickly search for a specific event by the given keyword.
A sampling of log analytics products includes DataSet, SolarWinds Security Event Manager and Microsoft Azure Log Analytics.
DataSet, formerly Scalyr, is marketed as a DevOps log monitoring and analysis tool because it consolidates logs for diagnosis and visualization from applications and systems. DataSet can filter out user-specified logs as well as graph metrics to show statistics such as percentiles, rates, distributions and trends.
SolarWinds Security Event Manager (SEM) is an example of a tool with a customizable dashboard. It can show data used by multiple accounts and filter events tied to security. SEM can alert users upon a warning event, which can be specified by the user. The dashboard can also contain a word cloud -- a chart showing where the most logs are generated.
Azure Log Analytics, not to be confused with the term log analytics, is part of a public cloud offering. It can be accessed independently or through other Azure products, such as Azure Security Center. Azure Log Analytics can analyze virtual machines via agents as well.
