Take advantage of eBPF's monitoring capabilities on Linux

What is eBPF?

The extended Berkeley packet filter is a framework that enables programs to run in the kernel space of a Linux system.

The eBPF utility is based on the Berkeley Packet Filter (BPF), which has been present in Linux and other Unix-like systems since the early 1990s. However, eBPF extends the features of BPF by collecting more types of data and making it easier to write programs that interact with that data.

The eBPF utility was introduced into Linux starting with kernel version 4.4. It's fully supported in kernels 4.9 and later.

Why is eBPF important?

To understand the value of eBPF, you must first understand the distinction that Linux kernels impose between user space and kernel space.

Typically, Linux applications run in what is known as user space or userland. These programs are abstracted from the kernel and cannot directly access information that the kernel controls, such as data about which memory and network resources each system process is using. Certain userland utilities, like lsof, make it possible to retrieve some of this data, given the right access permissions, but it is not available by default to user-space programs.

It was always possible to run programs in Linux's kernel space that enable the program to access data not available in user space. Historically, however, doing so required developers to create and insert a kernel module that contained the program. This presented several disadvantages; for example:

• Skill requirements. Writing kernel modules requires low-level knowledge of the kernel, which not all developers have.
• Stability. Modules with buggy code could destabilize the kernel and crash the entire system.
• Access management. Inserting modules usually requires root access, which might be unavailable to users who have a legitimate reason to access low-level data, like networking information, for observability purposes.
• Resource overhead. The more kernel modules inserted into a kernel, the more resources the kernel consumes. Thus, using kernel modules to gain low-level access to the system increases resource overhead.

Admins could also modify the Linux kernel's source code to run a kernel-mode program without using kernel modules. However, that approach is more complicated -- not to mention potentially insecure -- because it requires customizing and rebuilding an entire kernel.

For these reasons, running low-level monitoring programs on Linux systems wasn't always practical. Monitoring strategies therefore traditionally focused on data collected from individual applications running in userland.

With eBPF, however, developers and IT engineers can run kernel-mode programs that are easy to write -- when using an eBPF development framework, at least, as explained below -- and easy to deploy.

In addition, eBPF sandboxes programs, which virtually eliminates the potential security and stability risks from deploying a buggy eBPF program. Also, eBPF programs are efficient from a resource-consumption perspective, which means they add minimal resource overhead to the host system.

Writing eBPF programs

An eBPF program runs as bytecode. IT pros can write bytecode directly, but it's much easier to write eBPF programs using development frameworks like bcc or bpftrace. These frameworks enable coders to write eBPF programs in high-level languages, then compile the programs into bytecode automatically.

These development utilities are part of the reason why eBPF's use has grown in recent years, even though BPF has been around for decades. BPF was never very practical to use, but that's not the case with eBPF, which is built into the Linux kernel and supported by excellent development tooling.

Monitoring and observability with eBPF

An eBPF program supports a variety of use cases, such as security and reliability engineering. However, one of its most valuable use cases is for monitoring and observability.

In this context, eBPF makes it possible to collect system performance and reliability data with a degree of precision and granularity that is impossible with user-mode tools. With eBPF, teams that use eBPF can monitor on a process-by-process basis which packets are flowing where, which files are open or how much disk space is in use, for example.

Historically, to get this type of process-specific data was challenging. Admins could use tools like tcpdump to monitor packets or lsof to track open files. But tracing this data to individual processes was difficult, especially without the help of special kernel modules. With eBPF, admins can collect this type of data in commands that are as short as a single line.

Without eBPF, the best that admins could typically hope to achieve with monitoring and observability data was to detect unusual trends, such as a spike in network traffic. The process to trace those trends to the associated individual process to find the root cause was much more difficult -- especially if the process in question didn't expose metrics that monitoring and observability tools could ingest.

High-level applications, like a web server, usually expose such metrics; it's not too hard to determine the root cause of a performance issue in a high-level application. But what if the problem stems from a bug or configuration issue in a low-level resource, such as a network socket? Before eBPF, gaining visibility into the Linux kernel's dark corners was a significant challenge.

With eBPF, however, it is easy to get granular performance data from any process within a Linux system.

The eBPF utility doesn't solve every problem under the sun. It works only on Linux, so it's of no help for monitoring Windows-based environments. At time of publication, admins need root access to use most of its features. And while tools like bpftrace and bcc make it relatively easy to write and deploy eBPF programs, they are still maturing.