An eBPF tutorial to try out the bpftrace framework

EBPF development frameworks

An eBPF program is inserted into the kernel as bytecode. Developers can write eBPF bytecode directly, if they wish.

However, because bytecode is arcane and not easily readable by humans -- even experienced developers -- most programmers will write eBPF programs using a development framework like bcc or bpftrace, which are open source tools developed by the IO Visor Project. These frameworks enable developers to write high-level code that collects data through eBPF. Then, the frameworks automatically compile that code into eBPF bytecode that the kernel can execute.

Bpftrace is the simplest eBPF programming framework for most tasks. It provides a command-line utility, called bpftrace, that lets admins execute eBPF commands directly. Bpftrace is a great choice for straightforward eBPF programming tasks -- like monitoring block device activity or tracking which processes have which files open -- that don't involve a lot of conditionals or variables.

In contrast, bcc, which can be imported into Python and Lua applications, is a better tool for writing complex eBPF programs. But it also requires more effort to set up.

Write an eBPF program with bpftrace

For this short tutorial, we'll use bpftrace to write a simple eBPF program. The host environment is Fedora 34, though the steps below are generally the same on any modern Linux distribution. However, eBPF is fully supported only on Linux kernel versions 4.9 and later, so use a distribution with a 4.9 or newer kernel.

Install bpftrace

First, let's install bpftrace. Fedora provides a package, so we can install it with a simple dnf command:

dnf -y install bpftrace

Deploy an eBPF program

As mentioned above, bpftrace is a simple command-line utility that enables developers to write eBPF programs that are as short as a single line. For example, here's a short program -- based on one of the "one-liners" from the bpftrace GitHub repo -- that lists the processes' disk consumption:

bpftrace -e 'tracepoint:block:block_rq_issue { printf("%s %d\n", comm, args->bytes); }'

The output will look something like this:

Compositor 4096

kworker/0:1H 131072

kworker/0:1H 65536

kworker/0:1H 131072

WebExtensions 4096

WebExtensions 4096

kworker/0:1H 68608

The first column is the process name, and the second is disk size in bytes. You can modify the output by changing the printf function in the eBPF program. For example, to display the process ID in addition to the process name, the program would change to:

bpftrace -e 'tracepoint:block:block_rq_issue { printf("%s %d %d\n", comm, pid, args->bytes); }

Because eBPF programs trace data in real time, the command will continue to dump output to the terminal until killed with Control + C.

You can redirect the output to a file using the > operator if you want to save it there, or use bpftrace's built-in -o argument for saving output to a file. You can also pipe output to grep if you want to do regular expression searches.

Disable kernel lockdown

If bpftrace prints an error message that mentions kernel lockdown mode, ensure you disable lockdown mode before bpftrace will work. Lockdown mode, which is enabled by default on some Linux distributions, prevents eBPF from accessing kernel-level data.

There are several ways to disable kernel lockdown; the easiest is to press the Alt + SysRq + X key combination, which will disable kernel lockdown until the next reboot. You might need to enable the SysRq key first, which you can do by running the command echo "1" > /proc/sys/kernel/sysrq as root. You can also use mokutil to disable it permanently.

Next steps with eBPF programs

Above, we've covered the very basics to write and deploy an eBPF program. There are many other things you can do with bpftrace and other eBPF frameworks. The bpftrace main page is a good place to learn more about the eBPF programs you can write and deploy with this tool. And there is also bcc to incorporate eBPF into an application.