Practical eBPF in 30 Days

A hands-on, experiment-driven path through modern eBPF on Linux 7.1. Written in Head First style with rendered diagrams.

What you’ll know by Day 30

• How BPF programs are compiled, verified, JITed, and attached.
• The verifier’s type system and how to read its rejection messages fluently.
• CO-RE, BTF, and how libbpf relocates field offsets at load time.
• Tracing with fentry/fexit/tracepoints/uprobes; capturing stacks; building flame graphs.
• Networking BPF: XDP, tc/tcx, AF_XDP, cgroup_skb, sockops.
• Modern primitives: kfuncs, kptrs, struct_ops, BTF spelunking.
• The frontier: sched_ext (BPF-defined kernel schedulers).

Phases

How to use this

Each day is ~75–90 minutes. Format:

1. Concepts introduced today — what the new primitives are, before any code uses them.
2. The lab — runnable code with line-by-line walkthrough.
3. What to break, in order — deliberate mistakes that teach the most.
4. What to read in the kernel — specific files and functions in your kernel checkout.
5. Bullet Points — terse summary you can review before bed.
6. Check question — single recall prompt for tomorrow morning.

The most important step is the deliberate breakage. Don’t skip it. Five minutes of pain per lab; saves you weeks of confusion later.

Lab source and prerequisites

Start with the Lab environment page. It defines the Linux toolchain, pinned recursive dependencies, locked Linux v7.1 source, privilege boundary, and per-backend build commands.

Every published entry has a lab record under ebpf/labs/. Days 1–21, Day 24, and the reference capstone use checked-in source whose primary chapter listings are included from the same files compiled in CI. Day 22 and Days 25/27 wrap exact upstream v7.1 targets; Days 23/26 carry complete repo-owned derivatives built against that locked tree. make -C ebpf/labs check-coverage prevents a chapter from silently falling out of this contract.

Style notes

• Head First voice. Conversational, second-person, with the Verifier as a recurring character.
• Real diagrams. Every concept that benefits from visualization gets a rendered PNG.
• No bcc. Modern libbpf + CO-RE only.

Companion book

This book assumes you already know the kernel network stack. If you don’t, read the Linux Network Subsystem book first — it covers sk_buff, the RX/TX paths, netfilter, and other pieces that BPF programs hook into.

Lab environment

The book targets Linux 7.1. Reading and rendering the mdBook works on other platforms; compiling or running the eBPF labs requires Linux.

Use an isolated Linux host

Use a disposable VM or dedicated development machine where you have root. Tracing labs attach read-only programs, but later chapters exercise XDP, tc, tcx, AF_XDP, cgroups, TCP struct_ops, and sched_ext. Never learn those hooks on the interface carrying your only SSH session or on a production scheduler.

For runtime work the kernel needs BTF at /sys/kernel/btf/vmlinux, BPF syscall and JIT support, and each chapter’s hook/map/kfunc configuration. Linux 7.1 provides the required APIs when its corresponding BPF, networking, cgroup, TCP-congestion-control, and sched_ext options are enabled. The preflight checks what it can without changing the host; attachment failures still report the specific unavailable hook or kfunc.

Clone the pinned dependencies

git clone --recurse-submodules https://github.com/biefy/books.git
cd books

For an existing clone:

git submodule update --init --recursive

The repository pins:

• libbpf-bootstrap at fac4e8ddf011aead8e14962bf8db74542331264b, recursively including libbpf, bpftool, and architecture vmlinux.h;
• xdp-tools/libxdp v1.6.3 at 8fbad9f0af621a22aa87ff2520b3735915b1f0fd;
• the official Linux 7.1 archive and commit in ebpf/labs/linux-source.lock.json.

Do not replace these with moving branches when validating book code.

Install build prerequisites

You need Clang 17+ with the BPF target, a C compiler, GNU Make, Git, Python 3, libelf/zlib development headers, and m4 for the pinned libxdp build. The kernel-backed DCTCP and sched_ext builds add ordinary kernel build tools. On Ubuntu or Debian:

sudo apt update
sudo apt install -y \
  clang llvm git make gcc libc6-dev libelf-dev zlib1g-dev pkg-config python3 \
  curl xz-utils m4 flex bison pahole libssl-dev libcap-dev libunwind-dev \
  libdw-dev libpcap-dev bc rsync binutils iproute2 iperf3

Package names differ on other distributions. Repository scripts do not install packages, invoke sudo, mount filesystems, or silently fetch dependencies.

Run the check-only standalone preflight:

ebpf/labs/scripts/preflight.sh

Add --kernel after selecting/fetching the locked source and --runtime when you also want checks against the running host:

ebpf/labs/scripts/linux-source.sh fetch
ebpf/labs/scripts/preflight.sh --kernel --runtime

Build all lab sources

The quick path compiles every repository-owned standalone lab, including AF_XDP and the reference capstone:

make -C ebpf/labs check-standalone

The exhaustive check also builds the canonical Linux v7.1 DCTCP object and the sched_ext backends:

ebpf/labs/scripts/linux-source.sh fetch
make -C ebpf/labs check

Use an existing exact checkout with LINUX_SRC=/path/to/linux. The helper rejects other commits/releases. Build one chapter with make -C ebpf/labs dayNN; use day28-30 for the combined capstone. All generated artifacts stay under ebpf/labs/.output/ and make clean removes them. Kernel-backed wrappers build only the relevant v7.1 aggregate, convert its DWARF to BTF, and generate a narrow vmlinux.h; compilation never depends on the build host’s kernel BTF.

Coverage by backend

make check-coverage compares this curriculum mechanically with SUMMARY.md, manifest.json, the source tree, and the chapter includes.

Run without leaking state

Run ordinary loaders in the foreground and stop them with Ctrl-C. Their signal paths destroy skeleton-owned links and maps; nothing is pinned unless a chapter explicitly teaches pinning.

The privileged smoke dispatcher is opt-in and accepts an explicit day list:

EBPF_LABS_ALLOW_PRIVILEGED=1 \
  sudo --preserve-env=EBPF_LABS_ALLOW_PRIVILEGED \
  ebpf/labs/scripts/smoke.sh day01 day14

It owns its triggers, child PIDs, temporary files, namespaces/veths, qdiscs, and cgroups and cleans them in reverse order. TCP struct_ops registration additionally requires EBPF_LABS_ALLOW_STRUCT_OPS=1; sched_ext replacement requires EBPF_LABS_ALLOW_SCHED_EXT=1. Run either only on a disposable compatible VM. Hosted CI performs compile/link/BTF and locked-source builds only; it never performs privileged attachment.

Day 1 — First fentry program with ringbuf

Today’s mission: spy on every file deletion on your system, in real time, with a program that runs inside the kernel and can’t crash it. Along the way, meet the whole machine that makes this possible — the BPF virtual CPU, the one syscall that loads everything, the just-in-time compiler, and the little code-patching trick that lets you tap a running kernel without a hiccup. Total time: ~120 minutes.

So you want to spy on a kernel function

You’d think this would be hard. The kernel is busy. It runs thousands of functions per second across dozens of CPUs. It does not know you exist. How do you get it to tap you on the shoulder every time something calls filename_unlinkat?

You install a doorbell. That’s fentry.

Here’s the secret on x86_64 builds with function tracing enabled. Traceable kernel functions reserve an entry patch site, often shown as a 5-byte NOP slot. It sits there doing nothing until ftrace/BPF turns it into an entry call path.

filename_unlinkat:
    nop   nop   nop   nop   nop      ← these 5 bytes are reserved for you
    push  %rbp                       ← the actual function body starts here
    mov   %rsp, %rbp
    ...

Whose 5 bytes are those, anyway?

That slot is not BPF’s, and BPF didn’t put it there. It exists because the kernel is built with function-entry instrumentation (-pg / -mfentry): the compiler emits a call __fentry__ at the top of every traceable function (the entry stub is SYM_FUNC_START(__fentry__) at arch/x86/kernel/ftrace_64.S:148). At boot, ftrace — the kernel’s built-in function tracer, around since 2008 — overwrites each of those call sites with a 5-byte NOP. The NOP it writes is literally x86_nops[5] (ftrace_nop_replace() returns it at arch/x86/kernel/ftrace.c:66; the table is at arch/x86/kernel/alternative.c:91).

So the patch site is ftrace’s. fentry/BPF doesn’t own it — it borrows it by registering an ftrace “direct call” that repoints the site at your trampoline. You don’t need to learn ftrace today; just hold the one-line model: the compiler reserves the slot, ftrace owns it, fentry hangs your program off it.

When you attach an fentry program, the kernel atomically patches that reserved site into the architecture’s ftrace/BPF entry path. That path reaches a generated trampoline. The trampoline saves arguments, calls your BPF program with them, restores everything, and then lets filename_unlinkat run as if nothing happened. The exact instruction is architecture- and config-dependent; the important model is patch site → trampoline → original function body.

What the trampoline is actually for: an ABI bridge

“Saves arguments, calls your program with them” sounds like magic until you ask: why does it need to save anything? The answer is that the function being traced and your BPF program are called two incompatible ways.

filename_unlinkat is an ordinary C function. On x86-64, the System V ABI says its first argument arrives in register rdi, its second in rsi, and so on:

int filename_unlinkat(int dfd, struct filename *name)   /* fs/namei.c:5536 */
/*                        ↑ rdi          ↑ rsi                              */

A BPF program, by contrast, is not called with arguments spread across native registers. By convention every BPF program receives exactly one pointer — a pointer to a u64[] context array — in its first BPF register, R1 — the BPF VM’s registers get their full introduction in “Meet the cast” below; for now just hold “R1 = the one pointer your program is handed.” R1 → ctx, and that’s it. The program reads ctx[0], ctx[1], … to get its inputs.

Those two calling conventions don’t line up. Native code puts dfd in rdi; BPF code expects it as ctx[0]. Bridging that gap is the trampoline’s entire job:

1. On entry it spills the native argument registers into a u64 ctx[] array on the stack: ctx[0] = rdi (dfd), ctx[1] = rsi (name).
2. It calls your BPF program with R1 pointing at that array.
3. On return it restores the registers and falls through into the real body of filename_unlinkat.

That ctx[] array is the “argument array” the BPF_PROG macro unpacks. When you write

int BPF_PROG(on_unlink, int dfd, struct filename *name)

the macro (at tools/lib/bpf/bpf_tracing.h:672) expands to read ctx[0] as dfd and ctx[1] as name, casting each to the type you declared. This is the concrete payoff of BTF (below): BTF tells fentry the argument types, so the trampoline knows how many slots to spill and BPF_PROG knows how to cast them.

The trampoline isn’t per-program, either: there’s one trampoline per attach target, reference-counted and shared across every program attached there. That’s why the kernel keys them in a hash table — the “What to read” pointer about bpf_trampoline below. Frame it this way: the trampoline’s first job is the argument marshalling; the hash-table bookkeeping is just how the kernel avoids building a second one for the same target.

There are no Dumb Questions

Q: Patching kernel code while it’s running. Isn’t that lunacy?

A: It would be, except the kernel has been doing this since 2008 — it’s how ftrace works. The 5-NOP slot is reserved by the compiler at build time specifically so the kernel can patch it later. The patch goes through the kernel’s int3-based text-poke machinery, which is safe against concurrent instruction fetches on every CPU (the file:line specifics are in the “What to read” trampoline bullet at the end). You can install and remove fentry hooks while the kernel runs your browser, a database, and a video call without a hiccup.

Q: Why not use kprobe? I keep seeing kprobe in old tutorials.

A: kprobe predates fentry and works differently. It overwrites the function’s first instruction with a software breakpoint (int3 on x86). The CPU traps, a handler runs your code, then it emulates the original instruction and continues. Works on any function. The trap is expensive; fentry’s direct entry path is typically several times cheaper than a kprobe trap, though exact numbers vary by CPU and kernel config. Use fentry whenever it’s available; kprobe only for the rare functions that lack BTF.

Q: What does “BTF” mean, and why do you keep mentioning it?

A: Hold that thought — we’ll meet BTF properly in a moment. For Day 1: trust that BTF is what tells fentry the argument types of filename_unlinkat, so your program can declare int dfd, struct filename *name and have the kernel hand them over correctly typed.

Sharpen your pencil

On x86_64, the compiler often reserves a 5-byte NOP slot at the start of a traceable function. Why 5? What instruction size is that designed to fit?

.
.
.

Answer: a near relative call or jmp on x86_64 is 5 bytes — 1 opcode byte plus a 4-byte signed offset. The reservation is sized so ftrace can patch the site without moving the function body. Other architectures use their own patch-site shape.

Meet the cast

Before you write code, here’s the full cast of characters.

eBPF in 90 seconds

A BPF program is C code compiled to a stripped-down virtual instruction set (BPF) and loaded into the kernel through the bpf() syscall. Before the kernel runs it, a static analyzer called the Verifier proves the program is safe — it terminates, never reads uninitialized memory, never deref’s a pointer it hasn’t proven valid, and stays within a bounded instruction count. If the program passes, the kernel JITs it to native instructions and runs it whenever the chosen attach point fires. The program cannot crash the kernel and cannot read arbitrary memory — every load is verified at load time. Programs talk to userspace through maps (typed shared data structures).

That paragraph hides three machines worth meeting properly: the virtual CPU your program runs on, the syscall that gets it into the kernel, and the compiler that makes it fast. Let’s take them one at a time — you’ll be reading their fingerprints in error messages within the hour.

The BPF virtual machine: 11 registers and an 8-byte instruction

“A stripped-down virtual instruction set” deserves more than a hand-wave, because the very first error you’ll hit today (Break 1) prints register names at you. So: BPF is a 64-bit RISC-like virtual ISA. It has exactly 11 registers, R0 through R10, each 64 bits wide (BPF_REG_0 = 0 … BPF_REG_10, __MAX_BPF_REG at include/uapi/linux/bpf.h:74). Every instruction is a fixed 8-byte encoding — struct bpf_insn at include/uapi/linux/bpf.h:80 — packing an opcode, a destination register, a source register, an offset, and an immediate.

The registers aren’t interchangeable. They have a calling convention, and you must know it to read verifier output:

Two things fall straight out of this table. First, R0 = helper return value is exactly why, in Break 1, the result of bpf_ringbuf_reserve lands in R0. Second, when the verifier prints R0=mem_or_null, it’s telling you: R0 holds the pointer the helper returned, and the verifier has tagged it “might be NULL.” Dereferencing it (writing *(u32 *)(r0+0)) without first checking for NULL is the instruction it rejects.

How does the verifier know R0 might be NULL? Because it walks your program one instruction at a time and tracks a type and value-range for every register as it goes. That bookkeeping lives in struct bpf_reg_state and is updated by routines like check_reg_arg in kernel/bpf/verifier.c — the same engine that prints the R0=mem_or_null text you’ll see in Break 1. (You’ll meet the Verifier as a character on Day 4; today you just need to recognize the register names when it complains.)

One more distinction that saves confusion later: there are two different instruction streams in play. The BPF bytecode is what the verifier reads and what struct bpf_insn encodes. The native machine code is what actually runs on the CPU — and a separate stage, the JIT, produces it. Don’t conflate them.

The bpf() syscall: one door, many commands

Everything — creating a map, loading a program, attaching it — enters the kernel through a single syscall: bpf(). There is exactly one entry point, SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size) at kernel/bpf/syscall.c:6385, and it’s multiplexed by a command enum (the first argument, cmd):

• BPF_MAP_CREATE — create a map and return a file descriptor. Dispatches to map_create() (kernel/bpf/syscall.c:1362).
• BPF_PROG_LOAD — submit a program’s instructions + BTF, run the Verifier, and return a file descriptor. Dispatches to bpf_prog_load() (kernel/bpf/syscall.c:2864).
• BPF_LINK_CREATE / the various attach commands — wire a loaded program to an attach point.

libbpf issues these for you, in sequence: create the maps, then load the program, then attach. (The command names live in include/uapi/linux/bpf.h.)

That map creation and program loading are separate commands is the whole reason Break 2 behaves the way it does. When you give the map the wrong type, the kernel rejects it during BPF_MAP_CREATE — before BPF_PROG_LOAD is ever issued, so the Verifier (which lives inside the load path) never even runs. “Fails at the loader, before the verifier” is not hand-waving; it’s literally a different cmd.

And notice the recurring noun: file descriptor. Maps, programs, and links are all referenced by fd returned from bpf(). That’s the substrate under everything you’ll touch in userspace — skel->maps.rb and bpf_map__fd(skel->maps.rb) in hello.c are just typed wrappers around a map fd. A map, in general, is a kernel-resident typed key/value object created by BPF_MAP_CREATE and shared between kernel and userspace by that fd. Ringbuf (below) is one map type; BPF_MAP_TYPE_ARRAY (the one Break 2 mis-uses) is another.

JIT: BPF bytecode becomes native machine code

Once the Verifier accepts your bytecode, the kernel doesn’t interpret it event by event — that would be slow. Instead the BPF JIT (“just-in-time” compiler) rewrites each BPF instruction into the equivalent native CPU instructions (x86-64 here). The translator is bpf_int_jit_compile() at arch/x86/net/bpf_jit_comp.c:3718; it walks the program in do_jit() (around :1652), emitting native code into a buffer described by struct jit_context (:310).

Three things to internalize:

• The JIT runs once, at load time, inside the BPF_PROG_LOAD path — not on every event. When the attach point fires, it jumps straight into already-compiled native code.
• On most production kernels the JIT is on by default, controlled by the bpf_jit_enable sysctl; setting it to 2 additionally dumps the JIT image for debugging (if (bpf_jit_enable > 1) at arch/x86/net/bpf_jit_comp.c:3843). An interpreter fallback exists, but it’s the slow path.
• This is why fentry + JIT is cheap enough to leave attached on a busy machine — the “no hiccup” claim from the first Dumb Question. At the attach point, there’s no trap and no interpreter loop: just native code calling native code.

(In that lifecycle picture, the JIT stage sits between “Verifier accepts” and “runs at attach point”: it turns the verified BPF bytecode into native x86 instructions, which is what actually executes when filename_unlinkat is called.)

The Verifier (a recurring character)

The Verifier: Hi. I’m the gatekeeper. Nothing runs in the kernel until I prove it’s safe. I read your program one instruction at a time. I track every register’s type. I track every memory region’s bounds. I track every reference you take. If you do something I can’t prove safe — touch a pointer that might be NULL, run a loop I can’t bound, leak a refcount — I reject your program. My error messages aren’t always pretty, but every one tells you the exact instruction where I lost faith. Read carefully. We’ll be working together a lot.

You’ll meet the Verifier on Day 4. For now, write code that pleases it without trying. (And now you know what “track every register’s type” means literally — R0 through R10, one type each, updated instruction by instruction.)

SEC() — the section-name convention

Every BPF program belongs to a section in the compiled object. The section name tells libbpf how to load and attach the program. The prefix is the program type; the suffix is the attach target:

SEC("fentry/filename_unlinkat")    // type=fentry, attach to kernel symbol filename_unlinkat
SEC("xdp")                    // type=xdp, attach point provided by userspace
SEC("tp/sched/sched_switch")  // type=tracepoint, on the sched_switch tracepoint

The full table lives in tools/lib/bpf/libbpf.c — search static const struct bpf_sec_def section_defs[] (:9987). Bookmark it. Whenever a SEC() “doesn’t work”, that’s the file you check.

BTF (BPF Type Format)

BTF is a compact debug-info-like format the kernel exposes about itself. The running kernel ships its complete type information at /sys/kernel/btf/vmlinux. BPF objects ship BTF about their own types. BTF powers four things you’ll use constantly:

1. Verifier type-checking of typed pointers (PTR_TO_BTF_ID).
2. CO-RE field-offset relocation at load time.
3. Typed argument unpacking for fentry/fexit/tp_btf programs — the ABI bridge from earlier, made concrete: it’s what lets BPF_PROG(on_unlink, int dfd, struct filename *name) resolve to the right slots and casts.
4. Kfunc signature matching (Day 20).

Source: kernel/bpf/btf.c.

vmlinux.h

A C header generated from kernel BTF, containing the kernel types a BPF program can name. You can generate one from a running host with bpftool btf dump file /sys/kernel/btf/vmlinux format c.

The repo-owned labs compile against the architecture header pinned with libbpf-bootstrap, not a file regenerated on every reader’s machine. That gives CI and local builds one stable type universe; at load time CO-RE still resolves the accessed fields against the running kernel’s /sys/kernel/btf/vmlinux. If you are developing outside this scaffold, generating vmlinux.h from a known reference kernel is the equivalent workflow.

CO-RE — Compile Once, Run Everywhere

The whole reason BPF programs aren’t fragile across kernel versions. You compile once against vmlinux.h from one kernel; libbpf relocates field offsets at load time using the target kernel’s BTF. The same .bpf.o runs on kernel 6.6 and kernel 7.1 even if struct layouts changed.

libbpf

The userspace library at tools/lib/bpf/. Opens your .bpf.o, applies CO-RE relocations using the running kernel’s BTF (bpf_object__relocate_core at tools/lib/bpf/libbpf.c:6082), calls the bpf() syscall to load programs and create maps — issuing BPF_MAP_CREATE then BPF_PROG_LOAD then the attach commands, in that order — attaches them, and returns handles. It is the canonical loader. Do not write your own.

Skeleton (*.skel.h)

An auto-generated header produced by bpftool gen skeleton hello.bpf.o > hello.skel.h. It gives userspace typed accessors for every map and program in your BPF object: skel->maps.rb, skel->progs.on_unlink. Under the hood each of those is a wrapper around the fd that bpf() returned. The generated file is ~200 lines of straightforward code — open it once, it stops feeling magic.

ringbuf — the event channel you’ll use today

A kernel→userspace event channel, and one map type among many. Multi-producer (any CPU), single-consumer (one userspace reader), preserves cross-CPU ordering. Producers don’t run fully lock-free: inside __bpf_ringbuf_reserve they serialize briefly via an internal per-ringbuf spinlock (raw_res_spin_lock_irqsave on rb->spinlock, kernel/bpf/ringbuf.c:478) while advancing the producer position, then each caller writes into its own disjoint slice.

Two APIs:

• bpf_ringbuf_output(&rb, data, sz, 0) — copy-style. Easy.
• bpf_ringbuf_reserve(&rb, sz, 0) → write directly → bpf_ringbuf_submit(ev, 0) — zero-copy. Preferred.

Replaced perfbuf (BPF_MAP_TYPE_PERF_EVENT_ARRAY) for most uses. Perfbuf is per-CPU, requires N userspace consumers, loses ordering across CPUs. Use perfbuf only if peak per-CPU write rate matters more than ordering. Source: kernel/bpf/ringbuf.c.

The lab

Setup (one time)

Start with the Lab environment page, then run these commands from the books repository on your Linux lab host:

git submodule update --init --recursive
cd ebpf/labs
./scripts/preflight.sh
make hello

The repository pins libbpf-bootstrap and all of its nested dependencies. The shared Makefile builds that exact libbpf and bpftool, selects the pinned vmlinux.h for your architecture, compiles day01/hello.bpf.c, generates .output/day01/hello.skel.h, and links .output/day01/hello. Nothing is installed system-wide.

You still need Clang 17 or newer with -target bpf, a C compiler and Make, plus libelf and zlib development headers. preflight.sh checks those inputs but never installs packages or invokes sudo.

hello.h — the shared event record

The producer and consumer include one header, so a layout change cannot silently desynchronize the two sides:

#ifndef PRACTICAL_EBPF_HELLO_H
#define PRACTICAL_EBPF_HELLO_H

struct hello_event {
    unsigned int pid;
    char comm[16];
};

#endif

hello.bpf.c — the kernel side

This listing is included from the file the lab build and CI compile:

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

#include "hello.h"

char LICENSE[] SEC("license") = "GPL";

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

SEC("fentry/filename_unlinkat")
int BPF_PROG(on_unlink, int dfd, struct filename *name)
{
    struct hello_event *event;

    (void)dfd;
    (void)name;

    event = bpf_ringbuf_reserve(&rb, sizeof(*event), 0);
    if (!event)
        return 0;

    event->pid = bpf_get_current_pid_tgid() >> 32;
    bpf_get_current_comm(&event->comm, sizeof(event->comm));
    bpf_ringbuf_submit(event, 0);
    return 0;
}

Walkthrough of every line that’s new:

• #include "vmlinux.h" — pulls in every kernel type, including struct filename used in the prototype.
• char LICENSE[] SEC("license") = "GPL"; — a load-time gate, not legal advice. The kernel rejects a non-GPL program only if it calls a GPL-only helper. Many of the most useful helpers are GPL-only (bpf_probe_read_kernel, bpf_get_current_task, bpf_get_stackid, …), but the four simple helpers this lab uses are not — so this line has no teeth yet today (see Break 3).
• #include "hello.h" — the shared struct hello_event sent through ringbuf. Ringbuf transports raw bytes, so both sides must agree on this layout.
• The SEC(".maps") block — modern map declaration syntax. The __uint(...) macros from bpf_helpers.h produce BTF the loader uses to know it’s a 256-KiB ringbuf. (This is the BTF that drives the BPF_MAP_CREATE command libbpf issues for rb.)
• SEC("fentry/filename_unlinkat") — attach point. filename_unlinkat is in fs/namei.c, called on every unlink() and unlinkat() syscall.
• BPF_PROG(on_unlink, int dfd, struct filename *name) — macro from bpf_tracing.h that unpacks the trampoline’s argument array into the typed parameters dfd and name (the ABI bridge from earlier, made concrete).
• bpf_ringbuf_reserve returns either a valid pointer or NULL (when the ringbuf is full). The return value lands in R0, the Verifier tags it mem_or_null, and it requires the null check before you write through it.
• bpf_get_current_pid_tgid() — reads the currently-running task (more on “the task” just below). The result is packed (tgid << 32) | pid; Linux’s userspace “PID” is the kernel’s TGID. >> 32 extracts the user-visible PID.
• bpf_get_current_comm — copies up to 16 bytes of the task’s comm field. Always 16, always null-padded.
• bpf_ringbuf_submit makes the reserved entry visible to the consumer.

A one-paragraph refresher: “the task” and where 16 comes from

Two of those helpers read “the current task,” and the magic number 16 shows up with no explanation — so, briefly: every schedulable thread in the kernel is represented by a struct task_struct. What BPF calls current is simply the task_struct of the thread running on this CPU right now, and bpf_get_current_pid_tgid / bpf_get_current_comm both read fields out of it. The comm field is a fixed-size, NUL-padded short thread name — the rm you’ll see in the lab output — declared as char comm[TASK_COMM_LEN] in struct task_struct (include/linux/sched.h:1173). And TASK_COMM_LEN is 16 (include/linux/sched.h:325). So the 16 in sizeof(event->comm) isn’t arbitrary; it’s the kernel’s compile-time size of that field. That’s the whole refresher — we’ll do a proper task_struct tour another day; today you just need “always 16” and “the task’s comm” to stop being unexplained constants. (The pid-vs-tgid split above already covered why we shift by 32.)

hello.c — the userspace side

The loader checks each libbpf step, validates the ringbuf record size, handles SIGINT/SIGTERM, and releases both the consumer and skeleton on every exit path:

#include <errno.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>

#include <bpf/libbpf.h>

#include "hello.h"
#include "hello.skel.h"

static volatile sig_atomic_t exiting;

static void handle_signal(int signal_number)
{
    (void)signal_number;
    exiting = 1;
}

static int install_signal_handlers(void)
{
    const struct sigaction action = {
        .sa_handler = handle_signal,
    };

    if (sigaction(SIGINT, &action, NULL) != 0 ||
        sigaction(SIGTERM, &action, NULL) != 0) {
        fprintf(stderr, "failed to install signal handlers: %s\n",
                strerror(errno));
        return -errno;
    }

    return 0;
}

static int handle_event(void *ctx, void *data, size_t size)
{
    const struct hello_event *event = data;

    (void)ctx;
    if (size != sizeof(*event)) {
        fprintf(stderr, "unexpected ringbuf record size: %zu\n", size);
        return 0;
    }

    printf("PID %u %.*s deleted a file\n", event->pid,
           (int)sizeof(event->comm), event->comm);
    return 0;
}

int main(void)
{
    struct hello_bpf *skel = NULL;
    struct ring_buffer *ringbuf = NULL;
    int err;

    setvbuf(stdout, NULL, _IOLBF, 0);

    err = install_signal_handlers();
    if (err)
        return 1;

    skel = hello_bpf__open();
    if (!skel) {
        fprintf(stderr, "failed to open hello BPF skeleton\n");
        return 1;
    }

    err = hello_bpf__load(skel);
    if (err) {
        fprintf(stderr, "failed to load hello BPF programs: %s\n",
                strerror(-err));
        goto cleanup;
    }

    err = hello_bpf__attach(skel);
    if (err) {
        fprintf(stderr, "failed to attach hello BPF programs: %s\n",
                strerror(-err));
        goto cleanup;
    }

    ringbuf = ring_buffer__new(bpf_map__fd(skel->maps.rb), handle_event,
                               NULL, NULL);
    err = libbpf_get_error(ringbuf);
    if (err) {
        ringbuf = NULL;
        fprintf(stderr, "failed to create ringbuf consumer: %s\n",
                strerror(-err));
        goto cleanup;
    }

    puts("Tracing unlink calls; press Ctrl-C to stop.");
    while (!exiting) {
        err = ring_buffer__poll(ringbuf, 100);
        if (err == -EINTR) {
            err = 0;
            continue;
        }
        if (err < 0) {
            fprintf(stderr, "ringbuf poll failed: %s\n", strerror(-err));
            goto cleanup;
        }
    }

    err = 0;

cleanup:
    ring_buffer__free(ringbuf);
    hello_bpf__destroy(skel);
    return err != 0;
}

bpf_map__fd(skel->maps.rb) is exactly the map fd that BPF_MAP_CREATE returned during load — the userspace side polls it; the kernel side writes into it.

Run it

make hello
sudo ./.output/day01/hello
# in another terminal:
scratch=$(mktemp -d /tmp/ebpf-day01.XXXXXX)
touch "$scratch/one" "$scratch/two"
rm "$scratch/one" "$scratch/two"
rmdir "$scratch"

Expected output:

PID 13421 rm deleted a file
PID 13421 rm deleted a file

On a busy machine you’ll also see lines whose comm is something other than rm — package managers, editors saving over temp files, /tmp churn. That’s not a bug; it’s the whole point. Your doorbell fires on every unlink/unlinkat in the system, not just yours. Pick out the two lines whose comm is rm.

Congratulations. You just installed a doorbell on a kernel function.

What to break, in order

This is the part you don’t skip. Every break teaches one concept.

Break 1 — Drop the null check

Remove the if (!event) return 0;. Rebuild. The verifier rejects with something like:

0: (85) call bpf_ringbuf_reserve#131
1: R0=mem_or_null(id=2,sz=20)
2: (7b) *(u32 *)(r0 +0) = r1
R0 invalid mem access 'mem_or_null'

That’s the Verifier saying: your register R0 is a pointer that might be NULL; you can’t dereference it without proving it isn’t. You now know exactly why it’s R0 and not some other register — R0 is where a helper’s return value lives by calling convention, so bpf_ringbuf_reserve’s maybe-NULL pointer is sitting in R0 when instruction 2 tries to write through it.

To see this with full detail, set kernel_log_level = 1 in your loader options:

LIBBPF_OPTS(bpf_object_open_opts, opts, .kernel_log_level = 1);
skel = hello_bpf__open_opts(&opts);

Or use bpftool prog load file.o /sys/fs/bpf/x and read stderr.

Break 2 — Wrong map type

Change BPF_MAP_TYPE_RINGBUF to BPF_MAP_TYPE_ARRAY. Now the loader fails before the Verifier even runs:

libbpf: map 'rb': failed to create: Invalid argument

This is the BPF_MAP_CREATE-vs-BPF_PROG_LOAD split from the syscall section, made visible. An ARRAY map needs a key size and value size that a bare SEC(".maps") ringbuf-style declaration doesn’t supply, so the kernel rejects it during BPF_MAP_CREATE — the very first command libbpf issues. BPF_PROG_LOAD, where the Verifier lives, never runs. That’s the difference between loader-time errors (map config wrong) and verifier-time errors (program logic unsafe). Different layers, different commands, different failure modes. Get used to noticing which one bit you.

Break 3 — Remove the LICENSE

Try the obvious thing first: delete the char LICENSE[] SEC("license") = "GPL"; line and rebuild. The program loads and runs exactly as before. Surprised? The GPL gate fires only when your program calls a GPL-only helper, and none of the four helpers in hello.bpf.c (bpf_ringbuf_reserve, bpf_ringbuf_submit, bpf_get_current_pid_tgid, bpf_get_current_comm) are GPL-only.

To make the gate bite, first give it something to gate on. Add a GPL-only helper call inside on_unlink:

(void)bpf_get_current_task();   // bpf_get_current_task is a GPL-only helper

Now delete the LICENSE line again and rebuild. This time the load fails:

cannot call GPL-restricted function from non-GPL compatible program

The lesson: the license string is a load-time gate with teeth only for GPL-only helpers. Put the LICENSE line back (and you can drop the bpf_get_current_task call again) before moving on.

Break 4 — Reserve more than you write

Sharpen your pencil

Before you run anything: what is sizeof(struct hello_event)? The struct is a 4-byte __u32 pid followed by a 16-byte char comm[16]. Add them up — you’ll see this number printed as size= in a moment.

. . .

Answer: 20. There’s no padding to worry about here — a __u32 needs 4-byte alignment and char[16] needs only 1, so the 4 + 16 layout packs with no gaps. Hold “20” in mind; the 20 → 21 shift is the whole point of this break.

The lesson here is that ringbuf records are sized at reserve time, not at submit time. The checked-in handle_event() deliberately rejects any size other than sizeof(*event), so replace that equality check with a lower-bound check and print the delivered size:

if (size < sizeof(*event)) {
    fprintf(stderr, "short ringbuf record: %zu\n", size);
    return 0;
}
printf("PID %u %.*s deleted a file (size=%zu)\n", event->pid,
       (int)sizeof(event->comm), event->comm, size);

Rebuild and run the unmodified program. Each line now ends with the record size:

PID 13421 rm deleted a file (size=20)

sizeof(struct hello_event) is 20 — a 4-byte pid plus a 16-byte comm. Now over-allocate the reservation while still writing only sizeof(*event) bytes:

event = bpf_ringbuf_reserve(&rb, sizeof(*event) + 1, 0);

The Verifier accepts it (you reserved enough). Rebuild, run, delete a file again:

PID 13421 rm deleted a file (size=21)

The record is one byte larger even though you wrote the same bytes — the length is fixed when you reserve, not when you submit. (That trailing byte is uninitialized; we don’t print it, the 20 → 21 size delta is the observable signal.) Restore the original sizeof(*event) reservation and exact-size consumer check before moving on.

What to read in the kernel

Open these files in your ~/code/linux checkout. Skim, don’t memorize.

• kernel/bpf/trampoline.c — top of the file. Find arch_prepare_bpf_trampoline and bpf_trampoline_get. The trampoline is per-(target, prog-list), reference-counted, and rebuilt when programs attach or detach (bpf_trampoline_update, kernel/bpf/trampoline.c:607). Notice bpf_trampoline is not just one stub — it’s a hash table keyed by attach target. (The arg-spill/restore code itself is emitted by arch_prepare_bpf_trampoline at arch/x86/net/bpf_jit_comp.c:3536, with the real work in __arch_prepare_bpf_trampoline at :3213 — this is the ABI bridge from the trampoline section, in the flesh.) The patch that repoints the site is installed by the int3-based text-poke machinery, smp_text_poke_* (smp_text_poke_int3_handler at arch/x86/kernel/alternative.c:2838, driving the batched struct smp_text_poke_loc array at :2781) — safe against concurrent instruction fetch on every CPU.
• kernel/bpf/ringbuf.c — bpf_ringbuf_reserve is short. Note the per-record header (struct bpf_ringbuf_hdr, :88) and the BUSY bit (BPF_RINGBUF_BUSY_BIT, set at :529) that lets submit and discard finalize race-free.
• tools/lib/bpf/libbpf.c — search find_sec_def (:10212). Scroll the section_defs[] table (:9987). You now know every prefix that exists. This file is also where CO-RE relocation gets kicked off (bpf_object__relocate_core, :6082).

Optional: read the generated skeleton

./.output/bpftool/bootstrap/bpftool gen skeleton .output/day01/hello.bpf.o

It prints the same skeleton stored at .output/day01/hello.skel.h. Read it once. You’ll see it’s just stamped-out boilerplate calling bpf_object__find_map_by_name and friends. Demystified.

Bullet Points

• BPF is a 64-bit virtual ISA: 11 registers R0–R10, fixed 8-byte struct bpf_insn. R0 = return/helper-return value, R1–R5 = args, R6–R9 = callee-saved, R10 = read-only frame pointer to a 512-byte stack. The Verifier tracks a type per register — that’s where R0=mem_or_null comes from.
• bpf() is one syscall, multiplexed by a command enum: BPF_MAP_CREATE (→ map fd), BPF_PROG_LOAD (runs the Verifier → prog fd), attach commands. Maps, progs, and links are all fds. libbpf issues them in sequence — which is why a bad map fails before the Verifier (Break 2).
• After the Verifier accepts bytecode, the JIT translates it to native machine code once at load time; the attach point then runs native code at native speed.
• BPF programs are C compiled to a verified-then-JITed instruction set; they cannot crash the kernel.
• fentry patches the 5-byte NOP slot — reserved by the compiler’s -mfentry instrumentation and owned by ftrace, borrowed by BPF via an ftrace “direct call” — with a jump to a generated trampoline. The trampoline bridges the ABI: it spills native arg registers (rdi, rsi, …) into a u64 ctx[] array, calls your program with R1 → ctx, and BPF_PROG casts the slots back to typed args. Typically several times cheaper than a kprobe trap.
• The patch is installed by the kernel’s int3-based text-poke machinery, smp_text_poke_*, safe against concurrent instruction fetch on every CPU.
• Use fentry wherever possible. Use kprobe only for functions without BTF.
• SEC() is the section-name convention libbpf uses to load and attach. The prefix names the program type and tells libbpf where to attach.
• vmlinux.h is generated from kernel BTF and gives you every kernel type by name; BTF also tells fentry the argument types so the trampoline and BPF_PROG agree on the slots.
• CO-RE lets you compile once and run on any kernel; libbpf relocates field offsets at load time.
• ringbuf is one map type — multi-producer single-consumer with cross-CPU ordering — your default kernel→userspace channel.
• current is the running thread’s struct task_struct; comm is its 16-byte (TASK_COMM_LEN) NUL-padded name — that’s where “always 16” comes from.
• Two failure layers exist: loader-time (map/object misconfigured) and verifier-time (program logic unsafe).
• Always check the return of bpf_ringbuf_reserve and bpf_map_lookup_elem — both can return NULL (in R0) and the Verifier knows it.

Check question

If two CPUs concurrently call bpf_ringbuf_reserve(&rb, 64, 0), can the records overlap?

Tomorrow

Day 2: add a hash map. Per-PID counters. The first lesson on map element lifecycle (and how a tracer that doesn’t clean up its maps becomes a slow ENOSPC poison-pill).

Day 2 — Hash maps and per-PID counters

Today’s mission: count unlink() calls per PID, dump the running totals on demand. Learn what a hash map is — the bucket array, the lockless lookup, the per-CPU variant — what the Verifier knows about lookups, how userspace reaches into kernel map memory through a file descriptor, and why your tracer must clean up after itself or it dies. Total time: ~110 minutes.

The shared notebook

Yesterday you sent events to userspace one at a time. That works for streaming, but doesn’t help when you want a summary — “how many times did each PID call unlink?”

You want a shared notebook between kernel and userspace. The BPF program writes counts. Userspace reads them when it wants. The notebook lives in kernel memory, persists across program invocations, and survives until you tear it down.

That’s a map. Today’s flavor: BPF_MAP_TYPE_HASH.

But “shared notebook” hides three things that will bite you if you don’t understand them, and the rest of today is about making them obvious:

1. How userspace even touches a kernel map. The BPF side and the userspace side talk to the same map through two completely different doors — and, confusingly, through two functions with the same name.
2. Why the kernel can hand your BPF program a raw pointer into map memory and trust you to dereference it without ever taking a lock. (Spoiler: RCU.)
3. Why a per-CPU map dodges races entirely — which means knowing what “per-CPU memory” actually is.

We’ll teach each as we hit the part of the lab that depends on it.

Meet the cast

BPF_MAP_TYPE_HASH — the bread-and-butter map

A kernel-resident hash table. Fixed-capacity (max_entries), key/value sizes declared at creation, accessible from both BPF and userspace. Internally it’s a power-of-two-sized array of buckets, each holding a chained list of {key, value} pairs. Lookup hashes the key, masks it down to a bucket index, and walks the chain.

You can see every one of those words in the source. In kernel/bpf/hashtab.c:

/* kernel/bpf/hashtab.c:80 */
struct bucket {
    struct hlist_nulls_head head;   /* the chain of elements */
    rqspinlock_t            raw_lock;
};

The bucket index is just the hash masked to the array size — and because the array is a power of two, that mask is a single &:

/* kernel/bpf/hashtab.c:686 — select_bucket() */
return &__select_bucket(htab, hash)->head;   /* index = hash & (n_buckets - 1) */

The bits worth remembering today:

• Lookup returns a pointer into map memory — not a copy. You can read it. You can atomically update it. What you cannot do is assume it’s non-NULL. And while the element won’t be freed under you (RCU guarantees that — see below), its contents can change concurrently, so bumping a counter needs an atomic.
• max_entries is a hard cap. Once full, inserts fail with -E2BIG (is_map_full → ERR_PTR(-E2BIG) at hashtab.c:1108). There is no automatic eviction. (LRU and LRU-percpu variants exist if you want eviction; see Day 11.)
• No locks for you. The kernel handles bucket-level synchronization internally — we’ll see exactly how in a moment, because it’s the reason the returned pointer is safe to dereference. For atomic value updates, you use __sync_fetch_and_add and friends on the value pointer.
• Per-CPU variant (BPF_MAP_TYPE_PERCPU_HASH): same shape, but each CPU has its own value slot. No atomics needed; userspace must sum across CPUs to get a total. Use it when contention matters (high-rate counters, e.g. XDP).

Source: kernel/bpf/hashtab.c. The lookup is htab_map_lookup_elem (:752); the update is htab_map_update_elem (:1171). Look at the file once. It’s not magic — and by the end of today you’ll have read the parts that matter.

Two doors into one map: the map fd, and the two functions both called bpf_map_lookup_elem

Here is a thing the tutorials never tell you, and it will confuse you the first time you put the BPF side and the userspace side on one screen: bpf_map_lookup_elem is the name of two completely different functions with opposite contracts. You will write both today, in two different files. Let’s separate them now so you never trip on it.

First: what is a map, from each side? A loaded map is a single in-kernel object — that bucket array we just looked at. Neither the BPF program nor your userspace dumper holds a pointer to it directly. They reach it two different ways:

• The BPF program refers to the map by address of its declaration (&counts). At load time the kernel rewrites that into a direct reference to the kernel object. The program runs inside the kernel, so when it asks for an element it gets a bare pointer straight into live map memory.
• Userspace cannot hold a kernel pointer. It refers to the map through an integer file descriptor — the same kind of small int as an open file. bpf_map__fd(skel->maps.counts) pulls that fd out of the skeleton. (You used the exact same bpf_map__fd for the ringbuf yesterday and I never told you what the int was — now you know: it’s a handle to a kernel object, nothing more.)

Now the two functions. They have the same name and do almost-opposite things:

The userspace side is a thin wrapper. Open tools/lib/bpf/bpf.c and you’ll find it does nothing but fill a struct and issue a syscall:

/* tools/lib/bpf/bpf.c:407 */
int bpf_map_lookup_elem(int fd, const void *key, void *value)
{
    union bpf_attr attr;
    memset(&attr, 0, attr_sz);
    attr.map_fd = fd;
    attr.key    = ptr_to_u64(key);
    attr.value  = ptr_to_u64(value);          /* a buffer to copy INTO */
    ret = sys_bpf(BPF_MAP_LOOKUP_ELEM, &attr, attr_sz);
    return libbpf_err_errno(ret);
}

Every userspace map call follows this exact pattern — fill a union bpf_attr, call sys_bpf(...):

• bpf_map_update_elem(fd, …) → sys_bpf(BPF_MAP_UPDATE_ELEM, …) (bpf.c:390)
• bpf_map_delete_elem(fd, …) → sys_bpf(BPF_MAP_DELETE_ELEM, …) (bpf.c:469)
• bpf_map_get_next_key(fd, …) → sys_bpf(BPF_MAP_GET_NEXT_KEY, …) (bpf.c:498, syscall issued at :509)

That last one, bpf_map_get_next_key, is how userspace iterates a map: call it with one key, it hands you the next. There’s no single syscall behind bpf_map_get_next_key that walks the whole map for you — it hands back one key per call, so the get_next_key loop lives in userspace. (A separate batch API, BPF_MAP_LOOKUP_BATCH, can pull many entries per syscall — that’s what bpftool map dump uses — but it’s not what get_next_key does.) This is why the dump loop in count.c below is an ordinary userspace while loop, not BPF code.

So when you see bpf_map_lookup_elem on the BPF side returning a pointer you dereference, and bpf_map_lookup_elem on the userspace side copying into a buffer you read — same name, opposite contract. Keep them straight and the rest of today is easy.

Why the kernel trusts you with a raw pointer: RCU lookups and a per-bucket lock

Back to a claim I made above and then walked past: lookup returns a pointer into live map memory, no lock required, and you may dereference it. Stop and feel how scary that should be. Another CPU could be inserting or deleting in that same bucket at the same instant. How is it safe for your program to walk the chain and dereference an element with no lock at all?

The answer is two different synchronization mechanisms for two different jobs — and knowing which is which explains every “do I need an atomic here?” question you’ll ever ask about hash maps.

Job 1 — reads (lookup): RCU, no lock. RCU (Read-Copy-Update) is a kernel synchronization scheme where readers never take a lock and never block, and freeing is deferred until every in-flight reader has finished — those two guarantees are all we need here. Each bucket’s chain isn’t a plain linked list; it’s an RCU-protected hlist_nulls (struct hlist_nulls_head head; in that struct bucket). Two properties make a lockless walk safe:

• Readers run in an RCU read-side critical section and take no lock. That is literally why the helper can hand your program a raw pointer — there’s no lock to hold or drop. The source even labels the two walkers by whether they need the lock:

  /* hashtab.c:691 — needs the bucket lock */
  /* this lookup function can only be called with bucket lock taken */
  static struct htab_elem *lookup_elem_raw(...)
  
  /* hashtab.c:705 — the RCU read-side walk, used by the lookup helper */
  /* can be called without bucket lock. it will repeat the loop in
   * the unlikely event when elements moved from one bucket into another
   * while link list is being walked */
  static struct htab_elem *lookup_nulls_elem_raw(...)
  

• Elements are RCU-freed. When a writer removes an element, the memory isn’t reclaimed until all in-flight readers have finished. So the element your program is looking at cannot be deallocated out from under you while your program runs. Your null-check-then-dereference is racing only against the contents of the value (which is why you still need an atomic to bump a counter), never against use-after-free. That _nulls flavor of the list exists precisely so a lockless reader can detect “this element hopped to another bucket mid-walk” and restart the loop (the goto again in lookup_nulls_elem_raw).

Job 2 — writes (insert / replace / delete): the per-bucket spinlock. That second field in the bucket, rqspinlock_t raw_lock, is taken only by writers. htab_map_update_elem grabs it before touching the chain:

/* hashtab.c:1217, inside htab_map_update_elem */
ret = htab_lock_bucket(b, &flags);

This is the “bucket-level synchronization” I mentioned, and the “bucket lock” the Check question at the bottom names. Three things to notice:

• It’s one lock per bucket, so two writes to different keys that land in different buckets never contend. Contention is only between keys that collide into the same bucket.
• It serializes writers against each other and against sys_bpf() calls from userspace — the file’s big block comment above struct bucket (hashtab.c:35) spells out both scopes (“Serializing concurrent operations from BPF programs on different CPUs” and “between BPF programs and sys_bpf()”).
• It’s a raw, resilient queued spinlock (rqspinlock_t) on purpose: BPF programs run in atomic contexts (perf, kprobes, tracing), where you cannot take a sleeping lock. A raw spinlock is safe to grab from there. The comment block walks through exactly why that choice is forced.

The punchline you’ll use constantly: lookup gives you an unlocked live pointer; the bucket lock protects insert/delete, not your writes to the value. So two CPUs that both looked up the same slot and both want to += 1 are not protected by anything — which is exactly why the counter increment must be an atomic __sync_fetch_and_add. Different paths, different synchronization.

BPF_MAP_TYPE_PERCPU_HASH — and what “per-CPU” actually means

The check-question answer at the bottom recommends a per-CPU hash to dodge the increment race entirely. To understand why it dodges the race, you need to know what “per-CPU memory” means — nothing earlier in this book has taught it, so here it is.

The intuition. A per-CPU value is not one slot shared by everyone. The kernel allocates one independent copy of the value for every logical CPU. When a BPF program reads or writes that value, it transparently touches only the copy belonging to the CPU it is currently running on. Two CPUs working on “the same key” are physically touching different memory. There is no sharing to race over.

That single fact has three consequences:

• No atomics, no bucket contention on the value. Each CPU bumps its private copy with an ordinary load/store. You’ve traded kernel-side contention for userspace work.

• Read-out costs more. When userspace calls bpf_map_lookup_elem(fd, &key, …) on a per-CPU map, the value it copies out is an array — one entry per CPU. To get the logical total you must sum across all CPUs. The single-__u64 dumper in count.c below would have to change to read an array and add it up.

• Values are size-bounded. A per-CPU value is allocated from a special per-CPU allocator (the pcpu_ma in struct bpf_htab), and the rounded-up value size must fit PCPU_MIN_UNIT_SIZE:

  /* hashtab.c:458, in htab_map_alloc_check */
  /* percpu map value size is bound by PCPU_MIN_UNIT_SIZE */
  if (percpu && round_up(attr->value_size, 8) > PCPU_MIN_UNIT_SIZE)
      return -E2BIG;
  

  A regular hash value isn’t bounded this way — so if your values get large, per-CPU has a real ceiling.

When to use which. Reach for BPF_MAP_TYPE_PERCPU_HASH (include/uapi/linux/bpf.h:1005) when the hot path is contended — high-rate counters, XDP per-packet stats. Reach for a plain BPF_MAP_TYPE_HASH (:1001) plus __sync_fetch_and_add when contention is low and you want one authoritative slot you can read without summing. (BPF_MAP_TYPE_LRU_HASH at :1009 is a third option when you also want eviction; Day 11.)

PTR_TO_MAP_VALUE_OR_NULL — your first taste of the Verifier’s type system

When you call bpf_map_lookup_elem, the Verifier marks the returned register with type PTR_TO_MAP_VALUE_OR_NULL. That type means: might be a valid pointer to a map value, might be NULL — I don’t know yet.

You cannot dereference a pointer of that type. Until you prove it’s non-NULL, the Verifier rejects every load through it.

How do you prove it? With a null check:

__u64 *cnt = bpf_map_lookup_elem(&counts, &pid);
if (!cnt)
    return 0;
*cnt += 1;   // ← now legal: cnt is PTR_TO_MAP_VALUE

Inside the if (!cnt) branch, the Verifier knows cnt == NULL and pushes that branch into “this code is unreachable past this return.” Outside the branch, it knows cnt != NULL — it transitions the register’s type from PTR_TO_MAP_VALUE_OR_NULL to PTR_TO_MAP_VALUE. The deref becomes legal.

You’ll deepen this on Day 4. For today: always check the lookup result against NULL. (And note the connection back to RCU: the reason a non-NULL PTR_TO_MAP_VALUE is safe to dereference at all is that the element is RCU-freed — the Verifier guarantees you checked for NULL; RCU guarantees the memory won’t vanish mid-program.)

There are no Dumb Questions

Q: Why does the kernel make me check for NULL? Why not just return a sentinel value or pre-allocate?

A: Pre-allocation would mean the map guarantees every key has a value, which is exactly what BPF_MAP_TYPE_ARRAY does (with integer keys). For hash maps, the answer “this key isn’t in the map” is meaningful — you need to know whether to insert or update. Sentinels would require the kernel to runtime-check every dereference. The Verifier’s static check pushes that cost to compile time, where it costs nothing at runtime.

Q: I’m tempted to use a global variable instead of a map. Why not?

A: Global variables in BPF programs do exist (.bss/.data sections backed by BPF_MAP_TYPE_ARRAY of size 1 under the hood — see Day 7). They’re great for config and singletons. But for per-key state (counts per PID, latencies per syscall, denylist of IPs), you need a hash. Globals can’t be looked up by key.

Q: How does the kernel pick a hash function?

A: htab_map_hash (kernel/bpf/hashtab.c:674) computes the hash — it calls jhash2 when the key size is a multiple of 4 bytes (key_len % 4 == 0) and jhash otherwise (both are Jenkins-hash variants). The bucket index is then hash & (n_buckets - 1), done in __select_bucket. The hash is fixed; you don’t get to choose. For very large keys this matters; for the 4-byte PIDs we’re using today it doesn’t.

BPF_ANY / BPF_NOEXIST / BPF_EXIST — update flags

bpf_map_update_elem takes a flag:

• BPF_ANY — insert if absent, update if present.
• BPF_NOEXIST — insert if absent, fail with -EEXIST if present.
• BPF_EXIST — update if present, fail with -ENOENT if absent.

Use BPF_NOEXIST defensively when you want to be sure you’re not stomping someone else’s entry. Use BPF_ANY when you don’t care. The semantics live in check_flags (hashtab.c:1156), which returns -EEXIST for the BPF_NOEXIST-but-present case and -ENOENT for the BPF_EXIST-but-absent case — and the flag values themselves are in include/uapi/linux/bpf.h:1392 (BPF_ANY=0, BPF_NOEXIST=1, BPF_EXIST=2).

__sync_fetch_and_add — atomics in BPF

BPF programs don’t have access to most C library facilities, but they do have GCC-style sync builtins. __sync_fetch_and_add(p, n) atomically adds n to *p and returns the old value. Compiles to BPF_XADD (atomic add) or BPF_ATOMIC (general atomic) instructions — both are opcode 0xc0, with BPF_XADD being the legacy name (include/uapi/linux/bpf.h:23-24). The JIT lowers these to native LL/SC or lock xadd depending on architecture.

Why use it? Because (as the RCU section just showed) the lookup pointer is unlocked and concurrent CPUs can write the same slot — without an atomic you lose counts.

The lab

Continue from Day 1 in the repository’s ebpf/labs directory. The listings below are included from the source compiled by make count and CI.

count.bpf.c — swap yesterday’s ringbuf for a hash map

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

char LICENSE[] SEC("license") = "GPL";

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 8192);
    __type(key, __u32);
    __type(value, __u64);
} counts SEC(".maps");

SEC("fentry/filename_unlinkat")
int BPF_PROG(on_unlink)
{
    __u32 pid = bpf_get_current_pid_tgid() >> 32;
    __u64 *count;
    __u64 one = 1;

    count = bpf_map_lookup_elem(&counts, &pid);
    if (count) {
        __sync_fetch_and_add(count, 1);
        return 0;
    }

    if (bpf_map_update_elem(&counts, &pid, &one, BPF_NOEXIST) != 0) {
        /* Another CPU may have inserted the same PID after our lookup. */
        count = bpf_map_lookup_elem(&counts, &pid);
        if (count)
            __sync_fetch_and_add(count, 1);
    }

    return 0;
}

What’s new:

• The map is declared just like ringbuf: a struct in SEC(".maps") whose macros set type/size/key/value through BTF.
• bpf_map_lookup_elem(&counts, &pid) — the BPF-side function from the table above. It takes the map by address and returns a live pointer. Note we pass the address of pid, not the value. The kernel reads key_size bytes starting from that address. If you accidentally passed pid (a value), you’d be telling the kernel to interpret an integer as a pointer. The Verifier catches this.
• The insert uses BPF_NOEXIST instead of BPF_ANY. If two CPUs both observe a miss, one inserts and the other gets -EEXIST; the losing path then looks up the new entry and atomically contributes its increment. BPF_ANY would instead let the loser overwrite the winner’s count back to 1. A per-CPU hash is still the better high-contention design, but this retry closes the first-insert race without changing today’s map type.

count.c — userspace dumper

Same fentry hook as Day 1, but the output channel changes from a ringbuf to a map. The loader periodically iterates the map, starts with a NULL previous key, distinguishes normal ENOENT exhaustion from iteration errors, and destroys the skeleton on SIGINT/SIGTERM:

#include <errno.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

#include <bpf/bpf.h>
#include <bpf/libbpf.h>

#include "count.skel.h"

static volatile sig_atomic_t exiting;

static void handle_signal(int signal_number)
{
    (void)signal_number;
    exiting = 1;
}

static int install_signal_handlers(void)
{
    const struct sigaction action = {
        .sa_handler = handle_signal,
    };

    if (sigaction(SIGINT, &action, NULL) != 0 ||
        sigaction(SIGTERM, &action, NULL) != 0) {
        fprintf(stderr, "failed to install signal handlers: %s\n",
                strerror(errno));
        return -errno;
    }

    return 0;
}

static int dump_counts(int map_fd)
{
    const __u32 *previous = NULL;
    __u32 key;
    __u32 next;
    __u64 value;
    int ret;

    puts("--- snapshot ---");
    errno = 0;
    while ((ret = bpf_map_get_next_key(map_fd, previous, &next)) == 0) {
        if (bpf_map_lookup_elem(map_fd, &next, &value) == 0)
            printf("PID %u: %llu unlinks\n", next,
                   (unsigned long long)value);
        else if (errno != ENOENT)
            fprintf(stderr, "lookup failed for PID %u: %s\n", next,
                    strerror(errno));

        key = next;
        previous = &key;
        errno = 0;
    }

    if (ret != 0 && errno != ENOENT) {
        fprintf(stderr, "map iteration failed: %s\n", strerror(errno));
        return -errno;
    }

    return 0;
}

int main(void)
{
    struct count_bpf *skel = NULL;
    int map_fd;
    int err;

    setvbuf(stdout, NULL, _IOLBF, 0);

    err = install_signal_handlers();
    if (err)
        return 1;

    skel = count_bpf__open();
    if (!skel) {
        fprintf(stderr, "failed to open count BPF skeleton\n");
        return 1;
    }

    err = count_bpf__load(skel);
    if (err) {
        fprintf(stderr, "failed to load count BPF programs: %s\n",
                strerror(-err));
        goto cleanup;
    }

    err = count_bpf__attach(skel);
    if (err) {
        fprintf(stderr, "failed to attach count BPF programs: %s\n",
                strerror(-err));
        goto cleanup;
    }

    map_fd = bpf_map__fd(skel->maps.counts);
    puts("Counting unlink calls; press Ctrl-C to stop.");
    while (!exiting) {
        unsigned int remaining = sleep(2);

        if (exiting)
            break;
        if (remaining != 0)
            continue;

        err = dump_counts(map_fd);
        if (err)
            goto cleanup;
    }

    err = 0;

cleanup:
    count_bpf__destroy(skel);
    return err != 0;
}

Three things to connect back to the “two doors” section:

• bpf_map__fd(skel->maps.counts) pulls the integer fd out of the skeleton — that’s userspace’s only handle on the kernel map.
• bpf_map_get_next_key(fd, previous, &next) walks the map. Calling it with a NULL previous-key pointer returns the first key; calling it with the previous key returns the next. It returns -1 with errno == ENOENT when iteration is exhausted; the checked-in loader treats other errors as failures instead of spinning forever. Each call is one sys_bpf(BPF_MAP_GET_NEXT_KEY, …). This is why the loop lives in userspace.
• bpf_map_lookup_elem(fd, &next, &val) here is the userspace function — it copies the value into val, which we then read. (Contrast with the BPF side, which got a live pointer.)

Run it

In terminal 1:

make count
sudo ./.output/day02/count

In terminal 2, generate work inside a directory this lab owns. One rm process performs all 100 unlinks, so they aggregate under one PID:

scratch=$(mktemp -d /tmp/ebpf-day02.XXXXXX)
for i in $(seq 1 100); do touch "$scratch/x$i"; done
rm "$scratch"/x*
rmdir "$scratch"
# wait 2s for the next snapshot, then Ctrl-C the loader in terminal 1

If you rm inside the loop (touch ... && rm ...), the shell forks a brand-new rm process every iteration, so each unlink comes from a different PID and the map fills with ~100 entries of value 1 — the exact opposite of what we’re demonstrating. Batching the deletes makes one process issue all 100 unlinkat calls.

Expected — the snapshot also lists other PIDs from unrelated background unlink activity (systemd, systemd-logind, etc.), so look for the line showing 100:

--- snapshot ---
PID 14392: 100 unlinks

You can also dump from the pinned bpftool without writing the userspace iterator:

sudo ./.output/bpftool/bootstrap/bpftool map dump name counts

Ctrl-C sets the loader’s exit flag; after an interrupted sleep it destroys the skeleton, which detaches the fentry link and closes the map. The scratch directory is already gone.

What to break, in order

Break 1 — Drop the null check (rerun of Day 1, in a real context)

Change to:

__u64 *cnt = bpf_map_lookup_elem(&counts, &pid);
*cnt += 1;

Verifier rejects:

; *cnt += 1;
R1 invalid mem access 'map_value_or_null'

The *cnt += 1 is a direct store through a PTR_TO_MAP_VALUE_OR_NULL register, so the verifier’s check_mem_access falls through to its catch-all and prints invalid mem access 'map_value_or_null' (kernel/bpf/verifier.c). The type=... expected=... format is a different message — it comes from the helper-argument type checker when you hand a possibly-NULL pointer to a helper, not when you dereference it yourself. The exact register number depends on how clang allocates the lookup result, so yours may read R0 instead of R1.

Same lesson as Day 1, now with a hash map. The Verifier’s type system does not distinguish between “ringbuf reserve might fail” and “hash lookup might miss” — both produce PTR_TO_MAP_VALUE_OR_NULL (or PTR_TO_MEM_OR_NULL), both must be checked.

Break 2 — max_entries = 0

Change max_entries to 0 and rebuild. The loader fails:

libbpf: map 'counts': failed to create: Invalid argument

Map config errors fail before the Verifier runs. Read kernel/bpf/hashtab.c:htab_map_alloc_check (:407) to see the validation rules — including the explicit attr->max_entries == 0 || attr->key_size == 0 || attr->value_size == 0 → -EINVAL check at hashtab.c:446, which is the Invalid argument you just hit.

Break 3 — Wrong key size

Change __type(key, __u32) to __type(key, __u64) but keep passing __u32 pid (4 bytes) as the lookup key. The lookup will use 8 bytes starting at &pid, reading 4 bytes of stack garbage. You’ll see weird behavior: many keys, each one off by random bytes from the actual PID.

The Verifier might catch this (depending on how pid is laid out on the stack); it might not. Lesson: map key/value types must exactly match how you call into them. Mismatches don’t reliably fail at load time.

Break 4 — The “tracer that won’t clean up” pattern (preview of Day 9)

This break uses the Day 1 fentry+ringbuf program but with a twist:

SEC("fentry/vfs_read")
int BPF_PROG(on_read)
{
    __u64 tid = bpf_get_current_pid_tgid();
    __u64 ts = bpf_ktime_get_ns();
    bpf_map_update_elem(&counts, &tid, &ts, BPF_ANY);
    return 0;
}

Run for a minute. Watch the map fill:

sudo ./.output/bpftool/bootstrap/bpftool map dump name counts | wc -l

Without bpf_map_delete_elem somewhere, every TID that ever called vfs_read gets a permanent entry. Once you hit max_entries, new TIDs silently fail to insert. This is the bug Day 9 is about — but the lesson is here too: map entries don’t expire on their own. If you insert, plan to delete.

What to read in the kernel

• kernel/bpf/hashtab.c — open it, read htab_map_lookup_elem (small) and htab_map_update_elem (longer). Note the per-bucket lock (htab_lock_bucket, :149; taken in update at :1217) and the two chain walkers — lookup_elem_raw (“can only be called with bucket lock taken”, :691) vs the lockless RCU lookup_nulls_elem_raw (:705). The file is ~2700 lines but you only need the lookup/update path today.
• kernel/bpf/verifier.c — search mark_ptr_or_null_regs (:16060; the per-register worker mark_ptr_or_null_reg is at :16015). This is the function that flips a register’s type from PTR_TO_MAP_VALUE_OR_NULL to PTR_TO_MAP_VALUE after a null check. Don’t try to read the whole verifier — just this function and its callers. Maybe 50 lines.
• include/uapi/linux/bpf.h — search BPF_MAP_TYPE_. Skim every entry. You don’t need to know each one yet; just know they exist. We’ll use about half of them across these 30 days.

Bullet Points

• A map is the kernel-resident shared notebook between BPF and userspace. BPF_MAP_TYPE_HASH is the workhorse — a power-of-two array of buckets, index = hash & (n_buckets-1).
• Two doors into one map. The BPF program names the map by address and gets a live pointer; userspace names it by integer fd (bpf_map__fd) and the libbpf bpf_map_* calls are thin wrappers around sys_bpf() that copy values in/out. bpf_map_lookup_elem is the name of both functions, with opposite contracts.
• Lookup returns a pointer into live map memory, not a copy. The Verifier marks it PTR_TO_MAP_VALUE_OR_NULL until you check.
• Lockless reads, locked writes. Lookups walk an RCU hlist_nulls with no lock (elements are RCU-freed, so the returned pointer can’t be freed under you); inserts/deletes take a per-bucket raw spinlock. That split is why you still need an atomic for the value.
• Always check bpf_map_lookup_elem for NULL before dereferencing.
• max_entries is a hard cap with no automatic eviction. Inserting into a full hash map returns -E2BIG.
• Atomic updates use __sync_fetch_and_add — the lookup gave you an unlocked pointer, so concurrent increments race and lose counts without it.
• BPF_MAP_TYPE_PERCPU_HASH gives every CPU its own value copy: no atomics, no contention, but userspace must sum across CPUs to read a total, and per-CPU value size is bounded by PCPU_MIN_UNIT_SIZE. Use it when the hot path is contended.
• Don’t insert without a plan to delete. Maps don’t expire entries; tracers that don’t clean up degrade silently into oblivion.

Check question

Two CPUs simultaneously execute the program for the same PID, both observe cnt == NULL, and both try bpf_map_update_elem(..., BPF_NOEXIST). What happens?

Tomorrow

Day 3: CO-RE in anger. Read task->real_parent->tgid from BPF in a way that survives kernel upgrades. The first time vmlinux.h and CO-RE relocations earn their keep.

Day 3 — CO-RE: read task->real_parent->tgid and survive a kernel upgrade

Today’s mission: read the parent PID of every process that calls unlink. Do it in a way that runs unchanged on the kernel you compiled against and on next year’s kernel after the layout drifts. Total time: ~95 minutes.

The fragile pre-CO-RE world

Before CO-RE existed, BPF tracing programs had two ugly choices:

1. bcc-style runtime compilation. Ship the BPF source as a string. The userspace tool detects the running kernel, calls Clang at runtime against the current kernel’s headers, compiles the program then and there, loads it. Slow startup (Clang takes ~seconds), needs Clang+headers on every target machine, and breaks subtly when headers don’t match a running kernel built from a slightly different config.

2. Hardcoded offsets. Read kernel struct fields with bpf_probe_read_kernel(&out, sizeof(out), (char *)task + 1872). Works on exactly the kernel you derived 1872 from. Breaks the moment that struct grows a field.

Both options are why pre-2020 BPF tooling felt brittle. CO-RE killed both.

And the structure that breaks hardcoded-offset tools worst is the one we’re going to read today: struct task_struct. So before we touch CO-RE, let’s meet it — because half of why CO-RE exists is to read this one struct safely.

The structure that motivates everything: task_struct

Every runnable thread in the kernel — every process, every thread inside a process, every kernel worker — is described by exactly one struct task_struct. It is the kernel’s process/thread descriptor: the big bag of state the scheduler, the signal code, the VFS, and the credential machinery all hang off of. It is genuinely large — hundreds of fields, well over a kilobyte — holding scheduling state, the open-file table, credentials, parent/child links, and a short human-readable name.

A few fields are all we need today. In v7.1 (include/linux/sched.h):

pid_t pid;                              /* sched.h:1063 — per-thread ID      */
pid_t tgid;                             /* sched.h:1064 — thread-group ID    */
struct task_struct __rcu *real_parent;  /* sched.h:1077 — the (original) parent */
struct task_struct __rcu *parent;       /* sched.h:1080 — recipient of SIGCHLD  */
char comm[TASK_COMM_LEN];               /* sched.h:1173 — 16-byte thread name   */

current: the task that’s running right now

You never get a task_struct out of thin air — you ask “who is running on this CPU at this instant?” The kernel’s answer is the macro current. It’s a per-CPU/arch construct that yields the task_struct * of whatever thread is executing right now (on x86_64 it’s read from a per-CPU variable; other arches stash it in a register). Whenever kernel code says current->pid, it means “the PID of the thread that triggered this code path.”

BPF gets at it through a helper. bpf_get_current_task_btf() literally returns current — its kernel implementation is one line:

/* kernel/trace/bpf_trace.c:766 */
BPF_CALL_0(bpf_get_current_task_btf)
{
	return (unsigned long) current;   /* :768 */
}

So when your program calls bpf_get_current_task_btf() inside the unlink hook, you get the task_struct of the process that called unlink. That’s the anchor for everything else today.

pid vs tgid: the kernel’s PID is not userspace’s PID

Here’s the trap that bites everyone once. In the kernel, pid is the per-thread ID and tgid (thread-group ID) is the leader thread’s pid. A single-threaded process has pid == tgid. A process with four threads has four distinct pids but one shared tgid.

Userspace getpid() returns the tgid. So:

• The user-visible PID of a task is its tgid.
• The user-visible PPID is the parent’s tgid — i.e. real_parent->tgid.

(Day 1 already noted “userspace PID = kernel TGID” in passing when we used bpf_get_current_pid_tgid; this is why. That helper packs tgid in the high 32 bits and pid in the low 32 — >> 32 gives you the user-visible PID.)

That single fact is why the lab reads real_parent->tgid and not real_parent->pid: we want the PPID a human would recognize from ps, which is the parent’s thread-group ID.

real_parent vs parent

There are two parent pointers, and the kernel comment block at sched.h:1071 explains why:

• real_parent (sched.h:1077) — the original/biological parent. Used for reparenting and ptrace bookkeeping. This is the “who actually forked me” link.
• parent (sched.h:1080) — the task that receives SIGCHLD and wait4() reports. Usually identical to real_parent, but under ptrace (a debugger, strace) the parent becomes the tracer while real_parent stays the true parent.

The lab deliberately uses real_parent so the true parent shows up even if something is tracing the process.

comm[16]: the thread name

comm is a fixed 16-byte (TASK_COMM_LEN, sched.h:325) NUL-padded thread name — "bash", "rm", "sshd". It’s the same buffer Day 1’s bpf_get_current_comm helper copied out for you. Today we read it directly via CO-RE instead, which is the whole point: we’re going to walk into task_struct by hand.

And this is why CO-RE matters. task_struct grows and reshuffles constantly between kernel versions — a new scheduler field here, a reordered block there. Every one of those changes shifts the byte offset of comm, tgid, real_parent. A tool that hardcoded (char *)task + 1872 is broken by the next -rc. CO-RE is the fix.

What CO-RE actually does

You write field accesses by name:

__u32 ppid = BPF_CORE_READ(task, real_parent, tgid);

The compiler emits an instruction that loads a value at offset 0xC0RE0001 (a placeholder — Clang fills the actual relocation index there). Alongside the instruction, it emits a CO-RE relocation record that says: “the offset I just used is bogus; at load time, look up the byte offset of task_struct.real_parent in the target kernel’s BTF and patch this instruction with the real value.”

When libbpf loads the program, it walks every CO-RE relocation, looks up the field by name in /sys/kernel/btf/vmlinux, computes the offset, and patches the instruction. Same .o file, different patched offsets per kernel.

This is why your Day 1 program’s BPF_PROG(on_unlink, int dfd, struct filename *name) worked without you knowing the byte offset of anything — every kernel-type access went through this name-based relocation.

There are no Dumb Questions

Q: Why doesn’t C just do this normally? My userspace programs include kernel headers and read fields by name.

A: Userspace #include <linux/sched.h> reads the headers from when you compiled. If the running kernel has a different layout, you crash or read wrong bytes — that’s why people warn against using kernel headers from outside /proc. CO-RE inverts the model: you build BPF programs as if struct layouts are unknown until load time, then libbpf resolves them against ground truth.

Q: What if a field doesn’t exist on the target kernel?

A: There are CO-RE primitives for that:

• bpf_core_field_exists(task->real_parent) — returns 1 if the field exists, 0 otherwise.
• bpf_core_type_exists(struct foo) — same for whole types.
• bpf_core_enum_value_exists(...) — same for enum members.

If you check existence and skip access, libbpf will also patch the access instruction to a no-op when the field is missing. You can ship one program that gracefully degrades across kernel versions.

Q: Are direct typed-pointer derefs the same as BPF_CORE_READ?

A: Almost. With BTF and the right program type, the Verifier accepts task->real_parent->tgid and the compiler emits CO-RE-relocated instructions. Both are memory-safe: BPF_CORE_READ expands to bpf_probe_read_kernel (a helper call that zero-fills on a bad pointer), while a direct deref of a walked PTR_TO_BTF_ID is rewritten by the verifier to a fault-protected BPF_PROBE_MEM load that also zero-fills. The real difference is speed and applicability: direct deref is an inline load (no call) but needs a verifier-typed pointer, while BPF_CORE_READ is slower but works even when the verifier can’t type the pointer (kprobe pt_regs casts, non-BTF program types). For arguments handed to you by the kernel (fentry params, bpf_get_current_task_btf()), direct deref is ideal. (We unpack exactly why one is faster two sections down.)

Sharpen your pencil

Your Day 1 and Day 2 programs read bpf_get_current_pid_tgid() >> 32 to get the PID. That doesn’t go through CO-RE — there’s no struct field involved. But the BPF program still has to know it’s running on Linux x86_64 vs ARM64. What handles that abstraction?

.
.
.

Answer: the helper itself. bpf_get_current_pid_tgid is implemented in kernel/bpf/helpers.c and is called the same way regardless of arch. Architecture-specific work happens inside the kernel implementation. CO-RE is for accessing kernel data structures whose layout differs across kernel versions, not for arch-portable helper calls.

Why BPF can’t just dereference a pointer: bpf_probe_read_kernel

We keep saying “BPF_CORE_READ fault-handles each hop” and “direct deref is faster but assumes the pointer is valid.” To make that concrete you need to know what’s underneath — and the answer is a helper called bpf_probe_read_kernel.

Here’s the problem. A BPF program runs in kernel context. If it blindly dereferenced an arbitrary kernel pointer and that pointer were NULL, stale, or just wrong, it would fault in kernel context — and a fault in the kernel is not a friendly segfault, it’s an oops. BPF is supposed to be safe to load on a production box. So the kernel does not let a BPF program follow just any pointer. It offers exactly two sanctioned ways to touch kernel memory.

Way 1 — the bpf_probe_read_kernel(dst, size, src) helper. It copies size bytes from src into dst, but it does the copy through copy_from_kernel_nofault, the kernel’s “try this load, but catch the fault instead of crashing” primitive. If the read faults, the helper doesn’t crash — it zero-fills dst and returns -EFAULT:

/* include/linux/bpf.h:3387 — bpf_probe_read_kernel_common, the plain kernel read path */
ret = copy_from_kernel_nofault(dst, unsafe_ptr, size);  /* :3392 */
if (unlikely(ret < 0))
	memset(dst, 0, size);     /* :3394 — the zero-fill-on-fault path */
return ret;

(The string and user variants live in kernel/trace/bpf_trace.c and use a different primitive: bpf_probe_read_user_common memsets at :179, bpf_probe_read_user_str_common at :216, bpf_probe_read_kernel_str_common at :266. Only the non-string kernel read shown above is in bpf.h.)

/* kernel/trace/bpf_trace.c:235 */
BPF_CALL_3(bpf_probe_read_kernel, void *, dst, u32, size,
	   const void *, unsafe_ptr)
{
	return bpf_probe_read_kernel_common(dst, size, unsafe_ptr);   /* :238 */
}

That memset(dst, 0, size) is the source of the chapter’s “BPF_CORE_READ returns a default 0 on bad pointers” behavior. The zero doesn’t come from CO-RE — it comes from this helper’s error path. A bad hop gives you a clean 0, not a crash.

Way 2 — verifier-trusted typed pointers. When the Verifier can prove a pointer’s type and trustworthiness, it lets the JIT emit a plain hardware load with no helper call at all. No copy_from_kernel_nofault, no fault net — just a ldx. This is the fast path, and it’s why direct deref (task->real_parent->tgid) beats the macro. We meet the register type that unlocks it in the next section.

So now the two forms decompose cleanly:

• BPF_CORE_READ = (CO-RE relocation to patch the offset) + (bpf_probe_read_kernel to fault-protect the load), chained once per pointer hop.
• Direct deref = (CO-RE relocation to patch the offset) + (plain trusted load, no fault net).

Both keep the relocation. Only BPF_CORE_READ keeps the safety net. You can see the marriage of relocation-and-probe-read right in the header:

/* tools/lib/bpf/bpf_core_read.h:312 */
bpf_probe_read_kernel(dst, sz, (const void *)__builtin_preserve_access_index(src))

The __builtin_preserve_access_index(src) is the CO-RE relocation (it records “this offset must be patched against target BTF”); the bpf_probe_read_kernel(...) wrapped around it is the fault protection. One macro, both jobs.

One licensing consequence. bpf_probe_read_kernel’s proto is GPL-only:

/* kernel/trace/bpf_trace.c:241 */
const struct bpf_func_proto bpf_probe_read_kernel_proto = {
	.func		= bpf_probe_read_kernel,
	.gpl_only	= true,        /* :243 */
	...
};

This ties straight back to Day 1’s LICENSE gate (Day 1, Break 3): the instant your program uses BPF_CORE_READ, it routes through this GPL-only helper, so it genuinely needs char LICENSE[] SEC("license") = "GPL";. Drop the GPL string and the loader rejects the program with “helper call is not allowed in non-GPL program.” That’s not bureaucracy — it’s this proto’s .gpl_only flag.

PTR_TO_BTF_ID: the trusted, typed pointer that lets you skip the safety net

The fast path above hinges on one Verifier concept, so let’s name it.

Refresher (Day 2): the Verifier tags every register with a type and tracks what you may legally do with it. Recall PTR_TO_MAP_VALUE_OR_NULL from yesterday — the type a bpf_map_lookup_elem result carries until you NULL-check it. You couldn’t dereference it until you proved it non-NULL.

New today: PTR_TO_BTF_ID is a register type that says “this pointer points at a specific kernel type, identified by a BTF type id.” Because the register carries the type, the Verifier can type-check field accesses against the real struct layout and permit a direct load of a field at the relocated offset. No helper call needed — that’s the plain ldx from the previous section.

The _TRUSTED variant adds one more guarantee: the pointer is non-NULL and not stale. So no NULL check is required before deref — unlike a map-value pointer, which always might be NULL.

This is exactly why bpf_get_current_task_btf() is special. Look at its proto:

/* kernel/trace/bpf_trace.c:771 */
const struct bpf_func_proto bpf_get_current_task_btf_proto = {
	.func		= bpf_get_current_task_btf,
	.gpl_only	= true,
	.ret_type	= RET_PTR_TO_BTF_ID_TRUSTED,                  /* :774 */
	.ret_btf_id	= &btf_tracing_ids[BTF_TRACING_TYPE_TASK],    /* :775 */
};

RET_PTR_TO_BTF_ID_TRUSTED + a ret_btf_id pointing at the task_struct BTF id means: the Verifier marks the returned register a trusted, non-NULL PTR_TO_BTF_ID of type task_struct. That’s why the lab can write task->real_parent->tgid with zero NULL checks on task — the Verifier already knows it’s a valid task pointer.

Contrast the older helper, bpf_get_current_task (bpf_trace.c:755), whose proto has .ret_type = RET_INTEGER — it hands back an opaque u64 you have to cast yourself, with no type information for the Verifier. That’s the “older variant that returned an opaque u64 you had to cast” the cast notes mention. The _btf version exists precisely to give the Verifier the type.

This is the first of Day 1’s “four things BTF powers” (Day 1: “Verifier type-checking of typed pointers (PTR_TO_BTF_ID)”) finally made concrete. Forward pointer: Day 4 formalizes the Verifier’s full type lattice. For today, hold one sentence: PTR_TO_BTF_ID = a typed, trusted kernel pointer the Verifier lets you dereference directly.

Meet the cast

bpf_get_current_task_btf — typed access to current

Returns a struct task_struct * with the BTF type tag attached (RET_PTR_TO_BTF_ID_TRUSTED, as we just saw). The Verifier knows it’s a valid, non-NULL kernel pointer. You can deref fields directly. Cheaper than bpf_get_current_task (the older variant that returned an opaque u64 you had to cast).

BPF_CORE_READ(task, a, b, c) — chained CO-RE-aware reads

Expands to a series of bpf_probe_read_kernel calls walking each pointer hop, with CO-RE relocations on each field access. Equivalent to:

struct task_struct *p1 = task;
struct task_struct *p2;
__u32 result;
bpf_probe_read_kernel(&p2, sizeof(p2), &p1->real_parent);  // CO-RE: real_parent offset
bpf_probe_read_kernel(&result, sizeof(result), &p2->tgid); // CO-RE: tgid offset

The macro saves you that boilerplate and fault-handles each hop, as the bpf_probe_read_kernel section showed — a bad pointer anywhere in the chain yields a clean 0 instead of a crash.

BPF_CORE_READ_INTO(dst, src, a, b, c)

Same thing but writes the final value into *dst instead of returning it. Use when the field is larger than 8 bytes (e.g., reading task->comm[16] into a buffer).

BPF_CORE_READ_STR_INTO(dst, src, a, b, c)

Reads a NUL-terminated string. Bounds the read at sizeof(dst). Returns the actual length copied.

The lab

Continue in the repository’s ebpf/labs directory. make parent compiles the exact anchored listings shown below.

parent.h — the shared event record

#ifndef PRACTICAL_EBPF_PARENT_H
#define PRACTICAL_EBPF_PARENT_H

struct parent_event {
    unsigned int pid;
    unsigned int ppid;
    char comm[16];
    char parent_comm[16];
};

#endif

parent.bpf.c

#include "vmlinux.h"
#include <bpf/bpf_core_read.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

#include "parent.h"

char LICENSE[] SEC("license") = "GPL";

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

SEC("fentry/filename_unlinkat")
int BPF_PROG(on_unlink)
{
    struct task_struct *task;
    struct parent_event *event;

    task = (struct task_struct *)bpf_get_current_task_btf();
    event = bpf_ringbuf_reserve(&rb, sizeof(*event), 0);
    if (!event)
        return 0;

    __builtin_memset(event, 0, sizeof(*event));
    event->pid = bpf_get_current_pid_tgid() >> 32;
    event->ppid = BPF_CORE_READ(task, real_parent, tgid);
    BPF_CORE_READ_STR_INTO(&event->comm, task, comm);
    BPF_CORE_READ_STR_INTO(&event->parent_comm, task, real_parent, comm);

    bpf_ringbuf_submit(event, 0);
    return 0;
}

What’s new:

• bpf_get_current_task_btf() returns a typed struct task_struct *. Its proto’s return type is RET_PTR_TO_BTF_ID_TRUSTED, so the Verifier marks the register a trusted, non-NULL PTR_TO_BTF_ID — you can deref its fields directly with no NULL check.
• __builtin_memset(event, 0, sizeof(*event)) initializes the whole reserved record before helper reads. If a fault-tolerant string read fails, userspace receives an empty string instead of stale ringbuf bytes.
• BPF_CORE_READ(task, real_parent, tgid) walks task → real_parent → tgid (recall: tgid is the user-visible PID, so the parent’s tgid is the PPID). Each hop is a CO-RE-relocated bpf_probe_read_kernel.
• BPF_CORE_READ_STR_INTO(&event->comm, task, comm) reads task->comm (a 16-byte char array, not a pointer). Note comm is the last arg; for non-pointer fields you don’t need a final * deref hop.
• Unlike Day 1, BPF_PROG(on_unlink) takes no extra params — we don’t need the unlink arguments (dfd/name) today, because bpf_get_current_task_btf() hands us the task directly.

Userspace parent.c

The loader mirrors Day 1’s checked error and signal paths, validates the shared struct parent_event, and releases both ringbuf and skeleton on exit:

#include <errno.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>

#include <bpf/libbpf.h>

#include "parent.h"
#include "parent.skel.h"

static volatile sig_atomic_t exiting;

static void handle_signal(int signal_number)
{
    (void)signal_number;
    exiting = 1;
}

static int install_signal_handlers(void)
{
    const struct sigaction action = {
        .sa_handler = handle_signal,
    };

    if (sigaction(SIGINT, &action, NULL) != 0 ||
        sigaction(SIGTERM, &action, NULL) != 0) {
        fprintf(stderr, "failed to install signal handlers: %s\n",
                strerror(errno));
        return -errno;
    }

    return 0;
}

static int handle_event(void *ctx, void *data, size_t size)
{
    const struct parent_event *event = data;

    (void)ctx;
    if (size != sizeof(*event)) {
        fprintf(stderr, "unexpected ringbuf record size: %zu\n", size);
        return 0;
    }

    printf("PID %u (%.*s) ppid %u (%.*s) deleted a file\n",
           event->pid, (int)sizeof(event->comm), event->comm,
           event->ppid, (int)sizeof(event->parent_comm), event->parent_comm);
    return 0;
}

int main(void)
{
    struct parent_bpf *skel = NULL;
    struct ring_buffer *ringbuf = NULL;
    int err;

    setvbuf(stdout, NULL, _IOLBF, 0);

    err = install_signal_handlers();
    if (err)
        return 1;

    skel = parent_bpf__open();
    if (!skel) {
        fprintf(stderr, "failed to open parent BPF skeleton\n");
        return 1;
    }

    err = parent_bpf__load(skel);
    if (err) {
        fprintf(stderr, "failed to load parent BPF programs: %s\n",
                strerror(-err));
        goto cleanup;
    }

    err = parent_bpf__attach(skel);
    if (err) {
        fprintf(stderr, "failed to attach parent BPF programs: %s\n",
                strerror(-err));
        goto cleanup;
    }

    ringbuf = ring_buffer__new(bpf_map__fd(skel->maps.rb), handle_event,
                               NULL, NULL);
    err = libbpf_get_error(ringbuf);
    if (err) {
        ringbuf = NULL;
        fprintf(stderr, "failed to create ringbuf consumer: %s\n",
                strerror(-err));
        goto cleanup;
    }

    puts("Tracing unlink callers and parents; press Ctrl-C to stop.");
    while (!exiting) {
        err = ring_buffer__poll(ringbuf, 100);
        if (err == -EINTR) {
            err = 0;
            continue;
        }
        if (err < 0) {
            fprintf(stderr, "ringbuf poll failed: %s\n", strerror(-err));
            goto cleanup;
        }
    }

    err = 0;

cleanup:
    ring_buffer__free(ringbuf);
    parent_bpf__destroy(skel);
    return err != 0;
}

Run it

In terminal 1:

make parent
sudo ./.output/day03/parent

In terminal 2:

scratch=$(mktemp -d /tmp/ebpf-day03.XXXXXX)
touch "$scratch/x"
rm "$scratch/x"
rmdir "$scratch"

Expected:

PID 24501 (rm) ppid 24450 (bash) deleted a file

That parent is real_parent->tgid and real_parent->comm working exactly as the task_struct section promised: rm’s real parent is the shell that forked it, and its tgid is the PPID ps would show.

Press Ctrl-C in terminal 1 when the event appears. The loader frees the ringbuf and destroys the skeleton, detaching its fentry link without a broad pkill.

Inspect what CO-RE actually emitted

This is the most important step today. You need to see the relocations to believe they exist.

Disassemble the object with relocations interleaved (the -r flag is what makes the CO-RE records appear — -d alone never prints them):

llvm-objdump -dr .output/day03/parent.bpf.o | grep CO-RE

(If llvm-objdump isn’t found, your distro may only ship versioned binaries — try llvm-objdump-21 or llvm-objdump-18. Either works; the -dr flags and output are identical.)

Recent LLVM annotates the BPF disassembly with the CO-RE relocation records read from the object’s .BTF.ext section. You’ll see one line per kernel-field access:

0000000000000060:  CO-RE <byte_off> [13] struct task_struct::real_parent
00000000000000a0:  CO-RE <byte_off> [13] struct task_struct::tgid
00000000000000e8:  CO-RE <byte_off> [13] struct task_struct::comm

(addresses and the type index [13] vary by build). Drop the | grep CO-RE to see the full disassembly: the load instruction just above each relocation carries the compile-time byte offset Clang baked in — e.g. r1 = 0xae0 for real_parent — not a placeholder like 0x0. libbpf rewrites that immediate at load time if the running kernel’s layout differs. CO-RE relocation records live in the ELF section .BTF.ext (the object also has .BTF, .rel.BTF, and .rel.BTF.ext); there is no .relo.btf section.

To inspect the embedded types instead:

./.output/bpftool/bootstrap/bpftool btf dump file .output/day03/parent.bpf.o

You’ll see your maps (VAR 'rb' and the .maps DATASEC), your on_unlink program function, and references to kernel types like task_struct.real_parent. (struct parent_event is used only inside function code and is not reachable from a map, global, or function prototype, so the compiler does not emit it into the object’s BTF.)

Aside — shipping minimal BTF. ./.output/bpftool/bootstrap/bpftool gen min_core_btf /sys/kernel/btf/vmlinux min.btf .output/day03/parent.bpf.o writes a minimal BTF file containing only the types your program references (it prints nothing to stdout). That’s for portability — shipping a tiny BTF alongside your .o for kernels without /sys/kernel/btf/vmlinux — not for inspecting relocations.

To watch the relocations get applied at load time, you need libbpf’s own debug log, not the kernel verifier log. The two are different: .kernel_log_level feeds bpf_attr.log_level, which controls the in-kernel verifier log (a disassembled, post-relocation instruction dump plus verifier state). CO-RE patching happens in libbpf userspace before the BPF_PROG_LOAD syscall, so the verifier log contains no CO-RE string and no relocation provenance — grepping it for CO-RE finds nothing.

The patching messages come out of libbpf’s print callback at the LIBBPF_DEBUG level. Easiest from the shell — load with bpftool’s -d flag (which sets libbpf to LIBBPF_DEBUG) and grep for the relocation lines:

sudo ./.output/bpftool/bootstrap/bpftool -d prog load .output/day03/parent.bpf.o /sys/fs/bpf/parent 2>&1 | grep relo
sudo rm -f /sys/fs/bpf/parent   # clean up the pin

You’ll see lines like prog 'on_unlink': relo #N: ... patched insn .... From C, register a print callback and raise the level before open/load:

static int dbg(enum libbpf_print_level lvl, const char *fmt, va_list ap)
{
    return vfprintf(stderr, fmt, ap);
}
// in main(), before parent_bpf__open():
libbpf_set_print(dbg);

Note libbpf’s default print callback only emits WARN to stderr; the INFO/DEBUG relocation lines are suppressed unless you install your own callback or use a debug-enabled loader.

What to break, in order

Break 1 — Use a non-existent field

Add this line:

__u32 fake = BPF_CORE_READ(task, this_field_does_not_exist);

You might expect this to compile and only blow up at load time. It doesn’t — the build fails immediately:

error: no member named 'this_field_does_not_exist' in 'struct task_struct'

Here’s why: BPF_CORE_READ doesn’t reference fields by an opaque name string. It expands to the real C member-access expression (task->this_field_does_not_exist) wrapped in __builtin_preserve_access_index, and Clang type-checks that member against the struct task_struct in your vmlinux.h. A name that exists in no BTF at all is a plain C error. (bpf_core_field_exists(task->this_field_does_not_exist) fails the same way — it also emits the member-access expression.)

So to exercise the genuine load-time relocation-failure path — the one that prints

libbpf: prog 'on_unlink': relo #N: failed to relocate ...

you need a field that exists in your build’s BTF/vmlinux.h but is absent on the target kernel. The realistic way to trigger it is to compile against a newer vmlinux.h and load on an older kernel that lacks the field.

For the graceful-degradation demo, use a field that actually exists so the program compiles, and gate it on bpf_core_field_exists:

__u32 fake = 0;
if (bpf_core_field_exists(task->pid))
    fake = BPF_CORE_READ(task, pid);

The point of bpf_core_field_exists is fields that appear or disappear across kernel versions: libbpf evaluates the check at load time against the target kernel’s BTF and patches the branch to a no-op when the field is absent, so a single .o runs on kernels that have or lack the field. It is not a way to reference a name that exists in no BTF at all.

Break 2 — Direct deref vs BPF_CORE_READ

Replace the macro form with direct deref:

event->ppid = task->real_parent->tgid;

Compiles. Verifier accepts. Runs. Faster than BPF_CORE_READ because no bpf_probe_read_kernel call — this is the trusted-PTR_TO_BTF_ID plain-load fast path we dissected earlier. The Verifier already knows task is a non-NULL task_struct, so it lets the JIT emit a bare load with no fault net.

But this is not a safety downgrade, and that’s the subtle part. When the verifier accepts task->real_parent->tgid, it doesn’t emit a bare load for the walked hop. real_parent is reached by following a pointer, so the resulting register is an untrusted PTR_TO_BTF_ID, and the verifier rewrites that tgid load into a BPF_LDX | BPF_PROBE_MEM with an exception-table entry (kernel/bpf/fixups.c). On a fault the load is skipped and the destination register is zero-filled — the same 0 you’d get from BPF_CORE_READ. So if real_parent were ever NULL, ppid comes back 0 and the event still fires; it does not crash, terminate, or lose the event. Only loads through genuinely trusted pointers (like task itself) stay bare — and those are exactly the pointers the verifier has proven non-NULL, so the NULL-deref scenario can’t arise for them.

So both forms are memory-safe and both zero-fill on a bad pointer. The real reasons to choose between them:

• BPF_CORE_READ emits a bpf_probe_read_kernel helper call (slower), but works even when the verifier can’t hand you a typed pointer — e.g. a kprobe’s pt_regs cast, or a non-BTF program type. It also keeps the whole chained a, b, c expression portable.
• Direct deref of a verifier-typed PTR_TO_BTF_ID emits an inline load (faster, no call). The verifier auto-rewrites the walked/untrusted hops to BPF_PROBE_MEM so they fault-protect and zero-fill too; only the proven-non-NULL trusted hop stays bare.

Break 3 — Forget #include <bpf/bpf_core_read.h>

BPF_CORE_READ is undefined; compile fails. Trivial, but worth noting: BPF helper headers are split across bpf_helpers.h (general), bpf_tracing.h (BPF_PROG, ctx unpacking, PT_REGS_*), bpf_core_read.h (CO-RE macros). Forget any of them and the failure mode is “macro X not defined.”

What to read in the kernel

• tools/lib/bpf/relo_core.c — the CO-RE engine in userspace. bpf_core_calc_relo_insn (relo_core.c:1297) computes the relocation value from target-kernel BTF, and bpf_core_patch_insn (relo_core.c:1041) writes it into the instruction’s offset/immediate. ~300 lines, accessible if you know what to look for.
• tools/lib/bpf/btf.c — read btf__find_by_name_kind (btf.c:1166). This is how libbpf finds types in BTF by name. CO-RE is built on top of it.
• tools/lib/bpf/bpf_core_read.h — search BPF_CORE_READ. The macros expand to a chain of bpf_core_read (kernel-side) calls; bpf_core_field_exists is at bpf_core_read.h:187-188. Reading them once eliminates magic.
• include/uapi/linux/btf.h — the BTF_KIND_* enums. BTF has ~19 kinds (int, ptr, array, struct, union, enum, fwd, typedef, volatile, const, restrict, func, func_proto, var, datasec, float, decl_tag, type_tag, enum64). BTF_KIND_ENUM64 = 19 at uapi/btf.h:92, with NR_BTF_KINDS at :94 and BTF_KIND_MAX at :95. struct btf_type is also defined here (at :43); include/linux/btf.h only forward-declares it (at :113). Skim.
• include/linux/sched.h — the struct task_struct we read today: pid (1063), tgid (1064), real_parent (1077), parent (1080), comm (1173). Skim the field comments to feel how much state hangs off one task.

Bullet Points

• CO-RE = Compile Once, Run Everywhere. Field offsets are resolved at load time using target-kernel BTF.
• task_struct is the per-thread descriptor; current is the running one; bpf_get_current_task_btf() returns it. User-visible PID = tgid, user-visible PPID = real_parent->tgid; comm[16] is the thread name.
• BPF can’t blindly deref kernel pointers. bpf_probe_read_kernel copies via copy_from_kernel_nofault and zero-fills on fault (the source of “default 0”); it’s GPL-only, which is why CO-RE programs need SEC("license")="GPL".
• PTR_TO_BTF_ID (_TRUSTED) = a typed, non-NULL kernel pointer the Verifier lets you deref directly with no NULL check and no helper — the fast path behind direct deref.
• Use BPF_CORE_READ(a, b, c, d) to chain field accesses through pointer hops with fault-handling on each hop.
• Use direct deref (task->real_parent->tgid) when the pointer chain is trusted and you want speed.
• Handle missing fields with bpf_core_field_exists — libbpf will patch the access to a no-op when the field is absent.
• BTF kinds you’ll meet: STRUCT, UNION, ENUM, INT, PTR, ARRAY, FUNC, TYPEDEF. About 19 total.

Check question

You compile your program against kernel A. Field task_struct.real_parent is at offset 1872. You ship the .o to a machine running kernel B where the same field is at offset 1920. Your program does BPF_CORE_READ(task, real_parent, tgid). Walk through what happens at load time.

Tomorrow

Day 4: meet the Verifier properly. We’ll spend a whole day deliberately tripping over PTR_TO_MAP_VALUE_OR_NULL rejections in five different shapes — because every BPF programmer, no matter how senior, hits this error monthly. The goal: stop being surprised by the rejection and start reading the log fluently. (Today’s PTR_TO_BTF_ID was a preview of the type lattice we formalize there.)

Day 4 — Meet the Verifier (PTR_TO_MAP_VALUE_OR_NULL boot camp)

Today’s mission: stop being surprised by the most common BPF rejection. Trip over PTR_TO_MAP_VALUE_OR_NULL in five different shapes; read the log on each; learn what “register state” means — down to the individual registers and opcode bytes the log is built from. Total time: ~120 minutes. No new functionality today — this is pure verifier intuition.

Why a whole day on one error

Because you’ll see this rejection more than any other. Days 1–3 dodged it by always writing if (!cnt) return 0; immediately after lookups. That’s the trained-monkey response. Today is when that response becomes understanding.

The Verifier is not arbitrary. It runs an abstract interpreter over your program — for every instruction it tracks each register’s type, bounds, reference state, and which branch you’re on. When it can’t prove a memory access safe, it rejects. The error log tells you exactly which register and which instruction; if you can read register state, you can fix any rejection in under a minute.

But there’s a catch. Every line of that log is dense with notation the BPF tutorials never explain: R0, R10, (79), *(u64 *)(r0 +0), id=2, fp0. Today is titled a “register state boot camp,” so before we break anything we’re going to actually learn the alphabet: the eleven registers, the opcode bytes, and the two kinds of id. Then the five rejections read themselves.

The eleven registers: BPF’s whole register file

Every line in a verifier log mentions registers named R0 through R10. That is not a sampling — that is the entire register file. BPF has exactly eleven 64-bit registers, no more: ten general-purpose registers (R0–R9) plus one read-only frame pointer (R10):

/* include/uapi/linux/bpf.h:62-74 */
enum {
    BPF_REG_0 = 0,
    BPF_REG_1,
    ...
    BPF_REG_10,
    __MAX_BPF_REG,
};
/* include/uapi/linux/bpf.h:77-78 */
/* BPF has 10 general purpose 64-bit registers and stack frame. */
#define MAX_BPF_REG  __MAX_BPF_REG   /* == 11 slots: R0-R9 + R10 */

The Verifier keeps a struct bpf_reg_state for each of these eleven slots, at every instruction, on every path. When the log prints R0=map_value_or_null(...) R10=fp0, it’s dumping that per-register state table.

What makes the eleven registers legible is that they aren’t interchangeable — they follow a fixed calling convention, and the Verifier enforces it. This convention is the single most useful thing to memorize today, because it explains nearly every register name you’ll read:

The kernel documentation states the call rule directly:

After kernel function call, R1-R5 are reset to unreadable and R0 has a return type of the function. Since R6-R9 are callee saved, their state is preserved across the call. — Documentation/bpf/verifier.rst:32-35

And R10:

Though R10 is correct read-only register and has type PTR_TO_STACK… — Documentation/bpf/verifier.rst:81-85

R10’s type is PTR_TO_STACK — “reg == frame_pointer + offset” (include/linux/bpf.h:996). You can read and write stack slots through it (*(u32 *)(r10-4)), but you can never reassign R10 itself — the Verifier rejects writes to it. The stack it points at is capped:

/* include/linux/filter.h:98 */
#define MAX_BPF_STACK 512

512 bytes. That’s your entire automatic-storage budget per program frame.

Why does R0 have to be the return register? Because the ABI says so, full stop:

BPF allows only register R0 to be used as return value. — Documentation/bpf/bpf_design_QA.rst:41

Hold onto one consequence, because it bites later today: a lookup result you copy with r0 → r6 survives a later helper call (R6 is callee-saved), but r0 → r1 does not (R1 is clobbered by the next call). That’s also why the Verifier’s id tracking — covered in “Two different ids” below — has to follow a value as it hops between registers.

Reading the pseudo-assembly: the (hex) opcode and the instruction format

Every trace line looks like 7: (79) r1 = *(u64 *)(r0 +0). Three parts: the instruction index (7:), the raw opcode byte in parentheses ((79)), and a human-readable spelling of what it does. The opcode byte is the part nobody explains — let’s decode it, because the whole rejection hinges on recognizing one opcode.

A BPF opcode is one byte, built by OR-ing three fields: instruction class | mode/op | source/size. You don’t need the full table; you need the handful that appear in these labs:

The (15)/(55) jumps are the comparison-against-zero that mark_ptr_or_null_regs keys off: spotting one of them between the (85) call and a (79) load is how you tell, by eye, whether the OR_NULL → MAP_VALUE transition had a chance to fire.

These bytes are just the header constants OR’d together. From include/uapi/linux/bpf_common.h (classes BPF_LD 0x00, BPF_LDX 0x01, BPF_ST 0x02, BPF_STX 0x03, BPF_ALU 0x04, BPF_JMP 0x05; modes BPF_IMM 0x00, BPF_MEM 0x60; sizes BPF_W 0x00, BPF_DW 0x18) and include/uapi/linux/bpf.h (BPF_ALU64 0x07, BPF_MOV 0xb0, BPF_DW 0x18, BPF_CALL 0x80):

• (79) = BPF_LDX(0x01) | BPF_MEM(0x60) | BPF_DW(0x18) — a load.
• (85) = BPF_JMP(0x05) | BPF_CALL(0x80) — a call.
• (b7) = BPF_ALU64(0x07) | BPF_MOV(0xb0) with the K (immediate) source.

You do not have to memorize the table. You need exactly one reflex: any LDX opcode (0x61/0x69/0x71/0x79) is a load — a dereference. And a dereference is precisely what the OR_NULL rule forbids until you’ve checked for NULL.

The size suffix in *(u64 *) vs *(u32 *) is the size field: BPF_W = word/4 bytes, BPF_DW = double-word/8 bytes. That’s not noise — it echoes your C types directly. The key store is *(u32 *) because the map key is __u32 (4 bytes); the value deref is *(u64 *) because the value is __u64 (8 bytes). The widths in the log are your struct, reflected back.

Practical reading rule: when a rejection names a register, scan the opcode column for the LDX/STX (0x6x/0x7x) instruction that touches it. In all five rejections today that instruction is the same: (79) r1 = *(u64 *)(r0 +0) — the single load through R0 that the missing check failed to protect.

How the Verifier walks your program

The Verifier starts at instruction 0 with default register state. For each instruction:

1. It updates register types based on the operation (add, load, call, etc.).
2. On conditional jumps, it forks: explores both branches with appropriate state (e.g., on if (!r0), the false branch knows r0 != 0; the true branch knows r0 == 0).
3. When two branches reconverge, it merges register states (intersection).
4. State pruning. If the current state is “compatible with” a previously-explored state at the same instruction, it stops — it already knows this leads to acceptance. (This is the verifier’s key efficiency trick — without it, branchy programs explode exponentially.)
5. If it hits the complexity budget (1M instructions explored across all paths), it gives up: BPF program is too large.
6. Every path must reach a return/exit instruction safely.

Source: kernel/bpf/verifier.c. The function do_check is the main loop; bpf_is_state_visited (in kernel/bpf/states.c:1202, split out of verifier.c in the 2025 refactor) is the pruning check; mark_ptr_or_null_regs is what we’ll focus on today. The 1M budget is BPF_COMPLEXITY_LIMIT_INSNS (include/linux/bpf.h:2261 — #define BPF_COMPLEXITY_LIMIT_INSNS 1000000 /* yes. 1M insns */), checked against env->insn_processed in do_check (verifier.c:17705).

The state machine for bpf_map_lookup_elem’s return value

When you call bpf_map_lookup_elem, R0 is marked PTR_TO_MAP_VALUE_OR_NULL with an id attached. That id links the not-yet-resolved-NULL nature of this pointer across all flow paths (more on what kind of id, next). The Verifier treats:

• PTR_TO_MAP_VALUE_OR_NULL — deref forbidden.
• PTR_TO_MAP_VALUE — deref OK, with bounds equal to the map’s value size.
• SCALAR_VALUE = 0 — what the register becomes inside the r == 0 branch.

The transition PTR_TO_MAP_VALUE_OR_NULL → PTR_TO_MAP_VALUE happens via mark_ptr_or_null_regs after a comparison against zero — but only on the branch where the comparison would prove non-NULL.

In v7.1 these types are just a base type OR’d with a “maybe NULL” flag (include/linux/bpf.h:1026-1034):

PTR_TO_MAP_VALUE_OR_NULL = PTR_MAYBE_NULL | PTR_TO_MAP_VALUE,
PTR_TO_SOCKET_OR_NULL    = PTR_MAYBE_NULL | PTR_TO_SOCKET,
PTR_TO_BTF_ID_OR_NULL    = PTR_MAYBE_NULL | PTR_TO_BTF_ID,

The helper that produces our OR_NULL says so in its prototype (kernel/bpf/helpers.c:54): bpf_map_lookup_elem_proto has .ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL. That’s where R0’s type comes from — check_helper_call reads it off the proto.

Two different ids: nullness-linkage vs acquired reference

Here’s a subtlety that the logs blur together. A struct bpf_reg_state carries two distinct u32 fields (include/linux/bpf_verifier.h:155 and :195):

• id — general linkage. Links registers that share a not-yet-resolved property, e.g. the same maybe-NULL-ness.
• ref_obj_id — a handle on an acquired reference that must be explicitly released before the program exits.

For bpf_map_lookup_elem, the OR_NULL pointer carries an id only. It is free — no release required — so a bare if (!v) return 0; fully discharges it. That’s why all of today’s labs need nothing but a NULL check. The number you read as id=2 in the log is this general linkage id.

That is the whole intuition you need for today. The rest of this section — how the release machinery works for the reference-bearing OR_NULL types — is an optional going-deeper aside. Today’s map-value labs never trigger it; you can skip to “The Lab” and come back when a later chapter rejects a socket or ringbuf program with an “Unreleased reference” error.

Going deeper (optional): how releases actually fire

Now look at what mark_ptr_or_null_regs actually does (kernel/bpf/verifier.c:16060-16078):

static void mark_ptr_or_null_regs(struct bpf_verifier_state *vstate, u32 regno,
                                  bool is_null)
{
    struct bpf_func_state *state = vstate->frame[vstate->curframe];
    struct bpf_reg_state *regs = state->regs, *reg;
    u32 ref_obj_id = regs[regno].ref_obj_id;
    u32 id = regs[regno].id;

    if (ref_obj_id && ref_obj_id == id && is_null)
        /* regs[regno] is in the " == NULL" branch. */
        WARN_ON_ONCE(release_reference_nomark(vstate, id));

    bpf_for_each_reg_in_vstate(vstate, state, reg, ({
        mark_ptr_or_null_reg(state, reg, id, is_null);
    }));
}

It consults both fields. The bpf_for_each_reg_in_vstate loop walks every register sharing the comparison’s id and flips its type on each branch — that’s the universal part. But the if above it checks ref_obj_id: if the OR_NULL register held a real acquired reference and we just proved it NULL, it releases that reference. Map-value-or-null never minted a ref_obj_id (its proto doesn’t acquire), so that branch is skipped.

This sharpens the chapter’s later “all OR_NULL types follow the same pattern” claim. The NULL-check transition is universal. But only the reference-bearing OR_NULL types carry a ref_obj_id you must additionally resolve:

Type	id	ref_obj_id	Obligation
PTR_TO_MAP_VALUE_OR_NULL	set	0	check discharges it; return-on-NULL is free
PTR_TO_SOCKET_OR_NULL	set	set	must release, or “Unreleased reference”
PTR_TO_MEM | PTR_MAYBE_NULL (ringbuf reserve)	set	set	must submit/discard, or “Unreleased reference”

So if a future chapter shows you a bpf_ringbuf_reserve or socket lookup that loads clean but still gets rejected with an unreleased-reference error, that’s the ref_obj_id talking — same number space in the log, completely different obligation. Today’s map-value labs never hit it.

There are no Dumb Questions

Q: Why does the verifier need an “id” on the pointer? Isn’t the type enough?

A: The id links multiple registers that share the same not-yet-resolved-NULL state. If you copy r0 to r6 before checking, then check r0, the Verifier needs to know r6’s NULL-ness was resolved too. The id is how bpf_for_each_reg_in_vstate finds every register to flip. (For map values this is the linkage id; reference-bearing types additionally carry a ref_obj_id — see above.)

Q: What if I check r0 != NULL instead of !r0?

A: Same effect. The Verifier tracks the comparison and applies the OR_NULL → not-OR-NULL transition on the branch that proves non-NULL, regardless of which boolean form you used.

Q: Is PTR_TO_MAP_VALUE_OR_NULL the only “OR_NULL” type?

A: No — there are several. PTR_TO_SOCKET_OR_NULL, PTR_TO_SOCK_COMMON_OR_NULL, PTR_TO_TCP_SOCK_OR_NULL, and PTR_TO_BTF_ID_OR_NULL (some kfuncs) are all explicit register types in enum bpf_reg_type (include/linux/bpf.h:1026-1034). Ringbuf-reserve memory is OR_NULL too, but at the helper-prototype level (RET_PTR_TO_RINGBUF_MEM_OR_NULL, include/linux/bpf.h:908) — there is no PTR_TO_MEM_OR_NULL register type, just PTR_TO_MEM | PTR_MAYBE_NULL. They all follow the same NULL-check pattern: deref forbidden until proven non-NULL via a check. The reference-bearing ones (socket, ringbuf mem) also require you to release the reference — the deref rule is shared, the release obligation is not. (Packet pointers are not in this family: skb->data is never OR_NULL; its bounds are proven by comparing against PTR_TO_PACKET_END via find_good_pkt_pointers, a different mechanism — there is no PTR_TO_PACKET_OR_NULL type.)

The Lab: five rejections in five shapes

You don’t need a working program today. You need a single source file you mutate and reload.

Setup

Use any of yesterday’s programs as the base. We’ll mutate it in place.

reject.bpf.c — the base that loads clean, straight from the lab tree:

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

char LICENSE[] SEC("license") = "GPL";

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 1024);
    __type(key, __u32);
    __type(value, __u64);
} m SEC(".maps");

SEC("fentry/filename_unlinkat")
int BPF_PROG(rej)
{
    __u32 key = 0;
    /* WE WILL EDIT THE BODY BELOW FIVE TIMES */
    __u64 *v = bpf_map_lookup_elem(&m, &key);
    if (!v) return 0;
    *v += 1;
    return 0;
}

Build it once. It loads. Now break it five ways and read the log each time.

Set verbose loading in your reject.c:

LIBBPF_OPTS(bpf_object_open_opts, opts, .kernel_log_level = 1);
struct reject_bpf *skel = reject_bpf__open_opts(&opts);
if (reject_bpf__load(skel)) {
    fprintf(stderr, "load failed (this is expected)\n");
    return 1;
}

Or load with bpftool prog load reject.bpf.o /sys/fs/bpf/x and the log goes to stderr.

A successful load pins the program at /sys/fs/bpf/x. Rejected loads don’t pin, but the ones that load clean (the base program, Rejection 5’s first snippet, Rejection 3 with a compile-time-constant key) do — so the next successful load to the same path fails with Error: failed to pin program: File exists, which is not a verifier problem. Remove the pin before re-loading: sudo rm -f /sys/fs/bpf/x.

Rejection 1 — The bare deref

__u64 *v = bpf_map_lookup_elem(&m, &key);
*v += 1;       // no null check
return 0;

Expected log (essence):

0: (b7) r1 = 0           ; key = 0
1: (63) *(u32 *)(r10-4) = r1
2: (bf) r2 = r10
3: (07) r2 += -4
4: (18) r1 = 0xffff...   ; map fd
6: (85) call bpf_map_lookup_elem#1
7: R0=map_value_or_null(id=2,off=0,ks=4,vs=8) R10=fp0
7: (79) r1 = *(u64 *)(r0 +0)
R0 invalid mem access 'map_value_or_null'

processed 8 insns ...

How to read this — and now you can read every token:

• Lines 0–6 are the instruction trace, and you can decode each opcode: (b7) MOV-imm sets key = 0; (63) STX-word stores it to the stack slot r10-4 (that’s the &key you’ll pass); (bf) MOV-reg copies the frame pointer r2 = r10; (07) ADD-imm does r2 += -4 so R2 now holds &key; (18) LD-imm-dw loads the map address into R1; (85) CALL invokes bpf_map_lookup_elem. The calling convention is right there: key in R2, map in R1, result in R0.
• Line 7 shows register state right after the call: R0 is map_value_or_null with id=2 (the linkage id — ref_obj_id is 0, unprinted), offset 0, key size 4 (ks=4, from verbose_a("ks=%d,vs=%d", ...) at kernel/bpf/log.c:675), value size 8 (vs=8). R10=fp0 just means R10 is the frame pointer at offset 0 — read-only, as always.
• The next line is (79) — LDX-dw, a load through R0. That’s the dereference. R0 is still map_value_or_null. Forbidden.
• The violation line: R0 invalid mem access 'map_value_or_null', emitted by verbose(env, "R%d invalid mem access '%s'\n", ...) at kernel/bpf/verifier.c:6408 (and a twin at :6565). It names the register (R0) and the type that failed (map_value_or_null).

The Verifier did exactly what the diagram showed. You skipped the mark_ptr_or_null_regs transition. Fix: add if (!v) return 0;.

Rejection 2 — Deref before check

__u64 *v = bpf_map_lookup_elem(&m, &key);
*v += 1;       // deref before check
if (!v) return 0;

Same log as Rejection 1. The Verifier walks instructions in order; the late check doesn’t help the early deref. The (79) load still executes while R0 is map_value_or_null. Lesson: the null check must be before the deref. Compiler-style “this branch makes the early line unreachable” doesn’t apply — the Verifier evaluates instructions in program order.

Rejection 3 — Conditional null check (verifier path tracking)

__u64 *v = bpf_map_lookup_elem(&m, &key);
if (key == 0) {
    if (!v) return 0;
}
*v += 1;       // is v guaranteed non-NULL here?

(For this to demonstrate anything, key must be runtime-unknown — change the base to __u32 key = bpf_get_current_pid_tgid();. If key were the compile-time 0 from the setup, Clang would constant-fold if (key == 0) to always-true and the program could actually load.)

Verifier rejects. Why? Inside if (key == 0), you proved v non-NULL. But the Verifier explores both branches:

• key == 0 branch: v is checked, then *v += 1 runs with v proven non-NULL. OK.
• key != 0 branch: skips the inner if, falls through to *v += 1. v is still map_value_or_null. Reject.

The error message points at the deref instruction, with state showing R0 is still or-null:

N: R0=map_value_or_null(id=2,off=0,ks=4,vs=8) ...
N: (79) r1 = *(u64 *)(r0 +0)
R0 invalid mem access 'map_value_or_null'

The failing instruction number N is higher than Rejection 1 (the extra if (key == 0) branch adds instructions), but the violation line is identical: on the key != 0 fall-through path R0 never made the OR_NULL → MAP_VALUE transition. (Exact instruction numbers and the id= value vary by kernel version and path.) The Verifier is right — your code is buggy. Fix:

if (!v) return 0;
*v += 1;

Or check unconditionally before deref. Always make the check dominate the use.

Rejection 4 — Reset id by reassignment

__u64 *v = bpf_map_lookup_elem(&m, &key);
if (!v) return 0;
v = bpf_map_lookup_elem(&m, &key);  // re-lookup
*v += 1;                             // does the second result need a new check?

Yes. Each call to bpf_map_lookup_elem returns a fresh PTR_TO_MAP_VALUE_OR_NULL with a new id. The first check resolved the first lookup. The second lookup is unrelated. Verifier rejects the deref of the second result:

N: R0=map_value_or_null(id=3,off=0,ks=4,vs=8) ...   ; note id=3 — a NEW id
N: (79) r1 = *(u64 *)(r0 +0)
R0 invalid mem access 'map_value_or_null'

Compare the id= to Rejection 1’s id=2: the second bpf_map_lookup_elem minted a brand-new linkage id the first if (!v) never resolved, and it fails at a higher instruction number than Rejection 1’s deref. That fresh id is the concrete signal of “new check needed per call.” Remember from the register-file section: the first check flipped every register sharing id=2; nothing was sharing the new id=3. (Exact id values and instruction numbers vary by kernel version.)

Lesson: null state is per-call, not per-variable name. Re-check after every lookup, even if you wrote the same code.

Rejection 5 — Loop and lose track

__u64 *v;
for (int i = 0; i < 3; i++) {
    v = bpf_map_lookup_elem(&m, &key);
    if (!v) continue;
    *v += 1;
}
return 0;

This loads on a modern verifier: v is assigned once per iteration, and every iteration continues past the deref when v is NULL, so the single check dominates the loop body. Now move the guard so each iteration mints a fresh OR_NULL that a once-only check can’t cover:

__u64 *v;
for (int i = 0; i < 3; i++) {
    v = bpf_map_lookup_elem(&m, &key);  // fresh OR_NULL each iteration
    if (i == 0 && !v) return 0;         // only checked on iteration 0
    *v += 1;                            // iterations 1, 2: v unchecked
}
return 0;

This is rejected. The if (i == 0 && !v) guard only covers the first iteration; on iterations 1 and 2 the re-lookup hands back a fresh PTR_TO_MAP_VALUE_OR_NULL that nothing checked before *v += 1:

N: R0=map_value_or_null(id=N,off=0,ks=4,vs=8) ...
N: (79) r1 = *(u64 *)(r0 +0)
R0 invalid mem access 'map_value_or_null'

Contrast the two snippets: the first loads because v is checked-then-used with no skipped check, so the check dominates every unrolled iteration; the second rejects because the per-iteration re-lookup creates an unchecked OR_NULL on the i > 0 paths. (Exact instruction numbers and id= values vary by kernel version.)

Lesson: the check must dominate the use on every execution path. Loops, branches, retries — all paths.

How to read a verifier log fast

Verbose logs look intimidating. They’re really three things stacked:

1. Instruction trace. Each line shows one BPF instruction in pseudo-assembly: index: (opcode) spelling. You now know the opcode bytes — scan the (xx) column for an LDX/STX (0x6x/0x7x) to find the loads and stores.
2. Register state. After interesting instructions (calls, jumps, exits) the Verifier prints the per-register table: R0, R1, … R10. Recall the calling convention from “The eleven registers” above — R0 returns, R1–R5 are clobbered scratch, R6–R9 survive calls, R10 is the read-only frame pointer.
3. Final error. A one-line rejection naming the offending register, type, and offending instruction.

Tactics:

• Search the log for “R0 invalid” or “R0 type=”. That’s where the violation is.
• Walk backward from there a few instructions to see what the Verifier knew — find the (85) call that produced the OR_NULL, and check whether a NULL-comparison jump appears between it and the failing (79) load.
• The instruction numbers match llvm-objdump-21 -d output of your .bpf.o — you can correlate to source if you compile with -g.

The full log goes to kern_log if your loader doesn’t capture it; sudo dmesg | tail -200 after a load failure usually has it (reading the kernel buffer needs root when kernel.dmesg_restrict=1, which is the default on this box).

What to read in the kernel

• include/uapi/linux/bpf.h — the register enum (BPF_REG_0 .. BPF_REG_10, lines 62-78) and the opcode constants (BPF_ALU64, BPF_MOV, BPF_DW, BPF_CALL). The instruction classes and modes live in include/uapi/linux/bpf_common.h. Skim both once so the (79)/(85) notation stops being opaque.
• kernel/bpf/verifier.c:
  • do_check — the main loop. Don’t read it all; just see the structure and the BPF_COMPLEXITY_LIMIT_INSNS check (:17705).
  • mark_ptr_or_null_regs (:16060) — ~20 lines. The function that flips OR_NULL types after a comparison, and (for reference-bearing types) releases the ref_obj_id.
  • check_helper_call — how the Verifier knows what type a helper returns. The metadata comes from each helper’s bpf_func_proto (e.g. bpf_map_lookup_elem_proto in kernel/bpf/helpers.c:50).
• include/linux/bpf.h — see enum bpf_reg_type near PTR_TO_MAP_VALUE_OR_NULL (:1026). Read the comment. This is the canonical list of register types — bookmark it. MAX_BPF_STACK = 512 is in include/linux/filter.h:98.
• include/linux/bpf_verifier.h — struct bpf_reg_state, in particular u32 id (:155) and u32 ref_obj_id (:195). The two-id distinction lives here.
• tools/testing/selftests/bpf/progs/verifier_map_ret_val.c — each test is a 3-line program intended to be rejected, with the expected error embedded. Line 39 has __failure __msg("R0 invalid mem access 'map_value_or_null'") — the exact rejection you produced today.

Today’s experiment

Beyond mutating reject.bpf.c five times, watch the real opcodes:

# Disassemble your compiled object and find the (79) load:
# (the binary is versioned — use the one matching your clang toolchain)
llvm-objdump-21 -d reject.bpf.o

The byte columns there match the (xx) in the verifier log one-to-one. Find the LDX double-word load and confirm it’s the instruction the rejection names. Then add if (!v) return 0;, recompile, and watch that the load now appears after a (15)/(55) JEQ/JNE-against-zero jump — the comparison that lets mark_ptr_or_null_regs flip R0 before the deref.

Bullet Points

• BPF has exactly eleven registers, R0–R10, and the Verifier tracks a bpf_reg_state for each. R0 = return value, R1–R5 = args/scratch (clobbered by calls), R6–R9 = callee-saved, R10 = read-only frame pointer over a 512-byte stack.
• A verifier log line is index: (opcode) spelling. The (xx) is the raw opcode byte; LDX opcodes (0x6x/0x7x) are loads = dereferences, exactly what OR_NULL forbids. The *(u64*)/*(u32*) width echoes your C types.
• The Verifier walks every path of your program with abstract interpretation.
• Every register has a type (e.g., SCALAR_VALUE, PTR_TO_MAP_VALUE_OR_NULL, PTR_TO_BTF_ID).
• PTR_TO_MAP_VALUE_OR_NULL cannot be dereferenced until a NULL check transitions it to PTR_TO_MAP_VALUE via mark_ptr_or_null_regs.
• A reg carries two ids: id (nullness linkage, what map-value uses — free to discharge) and ref_obj_id (an acquired reference you must release, used by socket/mem-or-null). A bare if (!v) return 0; fully discharges a map value.
• Each bpf_map_lookup_elem call returns a fresh OR_NULL with a new id — re-check every time.
• Null check must dominate every dereference path — branches, loops, retries.
• Verifier log = instruction trace + register state + final error. Search the log for “invalid mem access” to land on the violation.
• All these rules apply to every OR_NULL type: socket-or-null, sock-common-or-null, tcp-sock-or-null, ringbuf-mem-or-null, btf-id-or-null, etc. The deref rule is shared; reference-bearing types additionally demand a release. (Packet pointers are not OR_NULL — they’re range-checked against PTR_TO_PACKET_END instead.)
• The Verifier’s complexity budget is 1M instructions explored (BPF_COMPLEXITY_LIMIT_INSNS). Hit it and you get “BPF program is too large.” (Day 5 covers this.)

Check question

The Verifier’s mark_ptr_or_null_regs runs after each conditional jump that compares an OR_NULL register against zero. Why does it have to run on both branches, not just the “non-NULL” one?

Tomorrow

Day 5: bounded loops. The verifier’s other big “I can’t prove this terminates” rejection. We meet bpf_loop, #pragma unroll, and the path-explosion problem (where the 1M BPF_COMPLEXITY_LIMIT_INSNS budget bites and you get “BPF program is too large”). End of phase 1: workflow + verifier intuition. Phase 2 (tracing specialization) starts Day 6.

Day 5 — Bounded loops, bpf_loop, and the path-explosion problem

Today’s mission: understand why “infinite loop” rejection is really “I can’t prove this terminates,” learn what the Verifier actually tracks about a loop counter, and learn the four ways to write loops it accepts (plus iterators, a loop-shaped bonus). End of phase 1. Total time: ~110 minutes.

“I’m just doing a for loop, why is it complaining?”

The Verifier’s contract is every BPF program must terminate within a bounded number of instructions. The reason is operational, not philosophical: BPF runs in kernel contexts where preemption may be disabled, IRQs may be off, scheduling may not happen. A loop that runs forever in such a context wedges the kernel — there is no watchdog that can yank you out. So the Verifier statically refuses any program where it can’t prove termination.

That’s the why. The how has three moving parts that the rest of this chapter leans on constantly, so we build all three before we touch a single loop:

1. A numeric range for every register — the abstract domain the Verifier uses to prove a counter reaches its exit. (New today.)
2. The control-flow-graph pass — a separate stage that finds the back-edge (the loop) before the main walk even starts. (New today.)
3. More than one complexity ceiling — Day 4 taught you the 1M instruction budget; there is a second, smaller cap that branchy loops hit first. (New today.)

Day 4 introduced the Verifier as abstract interpretation: it walks the program symbolically, and every register carries a type and a reference id. Today is about the part Day 4 only named in passing as “bounds” — the numeric part of that same register state. Loops are where it earns its keep.

What the Verifier knows about a number

Here is the single idea the whole chapter hinges on, and most tutorials never state it:

For every register holding a SCALAR_VALUE, the Verifier does not track the value. It tracks a range of possible values — a little box the true value is guaranteed to live inside. Loop termination is proved by watching that box shrink toward the exit condition, not by reading your C source.

Concretely, each scalar register carries five things at once (include/linux/bpf_verifier.h):

struct tnum var_off;        /* line 117 — known-bits mask */
s64 smin_value;             /* line 123 — minimum possible (s64)value */
s64 smax_value;             /* line 124 — maximum possible (s64)value */
u64 umin_value;             /* line 125 — minimum possible (u64)value */
u64 umax_value;             /* line 126 — maximum possible (u64)value */
s32 s32_min_value;          /* line 127 — 32-bit sub-register bounds, too */

Two complementary representations of “what could this number be”:

• The interval — four bounds, signed and unsigned: smin/smax and umin/umax. “This register is somewhere in [umin_value, umax_value].” A brand-new unknown scalar (say, the return of bpf_get_prandom_u32()) starts at the widest possible box: umin=0, umax=U64_MAX. The Verifier knows nothing, so the box is everything.
• The tnum (var_off) — a known-bits mask. For each bit position it records “this bit is definitely 0,” “definitely 1,” or “unknown.” It captures facts the interval can’t, like “the low 7 bits are unknown but every higher bit is zero” — which is exactly what a bitmask operation produces.

Every operation on a register updates both. Add a constant, the interval slides. Mask with & 0x7f, the tnum’s high bits snap to known-zero. Compare in a branch, each side of the branch narrows the box. The Verifier propagates these forward instruction by instruction — this is the “abstract interpretation” from Day 4, made numeric.

Why this is the whole game for loops

A loop is a back-edge: a jump to an earlier instruction. To accept it, the Verifier must prove the back-edge eventually stops being taken. It does this with the range domain:

• for (i = 0; i < 64; i++) — the comparison i < 64 is a hard fact the Verifier folds straight into i’s box: on the loop-body side, umax_value(i) becomes 63. Every iteration adds 1, the box marches upward, and because umax is pinned at 63 the Verifier can prove the exit is reached. Termination provable. The bound 64 is independent of anything else in the program.

• for (i = 0; i < n; i++) where n is an unknown scalar — n’s box is [0, U64_MAX], so i < n tells the Verifier only that i < U64_MAX. That leaves umax_value(i) at U64_MAX. The box never shrinks to a provable exit. Termination unprovable — and that is the reason this is rejected, not some surface-level “n looks variable” heuristic.

This is why answer 3 of the quiz below works: i < n && i < 64 keeps the unbounded n and adds the static i < 64. The 64 caps umax_value(i) regardless of n, so the box shrinks and the loop is provable. The “min trick” is literally “give the range tracker a constant ceiling it can use.”

How the Verifier handles loops: the CFG pass and back-edges

Before the main per-instruction walk even begins, the Verifier runs a separate pass that builds a control-flow graph (CFG) of your program and classifies every jump. This is bpf_check_cfg, invoked as its own stage:

/* kernel/bpf/verifier.c:20033 */
ret = bpf_check_cfg(env);

A CFG is just instructions as nodes and jumps as edges. The pass does a depth-first walk and labels each edge. The one that matters today is the back-edge: a jump to an instruction the DFS has already entered but not yet finished — which is precisely the definition of a loop. When the pass finds one it can’t allow, it says so by line and instruction number:

/* kernel/bpf/cfg.c:138 */
verbose(env, "back-edge from insn %d to %d\n", t, w);

On old or unprivileged paths, a back-edge here is a hard rejection — the Verifier predates bounded-loop support and simply refuses any loop. On modern bounded-loop kernels (5.3+) the back-edge is allowed at this stage, and the question “does it terminate?” is deferred to the range tracking we just built. So the CFG pass is where the Verifier discovers the loop; the range domain is where it judges the loop.

Three ceilings, three different rejection messages

Day 4 taught you the 1-million-instruction budget. That is not the only ceiling, and that is why the failures in this chapter don’t all say the same thing. There are three, and learning to map a message to its cause is half of reading verifier logs:

The third one is the surprise. The insns budget counts how many instructions the Verifier has processed across all paths:

/* kernel/bpf/verifier.c:17705 */
if (++env->insn_processed > BPF_COMPLEXITY_LIMIT_INSNS) {
        verbose(env, ...
                "BPF program is too large. Processed %d insn\n", ...

Separately, the depth of the explore stack — how many pending branch-points the DFS is juggling — is capped much lower:

/* kernel/bpf/verifier.c:1733 */
if (env->stack_size > BPF_COMPLEXITY_LIMIT_JMP_SEQ) {
        verbose(env, "The sequence of %d jumps is too complex.\n", ...

A branchy loop overflows the 8192 jump-sequence stack first, long before it ever processes a million instructions — which is exactly the stress-test result you’ll see later at ~114K insns, well under 1M. When the chapter says “not the message you might expect,” this table is the answer.

So the loop story end-to-end: the CFG pass (1) finds the back-edge, range tracking (2) tries to prove the counter’s box reaches the exit, and the two budgets (3) cap how much exploring that proof is allowed to cost.

There are no Dumb Questions

Q: 1M instructions sounds like a lot. How do programs actually hit it?

A: The 1M is paths explored across all branches, not instructions executed at runtime. A 100-iteration loop with 5 branches in the body explodes to ≈100×2⁵ = 3200 paths, each with maybe 50 instructions, that’s 160K — a lot already. Add nested loops or longer bodies and you breach 1M. The Verifier prunes aggressively but path-explosion is real.

Q: Why does state pruning sometimes work and sometimes not?

A: State pruning compares register states between two visits to the same instruction. If state A is “weaker than or equal to” state B (every register in A has a wider type than B), B can be pruned: anything safe in A is safe in B. The trick is that scalar values often narrow (the loop counter goes up by 1), making each iteration’s state strictly different from the prior one. Modern verifiers use scalar widening tricks to coerce many iterations into one merged state. See the liveness.c and states.c files added in 2025–2026. (“Narrows” here means exactly the range domain above: each iteration’s [umin,umax] for the counter is a different box, so the states don’t match unless the Verifier deliberately widens them back together.)

Q: I see #pragma unroll in old tutorials. Should I use it?

A: Sometimes. #pragma unroll tells Clang to expand a loop at compile time. The Verifier then sees a straight-line program with no back-edge — definitely terminates. Works for small fixed counts. Doesn’t scale: unrolling 1000 iterations produces 1000× the code, blowing past instruction limits. Use bpf_loop for non-trivial counts.

Sharpen your pencil

Predict which of these the Verifier accepts.

1. for (int i = 0; i < 10; i++) sum += i;
2. for (int i = 0; i < n; i++) sum += i; (where n is from bpf_get_prandom_u32())
3. for (int i = 0; i < n && i < 100; i++) sum += i; (same n)
4. int i = 0; while (1) { if (i >= 10) break; sum += i; i++; }

.
.
.

Answers: 1 ✓ (constant bound — umax(i) pinned at 9, box shrinks, easy unroll). 2 ✗ (“infinite loop” — n’s box is [0,U64_MAX], so umax(i) stays U64_MAX and termination is unprovable). 3 ✓ (the i < 100 is the static-cap trick from the range section — it caps umax(i) at 99 regardless of n). 4 ✓ (functionally identical to 1 — the if (i >= 10) break gives the range tracker the same umax(i) <= 9 fact).

Meet the four ways to loop

1. Constant-bound for (default first choice)

for (int i = 0; i < 16; i++)
    process(i);

The Verifier unrolls; works for small bounds. The constant 16 caps umax_value(i) immediately, so the back-edge is provable. Default first choice when the count is small (≤ ~64).

2. Variable-bound with a static cap (the “min” trick)

for (int i = 0; i < n && i < 64; i++)
    process(i);

Even when n is dynamic, the i < 64 clause pins umax_value(i) at 63 — a hard fact independent of n. The Verifier’s range box shrinks each iteration, so it accepts. Useful when n is “the smaller of: a parsed length, or our buffer size.”

3. #pragma unroll (deprecated for non-trivial counts)

#pragma unroll
for (int i = 0; i < 8; i++)
    out[i] = in[i] ^ 0xff;

Compiles to a straight-line sequence — no back-edge, so the CFG pass never even sees a loop, and the range tracker has nothing to prove. Fine for tiny counts. Bad for anything ≥ ~64 because it bloats program size.

4. bpf_loop (the modern answer for any non-trivial count)

static int my_cb(__u32 i, void *ctx) {
    /* ... */
    return 0;   /* return 1 to break early */
}

bpf_loop(1000000, my_cb, &ctx_struct, 0);

The Verifier verifies the callback once as a single iteration. Then it treats bpf_loop as a black-box helper that runs your verified callback up to N times. Complexity is independent of N.

This is the breakthrough that came in kernel 5.17 (commit e6f2dd0f8067 — “bpf: Add bpf_loop helper”). Before it, you were stuck unrolling. After, loops in BPF feel like loops in real C. But it has its own model, and it’s worth understanding why it works — that’s the next section.

What a BPF callback actually is

The exact line numbers below track the 7.1 tree; the symbol names (bpf_loop_proto, set_loop_callback_state, the ARG_* type tags) are the stable anchors to grep for on any kernel.

bpf_loop does something almost no other BPF program does: it takes a function pointer as an argument. To handle that, the Verifier has a dedicated argument type, and you can see it right in the helper’s prototype (kernel/bpf/bpf_iter.c:753-761; the two arg-types quoted below are at :758-759):

const struct bpf_func_proto bpf_loop_proto = {
    ...
    .arg2_type  = ARG_PTR_TO_FUNC,            /* the callback function */
    .arg3_type  = ARG_PTR_TO_STACK_OR_NULL,   /* the ctx — must be stack */
    ...
};

Two things to unpack here, and they explain the entire lab.

ARG_PTR_TO_FUNC — verify the callee once, as one generic iteration. When the Verifier sees the function-pointer argument, it treats your callback as a separate subprogram and walks it symbolically one time, with the index i modeled as a fully unknown scalar — set_loop_callback_state (verifier.c:9661) just marks R1 as a bare SCALAR_VALUE, leaving its range at the widest [0, U64_MAX] box. It is not narrowed to [0, nr_loops); the Verifier deliberately treats the index as completely opaque so that one symbolic walk covers every iteration regardless of nr_loops. It does not stamp out N copies of the body. That single symbolic walk is the whole reason cost is independent of N. The machinery that sets this up has a name:

/* kernel/bpf/verifier.c:9661 */
static int set_loop_callback_state(struct bpf_verifier_env *env, ...

invoked at verifier.c:10447 when the Verifier processes the bpf_loop call.

The callback contract is enforced, not advisory. Your callback must return 0 (continue) or 1 (stop). At the callback’s return point the Verifier applies the same range machinery from the start of this chapter, demanding R0 (the return register) lie in [0, 1]:

/* kernel/bpf/verifier.c:9896 */
"At callback return", "R0"

and the out-of-range complaint itself is the generic range message:

/* kernel/bpf/verifier.c:303 */
verbose(env, " should have been in [%d, %d]\n", range.minval, range.maxval);

This is the bridge that ties the chapter together: the scalar range domain you learned for proving loop termination is the exact same mechanism that checks a callback’s return value. “Bounded” and “in the allowed range” are different questions, and Break 4 below trips precisely on this.

Why the ctx must be stack memory. The 3rd argument is ARG_PTR_TO_STACK_OR_NULL — it must point at stack memory (fp = frame pointer). Hand it a raw PTR_TO_MAP_VALUE (a pointer into a map, like the one bpf_map_lookup_elem returns) and the Verifier rejects it:

R3 type=map_value expected=fp

That’s why the lab wraps its map pointer in a tiny stack struct and passes that struct’s address — it launders the map pointer through a stack slot to satisfy ARG_PTR_TO_STACK_OR_NULL. The wrapper isn’t a style choice; it’s the contract.

The runtime side is mundane. All the cleverness is on the verification side. At runtime, bpf_loop is an ordinary C for loop in the kernel (kernel/bpf/bpf_iter.c:743-751):

for (i = 0; i < nr_loops; i++) {
        ret = callback((u64)i, (u64)(long)callback_ctx, 0, 0, 0);
        /* return value: 0 - continue, 1 - stop and return */
        if (ret)
                return i + 1;
}

It calls your callback up to nr_loops times and stops early on a non-zero return. There is a runtime ceiling, too: nr_loops > BPF_MAX_LOOPS returns -E2BIG (bpf_iter.c:740-741), where BPF_MAX_LOOPS = 8 * 1024 * 1024 (include/linux/bpf.h:2268). So the “magic” is verify-once on the Verifier side; execution is a plain loop.

The catch: the callback can’t directly access local variables of the calling function. You pass state through the ctx pointer (last argument) — which, as we just saw, must live on the stack.

Bonus: Iterators (a fifth, loop-shaped tool)

For walking kernel data structures (tasks, sockets, cgroups, BTF), there are dedicated BPF iterators — task_iter, sock_iter, etc. These are programs, not loops, so they sit outside the four-way framing above. We meet them on Day 17. For now: just know they exist as a loop-shaped tool.

The lab

loops.bpf.c — try every shape

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

char LICENSE[] SEC("license") = "GPL";

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, 1);
    __type(key, __u32);
    __type(value, __u64);
} sum SEC(".maps");

/* helper to get the singleton sum slot */
static __u64 *get_sum(void) {
    __u32 z = 0;
    return bpf_map_lookup_elem(&sum, &z);
}

/* shape 1: constant bound */
SEC("fentry/filename_unlinkat")
int BPF_PROG(loop_const)
{
    __u64 *s = get_sum();
    if (!s) return 0;
    /* The per-iteration helper call keeps clang -O2 from constant-folding
       the loop into a single `*s += 120`. Without it there is no loop left
       for the Verifier to see — and the comparison below would be bogus. */
    for (int i = 0; i < 16; i++)
        *s += bpf_get_prandom_u32();
    return 0;
}

/* shape 4: bpf_loop callback. bpf_loop's ctx argument is
   ARG_PTR_TO_STACK_OR_NULL, so it must point at *stack* memory. Passing the
   raw map-value pointer (a PTR_TO_MAP_VALUE) is rejected:
       R3 type=map_value expected=fp
   so we wrap the map pointer in a small stack struct and pass its address. */
struct cb_ctx { __u64 *s; };
static int cb(__u32 i, void *ctx)
{
    struct cb_ctx *c = ctx;
    *c->s += i;
    return 0;
}

SEC("fentry/filename_unlinkat")
int BPF_PROG(loop_helper)
{
    __u64 *s = get_sum();
    if (!s) return 0;
    struct cb_ctx c = { .s = s };
    bpf_loop(10000, cb, &c, 0);
    return 0;
}

The two programs share a 1-element array map (a “global counter” pattern — recall the array-map singleton trick from Day 2). Build and load both. Note cb_ctx: it exists for exactly one reason — to give bpf_loop a stack address (&c) instead of the raw map pointer s, satisfying ARG_PTR_TO_STACK_OR_NULL.

Inspect the verifier’s effort

Both programs above load successfully — and that changes how you measure them. bpftool prog load only prints a verifier log when a program is rejected, so on a clean load it shows you nothing useful. To see the effort on a successful load, use one of these two paths.

Path 1 — veristat (the purpose-built tool). It load-tests every program in an object file and prints a per-program stats table:

sudo veristat loops.bpf.o

The Insns and States columns are exactly the comparison we want — no attach needed. (veristat ships in the kernel tree under tools/testing/selftests/bpf and may need building/installing; it is not in $PATH by default.)

Path 2 — your own loader. Set the verifier log level before load and read the bottom line of the printed log:

LIBBPF_OPTS(bpf_object_open_opts, opts, .kernel_log_level = 1);

The last line of loop_helper’s log looks like:

processed 24 insns (limit 1000000) ...

processed N insns is the path-exploration cost. The exact counts are kernel- and compiler-version dependent, so capture your own rather than trusting any number printed here — but the relationship is the lesson:

• loop_const (16-iter inline) verifies every unrolled iteration, so its insn count scales with the iteration count.
• loop_helper (10K bpf_loop) verifies the callback once — that’s the set_loop_callback_state single-walk in action — so its cost is tiny and independent of N — about 24 insns on this 7.1 kernel.

bpf_loop is cheaper for the Verifier even though it does more work at runtime, because the callback verifies once.

Stress test: when does inline break?

Try replacing loop_const with a branchy, bounded inline loop:

for (int i = 0; i < N; i++) {
    __u64 v = bpf_get_prandom_u32();
    if (v & 1) *s += v;
    else       *s -= v;
}

Each N needs a fresh build, so: edit N, rebuild (make, or rerun your clang command), reload, repeat. Walk N up: 16, 64, 256, 1024, 10000, and watch processed insns grow non-linearly. N=1024 still loads; around N=10000 on a 7.x kernel it is rejected — but not with the “too large” message you might expect. A branchy body overflows the jump-sequence stack (BPF_COMPLEXITY_LIMIT_JMP_SEQ = 8192, the second of the three ceilings) long before the 1,000,000-instruction budget:

The sequence of 8193 jumps is too complex.
processed 114705 insns (limit 1000000)

Note it failed at ~114K insns — far below the 1M limit. That’s the table from earlier doing its work: this is the jump-sequence depth cap (verifier.c:1733), not the insns-processed budget (verifier.c:17705). The flat BPF program is too large. Processed 1000001 insn message comes from a genuinely unbounded loop instead; see Break 1.

Replace the for with bpf_loop. Loads in milliseconds. The branch in the body × 10000 iterations was 2¹⁰⁰⁰⁰ paths if the Verifier didn’t prune — bpf_loop makes it just paths(callback), because the callback verifies once with i modeled as a fully unknown (unbounded) scalar.

What to break, in order

Break 1 — Unbounded loop

__u32 n = bpf_get_prandom_u32();
for (__u32 i = 0; i < n; i++)
    *s += bpf_get_prandom_u32();   /* per-iteration helper call: cannot be folded */

Note the body. If you write *s += i;, clang -O2 recognises the closed form n*(n-1)/2 and compiles it to straight-line arithmetic with no back-edge — the program then loads fine and this break never fires. The per-iteration bpf_get_prandom_u32() defeats that folding so a real unbounded loop survives to the Verifier.

Reject — n is an unknown scalar, so its range box is [0, U64_MAX], which leaves umax_value(i) at U64_MAX: the Verifier can never prove the counter reaches the exit, so it explores until it exhausts the complexity budget:

BPF program is too large. Processed 1000001 insn

This is the insns-processed ceiling (BPF_COMPLEXITY_LIMIT_INSNS, verifier.c:17705). On older kernels — pre-5.3, before bounded-loop support — the CFG pass rejected this earlier as back-edge from insn N to M from the check_cfg DAG check (cfg.c:138); that wording no longer fires for this case on modern kernels, which allow the back-edge and defer the verdict to range tracking. Add && i < 1024 — the static-cap trick from the range section gives umax_value(i) a hard ceiling of 1023.

Break 2 — Bound that isn’t visible

__u32 n = bpf_get_prandom_u32() % 100;
for (__u32 i = 0; i < n; i++)
    *s += bpf_get_prandom_u32();   /* non-foldable body — keeps a real loop */

(As in Break 1, the body must be something clang can’t fold: with *s += i; the whole loop collapses to n*(n-1)/2 and no loop reaches the Verifier, so there is no range to track and the lesson is lost.)

The Verifier might accept this on modern kernels (it can track n’s range as [0, 99] and thus umax_value(i) <= 98). On older kernels it rejects, because modulo arithmetic doesn’t always propagate a tight umax through the scalar tracker — the range can stay wide even though % 100 logically bounds it. Compare that to a bitwise AND, which the tnum captures immediately:

n = n & 0x7f;        // 0..127, verifier definitely sees the bound

& 0x7f sets every bit above bit 6 to known-zero in n’s tnum (var_off), which forces umax_value(n) to 127 on the spot. The high bits become a hard fact, so i < n now caps umax_value(i) at 126 and the loop is provable. Lesson: a bitmask updates the known-bits domain directly; modulo may not — be explicit when you can be.

Break 3 — Hit the complexity budget

A pattern that looks pathological:

char buf[256];
bpf_get_current_comm(buf, sizeof(buf));
for (int i = 0; i < 256; i++) {
    if (buf[i] == '/') { /* do A */ }
    else if (buf[i] == ' ') { /* do B */ }
    else { /* do C */ }
}

Surprise: on a modern (7.x) Verifier this loads cleanly. The branch bodies are empty, so every path leaves register/scalar state unchanged and state pruning collapses them all — the 256 iterations never explode. To actually breach the budget you need both a high iteration count (well past N=10000 on 6.6+, as in the stress test above) and branch bodies that diverge register state so the paths can’t be pruned. Give each branch a state-changing side effect and raise the count:

for (int i = 0; i < N; i++) {
    if      (buf[i] == '/') *s += i;
    else if (buf[i] == ' ') *s -= i;
    else                    *s ^= i;
}

Now the paths diverge (each branch leaves a different register-state box, so pruning can’t merge them) and, at a large enough N, you overflow the jump-sequence stack: The sequence of NNNN jumps is too complex. — the same BPF_COMPLEXITY_LIMIT_JMP_SEQ cap from the stress test, not the 1M insns budget. Fix: refactor with bpf_loop, or reduce the iteration count to the longest sensible string length.

Break 4 — Callback returning the wrong thing

Wire a callback that returns something other than 0 or 1 into the (now-loadable) loop_helper from the lab:

static int cb(__u32 i, void *ctx) {
    return 2;     // valid returns: 0 (continue) or 1 (break)
}

Verifier rejects:

At callback return the register R0 has smin=2 smax=2 should have been in [0, 1]

Read that message through the range domain. smin=2 smax=2 means R0 is the exact constant 2 — a perfectly bounded value; its box is the single point {2}. It is rejected anyway, because the callback-return range check (verifier.c:9896, “At callback return”, “R0”) demands R0 lie in [0, 1], and should have been in [0, 1] is the generic range complaint at verifier.c:303. “Bounded” and “in the allowed range” are different questions: the value is bounded, just bounded to the wrong range. This is the same scalar-range machinery that proves loop termination, now applied at the subprogram’s exit. (Exact register fields vary by kernel version. You can only reach this check once loop_helper’s ctx is fixed, otherwise the R3 type=map_value expected=fp error fires first.)

What to read in the kernel

• kernel/bpf/bpf_iter.c — search BPF_CALL_4(bpf_loop (line 727). The runtime implementation is the tight for loop at lines 743–751 calling the verified callback; the proto at 753–761 declares ARG_PTR_TO_FUNC (:758) and ARG_PTR_TO_STACK_OR_NULL (:759). The Verifier knows about this helper specifically and verifies the callback once via set_loop_callback_state (verifier.c:9661).
• include/linux/bpf_verifier.h — read struct bpf_reg_state around lines 117–127: var_off (the tnum) plus the smin/smax/umin/umax interval and the 32-bit sub-register bounds. This is the abstract domain. Spend five minutes here; it pays off for every future verifier log.
• kernel/bpf/verifier.c:
  • search bpf_is_state_visited (defined in kernel/bpf/states.c) — the heart of state pruning.
  • search process_bpf_exit_full — how state propagates across paths at exit. For the liveness machinery it leans on, see kernel/bpf/liveness.c.
  • line 20033 bpf_check_cfg(env) — the CFG pass that finds back-edges.
• kernel/bpf/cfg.c — line 138, the back-edge from insn %d to %d message, so you can see where the loop is actually discovered.
• tools/testing/selftests/bpf/progs/bpf_loop.c — official examples of every loop shape.
• Documentation/bpf/verifier.rst — the kernel’s own doc on how the Verifier handles loops and value tracking. Worth one read.

Bullet Points

• The Verifier rejects unbounded loops because it must prove termination statically — there’s no kernel watchdog to recover from infinite BPF.
• Every scalar register carries a numeric range, not just a type: signed/unsigned min/max plus a tnum (var_off) known-bits mask. Loop bounds are proved by watching this box shrink toward the exit, not by reading C source.
• A back-edge (loop) is found by the separate CFG pass (bpf_check_cfg); termination is then judged by the range domain. i < 64 pins umax(i)=63; an unbounded i < n leaves it at U64_MAX and is unprovable.
• Three ceilings, three messages: back-edge from insn N to M (CFG pass), BPF program is too large (1M insns-processed budget), The sequence of N jumps is too complex (8192 jump-sequence stack — branchy loops hit this first).
• Constant-bound for is fine for small counts (≤ ~64); variable bound with static cap (i < n && i < CAP) hands the range tracker a constant ceiling; #pragma unroll removes the back-edge entirely (tiny counts only).
• bpf_loop(N, cb, ctx, 0) is the modern answer: the callback (a subprogram passed via ARG_PTR_TO_FUNC) is verified once as a single generic iteration, so cost is independent of N. At runtime it’s a plain kernel for loop. The ctx must be stack memory (ARG_PTR_TO_STACK_OR_NULL) — hence the stack wrapper.
• The callback contract (return 0 or 1) is enforced by the same range machinery: R0 must be in [0,1] at return — “bounded” ≠ “in the allowed range” (see Break 4).
• A bitwise & 0x7f sets known-zero bits in the tnum and snaps umax to 127 immediately; modulo may not propagate a tight max — be explicit.

Check question

A loop iterates 100 times. Inside, you have an if/else with 4 conditions (so 4 branches at one point). Roughly how many paths does the Verifier explore without state pruning? And with bpf_loop instead of an inline loop?

End of Phase 1

You now have:

• A working libbpf+CO-RE+ringbuf+hashmap workflow.
• The instinct to null-check after every lookup or reserve.
• The vocabulary to read verifier logs — including the three distinct rejection messages and what each one means.
• An understanding of what the Verifier does — abstract interpretation over a numeric range domain, with state pruning, under (more than one) complexity budget.

That’s enough foundation to do real BPF work. Phase 2 (Days 6–13) specializes you into tracing: typed function tracing with fentry/fexit at scale, tracepoints, stack traces, uprobes, and sleepable programs.

Tomorrow

Day 6: typed function tracing with fentry/fexit at scale. We attach to kernel functions by name and read their arguments with full type information — no more PT_REGS_PARM guesswork. You’ll see how BTF lets fentry know the real signature of the function you hook, and why that makes argument access both safer and faster than the kprobe path.

Day 6 — fentry/fexit and the latency-measurement pattern

Today’s mission: measure the latency of every vfs_read on your system. Build a map keyed by thread, store the entry timestamp on call-in, retrieve it on call-out, emit duration to userspace. Along the way: meet fexit and understand exactly how it differs from the older kretprobe (the return-address swap, the instance pool, why bursts get missed), learn what vfs_read actually is and what its return value means, and pick the right per-thread key. Total time: ~110 minutes.

Phase 2 starts here. Days 1–5 made you fluent with libbpf, CO-RE, maps, and the Verifier. Days 6–13 turn that fluency into tracing: typed kernel-function tracing at scale, tracepoints, stack traces, uprobes, and sleepable programs. By Day 13 you’ll be able to write the kind of tracer that ships in production observability tools.

The pattern that powers every tracer

Most tracing problems reduce to one shape:

“When event A happens, remember something. When the matching event B happens, retrieve and act on it.”

For latency: A = function entry, B = function exit. Remember the timestamp on A; on B compute now - remembered.

Yesterday you saw fentry. Today you meet its other half — fexit — and use them as a pair to do the most common BPF tracing pattern in the world.

The fentry program runs before the function body. The fexit program runs after it. Both are connected to the same trampoline; the kernel installs them as a coordinated pair when you load the BPF object. Between entry and exit, the function does its work — could be 100 ns, could be 100 ms.

What is vfs_read, and what does its return value mean?

Before we hook anything, know your target. Today we trace vfs_read, and the lab only makes sense if you know two facts about it.

First: vfs_read is the kernel’s single central entry point for the read(2)/pread(2) family of syscalls. Every file-descriptor read — whether the fd is a regular file, a socket, a pipe, /dev/zero, or a procfs node — funnels through vfs_read before the kernel dispatches to that particular filesystem’s own read handler. That makes it a perfect chokepoint: hook vfs_read once and you see all reads on the box. (You actually used fentry/vfs_read back in a Day 2 break without explanation — now you know why it’s such a popular target.)

Second: its signature is exactly the four typed arguments our fentry program will declare, and its return type is exactly what fexit captures. In v7.1:

/* fs/read_write.c:554 */
ssize_t vfs_read(struct file *file, char __user *buf, size_t count, loff_t *pos)

• struct file *file — the open file the fd refers to (Day 1 noted struct file lives in vmlinux.h; that’s all we need).
• char __user *buf — the userspace destination buffer.
• size_t count — how many bytes the caller asked for.
• loff_t *pos — pointer to the file offset.

The return type ssize_t is the contract our fexit program reads. On success it’s the number of bytes actually read — which can be less than count (a short read, or EOF returning 0). On failure it’s a negative errno. You can see all three failure returns right at the top of the function:

/* fs/read_write.c:558-563 */
if (!(file->f_mode & FMODE_READ))
    return -EBADF;
if (!(file->f_mode & FMODE_CAN_READ))
    return -EINVAL;
if (unlikely(!access_ok(buf, count)))
    return -EFAULT;

and the success value comes straight from the filesystem’s own read op:

/* fs/read_write.c:571-573 */
if (file->f_op->read)
    ret = file->f_op->read(file, buf, count, pos);
else if (file->f_op->read_iter)
    ret = new_sync_read(file, buf, count, pos);

That’s why today’s lab prints the return value as a signed number labelled “bytes”: vfs_read → 2543 bytes is a successful 2543-byte read; a negative number would be an errno. We don’t need the full VFS read path (->read vs ->read_iter dispatch) — just “central read chokepoint” and “return = bytes-or-negative-errno,” so the output isn’t magic.

Meet fexit

fexit is the counterpart to fentry: it attaches at function return. But unlike fentry, you also get the return value handed to you as a typed argument. That last argument matters — it’s what makes fexit strictly more powerful than the older kretprobe.

To appreciate why fexit is the better tool, you have to understand how the old return-probe machinery works — because the chapter is about to contrast against it repeatedly, and a kretprobe hooks a function’s return in a fundamentally trickier way than the entry kprobe you already know.

Refresher: the entry kprobe (Day 1)

Day 1 already explained the entry kprobe: it overwrites the probed function’s first instruction with an int3 breakpoint byte. When the CPU executes that byte it traps, the kprobe handler runs, then the original instruction is single-stepped or emulated and execution continues. On x86 the emulation helpers are right there — int3_emulate_call, int3_emulate_ret, int3_emulate_jmp (arch/x86/kernel/kprobes/core.c:507-525). We do not re-teach that here. The entry hook is solved.

The new question is: how do you run code at the return? There’s no single instruction to breakpoint — a function can ret from many places, and the same code runs for every caller.

How a kretprobe hijacks the return address

A kretprobe is layered on top of an entry kprobe, and it plays a stack trick:

1. At function entry, the kprobe handler reads the function’s real return address off the kernel stack and saves it in a per-call object. Then it overwrites that saved return address on the stack with the address of a kernel trampoline.
2. The function runs normally and eventually does ret. But ret now pops the trampoline address, not the real caller — so control lands in the trampoline.
3. The trampoline runs your return handler (which can read the return value), then jumps to the real saved caller address so the program continues as if nothing happened.

On x86 that trampoline is hand-written assembly whose address is what gets stuffed onto the stack:

/* arch/x86/kernel/rethook.c:26-38 */
"arch_rethook_trampoline:\n"
    ...
    "   pushq $arch_rethook_trampoline\n"   /* fake return addr for the unwinder */
    ...
    "   call arch_rethook_trampoline_callback\n"

The instance pool. To save that real return address you need one object per in-flight call. Recursion, preemption, and many CPUs all mean several calls to the same function can be “in flight” at once, each needing its own saved address. So the kernel pre-allocates a fixed pool of these objects, sized by maxactive. The fields live right in struct kretprobe:

/* include/linux/kprobes.h:150-154 */
int maxactive;
int nmissed;
...
struct rethook *rh;

and the per-call save object is:

/* include/linux/kprobes.h:162-164 */
struct kretprobe_instance {
    ...
    struct rethook_node node;
    ...
};

The in-tree comment spells out the sizing contract: “maxactive - The maximum number of instances of the probed function that can be active concurrently” and “nmissed - tracks the number of times the probed function’s return was ignored, due to maxactive being too low” (include/linux/kprobes.h:135-138). If more calls are in flight simultaneously than the pool size, the extra returns simply aren’t probed — the kernel bumps nmissed and moves on. That’s what “bursts can exhaust the instances pool” means, made concrete: a sudden flood of concurrent calls silently loses return events.

The unwinder problem. Because the kretprobe rewrote the saved return address, anything that walks the kernel stack to reconstruct caller frames sees the trampoline address where the true caller should be. In the kernel this means the in-kernel stack unwinders (ORC on x86_64, frame-pointer unwinding) that back dump_stack(), /proc/<pid>/stack, perf/ftrace backtraces, and bpf_get_stack — there is no C++ exception machinery in the kernel; it’s all C. (The x86 trampoline even pushes a fake $arch_rethook_trampoline frame to tell the unwinder “this is a rethook,” precisely because the real return address is hidden.) This is the “stack-modifying tricks break unwinding” remark, made concrete.

Why fexit avoids all of that

fexit is part of the same trampoline fentry uses. The trampoline saves arguments on entry, calls fentry programs, calls the original function, gets the return value, appends it to the context array, then calls fexit programs.

Compare the two mechanisms point for point:

• kretprobe rewrites the saved return address, needs a finite per-probe instance pool, drops returns (nmissed++) on bursts, and confuses stack unwinders. Effective, but invasive.
• fexit never touches the return address. The original function returns normally to its real caller; the trampoline captured the return value alongside, so there’s no stack rewrite, no finite pool, and no missed-on-burst failure mode.

That’s the whole argument: prefer fexit wherever it’s available — any function with BTF info (essentially every non-__attribute__((always_inline)) kernel function on a modern build). Use kretprobe only when fexit isn’t available.

There are no Dumb Questions

Q: Are fentry and fexit two separate hooks, or one?

A: One hook with two callbacks, conceptually. The kernel builds a single trampoline per attach target that invokes all attached fentry programs and all attached fexit programs in sequence. You can attach an fentry without an fexit, or vice versa. They share the trampoline infrastructure (kernel/bpf/trampoline.c) but are independent BPF programs from your perspective.

Q: What if the function never returns (panic, BUG_ON)?

A: Then fexit doesn’t run. Your map entry stays in the hash table indefinitely. We’ll see this exact failure mode in today’s “what to break” — it’s the source of slow leaks in tracers that run for weeks.

Q: Can a function be called recursively?

A: Yes, and it’s the case where TID-as-key breaks down. Today’s check question at the end walks through exactly why. For most kernel functions you trace, recursion isn’t an issue.

Meet bpf_ktime_get_ns

__u64 ts = bpf_ktime_get_ns();

Returns nanoseconds on the monotonic clock (CLOCK_MONOTONIC): time advances steadily and never goes backward, but it excludes any interval the machine spent suspended, so it isn’t literally “elapsed since boot.” That’s exactly what you want for measuring a latency delta — both reads use the same clock and the difference is meaningful. Cheap — implemented as ktime_get_mono_fast_ns under the hood, no locking, single read of a kernel timekeeping cache. ~10ns to invoke. It is not calendar/wall-clock time and you can’t correlate it with userspace clock_gettime(CLOCK_REALTIME) timestamps. If you need elapsed real time that also counts suspend, use bpf_ktime_get_boot_ns (CLOCK_BOOTTIME — still a monotonic count from boot, just inclusive of suspend, not wall-clock). The closest thing to wall-clock here is bpf_ktime_get_tai_ns (CLOCK_TAI, which tracks UTC up to a fixed leap-second offset).

TID vs TGID — pick the right key

This is the bug that bites everyone the first time they write a per-thread tracer. Read this carefully before writing code today.

bpf_get_current_pid_tgid() returns a packed u64:

+------------------+------------------+
| TGID (upper 32)  | TID (lower 32)   |
+------------------+------------------+

Linux kernel terminology and userspace terminology don’t match:

• What userspace calls “PID” is the kernel’s TGID (thread group ID — the leader thread’s TID).
• What the kernel calls TID is the per-thread identifier.

A multi-threaded process has many TIDs sharing one TGID. Inside bpf_get_current_pid_tgid:

__u32 tgid = bpf_get_current_pid_tgid() >> 32;       // userspace "PID"
__u32 tid  = bpf_get_current_pid_tgid() & 0xffffffff; // unique per thread

For tracking concurrent in-flight calls, you need per-thread granularity:

If you key by TGID, four threads of the same process all hit the same map slot, overwriting each other’s timestamps. The latency reading is garbage. Use TID.

Sharpen your pencil

Why doesn’t the kernel just give you a single “thread identifier” without ambiguity? Why does bpf_get_current_pid_tgid pack two fields into one return value?

.
.
.

Answer: because one helper call is cheaper than two. The kernel only ever exposed the combined helper bpf_get_current_pid_tgid (helper id 14) — there was never a separate bpf_get_current_pid or bpf_get_current_tgid in the UAPI. The single call returns both fields packed together, mirroring what task_struct itself carries: both pid (kernel meaning) and tgid. The body is literally return (u64) task->tgid << 32 | task->pid; (kernel/bpf/helpers.c:225-233). The naming confusion is historical — userspace called processes “PIDs” before threads existed.

The lifecycle, explicitly

Walk through what the map looks like over time when two threads concurrently read:

Each thread enters → its TID gets a slot. Each thread exits → its TID’s slot is read and deleted. The map is empty when nothing is in flight. The delete is critical — without it, the map fills as threads come and go.

The lab

latency.h — the shared event record

The producer and consumer share one struct through a header, exactly as Days 1 and 3 did:

#ifndef PRACTICAL_EBPF_LATENCY_H
#define PRACTICAL_EBPF_LATENCY_H

struct event {
    unsigned int pid;
    unsigned int tid;
    unsigned long long dur_ns;
    long long ret;
    char comm[16];
};

#endif

latency.bpf.c

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

#include "latency.h"

char LICENSE[] SEC("license") = "GPL";

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, __u64);    /* tid */
    __type(value, __u64);  /* ns timestamp */
} starts SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

SEC("fentry/vfs_read")
int BPF_PROG(on_enter, struct file *f, char *buf, size_t n, loff_t *pos)
{
    __u64 tid = bpf_get_current_pid_tgid() & 0xffffffff;
    __u64 ts = bpf_ktime_get_ns();
    bpf_map_update_elem(&starts, &tid, &ts, BPF_ANY);
    return 0;
}

SEC("fexit/vfs_read")
int BPF_PROG(on_exit, struct file *f, char *buf, size_t n, loff_t *pos, ssize_t ret)
{
    __u64 id = bpf_get_current_pid_tgid();
    __u64 tid = id & 0xffffffff;
    __u64 *ts = bpf_map_lookup_elem(&starts, &tid);
    if (!ts)
        return 0;
    __u64 dur = bpf_ktime_get_ns() - *ts;
    bpf_map_delete_elem(&starts, &tid);

    struct event *e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
    if (!e) return 0;
    e->pid = id >> 32;
    e->tid = tid;
    e->dur_ns = dur;
    e->ret = ret;
    bpf_get_current_comm(&e->comm, sizeof(e->comm));
    bpf_ringbuf_submit(e, 0);
    return 0;
}

The fentry signature (struct file *f, char *buf, size_t n, loff_t *pos) is exactly the four vfs_read arguments we read off fs/read_write.c:554 above. The fexit signature is those same four args plus the captured ssize_t ret — the byte-count-or-negative-errno we discussed, which is what e->ret carries to userspace.

What’s new since Day 1–5

• Two programs in one object, sharing maps. The skeleton exposes both as skel->progs.on_enter and skel->progs.on_exit. Both auto-attach when you call latency_bpf__attach(skel).
• BPF_PROG(on_exit, ..., loff_t *pos, ssize_t ret) — fexit receives all function arguments first, then the return value as the last parameter. The BPF_PROG macro knows fexit programs receive one extra ctx slot. We meet BPF_PROG’s mechanics in detail on Day 7.
• bpf_map_update_elem(..., BPF_ANY) — insert-or-update. Note we use BPF_ANY here, not BPF_NOEXIST, because we don’t care if the same TID had a stale entry from a missed fexit (e.g., the previous call panicked). Overwriting is correct.
• bpf_map_lookup_elem returns a pointer into live map memory. You can read it directly. The Verifier requires the null check.
• bpf_map_delete_elem — do not skip this. It’s the line that prevents the map from filling forever.

latency.c

Standard ringbuf consumer. Print:

printf("PID %u TID %u vfs_read → %lld bytes in %llu µs (%s)\n",
       e->pid, e->tid, (long long)e->ret,
       e->dur_ns / 1000, e->comm);

(We don’t capture the request size in this minimal version. Add it as an exercise.)

Run it

Use two terminals so the tracer and the workload don’t fight over one shell.

Terminal A — build and run the tracer in the foreground, and wait until it has attached before generating work (a vfs_read issued before latency_bpf__attach() returns is silently missed):

make
sudo ./latency

Terminal B — once the tracer is live, generate some reads:

cat /etc/passwd > /dev/null
dd if=/dev/zero of=/dev/null bs=1k count=100

Expected (in Terminal A):

PID 14001 TID 14001 vfs_read → 2543 bytes in 12 µs (cat)
PID 14002 TID 14002 vfs_read → 1024 bytes in 8 µs (dd)
PID 14002 TID 14002 vfs_read → 1024 bytes in 6 µs (dd)
...

The → N bytes column is vfs_read’s return value: 2543 bytes read by cat, 1024 per dd block. A negative value there would be an errno (e.g. -9 for -EBADF) — that’s the contract we read off fs/read_write.c:558-573.

This traces every vfs_read on the box, not just cat/dd — sshd, systemd, journald, and the shell itself all read constantly, so those two lines are buried in a flood of unrelated events. To isolate them, pipe Terminal A through grep -E 'cat|dd' (or filter by comm in the consumer). Stop the tracer with Ctrl-C in Terminal A when done. (If you prefer to background it with sudo ./latency &, remember to sleep 1 before generating work so attach completes, and sudo pkill latency to stop it.)

You just measured every kernel-side vfs_read on your system in real time, with typed argument access and a few hundred nanoseconds of overhead per call. This is what eBPF is for.

What to break, in order

Break 1 — Forget the bpf_map_delete_elem

You saw a preview of this back in Day 2 (Break 4, flagged as a Day 9 teaser); here it’s the actual failure mode of today’s tracer, so it earns a full walk-through. Day 2 used the quick-and-dirty wc -l counter — below we teach the dependable -j | jq length form instead, which is the count you should trust.

Comment it out, then make and relaunch a fresh tracer (sudo pkill latency; make && sudo ./latency &) — the starts map only exists while the loaded program is running, so bpftool map ... name starts finds nothing if the tracer isn’t up. Drive load from another terminal with find /usr -type f -exec cat {} + >/dev/null, then periodically read the live entry count. The reliable, version-independent way is the JSON dump piped to jq length:

sudo bpftool map dump -j name starts | jq length

21

(Plain bpftool map dump name starts prints the entries themselves — on the v7.x bpftool here as key: … value: … lines with a trailing Found N elements footer; only the -j flag emits a JSON array. The format varies across builds, which is exactly why the -j | jq length form above is the dependable counter. Don’t use bpftool map show name starts for this — it reports only the static max_entries 10240 ceiling and key/value sizes, never the live count. And don’t pipe map dump through wc -l: it counts formatting/footer lines, not elements.)

Watch N climb. Slowly at first, then steady. As it plateaus toward max_entries=10240, bpf_map_update_elem silently fails on every new TID. Your tracer’s coverage degrades to “only TIDs that were already in flight when the map filled.”

This is the slow-poison failure mode. Always delete on exit.

Break 2 — Use TGID instead of TID

In both programs, key the map on the TGID — bpf_get_current_pid_tgid() >> 32 — instead of the TID. (Note the entry program computes tid directly and has no id variable, so you’re changing the masked expression in each, not reusing an id.) The collision only shows up when several threads of one process (one shared TGID, many TIDs) read concurrently — separate processes each have a distinct TGID, so they never clash. Drive that with a single multi-threaded reader:

python3 - <<'EOF'
import threading, time
def r():
    while True:
        with open('/etc/passwd') as f:
            f.read()
for _ in range(8):
    threading.Thread(target=r, daemon=True).start()
time.sleep(10)
EOF

All 8 threads share one TGID and all call vfs_read, so a TGID-keyed map collides on every concurrent read: when thread B’s fentry overwrites thread A’s start timestamp with B’s own (later) timestamp, A’s fexit then reads B’s timestamp and computes a bogus delta. Watch your output. Two symptoms dominate. First, understated durations: bpf_ktime_get_ns is monotonic, so the timestamp sitting in the slot is essentially always earlier than the now a later fexit reads — the subtraction stays positive but tiny (now - ts_B instead of the true now - ts_A), so latencies come out implausibly low. Second, the event count drops: whichever thread’s fexit runs first deletes the shared slot, so the other thread’s fexit finds no entry (bpf_map_lookup_elem returns NULL) and silently bails. (A giant ~10^18 “underflowed” duration is not the expected symptom with a monotonic clock — that would require the stored timestamp to be greater than now, which only happens in a narrow race where another CPU’s fentry rewrites the slot value between this fexit’s clock read and its dereference; it’s a rare artifact, not the headline.) Contrast with TID keying: one entry per thread, so deltas stay correct and no events are lost.

Lesson stays: TID for synchronous per-thread tracking.

Break 3 — Drop the null check in fexit

__u64 *ts = bpf_map_lookup_elem(&starts, &tid);
__u64 dur = bpf_ktime_get_ns() - *ts;   // no null check

Verifier rejects with the same R0 type=map_value_or_null error from Day 4. Re-read the log; the rejection should now feel completely unsurprising.

Break 4 — Convert to kretprobe

Change SEC("fexit/vfs_read") to SEC("kretprobe/vfs_read"). Rebuild. Things break:

1. BPF_PROG macro doesn’t know about kretprobe — use BPF_KRETPROBE instead.
2. The argument list is wrong. kretprobe’s ctx is struct pt_regs *, not the typed args + return. You can only get the return value via PT_REGS_RC(ctx).
3. You can’t access f, buf, n, or pos — they were the entry arguments and are gone by now.

This is the operational reason fexit is strictly better. With fexit you have entry args + return value; with kretprobe you have only return value (PT_REGS_RC reads it out of the trampoline’s saved registers) — and, as the mechanism section showed, you also inherit the instance pool, the nmissed burst drops, and the unwinder confusion.

The full equivalent kretprobe version:

SEC("kretprobe/vfs_read")
int BPF_KRETPROBE(on_exit_kretprobe, ssize_t ret)
{
    /* same body, but only `ret` is available */
}

Break 5 — A function that doesn’t always return

Try kernel_clone (the function that implements fork). Most calls return; a few may not (do_exit paths, exotic flags). Run for a long time. Check that the map stays bounded — kernel_clone is well-behaved here. Now try to attach fexit to do_exit itself:

SEC("fentry/do_exit")
int BPF_PROG(on_exit_enter) {
    /* ... store ... */
}

SEC("fexit/do_exit")
int BPF_PROG(on_exit_exit) {
    /* do_exit never returns */
}

On 7.1 this doesn’t even load. The Verifier knows do_exit is marked __noreturn, and it refuses to attach fexit to a function that can never return:

Attaching fexit/fsession/fmod_ret to __noreturn function 'do_exit' is rejected.

The kernel keeps a noreturn_deny BTF set (kernel/bpf/verifier.c) — do_exit, do_group_exit, make_task_dead, __module_put_and_kthread_exit, and friends — and rejects fexit/fmod_ret on any of them at load time. So the leak you might expect here never happens: the Verifier protects you up front.

The leak lesson still stands, though — it just applies to functions the denylist can’t catch. A function that returns on most paths but conditionally doesn’t (a goto into a do_exit call, an unwound error path that calls panic) isn’t __noreturn, so fexit attaches happily — and then silently never fires on the non-returning path, leaking that thread’s map entry. Not every function has a sensible fexit pair. For those, you need a different cleanup strategy (a periodic sweep from userspace, or a tracepoint at task termination such as sched_process_exit).

What to read in the kernel

• kernel/bpf/trampoline.c — open it. Search arch_prepare_bpf_trampoline. The trampoline is generated assembly that saves arguments, calls fentry programs, calls the original function, captures the return value into the ctx array, and calls fexit programs. The ASM is per-arch (x86_64, arm64, etc.) but the structure is consistent.
• tools/lib/bpf/bpf_tracing.h — search BPF_PROG. The macro is ~30 lines of variadic-template-style C macros. Read it once. You’ll see how the u64 *ctx array gets unpacked into typed parameters via ((__u64 *)ctx)[N] casts. After this, the macro stops feeling magic.
• kernel/trace/bpf_trace.c — search bpf_get_func_arg and bpf_get_func_ret (bpf_get_func_ret_proto is at bpf_trace.c:1223). These are helper-based access patterns for the same data, used when BPF_PROG’s positional args don’t fit (e.g., variadic kernel functions).
• arch/x86/kernel/rethook.c — the hand-written arch_rethook_trampoline (:26) whose address a kretprobe swaps onto the stack. Read it once after today and the return-address trick stops being abstract.
• Documentation/bpf/libbpf/program_types.rst — the official doc listing tracing program types (fentry, fexit, fsession). One read, optional.

Bullet Points

• vfs_read is the kernel’s central read chokepoint (fs/read_write.c:554); its ssize_t return is bytes-read on success or a negative errno on failure — that’s the “→ N bytes” you print.
• fexit attaches at function return; receives original arguments and the return value as typed parameters.
• A kretprobe hooks the return by swapping the saved return address on the stack for a trampoline at entry, using a finite instance pool (maxactive); bursts beyond the pool are dropped and counted in nmissed, and the swap confuses stack unwinders.
• Use fexit > kretprobe wherever fexit is available (any function with BTF): same trampoline as fentry, no return-address rewrite, no pool, no missed-on-burst. kretprobe is legacy.
• bpf_ktime_get_ns() returns CLOCK_MONOTONIC nanoseconds (excludes suspend, not wall-clock); cheap, ~10ns to call. Perfect for a latency delta, useless for calendar time.
• Key per-thread tracers by TID, not TGID. Multi-threaded processes will collide on TGID-keyed maps.
• The latency pattern: fentry stores entry timestamp keyed by TID; fexit retrieves it, computes duration, deletes the entry, emits the event.
• Don’t forget bpf_map_delete_elem. Tracers that don’t clean up degrade silently as their map fills.
• Some kernel functions don’t return (e.g., do_exit); fexit-based tracking can’t handle them. The Verifier even rejects fexit on __noreturn functions up front. Use a different mechanism (tracepoints, periodic sweeps).
• BPF_PROG macro unpacks typed arguments from the trampoline’s u64 *ctx array — same macro for fentry and fexit; fexit just has one extra arg (the return value) at the end.

Check question

You attach an fentry+fexit pair to a function that, on some paths, calls itself recursively. The latency map is keyed by TID. Does the measurement still work?

Tomorrow

Day 7: the BPF_PROG macro demystified. The ctx array, PT_REGS_* for kprobes, function argument access patterns, and the helper-vs-kfunc distinction. Less hands-on building, more “now you understand what the macros are doing.”

Day 7 — BPF_PROG demystified, ctx layouts, helpers vs kfuncs

Today’s mission: stop thinking of BPF_PROG/BPF_KPROBE/BPF_KRETPROBE as magic. See what they expand to. Understand why each program type hands you arguments in a different shape — a typed trampoline array, a saved-register snapshot, or a copied event struct — and why the Verifier trusts some of those shapes and not others. Total time: ~120 minutes. Light on coding, heavy on understanding.

Why this day exists

Yesterday you wrote BPF_PROG(on_exit, struct file *f, char *buf, size_t n, loff_t *pos, ssize_t ret) and it just worked. Today we open the hood.

This matters because:

• When you start using non-trivial program types (kprobe, raw tracepoint, sk_msg, sock_ops), the macro and ctx access differ.
• When the macro doesn’t fit your case (variadic functions, exotic signatures), you need to access ctx by hand.
• When the Verifier complains about a register’s type, knowing where that type came from in the proto is the difference between minutes and hours of debugging.

There are three pieces of background you’ve never been shown that every example today leans on:

1. struct pt_regs — the saved-register snapshot a kprobe hands you, and why offset 0 of it is not the first argument.
2. bpf_printk and trace_pipe — the debug-output channel today’s lab reads. Days 1–6 sent data out through ringbuf and maps; today is the first time you’ll print straight to the kernel’s shared trace buffer.
3. What a tracepoint actually is — the TRACE_EVENT macro, the generated trace_event_raw_* struct, the format file, and the raw-args array. Three of today’s five program flavors hang on this.

We’ll teach each as we hit the part that depends on it.

What BPF_PROG actually does

The trampoline calls your BPF program with a single argument — unsigned long long *ctx — pointing at an array of u64 values, one per kernel-function argument (plus, for fexit, the return value at the end).

You don’t want to write (struct file *)ctx[0] everywhere. So BPF_PROG exists.

The macro is variadic-template-style C — it generates an outer function that the BPF program calls (taking only ctx), and an inner __always_inline function that takes your typed parameters. The outer function casts each ctx[N] slot to the right type and forwards into the inner.

You can read the actual macro at tools/lib/bpf/bpf_tracing.h:672 (the #define BPF_PROG(name, args...) line). It’s about 30 lines of variadic macros. Worth opening once. After you do, there’s no more magic — BPF_PROG is a code generator that saves you from writing position-based casts. The line that does the real work is its forwarding through ___bpf_ctx_cast, which expands to ctx[0], ctx[1], ctx[2]… — literal indices into that u64 array.

There are no Dumb Questions

Q: How does the macro know how many arguments my function has?

A: C variadic macros (__VA_ARGS__) plus a counting trick (___bpf_narg) that uses preprocessor recursion to figure out the arity. bpf_tracing.h then dispatches through ___bpf_ctx_cast to the right ___bpf_ctx_cast0..___bpf_ctx_cast12 slot-caster (supporting up to 12 args). The actual implementation is gnarly C macro magic but the concept is simple.

Q: What if my function has more than 12 arguments?

A: You’re using the wrong tool. Most kernel functions have ≤ 6 args (matching the System V calling convention). For variadic kernel functions, use bpf_get_func_arg(ctx, N, &out) — a helper that reads the Nth argument by index and writes it to out (BPF_FUNC_get_func_arg = 183, include/uapi/linux/bpf.h:6088; implemented at kernel/trace/bpf_trace.c:1194). We won’t use it today.

Q: Why does BPF_PROG’s syntax look weird? Why isn’t it just a normal function?

A: Because the BPF program’s real signature must be int (*)(unsigned long long *) — that’s what the trampoline calls. Without the macro, you’d write that signature and unpack ctx[] manually. The macro pretends you wrote a normal C function but generates the unpacking glue.

Two shapes of ctx: the trampoline array vs. the register snapshot

Before the five-flavor chart, you need to see clearly that the word “ctx” means two physically different things depending on program type. Get this and half of today’s surprises evaporate.

Shape A — the trampoline u64 ctx[] (fentry/fexit/tp_btf). This is the array you met on Day 1: the trampoline saves the traced function’s arguments into a small u64[] and hands you a pointer to it. ctx[0] is the first typed argument, ctx[1] the second, and so on. BPF_PROG indexes straight into it (___bpf_ctx_cast → ctx[0], ctx[1]…), which is why BPF_PROG’s forwarding lives at bpf_tracing.h:672.

Shape B — the struct pt_regs * snapshot (kprobe/kretprobe). A kprobe doesn’t get a tidy array. Recall from Day 1: a kprobe fires via the int3 trap — the kernel overwrites the first byte of the traced instruction with a breakpoint, and when the CPU hits it the trap handler runs. At that moment the handler captures every general-purpose register into a struct pt_regs and hands your program a pointer to that. So a kprobe’s ctx is a frozen photograph of the CPU registers, not a per-argument array.

That difference is the whole reason BPF_KPROBE is a separate macro from BPF_PROG. BPF_KPROBE (tools/lib/bpf/bpf_tracing.h:816) declares ctx as struct pt_regs * and, instead of indexing an array, forwards each argument through PT_REGS_PARM1, PT_REGS_PARM2, … — macros that read a named register slot out of the snapshot.

Why offset 0 of pt_regs is the saved r15, not your first argument

Here is the layout that makes every “+112”, “+0x70”, “+0x60” you’ll see today legible. The x86-64 struct pt_regs is at arch/x86/include/asm/ptrace.h:103, and its fields are laid out in this exact order:

/* arch/x86/include/asm/ptrace.h:103 — x86-64 struct pt_regs */
unsigned long r15;   /* offset 0x00  ← FIRST member */
unsigned long r14;   /* 0x08 */
unsigned long r13;   /* 0x10 */
unsigned long r12;   /* 0x18 */
unsigned long bp;    /* 0x20 */
unsigned long bx;    /* 0x28 */
unsigned long r11;   /* 0x30 */
unsigned long r10;   /* 0x38 */
unsigned long r9;    /* 0x40 */
unsigned long r8;    /* 0x48 */
unsigned long ax;    /* 0x50 */
unsigned long cx;    /* 0x58 */
unsigned long dx;    /* 0x60  ← arg3 */
unsigned long si;    /* 0x68  ← arg2 */
unsigned long di;    /* 0x70  ← arg1 */
/* ... orig_ax, ip, cs, flags, sp, ss ... */

Two facts fall straight out of this picture:

• Byte offset 0 is r15 — a callee-saved register that has nothing to do with your function’s arguments. This is exactly why Break 2 below produces garbage: when you (wrongly) put BPF_PROG on a kprobe, the macro reads “ctx[0]” = the 8 bytes at offset 0 = the saved r15, casts it to struct file *, and you’re off into the weeds.
• di sits 15th in the struct → offset 14 × 8 = 112 = 0x70. That’s the number you’ll see the kprobe program load from in both the verifier log (r1 +112) and the disassembly (r1 + 0x70). dx is 13th → offset 96 = 0x60 (arg3).

The layout is deliberately arch-specific. There’s a separate 32-bit struct pt_regs at arch/x86/include/asm/ptrace.h:12 that starts bx, cx, dx, si, di, … — a completely different order. The offsets only make sense once you’ve pinned down the architecture, which is why the build needs -D__TARGET_ARCH_x86 (more on that at the lab).

Mapping System V arguments onto register slots

Recall from Day 1: the System V x86-64 calling convention passes the first integer/pointer argument in rdi, the second in rsi, the third in rdx. The PT_REGS_PARM* macros are just that convention written down. In tools/lib/bpf/bpf_tracing.h:

/* tools/lib/bpf/bpf_tracing.h:87 */
#define __PT_PARM1_REG di
#define __PT_PARM2_REG si
#define __PT_PARM3_REG dx
...
/* tools/lib/bpf/bpf_tracing.h:492 */
#define PT_REGS_PARM1(x) (__PT_REGS_CAST(x)->__PT_PARM1_REG)

So PT_REGS_PARM1(ctx) reads ctx->di, PT_REGS_PARM2(ctx) reads ctx->si, PT_REGS_PARM3(ctx) reads ctx->dx. This is the bridge that makes the disassembly’s “+0x70 = di = PT_REGS_PARM1 = f” and “+0x60 = dx” lines readable: argument 1 of vfs_read is struct file *f, it was passed in rdi, the trap saved rdi at offset 0x70, and the kprobe program loads f from there.

Contrast the two shapes one more time: fentry hands you the trampoline’s u64 ctx[] where ctx[0] genuinely is the typed first argument; kprobe hands you pt_regs where you must dig the argument out of a named register slot. That is why BPF_PROG indexes ctx[] and BPF_KPROBE wraps PT_REGS_PARM*. Same data, two different containers.

What a tracepoint actually is

Three of today’s five flavors (regular tracepoint, raw tracepoint, tp_btf) hook a tracepoint, and you’ve only ever seen the word in passing. Let’s make it concrete, because the structs those flavors declare come from a specific macro and live in a specific file.

A tracepoint is a named, static instrumentation site compiled into the kernel — sched_switch, sys_enter, kfree_skb, and hundreds more. Unlike fentry or kprobe, which hook an arbitrary function you name, a tracepoint is a deliberately placed, stable hook with a declared field schema. Kernel developers put it where it’s useful and promise to keep its fields meaningful across releases.

Every tracepoint is declared with the TRACE_EVENT() macro:

/* include/linux/tracepoint.h:671 */
#define TRACE_EVENT(name, proto, args, struct, assign, print) ...

That one macro generates a lot of code. The piece you care about today is a C struct describing the event’s copied payload:

/* include/trace/trace_events.h:62 — generated for every tracepoint */
struct trace_event_raw_##name {
    ...
};

So sched_switch gets a struct trace_event_raw_sched_switch, sys_enter gets a struct trace_event_raw_sys_enter, and so on. These structs are compiled into the kernel and end up in its BTF, which is why they appear in your vmlinux.h and why the flavor-3 program below can just declare struct trace_event_raw_sys_enter *ctx without defining it.

The crucial property: the kernel copies the chosen fields into that struct. When the tracepoint fires, it fills a trace_event_raw_* instance with the selected values and hands your program a pointer to those copied bytes — not to live kernel objects. That single fact explains Break 3 at the bottom: you can read ctx->prev_comm directly (it’s an embedded char array, copied in for you), but you cannot reach the parent task, because no pointer to it was ever copied.

The format file: a tracepoint’s public schema

Because a tracepoint has a declared schema, the kernel exposes it in human-readable form under tracefs:

sudo cat /sys/kernel/tracing/events/raw_syscalls/sys_enter/format

For sys_enter you’ll see fields including long id and unsigned long args[6]. (Note the path is raw_syscalls/, not syscalls/: the syscalls/ directory holds only the per-syscall nodes like sys_enter_read, whose format lists typed fields fd/buf/count. The generic id + args[6] schema that the flavor-3 program reads is the raw_syscalls:sys_enter tracepoint.) That’s not a coincidence — it comes straight from the declaration in include/trace/events/syscalls.h:18:

/* include/trace/events/syscalls.h:18 */
TRACE_EVENT_SYSCALL(sys_enter,
    TP_PROTO(struct pt_regs *regs, long id),
    ...
    TP_STRUCT__entry(
        __field( long,          id        )
        __array( unsigned long, args, 6   )
    ),
    ...

The __array(unsigned long, args, 6) is why the flavor-3 program reads ctx->args[0] to get the syscall’s first argument (the fd for read()): the kernel copied all six syscall arguments into that args[6] array for you. The format file is the contract; ctx->args[0] is you reading it.

Raw tracepoints: skip the copy, get positional u64s

A raw tracepoint (flavor 4) is the same instrumentation site without the copy step. Instead of a typed trace_event_raw_* struct, the kernel hands you:

/* include/uapi/linux/bpf.h:7286 */
struct bpf_raw_tracepoint_args {
    __u64 args[0];
};

That’s it — a bare array of __u64, one slot per TP_PROTO argument, unmodified. For sched_switch the TP_PROTO is (bool preempt, struct task_struct *prev, struct task_struct *next, …), so ctx->args[0] is the preempt bool, ctx->args[1] is a task_struct * you cast by hand, ctx->args[2] the next one. No fields are copied, no schema is applied; you unpack by position and you’re responsible for the casts.

tp_btf (flavor 5, detailed tomorrow) is the same raw argument array, but BTF-typed: the Verifier knows args[1] is a struct task_struct * and marks it PTR_TO_BTF_ID (recall from Day 1: a trusted, typed kernel pointer the Verifier will let you dereference), so you can dereference it directly. Raw args, but trusted.

Argument access by program type — five flavors

Different program types access arguments five different ways. This is the chart you’ll come back to whenever you’re confused about a “wrong type” verifier rejection — and now you have the background to read every row of it.

1. fentry/fexit — typed args from BTF

SEC("fentry/vfs_read")
int BPF_PROG(p, struct file *f, char *buf, size_t n, loff_t *pos) {
    loff_t cur = f->f_pos;   // direct deref allowed
}

The trampoline gives you a u64 *ctx. BPF_PROG casts each slot to your declared type. Because BTF has the original function’s signature, the cast is meaningful — f really is a struct file * pointing at a live kernel object. The Verifier marks f as PTR_TO_BTF_ID (typed kernel pointer), allowing direct dereference of fields.

Pros: typed, fast, direct deref. Cons: requires BTF (essentially always present in modern kernels).

2. kprobe/kretprobe — pt_regs from a trap

SEC("kprobe/vfs_read")
int BPF_KPROBE(p, struct file *f, char *buf, size_t n, loff_t *pos) {
    /* f is pt_regs->di on x86_64 */
}

BPF_KPROBE is a different macro that knows ctx is struct pt_regs * (a saved register snapshot from the entry trap). It expands to ((struct file *)PT_REGS_PARM1(ctx)) for the first argument, etc. The values are correct — they’re whatever was in RDI, RSI, etc. when the int3 fired — but the pointers are not marked as PTR_TO_BTF_ID because the Verifier can’t trust them in a kprobe context (the function’s BTF signature isn’t bound to the regs).

You access fields via bpf_probe_read_kernel:

loff_t pos;
bpf_probe_read_kernel(&pos, sizeof(pos), &f->f_pos);

This is why kprobes feel clunkier than fentry. Same data, more friction.

3. tracepoint/... — copied event context

SEC("tracepoint/syscalls/sys_enter_read")
int p(struct trace_event_raw_sys_enter *ctx) {
    int fd = (int)ctx->args[0];
}

The regular tracepoint program sees a stable kernel-defined event struct. The kernel copies the relevant fields into that per-tracepoint struct, and ctx points at the copied bytes. You don’t get live pointers. Useful when you want the stable event format in /sys/kernel/tracing/events/.../format.

4. raw_tracepoint/... — raw tracepoint argument array

SEC("raw_tracepoint/sched_switch")
int p(struct bpf_raw_tracepoint_args *ctx) {
    bool preempt = (bool)ctx->args[0];
    struct task_struct *prev = (void *)ctx->args[1];
    struct task_struct *next = (void *)ctx->args[2];
}

Raw tracepoints skip the copied event struct and expose the tracepoint arguments as raw u64 slots. You unpack by position. The verifier does not give you the same typed-argument ergonomics as tp_btf, so new code usually prefers tp_btf when BTF is available.

5. tp_btf — typed kernel pointers from a tracepoint

SEC("tp_btf/sched_switch")
int BPF_PROG(p, bool preempt, struct task_struct *prev, struct task_struct *next) {
    bpf_printk("%s -> %s", prev->comm, next->comm);  // direct deref
}

This is the modern typed interface to the same tracepoint event. The kernel hands you typed pointers (BTF-tagged) rather than copied bytes or raw positional slots. Direct deref works. Use tp_btf instead of raw tracepoint for new code when BTF is available — we’ll see this in detail tomorrow.

Sharpen your pencil

A function int foo(struct file *f, void *data, size_t len) is in the kernel. You attach an fentry. You want to read f->f_inode->i_ino. Three ways to do it:

1. __u64 ino = f->f_inode->i_ino;
2. __u64 ino = BPF_CORE_READ(f, f_inode, i_ino); (CO-RE, from Day 3 — walks the pointer chain with relocations, each hop via bpf_probe_read_kernel)
3. __u64 ino; bpf_probe_read_kernel(&ino, sizeof(ino), &f->f_inode->i_ino);

Which works? Which is preferred?

.
.
.

Answers: All three load. (1) and (2) work cleanly because f is PTR_TO_BTF_ID and the Verifier proves the chain. (1) is fastest: it’s a direct verified load — the Verifier permits the deref as a plain BPF load (LDX) and records it in the kernel exception table for fault handling, so there’s no helper call and nothing lowers to bpf_probe_read_kernel. (2) is safe-by-default (returns 0 on bad pointer) and is what (3) is the explicit form of — BPF_CORE_READ lowers to bpf_probe_read_kernel. Prefer (1) for trusted chains, (2) for chains where any hop could be NULL. (3) is rarely written by hand anymore.

Helpers vs kfuncs — the two extension mechanisms

Yesterday you used bpf_get_current_pid_tgid, bpf_ktime_get_ns, bpf_map_lookup_elem. Those are helpers — a frozen UAPI list of functions BPF programs can call. You declared them via bpf_helpers.h, the linker resolves them to enum values (BPF_FUNC_get_current_pid_tgid = 14, etc.), and at load time the verifier/JIT resolves the call against a per-program-type proto table.

But there’s a second, newer mechanism: kfuncs. Functions declared in any kernel C file, registered via BTF_KFUNCS_START, that BPF programs can call by name (matched against kernel BTF at load time).

Why both exist:

• Helpers are frozen UAPI. Adding a helper is a forever commitment; removing or changing one breaks user programs. Process to add a helper is heavyweight.
• kfuncs are not UAPI. They can evolve, be added, removed, renamed. The maintainers can change a kfunc’s signature in the next release; user programs that use the old name fail to load on the new kernel. Lower commitment.

Around 2022, the kernel community decided to stop adding new helpers and add functionality as kfuncs instead. This is why your “use this BPF feature” docs from 2024+ talk about kfuncs more than helpers.

You’ll meet kfuncs properly on Day 20. For now, just know: when you see extern struct task_struct *bpf_task_acquire(struct task_struct *) __ksym;, that’s a kfunc. The __ksym attribute tells libbpf “match this name to a kfunc in kernel BTF at load time.”

There are no Dumb Questions

Q: How do I know if bpf_xxx is a helper or a kfunc?

A: Helpers are listed in include/uapi/linux/bpf.h (the enum bpf_func_id table starting with BPF_FUNC_unspec). Kfuncs aren’t UAPI; their list lives in source files via BTF_KFUNCS_START blocks. bpftool feature probe shows what’s available on your kernel. Or just check: does it appear in <bpf/bpf_helpers.h>? If yes, helper. If no but you see __ksym, kfunc.

Q: Are helpers going away?

A: No. Existing helpers are stable forever (UAPI). New functionality comes as kfuncs. So the helper list will keep working but won’t grow.

bpf_printk and the trace_pipe debug channel

Today’s lab is the first in this book to use bpf_printk, so let’s be precise about what it is — because it does not go through the ringbuf or maps you’ve used on Days 1–6.

bpf_printk is not a function — it’s a libbpf macro. It builds a format string and dispatches to a helper:

/* tools/lib/bpf/bpf_helpers.h:341 */
#define bpf_printk(fmt, args...) ___bpf_pick_printk(args)(fmt, ##args)

The dispatcher ___bpf_pick_printk (bpf_helpers.h:334) picks the lowering by argument count. For ≤3 args it selects the __bpf_printk path (bpf_helpers.h:293), whose underlying helper is bpf_trace_printk — BPF_FUNC_trace_printk = 6 in the UAPI enum:

/* include/uapi/linux/bpf.h:5911 */
FN(trace_printk, 6, ##ctx) \

For 4 or more args it selects a different path — __bpf_vprintk (bpf_helpers.h:304) — which calls a separate helper, bpf_trace_vprintk (BPF_FUNC_trace_vprintk = 177, include/uapi/linux/bpf.h:6082). So the underlying helper is not the same either way: FN 6 for the short form, FN 177 for the wide form.

Where does the output go? Not into your own ringbuf or map — into the kernel’s shared ftrace ring buffer, a single system-wide buffer that every bpf_printk user writes to. You read it from userspace through a tracefs file:

sudo cat /sys/kernel/tracing/trace_pipe

trace_pipe is a blocking, drain-on-read pipe: cat it and it blocks until a line appears, and reading a line consumes it. Because the buffer is global, you’ll see lines from every BPF program on the system that calls bpf_printk, interleaved. That’s exactly why today’s “observe all three” experiment can simply cat trace_pipe and watch three different programs’ output land in one stream.

How many format arguments? Up to 12. Each path has its own ceiling. The bpf_trace_printk helper (the ≤3 path) builds a fixed three-slot array:

/* kernel/trace/bpf_trace.c:359 */
#define MAX_TRACE_PRINTK_VARARGS 3
/* kernel/trace/bpf_trace.c:362 */
BPF_CALL_5(bpf_trace_printk, char *, fmt, u32, fmt_size, u64, arg1,
           u64, arg2, u64, arg3)
{
    u64 args[MAX_TRACE_PRINTK_VARARGS] = { arg1, arg2, arg3 };
    ...
}

But that MAX_TRACE_PRINTK_VARARGS = 3 is the limit of the legacy bpf_trace_printk helper only — not of the bpf_printk macro. Pass a fourth value and the dispatcher transparently switches you to bpf_trace_vprintk (FN 177), which packs the args into a u64[] and allows up to MAX_BPRINTF_VARARGS = 12 (include/linux/bpf.h:3875; the bound is enforced in bpf_trace.c:424 as data_len > MAX_BPRINTF_VARARGS * 8). So bpf_printk("%d %d %d %d", a, b, c, d) compiles and loads fine — only a 13th argument is rejected. Today’s lab passes only two values per call because that’s all each line needs, not because of a ceiling.

Framing: bpf_printk is a debugging and teaching tool. It’s global, low-throughput, and serializes all writers — fine for “did my program fire and what did it see?”, wrong for production. Real tracers stream through the ringbuf and maps you already learned. That’s exactly why the book waited until a “poke at the macros” day to introduce it.

Today’s lab — small, focused

We’re not building a new tracer today. We’re poking at what the macros emit.

inspect.bpf.c

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

char LICENSE[] SEC("license") = "GPL";

/* Same logic, three different program types */

SEC("fentry/vfs_read")
int BPF_PROG(via_fentry, struct file *f, char *buf, size_t n, loff_t *pos)
{
    bpf_printk("fentry: f=%p n=%zu", f, n);
    return 0;
}

SEC("kprobe/vfs_read")
int BPF_KPROBE(via_kprobe, struct file *f, char *buf, size_t n, loff_t *pos)
{
    bpf_printk("kprobe: f=%p n=%zu", f, n);
    return 0;
}

SEC("tracepoint/syscalls/sys_enter_read")
int via_tp(struct trace_event_raw_sys_enter *ctx)
{
    int fd = (int)ctx->args[0];
    bpf_printk("tp: fd=%d", fd);
    return 0;
}

Notice each bpf_printk passes at most two values (f and n, or just fd) — not because of any ceiling (the macro supports up to 12), but simply because that’s all each line needs. And via_tp reaches the fd through ctx->args[0], exactly the args[6] slot the sys_enter format file declares.

First generate the type header if you don’t already have it from Day 1, then build the object. The -D__TARGET_ARCH_x86 is required: BPF_KPROBE expands to the arch-specific PT_REGS_PARM* macros (which select di/si/dx for x86-64), and without it the compile fails — there’d be no way to know which register order to use.

# once per kernel, if you don't already have vmlinux.h from Day 1:
sudo bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

clang -g -O2 -D__TARGET_ARCH_x86 -target bpf -c inspect.bpf.c -o inspect.bpf.o

The verifier only runs when a program is loaded, and the detailed per-instruction register state only prints at log level 2. Copy Day 6’s loader to inspect.c, point it at inspect.bpf.o, and raise the log level in the open options — the LIBBPF_OPTS line below is wired into the object-open call, which is what makes it take effect:

LIBBPF_OPTS(bpf_object_open_opts, opts, .kernel_log_level = 2);
struct bpf_object *obj = bpf_object__open_file("inspect.bpf.o", &opts);
bpf_object__load(obj);   /* verifier runs here; its log prints to stderr */

make
sudo ./inspect 2>&1 | less

If you have veristat (it ships in the kernel tree under tools/testing/selftests/bpf/, so you may need to build it), you can skip the loader entirely and dump the same per-instruction log: sudo veristat -v -l 2 inspect.bpf.o 2>&1 | less — -v is what emits the log and -l 2 is the log level that prints the R1_w=... register-state lines.

Inspect the verifier log for each program

For via_fentry, you’ll see early register state lines like:

0: R1=ctx() R10=fp0
1: (79) r1 = *(u64 *)(r1 +0)
2: R1_w=trusted_ptr_file(off=0,...)

R1_w=trusted_ptr_file — the Verifier sees R1 as a trusted (BTF-validated) pointer to struct file. That’s why f->f_pos works. Note the load is from +0 — ctx[0], the first slot of the trampoline array.

For via_kprobe:

0: R1=ctx()
1: (79) r1 = *(u64 *)(r1 +112)   /* PT_REGS_PARM1: di on x86_64 */
2: R1_w=scalar()

R1_w=scalar() — kprobe ctx access produces an untrusted scalar, not a typed pointer. That’s why direct deref f->f_pos is forbidden. And there’s the +112 from the pt_regs layout: offset 112 is di, the saved rdi, which held f.

For via_tp:

0: R1=ctx()
1: (61) r2 = *(u32 *)(r1 +16)
2: R2_w=scalar(...)

The tp ctx is a typed struct (provided by vmlinux.h); reads are scalar values from the copied event buffer — exactly the “kernel copied the fields in” model from the tracepoint section.

Inspect what BPF_PROG actually generated

Disassemble the same inspect.bpf.o you built above:

llvm-objdump -dr inspect.bpf.o | head -40

0000000000000000 <via_fentry>:
       0:	r4 = *(u64 *)(r1 + 0x10)
       1:	r3 = *(u64 *)(r1 + 0x0)
       2:	r1 = 0xa ll
       4:	w2 = 0x13
       5:	call 0x6
       6:	w0 = 0x0
       7:	exit

Because the inner ____via_fentry is static __always_inline and we compile at -O2, it is inlined into the single emitted function via_fentry — there is no separate inner function and no call to it. The only call is the bpf_trace_printk helper (call 0x6 — and yes, that 0x6 is BPF_FUNC_trace_printk = 6 from the UAPI enum). You’ll see via_fentry load f from *(u64 *)(r1 + 0x0) (ctx[0]) into r3 and n from *(u64 *)(r1 + 0x10) (ctx[2]) into r4 — the buf/ctx[1] slot at +0x8 is dead and elided. Run the same for via_kprobe and you’ll see it load from pt_regs offsets instead: *(u64 *)(r1 + 0x70) is di = PT_REGS_PARM1 = f (and +0x60 is dx). The two-function shape the macro creates collapses into one function in the object — the macro is purely a source-level code generator.

Observe all three reach the same read

The three programs each print via bpf_printk. Attach them and watch one vfs_read flow through all three paths. Auto-attach the object, then read the trace buffer while you trigger a read — trace_pipe is the shared ftrace buffer from the section above, so all three programs’ lines surface in the one stream:

sudo bpftool prog loadall inspect.bpf.o /sys/fs/bpf/insp autoattach
sudo cat /sys/kernel/tracing/trace_pipe &
dd if=/etc/hostname of=/dev/null bs=64 count=1

You’ll see three lines for the same call (addresses differ on your box; the point is the two f= values match):

             dd-20461   [001] ...21  9183.441: bpf_trace_printk: tp: fd=3
             dd-20461   [001] ...21  9183.441: bpf_trace_printk: fentry: f=ffff8e0c1a3b4500 n=64
             dd-20461   [001] ...21  9183.441: bpf_trace_printk: kprobe: f=ffff8e0c1a3b4500 n=64

fentry and kprobe both hook vfs_read, so the f= pointer for the same call is identical — even though fentry got it from BTF (ctx[0]) and kprobe from pt_regs->di. Same data, two paths. The tracepoint sits one layer up (the read() syscall entry, which later calls vfs_read), which is why it prints fd rather than f. Clean up:

sudo kill %1 2>/dev/null
sudo rm -rf /sys/fs/bpf/insp

What to break, in order

Break 1 — Try to direct-deref in a kprobe

SEC("kprobe/vfs_read")
int BPF_KPROBE(p, struct file *f) {
    loff_t pos = f->f_pos;          /* direct deref */
    bpf_printk("pos=%lld", pos);
    return 0;
}

Drop this into inspect.bpf.c, rebuild, and try to load it — bpftool prog loadall runs the verifier and prints its log to stderr on failure:

clang -g -O2 -D__TARGET_ARCH_x86 -target bpf -c inspect.bpf.c -o inspect.bpf.o
sudo bpftool prog loadall inspect.bpf.o /sys/fs/bpf/x

The verifier rejects it (no pin is created on failure, so there’s nothing to clean up):

R1 invalid mem access 'scalar'

Because in kprobe context, R1 (the cast from pt_regs->di) is a scalar, not a PTR_TO_BTF_ID. Fix:

loff_t pos;
bpf_probe_read_kernel(&pos, sizeof(pos), &f->f_pos);

Break 2 — Use BPF_PROG with kprobe

SEC("kprobe/vfs_read")
int BPF_PROG(p, struct file *f) { ... }

This one loads (verify with sudo bpftool prog loadall inspect.bpf.o /sys/fs/bpf/x — no verifier error — then sudo rm -rf /sys/fs/bpf/x), but the argument access is wrong. BPF_PROG reads ctx[0] — but kprobe’s ctx is pt_regs *, not the trampoline ctx array. As the pt_regs layout showed, offset 0 of struct pt_regs on x86-64 is the saved r15, a callee-saved register that has nothing to do with the first function argument. So BPF_PROG casts the leftover r15 to struct file *: pure garbage. Use BPF_KPROBE for kprobes — it reads PT_REGS_PARM1 = di = offset 112, where f actually lives.

Break 3 — Direct deref in a regular tracepoint

SEC("tracepoint/sched/sched_switch")
int p(struct trace_event_raw_sched_switch *ctx) {
    bpf_printk("%s", ctx->prev_comm);   /* direct read of copied bytes */
}

Works — sudo bpftool prog loadall inspect.bpf.o /sys/fs/bpf/x reports the program loaded with no verifier error (clean up with sudo rm -rf /sys/fs/bpf/x). The ctx->prev_comm is a copied char array embedded in the event struct (the kernel copied it in via TRACE_EVENT), not a pointer. No deref needed.

But:

SEC("tracepoint/sched/sched_switch")
int p(struct trace_event_raw_sched_switch *ctx) {
    /* try to read the parent task's comm */
    /* you can't! ctx doesn't have a task pointer, just the values */
}

Regular tracepoint/... programs give you what the kernel chose to copy. The format file for sched_switch lists prev_comm, prev_pid, next_comm, … — copied scalars and char arrays, no task_struct pointer. Since no pointer was copied, there’s nothing to follow to the parent. For more, switch to tp_btf (Day 8) where you get live typed pointers; use raw_tracepoint/... only when you explicitly want raw positional tracepoint args.

What to read in the kernel

• tools/lib/bpf/bpf_tracing.h — the canonical reference for BPF_PROG (:672), BPF_KPROBE (:816), BPF_KRETPROBE, BPF_KSYSCALL, and the PT_REGS_* family (PT_REGS_PARM1 at :492, the __PT_PARM1_REG di definitions from :87). Read all of it. It’s ~930 lines but most is per-arch macros for register names.
• arch/x86/include/asm/ptrace.h — the 64-bit struct pt_regs (:103) whose field order (r15 first, di at offset 112) is what makes the +0x70/+0x60 loads legible, plus the separate 32-bit struct at :12 proving the layout is arch-specific.
• include/uapi/linux/bpf.h — search enum bpf_func_id. The complete helper list. ~200 entries. Skim — recognize the categories (skb manipulation, map ops, networking, tracing, time, random, etc.). Note trace_printk = 6 (:5911) and get_func_arg = 183 (:6088), and the struct bpf_raw_tracepoint_args at :7286.
• include/linux/tracepoint.h and include/trace/trace_events.h — TRACE_EVENT() (tracepoint.h:671) and the struct trace_event_raw_##name { … } generation site (trace_events.h:62) that produces the structs your tracepoint programs declare.
• include/trace/events/syscalls.h — the sys_enter declaration (:18) with __array(unsigned long, args, 6), proving why ctx->args[0..5] works.
• kernel/bpf/helpers.c — search bpf_get_current_pid_tgid_proto. Each helper has a bpf_func_proto struct that declares its argument types, return type, and which program types can call it. This is what the Verifier consults when type-checking a helper call.
• kernel/trace/bpf_trace.c — MAX_TRACE_PRINTK_VARARGS (:359) and bpf_trace_printk_proto (:386), the helper bpf_printk lowers to; plus get_func_arg (:1194).
• net/core/filter.c — search bpf_helper_changes_pkt_data or xdp_func_proto. Per-program-type proto tables that gate which helpers a given program can call.
• Documentation/bpf/btf.rst and Documentation/bpf/kfuncs.rst — official docs, one read each.

Bullet Points

• BPF_PROG is a code generator that unpacks a u64 *ctx array into typed parameters via casts (ctx[0], ctx[1]…).
• BPF_KPROBE/BPF_KRETPROBE are a different macro: ctx is struct pt_regs *, a saved-register snapshot from the int3 trap, unpacked via PT_REGS_PARM* (= named register slots).
• On x86-64 struct pt_regs starts with r15 at offset 0, with dx/si/di at 0x60/0x68/0x70 — so PT_REGS_PARM1 reads di at offset 112, and a stray BPF_PROG on a kprobe reads r15 = garbage.
• fentry/fexit give you BTF-typed pointers — direct deref OK.
• kprobe gives you scalars from registers — must use bpf_probe_read_kernel.
• A tracepoint is a static TRACE_EVENT-declared hook; it generates a struct trace_event_raw_<name> and a /sys/.../format schema. The kernel copies fields in, so you read values but get no live kernel pointers.
• Regular tracepoint gives you copied event bytes (ctx->args[N], embedded char arrays).
• Raw tracepoint gives you positional raw args in struct bpf_raw_tracepoint_args { __u64 args[]; } — the unmodified TP_PROTO arguments, cast by hand.
• tp_btf gives you those same raw args but BTF-typed (PTR_TO_BTF_ID) — direct deref, tracepoint stability — modern preference.
• bpf_printk is a macro that dispatches by arg count: ≤3 args lower to the bpf_trace_printk helper (FN 6, max 3), 4–12 args to a separate bpf_trace_vprintk helper (FN 177, max MAX_BPRINTF_VARARGS = 12). Output lands in the shared ftrace buffer read via /sys/kernel/tracing/trace_pipe. Debug-only, global, serialized — production uses ringbuf/maps.
• Helpers are a frozen UAPI list in include/uapi/linux/bpf.h. Kfuncs are non-UAPI in-tree functions exposed via BTF; they can evolve. New BPF features are added as kfuncs, not helpers. Helper allowance is per program type.

Check question

Why can your fentry program directly deref f->f_pos, but your kprobe program cannot?

Tomorrow

Day 8: tp_btf vs raw tracepoint in depth. We trace sched_switch, get typed task_struct * pointers for both prev and next, and observe scheduling latency without writing any kprobe glue.

Day 8 — Tracepoints, raw vs tp_btf, and how to find them

Today’s mission: measure scheduler latency by hooking sched_switch. But first understand what that even means — what a context switch is, why a task “waits,” and exactly what number the lab computes. Then learn the anatomy of the TRACE_EVENT macro that makes a tracepoint exist, how the kernel dispatches a firing tracepoint to its listeners, and why tp_btf is the cheap, powerful way in. Compare raw tracepoint and tp_btf attach modes. Learn how to discover every tracepoint on your kernel and read its format. Total time: ~110 minutes.

Why tracepoints when fentry exists?

Fentry attaches to function entries. That’s perfect when there’s a function whose name describes the event you care about (vfs_read, filename_unlinkat). But many interesting events don’t map to a single function — or they’re emitted from inside a function that does many things.

Tracepoints are explicit instrumentation points added by kernel developers via TRACE_EVENT(...) macros. They name the event, specify the data fields, and live forever as part of the kernel’s API contract.

When you write a tracer for “every context switch,” you don’t want to attach to __schedule and figure out which paths inside it correspond to actual switches. You want sched_switch — the tracepoint that fires once per actual switch with prev and next already determined.

Tracepoints exist for: scheduler events, block I/O, networking, memory allocation, filesystem operations, syscalls, and many more.

First: what is a context switch, and what does “scheduling latency” mean?

The whole lab today computes a number it calls wait_ns and prints lines like firefox [4001] waited 152 µs. Before we hook anything, you need to know what that number is — otherwise it’s just a digit with a unit. So let’s build the scheduler model from scratch. You met task_struct on Day 3 (the kernel’s per-process descriptor); nothing earlier taught you how the scheduler runs those tasks. Here it is.

One CPU runs exactly one task at a time. That’s the whole problem the scheduler solves. You have hundreds of runnable threads and (say) 8 logical CPUs, so at any instant at most 8 threads are physically executing. Everyone else is waiting for a turn.

A run queue is the waiting room. Each CPU has its own run queue (struct rq) — the set of tasks that are runnable (ready to execute, not blocked on I/O or a lock) but aren’t the one currently on the CPU. The scheduler’s job is to repeatedly pick the next task to run from that queue.

A context switch is the hand-off. When the scheduler decides task B should run instead of task A, it performs a context switch: save A’s CPU register state and stack pointer, load B’s, and resume B where it last left off. To the running code, nothing happened — A simply “froze” mid-instruction-stream and will thaw later exactly where it stopped. The function that performs the actual register/stack swap is context_switch():

/* kernel/sched/core.c:5329 */
context_switch(struct rq *rq, struct task_struct *prev,
               struct task_struct *next, struct rq_flags *rf)

Note the two arguments already named prev and next — the outgoing and incoming task_struct. By the time the switch happens, the scheduler has already chosen who runs next.

sched_switch fires once per switch, after the choice is made. A single function drives every voluntary switch (a task blocks on I/O) and every preemptive switch (a task’s time slice expires, or a higher-priority task woke up): __schedule().

/* kernel/sched/core.c:7017 */
static void __sched notrace __schedule(int sched_mode)

Inside it, just before handing off to context_switch, the kernel emits the tracepoint with the already-decided prev and next:

/* kernel/sched/core.c:7186 */
trace_sched_switch(preempt, prev, next, prev_state);

That trace_sched_switch(...) call is the tracepoint firing. Those four arguments — preempt, prev, next, prev_state — are exactly the arguments your tp_btf program will receive. Hold that thought; we’ll cash it in below.

So what is “scheduling latency”? Informally: the gap between the moment a task became runnable (it was put on the run queue) and the moment the CPU actually picked it and started running it. During that gap the task sits in the run queue while other tasks use the CPU. A long gap means the task was ready but couldn’t get a turn — that’s scheduling delay, and it’s what makes an interactive app feel laggy under load.

Be precise about what THIS lab measures. We are not going to install two probes. We record one timestamp per task on every switch-out (when a task becomes prev), and when that same tid later shows up as next (switched back in), we report now - last_run. That is the wall-clock time the task spent off-CPU between two of its own on-CPU stints — the time it was not running. That’s an excellent, cheap proxy for scheduling delay, but it is not the textbook wakeup-to-run latency. The textbook version would also need sched_wakeup to mark the exact instant the task became runnable (it might have been blocked on disk for much of that off-CPU gap, which isn’t the scheduler’s fault). State this to yourself every time you read the output: wait_ns = elapsed off-CPU time between two runs of the same task. Trust the number for what it is.

There’s one more payoff from knowing prev and next are full struct task_struct * (Day 3’s descriptor): tp_btf hands you those live typed pointers, so you can read prev->mm, prev->cgroup, prev->real_parent, and anything else in the task. A regular tracepoint, as we’ll see, only hands you a handful of copied fields. That difference is the spine of today’s chapter.

The three BPF-facing ways to attach to a tracepoint

For tracepoint events, BPF has three related section families:

Regular tracepoint (SEC("tracepoint/group/event")) gives you a struct of copied bytes — whatever fields the tracepoint definition chose to copy out. Stable across kernel versions because the tracepoint format is part of the kernel’s API.

Raw tracepoint (SEC("raw_tracepoint/event")) gives you raw positional arguments in struct bpf_raw_tracepoint_args. It avoids the copied event struct, but you unpack by index and lose typed-argument ergonomics.

tp_btf (SEC("tp_btf/event")) gives you the original typed kernel pointers the tracepoint received as arguments. You can dereference fields directly when the pointer is valid for the callback. Same hook, more power.

Same event, three BPF-facing interfaces. The kernel dispatches whichever listener types are attached:

When a tracepoint fires, regular tracepoint/... listeners get the copied event struct, raw_tracepoint/... listeners get raw argument slots, and tp_btf/... listeners get typed arguments. Cost is bounded: the event-struct copy only happens for regular tracepoint listeners.

That “cost is bounded” claim is load-bearing for the whole “use tp_btf, it’s cheaper” thesis — so let’s ground it, because right now it’s just an assertion.

How a firing tracepoint actually reaches its listeners

A tracepoint is not an if in the hot path. It’s a registered list of probe callbacks sitting behind a near-zero-cost static branch. When the tracepoint is off (no listeners), a static-key patch turns it into a fall-through — the CPU pays essentially nothing. When at least one listener attaches, the branch is patched live to call into the dispatch machinery.

That dispatch machinery is a generated per-event function, __traceiter_<event>. Every TRACE_EVENT expands through a chain of macros (TRACE_EVENT → DECLARE_TRACE_EVENT → __DECLARE_TRACE → __DECLARE_TRACE_COMMON), and it is __DECLARE_TRACE_COMMON that generates it:

/* include/linux/tracepoint.h:267 */
extern int __traceiter_##name(data_proto);

and the firing site reaches it through a static call (on a CONFIG_HAVE_STATIC_CALL kernel — the common case today):

/* include/linux/tracepoint.h:227 */
static_call(tp_func_##name)(__data, args);

(On kernels without CONFIG_HAVE_STATIC_CALL, the #else branch at line 231 reaches the same iterator with a plain indirect call: __traceiter_##name(NULL, args).)

When sched_switch fires, __traceiter_sched_switch walks the list of attached probe callbacks and calls each one with the raw TP_PROTO arguments — (preempt, prev, next, prev_state). Crucially: nothing is copied at this layer. The iterator just passes the pointers the kernel already had in registers to each registered function.

So where does the “copied event struct” come from? It’s itself just one probe callback on that list. The probe that performs the TP_fast_assign memcpy (the flat prev_comm@8, prev_pid@24, … struct) is registered only when a regular tracepoint/... (or ftrace/perf) listener attaches. If the only listener is a tp_btf or raw program, that copy probe is never on the list, so the memcpy never runs. That is precisely why the cost is “bounded”: you pay the copy if and only if someone asked for the copied form.

Where BPF plugs in. BPF raw and tp_btf programs attach through the same raw-tracepoint path. libbpf opens a BPF_TRACE_RAW_TP link:

/* kernel/bpf/syscall.c:4388 */
case BPF_TRACE_RAW_TP:

The attach resolves the event name to its raw event map:

/* kernel/trace/bpf_trace.c:2051 */
struct bpf_raw_event_map *bpf_get_raw_tracepoint(const char *name)

and when the tracepoint fires, the BPF dispatch probe on the iterator’s list is __bpf_trace_run, which invokes your program with the raw argument slots:

/* kernel/trace/bpf_trace.c:2073 */
void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args)

tp_btf differs from raw in exactly one way: the verifier tags those raw argument slots with their BTF types, so instead of opaque u64s you get a typed struct task_struct *prev. Same path, same __bpf_trace_run, same zero-copy delivery — just type information layered on top. Because tp_btf rides this raw path, it pays neither the per-event struct copy nor a tracefs format lookup: it just receives the pointers the kernel already had in registers. This is the concrete grounding under the chapter’s “tp_btf goes through the same raw-tracepoint path” sentence.

There are no Dumb Questions

Q: If tp_btf is strictly better, why does raw tracepoint still exist?

A: Three reasons. First, history — raw tracepoints predate BTF. Second, raw tracepoints work on kernels without BTF (rare today, but possible). Third, raw tracepoint structs are explicitly stable — prev_comm will always be a 16-byte char array; the kernel maintainers commit to that. tp_btf gives you live task_struct *, but task_struct itself isn’t stable across versions (you’d need CO-RE for any field access).

Q: How do tracepoints differ from kprobe?

A: Tracepoints are explicitly placed by kernel developers as part of the kernel’s API. Their format is stable. Kprobes are dynamic — you can attach to any function, but the function isn’t part of any contract; it can be renamed, removed, or have its signature change in the next release.

Q: What about raw_tracepoint (with the underscore)?

A: That’s the raw positional-argument mode: ctx->args[0], ctx->args[1], and so on. It is distinct from SEC("tracepoint/..."), which receives a copied event struct. tp_btf supersedes raw tracepoints for most new code on BTF-enabled kernels because it gives typed arguments instead of positional slots.

How to find tracepoints on your kernel

# All tracepoints, grouped by subsystem:
sudo ls /sys/kernel/tracing/events/

# All sched events:
sudo ls /sys/kernel/tracing/events/sched/

# Format of a specific tracepoint (the struct you'd get from raw):
sudo cat /sys/kernel/tracing/events/sched/sched_switch/format

These all need sudo: the tracefs root /sys/kernel/tracing is mounted mode 0700 root:root, so unprivileged users can’t even traverse into events/. (The event subdirectories below it are 0755, and each format file is 0440 root:root, but the root directory gates access to everything underneath.) You’ll see the struct layout, e.g.:

name: sched_switch
ID: 310
format:
	field:char prev_comm[16];	offset:8;	size:16;	signed:0;
	field:pid_t prev_pid;	offset:24;	size:4;	signed:1;
	field:int prev_prio;	offset:28;	size:4;	signed:1;
	field:long prev_state;	offset:32;	size:8;	signed:1;
	field:char next_comm[16];	offset:40;	size:16;	signed:0;
	field:pid_t next_pid;	offset:56;	size:4;	signed:1;
	field:int next_prio;	offset:60;	size:4;	signed:1;
...

Those prev_*/next_* fields are exactly the copied struct a regular tracepoint/... program receives.

Where the format file and the copied bytes actually come from

You just saw two seemingly-unrelated descriptions of the same event: a flat format file with byte offsets, and (earlier) a TP_PROTO(...) C argument list. How does one tracepoint yield both? The answer is the anatomy of the TRACE_EVENT() macro, and once you see it, every “where did that field come from” question dissolves.

Open include/trace/events/sched.h and look at the macro for sched_switch. It has three parts that map directly onto the three attach modes:

/* include/trace/events/sched.h:220 */
TRACE_EVENT(sched_switch,

	/* include/trace/events/sched.h:222 — the C argument list */
	TP_PROTO(bool preempt,
		 struct task_struct *prev,
		 struct task_struct *next,
		 unsigned int prev_state),

	/* include/trace/events/sched.h:229 — the flat per-event struct layout */
	TP_STRUCT__entry(
		__array( char, prev_comm, TASK_COMM_LEN )
		__field( pid_t, prev_pid )
		__field( int, prev_prio )
		__field( long, prev_state )
		__array( char, next_comm, TASK_COMM_LEN )
		__field( pid_t, next_pid )
		__field( int, next_prio )
	),

	/* include/trace/events/sched.h:239 — code that fills the flat struct */
	TP_fast_assign(
		memcpy(__entry->prev_comm, prev->comm, TASK_COMM_LEN); /* :240 */
		__entry->prev_pid = prev->pid;
		/* ...one assignment per field... */
	),
	...
);

Read the three parts as three answers:

• TP_PROTO is the C argument list. These become your tp_btf BPF_PROG parameters and the raw args[] slots. This is the contract the raw path delivers.
• TP_STRUCT__entry declares the layout of the flat per-event struct. This is exactly what the format file describes — prev_comm first (offset 8 after an 8-byte common header), then prev_pid, and so on. A regular tracepoint/... program receives a pointer to a struct shaped like this.
• TP_fast_assign is the code that copies live data into that flat struct when the event fires. The memcpy(__entry->prev_comm, prev->comm, TASK_COMM_LEN) at line 240 is the literal copy — the “copy cost” we said only happens for regular-tracepoint listeners. It’s the body of that one copy-probe on the iterator’s list.

So the offsets you cat out of the format file (prev_comm@8, prev_pid@24, prev_state@32, …) are generated from TP_STRUCT__entry, and the bytes that fill them at runtime are generated from TP_fast_assign. Two faces, one macro.

This is also the airtight reason a regular tracepoint can never reach prev->mm: TP_fast_assign only copied the handful of fields the kernel author chose to put in TP_STRUCT__entry. There’s no mm in there, so there’s nothing to read. tp_btf instead receives the un-copied TP_PROTO pointer prev, so every field of task_struct is reachable.

One small consistency you’ll lean on: TASK_COMM_LEN is 16:

/* include/linux/sched.h:325 */
TASK_COMM_LEN = 16,

That’s why both the kernel’s __array(char, prev_comm, TASK_COMM_LEN) and your own struct event { char comm[16]; } agree, and why __builtin_memcpy(dst, src, sizeof(dst)) of 16 bytes is exactly right.

For a BPF-aware view of what your kernel exposes, perf lists every registered tracepoint by name:

sudo perf list 'sched:*'   # all sched tracepoints

  sched:sched_migrate_task                           [Tracepoint event]
  sched:sched_process_exec                           [Tracepoint event]
  sched:sched_process_fork                           [Tracepoint event]
  sched:sched_switch                                 [Tracepoint event]
  sched:sched_wakeup                                 [Tracepoint event]
  ...

(Note: sudo bpftool perf list is not a way to discover tracepoints — it lists BPF programs currently attached to perf events, so on an idle box with nothing loaded it prints nothing at all.)

Common families you should know about:

Sharpen your pencil

A workload runs slowly. You want to see which tasks the scheduler is preempting. You could:

1. fentry on __schedule().
2. Raw tracepoint on sched_switch.
3. tp_btf on sched_switch.

Which gives you the cleanest data with least friction?

.
.
.

Answer: (3). __schedule runs for many reasons; you’d have to figure out which calls are switches (and as we saw, __schedule is the single function behind every voluntary and preemptive switch — one function, many reasons). Raw sched_switch gives you the right event but only copied fields (no access to e.g. prev->mm). tp_btf gives you live task_struct *prev, *next — read whatever you want.

tp_btf signature — how to know the arguments

For an fentry program, the function’s signature is in BTF — straightforward. For a tp_btf program, the tracepoint’s signature is what matters. Tracepoints are defined via TRACE_EVENT() macros, and the args are the parameters declared in TP_PROTO (which, you now know, is the same list __traceiter_sched_switch passes unchanged to every listener).

Look in include/trace/events/sched.h:

TRACE_EVENT(sched_switch,
    TP_PROTO(bool preempt,
             struct task_struct *prev,
             struct task_struct *next,
             unsigned int prev_state),
    ...
);

So your tp_btf program is:

SEC("tp_btf/sched_switch")
int BPF_PROG(on_switch, bool preempt, struct task_struct *prev, struct task_struct *next, unsigned int prev_state)

Four args matching the TP_PROTO (the trailing prev_state was added in 5.18 — commit 9c2136be0878, “sched/tracing: Append prev_state to tp args instead”).

For tracepoints whose TP_PROTO you don’t know offhand, bpftool btf dump file /sys/kernel/btf/vmlinux | grep btf_trace_sched_switch will show you the typedef.

The lab

schedlat.h — the shared event record

#ifndef PRACTICAL_EBPF_SCHEDLAT_H
#define PRACTICAL_EBPF_SCHEDLAT_H

struct event {
    unsigned int prev_pid;
    unsigned int next_pid;
    unsigned long long wait_ns;     /* how long next was waiting */
    char prev_comm[16];
    char next_comm[16];
};

#endif

schedlat.bpf.c — measure scheduling latency per task

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

#include "schedlat.h"

char LICENSE[] SEC("license") = "GPL";

/* When was this task last switched OUT (i.e. when did it stop running)? */
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 16384);
    __type(key, __u32);    /* tid */
    __type(value, __u64);  /* timestamp captured on switch-out */
} last_run SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

SEC("tp_btf/sched_switch")
int BPF_PROG(on_switch, bool preempt, struct task_struct *prev, struct task_struct *next, unsigned int prev_state)
{
    __u64 now = bpf_ktime_get_ns();
    __u32 prev_tid = prev->pid;   /* live ptr deref! */
    __u32 next_tid = next->pid;

    /* Record when prev was scheduled out */
    bpf_map_update_elem(&last_run, &prev_tid, &now, BPF_ANY);

    /* How long was next waiting? */
    __u64 *t = bpf_map_lookup_elem(&last_run, &next_tid);
    if (!t)
        return 0;     /* first time we see next; no wait time */

    __u64 wait = now - *t;
    if (wait < 1000)
        return 0;     /* skip < 1µs noise */

    struct event *e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
    if (!e) return 0;
    e->prev_pid = prev_tid;
    e->next_pid = next_tid;
    e->wait_ns = wait;
    /* Direct deref of comm[16] — array, not pointer */
    __builtin_memcpy(e->prev_comm, prev->comm, sizeof(e->prev_comm));
    __builtin_memcpy(e->next_comm, next->comm, sizeof(e->next_comm));
    bpf_ringbuf_submit(e, 0);
    return 0;
}

What’s new

• SEC("tp_btf/sched_switch") — tp_btf attach. Args match TP_PROTO of sched_switch. (prev and next are the same task_struct * the kernel passed to trace_sched_switch at core.c:7186.)
• Direct deref of prev->pid, next->pid, prev->comm — works because prev and next are PTR_TO_BTF_ID (live, typed kernel pointers). prev->pid reads the field at include/linux/sched.h:1063; prev->comm reads the 16-byte array at include/linux/sched.h:1173. A regular tracepoint could read these too (they’re in the copied struct) — but only these. The win is that you could also read prev->mm, prev->cgroup, etc., none of which TP_fast_assign copied.
• The pattern is more interesting than a single function tracer. We’re tracking every task. The map fills up to one entry per active TID and naturally bounds itself (active TIDs ≪ max_entries on most systems).
• if (wait < 1000) return 0; — filter sub-µs noise. The kernel switches frequently enough that without filtering, you’d flood the ringbuf.
• Remember what wait_ns is. It’s now - last_run[next_tid]: the off-CPU gap between this task’s two consecutive on-CPU stints. A proxy for scheduling delay, not strict wakeup-to-run latency.

schedlat.c — userspace consumer

Same pattern. Print:

printf("%s [%u] waited %llu µs after %s [%u]\n",
       e->next_comm, e->next_pid, e->wait_ns / 1000,
       e->prev_comm, e->prev_pid);

Run

make
sudo ./schedlat 2>&1 | head -50

Expected: a stream of switches with wait times, e.g.:

firefox [4001] waited 152 µs after kworker/u8:2 [42]
chromium [5002] waited 89 µs after firefox [4001]
...

Spikes (> 1ms) often correlate with workload, lock contention, scheduler decisions worth investigating.

What to break, in order

Break 1 — Convert to regular tracepoint

SEC("tracepoint/sched/sched_switch")
int on_switch(struct trace_event_raw_sched_switch *ctx)
{
    __u32 prev_tid = ctx->prev_pid;
    __u32 next_tid = ctx->next_pid;
    /* ... no struct task_struct * available ... */
}

This compiles and works for prev_pid, next_pid, prev_comm, next_comm (all in the copied struct — the exact fields TP_STRUCT__entry declared and TP_fast_assign filled). But there’s no way to reach prev->real_parent, prev->cgroup, prev->mm, etc. — those live in the task_struct, and a regular tracepoint only hands you the copied event fields, not the live pointer. (And note: attaching this regular tracepoint/... program is what puts the TP_fast_assign copy probe on the iterator’s list — so now the per-event memcpy actually runs.)

For most tracers, tp_btf is more ergonomic. Reach for regular tracepoint/... only when:

• You explicitly want stability (the raw struct format won’t change).
• The kernel doesn’t have BTF (rare; pre-5.4).
• You want to be portable across BPF runtimes that lack BTF awareness.

Break 2 — Attach to a non-existent tracepoint

SEC("tp_btf/this_tracepoint_is_not_real")

Load fails:

libbpf: prog 'on_switch': failed to find kernel BTF type ID of 'this_tracepoint_is_not_real'

The tracepoint name is verified against kernel BTF. Typo-proof.

Break 3 — Wrong signature

SEC("tp_btf/sched_switch")
int BPF_PROG(p, struct task_struct *prev, struct task_struct *next)
{ /* missing 'bool preempt' first arg */ }

Loads, runs, but prev is actually bool preempt cast to a pointer — garbage. Direct deref segfaults the BPF program (Verifier kills the run).

The Verifier should catch this if you compile against the right BTF — but if it doesn’t, the symptom is “program loads but data is wrong.” Always check TP_PROTO.

Break 4 — Use BPF_CORE_READ when direct deref would do

__u32 next_tid = BPF_CORE_READ(next, pid);   /* helper-call-based */

Works. Slower than direct deref (~50ns extra for the helper call). For trusted, well-typed pointers like next here, direct deref is preferred.

But: if next could be NULL or invalid, BPF_CORE_READ returns 0 instead of crashing. So for edges where you’re chasing a chain (next->mm->start_brk), prefer BPF_CORE_READ so any NULL hop is silently zero-handled.

What to read in the kernel

• include/trace/events/sched.h — definitions of all sched/* tracepoints. Search TRACE_EVENT(sched_switch (line 220). The macro expands into a lot of code, but the three parts that matter are TP_PROTO (line 222, your tp_btf args), TP_STRUCT__entry (line 229, the format/copied-struct layout), and TP_fast_assign (line 239, the copy code — note the memcpy at line 240).
• include/linux/tracepoint.h — the macro machinery. Skim. Note __DECLARE_TRACE and __DECLARE_TRACE_COMMON (the helper TRACE_EVENT expands through), the generated __traceiter_##name (line 267), and how the firing site reaches it via __DO_TRACE_CALL (lines 227/231: a static_call when CONFIG_HAVE_STATIC_CALL, else a plain indirect call).
• kernel/tracepoint.c — what happens when a tracepoint fires. The generated __traceiter_<event> iterator walks the registered probe list.
• kernel/trace/bpf_trace.c — search bpf_get_raw_tracepoint (line 2051). This is how BPF programs attach to raw tracepoints; the per-event dispatch into your program is __bpf_trace_run (line 2073). tp_btf goes through the same raw-tracepoint path (BPF_TRACE_RAW_TP, kernel/bpf/syscall.c:4388): the link is set up via bpf_raw_tracepoint_open / bpf_get_raw_tracepoint, just with BTF-typed arguments.
• kernel/sched/core.c — the scheduler core. See __schedule (line 7017), context_switch (line 5329), and the tracepoint firing site trace_sched_switch(preempt, prev, next, prev_state) (line 7186) — the source of everything today’s lab measures.
• tools/testing/selftests/bpf/progs/cgrp_ls_tp_btf.c — official examples using tp_btf.

Bullet Points

• A CPU runs one task at a time; runnable-but-waiting tasks sit on a per-CPU run queue. A context switch (context_switch, core.c:5329) saves prev and loads next; __schedule (core.c:7017) drives every switch and fires sched_switch (core.c:7186) with prev/next already chosen.
• Today’s wait_ns is off-CPU time between a task’s two runs (now - last_run[next_tid]) — a proxy for scheduling delay, not strict wakeup-to-run latency (that needs sched_wakeup).
• A TRACE_EVENT has three parts: TP_PROTO (your tp_btf/raw args), TP_STRUCT__entry (the format file + copied struct), and TP_fast_assign (the memcpy that fills the copy). A regular tracepoint can only see what TP_fast_assign copied — which is why it can’t reach prev->mm.
• A firing tracepoint runs __traceiter_<event>, which walks a probe list and calls each listener with the raw TP_PROTO args — no copy at this layer. The struct copy is one optional probe, present only when a tracepoint//ftrace/perf listener attaches — that’s why the cost is bounded.
• Tracepoints are explicit instrumentation hooks placed by kernel devs; their format is stable across versions.
• Tracepoint events have three BPF section families: tracepoint/... (copied struct), raw_tracepoint/... (raw positional args), and tp_btf/... (typed BTF args). BPF raw and tp_btf share the path through bpf_get_raw_tracepoint/__bpf_trace_run (BPF_TRACE_RAW_TP); tp_btf just adds BTF types.
• Use tp_btf for new code. Same hook, lower overhead (no copy, no tracefs lookup), full field access via direct deref.
• The TP_PROTO(...) of a tracepoint defines your tp_btf BPF program’s argument list.
• Discover tracepoints in /sys/kernel/tracing/events/.
• Common families: sched/*, block/*, net/*, syscalls/*, kmem/*, filemap/*.
• Prefer fentry over tracepoint/syscalls/sys_enter_xxx (faster on the syscall path).
• Direct deref works on tp_btf args; BPF_CORE_READ for chains where any hop could be NULL.

Check question

You attach a tp_btf to sched_switch. Inside your program, you save prev (a task_struct *) into a hash map for use later. Will dereferencing it later still work?

Tomorrow

Day 9: stack traces. BPF_MAP_TYPE_STACK_TRACE, bpf_get_stackid, kernel vs user stacks, and how to fold output for flame graphs.

Day 9 — Stack traces and the path to flame graphs

Today’s mission: capture the kernel call stack at every vfs_read, aggregate by unique stack, and produce data ready for a flame graph. Along the way you’ll learn what a native call stack actually is, how the kernel unwinds it without frame pointers, and why turning a raw return address back into a function name is a userspace job split across /proc/kallsyms, /proc/PID/maps, and a pile of base arithmetic. Total time: ~120 minutes.

Why stack traces matter

Counters tell you “this happened 1000 times.” Stack traces tell you who caused it. Combined with rate aggregation, they reveal hotspots — which user code, through which kernel paths, accounts for the bulk of an event’s occurrences.

This is how perf top, bpftrace, and most profiling tools work under the hood. By Day’s end you’ll know how to write your own.

But “capture the stack” hides four things that will bite you the moment you look at the raw output, and most of today is about making them obvious:

1. What a native call stack even is — and how the kernel walks it backwards (the thing bpf_get_stackid does for you).
2. What the syscall-entry frames are — why every read() shares an identical base of kernel frames.
3. How a raw kernel return address becomes a function name — /proc/kallsyms, and why an exact match almost never works.
4. How a raw user return address becomes a function name — ASLR, load bases, and addr2line.

We’ll teach each as we hit the part of the lab that needs it.

First: what is a call stack, and how do you walk it backwards?

Before we touch any BPF, we have to be precise about the thing we’re capturing. When one function calls another on x86_64, the CALL instruction does one concrete thing: it pushes the return address — the address of the instruction right after the call — onto the stack, then jumps. When the callee runs RET, it pops that address and resumes there. So at any instant, the stack contains a chain of return addresses: one per function currently in flight, innermost (the leaf, where the CPU is right now) at the top.

Unwinding = recovering that ordered list of return addresses, leaf-first. That list is the stack trace. Unwinding produces exactly this — an array of return addresses (the internal trace->ip[]), innermost first; bpf_get_stackid hashes that array, stores it, and returns a stackid (see below).

Don’t confuse this with the BPF stack from Day 1/4. Day 1 taught the BPF program’s own 512-byte stack addressed through register R10 — that’s scratch space the verifier polices for your eBPF code. Today’s call stack is the CPU’s native kernel/user stack: the real x86 stack of return addresses laid down by CALL instructions in compiled kernel and userspace code. Different stack, different purpose. bpf_get_stackid walks the native one.

So how do you walk it backwards? You have a stack pointer and you know the top frame’s instruction pointer — but the stack is just bytes. Where does one frame end and the next begin? There are two answers.

Frame-pointer unwinding (the simple way)

With -fno-omit-frame-pointer, the compiler dedicates register %rbp to point at a known spot in the current frame: the saved-%rbp/return-address pair. Each function’s prologue pushes the caller’s %rbp and sets %rbp to the new frame. The result is a linked list: %rbp points at [saved %rbp | return addr], and the saved %rbp points at the previous frame’s pair, and so on. A walker just follows the chain, reading a return address out of each node.

Cheap and simple — but it costs a general-purpose register (%rbp can’t be used for anything else), so for years amd64 distros compiled everything with -fomit-frame-pointer to claw that register back. That single decision is why user-stack walks fail on older binaries: there’s no %rbp chain to follow.

ORC unwinding (how the x86_64 kernel does it)

The kernel can’t afford to lose %rbp, yet it still needs reliable stacks. The fix: an out-of-band table built at compile time. The kernel image ships two arrays — __start_orc_unwind_ip[] / __start_orc_unwind[] (declared extern at arch/x86/kernel/unwind_orc.c:30) — that, keyed by any instruction pointer, describe how to find the previous frame without a frame pointer: “at this IP, the stack pointer is at such-and-such offset, the return address lives there.” orc_find() (unwind_orc.c:209) does the per-IP lookup, and unwind_get_return_address() (unwind_orc.c:380) produces each frame. The common frame type it resolves is ORC_TYPE_CALL (the case ORC_TYPE_CALL in the unwind step, unwind_orc.c:611).

This is why kernel stacks “always work” while user stacks need frame pointers: the kernel carries its own unwind table; userspace binaries don’t (their equivalent, DWARF .eh_frame, is too expensive for the kernel to parse from a BPF context — more on that below).

Both methods produce the same result: the ordered trace->ip[] array of return addresses, which bpf_get_stackid hashes. The captured array lands in a struct perf_callchain_entry { u64 nr; u64 ip[]; } (include/linux/perf_event.h:59) — nr frames, then the IPs.

One subtlety that haunts symbolization

Look again at what CALL pushes: the address of the instruction after the call. So every captured frame (except the leaf) is a return address that points into the middle of a function, one instruction past a call site — never at the function’s start. Hold that thought; it’s the root cause of the kallsyms “nearest-preceding-symbol” trick we hit later.

Meet BPF_MAP_TYPE_STACK_TRACE

A purpose-built map type. You don’t store key/value pairs in it directly — you call bpf_get_stackid(ctx, &stacks, flags) and the kernel:

1. Walks the current stack (frame pointers or ORC, per the section above).
2. Hashes the frame array into a 32-bit stackid.
3. Stores the frames keyed by stackid (deduplicated — same stack → same id).
4. Returns stackid to your program.

You then use stackid as a key in a normal map (e.g., a hash map of stackid → count) to aggregate. The helper itself is BPF_CALL_3(bpf_get_stackid, ...) at kernel/bpf/stackmap.c:323; it deliberately returns unsymbolized frames — turning addresses into names is a userspace concern (we’ll see exactly why below).

The dedup matters: if 10 million events all happen with the same stack, you store the frame array once and count it 10 million times against one stackid. Memory is bounded by the number of unique stacks, not events.

Kernel stack vs user stack

The flag argument to bpf_get_stackid (and its sibling bpf_get_stack) decides which stack you walk:

• Default (flag=0): kernel stack. Walked via the ORC unwinder on x86_64 (default since 4.14) or frame pointers — the two mechanisms from the section above. Always works in any tracing context, because the kernel ships its own ORC table.
• BPF_F_USER_STACK: userspace stack of the current task. Works only if the user binary was compiled with frame pointers. The DWARF unwind info (.eh_frame) that would otherwise rescue a frame-pointer-omitted binary is too expensive for the kernel to parse from BPF context, so the kernel can’t unwind userspace from DWARF — no frame pointers, no user stack. Modern Linux distributions started building -fno-omit-frame-pointer by default starting with Fedora 38 in 2023, and Ubuntu 24.04 in 2024, to make this useful again.
• BPF_F_FAST_STACK_CMP: hash-only comparison (faster, no full frame walk for dedup).
• BPF_F_REUSE_STACKID: allow stackid reuse across capture sessions.

To get a combined kernel+user stack, you call bpf_get_stack twice (not bpf_get_stackid) — once with each flag — and combine in userspace before symbolizing.

There are no Dumb Questions

Q: How accurate are kernel stacks?

A: Very, in 99% of cases. The ORC unwinder is precise. Edge cases: in-flight kprobes that occur during function prologue/epilogue may produce off-by-one frames. Highly optimized leaf functions can be tail-called and not appear in the stack at all. For most tracing this is fine.

Q: Why doesn’t user stack walking “just work” like the kernel side?

A: Userspace binaries on amd64 have been compiled with -fomit-frame-pointer for years (default in distros until Fedora 38 in 2023 and Ubuntu 24.04 in 2024 reversed it). Without frame pointers, walking back through stack frames requires DWARF unwind info (.eh_frame) — which is expensive and the kernel can’t process. Modern distros are reversing course (Fedora, Ubuntu now ship frame-pointer-enabled libc and friends), but legacy binaries lack the metadata.

Q: How do I symbolize JIT’d languages (Java, Node.js)?

A: They expose runtime symbol files. perf-PID.map is a text file in /tmp mapping JIT addresses to function names. JIT runtimes (HotSpot, V8) generate these. Your userspace symbolizer reads them alongside ELF tables.

What the syscall-entry frames actually are

The exercise below leans on a specific 5-frame kernel chain for cat /etc/passwd, and the flame graph later calls the wide base “the common syscall-entry frames.” On Day 7 you saw that the read tracepoint sits one layer up at syscall entry while fentry/vfs_read hooks the bottom — but not the rungs between those two points. Here’s the full ladder — you don’t need to re-derive the syscall mechanism, just see the rungs.

A read() does not jump straight into vfs_read. On x86_64 it traverses a fixed dispatch ladder:

1. The hardware SYSCALL instruction lands at the assembly stub entry_SYSCALL_64.
2. That calls the C dispatcher do_syscall_64(regs, nr) (arch/x86/entry/syscall_64.c:87).
3. Which dispatches through the syscall table to __x64_sys_read — the arch wrapper that SYSCALL_DEFINE3(read, ...) (fs/read_write.c:724) expands to.
4. Which calls the thin ksys_read() (fs/read_write.c:706).
5. Which finally calls vfs_read() (fs/read_write.c:554) — the leaf you’re tracing.

This is why every sample shares a wide common base in the flame graph: every read() on the box enters through the identical entry_SYSCALL_64 → do_syscall_64 prefix; only the leaf paths diverge. It also fixes the leaf ordering in a folded line: process (cat) at the base, then the entry frames, with vfs_read — the function you traced — on top.

Sharpen your pencil

You write a tracer that captures kernel + user stacks for every vfs_read. Your test workload is cat /etc/passwd from bash. Roughly how many unique stackid values do you expect?

.
.
.

Answer: A small handful. cat’s call to read() always traverses roughly the same userspace path (libc read wrapper → syscall instruction). Bash’s cat invocation similarly goes through the same path. The kernel side is also stable: entry_SYSCALL_64 → do_syscall_64 → __x64_sys_read → ksys_read → vfs_read. Maybe 1–3 unique stacks total. Now run on a complex workload (Firefox loading a page) and you’ll see hundreds — that’s the dedup paying off.

How deep can a stack go? The hard 127-frame cap

Before you size the map, know the ceiling. When you create a BPF_MAP_TYPE_STACK_TRACE map, the kernel requires value_size / 8 (8 bytes per u64 frame) to be <= sysctl_perf_event_max_stack up front. So asking for 256 frames does not silently clamp — map creation is rejected with -EINVAL (stack_map_alloc, stackmap.c:113). value_size / 8 must be <= sysctl_perf_event_max_stack at creation time.

• PERF_MAX_STACK_DEPTH is 127 by default (include/uapi/linux/perf_event.h:1285), and int sysctl_perf_event_max_stack = PERF_MAX_STACK_DEPTH; (kernel/events/callchain.c:23). The runtime knob is /proc/sys/kernel/perf_event_max_stack (registered at callchain.c:306). It’s shared with the perf subsystem because bpf_get_stackid reuses perf’s callchain machinery.
• stack_map_calculate_max_depth() (kernel/bpf/stackmap.c:53) computes the runtime depth from value_size / elem_size and reconciles it with the sysctl: if (max_depth > curr_sysctl_max_stack) return curr_sysctl_max_stack;. This runtime clamp only matters when the sysctl is lowered after the map was created (or for the BPF_F_SKIP_FIELD adjustment) — not as a creation-time ceiling. The creation-time gate is separate: stack_map_alloc outright rejects a map whose value_size / 8 > sysctl_perf_event_max_stack with -EINVAL (stackmap.c:113–114).

Practical consequence: the lab’s MAX_STACK_DEPTH = 64 is comfortably under the cap. Choosing 64 vs 127 trades memory-per-unique-stack against truncation risk on deep stacks. Going above 127 buys nothing unless you also raise the sysctl.

The end-to-end flame graph pipeline

Capture in BPF, aggregate by stackid, read in userspace, symbolize each frame, fold into the textual format flamegraph.pl expects:

thread_or_pid;fn1;fn2;fn3;fn4 1234

Then pipe through flamegraph.pl (Brendan Gregg’s tool) or upload to speedscope.app.

The lab

stacks.bpf.c

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

char LICENSE[] SEC("license") = "GPL";

#define MAX_STACK_DEPTH 64

struct {
    __uint(type, BPF_MAP_TYPE_STACK_TRACE);
    __uint(max_entries, 16384);
    __uint(key_size, sizeof(__u32));
    __uint(value_size, MAX_STACK_DEPTH * sizeof(__u64));
} stacks SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 16384);
    __type(key, __u64);     /* (kstack_id << 32) | ustack_id */
    __type(value, __u64);   /* count */
} counts SEC(".maps");

SEC("fentry/vfs_read")
int BPF_PROG(on_read)
{
    __s64 kid = bpf_get_stackid(ctx, &stacks, 0);
    __s64 uid = bpf_get_stackid(ctx, &stacks, BPF_F_USER_STACK);

    /* Negative return = failure to capture (returns a negative errno:
       -EFAULT/-EEXIST/-ENOMEM, e.g., user stack with no frame ptrs). */
    if (kid < 0 && uid < 0)
        return 0;

    __u64 key = ((__u64)(kid & 0xffffffff) << 32) | (uid & 0xffffffff);
    __u64 *c = bpf_map_lookup_elem(&counts, &key);
    if (c) {
        __sync_fetch_and_add(c, 1);
    } else {
        __u64 one = 1;
        bpf_map_update_elem(&counts, &key, &one, BPF_NOEXIST);
    }
    return 0;
}

What’s new (and what’s recycled)

• The attach is old news. SEC("fentry/vfs_read") is the same BTF-typed entry trampoline on vfs_read you used on Day 6, and BPF_PROG(on_read) is the argument-unpacking macro demystified on Day 7. Nothing new in how we attach — vfs_read (fs/read_write.c:554) is the identical target from the Day 6 lab. Today’s novelty is entirely inside the handler.
• Two bpf_get_stackid calls — one kernel (flag 0), one user (BPF_F_USER_STACK).
• Negative return checks. The helpers return a negative errno (-EFAULT/-EEXIST/-ENOMEM) on failure (e.g., user-stack walk hit a missing frame pointer, or the stack map slot was already taken by a different stack).
• Composite key: pack both stackids into one u64 so identical (kernel, user) pairs share a counter. This (kid << 32) | uid packing is the only genuinely new twist here — two stackids collapsed into one hash-map key.
• The counter logic is pure Day 2. Recall from Day 2: a hash-map lookup hands back an unlocked live pointer, so the concurrent increment must be __sync_fetch_and_add; and the BPF_NOEXIST insert can lose a racing increment (a cosmetic miss). Same count.c pattern, same check question — no need to re-derive it.

stacks.c — userspace dumper + symbolizer

Outline (you can use blazesym or write a minimal kallsyms parser). The iteration loop is the same userspace bpf_map_get_next_key walk from Day 2 — one key per call, the loop lives in userspace:

/* every 5s, dump top stacks by count: */
while (!exiting) {
    sleep(5);

    /* iterate counts map */
    __u64 key = 0, next;
    __u64 val;
    int cnt_fd = bpf_map__fd(skel->maps.counts);
    int stk_fd = bpf_map__fd(skel->maps.stacks);

    while (bpf_map_get_next_key(cnt_fd, &key, &next) == 0) {
        bpf_map_lookup_elem(cnt_fd, &next, &val);
        __u32 kid = next >> 32, uid = next & 0xffffffff;

        __u64 kframes[64] = {0}, uframes[64] = {0};
        bpf_map_lookup_elem(stk_fd, &kid, kframes);
        bpf_map_lookup_elem(stk_fd, &uid, uframes);

        /* Print folded: stack;...; count */
        printf("[count=%llu]\n", val);
        for (int i = 0; i < 64 && kframes[i]; i++)
            printf("  K %llx\n", kframes[i]);
        for (int i = 0; i < 64 && uframes[i]; i++)
            printf("  U %llx\n", uframes[i]);
        printf("\n");

        key = next;
    }
}

For real symbolization, link against libblazesym (modern, fast) or write a kallsyms parser. We won’t write one today — Day 9 is about capturing the data; symbolization is plumbing. But you should understand what that plumbing does, because the next two sections walk it by hand.

Run it

make
sudo ./stacks &        # job %1; prints a dump every 5s
# Generate work, then wait for the next 5-second dump to see it:
find /usr -name "*.so" > /dev/null
cat /etc/passwd > /dev/null
sleep 5

Output is interval-driven — nothing prints until the next 5-second tick. When you’re done, stop the background dumper (otherwise it keeps printing as root forever):

kill %1        # or: sudo pkill stacks

You’ll see frames printed as raw hex (K <addr> / U <addr>). Those are the raw return addresses from trace->ip[] — exactly the values the section on call stacks promised. Resolving them by hand is fiddly, and the two halves (kernel vs user) work completely differently. Let’s do each by hand once; that’s the whole reason bpftool/libblazesym exist.

Symbolizing kernel frames: /proc/kallsyms

A running kernel has no ELF symbol table the way a binary on disk does — the image is loaded and relocated. Instead the kernel exposes its runtime symbol table at /proc/kallsyms: one line per symbol, address type name, e.g. ffffffff... T vfs_read. A userspace symbolizer turns a kernel return address into a name by looking it up here. Two gotchas:

• kptr_restrict zeros the addresses for non-root. This sysctl governs visibility; at the default value 1, an unprivileged reader sees every address as 0000000000000000. Grep as a normal user and you find zeros. You must read it as root. (It’s a hardening measure: leaking kernel addresses defeats KASLR.)
• kallsyms lists symbol start addresses, not ranges — and isn’t address-sorted on disk. Recall the subtlety from the call-stack section: a captured frame is a return address in the middle of a function, never at its start. So an exact match almost never succeeds. The fix is nearest-preceding-symbol: sort lexicographically (which equals numeric order for the fixed-width 16-hex-digit kernel addresses), then take the greatest symbol address <= your frame address.

# ADDR is one of the K <addr> values, e.g. ffffffff9a0ea100
sudo sort /proc/kallsyms | awk -v a=ADDR '$1 <= a {s=$0} END {print s}'
# ffffffff9a0ea0e0 T vfs_read

The frame address (ffffffff9a0ea100) is greater than vfs_read’s start (ffffffff9a0ea0e0) and less than the next symbol’s start, so it resolves to vfs_read — a return address a few bytes inside it. This is precisely the lookup bpftool/libblazesym automate.

Symbolizing user frames: load bases and ASLR

Kernel frames live in one address space; user frames are runtime virtual addresses inside the traced process, and that’s where it gets harder. addr2line and a binary’s ELF symbol table speak in file offsets — the link-time addresses baked into the .so/executable. The captured frame is a runtime address. The two differ by the load base:

file_offset = runtime_addr - load_base

Why isn’t load_base just zero? Because Position-Independent Executables (PIE) and all shared libraries (libc, etc.) are mapped at a randomized base each run — that’s ASLR. To recover the file offset you must subtract the base, and the base is the lowest mapped address of that object’s executable segment, found in the left column of the matching line of /proc/PID/maps.

# A is a U <addr> runtime value; PID is the traced process.
# Find B = the base of the object that contains A:
sudo grep -i 'r-xp' /proc/PID/maps      # locate the segment whose range covers A; B is its start
addr2line -f -e /path/to/object $((A - B))

The base subtraction is mandatory for PIE executables and shared libraries. The one exception: a classic non-PIE ET_EXEC is linked at a fixed address (its main text base is effectively 0), so you pass the runtime address straight to addr2line — which is the carve-out you’ll see in tools. Two beginner traps produce ??:0: forgetting -e <object> (wrong object) and forgetting the subtraction (wrong base). This per-object bookkeeping across many mapped libraries is the bulk of what libblazesym/bpftool do for you.

(There’s a robustness escape hatch: BPF_F_USER_BUILD_ID (include/uapi/linux/bpf.h:6174) captures a build-id + offset instead of a bare address, so offline symbolization survives ASLR and even a different machine. We won’t use it today, but it’s why that mode exists.)

This hand-resolution — kallsyms nearest-preceding for kernel, base-subtract-then-addr2line for user — is exactly the plumbing libblazesym / bpftool automate, which is why we deferred symbolization in the lab.

Pipe to a flame graph

Get flamegraph.pl once (it isn’t packaged on any distro — grab it from Brendan Gregg’s repo):

git clone https://github.com/brendangregg/FlameGraph
export PATH="$PATH:$PWD/FlameGraph"

Then:

sudo ./stacks --folded > out.folded
flamegraph.pl < out.folded > out.svg
# Open out.svg in any browser. On a headless box, copy it to your laptop,
# or just drag out.folded onto https://speedscope.app (no local browser needed).
# firefox out.svg   # only if you actually have a desktop browser

--folded is required reader work — the lab’s stacks.c only prints the %llx debug dump above. You need to parse argv, symbolize each frame (using the kallsyms and addr2line mechanics you just learned), and emit one line per unique stack in root-to-leaf order (process at the base, leaf on top), with the count last. Building on the loop above, that’s roughly:

printf("%s", comm);                       /* process name at the base */
for (int i = u_n - 1; i >= 0; i--)        /* user frames, outermost first */
    printf(";%s", sym_user(uframes[i]));
for (int i = k_n - 1; i >= 0; i--)        /* then kernel frames */
    printf(";%s", sym_kernel(kframes[i]));
printf(" %llu\n", val);                   /* count */

A correct folded line for the cat /etc/passwd workload looks like this — the same kernel ladder from the Sharpen-your-pencil answer, leaf (vfs_read) on top:

cat;__libc_read;entry_SYSCALL_64;do_syscall_64;__x64_sys_read;ksys_read;vfs_read 137

In the resulting SVG: the wide base is the common process / syscall-entry frames shared by every sample, narrow towers are the divergent call paths, each frame’s width is proportional to its count, and vfs_read sits near the top of every tower (it’s the function you traced).

What to break, in order

Break 1 — Stack walk failure on BPF_F_USER_STACK

If your binary has frame-pointer-omitted libraries (most older distros), you’ll see uid < 0 for many calls — there’s no %rbp chain to follow and no DWARF the kernel can use, exactly as the unwinding section predicted. Workaround: trace fentry of a userspace-heavy function in a frame-pointer-enabled binary. Or, on Fedora/Ubuntu 24+, modern libc has frame pointers and user stacks resolve.

To debug, check bpf_get_stackid’s return value directly: the program already guards on uid < 0 (lines __s64 uid = ...; if (kid < 0 && uid < 0)), and that negative value is the errno explaining the failed user-stack walk. bpftool prog tracelog is empty by default here — this lab emits nothing to the trace pipe — so to use it you must add a printk on the failure path, e.g. in on_read:

if (uid < 0)
    bpf_printk("user-stack walk failed: uid=%lld\n", uid);

Then watch sudo bpftool prog tracelog (or sudo cat /sys/kernel/debug/tracing/trace_pipe) to see the negative errno fire when a frame-pointer-omitted library breaks the walk.

Break 2 — Forgetting MAX_STACK_DEPTH

Set value_size too small (16 * sizeof(__u64)). Stacks deeper than 16 frames get truncated. You’ll see plausible-looking but incomplete stacks. Default to 64 unless you have a reason to be smaller — and remember from the cap section that going above 127 buys nothing without raising perf_event_max_stack.

Break 3 — Use bpf_get_stack instead of bpf_get_stackid

__u64 frames[64];
int n = bpf_get_stack(ctx, frames, sizeof(frames), 0);

This copies frames directly into a buffer you provide (BPF_CALL_4(bpf_get_stack, ...) at kernel/bpf/stackmap.c:514) instead of going through the dedup map. Useful when you’re emitting one stack per event (e.g., to a ringbuf) rather than aggregating. Trade: more memory traffic, no dedup; but you have the frames immediately without a second map lookup.

Break 4 — Increase stack rate, fill the map

Trace a high-rate event like tcp_recvmsg on a busy server. With many unique stacks, you’ll fill max_entries=16384. When a new stack hashes to a slot already holding a different stack, bpf_get_stackid returns -EEXIST and the new stack is dropped — that’s the if (bucket && !(flags & BPF_F_REUSE_STACKID)) return -EEXIST; branch at stackmap.c:298 (build-id path) / :305 (normal path). So genuinely new stacks are silently lost. Pass BPF_F_REUSE_STACKID and the behavior flips: the occupied slot is overwritten instead (the xchg(&smap->buckets[id], new_bucket) at stackmap.c:317), so you keep the newest stack but lose the old one (and any counts keyed on the old stackid now point at the wrong frames).

Lesson: size stacks and counts to expected unique-stack cardinality. For most production servers, 16K–64K is fine. Profilers often go to 1M.

What to read in the kernel

• kernel/bpf/stackmap.c — the implementation. stack_map_alloc builds the map’s own structure (a buckets[] array plus a per-CPU pcpu_freelist of stack buffers — not the generic htab_map_alloc; see struct bpf_stack_map at :26), and bpf_get_stackid (:323) is the helper. Note stack_map_calculate_max_depth (:53) where the 127-cap clamp lives. Read the file once; ~800 lines.
• arch/x86/kernel/unwind_orc.c — the ORC unwinder x86_64 uses. orc_find (:209) and unwind_get_return_address (:380) are the heart of it. Don’t read deeply; just know it exists and is faster/more reliable than frame pointers.
• kernel/bpf/stackmap.c — search bpf_get_stack (:514). The non-stackid version that copies frames directly.
• tools/lib/bpf/btf.c and tools/perf/util/symbol.c — for inspiration on symbolization. The selftests don’t have a clean example.
• tools/testing/selftests/bpf/progs/stacktrace_map.c — minimal example of the pattern.

External reference (skim once): https://www.brendangregg.com/flamegraphs.html

Bullet Points

• A native call stack is a chain of return addresses pushed by CALL instructions. Unwinding recovers them leaf-first; that ordered list is the trace. (Distinct from the BPF program’s own R10 stack from Day 1/4.)
• Two ways to unwind: frame pointers (%rbp linked list, cheap but costs a register — why user stacks fail) and ORC (an out-of-band .orc_unwind table the x86_64 kernel ships — why kernel stacks always work).
• BPF_MAP_TYPE_STACK_TRACE stores frame arrays keyed by hash; same stack → same stackid. Memory bounded by unique stacks.
• bpf_get_stackid(ctx, &stacks, flags) captures the current stack and returns a stackid. Use it as a key in another map for aggregation. BPF_F_USER_STACK walks userspace — only works with frame pointers.
• The syscall ladder entry_SYSCALL_64 → do_syscall_64 → __x64_sys_read → ksys_read → vfs_read is the common base every read() shares — the wide base of the flame graph.
• 127-frame cap: the kernel clamps captured depth to sysctl_perf_event_max_stack (default PERF_MAX_STACK_DEPTH = 127) regardless of value_size.
• Symbolization happens in userspace. Kernel frames: /proc/kallsyms, root-only (kptr_restrict=1 zeros it), nearest-preceding-symbol (return addr is mid-function). User frames: subtract the load base from /proc/PID/maps (ASLR/PIE), then addr2line — except non-PIE ET_EXEC.
• bpf_get_stack (no -id) copies frames directly to a buffer — preferred for per-event emission, not aggregation.
• Flame graph format: thread;fn1;fn2;fn3 count per line; flamegraph.pl produces interactive SVG.

Check question

Two CPUs simultaneously call bpf_get_stackid with identical stacks. Do they get the same stackid? Same map slot? Race?

Tomorrow

Day 10: uprobes. Tracing functions in userspace binaries from BPF. We’ll attach to bash’s readline() and see every command typed.

Day 10 — Uprobes: tracing userspace from BPF

Today’s mission: trace a function in a userspace binary. Watch every command typed into bash by hooking readline(). But first, understand the genuinely new machinery underneath: how an executable actually lives in memory, how the kernel patches a userspace instruction without corrupting it, and how it regains control when a function returns. Total time: ~110 minutes.

The other half of “what’s running on this machine”

Days 6–9 traced kernel functions. But your applications are mostly userspace — and many interesting events happen entirely in user code without touching syscalls (think: malloc, g_object_unref, PQexec, JIT compilation events).

Uprobes let BPF programs attach to userspace functions. The kernel patches the target binary’s executable pages with breakpoints; when any process executing that binary reaches the patched instruction, the trap fires and your BPF program runs.

It’s like kprobe — but the target is a function in /usr/bin/bash, /usr/lib/libssl.so.3, or your own myapp rather than vfs_read.

Here’s the thing the tutorials skip, though: tracing userspace code from the kernel is a genuinely different mechanism than the kernel-function patching you met earlier in the series. The closest kernel sibling is Day 7’s kprobe — the int3-trap path, not Day 1’s fentry (which patches a NOP→trampoline slot with no runtime trap). Three pieces of background make the difference, and the rest of today only makes sense once you have them:

1. How an executable lives in memory — it isn’t copied into RAM; it’s mmap’d from a file, and its pages are shared across every process running it. This is why one uprobe registration covers every bash on the system.
2. How the kernel patches a userspace instruction safely — it can’t just overwrite a shared, read-only, on-disk-backed page. It uses copy-on-write to get a private copy, writes a 1-byte int3 trap, and replays the displaced instruction out-of-line.
3. How the kernel regains control when a function returns — there’s no fixed instruction at a return site to patch, so it hijacks the return address. This is the whole trick behind uretprobes, which today’s lab depends on.

We’ll teach each as we hit the part of the lab that depends on it. Day 7’s kprobe used the same int3-trap primitive on kernel text in one shared address space; everything below is the userspace counterpart, and it is meaningfully different.

How an executable lives in memory: file-backed mmap, the page cache, and COW

Before we can patch a userspace instruction, we have to know where that instruction physically is. The intuition most people carry — “the program got loaded into RAM” — is wrong in a way that matters today.

A binary is not copied into RAM wholesale. When you run /usr/bin/bash, the loader does not slurp the file into memory. It mmap()s the file: it creates a file-backed mapping that says “this range of virtual addresses corresponds to these byte-offsets of this file.” The actual bytes are pulled in lazily, one page at a time, the first time the CPU touches them.

Where do those pages live? In the page cache — the kernel’s single, shared cache of file contents. A page of file data is keyed by (inode, file-offset): the inode identifies which file, the offset identifies which 4 KB chunk of it. (Recall the page from Day 1: physical RAM is handed out in PAGE_SIZE = 4 KB frames.) The crucial property:

Every process running the same binary maps the same physical pages. Two bash processes don’t each get their own copy of bash’s executable code. They both map the page-cache pages for (bash's inode, offset of .text), read-only and executable. One copy of the code in RAM, shared by all of them, plus the on-disk file backing it.

That shared page cache is exactly why a uprobe is system-wide. The breakpoint isn’t a property of a process — it’s a property of the inode. Because the probe is keyed to (bash-inode, readline's offset), the kernel walks every process (and every future mmap) that maps this inode and patches each one. It does not write once into the shared page-cache page to reach everybody — that would dirty a shared, file-backed page (exactly what the COW section below says it must avoid). The system-wide reach comes from the inode key plus the walk over all mappings, not from one shared write.

You can see the inode-identity in the kernel’s own data structure. A uprobe is identified by an inode pointer and a file offset — not a virtual address:

/* kernel/events/uprobes.c:62 — struct uprobe */
struct inode *inode;   /* the file the probe lives in (line ~69) */
loff_t        offset;  /* file-offset of the patched instruction (line ~74) */

And uprobe_register refuses a target that isn’t file-backed — there’d be no page cache to read the original instruction from:

/* kernel/events/uprobes.c:1402 */
/* copy_insn() uses read_mapping_page() or shmem_read_mapping_page() */
if (!inode->i_mapping->a_ops->read_folio &&
    !shmem_mapping(inode->i_mapping))
    return ERR_PTR(-EIO);

To remember the instruction it’s about to overwrite (so it can restore it on detach), the kernel reads the original byte straight out of the page cache, by offset:

/* kernel/events/uprobes.c:1058, inside __copy_insn() */
page = read_mapping_page(mapping, offset >> PAGE_SHIFT, filp);

offset >> PAGE_SHIFT is “which page of the file”; read_mapping_page pulls that page-cache page (__copy_insn at :1048, copy_insn at :1070).

Patching a shared page without corrupting it: copy-on-write

Now the problem. The page we want to patch is shared read-only across every process and backed by the on-disk file. If the kernel just wrote 0xCC into it, it would (a) hit every process at once with no control, and (b) risk dirtying a file-backed page. That’s not what we want — we want to patch one process’s view at a time.

The fix is copy-on-write (COW). COW is the trick that lets many mappings share one physical page until someone writes: at the moment of a write, the writer gets its own private writable copy of just that page, and everyone else keeps the original. The kernel forces that break deliberately before patching:

/* kernel/events/uprobes.c:518 — in uprobe_write() */
/*
 * When registering, we have to break COW to get an exclusive anonymous
 * page that we can safely modify. Use FOLL_WRITE to trigger a write
 * fault if required. ...
 */
if (is_register)
    gup_flags |= FOLL_WRITE | FOLL_SPLIT_PMD;   /* :526 */

It then verifies the page it got is a private anonymous folio before writing into it (the folio_test_anon check at :552). So the sequence per process is: break COW → the target process now has a private, writable, anonymous copy of that one page → write the int3 there. Other processes keep faulting on the clean shared page until they’re patched too.

This is what “COW protects other mappings” means, concretely. Day 7’s kprobe int3-poked kernel code: one address space, one copy, patched in place. Here the target is per-process userspace memory reached through a file-backed mapping, so the kernel must privately fork the page first. Genuinely different machinery.

The int3 trap and replaying the clobbered instruction

We’ve patched a byte. Two questions remain, and the second is the one every tutorial glosses over.

What byte, and what does it do? On x86_64 the breakpoint is a 1-byte instruction, 0xCC (int3), which raises the breakpoint exception:

/* arch/x86/include/asm/uprobes.h:20 */
#define UPROBE_SWBP_INSN      0xcc
#define UPROBE_SWBP_INSN_SIZE 1

set_swbp writes it; set_orig_insn restores it on detach — both go through the same uprobe_write_opcode / COW path above:

/* kernel/events/uprobes.c:612 */
int set_swbp(...)      { return uprobe_write_opcode(..., UPROBE_SWBP_INSN, true); }
/* kernel/events/uprobes.c:627 */
int set_orig_insn(...) { return uprobe_write_opcode(..., *(uprobe_opcode_t *)&auprobe->insn, false); }

When a thread executes that byte, the CPU traps into the kernel’s int3 exception entry (DEFINE_IDTENTRY_RAW(exc_int3) at arch/x86/kernel/traps.c:1004), which notifies the uprobe layer (uprobe_pre_sstep_notifier at kernel/events/uprobes.c:2871). The dispatcher handle_swbp (uprobes.c:2712) looks up the active uprobe by faulting address (find_active_uprobe_rcu, :2724) and runs your BPF program via handler_chain (:2549, the consumer’s handler is invoked at :2549–2566). The trap builds a pt_regs snapshot of the user registers — the same pt_regs you read with PT_REGS_PARM* on Day 7 — and hands it to your program.

Now the puzzle nobody explains. The original instruction is gone — you overwrote its first byte with 0xCC. After your BPF program runs, how does the thread execute the instruction it was supposed to, and continue?

The answer is execute-out-of-line (XOL). The kernel never removes the int3 during tracing; instead it keeps the saved original instruction and replays it elsewhere:

1. At first hit, the kernel allocates a per-process executable scratch page — the xol_area (struct xol_area at kernel/events/uprobes.c:109).
2. It copies the saved original instruction (arch-fixed-up if needed) into a slot there (xol_get_insn_slot at :1871; the bytes live in arch_uprobe.ixol[], arch/x86/include/asm/uprobes.h:33).
3. It points the user instruction pointer at that slot, single-steps that one instruction, then the arch post-XOL handler relocates the IP back into the original code stream — default_post_xol_op applies regs->ip += utask->vaddr - utask->xol_vaddr (arch/x86/kernel/uprobes.c:1240), landing right after the original instruction.

The 0xCC byte stays patched in the shared/COW page the whole time. So one uprobe hit costs: trap into the kernel (~50 ns) + your BPF program’s time + a single-step of the displaced instruction. Detach reverses everything — set_orig_insn writes the saved original byte back over the int3.

Why doesn’t this corrupt other threads? Because the int3 is never removed while tracing. Threads serialize through the trap; each one single-steps its own copy of the instruction in its own xol slot. No thread ever sees a half-patched instruction.

Tie it back to Day 7’s kprobe: same int3-trap primitive — overwrite the first byte, trap into the kernel, run your program. The difference is recovery: a kprobe single-steps the original instruction in place; a uprobe replays it out-of-line in a per-process XOL slot because the target is shared, file-backed userspace memory. (Day 1’s fentry is a different mechanism — a NOP→trampoline patch with no runtime trap.)

There are no Dumb Questions

Q: Does my uprobe affect other processes running the same binary?

A: Yes — by design. The breakpoint is per-binary-inode, not per-process (that’s the whole point of the page-cache section above: the patch is keyed to the inode, and every process maps the same inode’s pages). If you uprobe /usr/bin/bash:readline, every running bash and every future bash gets the breakpoint until you detach. This is what lets you do system-wide observability without restarting workloads.

Q: What if the target process exits while I’m attached?

A: Nothing breaks. The uprobe remains registered against the inode. New invocations get hooked. Old processes that exited just disappear. The uprobe lives until you detach (close the link FD).

Q: Can I uprobe a function in a JIT-compiled language (e.g., Java methods, V8 functions)?

A: Not with regular uprobes — those addresses don’t exist at attach time; they’re generated at runtime. The runtime would need to expose USDT probes (we’ll see those below) or generate perf-PID.map files for symbolization. For Java, the JVM can be configured to emit USDT events; tools like async-profiler use this.

Catching a return: return-address hijack and the trampoline

Everything above describes an entry uprobe: trap on a fixed instruction. But today’s lab hooks the return of readline() — because the line the user typed is readline’s return value, not an argument. And a return is a problem: there is no single fixed instruction at the return site to patch. A function can be called from a hundred places and ret to a hundred different addresses.

The kernel solves this with return-address hijacking. When the entry breakpoint fires for a function you’ve uretprobed, the kernel:

1. Reads the real return address off the user stack.
2. Saves it on a per-task stack of pending returns.
3. Overwrites the stack’s return-address slot with the address of a kernel-installed userspace trampoline (prepare_uretprobe at kernel/events/uprobes.c:2252; trampoline address from uprobe_get_trampoline_vaddr at :2223).

When the function later runs its normal ret, control lands on the trampoline. On the pinned v7.1 kernel, for a native 64-bit process (the lab traces native x86_64 bash) the trampoline is not a breakpoint — arch_uretprobe_trampoline() (arch/x86/kernel/uprobes.c:349) hands back a short syscall sequence for user_64bit_mode(regs):

/* arch/x86/kernel/uprobes.c:320–343 — uretprobe_trampoline_entry */
push %rax
push %rcx
push %r11
mov  $__NR_uretprobe, %rax
syscall
/* uretprobe_syscall_check: pop %r11; pop %rcx; ret; int3 */

So when readline returns into the trampoline, it executes the uretprobe(2) syscall rather than trapping on an int3. SYSCALL_DEFINE0(uretprobe) (uprobes.c:372) validates the caller IP, then calls uprobe_handle_trampoline(regs) (uprobes.c:401; generic side at kernel/events/uprobes.c:2635) — that’s where your uretprobe consumers run and the real return address is restored.

The older int3 trampoline is still in the tree, but it’s now the fallback for 32-bit/compat processes (the asm even notes “the compat process still uses standard breakpoint”; the trailing int3 after ret is just a guard). On that path, handle_swbp recognizes the trampoline address specially:

/* kernel/events/uprobes.c:2718, in handle_swbp() — compat/fallback path */
if (bp_vaddr == uprobe_get_trampoline_vaddr())
    return uprobe_handle_trampoline(regs);

At that point your uretprobe BPF program runs, PT_REGS_RC holds the return value (for readline, the malloc’d char * the user typed), and the kernel restores the saved real return address so the caller resumes none the wiser.

Because calls nest, the kernel keeps a per-task LIFO stack of pending returns — utask->return_instances (walked at uprobes.c:2019, :2062; pushed at :2307). Each hijacked frame pushes one entry; each return pops the matching one in last-in-first-out order. That’s why a uretprobe fires correctly through deep recursion and many stack frames.

This is also why the lab declares the return value as the “first parameter” (more below): at the return trap there are no argument registers left to read — the arguments are long gone — only the return register survives. It’s the exact analogue of the kretprobe PT_REGS_RC you met on Days 6–7.

Where to attach: ELF anatomy

Your target binary is an ELF file. To attach by name, libbpf has to turn binary:symbol into the file offset the kernel registers against the inode. That resolution runs through the binary’s symbol tables — and you need to know there are two of them, because today’s Break 2 depends on the difference.

An ELF file can carry two symbol tables:

• .symtab — the full symbol table: every function, including local/static helpers. This is what strip deletes to shrink a binary. A stripped /bin/bash has an empty .symtab.
• .dynsym — the dynamic symbol table: only the symbols needed for dynamic linking (a shared library’s exported functions, a binary’s imported symbols). It survives stripping, because the dynamic loader needs it at runtime to wire up shared libraries.

libbpf searches .dynsym first, then .symtab — and the source spells out exactly why that order, and why a missing section is normal rather than an error:

/* tools/lib/bpf/elf.c:278 */
int i, sh_types[2] = { SHT_DYNSYM, SHT_SYMTAB };
/* ...:306 — comment:
 * Search SHT_DYNSYM, SHT_SYMTAB for symbol. This search order is used because
 * if a binary is stripped, it may only have SHT_DYNSYM, and a fully-statically
 * linked binary may not have SHT_DYNSYM, so absence of a section should not be
 * reported as a warning/error.
 */

So the common path is just: write SEC("uprobe//path/to/binary:symbol_name"). libbpf opens the ELF (bpf_program__attach_uprobe at tools/lib/bpf/libbpf.c:12966 → elf_find_func_offset_from_file at :12817 → elf_find_func_offset at elf.c:277), finds the symbol, and computes its file offset from the symbol’s value and the ELF program-header layout. What the kernel ultimately registers is (inode, offset) — the offset must be page-alignable and within the file size (uprobe_register checks at kernel/events/uprobes.c:1407–1413).

A subtlety worth holding onto for Break 2: the symbol’s recorded value is a virtual address, and the file offset libbpf needs are not always identical. For the common executable code segment they usually coincide, which is why the nm -D value can be handed to the kernel directly below. When a symbol exists in neither table (a private static in a stripped binary), you fall back to supplying the file offset yourself:

SEC("uprobe//usr/bin/stripped_app:0x1234")

Tools map onto the tables directly: nm -D / objdump -T read .dynsym; objdump -t reads .symtab (empty on a stripped /bin/bash); objdump -d disassembles so you can eyeball an offset when no symbol exists at all.

Argument access in uprobes

This is prerequisite knowledge you already have. Recall from Day 7: for a trap-based probe the ctx is a struct pt_regs * register snapshot, and BPF_KPROBE/BPF_KRETPROBE expand PT_REGS_PARM1..6 to the System V AMD64 argument registers (rdi, rsi, rdx, rcx, r8, r9), with PT_REGS_RC = rax for the return value (the macros live in tools/lib/bpf/bpf_tracing.h; Day 7 verified PT_REGS_PARM1 maps to di).

The only uprobe-specific twist: these macros work unchanged for userspace targets, because the int3 trap snapshots the user registers into the very same pt_regs shape. So: same as kprobes — args in rdi…r9, return in rax.

For string arguments, you need bpf_probe_read_user_str (not _kernel):

char buf[64];
bpf_probe_read_user_str(buf, sizeof(buf), (void *)PT_REGS_PARM1(ctx));

The “_user” variant tells the kernel “this address is in userspace; use copy_from_user semantics, not direct deref.” Misusing _kernel on a user pointer doesn’t crash: the kernel-nofault helper checks the address range first and rejects a userspace pointer up front (on x86, copy_from_kernel_nofault_allowed returns false for addr < TASK_SIZE_MAX, so strncpy_from_kernel_nofault returns -ERANGE — an address-range rejection, not an actual page fault), and bpf_probe_read_kernel_str then zeroes the destination buffer on the negative return.

USDT: probes built into binaries

What if your application wants to be observable? Some major libraries (libc, libpthread, postgres, mysql, openssh) ship with USDT probes — user-space statically defined tracing. These are nop instructions placed by the developer at points of interest, accompanied by ELF metadata describing the arguments.

You attach with SEC("usdt//path:provider:probe"). libbpf reads the .note.stapsdt ELF section, finds the matching provider+probe, locates the nop offset, and installs a regular uprobe there.

Discover USDTs with:

sudo bpftrace -l 'usdt:/usr/lib/x86_64-linux-gnu/libc.so.6:*'

libc’s USDTs include lll_lock_wait, lll_lock_wait_private, setjmp, longjmp. Postgres has query__start, query__done. They’re a stable contract from the application.

Sharpen your pencil

Bash is a single-process per terminal session. You uprobe readline and observe nothing — but you’re definitely typing in another terminal. What’s wrong?

.
.
.

Answer: likely one of: (a) bash version mismatch — the bash on disk and the bash running might differ if you’re inside a shell started before a system upgrade. Restart your shell. (b) bash’s readline may be statically linked or in libreadline.so.8 rather than the bash binary itself — nm /usr/bin/bash | grep readline to confirm. If empty, attach to libreadline instead. (c) PATH issue — which bash gives one path; the running shell may have started from a different one (e.g., /bin/bash symlink resolution).

The lab

bashspy.h — the shared event record

#ifndef PRACTICAL_EBPF_BASHSPY_H
#define PRACTICAL_EBPF_BASHSPY_H

#define MAX_LINE 256

struct event {
    unsigned int pid;
    char comm[16];
    char line[MAX_LINE];
};

#endif

bashspy.bpf.c

The output channel is a ringbuf — recall it from Day 1: a kernel→userspace MPSC channel where you reserve a slot, fill it, and submit (zero-copy), and a single userspace consumer drains it via a callback. Nothing uprobe-specific; just the same pattern.

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

#include "bashspy.h"

char LICENSE[] SEC("license") = "GPL";

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

/* Track the prompt arg per TID so we can read the result on return.
 * For readline, the *return value* is what the user typed —
 * a malloc'd char* the caller frees.
 */

SEC("uretprobe//bin/bash:readline")
int BPF_KRETPROBE(on_readline_ret, const char *line)
{
    if (!line) return 0;

    struct event *e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
    if (!e) return 0;

    e->pid = bpf_get_current_pid_tgid() >> 32;
    bpf_get_current_comm(&e->comm, sizeof(e->comm));
    bpf_probe_read_user_str(&e->line, sizeof(e->line), line);

    bpf_ringbuf_submit(e, 0);
    return 0;
}

What’s new

• SEC("uretprobe//bin/bash:readline") — uretprobe (return) on bash’s readline. The kernel will hijack readline’s return address on entry and regain control at the trampoline, as the return section described. Adjust the path to your bash (which bash).
• BPF_KRETPROBE(on_readline_ret, const char *line) — for uretprobe, the first declared parameter after the function name is the return value, not an argument. (This is exactly why: at the return trap only PT_REGS_RC/rax survives — the argument registers are long gone.) Same as kretprobe.
• bpf_probe_read_user_str — user, not kernel. The pointer is in user memory.
• (const char *) cast in the parameter declaration — the macro will treat the return value as that type for size determination.

bashspy.c — userspace consumer

static int handle(void *ctx, void *data, size_t sz) {
    struct event *e = data;
    printf("[bash %u (%s)] %s\n", e->pid, e->comm, e->line);
    return 0;
}

Run

make
sudo ./bashspy &

# In another terminal, start a bash and type:
bash
echo hello
ls /tmp

Expected:

[bash 4001 (bash)] echo hello
[bash 4001 (bash)] ls /tmp

Don’t worry if your run doesn’t match this block line-for-line. The probe is system-wide, so you’ll also see the bash line you typed to launch the inner shell (the outer shell’s readline fired under a different PID), plus anything typed in any other interactive bash on the box. The PID will differ from 4001 — it’s arbitrary. And only interactive, readline-edited lines appear: commands fed via a script or a pipe never touch readline, so they won’t show up. Extra or missing lines here are normal, not a sign of a broken attach.

When you’re done, stop the spy and leave the test shell: run sudo kill %1 (or sudo pkill -f bashspy) in the first terminal to detach the uprobe, and exit the bash you spawned in the second terminal. Leaving bashspy running keeps the int3 breakpoint patched into every bash on the system — and an orphaned root daemon hanging around.

You’re now spying on every command typed into every bash on the system. For your own systems, treat this as a powerful debugging tool. For other people’s systems, this is the kind of thing that requires authorization.

What to break, in order

Break 1 — Path mismatch

SEC("uprobe//bin/bash:readline")

If your bash is at /usr/bin/bash (not /bin/bash), libbpf fails to attach:

libbpf: prog 'on_readline_ret': failed to attach: -ENOENT

The path has to resolve to the right inode — remember, the uprobe is registered against (inode, offset), so a wrong path means a wrong (or nonexistent) inode and the registration never happens. Fix: which bash and use the full path.

Break 2 — Stripped binary, missing symbol

Try a symbol that doesn’t exist in the table:

SEC("uprobe//bin/bash:internal_static_helper")

Resolution fails — libbpf can’t find internal_static_helper in .dynsym or .symtab. The job libbpf normally does for you is: search both symbol tables (dynsym first), look up the name, compute the file offset. When the name is gone (the binary is stripped, or the symbol is a private static that was never in .dynsym), you supply the offset yourself.

To see the fallback work, pick a symbol that does exist and grab its offset. /bin/bash is usually stripped of its .symtab, so objdump -t is empty — use the dynamic symbol table (.dynsym, which survives stripping) instead:

nm -D /bin/bash | grep -w readline
# 0000000000105e20 T readline

(objdump -T /bin/bash | grep -w readline shows the same thing — both read .dynsym.) That recorded value is a virtual address; for bash’s code segment it coincides with the file offset, so you can hand it to the kernel directly:

SEC("uprobe//bin/bash:0x105e20")

libbpf could have resolved readline by name — the offset form is exactly what you fall back to when the name isn’t available. (This 0x105e20 is bash-version-specific; always re-read it with nm -D for your binary rather than copying it verbatim.)

Break 3 — Use kernel-side _kernel_str on a user pointer

bpf_probe_read_kernel_str(&e->line, sizeof(e->line), line);  /* WRONG */

The helper returns negative and the destination is zeroed. On x86 the kernel-nofault path doesn’t even fault: copy_from_kernel_nofault_allowed rejects a userspace address by range (addr < TASK_SIZE_MAX), so strncpy_from_kernel_nofault returns -ERANGE and bpf_probe_read_kernel_str zeroes the buffer. Always _user for uprobe-derived pointers.

Break 4 — Convert uretprobe to uprobe

SEC("uprobe//bin/bash:readline")
int BPF_KPROBE(on_call, const char *prompt)
{
    /* prompt is the *argument* — what bash is asking */
}

Now you’re at entry, not return — a single int3 on readline’s first instruction, no return-address hijack. So you see prompt strings ("$ ", "PS1", etc.) read from rdi/PT_REGS_PARM1 instead of typed lines. To get the typed line, you need to be at return, where the function gives back its result in rax. That’s what uretprobe is for.

Break 5 — Multi-args with mixed types

SEC("uprobe//usr/lib/libssl.so.3:SSL_write")
int BPF_KPROBE(on_ssl_write, void *ssl, const void *buf, int num)
{
    char preview[32] = {0};
    bpf_probe_read_user(preview, sizeof(preview) - 1, buf);
    bpf_printk("SSL_write %d bytes: %s", num, preview);
    return 0;
}

You’re now reading plaintext that’s about to be encrypted — ssl from rdi, buf from rsi, num from rdx, exactly the System V ordering from Day 7. (Don’t deploy this on someone else’s system.) Note: the libssl path varies by distro and even by process. Find the library actually loaded by your target with cat /proc/<pid>/maps | grep -i ssl — commonly /usr/lib/x86_64-linux-gnu/libssl.so.3 on Debian/Ubuntu, /usr/lib/libssl.so.3 on Arch, /usr/lib64/libssl.so.3 on Fedora/RHEL — since a process may also bundle its own OpenSSL. Update the SEC string to match the path you find. (Because libssl.so.3 is a shared library, its .dynsym exports SSL_write even when stripped — no offset fallback needed.)

What to read in the kernel

• kernel/events/uprobes.c — overview. The function uprobe_register is the entry point. Note find_uprobe_rcu and set_swbp/set_orig_insn for the breakpoint patching mechanics.
• tools/lib/bpf/libbpf.c — search bpf_program__attach_uprobe. This is where libbpf opens the ELF, resolves symbols, and registers the uprobe via the kernel’s perf_event_open.
• tools/testing/selftests/bpf/progs/uprobe_multi.c — multi-uprobe (next day’s topic) examples.
• Documentation/trace/uprobetracer.rst — the kernel’s doc on uprobes from a tracing perspective.
• tools/lib/bpf/usdt.c — USDT support in libbpf. The .note.stapsdt parser is here.

External: Brendan Gregg’s USE method posts and bpftrace examples for uprobe usage patterns.

Bullet Points

• An executable isn’t copied into RAM — it’s mmap’d from a file, and its pages live once in the page cache, keyed by (inode, file-offset) and shared read-only/exec across every process running it.
• A uprobe is identified by (inode, offset), not a virtual address. That’s why it’s system-wide: the kernel walks every running and future process that maps the inode and patches each — no app changes, no restarts.
• Patching a userspace instruction uses copy-on-write. The kernel breaks COW to get a private writable page, then writes a 1-byte int3 (0xCC) — it never dirties the shared/on-disk page.
• The clobbered instruction is replayed out-of-line (XOL). The int3 stays patched; each thread single-steps its own saved copy in its own xol slot, then resumes after the original.
• Uretprobes hijack the return address. On entry the kernel saves the real return address on a per-task LIFO stack and redirects the return to a trampoline int3; at the return trap PT_REGS_RC holds the result.
• SEC("uprobe//path:symbol") for entry; SEC("uretprobe//...") for return.
• BPF_KPROBE/BPF_KRETPROBE macros work for uprobes too — same pt_regs ctx, args in rdi…r9, return in rax (Day 7).
• String args use bpf_probe_read_user_str (not _kernel).
• Two ELF symbol tables: .symtab (full, removed by strip) and .dynsym (dynamic, survives stripping). libbpf searches dynsym→symtab; when a symbol’s in neither, fall back to a literal file offset (nm -D / objdump -d).
• USDT probes are nops + ELF metadata, intentionally placed by app authors. Attached via SEC("usdt//path:provider:probe").

Check question

Why is bpf_probe_read_user_str necessary instead of just dereferencing the userspace string pointer directly?

Tomorrow

Day 11: multi-{u,k}probe — efficient attach to many functions at once. The newer batch-attach mechanism that scales to thousands of probes without thousands of perf_event_open syscalls.

Day 11 — Multi-probe: attach to many functions in one syscall

Today’s mission: trace every filesystem syscall (read, write, open, close, fsync, …) with one BPF program attached to all of them at once. Along the way, learn the third attach mechanism the kernel has — fprobe — that makes batch-attaching a thousand functions cheap, how a glob like vfs_* becomes a sorted array of addresses, and how one program tells its N probes apart. Total time: ~95 minutes.

The problem with one-by-one

Yesterday’s uprobes attach one function at a time. That works for a handful of probes. But what if you want to trace every syscall? Every function in the VFS layer? Every TCP entry point?

In the old model, you’d loop in userspace:

for (int i = 0; i < N; i++) {
    bpf_program__attach_kprobe(prog, false, names[i]);  /* one syscall each */
}

For N=200, that’s 200 perf_event_open syscalls, 200 kernel kprobe objects, 200 BPF links. Startup takes seconds. Memory grows linearly. Detach is similarly slow.

Multi-probe (added in kernel 5.18, commit 0dcac272540 for kprobe.multi) is the modern fix.

One syscall, one BPF link, N probes installed in one kernel-side batch.

But “one kernel-side batch” is doing a lot of work in that sentence, and the whole performance story of this chapter rests on how the batch is installed and run. Day 1 introduced fentry’s trampoline path and contrasted it with the older kprobe int3 trap; Day 6 added fexit. Today there’s a third attach mechanism, and it’s the one kprobe.multi rides on. Let’s name it before we lean on it.

What “fprobe-backed” actually means

Recall the two attach mechanisms Day 1 introduced (with fexit added in Day 6):

• kprobe patches the target instruction with an int3 software breakpoint. When the CPU hits it, it traps — a relatively expensive exception that vectors into the kprobe handler, which eventually calls your BPF program. One breakpoint per function.
• fentry/fexit (Days 1 and 6) use ftrace’s 5-byte NOP patch-site that every function carries at its entry, and route through a per-target BPF trampoline — a small chunk of generated code wired to one function. No trap; execution flows straight through. Cheaper per call, but one trampoline per attach target.

kprobe.multi is built on a third thing, also ftrace-based, called an fprobe.

The intuition first. Imagine you want a breakpoint on a thousand functions. The kprobe way is a thousand independent text-pokes — a thousand int3 bytes written into a thousand call sites, a thousand kernel objects to track. The fprobe way is one client that says to ftrace: “here is a sorted array of a thousand addresses; call me when execution reaches any of them.” ftrace already maintains the machinery to multiplex many call sites onto shared handler code (it’s how ftrace function tracing itself works). An fprobe is one customer of that machinery registered against a whole array at once.

Concretely, an fprobe is a single small struct:

/* include/linux/fprobe.h:62 */
struct fprobe {
    unsigned long       nmissed;          /* counter for missed events */
    unsigned int        flags;
    size_t              entry_data_size;
    fprobe_entry_cb     entry_handler;    /* called on function entry */
    fprobe_exit_cb      exit_handler;     /* called on function return */
    struct fprobe_hlist *hlist_array;     /* hash of the IPs this fprobe covers */
};

One struct fprobe, one entry_handler, an hlist_array of the addresses it covers. That’s why N functions install in one batch instead of N separate int3 patches.

fprobe is not fentry’s trampoline

It is tempting to assume fprobe reuses the per-target BPF trampoline Day 1 described. It does not, and the distinction matters. fprobe hangs off ftrace’s shared graph/ops machinery. You can see the fork in the source:

/* kernel/trace/fprobe.c:982 — register_fprobe_ips() */
if (fprobe_is_ftrace(fp))
    ret = fprobe_ftrace_add_ips(addrs, num);
else
    ret = fprobe_graph_add_ips(addrs, num);

Depending on whether the probe needs a return/exit handler, registration routes to ftrace’s ftrace_ops path or its function-graph (fgraph) path. Neither is a per-function BPF trampoline. The honest one-sentence statement is:

fprobe is ftrace-based like fentry — so it avoids the int3 trap — but it is ftrace_ops/fgraph, not a per-function BPF trampoline.

That distinction doesn’t weaken the performance argument; it sharpens it. Here is the real reason kprobe.multi beats a one-by-one kprobe loop, on both axes:

• Install is one ftrace operation over a sorted address array, not N separate int3 text-pokes. The kernel registers a single fprobe covering the whole addrs[] array (register_fprobe_ips(&link->fp, addrs, cnt)).
• At runtime there is no int3 trap. Execution reaches the shared ftrace handler directly — the exact same reason fentry beats kprobe in Day 1.

The ceiling, and the hard requirement

A single multi-attach is capped. The kernel defines:

/* kernel/trace/bpf_trace.c:44 */
#define MAX_KPROBE_MULTI_CNT (1U << 20)   /* ~1,048,576 */

and rejects anything larger up front in bpf_kprobe_multi_link_attach:

/* kernel/trace/bpf_trace.c — inside bpf_kprobe_multi_link_attach */
if (!cnt)
    return -EINVAL;
if (cnt > MAX_KPROBE_MULTI_CNT)
    return -E2BIG;

So the “1000 functions” number in this chapter has a real ceiling of ~1M addresses; over that you get -E2BIG. (And zero matches gets -EINVAL — hold that thought for Break 4.)

Finally, the whole mechanism is gated on a config option. kprobe.multi requires CONFIG_FPROBE=y. Without it, the entire attach entry point compiles down to a stub:

/* kernel/trace/bpf_trace.c:2890 — the #else /* !CONFIG_FPROBE */ branch */
int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
{
    return -EOPNOTSUPP;
}

That’s why, without CONFIG_FPROBE, attach fails outright with -EOPNOTSUPP rather than silently degrading to a slow loop. (The full prerequisite check is in the Run section below.)

From glob to addresses: who does what

SEC("kprobe.multi/vfs_*") hides a two-stage pipeline that is easy to blur into one step. The honest split is the opposite of the tempting guess: for a glob, userspace (libbpf) does both the glob expansion and the name→address resolution; the kernel just receives a ready-made array of addresses and registers it. Let’s walk both sides.

Userspace side (libbpf). Day 9 already taught you /proc/kallsyms: kptr_restrict zeroes the addresses for non-root, the file is not address-sorted, and you find a symbol by scanning for the nearest preceding entry. (We use exactly that fact for the lookup_ksym sketch later.) But for a SEC("kprobe.multi/glob") attach, libbpf may not need /proc/kallsyms at all — it depends on what tracefs exposes. The precedence is:

/* tools/lib/bpf/libbpf.c — inside bpf_program__attach_kprobe_multi_opts */
if (has_available_filter_functions_addrs())
    err = libbpf_available_kprobes_parse(&res);   /* modern: addr+name pairs */
else
    err = libbpf_available_kallsyms_parse(&res);   /* fallback: names + kallsyms */

On a modern kernel, tracefs exposes available_filter_functions_addrs — a file listing each instrumentable function as an address + name pair. libbpf_available_kprobes_parse reads addresses straight out of it, globs the names, and never opens /proc/kallsyms (this devbox has that file). On older kernels lacking it, libbpf falls back to libbpf_available_kallsyms_parse, which globs the names from available_filter_functions and then calls libbpf_kallsyms_parse (which fopens /proc/kallsyms) to turn those names into addresses. Either way the result is the same: libbpf produces a resolved res.addrs array in userspace. Using available_filter_functions (rather than raw kallsyms) for the name set matters: it already excludes notrace and inlined functions you could never attach to, so the glob can’t hand back a name that’s doomed to fail.

Kernel side. libbpf sets lopts.kprobe_multi.addrs = addrs and syms = NULL, so the kernel takes the address branch. It validates that exactly one of the two is set, then just copies the array in and moves on:

/* kernel/trace/bpf_trace.c — inside bpf_kprobe_multi_link_attach */
if (!!uaddrs == !!usyms)
    return -EINVAL;            /* exactly one of addrs[] or syms[] */
...
if (uaddrs) {                  /* glob path: addresses already resolved */
    if (copy_from_user(addrs, uaddrs, size))
        return -EFAULT;
} else {                       /* explicit names path: resolve in-kernel */
    err = copy_user_syms(&us, usyms, cnt);
    sort_r(us.syms, cnt, ...);
    err = ftrace_lookup_symbols(us.syms, cnt, addrs);
}

For a glob the kernel never sees names: it copy_from_users the address array and goes straight to register_fprobe_ips. The in-kernel name→address resolution (copy_user_syms → sort_r → ftrace_lookup_symbols) is the other branch — taken only when the caller passes an explicit opts.syms[] list or a single exact (wildcard-free) name. That path is where the chapter’s single-O(N)-pass-over-kallsyms story actually lives; see Break 1.

The single-pass detail (the usyms branch only). It is easy to assume ftrace_lookup_symbols is N independent log-N lookups into kallsyms. It is actually a single O(N) pass:

/* kernel/trace/ftrace.c:9262 — ftrace_lookup_symbols doc block */
/* ... kallsyms_on_each_symbol() with binary search into the sorted input
 * array.
 * Returns: 0 if all provided symbols are found, -ESRCH otherwise.
 */

The kernel walks kallsyms once (kallsyms_on_each_symbol), and for each symbol it sees, does a bsearch into your already-sorted name array. That’s why the names get sorted first: so the single pass can binary-search them. One sweep over kallsyms, not N sweeps. And if even one name doesn’t resolve, the whole attach fails with -ESRCH. Again, this only runs for the explicit-names path, not for a glob.

So the failure modes split by path:

• Glob, zero matches (vfs_xyz_nope* expands to nothing): caught in userspace — libbpf_available_kprobes_parse/libbpf_available_kallsyms_parse return -ENOENT when res.cnt == 0, before any syscall. The kernel’s if (!cnt) return -EINVAL never runs (it only fires if a caller explicitly passes cnt == 0 through the opts API). This is Break 4.
• Explicit names, one doesn’t resolve: the kernel’s ftrace_lookup_symbols returns -ESRCH. Only the usyms branch can hit this.

How to know which probe fired

When 50 functions all run the same BPF program, you need a way to tell them apart. The ctx (the saved registers) has the same shape for every target, so it can’t distinguish them — the only thing that differs is which function’s entry you’re sitting at. That’s what bpf_get_func_ip(ctx) gives you, and it’s worth understanding where that IP actually comes from, because it’s the load-bearing dispatch primitive for the whole chapter.

Where the IP comes from. When a target fires, fprobe calls the entry handler with the fentry_ip — the instrumented call site. The kprobe.multi handler converts that to the actual function entry address and stashes it in the per-invocation run context:

/* kernel/trace/bpf_trace.c:2592 — kprobe_multi_link_handler */
err = kprobe_multi_link_prog_run(link, ftrace_get_entry_ip(fentry_ip),
                                 fregs, false, data);

/* kernel/trace/bpf_trace.c:2319 — ftrace_get_entry_ip() */
unsigned long ip = ftrace_get_symaddr(fentry_ip);
return ip ? : fentry_ip;       /* resolved symbol address, or the call site */

So by the time your program runs, the resolved function entry address is already sitting in the run context.

bpf_get_func_ip is not one helper. The verifier swaps in a program-type-specific implementation depending on what kind of probe you are. For kprobe.multi it resolves to bpf_get_func_ip_kprobe_multi:

/* kernel/trace/bpf_trace.c:1080 */
BPF_CALL_1(bpf_get_func_ip_kprobe_multi, struct pt_regs *, regs)
{
    return bpf_kprobe_multi_entry_ip(current->bpf_ctx);
}

It just reads the entry IP that the handler already stashed in current->bpf_ctx. The selection happens in the verifier when it sees is_kprobe_multi(prog):

/* kernel/trace/bpf_trace.c:1327 */
case BPF_FUNC_get_func_ip:
    if (is_kprobe_multi(prog))
        return &bpf_get_func_ip_proto_kprobe_multi;

That’s why the helper is cheap: it’s a context read, not a symbol lookup at runtime. The expensive part (resolving the address) happened once, in the handler.

You typically use the returned IP as a key in a hash map: IP → count, or IP → behavior. Strip out bpf_get_func_ip (Break 3) and every one of the N probes increments the same counter, because nothing else in the program distinguishes them — attribution is gone.

One link owns all N attachments

Day 1 introduced bpf_link (BPF_LINK_CREATE wires a program to an attach point); Day 10 showed that a probe lives until you close the link’s FD. Nothing new to teach there — just one new point worth a sentence.

The same single link object covers all N functions:

/* kernel/trace/bpf_trace.c:2291 */
struct bpf_kprobe_multi_link {
    struct bpf_link link;
    struct fprobe   fp;          /* the one fprobe covering all N */
    unsigned long  *addrs;       /* the N resolved addresses */
    u64            *cookies;
    u32             cnt;         /* N */
    ...
};

/* kernel/trace/bpf_trace.c:2828 */
bpf_link_init(&link->link, BPF_LINK_TYPE_KPROBE_MULTI, ...);

One struct bpf_kprobe_multi_link, holding one struct fprobe, the addrs/cookies arrays, and the count. Closing its one FD detaches all N — versus the N separate links the one-by-one loop creates. That’s the whole “detach is one operation” claim, made concrete.

What the VFS layer is (and why vfs_* is the right glob)

The lab globs vfs_*. Day 6 first used vfs_read as an example target and Day 7 reused it, but no chapter has said what the VFS is — and you shouldn’t have to take “glob vfs_* to catch all filesystem activity” on faith.

The Virtual File System is the kernel’s filesystem-agnostic layer. Every read(), write(), open(), close() syscall funnels through a vfs_* entry point — vfs_read (fs/read_write.c:554), vfs_write (fs/read_write.c:668), vfs_open (fs/open.c:1074), and friends — before the kernel dispatches to the concrete filesystem driver (ext4, xfs, btrfs, …) that actually backs the file. The VFS is the common chokepoint; the concrete FS is what sits below it.

That single fact is why hooking vfs_* with one glob is the right aggregation point: it captures all filesystem activity in one place, no matter which filesystem each file lives on. You don’t need VFS internals beyond this — the companion linux-net book and general kernel background go deeper; here you only need to know vfs_* are the shared filesystem entry points.

There are no Dumb Questions

Q: Is multi-kprobe just syntactic sugar over a loop?

A: Functionally similar; operationally very different. The kernel-side install path is shared (one fprobe registered via ftrace against a list of addresses), the BPF link is a single object, and there’s no int3 trap at runtime. For 1000 functions, multi-kprobe attaches in ~10 ms; the loop version takes seconds.

Q: Can I use multi-probe with a wildcard pattern?

A: Yes. libbpf supports glob patterns in attach_kprobe_multi_opts.syms. tcp_* matches every TCP function. Internally libbpf reads tracefs available_filter_functions_addrs (or available_filter_functions plus /proc/kallsyms on older kernels) and resolves the glob to addresses in userspace, then hands the kernel a ready-made address array — the kernel just copy_from_users it and registers the fprobe. (The in-kernel name→address resolution only runs when you pass an explicit syms[] list instead of a glob.)

Q: Does multi-probe exist for fentry?

A: No — there’s no fentry.multi/fexit.multi. fentry/fexit build one trampoline per attach target, with no batched multi-attach variant in libbpf or the kernel. If you need to hook many functions at once with low per-call overhead, use kprobe.multi (which is fprobe-backed and very cheap) and read the return value via the kprobe ctx.

Q: Multi-uprobe?

A: Yes — added in 6.6. SEC("uprobe.multi/..."). Same idea for userspace targets.

The lab

multi.bpf.c

This listing is included from the file the lab build and CI compile:

#include “vmlinux.h” #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h>

char LICENSE[] SEC(“license”) = “GPL”;

struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(max_entries, 4096); __type(key, __u64); /* function ip / __type(value, __u64); / count */ } hits SEC(“.maps”);

SEC(“kprobe.multi/vfs_*”) int BPF_KPROBE(on_any_vfs) { __u64 ip = bpf_get_func_ip(ctx); __u64 *c = bpf_map_lookup_elem(&hits, &ip); if (c) { __sync_fetch_and_add(c, 1); } else { __u64 one = 1; bpf_map_update_elem(&hits, &ip, &one, BPF_NOEXIST); } return 0; }

SEC("kprobe.multi/vfs_*") — the suffix after kprobe.multi/ is a glob. libbpf expands it (via tracefs available_filter_functions, as we saw) into maybe 50 function names starting with vfs_, the kernel resolves those to addresses, and bpf_get_func_ip(ctx) returns the entry IP of whichever one fired — the key the hash map aggregates on.

multi.c — userspace

struct multi_bpf *skel = multi_bpf__open_and_load();
multi_bpf__attach(skel);

while (!exiting) {
    sleep(2);
    /* iterate hits map; resolve ip via /proc/kallsyms */
    int fd = bpf_map__fd(skel->maps.hits);
    __u64 key = 0, next, val;
    while (bpf_map_get_next_key(fd, &key, &next) == 0) {
        bpf_map_lookup_elem(fd, &next, &val);
        printf("%-30s %llu\n", lookup_ksym(next), val);
        key = next;
    }
}

lookup_ksym(addr) is a sketch, not supplied code — it’s a few-line function you write that scans /proc/kallsyms for the closest symbol at or below addr: parse each addr type name line, keep the entry with the largest address <= addr. (This is exactly the nearest-preceding-symbol scan Day 9 taught — and note it reads /proc/kallsyms, the symbolization file, not the available_filter_functions file the glob used.) Without it the program won’t link. To skip writing a symbolizer, print the raw IP with %llx instead and pipe each value through grep <hex> /proc/kallsyms by hand. Note /proc/kallsyms only exposes real addresses to root (kptr_restrict zeroes them otherwise), so run the binary with sudo.

Run

Prerequisite: kprobe.multi is fprobe-backed, so the kernel must be built with CONFIG_FPROBE=y (kernel >= 5.18), plus CONFIG_DEBUG_INFO_BTF=y for the vmlinux.h include. Check with grep -E 'CONFIG_FPROBE|CONFIG_DEBUG_INFO_BTF' /boot/config-$(uname -r) (or zgrep on /proc/config.gz). Without CONFIG_FPROBE the attach fails outright — bpf_kprobe_multi_link_attach is the -EOPNOTSUPP stub we saw above.

make
sudo ./multi &
# Generate work in another terminal:
find /etc -type f | xargs cat > /dev/null

Expected (assuming you wrote the lookup_ksym sketch above — otherwise the left column is raw hex IPs):

vfs_read       12453
vfs_open        2914
vfs_statx       1822
vfs_getattr_nosec  2914
...

Row order is arbitrary: the userspace loop walks the map with bpf_map_get_next_key, which returns keys in hash order, not ranked by count — so your rows may appear shuffled (the counts above aren’t sorted either). Only the per-function counts matter; pipe through sort -k2 -rn if you want them ranked.

You attached to ~50 functions in one syscall, watched all of them, and aggregated by function — without writing 50 separate handlers.

Same lesson as a one-liner

If you’d rather not assemble the C glue (in particular the lookup_ksym symbolizer), bpftrace gives you the whole lesson in one copy-pasteable program. The vfs_* glob is expanded and attached via kprobe.multi — the same path as SEC("kprobe.multi/vfs_*") in multi.bpf.c — and the func builtin plays the role of bpf_get_func_ip, telling you which of the vfs_* functions fired:

# Terminal 1 — one program, many functions, attributed by func():
sudo bpftrace -e 'kprobe:vfs_* { @[func] = count(); } interval:s:5 { exit(); }'
# Terminal 2 — generate VFS load:
find /etc -type f | xargs cat > /dev/null

Expected (counts vary run to run; the highest-traffic VFS paths dominate, and bpftrace prints @-maps sorted ascending by value):

Attached 78 probes

@[vfs_write]: 524
@[vfs_fstatat]: 1744
@[vfs_statx]: 2823
@[vfs_open]: 4719
@[vfs_fstat]: 4897
@[vfs_getattr_nosec]: 7742
@[vfs_read]: 20960

bpftrace resolves the names directly via func, so this version sidesteps the lookup_ksym glue entirely.

What to break, in order

Break 1 — Specific list instead of glob

SEC("kprobe.multi")
int BPF_KPROBE(p) { ... }

In userspace:

LIBBPF_OPTS(bpf_kprobe_multi_opts, opts);
const char *syms[] = {"vfs_read", "vfs_write", "vfs_open"};
opts.syms = syms;
opts.cnt = 3;
bpf_program__attach_kprobe_multi_opts(skel->progs.p, NULL, &opts);

This gives you precise control over which functions to attach. Use when a glob would over-match. (libbpf still feeds these names through copy_user_syms → sort_r → ftrace_lookup_symbols on the kernel side — the same resolution path, you’ve just skipped the glob expansion.)

Break 2 — Mixing offsets

const __u64 addrs[] = {0xffffffff812a4000, ...};
opts.addrs = addrs;     /* numeric addresses */
opts.cnt = N;

You can pass raw kernel addresses too. Useful when symbols are aliased or you have addresses from another source (perf, kallsyms parsed yourself). With opts.addrs the kernel skips name resolution entirely — it just copy_from_user(addrs, uaddrs) into the link’s array and goes straight to register_fprobe_ips. No ftrace_lookup_symbols step.

Break 3 — Forget bpf_get_func_ip

Without it, every probe just increments the same counter — you lose per-function attribution. Run, then realize you can’t tell which vfs_* was called. The reason is exactly what the dispatch section showed: the ctx (pt_regs) is identical in shape across all N targets, so the entry IP is the only thing distinguishing one invocation from another. Lesson: always identify the source when one program serves many probes.

Break 4 — Multi-probe on a function that doesn’t exist

SEC("kprobe.multi/this_function_does_not_exist*")

Glob expands to zero matches. libbpf fails attach with -ENOENT (the kernel side would reject cnt == 0 with -EINVAL; libbpf surfaces the empty resolution as -ENOENT). Friendly: silent acceptance with zero matches would mask bugs.

What to read in the kernel

• kernel/trace/bpf_trace.c — search bpf_kprobe_multi_link_attach. The function that handles the new link type.
• kernel/bpf/syscall.c — search BPF_TRACE_KPROBE_MULTI. The syscall dispatch.
• tools/lib/bpf/libbpf.c — search bpf_program__attach_kprobe_multi_opts. The userspace API.
• tools/testing/selftests/bpf/progs/kprobe_multi.c — official test/examples.

Bullet Points

• Multi-probe attaches one BPF program to many functions in a single syscall (~10ms for 1000 vs seconds for one-by-one).
• It’s fprobe-backed: one struct fprobe registered via ftrace against a sorted addr[] array (register_fprobe_ips). fprobe is ftrace-based like fentry — so no int3 trap — but it’s ftrace_ops/fgraph, not a per-function BPF trampoline. Cheap to install (one ftrace op, not N text-pokes) and cheap at runtime (no trap). Capped at MAX_KPROBE_MULTI_CNT = 1<<20; CONFIG_FPROBE=y required or attach returns -EOPNOTSUPP.
• Glob → addresses, resolved in userspace. For a glob, libbpf does both the expansion and the name→address resolution: it reads tracefs available_filter_functions_addrs (or available_filter_functions + /proc/kallsyms on older kernels), produces an address array, and the kernel just copy_from_users it and registers the fprobe — no in-kernel name resolution. The sort_r names + ftrace_lookup_symbols single O(N) pass over kallsyms (bsearch into the sorted input) runs only for an explicit syms[] list or an exact wildcard-free name.
• SEC("kprobe.multi/glob") for glob attach; or pass an explicit list via bpf_kprobe_multi_opts.syms, or raw addresses via .addrs (which skips name resolution).
• bpf_get_func_ip(ctx) lets the program know which function fired. It’s a per-program-type helper (bpf_get_func_ip_kprobe_multi) that just reads the entry IP the handler stashed in current->bpf_ctx — cheap, no runtime symbol lookup. The IP is the only discriminator; without it, all N probes increment one counter.
• One bpf_link (struct bpf_kprobe_multi_link, holding one fp and the addrs/cookies/cnt) owns all N attachments; closing its FD detaches all N.
• The VFS is the kernel’s filesystem-agnostic layer; every read/write/open/close syscall funnels through a vfs_* entry before dispatching to the concrete FS — which is why vfs_* is the right one-glob aggregation point.
• Variants: kprobe.multi, kretprobe.multi, uprobe.multi (6.6+). There is no fentry.multi/fexit.multi.

Check question

You attach kprobe.multi/tcp_* and observe bpf_get_func_ip returning some IPs that don’t match what cat /proc/kallsyms | grep tcp_ shows. What might cause this?

Tomorrow

Day 12: sleepable BPF programs. The constraint that’s been quietly limiting you (no helpers that fault) and how to relax it.

Day 12 — Sleepable BPF: programs that can fault

Today’s mission: trace openat and read the path string from userspace memory — something a normal BPF program literally cannot do. Along the way, learn what a page fault actually is (demand paging, minor vs major faults), what it means for kernel code to “sleep,” which contexts forbid sleeping and which allow it, and the real synchronization mechanism that protects sleepable programs (it is not SRCU — the kernel changed, and a lot of docs are stale). Total time: ~100 minutes.

The constraint you’ve been living under

Every BPF program you’ve written so far runs in atomic context. The kernel can’t schedule the CPU away mid-execution; the BPF program must run start-to-finish without yielding. This is what makes BPF so cheap (no context switches, no scheduling) and what gates all the other guarantees.

You met the consequences of that on earlier days without us naming the cause. On Day 2 the hash map’s bucket lock had to be a raw spinlock specifically “because BPF programs run in atomic contexts where you cannot take a sleeping lock.” On Day 5 the Verifier insisted your program terminate in bounded time because it runs where preemption and IRQs may be off. Today we finally turn that constraint over and look underneath it — and then we lift it.

A direct consequence of atomic context: any helper that might page-fault is forbidden. Hitting an unmapped userspace page would require resolving the fault — which can require scheduling the thread away, which atomic context can’t do.

Helpers in the forbidden list: bpf_copy_from_user (and its task variant bpf_copy_from_user_task). Before sleepable BPF existed, the answer was “you can’t do that from BPF.”

Sleepable BPF programs lift the restriction.

First, what does “sleep” even mean in the kernel?

The word “sleep” is doing a lot of load-bearing work today, and it does not mean “wait for a timer” or “delay.” In kernel terms, to sleep is to voluntarily block: the current thread is put on a wait queue, the scheduler picks some other runnable task and switches to it, and the original thread resumes later when whatever it was waiting for is ready. The CPU is not spinning — it goes and does useful work for someone else in the meantime.

The kernel even has a built-in annotation for “this function may block”:

/* include/linux/kernel.h:79 — might_sleep() is an annotation, not an action */
/* might_sleep - annotation for functions that can sleep */
# define might_sleep() \
    do { __might_sleep(__FILE__, __LINE__); might_resched(); } while (0)

might_sleep() (include/linux/kernel.h:90, with __might_sleep declared at :74) is a debugging tripwire: drop it into a function, and if that function is ever called from a context where sleeping is illegal, the kernel screams. Hold that thought — we’ll see the sleepable BPF trampoline call its faulting cousin might_fault() in a minute.

Faultable vs non-faultable contexts

So why can’t every kernel code path sleep? Because in some contexts there is no task to safely block, and blocking would deadlock or stall the CPU:

• Interrupt handlers and softirqs. A softirq — the deferred-interrupt context where packet-processing program types like XDP run — is not a schedulable entity. There’s no “thread” the scheduler can park and resume — so it cannot sleep. This is exactly why XDP can never be made sleepable.
• Code holding a spinlock, or with preemption explicitly disabled. Sleeping here would either deadlock (another CPU spinning forever on the lock you’re holding) or strand the CPU.

These are non-faultable / atomic contexts. Everything you’ve attached to so far — kprobes, XDP, tc, raw tracepoints — lives here.

• Ordinary process/thread context with preemption allowed is faultable. Syscall entry, most LSM hooks (Linux Security Module callbacks), BPF iterators. Here the current thread is a real schedulable task: it can be descheduled and resumed. So a major fault — or any helper that might sleep — is safe.

That last set is precisely the set of attach points that can be made sleepable. Keep the picture simple: sleepable-eligible = faultable = “there’s a real thread here we can safely put to sleep and wake up later.”

One subtlety worth a single sentence: even a sleepable BPF program runs under migrate_disable(), not full preemption — so it may sleep, but it stays pinned to the CPU it started on (so per-CPU map access stays correct). You’ll see that in the trampoline code below.

What “sleepable” actually means

The program is allowed to be in contexts where the kernel can schedule the current thread away. That means:

• It runs from a faultable code path (typically a syscall entry, an LSM hook, or an iterator).
• It can call helpers that may page-fault.
• It can take longer (microseconds — or, on a real fault, milliseconds — rather than nanoseconds) without breaking timing assumptions.
• It’s protected by a different RCU variant — RCU Tasks Trace — that allows readers to sleep. (Older write-ups call it SRCU. That’s wrong for modern kernels; we fix it properly below.)

You opt in with the .s/ suffix on the SEC name:

SEC("fentry.s/__x64_sys_openat")    /* sleepable fentry */
SEC("lsm.s/file_open")               /* sleepable LSM hook */
SEC("iter.s/task")                   /* sleepable iterator */

The Verifier tracks per-program sleepability. A sleepable program gets access to the wider helper set; a non-sleepable program calling a sleepable-only helper is rejected.

What a page fault actually IS

Before we can talk about why bpf_copy_from_user might sleep, you need to know what a page fault is and why a perfectly valid pointer can point at memory that isn’t there. This is the concept the whole chapter rests on, so we’ll build it from the ground up.

Demand paging: a valid pointer with no physical page behind it

When a process allocates memory — malloc, a growing stack, a freshly mmap’d region — the kernel records a VMA (virtual memory area): “addresses X through Y are legally yours.” But it does not immediately go find physical RAM to back every one of those addresses. That would be wasteful — most programs touch only a fraction of what they reserve. Instead the kernel waits until the program actually touches an address, and only then finds a physical page for it. This laziness is called demand paging.

The consequence is the thing that surprises everyone the first time: a pointer can be valid — it lands inside a mapped VMA, the kernel agrees you’re allowed to read it — and yet have no resident physical page behind it right now. “Valid address” and “address backed by RAM” are two different questions.

How does the hardware tell the difference? Through the page table. The CPU’s MMU translates every virtual address to a physical one by walking page-table entries (PTEs). If the PTE for an address says “not present,” the MMU can’t complete the translation and raises a page fault — a CPU trap into the kernel’s fault handler.

The handler’s decision, and minor vs major faults

The fault handler’s first job is to decide: is this a legal access into a real VMA, or a bug?

• Illegal (the address isn’t in any VMA — a wild pointer): deliver SIGSEGV. This is the segfault you know.
• Legal (inside a VMA, just not yet backed): fix it up — allocate a page, zero-fill it, copy-on-write it, or read it back from disk/swap — then install the PTE and let the faulting instruction re-run as if nothing happened.

When the access is legal, there are two very different costs, and the distinction is the entire reason sleepable BPF exists:

• Minor fault — the data is already in RAM; the kernel just needs to install a PTE pointing at it. This happens for a shared page, the shared zero page, or a page-cache hit (the file’s bytes are already cached). Fast, and the thread never blocks.
• Major fault — the page must be fetched from disk or swap. That’s I/O, which takes essentially forever in CPU terms. The faulting thread is descheduled (put to sleep) until the I/O completes, then woken and resumed. This is the case that requires a sleepable context.

On x86, the userspace fault entry is do_user_addr_fault (arch/x86/mm/fault.c:1207). Read two lines in it and the minor/major split jumps out: if (fault & VM_FAULT_MAJOR) (fault.c:1343) is the disk/swap branch, and if (unlikely(fault & VM_FAULT_RETRY)) (fault.c:1408) is the “we had to drop the mmap lock and may block” branch. Both route through the core resolver handle_mm_fault (mm/memory.c:6699). VM_FAULT_RETRY — the “we had to drop the lock and may block” signal — is returned by deeper fault helpers such as vmf_can_call_fault (mm/memory.c:3800) and __vmf_anon_prepare (mm/memory.c:3827) and propagated out through handle_mm_fault.

Tying it to the helper

Here’s why this matters for us. bpf_copy_from_user is, underneath, an ordinary copy_from_user:

/* kernel/bpf/helpers.c:659 — bpf_copy_from_user body */
int ret = copy_from_user(dst, user_ptr, size);
if (unlikely(ret)) {
    memset(dst, 0, size);   /* on failure, zero the dst... */
    ret = -EFAULT;          /* ...and report EFAULT */
}

copy_from_user touches the user address — and if the page isn’t present, it goes through the exact do_user_addr_fault → handle_mm_fault machinery above. If that’s a minor fault, the copy completes in microseconds and nobody sleeps. If it’s a major fault, the thread blocks until the page is read in. A non-sleepable program cannot be allowed anywhere near that second path, which is why the helper is gated. The helper even advertises the risk to the Verifier:

/* kernel/bpf/helpers.c:675 — the per-helper flag the Verifier checks */
.might_sleep = true,

Why a sleepable program is safe to detach: RCU Tasks Trace (not SRCU)

There’s one more piece. If a sleepable BPF program can sit suspended in the middle of a major fault for milliseconds, how does the kernel safely free that program when you detach it? It must guarantee no invocation is still in flight before it reclaims the program’s memory — otherwise it’d be freeing code that’s mid-execution.

You already know the shape of this answer from Day 2’s RCU section: readers run inside a read-side critical section, writers wait for a grace period (until all in-flight readers finish), and freeing is deferred so memory can’t vanish under a reader. That’s the same idea here — only the flavor of RCU is different, because classic RCU readers are not allowed to sleep (a sleeping reader would pin the grace period forever and starve writers). A sleepable BPF program obviously can sleep, so classic RCU won’t do.

The variant designed for exactly this is RCU Tasks Trace (a.k.a. tasks-trace RCU). Its read-side critical sections may block, and its writer-side grace period waits until every CPU has passed through a point where no tasks-trace reader is active.

You can see it in the trampoline. The wrapper that runs a sleepable program does not take an SRCU lock — it takes the tasks-trace read lock and disables migration (so it stays CPU-pinned even while it can fault):

/* kernel/bpf/trampoline.c:1250 — __bpf_prog_enter_sleepable() */
rcu_read_lock_trace();
migrate_disable();
might_fault();          /* "I might take a fault / sleep here" */

The recursion-aware pair __bpf_prog_enter_sleepable_recur / __bpf_prog_exit_sleepable_recur (trampoline.c:1221–:1247) bracket the program with rcu_read_lock_trace() / rcu_read_unlock_trace() the same way. And the kernel spells out the contrast in a comment — sleepable uses trace RCU, normal uses classic RCU:

/* kernel/bpf/trampoline.c:518 */
/* rcu_read_lock_trace to protect sleepable bpf progs
 * rcu_read_lock to protect normal bpf progs
 */

On detach, the kernel waits out a tasks-trace grace period before freeing — synchronize_rcu_tasks_trace() is the call, named explicitly in the syscall-side comment (kernel/bpf/syscall.c:157: “For sleepable BPF programs, synchronize_rcu_tasks_trace() should be used to wait for the completions of these programs”), with the fentry trampoline image freed after a tasks-trace grace period via call_rcu_tasks_trace (kernel/bpf/trampoline.c:562), which then chains call_rcu_tasks to drain the trampoline asm and normal progs. So the intuition you’d expect — “wait for a grace period before freeing so no in-flight invocation is mid-execution” — is exactly right. Only the RCU flavor name matters: it’s RCU Tasks Trace, not SRCU.

There are no Dumb Questions

Q: Why isn’t every BPF program sleepable by default?

A: Cost and reach. Non-sleepable programs use cheap classic RCU (~ns to enter/exit) and can attach to any context including IRQ handlers and softirqs. Sleepable programs use RCU Tasks Trace (more cost) and can only attach to faultable contexts (essentially syscall entries, LSM hooks, certain trampolines). Most tracers don’t need to fault, so non-sleepable is the right default.

Q: Can I make my XDP program sleepable?

A: No. XDP runs in softirq context — a non-faultable context by design (a softirq isn’t a schedulable thread, so there’s nothing to put to sleep). There’s no SEC("xdp.s/..."). If you need to do something faulting in response to a packet, send the metadata to userspace and do the work there.

Q: My fentry program needs to read a userspace string. Without sleepable, what are my options?

A: Use bpf_probe_read_user_str instead — it doesn’t fault on missing pages, it just returns -EFAULT. You get the data when the page happens to be in memory, and silently miss when not. For most observability use cases that’s acceptable. For correctness-critical reads, use sleepable.

The lab

openat_path.bpf.c

The producer and consumer share the event layout through one header:

#ifndef PRACTICAL_EBPF_OPENAT_PATH_H #define PRACTICAL_EBPF_OPENAT_PATH_H

struct event { unsigned int pid; char comm[16]; char path[256]; };

#endif

and the sleepable program is included from the source the lab build and CI compile (__user is defined away for the BPF target — it is a sparse annotation vmlinux.h does not carry):

#include “vmlinux.h” #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> #include <bpf/bpf_core_read.h>

#include “openat_path.h”

/* vmlinux.h does not carry the __user sparse annotation. Define it away so

• the syscall-argument pointer keeps its documentary __user tag and still
• compiles under -target bpf. */ #ifndef __user #define __user #endif

char LICENSE[] SEC(“license”) = “GPL”;

struct { __uint(type, BPF_MAP_TYPE_RINGBUF); __uint(max_entries, 256 * 1024); } rb SEC(“.maps”);

/* Sleepable fentry — the ‘.s/’ suffix gives access to bpf_copy_from_user. */ SEC(“fentry.s/__x64_sys_openat”) int BPF_PROG(on_openat, struct pt_regs *regs) { struct event *e; const char __user *upath; long ret;

/* The syscall wrapper passes pt_regs; PARM2 is the path string. */
upath = (const char __user *)PT_REGS_PARM2_CORE_SYSCALL(regs);
if (!upath)
    return 0;

e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
if (!e)
    return 0;

e->pid = bpf_get_current_pid_tgid() >> 32;
bpf_get_current_comm(&e->comm, sizeof(e->comm));

/* This is the magic — only allowed in sleepable programs: */
ret = bpf_copy_from_user(e->path, sizeof(e->path) - 1, upath);
if (ret < 0)
    e->path[0] = 0;   /* couldn't read; emit empty */

bpf_ringbuf_submit(e, 0);
return 0;

}

Why the syscall hook takes a single struct pt_regs *

There’s a snag worth pausing on. On Day 7 you learned that an fentry program on an ordinary kernel function gets the function’s arguments as typed kernel objects — BPF_PROG(on_x, struct file *f, int flags) just works, because the BTF signature says so. You also met pt_regs there as a saved register snapshot (for kprobes), and saw that each program type has its own ctx layout. So why, for __x64_sys_openat, are we suddenly back to a raw struct pt_regs * and a PARM2 macro instead of typed args?

Because a modern x86-64 syscall doesn’t have the signature you’d expect. SYSCALL_DEFINE4(openat, ...) (confirmed at fs/open.c:1381, where the 2nd logical arg filename is indeed const char __user *) expands into a thin wrapper:

long __x64_sys_openat(const struct pt_regs *regs);

The wrapper’s job is to pull the individual syscall arguments out of the saved register frame and hand them to the real worker (__do_sys_openat). So at the fentry attach point, __x64_sys_openat’s BTF signature genuinely has exactly one argument: a struct pt_regs * — not the four logical openat args. You can’t write BPF_PROG(on_openat, int dfd, const char *path, ...); the type information simply isn’t there. You take regs and decode it yourself.

PT_REGS_PARM2_CORE_SYSCALL(regs) is the accessor for “the 2nd syscall argument” under this wrapper convention. The _CORE_ variant is CO-RE-aware — it reads the right register field through BPF_CORE_READ, so it relocates correctly across kernels and arches (tools/lib/bpf/bpf_tracing.h:545; the plain PT_REGS_PARM2_SYSCALL at :544 is a direct field read). openat(dfd, path, flags, mode) → PARM2 is path. On other arches the wrapper prefix changes (__arm64_sys_openat), which is exactly why you want the CO-RE form.

Back to the program — what’s new

• SEC("fentry.s/__x64_sys_openat") — the .s/ makes it sleepable.
• __x64_sys_openat is the kernel’s system-call entry on x86-64. As just explained, it receives a single struct pt_regs * (the wrapper unpacks the logical args from the trap frame). On other arches the prefix differs (__arm64_sys_openat).
• PT_REGS_PARM2_CORE_SYSCALL(regs) — the CO-RE-aware macro that reads the second syscall argument across architectures. The path string is the second argument to openat(dirfd, path, flags, mode) (after dirfd).
• bpf_copy_from_user — the helper that may fault. Returns 0 on success, negative errno on failure. Only callable from sleepable programs — try it from a non-sleepable program and the Verifier rejects with sleepable helper bpf_copy_from_user#148 in <context>, where <context> names the non-sleepable context it found. (Func id 148 is fixed in the UAPI: FN(copy_from_user, 148, ...) at include/uapi/linux/bpf.h:6053.)

openat_path.c — userspace

Reuses the exact ringbuf consumer skeleton from Day 01 (the handle callback plus the ring_buffer__poll loop in main). Only the print line changes — it pulls pid, comm, and path off the event so the output matches the Run section below:

static int handle(void *ctx, void *data, size_t sz) {
    struct event *e = data;
    printf("PID %u (%s) opened %s\n", e->pid, e->comm, e->path);
    return 0;
}

Run

make
sudo ./openat_path &
# In another terminal:
ls /etc
cat /etc/passwd

Expected:

PID 4001 (ls) opened /etc
PID 4002 (cat) opened /etc/passwd

You read userspace memory from the kernel, possibly faulting if the page wasn’t resident, all without crashing.

One honest caveat: on a normal box the path string the caller just constructed is already page-resident (the syscall’s caller literally just wrote those bytes — a guaranteed minor fault at worst, usually no fault), so every bpf_copy_from_user returns 0 and no sleep is actually taken here. What sleepability buys you is the guarantee that if the page were missing — the major-fault case from the demand-paging section — the helper would safely sleep and page it in, whereas a non-sleepable program could not call the helper at all. To actually witness the fault path you would have to evict the page (e.g. madvise(MADV_DONTNEED) on the buffer) before the syscall. Printing ret from the consumer makes success-vs--EFAULT visible if you want to confirm.

What to break, in order

Break 1 — Drop the .s/ suffix

SEC("fentry/__x64_sys_openat")

Verifier rejects:

sleepable helper bpf_copy_from_user#148 in <context>

(<context> is filled in with the non-sleepable context the Verifier found — e.g., the program type or attach point.) This is the exact verbose(env, "sleepable helper %s#%d in %s\n", ...) string at kernel/bpf/verifier.c:10331, emitted from the if (fn->might_sleep && !in_sleepable_context(env)) check at :10330. Lesson: .s/ is what unlocks the sleepable helpers. Without it, the call is forbidden at verification time.

Break 2 — Try bpf_copy_from_user in an XDP program

SEC("xdp")
int xdp_prog(struct xdp_md *ctx) {
    char buf[16];
    bpf_copy_from_user(buf, sizeof(buf), (void *)0x12345678);
    return XDP_PASS;
}

Same rejection — helper not allowed. XDP can’t be made sleepable; it runs in softirq context, which (as we established) isn’t a schedulable thread and so is fundamentally non-faultable.

Break 3 — Emit large amounts to ringbuf

A sleepable fentry on __x64_sys_openat fires on every open(). On a busy system that’s thousands per second. Run on a heavy workload (find /usr | xargs cat > /dev/null) and the consumer can’t keep up — some records get dropped.

But you can’t see those drops with the obvious command. A ringbuf is a stream, not a key/value store — it has no iteration and no built-in per-map drop counter — so bpftool map dump simply can’t dump it:

sudo bpftool map dump name rb

It prints a useless Found 0 elements line and exits non-zero (exit 244 on bpftool v7.7.0; some builds instead produce no output at all — either way you learn nothing about drops). Real drop visibility needs an explicit counter you add yourself: a __u64 in a separate BPF_MAP_TYPE_ARRAY, incremented whenever bpf_ringbuf_reserve() returns NULL, then read with bpftool map dump name <counter_map>. You build exactly that in Day 13.

Break 4 — Use bpf_probe_read_user_str instead

bpf_probe_read_user_str(e->path, sizeof(e->path), upath);

This works in non-sleepable programs too. It doesn’t fault — it returns -EFAULT if the page isn’t resident. For tracing, the silent miss is usually acceptable; you trade a few dropped events for not needing sleepable.

When do you actually need sleepable? When dropping the event isn’t acceptable (security policy, audit logs), or when you need a helper that only exists in sleepable form (bpf_copy_from_user, bpf_copy_from_user_task).

A helper worth distinguishing from these: bpf_d_path (resolves a struct path to a string). It is not sleepable-only — its proto carries no .might_sleep flag. Instead it’s gated purely by a BTF allowlist: the .allowed callback (bpf_d_path_allowed in kernel/trace/bpf_trace.c:947, wired at bpf_d_path_proto’s .allowed field, :970) only permits it when you’re attached to one of a fixed set of functions (security_file_open, vfs_getattr, filp_close, dentry_open, …, in the btf_allowlist_d_path set), or from a BPF iterator. So if bpf_d_path is rejected, the fix isn’t .s/ — it’s attaching to an allowlisted function.

Contrast this with bpf_ima_inode_hash / bpf_ima_file_hash, which are gated by both mechanisms at once: their protos carry .might_sleep = true (kernel/bpf/bpf_lsm.c:181 and :200) and a .allowed allowlist callback (bpf_ima_inode_hash_allowed, :187). So they require a sleepable program (rejected by the same .might_sleep gate that rejects bpf_copy_from_user) and an allowlisted attach target. Only bpf_d_path is the clean “allowlist-but-not-sleepable” example.

What to read in the kernel

• kernel/bpf/trampoline.c — search __bpf_prog_enter_sleepable (:1250). The wrapper that takes the tasks-trace RCU read-side lock (rcu_read_lock_trace()), migrate_disable()s, and calls might_fault() around a sleepable program invocation. The comment at :518 lays out trace-RCU-vs-classic-RCU explicitly.
• kernel/bpf/verifier.c — search in_sleepable_context (:10253). How the Verifier decides which helpers a program may call; the .might_sleep gate is at :10330.
• kernel/bpf/helpers.c — bpf_copy_from_user body (:659) and its .might_sleep = true proto flag (:675).
• Documentation/RCU/Design/Requirements/Requirements.rst plus the Tasks RCU material — the relevant design notes are the Tasks-Trace RCU sections (the load-bearing code is __bpf_prog_enter_sleepable in trampoline.c). Ignore older “SRCU protects sleepable BPF” claims; that’s stale.
• tools/testing/selftests/bpf/progs/lsm.c — sleepable LSM example (see SEC("lsm.s/bprm_committed_creds") at :111).
• include/linux/bpf.h — search struct bpf_prog and find the sleepable:1 flag bit (:1794).

Bullet Points

• A page fault is a CPU trap taken when the MMU finds a “not present” PTE. Userspace memory is demand-paged: a pointer can be valid (inside a VMA) yet have no resident page. The handler does minor faults (page already in RAM → install PTE, no sleep) and major faults (read from disk/swap → thread descheduled). Major faults are the reason sleepable BPF exists.
• “Sleep” in the kernel = block/deschedule, not delay. Forbidden in non-faultable/atomic contexts (IRQ, softirq/XDP, spinlock held, preemption off); allowed in faultable contexts (syscall entry, most LSM hooks, iterators).
• Sleepable BPF programs can run in faultable contexts and call helpers that page-fault or schedule. Opt in via the .s/ suffix (SEC("fentry.s/..."), SEC("lsm.s/...")).
• Sleepable programs can use helpers marked .might_sleep: bpf_copy_from_user / bpf_copy_from_user_task (and the IMA hashers bpf_ima_inode_hash / bpf_ima_file_hash, which additionally require an attach-target allowlist). (bpf_d_path is not sleepable-only — it’s gated purely by a BTF allowlist of attach targets, with no .might_sleep flag.)
• Cost & protection: sleepable progs run under an RCU Tasks Trace read-side (rcu_read_lock_trace) plus migrate_disable() — slightly more expensive than classic RCU, but it permits sleeping. Detach waits a tasks-trace grace period (synchronize_rcu_tasks_trace) before freeing. (Not SRCU — that’s an outdated name.)
• Limit: only attach types that run in faultable contexts can be sleepable — fentry on syscalls, LSM hooks, iterators. Not XDP, softirq, or IRQ handlers.
• Modern x86-64 syscall hooks take a single struct pt_regs * (the SYSCALL_DEFINEn wrapper convention), so you decode args with PT_REGS_PARM*_CORE_SYSCALL, not typed BPF_PROG params.
• For tracing reads where occasional misses are OK, prefer bpf_probe_read_user_str in a non-sleepable program — cheaper and doesn’t require sleepable plumbing.

Check question

You write a sleepable fentry program that calls bpf_copy_from_user to read a 4 KB buffer. The user pointer is valid but the page isn’t currently in memory. What happens to the BPF program’s execution timeline?