Linux Network Subsystem in 30 Days

A hands-on, experiment-driven path through the Linux kernel network stack. Read kernel source, instrument with ftrace/perf/bpftrace, change sysctls, observe behavior. Verified against kernel 7.1.

What you’ll know by Day 30

• How a packet flows from wire to userspace, and back, through every kernel layer.
• The data structures (sk_buff, net_device, sock, dst_entry, fib_table, nf_conntrack) by name and by line number.
• The TCP state machine and how Linux’s CC algorithms (CUBIC, BBR, etc.) plug in.
• Netfilter, nftables, and conntrack — the modern packet-filter trio.
• Traffic control: qdiscs, classes, fq_codel.
• Modern features: kTLS, MPTCP, io_uring networking, drop_monitor.

Phases

How to use this

Each day is ~75–90 minutes. Most days follow this shape (loosely — not every section appears every day):

1. Concepts introduced today — what the new structures and paths are.
2. What to read in the kernel — specific files and functions in your kernel source checkout. Line numbers verified against kernel 7.1.
3. Today’s experiment — what to trace/modify/observe to see the structures running.
4. What to break — sysctls or configurations that demonstrate the effect (sometimes folded into the experiment).
5. Bullet Points.
6. Check question.

The labs lean on ftrace, perf, and bpftrace as observability tools. You don’t need to know how to write BPF programs to follow these labs — you’ll use one-liners and existing tools.

Tip: snapshot your sysctls

Some experiments adjust sysctls. Snapshot before:

sudo sysctl -a > sysctl.snapshot.$(date +%Y%m%d)

So you can diff back to a known-good state at the end of each day.

Companion book

After this, the Practical eBPF book builds on the kernel-internals foundation laid here. The plans are intentionally sequenced: kernel internals first, BPF as the application of that knowledge.

Day 1 — struct sk_buff: the universal packet container

Today’s mission: understand the most-touched data structure in Linux networking. Learn its anatomy, lifecycle, and the layout games it plays — and the hardware and memory machinery underneath that make those games necessary. Total time: ~75 minutes.

Phase 1 starts here. Days 1–5 cover the foundations: sk_buff, the RX/TX paths, NAPI, segmentation offloads, network namespaces. By Day 5, you’ll be able to read the kernel’s RX path top-to-bottom without losing the thread.

Why sk_buff matters more than any other structure

Every packet that crosses any layer of the kernel network stack rides in an sk_buff. RX from the NIC, TX to the wire, packet filtering, routing, encapsulation, segmentation — all operate on this structure. Knowing how it works is the difference between reading kernel networking code fluently and getting lost two function calls deep.

The structure lives at include/linux/skbuff.h (line 886 in 7.1) and the implementation at net/core/skbuff.c (~7500 lines as of 7.1 — about 80% of the file is utility functions).

Here’s the thing, though: the sk_buff doesn’t exist in a vacuum. It is the kernel’s answer to a very physical problem — a network card just dumped some bytes into your RAM, and now several different subsystems each want to look at those bytes without copying them around. To really understand why sk_buff is shaped the way it is, you need four pieces of background that most tutorials skip:

1. How a NIC actually delivers a packet into memory (descriptor rings + DMA).
2. How the kernel hands out memory in fixed-size pools (the slab allocator).
3. How physical memory is chopped into pages, and why you can’t always get a big contiguous chunk.
4. How the CPU reads memory in 64-byte lines, which dictates field ordering.

We’ll teach each of these as we hit the part of sk_buff that depends on it. By the end you’ll be able to read __alloc_skb and build_skb and know exactly what every line is fighting against.

Where packets come from: the NIC, descriptor rings, and DMA

Before we open the struct, let’s follow the bytes. When a frame arrives on the wire, the CPU is not involved in copying it into memory. That job belongs to the NIC itself, working through a mechanism called DMA.

DMA (Direct Memory Access) means the device reads and writes host RAM on its own, without the CPU moving bytes one load/store at a time. The CPU just tells the device where in RAM to put things; the device’s DMA engine does the transfer and signals when it’s done. For this to be safe, the buffer must be dma_map’d first: the kernel gives the device a bus address it can use, and the mapping makes sure CPU caches and device view of memory stay coherent.

So how does the NIC know where to put incoming frames? Through a RX descriptor ring:

• A ring is a fixed-size circular array of descriptors, living in host memory, that the driver sets up at init time.
• Each descriptor is a small record holding a DMA (bus) address plus status bits.
• At startup the driver pre-allocates a buffer (typically one page, or a slice of a page) for each descriptor and writes that buffer’s bus address into the descriptor. These are the “driver’s preallocated pages” you’ll see referenced everywhere.
• When a frame arrives, the NIC’s DMA engine writes the frame bytes directly into the buffer named by the next free descriptor, then flips a DONE bit in that descriptor and (eventually) raises an interrupt.

The crucial consequence: by the time the CPU runs, the packet bytes are already sitting in a page in RAM. The driver didn’t copy them there — the hardware did. The driver’s job in its receive routine (which on Linux runs inside NAPI poll — Day 2’s whole topic) is to walk the ring, find descriptors with the DONE bit set, and turn each already-populated DMA page into an sk_buff the stack can process.

That last step is where sk_buff enters, and it has two strategies:

• Small packets (e.g. a 64-byte ACK): copy the few bytes out of the DMA page into a fresh, small skb buffer, and immediately recycle the DMA page back into the ring. Cheap, because the copy is tiny.
• Large packets: don’t copy at all. Wrap the existing DMA page in an skb so that the skb’s data pointer points into the page the NIC already filled. This is the zero-copy receive path, and it’s exactly what build_skb() exists to do. We’ll meet it below.

TX is the mirror image. To transmit, the driver takes an outgoing skb, dma_maps its linear buffer and each page fragment, writes those bus addresses into TX-ring descriptors, and kicks the NIC. The NIC’s DMA engine reads the bytes straight out of those pages and onto the wire, then raises a completion so the driver knows it can unmap and free the skb.

Hold onto one idea: a packet’s payload often lives in independent pages that the kernel wants to reference, not copy — DMA pages on RX, file-cache pages on sendfile() TX. That single fact is why sk_buff is split into a small “descriptor + headers” part and a list of “page fragments,” which we’ll see in a moment.

Connect forward: Day 2’s RX path starts in the driver’s NAPI poll draining this exact ring. Today we only need to know the ring exists and that it leaves packet bytes pre-positioned in pages.

A common zero-copy RX path: DMA page → skb window

Here’s the part most treatments skip, and it’s the part that makes everything else click. On a common high-performance receive path, the driver gives the NIC a kernel-owned buffer for DMA, then wraps that same storage in an skb instead of copying the frame. This is an important path — not a contract every driver must follow.

A modern NIC and its driver commonly communicate through a descriptor ring: a fixed-size circular array where each receive descriptor names a DMA-capable buffer. In a page-pool-based driver, the handshake looks like this:

1. Before a packet arrives, the driver obtains a page (often from a page pool), maps it for DMA, and writes its DMA address into a free receive descriptor. The kernel owns the memory; the descriptor lends it to the NIC.
2. A frame lands. The NIC DMA-engines the bytes into that buffer, marks the descriptor done, and eventually causes NAPI polling to run.
3. The driver processes the completion. Native XDP, when supported and attached, can inspect the raw buffer here. After XDP_PASS, many drivers call build_skb or napi_build_skb: the helper allocates an skb descriptor and points skb->head at the caller-provided storage. That branch avoids a payload copy.
4. The skb travels up the stack — ip_rcv, routing, TCP — with its pointers describing bytes in that buffer.
5. The last data reference releases the head. head_frag tells skb_free_head() that the head came from a page or page fragment. Separately, a page-pool-aware driver calls skb_mark_for_recycle(), setting pp_recycle; only then does the free path try to return the page to its page pool. Without that marker, the page-fragment free path releases it normally.

The alternatives matter. Some drivers allocate an skb with napi_alloc_skb() and copy a small frame into it (a receive copybreak); others use different buffer managers. Generic XDP also runs later, after an skb already exists. So build_skb means “wrap caller-provided storage,” not “all RX is page-pool zero-copy.”

This is why build_skb exists alongside __alloc_skb. __alloc_skb allocates a descriptor plus fresh linear storage. build_skb allocates the descriptor around storage supplied by its caller; that storage may be a DMA-filled page, but the helper itself neither knows about a NIC ring nor opts the skb into page-pool recycling.

Three consequences fall out of the common page-backed path:

• head_frag and pp_recycle answer different questions. head_frag selects page-fragment release rather than kfree; pp_recycle asks the page-pool path to reclaim eligible storage.
• Native XDP can run before skb construction. A drop or redirect can therefore avoid allocating the skb descriptor at all. (Day 27.)
• The driver reserves headroom and tailroom deliberately. NET_SKB_PAD-style headroom permits later skb_push; tailroom must include aligned skb_shared_info. The exact layout is driver-specific.

The transmit side uses a related ring but not a perfect ownership mirror (Day 3). TCP may retain its original skb in the retransmit tree while a charged clone travels through IP, the qdisc, and the driver. The driver maps that downstream skb for a TX descriptor, and completion releases the downstream reference.

Hold onto the qualified picture: RX descriptor → driver-owned DMA buffer → optional native XDP → skb view (or copy) → stack → release/recycle. Days 2 and 3 walk it function by function.

The descriptor and the data

The sk_buff itself is a descriptor. The packet bytes live elsewhere — in a separately-allocated linear buffer plus, optionally, a tail of page fragments.

Four pointers in sk_buff define the linear region:

• head — start of the allocation. Doesn’t move after creation.
• data — first valid byte of the current header view. Moves as you push/pull headers.
• tail — one past the last valid byte.
• end — the boundary; the skb_shared_info trailer (fragment list + buffer refcounts, defined below) lives here.

The invariants:

• head ≤ data ≤ tail ≤ end
• headroom = data - head, tailroom = end - tail
• For linear-only skbs, len == tail - data and data_len == 0.

When the kernel pushes an outer header (e.g., adding an IP header to encapsulate), it does skb_push(skb, header_len) — which decrements data. Headroom shrinks; the prepended bytes are now part of the packet. The opposite is skb_pull — increment data, used when stripping a header you’ve already processed.

Why reserve headroom at all? NET_SKB_PAD explained

Notice that skb_push only works if there’s empty space before data to push into. That empty space is the headroom, and on the receive path the driver deliberately reserves some up front. Why?

Because as a received packet climbs the stack, layers frequently need to prepend bytes: re-insert a hardware-stripped VLAN tag, wrap the packet in an outer IP/UDP header for a tunnel, and so on. If there’s no headroom, each skb_push would have to reallocate the whole buffer (via pskb_expand_head) just to make room — expensive. Reserving a little headroom at allocation time means those pushes are nearly free.

The amount reserved by the RX allocators is NET_SKB_PAD, and it is not a magic constant — it’s defined relative to the cache line:

#define NET_SKB_PAD     max(32, L1_CACHE_BYTES)   /* include/linux/skbuff.h:3319 */

On x86_64, L1_CACHE_BYTES is 64 (CONFIG_X86_L1_CACHE_SHIFT=6 → 1 << 6 = 64), so NET_SKB_PAD = 64. Two things are happening with that value:

1. Enough room for outer headers. 64 bytes comfortably covers a VLAN tag, a tunnel header, etc., so the common prepends never reallocate.
2. Cache alignment. Because the pad equals the cache-line size, the packet’s first byte (data, where the Ethernet header begins) starts on a cache-line boundary — which the receive path and memcpys benefit from. (We’ll explain cache lines fully later; for now: aligned data is faster to touch.)

A separate, smaller pad you’ll see is NET_IP_ALIGN. Some architectures want the IP header 4-byte aligned. Since the Ethernet header is 14 bytes, a 2-byte pad before data makes byte 14 (the start of the IP header) land on a 4-byte boundary. But NET_IP_ALIGN = 2 (skbuff.h:3295) is only the generic fallback — it’s defined under #ifndef NET_IP_ALIGN. On x86 the arch overrides it to 0 (arch/x86/include/asm/processor.h:46: #define NET_IP_ALIGN 0), so no IP-alignment pad is inserted there at all — it’s not a per-driver choice to “skip” a 2-byte pad, the constant is simply zero. That’s why napi_alloc_skb reserving NET_SKB_PAD + NET_IP_ALIGN (skbuff.c:886) comes out to 64 + 0 = 64 on x86, matching the “~64 bytes” the headroom experiment promises. The 2-byte pad is honored only on alignment-sensitive architectures, where it costs nothing extra.

Keep NET_SKB_PAD = 64 in mind — it shows up again in the headroom experiment and in today’s Check question.

Page fragments — for big packets and zero-copy

Many real packets — especially large outbound ones from sendfile() or large GRO inbounds — don’t fit in one allocation. The kernel uses skb_shared_info (placed right after the linear buffer’s end) to chain page fragments:

struct skb_shared_info {
    __u8 nr_frags;
    skb_frag_t frags[MAX_SKB_FRAGS];   /* (netmem, offset, len) tuples; see below */
    struct sk_buff *frag_list;         /* chain of skbs (TSO) */
    /* ... */
};

data_len holds the byte count in fragments. len = (tail - data) + data_len. Most code uses helpers (skb_frag_size, skb_frag_page) rather than touching the fields directly.

What a “page” is, and why payload goes into frags

To see why the kernel bothers with a separate fragment list instead of one big buffer, you need the kernel’s unit of physical memory: the page.

• Physical RAM is managed in fixed page frames of PAGE_SIZE (4 KB on x86_64). Every allocation of physical memory is ultimately some number of these frames.
• The buddy allocator hands out runs of 2^order contiguous pages: order 0 = 4 KB, order 1 = 8 KB, order 3 = 32 KB, and so on. Higher orders require more physically contiguous memory.
• kmalloc returns physically contiguous memory and is therefore capped at a maximum size (KMALLOC_MAX_SIZE, slab.h:591). Above KMALLOC_MAX_CACHE_SIZE (slab.h:593) kmalloc stops using the size-bucketed slab caches and goes straight to the buddy/page allocator (still physically contiguous); the hard upper bound on one allocation is KMALLOC_MAX_SIZE. Asking for, say, one 64 KB physically contiguous block is unreliable on a busy system: there may be plenty of free pages, but not 16 of them in a row. That condition — free memory that isn’t contiguous — is fragmentation pressure.

Now the design makes sense. Think of a skb_frag_t as a (page, offset, size) reference — some bytes inside a page — which is the right first approximation. The literal struct in 7.1 is a (netmem, offset, len) tuple: struct skb_frag { netmem_ref netmem; unsigned int len; unsigned int offset; } (skbuff.h:361). The netmem_ref (include/net/netmem.h:140) is an opaque reference that encodes either a struct page or a net_iov (non-page memory, e.g. devmem-TCP), which is why a frag isn’t guaranteed to be backed by a struct page at all — skb_frag_page() first checks skb_frag_is_net_iov(frag) and returns NULL for the non-page case. Always touch a frag via the helpers skb_frag_page() / skb_frag_off() / skb_frag_size(), never the fields directly. The memory referenced by different frags need not be contiguous with each other or with the linear head. So a 64 KB GRO superpacket becomes:

• a tiny linear head holding just the Ethernet/IP/TCP headers (a small kmalloc, easy to satisfy), plus
• up to MAX_SKB_FRAGS page fragments holding the payload — each an ordinary 4 KB page grabbed from wherever it was free.

No 64 KB contiguous allocation needed, ever. And it’s exactly this structure that makes zero-copy possible: the frag pages can be the NIC’s DMA pages on RX, or sendfile()’d file-cache pages on TX — referenced, never copied.

MAX_SKB_FRAGS bounds how many frags one skb_shared_info holds. In v7.1 it’s configurable:

/* include/linux/skbuff.h */
#ifndef CONFIG_MAX_SKB_FRAGS
# define CONFIG_MAX_SKB_FRAGS 17
#endif
#define MAX_SKB_FRAGS CONFIG_MAX_SKB_FRAGS

and net/Kconfig declares config MAX_SKB_FRAGS with range 17 45 and default 17 — raising it helps GRO/BIG TCP pack more payload per skb.

There are no Dumb Questions

Q: Why isn’t the linear buffer just always big enough?

A: Because each skb potentially allocates its linear buffer at packet receive. For 64-byte ACKs that don’t carry payload, a 1500-byte allocation would waste memory. For 64KB GRO superpackets, a 64KB linear buffer is impractical (kmalloc max-order limits, fragmentation pressure). The split design lets the kernel allocate just enough for headers linearly and use page fragments for payload.

Q: What’s the cb[48] for?

A: It’s a per-packet scratchpad. Each protocol layer can stash state there. TCP uses TCP_SKB_CB(skb) to keep sequence numbers, flags, and SACK info. Across skb_clone the cb contents are copied verbatim into the clone — __copy_skb_header (net/core/skbuff.c:1552) does memcpy(new->cb, old->cb, sizeof(old->cb)) — so the clone starts with the same control data. Layers that need clone-independent state must reset cb themselves. 48 bytes is generous — most layers use a fraction.

Q: How big is a fresh sk_buff descriptor?

A: Around 230 bytes on x86_64 (verify: pahole sk_buff in your build directory after compiling — exact size is config-dependent). The structure has been carefully cache-line-aligned and the fields ordered for hot/cold separation. Read the comments around the field declarations to see what’s “RX hot” vs “TX hot.”

How the kernel hands out memory: the slab allocator

We’re about to read the allocation path, and it names several “caches.” Before that pays off, you need the slab allocator (Linux’s is called SLUB).

A general-purpose malloc is flexible but has overhead: it tracks variable-sized blocks, searches free lists, and fragments over time. The kernel allocates the same few object types over and over — millions of sk_buffs, for instance — so it uses a specialized scheme:

• The slab allocator carves whole pages into fixed-size object slots.
• A kmem_cache is a pool specialized for one object size/type. Allocation is essentially “pop a free slot,” free is “push it back” — O(1), with excellent cache behavior because objects of the same type cluster together.
• General-purpose kmalloc(n) is built on top of a set of size-bucketed caches (kmalloc-512, kmalloc-2k, …). kmalloc(n) rounds n up to the next bucket and pulls from that cache.

Now here’s the key structural insight about sk_buff: the descriptor and the data come from two different allocators.

• The descriptor (struct sk_buff itself) comes from a dedicated, fixed-size cache so every skb is the same size and stays hot in cache. In v7.1 this cache is named "skbuff_head_cache" and reached via net_hotdata.skbuff_cache:

  /* net/core/skbuff.c:5189, in skb_init() */
  net_hotdata.skbuff_cache = kmem_cache_create_usercopy("skbuff_head_cache",
                          sizeof(struct sk_buff), 0,
                          SLAB_HWCACHE_ALIGN | SLAB_PANIC | FLAG_SKB_NO_MERGE, ...);
  

• The linear data buffer is a separate allocation, sized to the packet. That’s why __alloc_skb allocates twice.

There are three skb-related caches created in skb_init():

Fclones. When code knows it will clone an skb soon (TCP does this constantly — it keeps a copy for retransmission while sending a clone down the stack), it sets SKB_ALLOC_FCLONE. That allocates from skbuff_fclone_cache, whose object is a struct sk_buff_fclones { struct sk_buff skb1; struct sk_buff skb2; refcount_t fclone_ref; } (skbuff.h:1396). The anticipated clone (skb2) is right there in the same allocation — no second descriptor allocation needed when the clone happens.

Per-CPU fast path. RX is hot, so there’s a per-CPU stash of recycled descriptors called the napi_alloc_cache (a per-CPU skb_cache[] array, skbuff.c:231). napi_skb_cache_get() (skbuff.c:284) pops from it; when empty it refills in bulk via kmem_cache_alloc_bulk(net_hotdata.skbuff_cache, ...). Allocating an skb with SKB_ALLOC_NAPI (or simply allocating in softirq context) pulls a descriptor from this per-CPU cache and skips the slab allocator’s slow path entirely. That’s the “per-CPU caching” the NAPI allocator advertises.

GFP flags. Every allocation takes a gfp_mask (e.g. GFP_KERNEL, GFP_ATOMIC). It tells the allocator what it’s allowed to do — most importantly, whether it may sleep. The RX path runs in softirq/IRQ (atomic) context where sleeping is forbidden, so it uses GFP_ATOMIC. That’s why the allocation functions thread a gfp_mask (and a NUMA node) through everything.

sk_buff lifecycle: cradle to grave

Now the allocation paths read cleanly, because you know what every cache and flag means.

Allocation paths (all in net/core/skbuff.c):

• __alloc_skb(size, gfp_mask, flags, node) — the workhorse. Allocates the descriptor from net_hotdata.skbuff_fclone_cache (when SKB_ALLOC_FCLONE is set, useful for skbs you’ll clone) or the default skbuff_head_cache — or, with SKB_ALLOC_NAPI, from the per-CPU napi_alloc_cache. Then allocates the linear buffer separately via kmalloc_reserve (skbuff.c:604), which pulls from skb_small_head_cache for small heads or falls back to the size-bucketed kmalloc caches for larger ones. Two allocations, two allocators — exactly as the slab section predicted. (__alloc_skb is at skbuff.c:672.)
• __netdev_alloc_skb(dev, len, gfp_mask) — drivers’ RX-side allocator. Its buffer “has NET_SKB_PAD headroom built in” (the kernel doc comment says so verbatim at skbuff.c:753) for cheap header insertion.
• napi_alloc_skb(napi, len) — NAPI fast-path allocator with the per-CPU descriptor caching described above; also reserves NET_SKB_PAD.
• build_skb(data, frag_size) / slab_build_skb(data) — wrap a pre-existing buffer (the driver’s preallocated DMA page) into an skb. This is the zero-copy receive path from the NIC section. Instead of allocating a data buffer and copying the frame in, it allocates only a descriptor (kmem_cache_alloc(net_hotdata.skbuff_cache, ...)) and points the skb at the page the NIC already filled. __build_skb is at skbuff.c:488; build_skb at skbuff.c:506 then sets skb->head_frag = 1 (the skb’s head is a page fragment, skbuff.h:828) and calls skb_propagate_pfmemalloc(...). napi_build_skb() (skbuff.c:574) is the NAPI-context variant that also sources its descriptor from the per-CPU cache. The doc comment on __build_skb spells out the RX model we taught: “Before IO, driver allocates only data buffer where NIC put incoming frame… RX rings only contain data buffers, not full skbs.”

Cloning paths — and to read these you need the refcount model, so let’s build it first.

Two refcounts, not one: skb->users vs dataref

skb_clone makes a cheap copy of an skb. To understand why it’s cheap and what it’s safe to do afterward, you must know that an skb tracks two completely separate reference counts.

1. skb->users — references to the descriptor.

refcount_t users;   /* include/linux/skbuff.h:1099 */

This counts how many places hold this sk_buff pointer. skb_get() bumps it; kfree_skb() decrements it and only actually frees when it hits zero. skb_shared(skb) is literally refcount_read(&skb->users) != 1 (skbuff.h:2112). Two subsystems queuing the same skb use this.

2. skb_shared_info.dataref — references to the data buffer.

atomic_t dataref;   /* include/linux/skbuff.h:612 */

This counts how many sk_buff descriptors point at the same data buffer. This is the one skb_clone touches: a clone gets a brand-new descriptor but, instead of copying the bytes, increments dataref. That is why cloning is cheap — no payload copy.

Because clones share the buffer, you must not blindly write to a cloned skb’s headers — you’d corrupt the other holder’s view. So dataref is cleverly split into two halves (SKB_DATAREF_SHIFT = 16, skbuff.h:658):

• the low 16 bits count the overall number of references to the buffer;
• the high 16 bits count how many of those are payload-only (header-release) references.

The kernel’s own doc block (skbuff.h, “DOC: dataref and headerless skbs”) explains it: TCP marks an skb nohdr via __skb_header_release(), which sets dataref = 1 + (1 << SKB_DATAREF_SHIFT) (skbuff.h:2101) so lower layers know they may prepend headers into the shared buffer safely. skb_header_cloned() (skbuff.h:2070) computes (dataref & MASK) - (dataref >> SHIFT) to decide whether the headers are safe to modify in place.

This gives you the precise meaning of two helpers you’ll see everywhere:

• skb_cloned(skb) (skbuff.h:2031) is true when the data buffer is shared ((dataref & SKB_DATAREF_MASK) != 1). If it’s true, you must not write headers without first doing a copy-on-write via pskb_expand_head.
• skb_shared(skb) tests the descriptor count (users != 1), a different question entirely.

With that, the cloning paths are obvious:

• skb_clone(skb, gfp) (skbuff.c:2088) — new descriptor, shares the data buffer (bumps dataref). Cheap. Used heavily in packet sockets (tcpdump) and netfilter LOG. (If an fclone is available it reuses the pre-allocated skb2 — no descriptor allocation at all.)
• skb_copy(skb, gfp) — full copy of both descriptor and data (fresh buffer, dataref = 1). Slow; only when you must mutate the payload.
• pskb_copy(skb, gfp) — copies the linear head but shares the frag pages (bumps the pages’ refcounts).

This two-level scheme is why tcpdump, which clones every packet, stays cheap per packet yet still shows up measurably at high rates — today’s clone-tracing experiment makes that visible.

Free paths:

• kfree_skb(skb) — the standard release. Decrements refcounts (both users and, when users hits zero, dataref on the buffer); frees the data only if it’s the last reference.
• kfree_skb_reason(skb, reason) — newer; takes a enum skb_drop_reason so the kernel’s drop monitor can attribute the drop. Always prefer this in new code. The reason enum is in include/net/dropreason-core.h.
• consume_skb(skb) — same as kfree_skb but doesn’t trigger drop tracepoints (used for “successful” disposals).

The CPU’s view of memory: cache lines and field ordering

One last piece of background, and it explains the comments you’ll read in the struct and the whole point of today’s pahole lab.

The CPU does not read memory one byte at a time. It moves memory in fixed cache-line units — L1_CACHE_BYTES, 64 bytes on x86_64. Touch a single field and the CPU pulls that field’s entire 64-byte line into L1 cache. Touch a field in a different line, that’s a second line fetched.

This has a direct performance consequence for a struct touched as often as sk_buff:

• Hot/cold separation. If the fields a given code path uses are scattered across many cache lines, the path drags many lines into cache. If they’re grouped into the same line(s), the path touches few lines and runs faster. So sk_buff is laid out so that RX-hot fields cluster, TX-hot fields cluster, and rarely-used (cold) fields sit apart so they don’t pollute hot lines. That’s why the very first fields are next, prev, and the dev/sk area (skbuff.h:886) — the list-and-device fields the RX path hits first.

• Bit-packed flags. Boolean state like cloned:1, nohdr:1, fclone:2, head_frag:1, pfmemalloc:1, … is packed into shared __u8 bitfields (skbuff.h:956) partly to keep the hot section dense — many flags in one byte instead of one byte each.

• Descriptor alignment. Remember skbuff_head_cache was created with SLAB_HWCACHE_ALIGN? That makes every descriptor start on a cache-line boundary, so a given field’s offset maps predictably to a line.

• Separating head from skb_shared_info. Look again at the comment inside __alloc_skb (right before kmalloc_reserve):

  /* We do our best to align skb_shared_info on a separate cache line.
   * It usually works because kmalloc(X > SMP_CACHE_BYTES) gives aligned
   * memory blocks ... Both skb->head and skb_shared_info are cache line aligned.
   */
  

  The kernel places skb_shared_info at a cache-line boundary at the end of the data buffer so that skb->head (written on RX) and skb_shared_info (refcount fields touched by clones) don’t share a line — otherwise the line would bounce between CPUs.

Tie it back to NET_SKB_PAD = max(32, L1_CACHE_BYTES): that’s the same L1_CACHE_BYTES, used so the data start lands on a line boundary. The cache line is the hidden constant behind half the layout decisions in this file.

This is precisely what pahole reveals: it prints each field’s offset, the padding holes, and a /* size: N, cachelines: M, members: K */ footer showing how the struct fills its lines. Today’s lab teaches you to read that.

Today’s experiment

You don’t write code today. You inspect what’s already running.

See sk_buff allocations live

sudo bpftrace -e 'kprobe:__alloc_skb { @[comm, kstack] = count(); } interval:s:5 { exit(); }' | head -50

5 seconds of skb allocations grouped by caller. You’ll see drivers and protocol layers calling in.

Watch drops with reasons

sudo perf trace --no-syscalls -e skb:kfree_skb -- sleep 5

Output shows where in the kernel each dropped skb was disposed of, plus the drop reason if kfree_skb_reason was used.

Inspect skb size on your build

cd ~/code/linux
# After building vmlinux once (needs CONFIG_DEBUG_INFO_BTF=y or DWARF):
pahole -C sk_buff vmlinux

pahole reads the struct layout from the compiled vmlinux’s debug info, not from the header source — a .h has no offsets or padding until the compiler lays the type out. -C sk_buff selects just that struct (no grep needed); it prints the field-by-field layout with a /* size: N, ... */ footer. Reveals the size, padding, and field-by-field layout. Note how next, prev, dev, sk are at the top — that’s the cache-hot section that the RX path touches first. Now that you know what a cache line is, read the padding holes and the cachelines: count as a map of the hot/cold separation we just discussed.

Trace a packet’s headroom journey

sudo bpftrace -e '
fentry:ip_rcv { @h[comm] = lhist((uint64)args->skb->data - (uint64)args->skb->head, 0, 256, 16); }
interval:s:5 { exit(); }'

Histogram of headroom on packets entering ip_rcv. You’ll see most packets have ~64 bytes of headroom (NET_SKB_PAD). That 0 vs 64 split is the whole point of the headroom section: the RX allocators (__netdev_alloc_skb, napi_alloc_skb) pre-reserve NET_SKB_PAD so later skb_pushes of outer headers don’t reallocate, while plain __alloc_skb reserves nothing (some drivers reserve more on top for crypto offload or XDP).

Watch clones share-count under tcpdump

A packet socket capture clones every packet. Start a capture (-w /dev/null discards packets so tcpdump doesn’t spam your terminal):

sudo tcpdump -i any -w /dev/null &

Then trace skb_clone:

sudo bpftrace -e 'fentry:skb_clone { @[kstack] = count(); } interval:s:5 { exit(); }' | head -30

skb_clone fires on every packet because the capture path clones each one. Each clone is a fresh descriptor that merely bumps the buffer’s dataref (not a payload copy), which keeps per-packet cost low — yet at high packet rates even that cheap clone is measurable, which is why tcpdump adds overhead. Stop the capture when done:

sudo pkill tcpdump

What to read in the kernel

• include/linux/skbuff.h — struct sk_buff definition (line 886). Read the field comments. Then look at the helpers (skb_push, skb_pull, skb_reserve, skb_put). Also skim the bitfield block (~line 955) and the “DOC: dataref and headerless skbs” comment.
• net/core/skbuff.c — __alloc_skb (line 672), __build_skb (line 488), kfree_skb_reason, skb_clone (line 2088). Note the three caches created in skb_init (lines 5189–5208). ~7500 lines total, but the allocation path is < 100 lines.
• include/linux/skbuff_ref.h — refcount helpers; quick read.
• include/net/dropreason-core.h — the enum skb_drop_reason list (~124 reasons in 7.1). Skim. This is what you’ll see in perf trace.
• Documentation/networking/skbuff.rst — the official reference. One-time read.

Bullet Points

• A NIC delivers packets by DMA into pre-allocated pages named by RX-ring descriptors — bytes are in RAM before the CPU runs. The driver turns each done descriptor into an skb.
• sk_buff is the descriptor; data lives in a linear buffer + optional page fragments. A frag is a (netmem, offset, len) tuple — netmem references a page or a net_iov (use skb_frag_page()/skb_frag_off()/skb_frag_size()) — so payload can be non-contiguous pages — DMA pages or sendfile pages — and need never be copied.
• Pointer invariant: head ≤ data ≤ tail ≤ end. headroom, tailroom, len, data_len. RX allocators pre-reserve NET_SKB_PAD = max(32, L1_CACHE_BYTES) = 64 so skb_push of outer headers doesn’t reallocate.
• A 64KB linear buffer is impractical because kmalloc returns physically-contiguous memory capped at KMALLOC_MAX_SIZE, and fragmentation makes big contiguous runs scarce — hence the linear-head + page-frags split.
• The slab allocator gives O(1) fixed-size pools. The skb descriptor comes from skbuff_head_cache; the linear buffer from skb_small_head_cache/kmalloc buckets — two allocations. skbuff_fclone_cache pre-bundles a clone; the per-CPU napi_alloc_cache feeds the RX fast path. GFP_ATOMIC is used in atomic RX context.
• cb[48] is the per-packet scratchpad each protocol layer uses.
• Allocate via __alloc_skb / napi_alloc_skb / build_skb (zero-copy wrap of a DMA page); free via kfree_skb_reason.
• Two refcounts: skb->users counts descriptor references (skb_shared); skb_shared_info.dataref counts how many descriptors share the data buffer (split into all-refs / payload-only halves). skb_clone bumps dataref and shares data; skb_copy duplicates everything (dataref = 1). skb_cloned() ⇒ copy-on-write before touching headers.
• enum skb_drop_reason is the new-and-required way to attribute drops.
• The structure is large (~230 bytes; verify with pahole, config-dependent); the CPU reads memory in 64-byte cache lines, so fields are cache-line ordered for RX-hot/TX-hot/cold separation, the descriptor is SLAB_HWCACHE_ALIGNed, and skb_shared_info is placed on its own line — read the comments, and read pahole’s cachelines: footer.

Check question

You receive a packet at the NIC. The driver allocates an skb via napi_alloc_skb(napi, 1500). The packet is 100 bytes of Ethernet + IP + TCP. Walk through what the four pointers (head, data, tail, end) look like just after the driver finishes setup but before ip_rcv runs.

Tomorrow

Day 2: the RX path. NAPI poll → driver → __netif_receive_skb → ip_rcv. We trace a packet through every stage and see where each skb_* helper gets called — starting in the driver’s NAPI poll, draining the very RX descriptor ring we met today.

Day 2 — The RX path: from wire to ip_rcv

Today’s mission: trace a packet from the moment it hits the NIC to the moment it enters ip_rcv. See every step — and learn the five mechanisms the path is built on (softirqs, NAPI, XDP, GRO, packet_type demux) so nothing along the way is a black box. Total time: ~110 minutes.

The journey at a glance

Every received packet on a typical Linux box traverses a sequence of handoffs. Hardware DMAs the frame into kernel-owned RX memory, the IRQ only schedules work, NAPI polls the driver under a budget, native XDP can decide before an skb exists, XDP_PASS becomes an sk_buff, GRO may coalesce related TCP segments, and the core stack dispatches the packet to the right L3 protocol handler.

That sentence names five things that this whole chapter leans on but that you’ve never been formally taught: a softirq, the NAPI state machine, XDP, the GRO accumulator, and packet_type demultiplexing. Day 1 forward-referenced all of them but taught none. So before we walk the path, we’ll teach each one — intuition first, then the concrete v7.1 struct or function — and only then trace the packet, with every stage resting on something you already understand.

We’ll anchor everything to a specific file/function in your ~/code/linux checkout (line numbers from kernel 7.1).

Two things from Day 1 are load-bearing today and will not be re-taught:

• The RX descriptor ring + DMA. Recall from Day 1 — the NIC’s DMA engine writes each frame into a pre-allocated page named by an RX-ring descriptor and flips a DONE bit; the driver’s NAPI poll drains those done descriptors. The bytes are in RAM before the CPU runs.
• build_skb zero-copy wrap. build_skb wraps that already-filled DMA page (head_frag = 1) instead of copying it, giving zero-copy receive. We met it in Day 1’s lifecycle section.

Background 1: What a softirq actually is

The whole RX path runs in a softirq. Before we can say “the IRQ raises NET_RX_SOFTIRQ and the softirq runs net_rx_action,” you need to know what that machinery is.

The problem: a hardware IRQ handler must be tiny

When a NIC raises an interrupt, the CPU drops whatever it was doing and jumps to the driver’s IRQ handler. That handler runs in a brutally constrained world:

• It preempts the currently-running task (which never agreed to be interrupted).
• It runs on a small, dedicated IRQ stack.
• It cannot sleep — no blocking allocations, no waiting on locks that might sleep.
• It runs with interrupts of the same priority masked, so while it runs, other devices at that level are stalled.

Doing the full receive path — route lookup, netfilter, BPF, delivering to a socket — inside that handler would be a disaster: it would stall the machine and starve everything else. So Linux splits interrupt work into two halves:

• The top half is the actual hardware IRQ handler. It does the absolute minimum: acknowledge the device and schedule deferred work. Then it returns.
• The bottom half is that deferred work, run later in a relaxed context where it’s allowed to take its time.

A softirq is Linux’s primary bottom-half mechanism.

What “softirq” means concretely

A softirq is a statically-defined, per-CPU bottom half. “Statically defined” matters: there is a fixed, small enum of softirq vectors compiled into the kernel — you cannot create one at runtime. Here is the entire list (include/linux/interrupt.h:550):

enum
{
    HI_SOFTIRQ=0,
    TIMER_SOFTIRQ,
    NET_TX_SOFTIRQ,
    NET_RX_SOFTIRQ,
    BLOCK_SOFTIRQ,
    IRQ_POLL_SOFTIRQ,
    TASKLET_SOFTIRQ,
    SCHED_SOFTIRQ,
    HRTIMER_SOFTIRQ,
    RCU_SOFTIRQ,    /* Preferable RCU should always be the last softirq */
    NR_SOFTIRQS
};

Networking owns two of them: NET_RX_SOFTIRQ (receive) and NET_TX_SOFTIRQ (transmit). Each vector has one action function registered at boot via open_softirq() (kernel/softirq.c:806). The networking subsystem registers its two in net/core/dev.c:

open_softirq(NET_TX_SOFTIRQ, net_tx_action);   // net/core/dev.c:13288
open_softirq(NET_RX_SOFTIRQ, net_rx_action);   // net/core/dev.c:13289

So net_rx_action is the receive bottom half. The rest of today is mostly the story of what net_rx_action does.

Raising vs. running — they are not the same

Here’s the subtlety that trips people up. “Raising” a softirq does not run anything. It just sets a per-CPU pending bit. The hot-path raise is raise_softirq_irqoff (kernel/softirq.c:773) → __raise_softirq_irqoff (:799), which does exactly one meaningful thing:

or_softirq_pending(1UL << nr);   // kernel/softirq.c:803

That’s it — OR a bit into this CPU’s pending mask. The actual draining happens elsewhere: __do_softirq (kernel/softirq.c:654) checks the pending mask and runs each pending vector’s action function. It’s invoked on IRQ-exit (right after the top half returns) and from a few other points. So the top half “raises” (sets the bit and returns); moments later, on the way out of interrupt context, __do_softirq notices the bit and calls net_rx_action.

Why a packet flood can’t freeze your shell: ksoftirqd

Softirqs run at higher priority than user threads — so what stops an endless packet flood from making __do_softirq loop forever and starving your login shell? A safety valve. __do_softirq will restart processing at most MAX_SOFTIRQ_RESTART = 10 times (kernel/softirq.c:544). If softirqs are still pending after that, it gives up and wakes a per-CPU kernel thread, ksoftirqd (run_ksoftirqd, kernel/softirq.c:1068), to finish the work as a normal schedulable thread. That thread competes fairly with user tasks, so softirq work can be intense but can never indefinitely lock out user space. This hand-off is the concrete meaning of “softirqs are lower priority than hardware IRQs but a notch above user threads.”

Per-CPU is the through-line

Notice the recurring word: per-CPU. Each CPU has its own pending mask, runs its own __do_softirq, and has its own ksoftirqd. This is why everything downstream is per-CPU too — the poll_list NAPI uses, the softnet_data struct, the /proc/net/softnet_stat counters. When you see “per-CPU” later, it traces straight back to here.

Background 2: NAPI and the two budgets

You’ll constantly read “napi_schedule,” “poll_list,” and “the poll function processes packets under a budget.” Time to make those real — and to kill a common confusion: there are two different budgets, not one.

The NAPI idea: stop interrupting, start polling

A 10/40/100-Gbit NIC under load can raise millions of interrupts per second. Even with a tiny top half, that’s death by a thousand cuts — the IRQ overhead alone melts a CPU (“receive livelock”). NAPI (“New API,” though it’s been the API for ~20 years) fixes this with a trade: at the first packet, mask the device’s RX interrupt and switch to polling. The kernel then repeatedly asks the driver “got more?” until the ring drains, and only then re-enables the interrupt. One IRQ per burst, instead of one per packet.

struct napi_struct: the per-queue polling context

Each NIC receive queue registers a struct napi_struct — the context that ties a driver’s poll routine to the core’s polling loop (include/linux/netdevice.h:381):

struct napi_struct {
    unsigned long       state;       /* NAPI_STATE_SCHED bit lives here */
    struct list_head    poll_list;   /* links onto the per-CPU poll_list */
    int                 weight;      /* per-poll packet cap (default 64) */
    int                 (*poll)(struct napi_struct *, int);  /* driver callback */
    struct net_device   *dev;
    struct sk_buff      *skb;
    struct gro_node     gro;         /* the GRO accumulator (Background 4) */
    /* ... timers, ids, control-path fields ... */
};

The four fields that matter today: state (holds the NAPI_STATE_SCHED bit — “I’m scheduled to be polled”), poll_list (the list node that hangs this napi off the per-CPU poll list), weight (the per-poll budget — more below), and poll (the driver’s drain routine, e.g. e1000_clean, mlx5e_napi_poll).

The schedule → poll → complete cycle

Here is the full loop. Follow the NAPI_STATE_SCHED bit and the hardware IRQ as they flip:

1. IRQ fires. The driver’s top half calls napi_schedule (or __napi_schedule_irqoff on the hot path). Its first step, napi_schedule_prep (net/core/dev.c:6736), atomically sets NAPI_STATE_SCHED (and sets NAPI_STATE_MISSED if SCHED was already set). Then ____napi_schedule (net/core/dev.c:4957) adds the napi to this CPU’s poll list —

   list_add_tail(&napi->poll_list, &sd->poll_list);   // net/core/dev.c:4984
   ...
   raise_softirq_irqoff(NET_RX_SOFTIRQ);              // net/core/dev.c:4990
   

   — and raises NET_RX_SOFTIRQ (there’s the Background-1 mechanism). The driver also masks its own RX interrupt at this point. From now until step 3, this queue generates no IRQs.

2. Softirq runs net_rx_action, which walks the poll list and calls each napi’s ->poll(napi, weight). The driver drains done descriptors from the ring (the Day-1 ring), turning each into an skb.

3. Ring drained below weight. When the driver finds fewer packets than its weight, it knows the burst is over and calls napi_complete_done (net/core/dev.c:6771). That clears NAPI_STATE_SCHED and returns true to tell the driver the burst is over; the driver then re-enables its hardware RX interrupt (e.g. if (napi_complete_done(napi, work)) <write the device's IRQ-enable register>). Back to interrupt-driven mode until the next packet.

This is the exact mechanism behind “one IRQ per burst.” If instead the driver hits its weight (ring still full), it does not complete — it returns the full weight, stays on the poll list, and gets polled again. The interrupt stays masked while traffic is heavy.

Two budgets: weight (64) vs netdev_budget (300)

This is the part everyone conflates. There are two separate caps:

• weight — caps how many packets one ->poll call may process. It’s the value passed into ->poll. Default is NAPI_POLL_WEIGHT = 64 (include/linux/netdevice.h:2839). The per-NAPI poll wrapper reads it (__napi_poll, net/core/dev.c:7719): weight = n->weight; work = n->poll(n, weight);.

• netdev_budget — caps the total packets across all napis in a single net_rx_action softirq run. Default 300 (net/core/hotdata.c:14). There’s also a time cap, netdev_budget_usecs (net/core/hotdata.c:16, = 2 jiffies).

So in one softirq run, net_rx_action starts with a budget of 300 and spends it across however many queues are pending, where each individual poll call can claim at most 64. The loop (net/core/dev.c:7914):

int budget = READ_ONCE(net_hotdata.netdev_budget);   // 300, the OUTER budget
...
n = list_first_entry(&list, struct napi_struct, poll_list);
budget -= napi_poll(n, &repoll);                     // each poll capped at weight=64

Picture it as nested limits: the inner ring (one poll) holds 64; the outer ring (one softirq run) holds 300 spread across all queues.

time_squeeze and softnet_data

When net_rx_action exits the loop because budget went to zero or the time limit expired while work was still pending, it bumps a counter called time_squeeze and leaves the remaining napis to be polled in a later softirq (or punted to ksoftirqd). That counter is exactly what today’s softnet_stat lab watches: a rising time_squeeze means receive work is outrunning a single softirq window.

All of this per-CPU bookkeeping lives in struct softnet_data (DEFINE_PER_CPU_ALIGNED(struct softnet_data, softnet_data), net/core/dev.c:462): the poll_list, the processed/drop/time_squeeze counters. /proc/net/softnet_stat prints one line per CPU straight from these fields (net/core/net-procfs.c:145).

Stage 1: NIC → IRQ → softirq

Now Stage 1 reads as plain English. Modern NICs use interrupt coalescing: several packets per IRQ, configurable via ethtool -c. When the IRQ fires, the driver doesn’t process packets in IRQ context (Background 1 told you why it can’t). Instead its top half calls napi_schedule() (or __napi_schedule_irqoff on the hot path), which (Background 2):

1. Sets NAPI_STATE_SCHED (via napi_schedule_prep).
2. Adds the napi to the per-CPU poll_list.
3. Raises NET_RX_SOFTIRQ.

Separately, the driver masks its own RX interrupt in its IRQ handler (Background 2, step 1) — napi_schedule itself touches no hardware.

The softirq runs net_rx_action (net/core/dev.c:7914), which iterates the per-CPU poll list, calling each napi’s poll function under the two budgets you just learned (Background 2): the inner weight (64) per ->poll call, and the outer netdev_budget (300) across all napis in the run.

The core napi_poll wrapper invokes the driver’s registered ->poll (e.g. e1000_clean, mlx5e_napi_poll), which in turn call RX helpers like e1000_clean_rx_irq / mlx5e_poll_rx_cq.

Background 3: What XDP is

Stage 2 is built around XDP, and Day 27 covers it in full. But the RX narrative is load-bearing on the basics, so here’s just enough to follow the handoff.

XDP (eXpress Data Path) is an eBPF program the driver runs on the raw RX frame, before any sk_buff exists — the earliest possible software hook. Because it runs pre-skb, it can drop or redirect at line rate without paying the cost of skb allocation (the slab work and refcount machinery you learned in Day 1). That’s the whole point: a DDoS-drop or load-balancer redirect that never even builds a packet object.

What does XDP look at? An xdp_buff — a lightweight, stack-allocated descriptor over the DMA region (include/net/xdp.h:86):

struct xdp_buff {
    void *data;            /* first byte of the frame */
    void *data_end;        /* one past the last byte */
    void *data_meta;       /* scratch metadata area */
    void *data_hard_start; /* start of the buffer */
    /* ... */
};

Contrast that with the sk_buff you spent Day 1 on: no slab allocation, no users/dataref refcounts, no headroom-reservation machinery — just four pointers into the DMA page. The xdp_buff is thrown away the moment XDP returns. It’s a temporary lens over bytes, not a packet object.

An XDP program returns one of five action codes (enum xdp_action, include/uapi/linux/bpf.h:6548):

enum xdp_action {
    XDP_ABORTED = 0,   /* error path — drop + tracepoint */
    XDP_DROP,          /* free the frame in-driver, no skb ever built */
    XDP_PASS,          /* the ONLY code that proceeds to build an sk_buff */
    XDP_TX,            /* bounce the frame back out the same NIC */
    XDP_REDIRECT,      /* send to another device / CPU / AF_XDP socket */
};

XDP_ABORTED, XDP_DROP, XDP_TX, and XDP_REDIRECT all consume the frame right there — the normal stack never sees it. XDP_PASS is the single code that says “turn this into a real packet” — only then does the driver build an skb and enter the path the rest of this chapter describes. Stage 2’s entire handoff hinges on that one return value.

Native vs. generic. Native XDP runs inside the driver’s poll on the DMA buffer, pre-skb — fast, the design intent. Generic XDP (do_xdp_generic → netif_receive_generic_xdp, net/core/dev.c:5656/:5576) is a fallback for drivers without native support: it runs later, in the core stack, after an skb already exists. It’s slower (you’ve already paid for the skb) — which is why the chapter notes generic XDP “runs later.” Full XDP programming, maps, and AF_XDP are Day 27; this is just enough to read the RX handoff.

Stage 2: Driver → native XDP → skb → GRO

Inside the driver’s poll, for each completed RX descriptor (recall from Day 1 — the descriptor’s DONE bit is set and the frame is already DMA’d into its page):

1. Build an xdp_buff view of the DMA buffer. Native XDP runs while the packet is still just bytes in driver-owned RX memory — no sk_buff has been allocated yet (Background 3).
2. Call XDP if attached. XDP_DROP, XDP_TX, and XDP_REDIRECT consume the packet at the driver/XDP layer. Only XDP_PASS says, “turn this into a normal kernel packet.”
3. Wrap the DMA buffer in an skb. After XDP_PASS, modern drivers use build_skb / napi_build_skb (recall from Day 1: zero-copy — the driver already DMAed bytes into a page, so the skb’s head/data/tail just point at it and head_frag = 1). Generic XDP is the exception: it runs later on an already-created skb (net/core/dev.c:5576).
4. Set skb->protocol via eth_type_trans (net/ethernet/eth.c:155) — it resets mac_header to point at the Ethernet header (skb_reset_mac_header), pulls the 14-byte Ethernet header off data so data now points at the L3 header (eth_skb_pull_mac → skb_pull ETH_HLEN), and returns the EtherType. Hold that thought: the value it returns becomes the demux key in Stage 3.
5. Pass to GRO: the driver calls napi_gro_receive(napi, skb), a static inline in include/linux/netdevice.h:4286 that funnels into gro_receive_skb(&napi->gro, skb) (and dev_gro_receive) on the napi’s GRO accumulator. What GRO merges is the next section (Background 4).

Background 4: what GRO merges, and where 64KB lives

GRO (Generic Receive Offload) tries to merge consecutive segments of the same flow into one big skb before the stack sees it. A 64KB GRO superpacket means one trip up the stack instead of 40-something — you pay the route lookup, netfilter, and socket delivery once instead of per-segment.

Two prerequisites the bare phrase “merge same-flow segments” hides:

• What counts as “the same flow”? For GRO it’s the L3/L4 identity: matching src/dst IP, protocol, and TCP ports, with in-order, contiguous sequence numbers and compatible flags. gro_list_prepare (net/core/gro.c:355) builds the comparison key, and per-protocol callbacks (tcp4_gro_receive, net/ipv4/tcp_offload.c:419) make the final mergeability call. A packet with mismatched flags (PSH/FIN) or a sequence gap is not merged — it flushes the held skb and starts fresh. So GRO is comparing flow identity and sequence continuity, not just “two TCP packets.”

• How does 64KB fit without a 64KB allocation? It doesn’t grow a linear buffer — each merged segment’s payload is appended as page fragments onto the held skb. (Refresher only — this is exactly the linear-head + MAX_SKB_FRAGS page-fragment design from Day 1; a 64KB superpacket is a tiny linear header plus a chain of page frags, no contiguous 64KB ever allocated.)

The accumulator is napi->gro (a struct gro_node, include/linux/netdevice.h:358). It flushes — handing the assembled skb onward via netif_receive_skb — when any of these happen: NAPI budget/weight is exhausted, gro_normal_one is called (net/core/gro.c:299), a non-mergeable packet arrives, or the GRO flush timeout fires (__gro_flush, net/core/gro.c:324).

Code to read: net/core/gro.c — dev_gro_receive (the workhorse, :474), the exported gro_receive_skb, gro_list_prepare, gro_complete. Tracing caveat: napi_gro_receive itself is a static inline in include/linux/netdevice.h (:4286), so it is not fentry-traceable — attach to gro_receive_skb instead.

Background 5: packet_type demultiplexing

Stage 3 says __netif_receive_skb_core “calls each registered packet_type” and “pt_prev->func dispatches to the L3 handler.” To read that you need to know what a packet_type is, how the kernel picks ip_rcv out of a list, and what the pt_prev trick buys.

A packet_type is a registration record

When a protocol wants to receive packets, it registers a struct packet_type (include/linux/netdevice.h:2968):

struct packet_type {
    __be16  type;        /* EtherType, e.g. htons(ETH_P_IP) — the demux key */
    /* ... */
    int     (*func)(struct sk_buff *, struct net_device *,
                    struct packet_type *, struct net_device *);  /* the callback */
    /* ... */
    struct list_head list;
};

It pairs an EtherType (type) with a callback (func). IPv4 registers one in net/ipv4/af_inet.c:1881:

static struct packet_type ip_packet_type __read_mostly = {
    .type = cpu_to_be16(ETH_P_IP),
    .func = ip_rcv,                  // af_inet.c:1883
};
...
dev_add_pack(&ip_packet_type);       // af_inet.c:2006

So ip_rcv isn’t called by name from the core — it’s looked up by EtherType. Where does that EtherType come from? eth_type_trans in Stage 2. It returned the frame’s EtherType into skb->protocol, and that value is the key the core hashes on now. That’s the connection Stage 2 set up.

Two lists: ptype_all and ptype_base

The kernel keeps protocol receivers in two structures:

• ptype_all — receivers that want every packet regardless of type. This is how AF_PACKET sockets and tcpdump tap the path.
• ptype_base[] — a hash table keyed by EtherType (net/core/dev.c:172), where exactly-one-type receivers live: ip_rcv (ETH_P_IP), the IPv6 handler, ARP. The bucket is ptype_base[ntohs(pt->type) & PTYPE_HASH_MASK] (net/core/dev.c:608).

__netif_receive_skb_core walks ptype_all first (taps), then indexes ptype_base by skb->protocol to find the one real L3 handler.

The pt_prev deferral trick

Watch how the core dispatches. Instead of calling each matching handler immediately, it remembers the last match in a pointer pt_prev, and only actually invokes a handler when it discovers a subsequent taker exists. The invoke goes through deliver_skb (net/core/dev.c:2485):

static int deliver_skb(struct sk_buff *skb, struct packet_type *pt_prev,
                       struct net_device *orig_dev)
{
    ...
    refcount_inc(&skb->users);                        // bump the descriptor refcount
    return pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
}

Why the indirection? Because deliver_skb does a refcount_inc(&skb->users) (the descriptor refcount from Day 1) for each receiver. In the overwhelmingly common case of exactly one receiver (just ip_rcv, no tcpdump running), the deferral lets the core call that single handler without the extra atomic bump — it knows there’s no other taker. This is precisely why Stage 3 dispatches via pt_prev->func() rather than calling ip_rcv directly.

Cross-link to Day 1: when there are multiple takers (e.g. tcpdump’s ptype_all socket and ip_rcv), deliver_skb gives each earlier taker another reference to the same skb descriptor by incrementing skb->users; it does not call skb_clone. The final taker consumes the original reference without that extra atomic increment.

Scope and batching beyond the global bucket

The two global structures are only the first layer. AF_PACKET adds scoped lists rather than making every socket another global protocol handler. ETH_P_ALL capture taps live in the receive network namespace’s ptype_all list or a device’s ptype_all; protocol-bound packet sockets can live in namespace/device ptype_specific lists. __netif_receive_skb_core visits all-protocol taps early, before ingress processing may redirect or change skb->dev. After ingress, VLAN, and any RX handler, it walks the global ethertype bucket and then the applicable namespace/original/current-device protocol-specific lists. ip_rcv is normally the final match from the global bucket.

The pt_prev deferral spans all of those walks. deliver_ptype_list_skb flushes the previous match with deliver_skb only when it discovers a subsequent one; deliver_skb increments skb->users before invoking that earlier callback. With one matching L3 handler and no tap, the final pt_prev reaches __netif_receive_skb_one_core, which dispatches it with INDIRECT_CALL_INET(..., ipv6_rcv, ip_rcv, ...) and avoids the extra reference operation entirely. Adding an ETH_P_ALL capture tap means the tap must be flushed before the later L3 consumer, which is one reason packet capture has measurable cost.

There is also a list receive path. IPv4 registers .list_func = ip_list_rcv alongside .func = ip_rcv. __netif_receive_skb_list_core classifies each skb through the same core, groups adjacent packets with the same final packet_type and original device, then calls that list callback. Handlers without one fall back to per-skb .func calls. The scalar deferral and list batching are two layers of the same optimization: avoid unnecessary references for one consumer, then preserve batches for a consumer that can process them together.

If no handler matches, pt_prev remains NULL, dev_core_stats_rx_dropped_inc records the unhandled protocol, and kfree_skb_reason releases the packet.

Stage 3: GRO → netif_receive_skb → __netif_receive_skb_core

When NAPI’s poll budget is exhausted (or gro_normal_one is called), accumulated GRO superpackets are flushed via netif_receive_skb:

int netif_receive_skb(struct sk_buff *skb)        // net/core/dev.c:6454

Which calls into:

static int __netif_receive_skb_core(struct sk_buff **pskb, bool pfmemalloc,
                                    struct packet_type **ppt_prev)  // line 5972

This is the function that does most of the work, and with Background 5 in hand every line of it makes sense:

• VLAN/ingress hook handling (__skb_push to re-add VLAN if hardware-stripped).
• Calls each registered packet_type — walk ptype_all (tcpdump’s AF_PACKET socket, AF_BRIDGE, etc.) then the ptype_base bucket for skb->protocol.
• tc ingress hooks run here (tcx/tc-bpf ingress).
• pt_prev->func() dispatches to the L3 protocol handler via the deferral trick.

For IPv4 packets, pt_prev->func is ip_rcv — registered statically in net/ipv4/af_inet.c as static struct packet_type ip_packet_type (Background 5).

Stage 4: ip_rcv and netfilter

int ip_rcv(struct sk_buff *skb, struct net_device *dev,
           struct packet_type *pt, struct net_device *orig_dev)  // net/ipv4/ip_input.c:603
{
    struct net *net = dev_net(dev);
    skb = ip_rcv_core(skb, net);
    if (skb == NULL)
        return NET_RX_DROP;
    return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING,
                   net, NULL, skb, dev, NULL,
                   ip_rcv_finish);
}

ip_rcv_core (net/ipv4/ip_input.c:499) does sanity checks (IP header length, version, checksum if not hw-validated) and trims the skb to the IP header’s claimed tot_len. Then NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, ...) runs all netfilter chains on PREROUTING (this is where iptables/nftables/conntrack get involved). Day 20 covers netfilter in detail.

If the hooks pass the skb, ip_rcv_finish (net/ipv4/ip_input.c:478) is called. It does the route lookup (ip_route_input_noref → FIB lookup) and then dst_input(skb), which calls ip_local_deliver (net/ipv4/ip_input.c:250) for local sockets, or ip_forward for routed traffic.

Where BPF can attach

The kernel exposes BPF hook points at four positions along the RX path: XDP at the driver (Background 3 — pre-skb), tcx at __netif_receive_skb_core, fentry on ip_rcv itself, cgroup_skb after socket lookup. We won’t write any BPF today — this map is here so the attach points have a clear place in this path. Today’s labs use one-liner BPF tools (bpftrace) only as an inspection mechanism, not for authoring.

There are no Dumb Questions

Q: If raising a softirq just sets a bit, what actually runs net_rx_action?

A: __do_softirq (kernel/softirq.c:654). It’s invoked on the way out of interrupt context (and a few other points), checks the per-CPU pending mask, and calls each pending vector’s action function. The top half only set the NET_RX_SOFTIRQ bit and returned; __do_softirq does the draining moments later. If it has to restart more than MAX_SOFTIRQ_RESTART (10) times, the rest is handed to ksoftirqd.

Q: I keep seeing “budget.” Which budget — 64 or 300?

A: Both, at different scopes. weight (64) is per single ->poll call — the cap passed into the driver. netdev_budget (300) is per net_rx_action softirq run, shared across all napis polled in that run. One softirq run can poll several queues; each individual poll claims at most 64, and the run stops once the cumulative 300 (or the time limit) is hit — bumping time_squeeze if work remains.

Q: Why does __netif_receive_skb_core dispatch through pt_prev->func() instead of just calling ip_rcv?

A: The pt_prev deferral (Background 5). It defers each delivery so it can skip an atomic skb->users bump in the common single-receiver case. ip_rcv is reached by EtherType lookup in ptype_base, not by name — the EtherType being the value eth_type_trans stored in skb->protocol back in the driver.

Q: An XDP program returned XDP_PASS. What changed?

A: Nothing yet — XDP_PASS is the only action that lets the driver proceed to build an sk_buff and enter the normal stack. The other four (ABORTED/DROP/TX/REDIRECT) consumed the frame while it was still just an xdp_buff over the DMA page, with no skb ever allocated.

Today’s experiment

Trace a real packet’s path.

Use ftrace to see the call chain

trace-cmd is not installed by default — sudo apt-get install -y trace-cmd (it needs CONFIG_FUNCTION_GRAPH_TRACER, on by default in typical kernels). This needs two terminals: the recorder blocks for 5 seconds, and you must fire the packet during that window.

In terminal 1, start recording:

sudo trace-cmd record -p function_graph \
    -g netif_receive_skb \
    -e net:netif_receive_skb \
    -O nofuncgraph-overhead \
    -O funcgraph-tail \
    sleep 5

In terminal 2, within those 5 seconds, generate one packet:

ping -c 1 8.8.8.8

After the recorder exits, render the trace:

sudo trace-cmd report | head -100

You’ll see the function-call tree: netif_receive_skb → __netif_receive_skb_one_core → __netif_receive_skb_core.constprop.0 → ip_rcv → ip_rcv_core → nf_hook_slow → ip_rcv_finish_core → ip_local_deliver → icmp_rcv. (The compiler appends a .constprop.N/.isra.N suffix to some of these symbols, and the outer ip_rcv_finish wrapper is inlined so what you actually see in the trace is ip_rcv_finish_core.) The leaf is icmp_rcv because a ping echo reply is an ICMP packet — icmp_rcv then calls icmp_echo. (Trigger a TCP flow instead — e.g. curl -s http://example.com >/dev/null — and the leaf becomes tcp_v4_rcv.) The pt_prev->func() dispatch (Background 5) goes through deliver_skb, but on this kernel deliver_skb is inlined into __netif_receive_skb_core so it does not show up as its own node in function_graph — don’t go hunting for it. To observe the dispatch directly, attach a kprobe instead: sudo bpftrace -e 'kprobe:deliver_skb { @[comm]=count(); }'.

Or use BPF for a custom view

sudo bpftrace -e '
fentry:ip_rcv { @ip[args->skb->dev->name] = count(); }
fentry:tcp_v4_rcv { @tcp[args->skb->dev->name] = count(); }
fentry:udp_rcv { @udp[args->skb->dev->name] = count(); }
interval:s:6 { exit(); }' &

# Generate receives during the window, then let it exit:
ping -c 5 -i 0.3 8.8.8.8 >/dev/null; curl -s http://example.com >/dev/null
wait

Per-protocol receive counts per interface. Typical output:

@ip[lo]: 4
@tcp[eth0]: 21
@udp[eth0]: 2
@udp[lo]: 4

@tcp/@udp are the reliable signal. Note the @ip map: on some virtual NICs (cloud/virtio) fentry:ip_rcv attaches but may not fire for the physical interface — you may see only @ip[lo] (or nothing) even with eth0 traffic flowing. On other boxes @ip[eth0] populates normally. Either way it’s a tracing-environment quirk, not a missing-packet problem; trust @tcp[eth0]/@udp[eth0] to confirm receives are happening.

Inspect the per-CPU RX state

cat /proc/net/softnet_stat

One line per CPU — these are the per-CPU softnet_data counters from Background 2. Every field is a 32-bit counter printed in hexadecimal (zero-padded %08x) with no header line — don’t read the values as decimal. In order the columns are: packets processed, dropped, time_squeeze (budget exhaustions), then several zeros, with received_rps near the end (exact trailing columns are kernel-version-dependent — see net/core/net-procfs.c:145). Convert one to decimal with printf '%d\n' 0x<value>. High time_squeeze means a softirq run kept hitting budget <= 0 (or the time limit) with work still pending — i.e. your netdev_budget is too small for the load.

Adjust the budget. Set it (in its own step, so you can observe the box running at the new value):

old_budget=$(cat /proc/sys/net/core/netdev_budget)
echo 600 | sudo tee /proc/sys/net/core/netdev_budget
cat /proc/sys/net/core/netdev_budget   # confirm it changed

Then, under sustained RX load (e.g. iperf3 -c <host> -P 16 from another box, or a packet flood), re-read /proc/net/softnet_stat repeatedly and watch the time_squeeze column. Be honest with yourself about what you’ll see: on an idle host time_squeeze never moves — it only increments when a softirq actually exhausts its budget under heavy receive load, and even under load it can stay flat on fast CPUs / multi-queue NICs. A non-moving counter is normal, not a sign the change failed (you already confirmed the change with the cat above).

Restore the original budget so the host isn’t left with changed RX scheduling behavior:

echo "$old_budget" | sudo tee /proc/sys/net/core/netdev_budget

What to read in the kernel

• kernel/softirq.c — the softirq engine. __do_softirq (line 654), raise_softirq_irqoff (line 773) → __raise_softirq_irqoff (line 799), open_softirq (line 806), run_ksoftirqd (line 1068), and #define MAX_SOFTIRQ_RESTART 10 (line 544).
• net/core/dev.c — the central RX dispatch.
  • ____napi_schedule (line 4957) — adds to poll_list (line 4984), raises NET_RX_SOFTIRQ (line 4990). (The NAPI_STATE_SCHED bit is set earlier, by napi_schedule_prep at line 6736.)
  • net_rx_action (line 7914) — the softirq loop; reads the 300 budget (line 7920).
  • __napi_poll (line 7719) — per-NAPI poll; reads weight and calls ->poll.
  • __netif_receive_skb_core (line 5972) — the workhorse; the packet_type walk.
  • netif_receive_skb (line 6454) — entry from drivers/GRO.
  • deliver_skb (line 2485) — the pt_prev->func() dispatch + users bump.
  • napi_complete_done (line 6771) — clears SCHED, returns true so the driver can re-arm its IRQ.
• net/core/hotdata.c — netdev_budget = 300 (line 14), netdev_budget_usecs (line 16).
• net/core/gro.c — GRO machinery. Read dev_gro_receive (line 474), gro_receive_skb, gro_list_prepare (line 355), gro_complete, __gro_flush (line 324). (napi_gro_receive is a static inline in netdevice.h:4286, not here — not fentry-traceable.)
• net/ipv4/ip_input.c — IPv4 receive. ip_rcv (line 603), ip_rcv_core (line 499), ip_rcv_finish (line 478), ip_local_deliver (line 250).
• net/ipv4/af_inet.c — search ip_packet_type (line 1881), see how ip_rcv is registered via dev_add_pack (line 2006).
• include/linux/netdevice.h — struct napi_struct (line 381), NAPI_POLL_WEIGHT 64 (line 2839), struct packet_type (line 2968), struct gro_node (line 358).
• include/linux/interrupt.h — the softirq vector enum (line 550).
• include/net/xdp.h — struct xdp_buff (line 86); include/uapi/linux/bpf.h — enum xdp_action (line 6548).

Bullet Points

• A softirq is a statically-defined, per-CPU bottom half. The top half (hardware IRQ) only acks the device and raises a pending bit; __do_softirq later runs the action. Networking’s RX vector is NET_RX_SOFTIRQ → net_rx_action. Overflow (>10 restarts) is punted to ksoftirqd so user space can’t starve.
• NAPI turns IRQ floods into one IRQ per burst: napi_schedule sets NAPI_STATE_SCHED + queues the napi (the driver masks the IRQ in its handler); napi_complete_done clears SCHED and returns true so the driver re-enables the IRQ when the ring drains. State machine driven by the NAPI_STATE_SCHED bit.
• Two budgets, not one: weight (NAPI_POLL_WEIGHT = 64) caps one ->poll call; netdev_budget (300) caps the total across all napis in one net_rx_action run. time_squeeze counts budget/time exhaustions with work still pending.
• Native XDP runs on an xdp_buff (4 stack pointers, no refcounts) before any skb exists; only XDP_PASS proceeds to build an skb. DROP/TX/REDIRECT/ABORTED consume the frame in-driver.
• build_skb wraps a pre-existing DMA buffer into an skb (zero-copy receive — recall Day 1).
• GRO (net/core/gro.c) coalesces consecutive same-flow segments (matching IP/proto/ports + contiguous sequence) into one skb whose payload is a chain of page frags (recall Day 1) — one trip up the stack instead of dozens.
• A packet_type pairs an EtherType with a callback; ip_rcv is registered as ip_packet_type and reached by hashing skb->protocol (the value eth_type_trans returned). ptype_all = taps (tcpdump); ptype_base[] = the L3 hash table. The pt_prev deferral skips an atomic users bump in the single-receiver case.
• The single most-touched RX function is __netif_receive_skb_core at net/core/dev.c:5972.
• After ip_rcv → netfilter PREROUTING → ip_rcv_finish → routing → ip_local_deliver or ip_forward.

Check question

Why does the kernel run softirqs (and thus most of the RX path) outside of hardware IRQ context?

Tomorrow

Day 3: the TX path. From sendmsg to the wire. Socket buffer accounting, queue disciplines, the driver’s ndo_start_xmit.

Day 3 — The TX path: from sendmsg to the wire

Today’s mission: trace a TCP packet from a userspace send() to the moment it leaves the NIC — and meet the four structures that gate it on the way out: the socket and its send buffer, the windows that decide what may go now, the qdisc that schedules it, and the neighbour entry that finally stamps on a MAC address. Total time: ~110 minutes.

The journey, mirrored

Yesterday’s RX path went wire → driver → IP → socket. Today’s TX path goes the other direction:

Six layers of work between your send() syscall and the bytes hitting the wire. Each one has a job and a place to fail.

But there’s a deeper asymmetry, and it’s worth naming up front because it reshapes how you read the code. The RX path is skb-centric. The TX path is sock-centric.

On RX, a packet arrived out of nowhere — the NIC DMA’d some bytes into a page, the driver wrapped them in an sk_buff (Day 1), and that skb climbed the stack as a self-contained object. Nothing “owned” it until a socket claimed it at the very top.

On TX, the story starts the other way around. The bytes begin life inside a process that already has an open connection — a long-lived object that remembers the peer’s address, the sequence numbers, how much the network can take, how much the receiver will accept. That object is struct sock, and on the way out, every skb is something the sock produces and owns. So before we can read a single line of tcp_sendmsg, we need to meet the sock.

Stage 0: the connection object — struct sock

Day 1 taught you the sk_buff: one packet. Day 2 followed packets up the RX path. But neither ever introduced the thing that owns packets on a connection. That’s the gap we close first, because the entire TX path hangs off it.

Here’s the intuition. A TCP connection is not a packet — it’s a relationship that lives for seconds, minutes, or hours and carries thousands of packets. Somebody has to remember, between one send() and the next:

• who the peer is, and the route to reach it,
• the next sequence number to use,
• how many bytes are out on the wire unacknowledged,
• how much buffer the kernel is still allowed to spend on this connection,
• a chain of packets that are queued up but not yet (fully) sent.

That somebody is struct sock — “the sock.” Where an sk_buff is one envelope, a sock is the whole mailbox: a long-lived endpoint object that owns a list of skbs and all the bookkeeping around them.

The send-side fields we care about today all live in include/net/sock.h:

struct sk_buff_head  sk_write_queue;   /* include/net/sock.h:491 — the chain of queued skbs */
int                  sk_wmem_queued;   /* include/net/sock.h:484 — memory charged for unfreed send-side skbs */
int                  sk_sndbuf;        /* include/net/sock.h:526 — the cap on those bytes */

sk_write_queue is an sk_buff_head — the same doubly-linked list of skbs you saw on RX queues — except here it hangs off the socket and holds packets that TCP has built but not yet handed to the wire. tcp_sendmsg appends to its tail; tcp_write_xmit drains from its head.

How one sock reaches the right protocol code: the proto vtable

When userspace calls send(), the kernel has a generic struct sock in hand but needs to run TCP’s send logic — not UDP’s, not raw IP’s. It avoids a giant if (protocol == TCP) ladder with a classic trick: a vtable of function pointers.

Every sock carries sk->sk_prot, a pointer to a struct proto:

struct proto {                              /* include/net/sock.h:1291 */
        int  (*sendmsg)(struct sock *sk, struct msghdr *msg, ...);  /* :1321 */
        /* ...recvmsg, connect, close, ... */
};

For a TCP socket, sk_prot points at tcp_prot, whose .sendmsg slot is wired to tcp_sendmsg:

/* net/ipv4/tcp_ipv4.c:3353 */
.sendmsg = tcp_sendmsg,

So sock_sendmsg → inet_sendmsg → sk->sk_prot->sendmsg(...) lands in tcp_sendmsg with no branching. The exact same indirection reappears one layer down at L3: the connection keeps an icsk->icsk_af_ops->queue_xmit pointer (include/net/inet_connection_sock.h:36) that, for AF_INET sockets, is ip_queue_xmit. Keep this vtable picture in mind — “dispatch by pointer, not by if” is how the whole stack stays protocol-agnostic.

Stage 1: syscall to socket

When userspace calls send(fd, buf, len, 0) (or write(fd, ...) on a socket), the kernel walks:

sys_sendto / sys_write
  → sock_sendmsg
    → inet_sendmsg
      → sk->sk_prot->sendmsg     // dispatch by protocol (the vtable above)

For TCP that’s tcp_sendmsg at net/ipv4/tcp.c:1447. The function locks the socket, then calls tcp_sendmsg_locked (line 1117) which is where the real work happens.

Stage 2: copy and queue

tcp_sendmsg_locked does two distinct things:

1. Copy bytes from userspace into kernel skbs. Allocate skbs via tcp_stream_alloc_skb, copy data via copy_from_iter (or, with MSG_ZEROCOPY, pin the user pages and point skb frags at them — no copy; recall the page-fragment design from Day 1). Append each skb to sk->sk_write_queue.

2. Maybe trigger transmission. Calls tcp_push (net/ipv4/tcp.c:741) which eventually invokes tcp_write_xmit (line 2963 in tcp_output.c).

The split matters: queueing is cheap; sending is gated. Appending to sk_write_queue just costs a copy and a list insert. Actually putting bytes on the wire requires the congestion window to be open, the receive window to permit it, Nagle constraints to be satisfied (all three explained in the next stage). So one send() may queue 1 MB of bytes but only transmit 64 KB right now — the rest sits in sk_write_queue waiting for ACKs.

The send buffer: sk_wmem_queued vs sk_sndbuf

If a socket could queue unbounded data, a fast writer talking to a slow network would balloon kernel memory without limit. So every sock has a send-buffer cap, and accounting to enforce it:

• sk_wmem_queued — total memory (the sum of skb->truesize) charged for every send-side skb the kernel hasn’t freed yet. That spans two queues: skbs still unsent in sk_write_queue and skbs already transmitted but not yet ACKed, which in v7.1 have migrated to the separate retransmit queue tcp_rtx_queue (an rbtree). It’s freed only when the data is finally ACKed. Note this is a memory figure — it includes headroom and per-skb overhead, not just payload bytes.
• sk_sndbuf — the cap on that memory.

The check is a one-liner in the header:

static inline bool __sk_stream_memory_free(const struct sock *sk, int wake)
{                                                  /* include/net/sock.h:1413 */
        if (READ_ONCE(sk->sk_wmem_queued) >= READ_ONCE(sk->sk_sndbuf))
                return false;                      /* buffer full → not free */
        /* ... */
}

When the buffer is full, tcp_sendmsg_locked consults sk_stream_memory_free (net/ipv4/tcp.c:1248) and then either:

• blocks — sk_stream_wait_memory (net/ipv4/tcp.c:1405) sleeps the calling thread until ACKs free space (default, blocking socket); or
• returns EAGAIN — for a non-blocking socket, so an event loop can come back later.

sk_sndbuf isn’t a fixed number — the kernel auto-tunes it per socket between the bounds of the net.ipv4.tcp_wmem (min, default, max) sysctl, growing it for connections that can productively use more. This is exactly the accounting ss -tim surfaces: the w value inside skmem:(...) is sk_wmem_queued (charged memory) and tb is sk_sndbuf. The Send-Q column is a separate quantity — write_seq - snd_una, the count of unacknowledged sequence (payload) bytes — closer in spirit to the in-flight/unsent backlog than to the memory counter. The two correlate but are measured in different units (payload sequence bytes vs charged memory, which includes per-skb overhead). Lab 2 makes them move.

Stage 2.5: what actually gates transmission

Stage 2 said sending is “gated” by three things. Those three terms — congestion window, receive window, Nagle — get a full treatment later (congestion control is Days 16–17), but you can’t read tcp_write_xmit today without a working picture, so here it is in one pass. The goal is narrow: understand why one send() can queue 1 MB yet transmit far less right now.

Think of the path to the peer as a pipe. Two separate limits decide how much may be “in the pipe” (sent but not yet acknowledged) at once — and, crucially, Linux measures them in different units and enforces them with two separate tests, never a single min():

• Congestion window — snd_cwnd (include/linux/tcp.h:225), measured in packets (MSS-sized segments — initial value TCP_INIT_CWND = 10 packets, not a byte count like the textbook cwnd). The sender’s own estimate of how many segments the network path can hold in flight without dropping. Nobody negotiates it; the sender grows it as ACKs arrive and shrinks it on loss. It’s the sender protecting the network from itself. Enforced by tcp_cwnd_test (net/ipv4/tcp_output.c:2323): in-flight packets must be below snd_cwnd.

• Receive window — snd_wnd (include/linux/tcp.h:223, commented “The window we expect to receive”), measured in bytes. The amount of buffer the peer advertised it can accept. This is flow control — it stops a fast sender from overrunning a slow receiver. Enforced by tcp_snd_wnd_test (net/ipv4/tcp_output.c:2380): the segment’s end sequence number must not pass snd_una + snd_wnd (tcp_wnd_end).

The rule that combines them:

A segment may go out only if it passes both independent gates: in-flight packets < snd_cwnd (the congestion test, in packets) and its end sequence ≤ snd_una + snd_wnd (the flow-control test, in bytes).

New data may go out only while there’s room under both tests. As ACKs return, in-flight drops, the gates open, and tcp_write_xmit releases more from sk_write_queue. The in-flight quantity the congestion test uses is tcp_packets_in_flight(tp) = packets_out − (sacked + lost) + retrans (include/net/tcp.h:1502). ss -tim’s unacked column (= tp->packets_out) approximates it; the two are equal only on a clean connection with no SACKed, lost, or retransmitted segments.

The third gate is Nagle’s algorithm. Intuition: if an app does many tiny writes (think a character at a time over SSH), naively each becomes its own minimum-size segment and floods the wire with “tinygrams,” mostly header. Nagle says: don’t send a new small segment while a previous small one is still unacknowledged — coalesce instead. Whether it’s disabled is recorded in tp->nonagle (include/linux/tcp.h:291); TCP_NODELAY sets it to “off” for latency-sensitive apps.

All three are applied in one loop:

/* net/ipv4/tcp_output.c:2963 */
static bool tcp_write_xmit(struct sock *sk, unsigned int mss_now, int nonagle, ...)

tcp_write_xmit walks sk_write_queue from the head and, for each skb, asks: does cwnd allow it? does snd_wnd allow it? does Nagle allow it? (And how big a TSO chunk to cut — Day 4.) It stops at the first skb the gates reject; the rest stay queued. The connection even records whether it was the cwnd that stopped it, in is_cwnd_limited (include/linux/tcp.h:234).

Forward reference: the mechanics of how cwnd grows and collapses (slow start, congestion avoidance, the CUBIC/BBR algorithms) are Phase 3, Days 13–19. Today you only need: queue is cheap, transmit is gated, and ACKs are what reopen the gate.

What stops one flow from drowning the qdisc: TCP Small Queues

Here’s a tension the staging above quietly creates. One send() can charge a megabyte of TCP write memory, while tcp_write_xmit may have a large congestion window available. Why doesn’t one bulk flow push that entire window into the qdisc and device queues at once? TCP Small Queues (TSQ) applies per-socket backpressure before each new transmit.

First separate two lifetimes that are easy to collapse into one “send queue”:

• A newly built data skb starts on sk_write_queue and is charged to sk_wmem_queued.
• After its first successful transmission, tcp_event_new_data_sent() unlinks that original skb from sk_write_queue and inserts it into tcp_rtx_queue, where TCP retains it until ACK/free for possible retransmission. Its sk_wmem_queued charge persists across that move; tcp_wmem_free_skb() removes the charge only when TCP releases the skb.
• tcp_transmit_skb() normally sends a clone of that retained TCP skb down through IP. The downstream clone is charged to sk_wmem_alloc and gets tcp_wfree as its destructor. This second counter represents memory still held by qdisc/device-side references, not the bytes TCP retains in its write and retransmit queues.

TSQ uses the downstream counter as its gate:

1. Before sending another skb, tcp_small_queue_check() computes min(max(2 * skb->truesize, sk_pacing_rate >> sk_pacing_shift), tcp_limit_output_bytes). With the usual shift of 10, the pacing term is about one millisecond of bytes, while the two-skb floor prevents tiny limits. Retransmit paths can multiply the limit. There is also an escape for an empty or single-entry retransmit tree so delayed completions cannot deadlock progress.
2. If sk_wmem_alloc remains above the limit, TSQ sets TSQ_THROTTLED and tcp_write_xmit stops. Unsent originals stay on sk_write_queue; already-sent originals remain in tcp_rtx_queue. Neither queue is the qdisc backlog.
3. When a downstream skb is freed, tcp_wfree() removes its charge. If the socket was throttled, it queues the socket on a per-CPU BH workqueue so TCP can try tcp_write_xmit again outside the qdisc-locking context.

The payoff is bounded per-flow occupancy below TCP: one socket cannot dump an arbitrary congestion window into qdisc and driver queues. That usually keeps latency much lower, but it is not a promise that the whole qdisc is empty — many flows can still build aggregate backlog, and some devices (notably Wi-Fi aggregation) need enough queued bytes for throughput. BBR’s deliberate pacing rate feeds the TSQ limit, while a pacing qdisc such as fq separately schedules when packets leave.

Stage 3: TCP header and the IP layer

tcp_transmit_skb builds the TCP header (sequence number, ACK, flags, window, checksum if not offloaded) on the skb being sent. (tcp_transmit_skb is a thin wrapper inlined into __tcp_transmit_skb, so that’s the name you’ll actually see in ftrace output.) Then it calls into IP via the L3 vtable slot we met in Stage 0:

icsk->icsk_af_ops->queue_xmit(...)
  → ip_queue_xmit                  // net/ipv4/ip_output.c:546

ip_queue_xmit builds the IP header (after a route lookup if the route isn’t cached), sets TTL/DSCP, then:

ip_local_out                       // net/ipv4/ip_output.c:125
  → __ip_local_out
    → NF_HOOK(NFPROTO_IPV4, NF_INET_LOCAL_OUT, ...)
  → dst_output
    → ip_output                    // net/ipv4/ip_output.c:428
      → NF_HOOK(NFPROTO_IPV4, NF_INET_POST_ROUTING, ...)
      → ip_finish_output
        → ip_finish_output2        // net/ipv4/ip_output.c:200

Two netfilter hooks on the way out

You already met the NF_HOOK machinery on RX (Day 2, Stage 4, where NF_INET_PRE_ROUTING fires inside ip_rcv; full netfilter treatment is deferred to Day 20). The TX path has its own two hook points — same mechanism, different positions:

• NF_INET_LOCAL_OUT (include/uapi/linux/netfilter.h:46) fires in __ip_local_out, just after the IP header is set for locally-generated packets.
• NF_INET_POST_ROUTING (:47) fires in ip_output, after routing, just before the device — the last place iptables/nftables can touch the packet.

No need to re-learn what a hook is; just place these two on the path.

The last L3 step: turning a next-hop IP into a MAC

ip_finish_output2 is where the packet finally gets a link-layer destination — and this is a subsystem no earlier chapter has introduced, so here’s the background you need.

Routing produced a next-hop IP address (the gateway, or the destination if it’s on-link). But the NIC can’t send to an IP — Ethernet frames are addressed by MAC address. Something must map next-hop-IP → MAC. That something is the neighbour subsystem: ARP for IPv4, NDP for IPv6. It resolves the mapping once and caches it in a struct neighbour, so the second packet to the same next hop doesn’t pay for resolution again.

Two outcomes when ip_finish_output2 calls neigh_output (include/net/neighbour.h:547):

• Resolved (fast path). If the neighbour is reachable and has a cached hardware header, neigh_hh_output (include/net/neighbour.h:507) just prepends the precomputed Ethernet header and sends. This is the moment the L2 header is finally pushed onto the skb — and it’s free because tcp_stream_alloc_skb (net/ipv4/tcp.c:927) reserved MAX_TCP_HEADER headroom up front (alloc_skb_fclone(MAX_TCP_HEADER) then skb_reserve). MAX_TCP_HEADER bakes in MAX_HEADER/LL_MAX_HEADER — the link-layer header space — so there’s already room in front of data to push the Ethernet header with no reallocation. (This is the same principle as Day 1’s headroom reservation, but the TX-side constant is MAX_TCP_HEADER, not the RX allocators’ much smaller NET_SKB_PAD.)

• Unresolved (slow path). neigh_resolve_output (include/net/neighbour.h:364) parks the skb on the neighbour’s arp_queue and fires an ARP request. The packet is sent later, when the reply arrives — or dropped if resolution times out. This is why a brand-new flow to a fresh next hop can briefly stall right here.

Forward reference: the neighbour subsystem in full — ARP/NDP state machine, NUD_* states, garbage collection — is a Phase 2 topic (Days 6–12). Today you only need: this is where the MAC is resolved and the Ethernet header is built, and where a packet can stall.

Stage 4: the device queue and the qdisc

dev_queue_xmit (a wrapper for __dev_queue_xmit at net/core/dev.c:4766) is the boundary between L3 and the device layer. Three new ideas live here: an egress BPF hook, TX-queue selection, and the qdisc — the software packet scheduler. Let’s build them in the order the code hits them.

Step 1 — the egress BPF hook (one line)

First, __dev_queue_xmit runs sch_handle_egress (net/core/dev.c:4524, called at dev.c:4807), which runs any attached tcx / tc-bpf egress programs via tcx_run (dev.c:4439). This is simply the egress sibling of the tcx ingress attach point you saw on the RX path (Day 2, “Where BPF can attach”). It runs before queue selection and the qdisc. No new BPF to learn here.

Step 2 — pick a TX queue (multi-queue NICs)

A modern NIC doesn’t have one transmit path — it exposes many hardware TX queues so different CPUs can transmit in parallel without fighting over a single lock. Before touching any qdisc, __dev_queue_xmit must decide which TX queue (and therefore which root qdisc) this skb uses:

queue_index = netdev_pick_tx(dev, skb, sb_dev);   /* net/core/dev.c:4736 */

netdev_pick_tx (net/core/dev.c:4691) chooses in priority order:

1. an explicit skb->queue_mapping if something upstream set one (e.g. XPS or socket steering) — that wins;
2. otherwise a flow hash, so every packet of one connection lands on the same queue and TCP’s ordering is preserved.

This is exactly why the Lab 3 output shows an mq root with one fq_codel leaf per hardware queue: one qdisc per TX queue.

Step 3 — the qdisc: what it is and why it exists

Here’s the new structure. Between the IP/device layer and the driver sits a queueing discipline (qdisc) — a per-TX-queue software FIFO-plus-scheduler. It exists because the driver/NIC can’t always take a packet this instant (the ring may be full, or you may want to rate-limit or fairly interleave flows). The qdisc is the kernel’s buffer-and-scheduling stage in front of the hardware: it can hold packets, reorder them, rate-limit them, and pick which flow goes next.

The whole qdisc abstraction comes down to two function pointers in struct Qdisc_ops:

int             (*enqueue)(struct sk_buff *skb, struct Qdisc *q,   /* include/net/sch_generic.h:314 */
                           struct sk_buff **to_free);
struct sk_buff *(*dequeue)(struct Qdisc *q);                       /* include/net/sch_generic.h:317 */

• enqueue — the stack hands a packet in. __dev_queue_xmit calls q->enqueue(skb, q, &to_free) on the chosen queue’s root qdisc.
• dequeue — the pump pulls the next packet out to give the driver.

The pump is qdisc_run → __qdisc_run (net/sched/sch_generic.c:440), a loop that calls q->dequeue, then sch_direct_xmit (net/sched/sch_generic.c:344), then netdev_start_xmit, then the driver — draining the queue as fast as the device will accept.

So the per-queue stats tc -s qdisc prints map directly onto this machinery:

• backlog — packets currently sitting in the qdisc, enqueued but not yet dequeued.
• drops — packets the qdisc threw away (queue full, or an active-queue-management algorithm like CoDel decided to).
• requeues — packets handed back because the driver returned NETDEV_TX_BUSY (ring full); the qdisc holds them to retry.

On an idle box all three read 0. That’s not a bug — it means nothing is backing up. Lab 4 forces backlog non-zero on purpose so you can watch it move.

Default qdisc: built-in vs configured

There’s a subtlety worth getting right. The kernel’s compiled-in default qdisc is pfifo_fast — a simple 3-band priority FIFO:

/* net/sched/sch_generic.c:37 */
const struct Qdisc_ops *default_qdisc_ops = &pfifo_fast_ops;

But most distributions change the runtime default to fq_codel (fair-queuing with CoDel AQM) via the net.core.default_qdisc sysctl, handled by set_default_qdisc (net/core/sysctl_net_core.c:595). So:

Default qdisc on most modern systems is fq_codel (selected via the net.core.default_qdisc sysctl; the kernel’s built-in default is still pfifo_fast).

Day 23 covers qdiscs in detail.

The steps inside __dev_queue_xmit, summarized

1. tcx/tc-bpf egress hook runs first (before TX queue selection and the qdisc lookup).
2. Pick a TX queue. netdev_pick_tx uses the socket’s cached TX queue (sk_tx_queue_get), then XPS (get_xps_queue), then a flow hash (skb_tx_hash). Modern NICs have many TX queues for parallelism.
3. Find the root qdisc on that queue (txq->qdisc).
4. Enqueue: q->enqueue(skb, q, &to_free).
5. Pump: qdisc_run → __qdisc_run → q->dequeue → sch_direct_xmit → netdev_start_xmit → driver’s ndo_start_xmit.

Stage 5: driver and hardware

netdev_start_xmit(skb, dev, txq, more) (include/linux/netdevice.h:5371) calls dev->netdev_ops->ndo_start_xmit(skb, dev) (include/linux/netdevice.h:1441). Each driver implements this differently — but fundamentally it’s the mirror of Day 1’s RX descriptor ring, run in reverse: build a DMA descriptor, write it into the TX ring, ring the NIC’s doorbell register. The NIC’s DMA engine then reads the bytes straight out of the skb’s pages and puts them on the wire (recall: zero-copy, the payload pages are referenced, not copied).

Return value:

• NETDEV_TX_OK (include/linux/netdevice.h:135) — submitted to the NIC.
• NETDEV_TX_BUSY (include/linux/netdevice.h:136) — the TX ring is full, so the driver refuses the skb; the qdisc holds it and retries (this is the requeues counter from Stage 4).

There are no Dumb Questions

Q: Where does TSO/GSO fit into this?

A: Day 4. Briefly: TSO (TCP Segmentation Offload) lets the kernel hand a 64KB skb to the NIC, and the NIC chops it into MSS-sized packets. GSO (Generic Segmentation Offload) does the same in software when the NIC doesn’t support TSO. Both happen late — at the qdisc/driver boundary, not during tcp_sendmsg.

Q: Why does the kernel sometimes block sendmsg vs return EAGAIN?

A: Socket flag. Default sockets block (sleep until space) — that’s sk_stream_wait_memory from Stage 2. Non-blocking sockets (O_NONBLOCK) return EAGAIN instead. epoll-based servers use non-blocking + EPOLLOUT notifications to know when the send buffer has drained enough to retry.

Q: What about MSG_ZEROCOPY?

A: A flag for send(). The kernel pins the user pages, points skb fragments at them directly (no copy into sk_write_queue — the frags reference the user pages), and notifies userspace via the socket’s error queue when transmission completes. Useful for very large transfers; userspace must hold the buffers until the notification.

Today’s experiment

Lab 1 — Trace a TCP send all the way through

Run the trigger inside the recorded command so the send is guaranteed to fire during the capture window — no second terminal, no race against sleep:

sudo trace-cmd record -p function_graph \
    -g tcp_sendmsg \
    -O nofuncgraph-overhead \
    -O funcgraph-tail \
    bash -c 'echo hello | nc -q 1 example.com 80; sleep 1'

sudo trace-cmd report | head -200

Prerequisite: this needs outbound TCP reachability and a completed handshake during the capture — any host you can actually reach works. example.com:80 reliably accepts the connection; 8.8.8.8:80 is filtered on many networks, so the handshake never completes and the payload tcp_sendmsg never fires. -q 1 is a BSD/traditional nc flag — on nmap’s ncat, drop it (use nc example.com 80).

You’ll see the call tree from tcp_sendmsg down through tcp_write_xmit, tcp_transmit_skb, ip_queue_xmit, eventually dev_hard_start_xmit, then the driver’s xmit. trace-cmd is global, so unrelated sends (e.g. background sshd) may appear too — your nc send is the one that ends in the full ip_queue_xmit → ... → dev_hard_start_xmit chain.

Lab 2 — Watch socket buffer accounting

On an idle box ss -tim shows only your SSH session, with the buffer/window counters static and near zero — the send-buffer accounting from Stage 2 is invisible until something is actively transmitting. So generate a sustained send first, then snapshot:

# Sustained upload so the send buffer actually has bytes in flight:
curl -s -o /dev/null -T /dev/zero --max-time 4 https://speed.cloudflare.com/__up &
ss -tim                      # watch the uploading socket while curl runs

Per-socket you get: send buffer used, congestion window, RTO, retransmits. ss does not print a field literally named wmem_queued; it surfaces the charged send-side memory (the sk_wmem_queued from Stage 2) as the w value inside skmem:(...). The Send-Q column is a different quantity — write_seq - snd_una, unacknowledged payload sequence bytes — which tracks the same backlog but in payload bytes rather than charged memory. While curl runs, the uploading socket (to :https, not your idle SSH session) shows a large Send-Q and skmem w, plus cwnd, unacked, and pacing_rate:

ESTAB  0  2765014  10.0.0.4:36872  162.159.140.220:https
  skmem:(r0,rb131072,t0,tb4194304,f2858,w2811094,o0,bl0,d0) cubic wscale:13,10
  ... cwnd:1950 ... unacked:266 ... pacing_rate 1.42Gbps delivery_rate 120Mbps

unacked (= tp->packets_out) is segments sent but not yet ACKed — it approximates the in-flight quantity from Stage 2.5 that the congestion test bounds (the exact figure is tcp_packets_in_flight(); the two diverge under loss/SACK). The skmem w value is the memory charged for not-yet-freed send-side skbs (sk_wmem_queued). Those are exactly the Stage 2 / Stage 2.5 quantities the check question asks you to compare. (Your cc may read bbr instead of cubic depending on net.ipv4.tcp_congestion_control.) If you have no internet egress, a local sink works but won’t fully exercise the buffer cap (loopback has no bottleneck, so it drains as fast as it fills).

Lab 3 — Inspect qdisc statistics

tc -s qdisc show dev eth0

Drops, requeues, current backlog — the three counters from Stage 4. On an idle box every one of these reads 0 — that is expected, not a failure:

qdisc mq 0: root
 Sent 1279377437 bytes 1467116 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: parent :1 limit 10240p flows 1024 quantum 1514 ...
 Sent 895133146 bytes 775553 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

(On a multi-queue NIC the root is mq with one fq_codel leaf per hardware TX queue — exactly the netdev_pick_tx fan-out from Stage 4, step 2 — so you’ll see several stanzas.) The next experiment forces backlog non-zero so you can actually watch it move.

Lab 4 — Force a backlog and watch

Warning: this throttles all egress on eth0. If you are connected over that interface, your SSH/management traffic competes with the test transfer for the same 1mbit and the session may stall — run this on a throwaway VM or a non-management NIC.

First capture the qdisc that is actually there so you can restore it exactly (the live default here is mq, not fq_codel), then apply the rate limit:

ORIG=$(tc qdisc show dev eth0 | awk 'NR==1{print $2}')   # remember the real default
sudo tc qdisc replace dev eth0 root tbf rate 1mbit burst 32kbit latency 50ms
# now egress is rate-limited; large transfers back up at the qdisc

Drive traffic that actually leaves eth0. The backlog only grows for traffic egressing this device — a 127.0.0.1 transfer goes through the lo qdisc and shows backlog 0b 0p, so loopback is not a substitute. On a second machine run iperf3 -s, then here:

iperf3 -c <that-host-IP> -t 30 &
watch -n1 'tc -s qdisc show dev eth0'   # the `backlog ...b ...p` line climbs

No second machine or iperf3? Any sustained upload over eth0 backs up the qdisc the same way:

curl -s -o /dev/null -T /dev/zero --max-time 20 https://speed.cloudflare.com/__up &
watch -n1 'tc -s qdisc show dev eth0'

Restore (drop back to whatever was really there, and stop the transfer):

kill %1 2>/dev/null                       # stop the backgrounded transfer
sudo tc qdisc del dev eth0 root           # restores the kernel/runtime default
# or, to put back exactly what you captured: sudo tc qdisc replace dev eth0 root "$ORIG"

What to read in the kernel

• include/net/sock.h — struct sock send-side fields: sk_write_queue (491), sk_wmem_queued (484), sk_sndbuf (526); the struct proto vtable (1291) and its sendmsg slot (1321); __sk_stream_memory_free (1413).
• net/ipv4/tcp.c — tcp_sendmsg (line 1447), tcp_sendmsg_locked (line 1117), tcp_push (741), the send-buffer wait at 1248/1405. The core of TCP user-side semantics.
• net/ipv4/tcp_output.c — tcp_write_xmit (line 2963), tcp_transmit_skb. Decides what to send when, applying cwnd/snd_wnd/Nagle.
• include/linux/tcp.h — snd_wnd (223), snd_cwnd (225), is_cwnd_limited (234), nonagle (291).
• net/ipv4/ip_output.c — ip_queue_xmit (line 546), ip_local_out (line 125), ip_output (428), ip_finish_output2 (200) and its neigh_output call (237).
• include/net/neighbour.h — neigh_output (547), neigh_hh_output (507), neigh_resolve_output (364).
• net/core/dev.c — __dev_queue_xmit (line 4766), sch_handle_egress (4524), netdev_pick_tx (4691), the qdisc dance.
• net/sched/sch_generic.c — __qdisc_run (440), sch_direct_xmit (344), pfifo_fast_ops (942), the qdisc pump; the default_qdisc_ops = &pfifo_fast_ops line (37).
• include/net/sch_generic.h — Qdisc_ops.enqueue (314) / dequeue (317).
• include/linux/netdevice.h — ndo_start_xmit (line 1441), netdev_start_xmit (5371), NETDEV_TX_OK/NETDEV_TX_BUSY (135–136).
• Documentation/networking/scaling.rst — how multi-queue TX works.

Bullet Points

• TX is sock-centric where RX was skb-centric: a struct sock is the long-lived connection that owns a chain of skbs; an sk_buff is just one packet.
• TX path: sendmsg → tcp_sendmsg → tcp_write_xmit → tcp_transmit_skb → ip_queue_xmit → dev_queue_xmit → qdisc → driver → wire.
• Protocol dispatch is a vtable: sk->sk_prot->sendmsg is tcp_sendmsg; icsk->icsk_af_ops->queue_xmit is ip_queue_xmit. No if-ladder.
• sk_write_queue holds queued-but-not-yet-transmitted skbs; after transmission, TCP retains the originals in tcp_rtx_queue until ACK/free. sk_wmem_queued spans both queues and counts their charged memory (skb->truesize) against the sk_sndbuf cap. Full buffer ⇒ block (sk_stream_wait_memory) or EAGAIN. Auto-tuned by tcp_wmem.
• tcp_write_xmit decides what to send now based on cwnd, snd_wnd, Nagle, TSO size. A segment goes out only if it clears two independent gates: in-flight packets < snd_cwnd (congestion, in packets — tcp_cwnd_test) and its end sequence ≤ snd_una + snd_wnd (flow control, in bytes — tcp_snd_wnd_test). Linux’s snd_cwnd is in MSS-sized packets, so the two can’t be min()’d. ACKs reopen the gates. (Full congestion control: Days 16–17.)
• Two netfilter hooks on TX: NF_INET_LOCAL_OUT and NF_INET_POST_ROUTING.
• Neighbour resolution (ARP/NDP) happens in ip_finish_output2: resolved ⇒ neigh_hh_output prepends the cached Ethernet header (using Day 1’s headroom); unresolved ⇒ skb parked on arp_queue + ARP sent. (Full neighbour subsystem: Phase 2.)
• __dev_queue_xmit runs tcx/tc-bpf egress, picks a TX queue (netdev_pick_tx), then enqueues to that queue’s root qdisc.
• A qdisc is a per-TX-queue software FIFO+scheduler with enqueue/dequeue; __qdisc_run pumps it to the driver. backlog/drops/requeues are its counters. Multi-queue NICs show an mq root with an fq_codel leaf per queue.
• Default qdisc on modern Linux is fq_codel (via net.core.default_qdisc); the built-in default is still pfifo_fast.
• Driver’s ndo_start_xmit is the final hand-off: write the TX-ring descriptor, ring the doorbell, NIC DMAs out. NETDEV_TX_BUSY = ring full ⇒ qdisc requeues.

Check question

You call send(fd, buf, 1MB, 0) on a TCP socket whose RTT is 100ms and whose BDP — bandwidth-delay product, the bandwidth times the round-trip time, i.e. the most data that can usefully be in flight at once — is much smaller than 1MB. The send returns immediately with the full 1MB written. Where are those bytes physically right now?

Tomorrow

Day 4: GRO, GSO, TSO. Why a 64KB packet enters ip_rcv even though Ethernet’s MTU is 1500.

Day 4 — GRO, GSO, TSO: segmentation offloads

Today’s mission: understand why a single skb that enters ip_rcv may represent 40+ wire packets, and why the kernel writes one 64 KB skb on TX. You’ll meet the segmentation unit (MSS), the three-field GSO marker the kernel stamps into every big skb, and the checksum contract that lets hardware finish the job — then watch all of it move under bpftrace. Total time: ~110 minutes.

The problem

Ethernet has an MTU of 1500 bytes. Most NICs can handle up to ~9000 bytes (jumbo frames) but not more. So one TCP connection sending a gigabyte through the kernel produces ~700,000 wire packets.

If the kernel ran its full TCP send code path (queue, build header, route lookup, qdisc, driver — the entire Day 3 journey) once per wire packet, the per-packet overhead would crush throughput. Same on RX: 700,000 calls up the Day 2 receive path is too many.

Three offload technologies move this work to either hardware or batched software so the stack runs at aggregate rate, not per-segment rate.

But before any of them make sense, you need to know the size of the chunk they all count in. The intro promised “40+ wire packets from one skb” — that number isn’t magic, it’s arithmetic, and the unit underneath it is the MSS.

Background 1: MTU, MSS, and the segmentation unit

You already know MTU — Day 3’s TX path ended at the wire, and the intro above names it: the largest L2 payload the link will carry, 1500 bytes on standard Ethernet. That’s the cap on one frame’s contents.

But TCP doesn’t think in frames; it thinks in segments. So there’s a second number, derived from the MTU, that every offload in this chapter actually counts in: the MSS — Maximum Segment Size. It’s the largest TCP payload one segment may carry, and it’s just the MTU with the headers subtracted:

MSS = MTU − IP header − TCP header
    = 1500 − 20 − 20
    = 1460 bytes        (no-options IPv4)

That’s it. One Ethernet frame holds 1500 bytes of L2 payload; of those, 20 go to the IP header and 20 to the TCP header, leaving 1460 bytes of actual data. So the wire packet a full segment becomes is 1460 (data) + 20 (TCP) + 20 (IP) + 14 (Ethernet) = 1514 bytes on the wire.

Two things make MSS less of a constant than it looks:

• It’s per-connection, not global. MSS is negotiated at SYN time (each side advertises what it can receive) and recomputed when the path MTU changes (PMTU discovery). A connection over a tunnel, or one that hit an ICMP “fragmentation needed,” runs a smaller MSS. So there is no single system-wide MSS — each connection carries its own.
• The kernel stores the value the segmenter must use, per-skb. Because MSS is per-connection and can change mid-flight, the segmentation step does not recompute “MTU minus headers” on the fly. The TCP TX path writes the MSS that applies to this skb into the skb itself, and the segmenter reads it back out. (That stored field is gso_size — Background 2.)

The arithmetic the whole chapter rests on

Now do the calculation once, because it’s the multiplier behind every histogram in today’s lab. The kernel builds one big skb carrying up to ~64 KB of payload (we’ll pin down that bound in a moment). At MSS 1460:

65536 bytes / 1460 bytes-per-segment ≈ 44.9  →  ceil = 45 segments

So one 64 KB skb becomes ~44 wire packets — and the stack ran its TX path once to produce all of them. That is the “40+ wire packets from one skb” the intro promised, and the coalescing ratios you’ll watch under bpftrace are this number running in reverse on RX.

Where does the per-skb MSS live, and where does the 64 KB ceiling come from? Two anchors:

• The segmenter reads the per-skb MSS via tcp_skb_mss() (include/net/tcp.h:1214), which simply returns TCP_SKB_CB(skb)->tcp_gso_size — the value TCP stashed in the skb’s control block.
• The “one big skb” is bounded by GSO_LEGACY_MAX_SIZE = 65536u (include/linux/netdevice.h:2446) and GSO_MAX_SEGS = 65535u (netdevice.h:2445). That 65536 is the “64 KB” this chapter quotes everywhere.

Keep MTU and MSS straight. This chapter switches between “MTU-sized skbs” (what GSO outputs) and “MSS-sized chunks” (how TSO is described) within a couple of paragraphs. They name the same wire packet from two ends: an MSS-sized payload (1460 B) wrapped in headers is an MTU-sized frame (1500 B of L2 payload). When you read “MSS-sized chunk,” picture the 1460 bytes of data; when you read “MTU-sized skb,” picture the same thing plus its IP/TCP headers.

Background 2: the GSO marker — gso_size, gso_segs, gso_type

Here is the structure the whole chapter quietly depends on. When the kernel decides an skb should be segmented later, it doesn’t change the skb’s shape — it just leaves a three-field note in the one place every layer of the stack already carries: the skb_shared_info trailer you met on Day 1 (the thing that sits past the linear buffer’s end and holds nr_frags, frags[], frag_list).

Day 1 taught those fragment fields. It did not teach the three GSO fields living in the same struct. They are:

struct skb_shared_info {
        /* ... nr_frags, frags[] from Day 1 ... */
        unsigned short  gso_size;   /* include/linux/skbuff.h:598 — MSS each segment carries */
        unsigned short  gso_segs;   /* skbuff.h:600 — how many segments this becomes */
        struct sk_buff  *frag_list; /* skbuff.h:601 — the Day 1 frag-list chain */
        unsigned int    gso_type;   /* skbuff.h:606 — bitmask naming the protocol */
};

Read them as a recipe:

• gso_size = the MSS each output segment should carry (1460 in our example). This is the per-skb MSS from Background 1 — the value the segmenter reads instead of recomputing MTU minus headers.
• gso_segs = how many segments this one skb will become (~44). This is the number today’s “What to break/observe” bullets tell you to multiply by when you want true per-wire-packet counts.
• gso_type = a bitmask naming the protocol, so the right segmentation callback is chosen. The bits are defined in skbuff.h: SKB_GSO_TCPV4 = 1 << 0 (skbuff.h:669), SKB_GSO_DODGY = 1 << 1 (skbuff.h:672, “this came from an untrusted source, validate it”), SKB_GSO_TCPV6 = 1 << 4 (skbuff.h:679).

So when the prose says an skb is “marked with SKB_GSO_TCPV4,” it means gso_type has that bit set. And when later text says the path “detects the GSO marker,” the literal test is skb_is_gso(skb) (skbuff.h:5267), which in v7.1 returns skb_shinfo(skb)->gso_size — a non-zero gso_size is “this skb wants segmenting.”

Who writes the marker

The TCP transmit path stamps it, right before handing the skb down the Day 3 path. In tcp_transmit_skb:

skb_shinfo(skb)->gso_segs = tcp_skb_pcount(skb);   /* net/ipv4/tcp_output.c:1704 */
skb_shinfo(skb)->gso_size = tcp_skb_mss(skb);      /* tcp_output.c:1705 */

That’s the key move. TCP fills in gso_size = MSS and gso_segs = segment count, and then the same struct rides unchanged through ip_queue_xmit → __dev_queue_xmit → qdisc → driver, exactly the journey Day 3 described. The skb carries its own segmentation recipe the whole way down; nothing in between has to understand it.

Who reads the marker, and the TSO-vs-GSO switch

The decision happens at the very bottom of the Day 3 TX path, in validate_xmit_skb() (net/core/dev.c:4019), called from the qdisc dequeue / sch_direct_xmit path right before the driver. It asks one question — netif_needs_gso(skb, features) (dev.c:4036):

• NIC can segment this gso_type in hardware (TSO). The big skb is handed to the driver’s ndo_start_xmit as-is — hardware does the split. netif_needs_gso returns false; nothing else happens.
• NIC can’t (GSO). The kernel calls skb_gso_segment() (dev.c:4039) to split the skb in software before the driver sees it.

This is the single switch that unifies the three technologies this chapter presents as separate: gso_type + NIC capability. It’s the same big skb either way — the only question is whether hardware or software does the split. Hold that idea; everything below is a special case of it.

Zoom in on that fork — the big skb’s GSO marker is the only thing that decides hardware-vs-software:

Background 3: ip_summed and checksum offload

There’s one more contract a big skb carries, and the checksum Q&A at the end of this chapter leans on it entirely: skb->ip_summed, which records who is responsible for the checksum — the stack or the NIC. Day 2 mentioned “checksum if not hw-validated” inside ip_rcv_core but never defined the states, so here they are. There are exactly four (include/linux/skbuff.h:248–251):

#define CHECKSUM_NONE         0   /* nobody has computed/verified it */
#define CHECKSUM_UNNECESSARY  1   /* RX: NIC already verified — stack may skip */
#define CHECKSUM_COMPLETE     2   /* RX: NIC handed up a raw sum for the stack to fold in */
#define CHECKSUM_PARTIAL      3   /* TX: stack wrote the pseudo-header sum; NIC finishes it */

The kernel’s own header documents this contract in a long comment block (skbuff.h:98–136). Two of these states are RX-side (UNNECESSARY, COMPLETE) — the NIC telling the stack what it already checked. The one that matters for segmentation is the TX-side CHECKSUM_PARTIAL.

Why offloads require CHECKSUM_PARTIAL

Think about it: a GSO/TSO skb has no per-segment checksums yet — they can’t exist, because the segments themselves don’t exist until the split happens. So what does the stack put in the checksum field of the one big skb?

It puts the pseudo-header checksum (the part computed over the IP addresses, protocol, and length — everything except the payload), and sets ip_summed = CHECKSUM_PARTIAL. That state is a promise: “the checksum is half-done; whoever splits this skb will finish each segment’s checksum over its payload.” For TSO the NIC finishes it; for GSO the software segmenter finishes it during the split. The header spells out this coupling directly: if gso_type is SKB_GSO_TCPV4/V6, TCP checksum offload is implied, and ip_summed is CHECKSUM_PARTIAL (skbuff.h:239–242).

This is exactly why the TSO-vs-GSO switch from Background 2 also reads ip_summed. Look again at netif_needs_gso (include/linux/netdevice.h:5480):

return skb_is_gso(skb) && (!skb_gso_ok(skb, features) ||
        unlikely((skb->ip_summed != CHECKSUM_PARTIAL) &&
                 (skb->ip_summed != CHECKSUM_UNNECESSARY)));

A GSO skb whose checksum is not PARTIAL (or UNNECESSARY) can’t be safely handed to hardware as-is, so it’s forced down the software segmentation path. Hardware-capability and the checksum contract together pick TSO over GSO.

And it’s why the segmenter re-initialises the checksum field per segment. __skb_gso_segment() (net/core/gso.c:88) calls skb_cow_head() precisely so it can write a fresh ->check field into each new TCP/UDP header (gso.c:97, “We’re going to init ->check field in TCP or UDP header”): the original skb carried only the partial (pseudo-header) sum, so every segment needs its real checksum finished. That’s the concrete reason the Q&A below tells you to read __skb_gso_segment when you debug a checksum issue.

One more parallel: checksum offload is itself an ethtool feature (rx-checksumming / tx-checksumming), sitting right next to the segmentation features today’s lab toggles. They interact — disabling tx-checksumming can disturb segmentation, because a non-PARTIAL skb gets pushed off the TSO path.

TSO — TCP Segmentation Offload

Now TSO reads as a special case of the switch. The kernel hands the NIC one large skb (up to 64 KB — GSO_LEGACY_MAX_SIZE) whose gso_type has SKB_GSO_TCPV4 (or SKB_GSO_TCPV6) set and whose ip_summed is CHECKSUM_PARTIAL. Because the NIC’s feature flags say it can segment that gso_type, netif_needs_gso returns false and the big skb goes straight to ndo_start_xmit. The NIC’s hardware then:

1. Reads the skb’s TCP header as a template.
2. Walks the payload in MSS-sized chunks (= gso_size bytes each — Background 1).
3. For each chunk, builds an IP+TCP header (cloning the template, advancing the TCP sequence number, incrementing the IP id, finishing the checksum the CHECKSUM_PARTIAL promise left half-done), prepends the Ethernet header, transmits. (Fields like TTL are copied straight from the template — every segment of one flow carries the same TTL; only seq, IP id, length, and checksum change per segment.)

Result: ~44 wire packets, one kernel call to ndo_start_xmit. Per-packet stack overhead drops ~44×.

Enable/check:

ethtool -k eth0 | grep tcp-segmentation-offload
# tcp-segmentation-offload: on
ethtool -K eth0 tso off

(The feature is named tcp-segmentation-offload, not tso — grepping for tso matches nothing.)

NIC must support it (most modern NICs do).

GSO — Generic Segmentation Offload

Same big skb, same marker — but the NIC can’t segment this gso_type, so the split happens in software, late in the TX path. Recall the qdisc → ndo_start_xmit boundary from Day 3: the big GSO skb rides that path unchanged, and segmentation is bolted onto the very end of it.

1. TCP builds one big skb just like for TSO, stamping gso_size/gso_segs/gso_type (Background 2).
2. The skb travels down through tcp_transmit_skb, ip_queue_xmit, dev_queue_xmit as if it were a single packet — exactly the Day 3 journey.
3. Just before ndo_start_xmit, in validate_xmit_skb() (net/core/dev.c:4019), netif_needs_gso() (dev.c:4036) sees the NIC can’t do it and calls skb_gso_segment() (dev.c:4039) → __skb_gso_segment (net/core/gso.c:88), which splits the skb into a chain of MTU-sized skbs (the L2 split runs through skb_mac_gso_segment, gso.c:37; the TCP-specific callback is tcp_gso_segment, net/ipv4/tcp_offload.c:133).
4. The driver receives the chain and transmits each.

GSO is universal — works on any NIC. The CPU cost of segmentation is real but smaller than running the full stack per packet.

GRO — Generic Receive Offload

The receive-side counterpart. Recall the NAPI poll loop and the napi_gro_receive → gro_receive_skb funnel from Day 2: GRO runs inside the driver’s NAPI poll, after XDP_PASS turns DMA bytes into an skb and before the stack entry. (Day 2 also established why you trace gro_receive_skb and not napi_gro_receive — the latter is a static inline in include/linux/netdevice.h:4286, not fentry-traceable; the exported entry is gro_receive_skb at net/core/gro.c:636, and the workhorse is dev_gro_receive at gro.c:474.)

The GRO engine compares each new skb against a list of “in flight” same-flow skbs. If it can merge (consecutive sequence numbers, same flow tuple, no flag changes), it appends payload to the existing one as page fragments (skb_gro_receive, gro.c:92 — never by growing the linear data buffer). Recall the linear-head + page-frags model from Day 1: a 64 KB GRO superpacket is a tiny linear header plus a chain of page frags, never a 64 KB contiguous allocation. (There’s a second assembly form — skb_shared_info.frag_list (skbuff.h:601, tested by skb_has_frag_list() at skbuff.h:4206) chains whole skbs rather than the frags[] page array; some GSO/frag-list paths use it. One line is enough — the frags[] model is the one you’ll meet most.)

Result: one ip_rcv call for what was ~44 wire packets.

Flush triggers:

• Different flow arrives.
• Timeout (per-device /sys/class/net/<dev>/gro_flush_timeout, or per-NAPI via netlink).
• NAPI poll exits — the GRO accumulator lives per-NAPI (napi->gro), so it must drain before the softirq yields the CPU.
• Special flag (FIN, RST, PSH).

There are no Dumb Questions

Q: Why doesn’t TSO break TCP correctness?

A: Because the segmenting NIC produces correct wire packets — same as if the kernel did it. From a peer’s perspective, traffic is indistinguishable. Only the local stack saves work. Same for GRO: the merged skb has the same bytes as the original segments would; the local stack just sees them as one.

Q: When should I disable these?

A: For latency-sensitive measurements (HFT, microsecond timing, packet capture for forensics) — GRO can hold a packet briefly waiting for a possible merge buddy. Cost is usually <100 µs but visible. For per-packet observability (BPF programs that need to see wire packets) — GRO at the driver hides them.

Q: How does GSO interact with checksum offload?

A: Closely — and Background 3 already gave you the machinery. The short version: no per-segment sums exist before the split, so the stack ships CHECKSUM_PARTIAL (pseudo-header sum only) and the splitter finishes each one — the NIC for TSO, __skb_gso_segment for GSO. It’s also why netif_needs_gso forces a non-PARTIAL skb down the software path. Re-read Background 3 if a checksum bug bites.

Pitfalls when offloads are on

The biggest gotcha is observability. If you trace ip_rcv and count packets, you’re counting GRO superpackets, not wire packets. Multiply by skb_shinfo(skb)->gso_segs (Background 2 — the segment count the kernel stamped in) if you have it; or disable GRO during measurement.

For BPF programs that hook into the stack (XDP runs before GRO; tc-bpf runs after), this matters: XDP sees per-segment, tc sees coalesced.

Today’s experiment

See offload state

ethtool -k eth0 | grep -E "segmentation-offload|receive-offload"

Output (typical):

tcp-segmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: on

The feature names are tcp-segmentation-offload / generic-*-offload — they don’t contain the substrings tso/gso/gro, so a grep -E "tso|gso|gro" would match only unrelated lines and never print these. Match the real names. (large-receive-offload is the hardware LRO cousin of GRO; it may read off or off [fixed] on NICs that don’t support it.)

Watch GRO in action

GRO coalesces many wire packets into one superpacket, so the entry probe (gro_receive_skb) fires once per arriving segment, while the post-GRO probe fires far fewer times with much larger skbs. We watch both the lengths and the call counts:

sudo bpftrace -e '
fentry:gro_receive_skb {
  @gro_lengths = lhist(args->skb->len, 0, 65536, 8192);
  @gro_calls = count();
}
tracepoint:net:netif_receive_skb {
  @postgro_lengths = lhist(args->len, 0, 65536, 8192);
  @postgro_calls = count();
}
interval:s:8 { exit(); }' &

# While that window is open, pull a real bulk download (server-less, follows redirects).
curl -sL -o /dev/null --max-time 6 \
  https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img
wait

Run the download inside the bpftrace window, so background the tracer (&) and wait. Typical output (numbers scale with the transfer):

@gro_calls: 237409          <- once per arriving segment, mostly small
@gro_lengths:
[0, 8K)           219020 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8K, 16K)          11732 |@@                                                |
 ...
@postgro_calls: 30527       <- ~8x fewer: GRO merged ~8 segments per superpacket
@postgro_lengths:
[0, 8K)            11056 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8K, 16K)           7203 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[56K, 64K)          3260 |@@@@@@@@@@@@@@@                                    |

The story is in the two *_calls: gro_calls ≫ postgro_calls (here ~8:1) is the coalescing ratio, and postgro_lengths spreads into the big buckets — those are the merged superpackets the stack processes once instead of eight times.

Why netif_receive_skb and not ip_rcv? tracepoint:net:netif_receive_skb is the post-GRO entry into the stack and fires on every NIC. fentry:ip_rcv works on bare-metal NICs but on many virtual NICs (cloud/virtio) it attaches yet never fires — you’d see an empty histogram and wrongly conclude GRO is off. Use the tracepoint; add fentry:ip_rcv { @ip = lhist(args->skb->len,0,65536,8192); } too if you want to confirm it on bare metal.

Disable GRO and re-measure

Now turn GRO off and run the same observation. The contrast is the whole point:

sudo ethtool -K eth0 gro off

sudo bpftrace -e '
tracepoint:net:netif_receive_skb {
  @postgro_lengths = lhist(args->len, 0, 65536, 8192);
  @postgro_calls = count();
} interval:s:8 { exit(); }' &
sleep 1
curl -sL -o /dev/null --max-time 6 \
  https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img
wait

# Always restore — leaving GRO off slows every later experiment:
sudo ethtool -K eth0 gro on

With GRO off, postgro_calls jumps roughly back to the segment count (here ~233 000 vs ~30 000 with GRO on — an ~8x increase), and postgro_lengths collapses almost entirely into the [0, 8K) bucket: the stack now runs once per wire-sized packet instead of once per superpacket. That extra per-packet work is the CPU cost GRO was hiding.

Per-segment counter

TSO is a transmit offload: the stack hands the NIC large skbs and the NIC segments them onto the wire. So you need an upload to see it (a download’s TX side is just small ACKs). Watch the size of skbs entering the device queue while pushing data out:

sudo bpftrace -e 'fentry:__dev_queue_xmit {
  @tx_skb_len = lhist(args->skb->len, 0, 65536, 8192);
} interval:s:6 { exit(); }' &

# Server-less upload sink; --max-time bounds it, the timeout exit is expected (|| true).
curl -s -o /dev/null -T /dev/zero --max-time 4 https://speed.cloudflare.com/__up || true
wait

Typical output — a strong spike in the top bucket, the 64 KB GSO/TSO skbs the stack handed down:

@tx_skb_len:
[0, 8K)              206 |@@@@@@                                            |
[8K, 16K)            120 |@@@                                               |
 ...
[56K, 64K)          1660 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|

Each of those ~64 KB skbs becomes ~40+ MTU-sized wire packets (that’s gso_segs from Background 2), but the stack ran its TX path once per skb. You can see the multiplication in the NIC counters too — the wire-packet count climbs far faster than the skb count:

ethtool -S eth0 | grep -iE 'tx_packets|tx_queue.*packets'   # NIC's per-queue / aggregate wire packets
cat /sys/class/net/eth0/statistics/tx_packets               # kernel-level tx_packets

The field is named tx_packets / tx_queue_N_packets — not tx_pkts, which matches nothing on any driver. Counter layout is driver-specific: on virtualized NICs (e.g. Azure/mlx5 VF) the tx_queue_N_packets lines may read 0 and the real counts live in vf_tx_packets / cpuN_tx_packets.

What to read in the kernel

• net/core/gso.c — segmentation engine. ~300 lines. __skb_gso_segment is the entry (gso.c:88); skb_mac_gso_segment does the L2 split (gso.c:37); note the skb_cow_head call (gso.c:97) that re-inits each segment’s checksum field.
• net/core/gro.c — coalescing engine. ~800 lines. dev_gro_receive is the workhorse (gro.c:474); gro_receive_skb is the exported entry (gro.c:636); napi_gro_receive itself is a static inline in netdevice.h:4286; per-protocol callbacks (tcp4_gro_receive) live in protocol files.
• net/ipv4/tcp_offload.c — TCP-specific GRO/GSO callbacks; tcp_gso_segment at tcp_offload.c:133.
• include/linux/netdev_features.h — NETIF_F_GSO_*, NETIF_F_TSO_*, NETIF_F_GRO_* flags.
• include/linux/skbuff.h — the GSO marker in skb_shared_info: gso_size (598), gso_segs (600), gso_type (606); the SKB_GSO_* bits (SKB_GSO_TCPV4 at 669); the ip_summed contract (CHECKSUM_* at 248–251, documented 98–136).
• net/core/dev.c — the TX split site: validate_xmit_skb (4019), netif_needs_gso test (4036), skb_gso_segment (4039).
• Documentation/networking/segmentation-offloads.rst — official guide.

Bullet Points

• MSS = MTU − IP − TCP = 1500 − 20 − 20 = 1460 B (no-options IPv4) — the chunk size every offload counts in. It’s per-connection, stored per-skb so the segmenter reads it (tcp_skb_mss(), tcp.h:1214) instead of recomputing. 64 KB / 1460 ≈ 44 segments from one skb.
• The GSO marker lives in skb_shared_info: gso_size (MSS, skbuff.h:598), gso_segs (count, 600), gso_type (protocol bitmask, 606). TCP stamps it in tcp_transmit_skb (tcp_output.c:1704–1705); the same big skb then rides the Day 3 TX path unchanged.
• One switch unifies all three: gso_type + NIC capability. validate_xmit_skb → netif_needs_gso (dev.c:4019/4036) hands the whole skb to hardware (TSO) or calls skb_gso_segment (dev.c:4039) for a software split (GSO).
• ip_summed is the checksum contract (CHECKSUM_NONE/UNNECESSARY/COMPLETE/PARTIAL, skbuff.h:248–251). Offloads require CHECKSUM_PARTIAL on TX: the stack writes only the pseudo-header sum and each segment’s real checksum is finished after the split (NIC for TSO, __skb_gso_segment→skb_cow_head for GSO).
• TSO: hardware segments large skbs into wire packets. Saves stack overhead ~44×.
• GSO: software segmentation late in TX (net/core/gso.c:__skb_gso_segment, gso.c:88). Same stack savings as TSO; works on any NIC.
• GRO: receive-side coalescing in the NAPI poll (net/core/gro.c:dev_gro_receive, gro.c:474) — recall the Day 2 funnel. One stack pass per superpacket; payload appended as page frags (Day 1) or, less commonly, chained via frag_list.
• All three controlled with ethtool -K. Default ON for all three on modern NICs.
• Side effects: per-packet observability is wrong unless you disable GRO or multiply by skb_shinfo(skb)->gso_segs.
• For latency-critical workloads, sometimes GRO is disabled (sub-100µs latency floor).

Check question

You run iperf3 and for the first instant of a transfer the reported throughput briefly reads above the NIC’s line rate, then settles. What’s happening?

Tomorrow

Day 5: network namespaces. Why a single kernel can run dozens of independent network stacks at once, and how struct net makes it work.

Day 5 — Network namespaces and struct net

Today’s mission: understand what a namespace even is, how a process is bound to one, and how the kernel keeps multiple independent network stacks isolated. Then create a network namespace, give it a virtual interface, run a process inside it, and watch one get born and die under bpftrace. End of Phase 1. Total time: ~110 minutes.

What a namespace IS (before we touch the network kind)

Days 1–4 never said the word namespace, so let’s build the idea from scratch — it’s the one piece of background everything today rests on.

A namespace is a kernel-wide scoping mechanism. Take one global resource — say, “the list of network interfaces,” or “the set of mounted filesystems,” or “the table of process IDs” — and make it appear as N independent copies instead of one shared thing. Two processes in different namespaces of the same kind look at the same global resource and each sees a private, isolated view. It’s the same trick a hypervisor plays with whole machines, except the kernel does it per-resource, in software, with almost no overhead.

Linux has several kinds of namespace, each scoping a different resource:

A container is, at bottom, just a process whose mnt, pid, net, … pointers have been swapped to fresh namespaces. There is no “container” object in the kernel — there’s a process holding a bundle of namespace pointers. Today we care about exactly one of those pointers: net.

The network namespace scopes the whole stack. Each one has its own:

• list of network interfaces,
• routing tables,
• netfilter rules (the kernel’s packet-filtering/firewall framework; nftables is its modern ruleset),
• sockets and bound ports,
• proc/sysctl views,
• conntrack table (connection tracking — the kernel’s record of active flows, used by NAT and stateful firewalling).

This is what makes containers and VMs (with TAP/MACVTAP) practical. Every Docker/Podman/Kubernetes pod runs in its own netns. Linux can spin up tens of thousands of them at once.

How a process is bound to its namespaces: nsproxy

So where does a process keep the pointer that says “this is my netns”? Not directly in task_struct. Every task points at a small shared bundle called struct nsproxy, which holds one pointer per namespace kind:

/* include/linux/nsproxy.h:32 */
struct nsproxy {
	refcount_t count;
	struct uts_namespace *uts_ns;
	struct ipc_namespace *ipc_ns;
	struct mnt_namespace *mnt_ns;
	struct pid_namespace *pid_ns_for_children;
	struct net           *net_ns;        /* <-- the network namespace */
	struct time_namespace *time_ns;
	struct time_namespace *time_ns_for_children;
	struct cgroup_namespace *cgroup_ns;
};

The field that matters today is net_ns, a struct net *. Every task in the same netns shares the very same struct net * — that pointer is the namespace. The nsproxy itself is refcounted (count) and shared between tasks that happen to have identical namespace bundles, which is why fork without CLONE_NEW* is cheap: the child just bumps the parent’s nsproxy.

So the chain you’ll see in code is: task_struct → nsproxy → net_ns → struct net. When kernel code wants “the current task’s network namespace” it walks exactly that path (the helper is current->nsproxy->net_ns, usually behind sock_net(sk) or dev_net(dev)).

struct net — one per namespace

Each namespace is one struct net (defined at include/net/net_namespace.h:62). Kernel networking functions take struct net *net as a first-class parameter — anywhere a function needs a routing table, a netfilter chain, a port allocator, it’s looked up via net. (We’ll see why that one pointer is enough to reach every subsystem’s per-ns state when we get to pernet_operations below.)

The “default” namespace at boot is init_net (net/core/net_namespace.c:49: struct net init_net;). The kernel’s first netdev (loopback, then physical NICs) all start there. New namespaces are empty until you put devices in them.

The inode that names a namespace: ns_common and nsfs

Here’s the part that makes today’s first lab make sense. Embedded inside struct net is a small common header shared by all namespace kinds:

/* include/net/net_namespace.h:97 — inside struct net */
struct ns_common	ns;

And ns_common carries the namespace’s identity, an inode number:

/* include/linux/ns/ns_common_types.h:111 */
struct ns_common {
	struct { refcount_t __ns_ref; } ____cacheline_aligned_in_smp;
	u32 ns_type;
	struct dentry *stashed;
	const struct proc_ns_operations *ops;
	unsigned int inum;          /* <-- the unique identity */
	/* ... */
};

That inum is a unique inode number assigned when the namespace is created. Be careful about which identity it is, though. Internally, the kernel identifies a namespace by its struct net * pointer, not by inum. “Are these two in the same netns?” is answered by net_eq(net1, net2), which is literally return net1 == net2 (include/net/net_namespace.h:302) — a pointer compare — and same-netns checks throughout the stack (e.g. inet_bind_bucket matching) call net_eq on pointers. The inum is the userspace-facing name, surfaced through nsfs as net:[<inum>]. Because each live struct net is assigned a unique inum, userspace can compare two net:[...] strings to answer “same netns?” and get the right answer for currently-live namespaces — but that is a userspace convenience built on a 1:1 mapping, not the comparison the kernel itself performs. (Inums are also recycled once a namespace is freed — the ns_common comment notes inum is “quickly recycled for non-initial namespaces” — so an inum is a stable identity only among live namespaces, whereas the pointer is the authoritative live key.)

How do you see that number from userspace? Through nsfs, a tiny internal pseudo-filesystem whose only job is to expose namespaces as files. Each kind of namespace shows up as a magic symlink under /proc/<pid>/ns/. Reading the net one (readlink /proc/<pid>/ns/net) prints net:[<inum>]. Same inum ⇒ same namespace; different inum ⇒ different struct net. That’s the entire mechanism behind the metadata lab later in this chapter.

The boot namespace init_net gets a well-known, fixed inode number rather than a freshly allocated one — NET_NS_INIT_INO = 0xEFFFFFF9 (include/uapi/linux/nsfs.h:54), wired in through ns_init_inum() (include/linux/ns/ns_common_types.h:144). So on every Linux box, init_net’s net:[...] ends in the same number; only the namespaces you create get fresh ones.

There are no Dumb Questions

Q: Is the loopback per-ns?

A: Yes. Each ns gets its own lo. So 127.0.0.1 in ns A is unrelated to 127.0.0.1 in ns B. Two processes (one in each) can both bind() 127.0.0.1:80 without conflict.

Q: Can ns A see ns B’s traffic?

A: Only via connecting devices. Common patterns: a veth pair (two virtual NICs that act as a wire — one in each ns), a Linux bridge that interconnects veths in init_net, or a physical NIC moved to one ns.

Q: Where does shared state live?

A: Hardware (NICs themselves), slab caches (sk_buff allocations are kernel-wide — recall the skbuff_head_cache and per-CPU napi_alloc_cache from Day 1), the scheduler. A NIC is per-ns in the sense that its dev->nd_net points to one ns — but the underlying hardware is one physical thing.

Setting up a netns

Linux exposes ns operations via iproute2:

# Create namespace 'red'
sudo ip netns add red

# Run a shell in it
sudo ip netns exec red bash

# (in that shell) network is empty:
ip link
# 1: lo: <LOOPBACK> ... state DOWN
ip route
# (nothing)

You’re inside a fully isolated network stack. Bring up loopback, you can localhost ping yourself, but nothing else is reachable.

Connecting to the outside

Two veth ends, one in each ns:

# in init_net:
sudo ip link add veth_red type veth peer name veth_red_peer
sudo ip link set veth_red_peer netns red

# inside 'red' (or use ip netns exec):
sudo ip netns exec red ip link set veth_red_peer up
sudo ip netns exec red ip addr add 10.99.99.2/24 dev veth_red_peer

# back in init_net:
sudo ip link set veth_red up
sudo ip addr add 10.99.99.1/24 dev veth_red

# now:
sudo ip netns exec red ping 10.99.99.1     # works
ping 10.99.99.2                              # works (init→red)

The figure shows the general pattern — substitute red for “netns A” and veth_red/veth_red_peer for vethA0/vethA1.

How struct net is assembled: pernet_operations

We keep saying struct net “aggregates all per-ns subsystem state.” But who fills it in? When ip netns add red runs, the kernel allocates one bare struct net — how do routing tables, netfilter chains, conntrack counters, and TCP state all appear inside it? And later, who tears them all back down? The answer is one elegant registration mechanism, and once you see it the chapter’s central claim (“most kernel net functions take struct net *net first”) stops being a slogan and becomes obvious.

Intuition: subsystems register init/exit callbacks

Picture every networking subsystem — IPv4 routing, netfilter, conntrack, TCP, nftables — as a tenant that wants a private room in every struct net that will ever exist. At boot, each tenant signs up once by handing the core a pair of callbacks: “when a new netns is born, call my init to furnish my room; when it dies, call my exit to clear it out.” The core keeps these signups in a list and replays them for every namespace.

The signup form is struct pernet_operations:

/* include/net/net_namespace.h:459 */
struct pernet_operations {
	struct list_head list;
	int  (*init)(struct net *net);   /* :483 — furnish my per-ns state   */
	void (*pre_exit)(struct net *net);
	void (*exit)(struct net *net);   /* :485 — tear it down               */
	void (*exit_batch)(struct list_head *net_exit_list);
	void (*exit_rtnl)(struct net *net, struct list_head *dev_kill_list);
	unsigned int * const id;         /* :490 — net_generic slot id        */
	const size_t size;               /* :491 — bytes to allocate for it   */
};

A subsystem registers it with register_pernet_subsys() (include/net/net_namespace.h:513), which just appends the node to a global list called pernet_list. That’s the whole registration. For example, IPv4 routing’s /proc plumbing:

/* net/ipv4/route.c:379 */
static struct pernet_operations ip_rt_proc_ops __net_initdata = { ... };
/* and registered at route.c:386 (plus sysctl_route_ops / ip_rt_ops / rt_genid_ops at :3782+) */
register_pernet_subsys(&ip_rt_proc_ops);

Dispatch: setup_net walks the list

When a namespace is created, setup_net() walks pernet_list front-to-back and calls every registered init:

/* net/core/net_namespace.c:436 */
static __net_init int setup_net(struct net *net)
{
	const struct pernet_operations *ops;
	/* ... */
	list_for_each_entry(ops, &pernet_list, list) {
		error = ops_init(ops, net);          /* :446 */
		if (error < 0)
			goto out_undo;
	}
	/* ... */
}

This loop is the assembly line. Each subsystem’s init fills in its slice of per-ns state, one after another, in registration order. That’s why a fresh namespace already has a working (if empty) routing table, netfilter hooks, and so on the instant ip netns add returns.

Two ways to store per-ns state: direct field vs net_generic

Where does a subsystem put its state? There are two strategies, and the chapter’s “What’s per-ns” list mixes both:

1. A direct field of struct net — for hot, core subsystems the state is embedded right in the struct: net->ipv4 (net_namespace.h:138), net->nf (:149), net->ct (:151), net->nft (:154). Fast to reach, no indirection.

2. net_generic — a side array — for everything else, baking a field into struct net for every module would bloat it. Instead, if a pernet_operations sets id and size, the dispatcher allocates the room on demand. Look at ops_init:

   /* net/core/net_namespace.c:120 */
   static int ops_init(const struct pernet_operations *ops, struct net *net)
   {
       void *data = NULL;
       if (ops->id) {
           data = kzalloc(ops->size, GFP_KERNEL);          /* allocate the room */
           err = net_assign_generic(net, *ops->id, data);  /* :131 — stash pointer in net->gen */
       }
       /* ... */
       if (ops->init)
           err = ops->init(net);                            /* then furnish it */
   }
   

   The pointer lands in net->gen (net_namespace.h:163), a resizable array indexed by the subsystem’s id. Code fetches it back with net_generic(net, id). This is exactly what nft_pernet(net) does under the hood — which is why the chapter calls nftables state “net_generic-backed”: it isn’t a direct field, it lives in net->gen[nft_id].

Either way, everything reachable from one struct net *. Direct field or net->gen[id], a subsystem starts from that single pointer and finds its per-ns slice. That is the real reason “most kernel net functions take struct net *net as the first parameter” — the pointer is the key to every per-ns room in the building.

Teardown is the mirror image

On destruction, cleanup_net (covered next) walks the same registrations and runs each exit callback — the matching “clear out the room” step. Because every registered subsystem must run its exit (flush conntrack, close sockets, free routing tables) one at a time, deleting a busy namespace is slow: it’s a serial walk over every tenant’s move-out.

What’s per-ns

The state per ns includes (the struct net figure above already previewed this inventory; here we mark which are direct fields vs net_generic-backed, now that you know the difference):

• Routing: net->ipv4.fib_main, ipv4.fib_default, ipv6.fib6_main_tbl — direct fields off net->ipv4. ip route only shows the current ns’s table.
• Netfilter: net->nf.hooks_ipv4[] (per-hook arrays, direct field), nft_pernet(net)->tables (net_generic-backed), per-ns conntrack stats in net->ct. nftables and conntrack state are independent.
• Sockets: the TCP bind hashtable (tcp_hashinfo.bhash[]) is itself global/shared, not per-ns — isolation comes from each bind bucket recording its owning netns. struct inet_bind_bucket carries ib_net, the bucket-match test is net_eq(ib_net(tb), net) && tb->port == port, and the net pointer is mixed into the hash via inet_bhashfn(net, port, ...). So two namespaces that both bind() 0.0.0.0:80 land in separate buckets keyed by their struct net inside the one shared table — same port, no conflict. (Only the ehash can optionally be made per-net via tcp_child_ehash_entries; net->ipv4.tcp_death_row.hashinfo is the per-net hashinfo pointer but it still points at the shared bind table by default.)
• Sysctls: most net.ipv4.* are per-ns (tcp_congestion_control, rp_filter, ip_forward).
• proc/net: each ns sees its own /proc/net, /proc/sys/net.

The life and death of a struct net

The chapter’s lifecycle lab traces a namespace being born (copy_net_ns/setup_net) and dying (cleanup_net). To read what you’re seeing — and to understand why deleting a netns sometimes doesn’t actually free it — you need the refcount model. Good news: it’s the same rule you already learned on Day 1.

Recall: the free-at-zero refcount

On Day 1 you saw sk_buff use a refcount_t (skb->users) that’s bumped by skb_get() and dropped by kfree_skb(), freeing only when it hits zero. struct net uses the identical pattern — only the structure being guarded differs. There’s nothing new to learn about how a refcount works; we’re just applying it to namespaces.

Tier 1: holders (get_net / put_net)

The primary count lives in the embedded ns_common.__ns_ref. Anything that needs the namespace to stay alive takes a reference:

• a non-kernel socket in that netns (sk_alloc → get_net_track; kernel sockets with sk_net_refcnt == 0 take only the passive count, not a holder),
• a task’s nsproxy → net_ns pointing at it (copy_net_ns takes a get_net for the non-CLONE_NEWNET case; a freshly created net starts with one __ns_ref),
• an open FD or bind-mount of the ns (the nsfs name keeps it pinned).

A netdev does NOT hold a reference to its netns. Its nd_net is a plain pointer: dev_net_set() → write_pnet(&dev->nd_net, net) (netdevice.h:2774) is a bare pointer write with no refcount — grep -cE 'get_net\b|get_net_track' net/core/dev.c returns 0. The only count a netdev ever touches is the separate passive (memory-guard) count, and only transiently inside rtnl_net_dev_lock/unlock. In fact the design is the opposite of “a device pins its netns”: because devices do not hold the namespace alive, the teardown machinery must actively evict them. default_device_ops (registered via register_pernet_device) runs default_device_exit_net at teardown, which moves migratable physical devices back to init_net and deletes virtual ones (veth). That eviction code exists precisely because a netdev cannot keep a netns alive — if it could, any netns containing a device would never be freed.

get_net(net) bumps the count; put_net(net) drops it:

/* include/net/net_namespace.h:276, :295 */
static inline struct net *get_net(struct net *net)
{
	ns_ref_inc(net);          /* bump ns_common.__ns_ref */
	return net;
}
static inline void put_net(struct net *net)
{
	if (ns_ref_put(net))      /* drop; true when it hit zero */
		__put_net(net);
}

There’s also maybe_get_net() (:282), the careful variant that fails (returns NULL) if the count is already zero — used when you have a pointer but aren’t sure the namespace is still alive.

Why the last put_net doesn’t free inline — the workqueue

Here’s the subtle part. When put_net drops the holder count to zero it calls __put_net, which does not free anything immediately:

/* net/core/net_namespace.c:745 */
void __put_net(struct net *net)
{
	ref_tracker_dir_exit(&net->refcnt_tracker);
	/* Cleanup the network namespace in process context */
	if (llist_add(&net->cleanup_list, &cleanup_list))
		queue_work(netns_wq, &net_cleanup_work);   /* :750 — punt to a workqueue */
}

It appends the dead namespace to a cleanup_list and queues work. Destruction is deferred to a workqueue.

A workqueue is the kernel’s mechanism for running deferred functions in a kernel thread — i.e. in process context, where the code is allowed to sleep. That’s distinct from the softirq/NAPI context you met on Day 2, which is atomic and may not sleep. Why does namespace teardown need to sleep? Because tearing down subsystems (synchronize_rcu(), flushing conntrack, closing sockets) blocks — and put_net can be called from anywhere, including atomic contexts where blocking is forbidden. So the rule is: drop the refcount cheaply and atomically, then hand the heavy, sleepy teardown off to a kernel thread.

The work item and its function are wired up statically:

/* net/core/net_namespace.c:743 */
static DECLARE_WORK(net_cleanup_work, cleanup_net);

cleanup_net (net/core/net_namespace.c:662) is the function that eventually runs on that workqueue thread. It walks the pernet_operations registrations in reverse and calls every subsystem’s exit — the teardown half of the assembly line from the previous section.

This is also a handy tracing fact: because cleanup_net is a real, non-inlined function registered via DECLARE_WORK, fentry:cleanup_net attaches reliably. The birth-side setup_net, by contrast, is static __net_init and may be inlined — which is exactly the caveat in the lifecycle lab below.

Tier 2: passive (the memory guard)

There’s a second, separate refcount on struct net:

/* include/net/net_namespace.h:66 */
refcount_t		passive;

Why two? The holders count answers “should this network stack stay operational?” The passive count answers a narrower question: “is anyone still touching this struct net’s memory?” A transient lookup might need to pin the bytes of the struct for a moment without keeping the whole stack alive. passive is initialized to 1 (net/core/net_namespace.c:410: refcount_set(&net->passive, 1)), and net_passive_dec() (:530) does the final teardown — freeing net->gen and ultimately kmem_cache_freeing the struct — when it reaches zero, after the operational teardown has already run. Two tiers: holders decide when the stack dies, passive decides when the memory is freed.

What ip netns delete actually does

Now the chapter’s central puzzle resolves cleanly. ip netns delete red only unlinks the name — the nsfs bind-mount under /var/run/netns/red. It does not force the namespace to die. If any process, socket, or open FD still holds a get_net reference, the holder count stays above zero, __put_net is never called, cleanup_net never runs, and the namespace persists invisibly until the last holder exits. The name is gone from ip netns list, but the struct net lives on. That is precisely the “what if a process is still inside?” problem: Linux holds struct net until the last reference goes away — process exit, FD close, etc.

Today’s experiment

See ns metadata

sudo ip netns add green
ls /var/run/netns/        # one entry per ns

# nsfs link to /proc/<pid>/ns/net:
readlink /proc/$$/ns/net           # current shell's netns (init_net)
sudo ip netns exec green readlink /proc/self/ns/net   # green's netns

net:[4026531833]    # init_net (your number differs)
net:[4026532243]    # green — a different inode

You’re looking at two ns_common.inum values resolved through nsfs — the userspace-visible name for each namespace (the kernel itself keys on the struct net * pointer, but each live net maps 1:1 to an inum, so distinct strings here mean distinct namespaces). Note the use of /proc/self in the second command, not /proc/$$. $$ is expanded by your outer interactive shell (which lives in init_net) before ip netns exec runs, so it would resolve the symlink for a process still in init_net and print the same inode twice. /proc/self is resolved by the readlink process that ip netns exec actually placed inside green, so the two inodes genuinely differ — that’s how the kernel knows which ns a process is in.

Per-ns sysctl

# In init_net — show YOUR actual value (don't assume a specific one):
cat /proc/sys/net/ipv4/tcp_congestion_control
# cubic (whatever your box uses — often cubic)

# What's even compiled in? (reno is always built in)
cat /proc/sys/net/ipv4/tcp_available_congestion_control
# reno cubic dctcp bbr htcp

# In green:
sudo ip netns exec green cat /proc/sys/net/ipv4/tcp_congestion_control
# cubic (a new ns inherits init_net's congestion control)

# Set green to reno — always built in, so guaranteed distinct from a default-cubic box:
sudo ip netns exec green sysctl -w net.ipv4.tcp_congestion_control=reno

# Confirm green changed but init_net did NOT:
sudo ip netns exec green cat /proc/sys/net/ipv4/tcp_congestion_control   # reno
cat /proc/sys/net/ipv4/tcp_congestion_control                            # unchanged (your original value)

Different values per ns — green now reads reno while init_net keeps its original value, proving the tables are independent. The per-ns sysctl table is at net->sysctls. (If your init_net somehow already runs reno, set green to cubic instead — the point is just to pick something different from init_net’s current value.)

Per-ns routing

# In red (with veth set up):
sudo ip netns exec red ip route
# 10.99.99.0/24 dev veth_red_peer scope link

# In init_net:
ip route
# (your normal routes, including 10.99.99.0/24 if the host added one)

Different tables, different views.

Watch ns lifecycle

sudo bpftrace -e '
fentry:setup_net  { printf("setup_net %p\n", (void *)args->net); }
fentry:cleanup_net { printf("cleanup_net work %p\n", (void *)args->work); }
'

# In another terminal:
sudo ip netns add demo
sudo ip netns delete demo

Caveat: setup_net is static __net_init (net/core/net_namespace.c:436), so it may be inlined or freed after boot and fentry:setup_net may not reliably attach. If it doesn’t fire, trace its caller copy_net_ns (net/core/net_namespace.c:549) instead — but note that copy_net_ns’s struct net * argument (args->old_net) is the OLD/parent namespace, not the new one. The freshly created struct net is the function’s RETURN value, so you must use fexit and retval:

sudo bpftrace -e 'fexit:copy_net_ns { printf("new net %p\n", retval); }'

(copy_net_ns also runs for CLONE without CLONE_NEWNET, in which case it just returns the old net pointer; the distinct heap addresses appear exactly when ip netns add runs.) cleanup_net is a work-queue function (registered via DECLARE_WORK at net/core/net_namespace.c:743) and attaches reliably — exactly as the lifecycle section explained.

You’ll see the ns being created and torn down. Tie it back to the refcount model: the cleanup_net line only fires once the holder count hit zero and __put_net queued the work — for a freshly created, empty demo with no lingering processes or sockets, that’s immediate.

What happens when you delete a netns

You just watched cleanup_net fire — recall from the refcount section that it only ran because demo had no lingering holders. (If a process, socket, or FD had still held a get_net reference, ip netns delete would have unlinked only the nsfs name and the struct net would have survived invisibly until that last holder went away.)

What to read in the kernel

• include/linux/nsproxy.h — struct nsproxy (line 32). See the one-pointer-per-namespace-kind layout; net_ns is the field today is about.
• include/net/net_namespace.h — struct net definition (line 62). Read all fields once. Note the embedded ns_common ns (line 97), the passive refcount (line 66), the gen pointer for net_generic (line 163), and the direct fields ipv4/nf/ct (lines 138/149/151). Also struct pernet_operations (line 459) and register_pernet_subsys (line 513).
• include/linux/ns/ns_common_types.h — struct ns_common (line 111); the inum field (line ~118) is the identity nsfs prints.
• net/core/net_namespace.c — the lifecycle: setup_net (line 436), ops_init/net_assign_generic (lines 120/83), copy_net_ns (line 549), __put_net (line 745), cleanup_net (line 662), net_passive_dec (line 530), and struct net init_net (line 49).
• include/linux/netdevice.h — search dev_net(dev). Most code calls this to get the ns from a netdev.
• Any *_pernet_ops registration — for example net/ipv4/route.c register_pernet_subsys(&ip_rt_proc_ops) (line 386). Each subsystem registers init/exit callbacks here.
• Documentation/admin-guide/sysctl/net.rst — which sysctls are per-ns, which are global.

What to break

Try moving a physical NIC to a netns

First find your real interface name — many cloud/test VMs use predictable names like ens5 or enp0s3, not eth0:

ip route get 1.1.1.1     # read the 'dev <name>' field; substitute it for eth0 below

sudo ip link set eth0 netns red    # **CAREFUL** — your SSH may go away

If you do this on the real interface you’re using, your SSH session disconnects (init_net no longer has eth0). For real testing, do it with a non-essential interface or via a console. Under the hood, moving the NIC just reassigns its dev->nd_net to red’s struct net — a plain pointer write, not a get_net reference. The device does not pin red: at red’s destruction default_device_exit_net would push eth0 back to init_net automatically.

To recover from console — note the device comes back down with its addresses flushed, so you must bring it up and re-acquire an address:

sudo ip netns exec red ip link set eth0 netns 1   # back to init_net
sudo ip link set eth0 up
sudo dhclient eth0        # if your distro uses DHCP; otherwise re-add the static address you had before

Watch conntrack count per ns

sudo ip netns exec red sysctl net.netfilter.nf_conntrack_count
# 0 (red has no traffic)

sysctl net.netfilter.nf_conntrack_count
# possibly thousands (init_net has all real traffic)

The two views are independent — net->ct is a per-ns slice furnished by conntrack’s own pernet_operations init when red was created.

Cleanup

These experiments leave persistent namespaces and a veth pair behind. Tear them down so you don’t accumulate stale interfaces and ns mounts:

sudo ip link del veth_red 2>/dev/null     # removes both ends of the pair
sudo ip netns delete green 2>/dev/null
sudo ip netns delete red 2>/dev/null      # also reaps the peer living inside it

Remember: each ip netns delete only drops the name. The struct net is actually freed only once its last holder (the veth you just deleted, any ip netns exec shells you left open) is gone and cleanup_net has run.

Bullet Points

• A namespace is a kernel scoping mechanism that turns one global resource into N isolated copies. Kinds: uts, ipc, mnt, pid, net, time, cgroup. A process points at all of them through task_struct → nsproxy; nsproxy->net_ns is the netns.
• Network namespace = struct net (net_namespace.h:62). One per ns; aggregates all per-ns subsystem state. The default ns is init_net.
• A namespace’s userspace identity is ns_common.inum, an inode number exposed via nsfs as /proc/<pid>/ns/net → net:[<inum>]. The kernel itself compares the struct net * pointer (net_eq is net1 == net2); the inum is a 1:1 userspace name for live namespaces (and is recycled after free). init_net’s is the fixed NET_NS_INIT_INO.
• Per-ns state is assembled by pernet_operations: each subsystem registers init/exit callbacks via register_pernet_subsys; setup_net walks the list calling each init, cleanup_net calls each exit. State lives either in a direct field (net->ipv4/nf/ct) or in net_generic (net->gen[id], e.g. nftables).
• That single mechanism is why most kernel net functions take struct net *net first — the pointer reaches every per-ns slice.
• struct net uses the same free-at-zero refcount rule as Day 1’s sk_buff: holders (get_net/put_net on ns_common.__ns_ref) keep the stack alive; a separate passive refcount guards the final memory free.
• The last put_net doesn’t free inline — __put_net queues cleanup_net on the netns_wq workqueue (process context, may sleep), distinct from Day 2’s softirq context.
• ip netns delete unlinks only the nsfs name; if any holder remains, the ns persists invisibly until the last one exits.
• veth pairs are the most common way to connect a ns to the outside world.
• Per-ns: routing, netfilter rules, conntrack, sockets, sysctls, /proc/net. NOT per-ns: hardware, slab caches, scheduler.
• ip netns add/del/exec for management; unshare(CLONE_NEWNET) to create programmatically.

Check question

Two TCP servers run, one in init_net and one in netns “red”, both bound to 0.0.0.0:80. Both have full read/write to their socket. A client in init_net connects to 127.0.0.1:80. Which server gets the connection?

End of Phase 1

You can now read the kernel network stack from sk_buff up. RX path, TX path, offloads, namespaces. The vocabulary is real — when you see dev_net(skb->dev), napi->poll, __qdisc_run, gso_segs, get_net(sock_net(sk)), you know what they’re doing.

Phase 2 (Days 6–12) walks the L2/L3 layers in detail: Ethernet, ARP, the FIB, IPv6, bridges, tunnels.

Day 6 — Ethernet, VLAN, and the L2 layer

Today’s mission: see what eth_type_trans does to every received packet — and learn the four pieces of wire-format background it quietly leans on (the Ethernet header layout, MAC-address bit semantics, network byte order, and the 802.1Q tag internals) so nothing along the way is a magic constant. Then create a VLAN device and watch tags appear and disappear. Total time: ~110 minutes.

Phase 2 starts here. Days 6–12 walk the L2/L3 layers in detail: Ethernet, ARP, IP routing, IPv6, bridges, tunnels.

What L2 means in Linux

The “Layer 2” code in Linux is small but ubiquitous. Every received frame passes through it; every transmitted frame has its Ethernet header built there. The implementation lives mostly in net/ethernet/eth.c (~640 lines).

The single most important function in this layer is eth_type_trans — called by every Ethernet driver on RX, every time. By the end of Day 2 you watched the driver call it in Stage 2 of the RX path: it sets skb->protocol, and that value becomes the demux key the core stack hashes on to pick ip_rcv. Today we open it up.

But eth_type_trans parses raw bytes off the wire, and to read it without hand-waving you need to know what those bytes are. So before we walk the function, three short background sections — intuition first, then the concrete v7.1 struct — on the Ethernet header, MAC-address bits, and byte order. After that the function reads as plain English. The fourth piece of wire-format background — the 802.1Q VLAN tag (Background 4) — comes after the function, in the VLAN section, because that’s where the function hands tagged frames off to the dedicated VLAN subsystem.

One thing from Day 1 is load-bearing today and will not be re-taught:

• The skb’s four pointers (head ≤ data ≤ tail ≤ end) and skb_pull/skb_push. Recall from Day 1 — skb->data is the first valid byte of the current header view, and skb_pull(skb, n) advances data by n bytes (stripping a header you’ve finished with). eth_type_trans does exactly one skb_pull of 14 bytes; that’s the whole “advance past the Ethernet header” trick.

Background 1: the 14 bytes of an Ethernet header

Before eth_type_trans can do anything, it has to know what the first bytes of a received frame mean. On Ethernet, every frame starts with a fixed 14-byte header, and it is gloriously simple — just three fields back to back, no options, no variable length.

struct ethhdr is exactly that (include/uapi/linux/if_ether.h:177-181):

struct ethhdr {
    unsigned char   h_dest[ETH_ALEN];   /* destination eth addr  */
    unsigned char   h_source[ETH_ALEN]; /* source ether addr     */
    __be16          h_proto;            /* packet type ID field  */
} __attribute__((packed));

with ETH_ALEN 6 (if_ether.h:32) and ETH_HLEN 14 (if_ether.h:34). Lay it out as a byte ruler:

• Bytes 0–5: h_dest — the destination MAC (who the frame is for).
• Bytes 6–11: h_source — the source MAC (who sent it).
• Bytes 12–13: h_proto — the 2-byte EtherType (what’s inside).

6 + 6 + 2 = 14 = ETH_HLEN. That is the exact number skb_pull advances by. When a frame arrives, the driver has placed skb->data on byte 0 (the start of h_dest). After eth_type_trans does skb_pull(skb, ETH_HLEN), skb->data lands precisely on byte 14 — the first byte of the L3 header (the IP header, usually). No parsing, no length math: the Ethernet header is a fixed size, so one fixed pull strips it. This is why the helper that does it, eth_skb_pull_mac, is so short (etherdevice.h:640-645):

static inline struct ethhdr *eth_skb_pull_mac(struct sk_buff *skb)
{
    struct ethhdr *eth = (struct ethhdr *)skb->data;  /* read the 14 bytes */
    skb_pull_inline(skb, ETH_HLEN);                   /* then step past them */
    return eth;
}

The EtherType is sometimes a length

Here’s the one wrinkle. The 2-byte h_proto field is overloaded, for historical reasons:

• If the value is ≥ 0x0600 (1536), it’s a real protocol type — “Ethernet II” framing. Examples: ETH_P_IP 0x0800 (if_ether.h:52), ETH_P_IPV6 0x86DD (:74), ETH_P_ARP 0x0806 (:54), ETH_P_8021Q 0x8100 (:71, VLAN), ETH_P_8021AD 0x88A8 (:93, QinQ).
• If the value is ≤ 1500, it’s a frame length, not a type — the original IEEE 802.3 framing, where the protocol was identified by an LLC header inside the payload instead.

The boundary between the two interpretations is the constant ETH_P_802_3_MIN 0x0600 (if_ether.h:127), whose comment says it outright: “If the value in the ethernet type is more than this value then the frame is Ethernet II. Else it is 802.3.” The kernel decides which case it’s in with eth_proto_is_802_3() (etherdevice.h:220). That is exactly why, at the very end, eth_type_trans returns eth->h_proto only if (eth_proto_is_802_3(...)) — and otherwise falls back to a dummy “this is really a length” protocol ID. So the “h_proto is a length field (< 1536)” aside later in this chapter is not magic: it’s the < 1536 / ≥ 0x0600 split made concrete.

Background 2: what a MAC address actually encodes

eth_type_trans classifies every frame by looking at the destination MAC (h_dest): is it for us, for everyone, for a group, or for someone else? To follow that — and to understand why the lab later invents the address 02:00:00:00:00:01 — you need to know that a MAC address isn’t just six random bytes. Two bits of the first octet carry meaning.

A MAC address is 48 bits / 6 octets. The two least-significant bits of the first octet are special:

• bit 0 — I/G (Individual/Group): 0 = unicast (this frame is addressed to one NIC); 1 = multicast/group (addressed to a set of NICs).
• bit 1 — U/L (Universal/Local): 0 = globally unique, burned in by the vendor (the top 3 octets are the vendor’s OUI); 1 = locally administered, i.e. software-assigned.

The kernel reads “the multicast bit” with the simplest possible test — is_multicast_ether_addr() is literally 0x01 & addr[0] (etherdevice.h:130). That single bit is the multicast bit this chapter keeps mentioning. And is_local_ether_addr() is 0x02 & addr[0] (the U/L bit, etherdevice.h:163).

Broadcast is a special case of multicast. The broadcast address ff:ff:ff:ff:ff:ff has every bit set — including bit 0 — so it is a multicast address, just the all-ones one. is_broadcast_ether_addr() checks all six octets equal 0xff (etherdevice.h:176). This is exactly why the classifier checks multicast first, then asks “is it the broadcast address specifically?” Watch eth_skb_pkt_type() do it (etherdevice.h:623-638):

static inline void eth_skb_pkt_type(struct sk_buff *skb,
                                    const struct net_device *dev)
{
    const struct ethhdr *eth = eth_hdr(skb);

    if (unlikely(!ether_addr_equal_64bits(eth->h_dest, dev->dev_addr))) {  /* not us */
        if (unlikely(is_multicast_ether_addr_64bits(eth->h_dest))) {      /* group bit set? */
            if (ether_addr_equal_64bits(eth->h_dest, dev->broadcast))
                skb->pkt_type = PACKET_BROADCAST;   /* the all-ones one */
            else
                skb->pkt_type = PACKET_MULTICAST;   /* some other group */
        } else {
            skb->pkt_type = PACKET_OTHERHOST;       /* unicast, but not our MAC */
        }
    }
    /* else: leaves the default PACKET_HOST — dst == our dev_addr */
}

The four outcomes are the pkt_type constants (include/uapi/linux/if_packet.h:26-29): PACKET_HOST 0, PACKET_BROADCAST 1, PACKET_MULTICAST 2, PACKET_OTHERHOST 3. Note the function only sets a value when the dst is not our dev_addr; the skb’s default pkt_type is already PACKET_HOST, so “addressed to us” needs no assignment.

Why 02:00:00:00:00:01 is the canonical “safe to invent” MAC

Now the lab’s magic constant decodes itself. Take the first octet 0x02 = binary 0000 0010:

• bit 0 = 0 → unicast. A NIC configured with this address treats frames sent to it as PACKET_HOST (us).
• bit 1 = 1 → locally administered. By definition it can’t collide with any vendor’s globally-unique OUI.

So 02:00:...:01 is guaranteed (a) to be a normal unicast address and (b) never to clash with real hardware. That’s why it’s the textbook address to assign to a throwaway interface — which is exactly what the “Set a non-default MAC” lab does.

Background 3: network byte order and the __be16 type

There’s one more thing you must know before reading eth_type_trans, and it’s the gotcha that makes today’s bpftrace experiment print a “wrong-looking” number. It’s about byte order.

A 2-byte field like h_proto holds a 16-bit number, and there are two ways to lay those two bytes out in memory:

• Big-endian (“network byte order”): most-significant byte first. The value 0x0800 is stored as the bytes 08 00.
• Little-endian: least-significant byte first. The same value 0x0800 is stored as 00 08.

The wire is always big-endian. Every multi-byte protocol field — EtherType, IP addresses, port numbers — travels most-significant-byte-first. But x86 and ARM hosts are little-endian. So if you grab the two raw wire bytes 08 00 and read them back as a host u16, the CPU interprets them in its order and you get 0x0008, not 0x0800:

That table is the entire explanation for today’s experiment printing type=0x0008 for an IP frame. The bytes are right; you’re just reading them in the wrong order.

__be16 / __be32 are the kernel’s annotations marking a value as big-endian (they’re checked by the sparse static analyzer). When you see __be16 h_proto, it’s a reminder: don’t do arithmetic or comparisons on this as a native integer until you convert it. The conversion helpers are:

• ntohs() / ntohl() — network-to-host, wire → host (16-bit / 32-bit).
• htons() / htonl() — host-to-network, host → wire.

The subtlety that ties Day 2 together

Here’s the part worth holding onto. eth_type_trans returns h_proto __be16, without swapping it — return eth->h_proto; at net/ethernet/eth.c:177, and the signature returns __be16 (eth.c:155). The driver stores that still-big-endian value straight into skb->protocol. So skb->protocol stays in network byte order end-to-end.

Where does the swap finally happen? At the L3 demux you met in Day 2. Handlers register their EtherType into a bucket with htons(pt->type) (the registration helper ptype_head() at net/core/dev.c:608), and at RX time __netif_receive_skb_core indexes that same hash table with ntohs() — ptype_base[ntohs(type) & PTYPE_HASH_MASK] where type = skb->protocol (net/core/dev.c:6147, with type assigned from skb->protocol at :6142). Registration and demux hash the same way. Recall from Day 2 — that’s how skb->protocol (big-endian) gets matched against the registered handlers to reach ip_rcv. So nothing along the RX path ever byte-swaps skb->protocol itself; the comparison is done in network order. This is precisely why a raw %04x print in bpftrace shows the swapped value: you’re seeing the honest, unswapped wire bytes.

The mental tool: when you print a wire field raw and it looks byte-swapped, read the low two hex digits, or wrap the field in bswap() in bpftrace.

eth_type_trans — the universal RX header parser

Now the function reads cleanly. eth_type_trans(skb, dev) (net/ethernet/eth.c:155) does four things in tight succession:

1. skb_reset_mac_header(skb) — record where the Ethernet header is by storing the offset of skb->data within the buffer (skb->data - skb->head); it’s at skb->data right now (Background 1: byte 0, the start of h_dest).
2. Read and pull, in one helper — eth_skb_pull_mac(skb). This single call both reads eth = (struct ethhdr *)skb->data (the three-field header from Background 1) and does the skb_pull(skb, ETH_HLEN) that advances skb->data past the 14-byte header. So by the end of this one step skb->data already points at the L3 payload — the read and the pull are not two separate stages.
3. Set skb->pkt_type (via eth_skb_pkt_type, called after the pull) based on destination MAC — exactly the multicast-first / broadcast / otherhost logic from Background 2. This still works even though the header was just pulled: eth_skb_pkt_type finds the header via eth_hdr(skb), which uses the mac_header offset recorded in step 1, not skb->data.
   • PACKET_BROADCAST if dst is ff:ff:ff:ff:ff:ff.
   • PACKET_MULTICAST if the multicast bit (addr[0] & 1) is set.
   • PACKET_OTHERHOST if dst is unicast but doesn’t match dev->dev_addr (would be dropped on a normal NIC; promiscuous mode keeps these). Otherwise it leaves the default PACKET_HOST when dst matches dev_addr (us).
4. Return the protocol ID — typically eth->h_proto (ETH_P_IP 0x0800, ETH_P_IPV6 0x86DD, or ETH_P_8021Q 0x8100, VLAN), but only if (eth_proto_is_802_3(...)) — the ≥ 0x0600 / < 1536 boundary from Background 1. There are two short-circuits before that: if the device uses DSA tagging (Distributed Switch Architecture, where an embedded switch prepends its own tag) it returns ETH_P_XDSA (0x00F8) without looking at the packet, and if h_proto is a length field (< 1536, i.e. below ETH_P_802_3_MIN 0x0600) it falls back to ETH_P_802_3 (IPX magic) or ETH_P_802_2 (802.2 LLC). The returned value is __be16 and is not byte-swapped (Background 3).

The driver assigns the return value to skb->protocol, then passes the skb to GRO / netif_receive_skb (recall the Day 2 RX path). From here the L3 dispatcher uses skb->protocol to find the right handler (ip_rcv for IP) — hashing it via ntohs() as Background 3 described.

Background 4: the 802.1Q VLAN tag

So far every frame has been 14 bytes of header then payload. A VLAN tag inserts 4 extra bytes into that header to label which virtual LAN the frame belongs to. Let’s decompose those 4 bytes — they’re the reason skb->vlan_tci and “VLAN ID 100” exist.

The full 802.1Q tag is 4 bytes (VLAN_HLEN 4, if_vlan.h:16), inserted between the source MAC and the EtherType field. It’s two halves:

• a 2-byte TPID (Tag Protocol IDentifier) — 0x8100 for 802.1Q, or 0x88A8 for the outer tag of stacked 802.1ad/QinQ. The TPID sits where the EtherType normally would, which is how the NIC knows a tag follows.
• a 2-byte TCI (Tag Control Information) — the actual label, decomposed below.

The original EtherType (e.g. 0x0800 for IP) moves to after the tag. The concrete struct makes this layout explicit (if_vlan.h:48-55):

struct vlan_ethhdr {
    unsigned char   h_dest[ETH_ALEN];        /* bytes  0–5  */
    unsigned char   h_source[ETH_ALEN];      /* bytes  6–11 */
    __be16          h_vlan_proto;            /* bytes 12–13: TPID, 0x8100   */
    __be16          h_vlan_TCI;              /* bytes 14–15: TCI            */
    __be16          h_vlan_encapsulated_proto; /* bytes 16–17: inner type   */
};

(struct vlan_hdr at if_vlan.h:35-38 is just the TCI plus the inner ethertype — the two __be16s that follow the TPID, not the full 4-byte tag.)

Decomposing the 16-bit TCI

The TCI packs three subfields into 16 bits:

• bits 15–13 (top 3): PCP — Priority Code Point, the 802.1p class of service (0–7). Mask VLAN_PRIO_MASK 0xe000, shift VLAN_PRIO_SHIFT 13.
• bit 12: DEI/CFI — Drop Eligible Indicator (formerly Canonical Format Indicator). Mask VLAN_CFI_MASK 0x1000.
• bits 11–0 (bottom 12): VID — the VLAN ID. Mask VLAN_VID_MASK 0x0fff.

(All four constants at if_vlan.h:73-77.) Twelve bits give VLAN_N_VID 4096 possible IDs (0 and 4095 are reserved, leaving 4094 usable — which is why the labs cap out there). The VID is the “VLAN ID 100” the labs use.

When a modern NIC strips the tag in hardware (HW acceleration), it stashes the TCI in skb->vlan_tci and the TPID in skb->vlan_proto. These overlay a single u32 for cheap presence-testing (include/linux/skbuff.h:1052-1055):

union {
    u32       vlan_all;        /* test this for "any tag present?" */
    struct {
        __be16 vlan_proto;     /* the TPID  */
        __u16  vlan_tci;       /* the TCI   */
    };
};

That’s why skb_vlan_tag_present(skb) is literally !!skb->vlan_all (if_vlan.h:82): if either half is non-zero, a tag is present. And the accessors mask the TCI apart exactly as the bit diagram shows — skb_vlan_tag_get_id(skb) is vlan_tci & VLAN_VID_MASK (if_vlan.h:84), skb_vlan_tag_get_prio shifts out the PCP, skb_vlan_tag_get_cfi tests the DEI bit.

The kernel handles VLANs in two ways:

1. HW acceleration — the vlan_all union path described above (NIC strips the tag on RX, stack sees an untagged frame plus metadata, tested with skb_vlan_tag_present). On TX the kernel hands the NIC a possibly-untagged skb and the NIC adds the tag if requested.

2. Software path — for NICs without HW VLAN, or for stacked VLANs (QinQ), vlan_do_receive in net/8021q/vlan_core.c parses the tag and dispatches.

A VLAN device (eth0.100) is a virtual netdev that filters frames by VLAN ID:

sudo ip link add link eth0 name eth0.100 type vlan id 100
sudo ip link set eth0.100 up
sudo ip addr add 192.168.100.5/24 dev eth0.100

Now eth0.100 is a fully-featured interface with its own routes, MTU, and IP. Frames with VLAN ID 100 arriving on eth0 get redirected to eth0.100’s RX path; transmits on eth0.100 get tagged and sent through eth0. That redirection is vlan_do_receive() (net/8021q/vlan_core.c:10): it reads the VID with skb_vlan_tag_get_id(skb) (the & VLAN_VID_MASK from the bit diagram) and looks up the matching device with vlan_find_dev(skb->dev, vlan_proto, vlan_id). If a device is registered for that (interface, VID) pair, the frame is steered to it; if not, the lookup returns NULL and vlan_do_receive returns false (the Check question hinges on this).

There are no Dumb Questions

Q: What’s the difference between skb->protocol and the type field?

A: They’re set from the same source (eth->h_proto), but skb->protocol is what the stack uses for dispatch. For VLAN-tagged frames, the kernel processes the tag, then sets skb->protocol to the inner type (h_vlan_encapsulated_proto). So a TCP-over-VLAN frame ends up with skb->protocol = ETH_P_IP after VLAN handling. (Both are stored big-endian — Background 3.)

Q: Can I have a VLAN inside a VLAN?

A: Yes — and there are two flavors. The simplest is plain 802.1Q-in-802.1Q double tagging, where both the outer and inner tags use TPID 0x8100: ip link add link eth0.100 name eth0.100.200 type vlan id 200 stacks a second default-protocol (0x8100) tag on top of the first. True QinQ (802.1ad) instead uses an 0x88a8 outer S-tag with an 0x8100 inner tag — but you must ask for that protocol explicitly, because the VLAN netlink default is 802.1Q (0x8100). Create the outer device with type vlan proto 802.1ad id 100, then stack the inner with type vlan proto 802.1q id 200. The bare command above does not produce an 0x88a8 outer tag.

Q: How does the kernel pick the source MAC for outbound frames?

A: From dev->dev_addr of the egress device. Modern NICs allow random MACs at boot (privacy via MACAddressPolicy=random in systemd-networkd) — and a randomly-generated MAC always has the locally-administered bit set (Background 2), so it never collides with a vendor OUI. The Ethernet header build happens in dev_hard_header → eth_header (net/ethernet/eth.c).

Today’s experiment

See eth_type_trans in action

sudo bpftrace -e '
fentry:eth_type_trans {
  $eth = (struct ethhdr *)args->skb->data;
  printf("dst=%02x:%02x:%02x:%02x:%02x:%02x type=0x%04x\n",
         $eth->h_dest[0], $eth->h_dest[1], $eth->h_dest[2],
         $eth->h_dest[3], $eth->h_dest[4], $eth->h_dest[5], $eth->h_proto);
}' &
sleep 3                 # let the fentry BTF probe attach before we trigger traffic
ping -c 5 8.8.8.8
sleep 1                 # let the last frames drain through the probe
sudo killall bpftrace

The sleep 3 matters: a BTF fentry probe takes a second or two to attach, and a single ping -c 1 completes in a few hundred milliseconds — without the delay the trigger (and the kill) happen before the probe is live and you see nothing. eth_type_trans fires on every received frame, so with the probe up for a few seconds you’ll get output even apart from the ping:

dst=00:22:48:7c:ffffffb3:ffffffef type=0x0008
dst=00:22:48:7c:ffffffb3:ffffffef type=0x0008
dst=00:22:48:7c:ffffffb3:ffffffef type=0x0008

You’ll see actual MAC addresses and EtherTypes flying through, and both gotchas from the background sections show up at once:

• The MAC octets print as signed bytes, so values ≥ 0x80 show sign-extended as ffffffb3 — read the low two hex digits (b3). (This is a bpftrace printing quirk, not the bit semantics from Background 2.)
• The type is byte-swapped. h_proto is __be16 (big-endian on the wire, Background 3), so on a little-endian host the raw %04x print shows IP as 0x0008 rather than 0x0800. Mentally swap the bytes, or wrap the field in bswap().

Create a VLAN and watch traffic

sudo ip link add link eth0 name eth0.100 type vlan id 100
sudo ip link set eth0.100 up
sudo ip addr add 10.100.0.1/24 dev eth0.100

# verify
ip -d link show eth0.100   # see vlan_id 100, vlan_protocol 802.1Q

# capture both:
sudo tcpdump -i eth0 -e -n vlan 100 &
sudo tcpdump -i eth0.100 -n &
sleep 1

# trigger: no peer answers, but the ARP request for 10.100.0.2
# egresses eth0 tagged with VID 100 (and untagged on eth0.100)
ping -c 3 -W1 10.100.0.2
sleep 1

You should see the same frame twice — on eth0 carrying the 802.1Q tag that -e exposes (vlan 100, p 0, ethertype ARP ...) and on eth0.100 already stripped/untagged. The vlan 100, p 0 is the TCI decoded for you: VID 100, PCP 0 (the TCI bitfields from Background 4). That side-by-side is the VLAN device doing tag insertion on egress and removal on ingress. The ping gets no replies (there’s no host at 10.100.0.2) — that’s fine; the tagged ARP requests are the point.

# cleanup
sudo pkill tcpdump            # stop both backgrounded captures (and drop promisc mode)
sudo ip link del eth0.100     # also removes the 10.100.0.1/24 address

See pkt_type stats

sudo bpftrace -e '
fexit:eth_type_trans {
  @pkt_types[args->skb->pkt_type] = count();
}
interval:s:5 { exit(); }'

PACKET_HOST=0, BROADCAST=1, MULTICAST=2, OTHERHOST=3 (the constants from Background 2). We attach at fexit (function return), not fentry: eth_type_trans is the function that sets pkt_type (via eth_skb_pkt_type), so at entry the field still holds its prior value (usually 0). By the time the function returns, the assignment is complete. fexit exposes args-> for the input arguments just like fentry.

What to break

Toggle promiscuous mode

sudo ip link set eth0 promisc on

# re-run the "See pkt_type stats" trace above while generating traffic

sudo ip link set eth0 promisc off   # restore

Now the kernel processes packets that aren’t addressed to your MAC (where pkt_type == PACKET_OTHERHOST — unicast frames whose dst doesn’t match dev_addr, Background 2). tcpdump enables this implicitly.

Caveat: on a mirrored/SPAN port, a hub, or a shared segment you’ll now see @pkt_types[3] (PACKET_OTHERHOST) appear. On an ordinary switched link or a cloud vNIC the switch never delivers other hosts’ unicast to your port, so key 3 may stay 0 even with promisc on — that’s expected, not a bug. Always promisc off afterward so you don’t leave the NIC in promiscuous mode.

Set a non-default MAC

WARNING: do not change the MAC of the interface carrying your SSH session. eth0 is usually the management interface on a cloud/test VM — changing its MAC (or bringing the link down to do so) drops your connection, and many drivers reject an address change while the link is up. Use a throwaway interface instead:

# Option A (safe): a dummy interface, so you never touch the SSH link.
sudo ip link add mac-test type dummy
sudo ip link set mac-test address 02:00:00:00:00:01
ip link show mac-test            # observe the new MAC
sudo ip link delete mac-test     # cleanup

# Option B: a real spare NIC (NOT the one carrying SSH).
IF=eth1                          # NOT your SSH interface; eth1 is illustrative —
                                 # substitute an actual spare NIC present on your box
orig=$(cat /sys/class/net/$IF/address)
sudo ip link set $IF down
sudo ip link set $IF address 02:00:00:00:00:01
sudo ip link set $IF up
ip link show $IF                 # confirm; re-run the eth_type_trans trace
sudo ip link set $IF down
sudo ip link set $IF address "$orig"   # restore the original MAC
sudo ip link set $IF up

We chose 02:00:00:00:00:01 deliberately (Background 2): first octet 0x02 has the unicast bit clear and the locally-administered bit set, so it’s guaranteed to be treated as a normal unicast address and guaranteed not to collide with any vendor’s burned-in OUI. Now eth_type_trans sees 02:00:00:00:00:01 as “us” — frames addressed there are PACKET_HOST. Useful for testing identity rules.

What to read in the kernel

• net/ethernet/eth.c — eth_type_trans (line 155), eth_header, header_ops. The whole file is ~640 lines.
• include/uapi/linux/if_ether.h — struct ethhdr (line 177), ETH_ALEN/ETH_HLEN, the ETH_P_* EtherType constants, and ETH_P_802_3_MIN (line 127, the type-vs-length boundary).
• include/linux/etherdevice.h — is_multicast_ether_addr (line 130), is_broadcast_ether_addr (line 176), eth_proto_is_802_3 (line 220), eth_skb_pkt_type (line 623), eth_skb_pull_mac (line 640).
• include/linux/if_vlan.h — struct vlan_ethhdr (line 48), the TCI masks (VLAN_VID_MASK etc., lines 73–77), skb_vlan_tag_present/skb_vlan_tag_get_id (lines 82–84).
• net/8021q/vlan_core.c — VLAN reception path (vlan_do_receive, line 10).
• net/8021q/vlan_dev.c — VLAN device implementation.
• include/linux/skbuff.h — search vlan_tci, vlan_proto, vlan_all (the union at line 1052).

Bullet Points

• The Ethernet header is 14 bytes (ETH_HLEN): h_dest[6] + h_source[6] + h_proto[2] (struct ethhdr). That fixed size is why skb_pull(skb, 14) lands skb->data exactly on the L3 header.
• The EtherType h_proto is overloaded: ≥ 0x0600 = real type (IP 0x0800, IPv6 0x86DD, ARP 0x0806, VLAN 0x8100); ≤ 1500 = a frame length (802.3). The boundary is ETH_P_802_3_MIN 0x0600, decided by eth_proto_is_802_3().
• A MAC’s first octet carries two flag bits: bit 0 = I/G (0 unicast, 1 multicast); bit 1 = U/L (0 global/vendor, 1 locally-administered). “The multicast bit” is addr[0] & 1. Broadcast ff:ff:ff:ff:ff:ff is just the all-ones multicast. 02:00:..:01 = safe synthetic unicast (local + not multicast).
• Network byte order is big-endian. __be16/__be32 mark wire values; convert with ntohs/htons. eth_type_trans returns h_proto unswapped, so skb->protocol stays big-endian end-to-end; the demux does the ntohs(). A raw print on little-endian x86 shows IP as 0x0008.
• eth_type_trans is the universal RX header parser; called from every Ethernet driver. Sets skb->dev, skb->mac_header and skb->pkt_type, advances skb->data past the L2 header, and returns the L3 protocol id (which the driver assigns to skb->protocol — eth_type_trans never writes that field itself).
• skb->pkt_type: HOST / BROADCAST / MULTICAST / OTHERHOST.
• VLAN = a 4-byte tag (VLAN_HLEN) between src MAC and EtherType: 2-byte TPID (0x8100, or 0x88a8 for QinQ) + 2-byte TCI. The TCI splits into PCP[15:13] + DEI[12] + VID[11:0]; 12-bit VID → 4094 usable IDs.
• HW VLAN acceleration: NIC strips tag, kernel reads skb->vlan_tci/vlan_proto (the vlan_all union). skb_vlan_tag_present = !!vlan_all; skb_vlan_tag_get_id = vlan_tci & VLAN_VID_MASK. Otherwise software path in net/8021q/.
• VLAN devices are virtual netdevs that filter+strip by VLAN ID; vlan_do_receive looks them up with vlan_find_dev.
• TX side: dev_hard_header → eth_header builds the L2 frame, using dev->dev_addr as the source MAC.

Check question

A frame arrives at eth0 with VLAN tag 100, but no eth0.100 device exists. What happens?

Tomorrow

Day 7: ARP and the neighbour subsystem. How the kernel learns its peers’ MAC addresses and what happens when neighbour entries go stale.

Day 7 — ARP and the neighbour subsystem

Today’s mission: see how Linux learns its peers’ MAC addresses, the state machine an ARP entry walks through, and what happens at scale — plus the four pieces of background that make the state machine make sense: the kernel’s notion of time, the kernel timer that drives the states, the ARP packet on the wire, and how a flat-looking “table” is really a hashed, garbage-collected cache. Total time: ~110 minutes.

ARP in one paragraph

When Linux wants to send an IP packet to some next-hop IP on a directly connected network, it needs the destination’s MAC. The protocol is ARP (RFC 826): broadcast a query, receive a reply, cache the result. The cache is what ip neigh shows.

But “ARP” in Linux is a specific case of a more general neighbour subsystem at net/core/neighbour.c. The same code handles IPv6 NDP (Neighbour Discovery Protocol) and InfiniBand IPoIB. The L4-protocol-specific bits live in net/ipv4/arp.c.

How does an arriving ARP frame even reach this code? Recall eth_type_trans from Day 6: it reads the Ethernet frame’s EtherType and stores it in skb->protocol, and the core stack uses that value to pick a handler. ARP frames carry ETH_P_ARP = 0x0806 (include/uapi/linux/if_ether.h:54), so they’re dispatched to arp_rcv rather than ip_rcv (0x0800). Same demux you saw on the RX path — just a different EtherType key.

Most of this chapter is about time: “REACHABLE for 30s,” “probe every 1s,” “stale after 60s.” So before anything else, we need to know what the kernel means by a second.

Background: jiffies and HZ — the kernel’s notion of time

The kernel can’t call gettimeofday() on every packet — that’s far too expensive for the hot path. Instead it keeps a single, dirt-cheap counter that a periodic timer interrupt bumps on a fixed schedule:

• A jiffy is one tick of the kernel’s periodic timer interrupt.
• The global counter jiffies increments by one on every tick. Reading it is a single memory load — this is the kernel’s cheap monotonic clock.
• HZ is the tick rate (ticks per second). It’s a compile-time constant — # define HZ CONFIG_HZ (include/asm-generic/param.h:8) — and common values are 100, 250, or 1000. On a HZ=1000 kernel, one jiffy is 1 ms; on HZ=250, one jiffy is 4 ms.

The key consequence: a duration in seconds is seconds * HZ jiffies. That’s why the kernel source writes 30 * HZ, 1 * HZ, 5 * HZ instead of literal millisecond counts — the same expression yields the right tick count whatever HZ the kernel was built with. Look at the ARP defaults (net/ipv4/arp.c:152):

struct neigh_table arp_tbl = {
    .parms = {
        .reachable_time = 30 * HZ,          /* REACHABLE lasts ~30s   */
        .data = {
            [NEIGH_VAR_MCAST_PROBES]   = 3,
            [NEIGH_VAR_UCAST_PROBES]   = 3,
            [NEIGH_VAR_RETRANS_TIME]   = 1 * HZ,   /* probe spacing ~1s */
            [NEIGH_VAR_DELAY_PROBE_TIME] = 5 * HZ, /* DELAY window ~5s  */
            [NEIGH_VAR_GC_STALETIME]   = 60 * HZ,  /* GC after ~60s     */
            /* ... */
        },
    },
    .gc_thresh1 = 128, .gc_thresh2 = 512, .gc_thresh3 = 1024,
};

So when this chapter says “REACHABLE for 30s” or “retrans 1s,” the struct literally holds 30 * HZ and 1 * HZ jiffies — not seconds, not milliseconds.

The 30s is a base, not a fixed window. .reachable_time = 30 * HZ is only the seed: each entry’s effective reachable_time is randomized by neigh_rand_reach_time (net/core/neighbour.c:130) to a value in (1/2)*base … (3/2)*base — roughly 15–45s — and re-randomized periodically. The point is to avoid a fleet of entries all expiring on the same jiffy. So “~30s” throughout this chapter means “30s base, 15–45s in practice.”

The neighbour’s timestamp fields are jiffies snapshots too. In struct neighbour (include/net/neighbour.h:140):

unsigned long confirmed;   /* neighbour.h:145 — jiffies at last reachability confirmation */
unsigned long updated;     /* neighbour.h:146 — jiffies at last state change            */

“Is this entry stale?” is therefore never wall-clock arithmetic; it’s a jiffy comparison like time_after(now, confirmed + reachable_time), where now = jiffies. You’ll see that exact pattern in the timer handler in a moment.

One footnote so the sysctls make sense. Because the unit is jiffies, the same .data[] slot is exposed to userspace under two names — a ticks form and a *_MS milliseconds alias (e.g. NEIGH_VAR_RETRANS_TIME vs RETRANS_TIME_MS). That’s why ip-sysctl.rst lets you read or write retrans_time (jiffies) or retrans_time_ms (milliseconds) for the same underlying value.

The ARP wire format

Before we cache MACs, let’s see what’s actually in an ARP packet — otherwise “request is broadcast, reply is unicast” and “gratuitous ARP” are just assertions. An ARP message is a tiny fixed header followed by four variable-length address fields. The fixed header is struct arphdr (include/uapi/linux/if_arp.h:145):

struct arphdr {
    __be16        ar_hrd;   /* hardware type (1 = Ethernet)        */
    __be16        ar_pro;   /* protocol type (0x0800 = IPv4)       */
    unsigned char ar_hln;   /* hardware address length (6 for MAC) */
    unsigned char ar_pln;   /* protocol address length (4 for IPv4)*/
    __be16        ar_op;    /* opcode: request or reply            */
};
/* then, for ARP-over-Ethernet (commented in the header at if_arp.h:156-159): */
/*   ar_sha[6]  sender hardware address (SHA)  */
/*   ar_sip[4]  sender IP address       (SIP)  */
/*   ar_tha[6]  target hardware address (THA)  */
/*   ar_tip[4]  target IP address       (TIP)  */

The opcode (ar_op) is what distinguishes the two messages (include/uapi/linux/if_arp.h:107-108):

• ARPOP_REQUEST = 1 — “who has TIP? tell SIP.” Sent to the Ethernet broadcast MAC ff:ff:ff:ff:ff:ff so every host on the segment sees it. The sender fills in SHA + SIP + TIP and leaves THA = 00:00:00:00:00:00 (it doesn’t know the target’s MAC yet — that’s the whole question).
• ARPOP_REPLY = 2 — “TIP is at THA.” The host that owns TIP fills in its own MAC as THA and sends it unicast straight back to the asker.

That’s exactly what a tcpdump -e capture shows in the lab below: one broadcast request with an empty target MAC, one unicast reply with it filled in. The parsing happens in arp_process (net/ipv4/arp.c:702), reached from arp_rcv (net/ipv4/arp.c:967); outgoing requests/replies are built and sent by arp_send (net/ipv4/arp.c:323).

Gratuitous ARP falls right out of this layout: it’s just a request whose TIP equals its SIP. Re-read that with the fields in hand — the node is announcing “I (SIP) now own this IP,” and because it’s a broadcast request, everyone updates their cache unsolicited. No reply is expected; the point is the announcement.

The neighbour table

struct neigh_table arp_tbl (net/ipv4/arp.c:152) holds all IPv4 neighbour entries. Each entry is a struct neighbour (include/net/neighbour.h:140) with:

• primary_key (neighbour.h:169): the next-hop IP address — the lookup key.
• ha[MAX_ADDR_LEN]: hardware address (MAC).
• dev: which netdev this neighbour is reachable through.
• nud_state: state machine (REACHABLE, STALE, INCOMPLETE, …).
• confirmed/updated: jiffies snapshots (the timestamps from the jiffies section).
• arp_queue: skbs waiting for resolution.

It’s a hash table, not a list

We keep saying “the table,” but it isn’t a flat list. arp_tbl.nht (include/net/neighbour.h:244) points at a struct neigh_hash_table (include/net/neighbour.h:201):

struct neigh_hash_table {
    struct hlist_head *hash_heads;   /* array of hash buckets        */
    unsigned int       hash_shift;   /* current size = 1 << shift     */
    __u32              hash_rnd[NEIGH_NUM_HASH_RND]; /* random seeds (=4, anti-collision) */
    struct rcu_head    rcu;
};

An entry lives in the bucket chosen by hashing its key (next-hop IP) plus dev. Each struct neighbour is linked into its bucket through struct hlist_node hash; (neighbour.h:141). A lookup — neigh_lookup (net/core/neighbour.c:625), or the fast inline __neigh_lookup — hashes the key, then walks just that one bucket’s hlist. As the population grows the table is rehashed to a bigger size, keeping chains short. That’s why a gateway with thousands of peers still resolves each next hop in roughly O(1).

Two details make the hot path safe and cheap:

• RCU read side. The bucket walk happens under RCU read-side protection — no lock on the per-packet lookup path. (RCU, in one sentence: readers proceed lock-free, and a removed object isn’t actually freed until all concurrent readers have finished, so a lookup can safely hold a pointer that another CPU is deleting.)
• Refcounts, freed via RCU. Each entry carries refcount_t refcnt; (neighbour.h:148) and a struct rcu_head rcu; (neighbour.h:166). This is the same free-at-zero refcount discipline you learned for sk_buff->users on Day 1, applied to neighbours: hold a reference while you use the entry, drop it when done, and the object is reclaimed only when the count hits zero and RCU says no reader can still see it.

Background: the timer that drives the state machine

Here’s the thing the state diagram never shows: states don’t change on their own. The engine that advances a neighbour from REACHABLE to STALE, or from INCOMPLETE to FAILED, is a per-entry kernel timer.

Each struct neighbour embeds one (include/net/neighbour.h:151):

struct timer_list timer;

A timer_list is the kernel’s basic one-shot timer: you arm it with a future expiry expressed in jiffies, and when jiffies reaches that value the kernel’s timer machinery calls the callback you registered. The neighbour subsystem wires the callback at construction time (net/core/neighbour.c:534):

timer_setup(&n->timer, neigh_timer_handler, 0);

Day 5 introduced workqueues — deferred work that runs in process context and may sleep — when tearing down a netns. A timer_list is a different mechanism Day 5 did not cover: its callback runs atomically (softirq context, cannot sleep) and fires at a scheduled jiffy, not “whenever a worker thread gets around to it.” The neighbour subsystem uses a timer for each entry’s per-entry state, and (separately) a delayed_work for table-wide garbage collection — we’ll meet the GC one later.

When the timer fires, neigh_timer_handler (net/core/neighbour.c:1103) runs. It reads now = jiffies (neighbour.c:1114), looks at nud_state and the timing constants, and either re-arms itself (still probing) or transitions the entry. The real transitions, straight from the source:

• REACHABLE → if now is past confirmed + reachable_time, demote to STALE (or DELAY if the entry was recently used). (neighbour.c:1120-1139)
• DELAY → if confirmation came in, promote back to REACHABLE; otherwise go to PROBE and start actively soliciting. (neighbour.c:1140-1162)
• PROBE / INCOMPLETE → re-arm at now + retrans_time to send the next probe — until the probe budget is exhausted.

That budget is the missing piece behind “6 probes / 3 probes.” It’s computed by neigh_max_probes (net/core/neighbour.c:1054):

static __inline__ int neigh_max_probes(struct neighbour *n)
{
    struct neigh_parms *p = n->parms;
    return NEIGH_VAR(p, UCAST_PROBES) + NEIGH_VAR(p, APP_PROBES) +
           (n->nud_state & NUD_PROBE ? NEIGH_VAR(p, MCAST_REPROBES)
                                     : NEIGH_VAR(p, MCAST_PROBES));
}

With ARP defaults (UCAST_PROBES = 3, MCAST_PROBES = 3, MCAST_REPROBES = 0, APP_PROBES = 0):

• INCOMPLETE (not yet in PROBE): 3 + 0 + 3 = 6 is the threshold.
• PROBE (already had a MAC, re-confirming): 3 + 0 + 0 = 3 is the threshold.

But the threshold and the number of packets actually sent differ for INCOMPLETE. When __neigh_event_send first drives an entry into INCOMPLETE it pre-seeds the probe counter to UCAST_PROBES = 3 (neighbour.c:1224, atomic_set(&neigh->probes, UCAST_PROBES)). The timer then increments the counter once per multicast solicitation, climbing 3 → 4 → 5 → 6; at 6 it equals the threshold and the entry fails. So only 3 multicast ARP requests are actually emitted in INCOMPLETE — and they are all multicast, because arp_solicit computes probes -= UCAST_PROBES (arp.c:376) which is always ≥ 0 here, forcing the multicast branch every time. No unicast probe is possible while INCOMPLETE: there is no MAC to unicast to. (Those UCAST_PROBES unicast probes belong to PROBE-state revalidation, where the entry already has a stale MAC.)

Once probes >= neigh_max_probes(neigh) for an INCOMPLETE or PROBE entry, the handler gives up (neighbour.c:1164-1169): INCOMPLETE → NUD_FAILED, PROBE → STALE/FAILED. That is the timer-driven mechanism behind “first ping takes ~1ms longer” (one resolution round-trip before the entry is REACHABLE) and “INCOMPLETE → FAILED after the probes.”

The state machine

A neighbour entry walks through states — and now you know the timer is what moves it:

• NONE — newly created, no resolution attempted yet.
• INCOMPLETE — ARP request sent, waiting for reply. Skbs going to this neighbour queue here.
• REACHABLE — fresh, recent confirmation (default reachable_time = 30s, i.e. 30 * HZ jiffies).
• STALE — has MAC but old. Next packet triggers a confirmation probe.
• DELAY — sent a packet expecting the L4 to confirm reachability quickly.
• PROBE — actively probing via ARP.
• FAILED — resolution timed out. INCOMPLETE sends ~3 multicast ARP requests ~1s apart, then fails; PROBE-state revalidation (entry already has a stale MAC) sends up to 3 unicast probes instead. (Full derivation — why the budget is 6 but only 3 are sent — is in the timer-background section above.)

The state machine is the canonical answer to “why does my first ping take 1ms longer than subsequent ones?” — first packet goes through INCOMPLETE → REACHABLE; later packets hit a cached entry directly.

Lookup on TX

The packet that triggers ARP: async resolution, a bounded queue, and two fast paths

ip_finish_output2 does not call ARP directly. It reaches the inline neigh_output(neigh, skb, skip_cache) helper, which makes two decisions: can it use a cached L2 header immediately, and if not, which function does neigh->output currently name? Follow the states rather than treating the pointer as the whole fast path:

1. First packet, no usable address. A new NUD_NONE entry has neigh->output = neigh->ops->output, which for ordinary ARP is neigh_resolve_output. neigh_event_send → __neigh_event_send changes NONE to INCOMPLETE, arms the timer, sends a solicitation, queues the supplied skb, and returns 1. neigh_resolve_output therefore does not transmit that skb; TX unwinds instead of blocking for an ARP round-trip.

2. Only unresolved packets queue. While the state is INCOMPLETE, skbs are parked on the per-neighbour arp_queue. Its arp_queue_len_bytes is bounded by QUEUE_LEN_BYTES; when adding a new skb would exceed the limit, the loop dequeues and drops the oldest entries with SKB_DROP_REASON_NEIGH_QUEUEFULL until the new skb fits. One unreachable neighbour therefore has a fixed memory budget.

3. The reply lands. arp_rcv → neigh_update stores the MAC and changes the state to REACHABLE. neigh_connect writes neigh->output = neigh->ops->connected_output, and neigh_update_process_arp_queue drains the unresolved queue. It relooks up the top-level neighbour for each skb before reinjecting it because qdiscs such as shapers can lead to a different neighbour object.

4. Connected output has two forms. On the ordinary call path, neigh_output() first checks NUD_CONNECTED and hh.hh_len. If both hold, neigh_hh_output() copies the cached L2 header under its sequence lock, pushes it, and calls dev_queue_xmit without calling neigh->output at all. If no header cache is usable, the helper falls through to the function pointer; generic neigh_connected_output() rebuilds the header with dev_hard_header and transmits it. hh_cache and neigh_connected_output are alternatives, not one combined fast path.

5. STALE is usable, not unresolved. When reachability ages, neigh_suspect restores the resolving output function. The next packet reaches neigh_resolve_output; __neigh_event_send changes STALE to DELAY but returns success, so that packet is sent immediately with the known MAC. If upper-layer confirmation does not arrive during delay_first_probe_time, the timer advances to PROBE and sends solicitations. A STALE packet does not enter arp_queue.

If INCOMPLETE probing exhausts its budget, neigh_timer_handler sets FAILED and neigh_invalidate removes queued skbs through ops->error_report before purging the remainder. For IPv4, arp_error_report calls dst_link_failure and frees the skb with SKB_DROP_REASON_NEIGH_FAILED; the exact user-visible error depends on the route/socket context.

This connects forward: routing chooses the next hop that the neighbour subsystem resolves, and the bounded oldest-first eviction pattern reappears in the queueing discussion on Day 23.

Which IPs actually get ARP’d? on-link vs via-gateway

The FAILED lab below hinges on a routing fact, so here’s the rule (the full FIB is Day 8). The kernel only ARPs for the next hop, and the next hop depends on routing:

• A destination on a directly-connected subnet (on-link) is its own next hop — the kernel ARPs for it directly.
• A destination outside every local subnet is reached via the gateway — so the kernel ARPs for the gateway, and never for the remote IP at all.

“Directly connected” just means the destination matches one of the interface’s configured subnet prefixes (e.g. 192.168.1.0/24 on eth0). This is the same next-hop IP that Day 3 said ip_finish_output2 hands to the neighbour subsystem: routing chooses the next-hop IP, neighbour resolves it to a MAC. And it’s confirmed by the table itself — arp_tbl has key_len = 4 and is keyed by primary_key (neighbour.h:169), i.e. one entry per next-hop IPv4 address.

This is why the “Watch FAILED state” lab must target an unused address inside your own subnet: only then is the destination its own next hop, so your packet triggers a real ARP that can time out to FAILED. An off-subnet address like 10.99.99.99 would just resolve the gateway (already cached) and never produce a FAILED entry for the address you typed.

There are no Dumb Questions

Q: What is “gratuitous ARP”?

A: A node announcing its MAC unsolicited (e.g., on interface up, or after a failover). Linux sends one as an arp_send(ARPOP_REQUEST, ETH_P_ARP, ...) with target IP == source IP, from inetdev_send_gratuitous_arp in net/ipv4/devinet.c. It updates other nodes’ caches without them having to ask. (With the wire format in hand: it’s a request whose TIP == SIP — see the ARP wire-format section.)

Q: What’s arp_announce for?

A: Sysctl that controls which IP the kernel uses as source for ARP requests. Default 0 (any local IP); 1 prefers same-subnet; 2 always uses the primary IP of the egress interface. Important for multi-homed servers.

Q: Why are there gc_thresh1/2/3?

A: gc_thresh3 is a hard cap; new ARP attempts beyond it fail. Heavy-traffic gateways routinely hit the default 1024 — symptom: neighbor table overflow! in dmesg. Bump via sysctl. (The exact mechanism is the next section.)

How garbage collection enforces gc_thresh1/2/3

The table can’t grow forever, so a per-table atomic counter, gc_entries, is bumped every time an entry is created — and creation consults the three thresholds before allocating. The check, in the create path (net/core/neighbour.c:507):

entries = atomic_inc_return(&tbl->gc_entries) - 1;
gc_thresh3 = READ_ONCE(tbl->gc_thresh3);
if (entries >= gc_thresh3 ||
    (entries >= READ_ONCE(tbl->gc_thresh2) &&
     time_after(now, READ_ONCE(tbl->last_flush) + 5 * HZ))) {
    if (!neigh_forced_gc(tbl) && entries >= gc_thresh3) {
        net_info_ratelimited("%s: neighbor table overflow!\n", tbl->id);
        NEIGH_CACHE_STAT_INC(tbl, table_fulls);
        goto out_entries;          /* returns -ENOBUFS */
    }
}

That’s the whole three-tier behaviour, grounded:

• Below gc_thresh1 (128): the periodic GC doesn’t even bother — neigh_periodic_work skips the bucket scan with if (atomic_read(&tbl->entries) < gc_thresh1) goto out; (net/core/neighbour.c:1000), and the out: path still re-queues the delayed_work, so the worker keeps rescheduling itself. (That periodic GC is a delayed_work running in the process context Day 5 introduced for netns teardown — no need to re-teach it.)
• At/above gc_thresh2 (512) and more than 5 s (5 * HZ) since the last flush: a forced synchronous GC runs, neigh_forced_gc (net/core/neighbour.c:253), reclaiming stale entries back down toward gc_thresh2.
• At/above gc_thresh3 (1024): if even forced GC can’t get back under the cap, creation fails with -ENOBUFS and emits the rate-limited "%s: neighbor table overflow!" — the exact dmesg symptom this chapter and the Check question describe.

Two things to keep straight:

• The data packet isn’t dropped because of this. The kernel just couldn’t create the neighbour entry; that surfaces as a resolution failure / brief hang, not a thrown-away packet at the GC layer. (That’s the grounding for the Check answer’s “it just refuses to create new entries.”)
• Reclamation targets STALE/old entries; NUD_PERMANENT entries are exempt and never GC’d. That’s why the static nud permanent pins in the labs survive GC pressure.

Today’s experiment

Before you start: the commands below use eth0 and the gateway 192.168.1.1 as examples. Replace them with your interface and default gateway — derive both from ip route show default:

IFACE=$(ip route show default | awk '{print $5; exit}')
GW=$(ip route show default | awk '{print $3; exit}')
echo "iface=$IFACE gateway=$GW"
# iface=eth0 gateway=10.0.0.1

Run verbatim against a machine whose interface is ens3/enp0s3 or whose gateway is different, and ip neigh flush dev eth0 fails with “Cannot find device” and ping 192.168.1.1 goes nowhere. Substitute $IFACE/$GW (or your real values) wherever you see eth0/192.168.1.1 below.

See your ARP table

ip neigh show
# 192.168.1.1 dev eth0 lladdr aa:bb:cc:11:22:33 REACHABLE
# 192.168.1.20 dev eth0 lladdr aa:bb:cc:44:55:66 STALE

Force an ARP from scratch

sudo ip neigh flush dev eth0
ping -c 1 192.168.1.1
ip neigh show
# entry now REACHABLE

Watch ARP packets

sudo tcpdump -i eth0 -n arp -e &
sudo ip neigh flush dev eth0
ping -c 1 192.168.1.1
sudo pkill tcpdump   # stop the background capture

You’ll see the ARP request (broadcast) and reply (unicast) — and now you can read them field-by-field: the request goes to ff:ff:ff:ff:ff:ff with ARPOP_REQUEST (op 1) and an empty target MAC; the reply comes back unicast with ARPOP_REPLY (op 2) and the target MAC filled in. The pkill is essential: a backgrounded tcpdump with no count/timeout otherwise runs forever, spamming your terminal with every later ARP on the link.

Trace neighbour state changes

sudo bpftrace -e '
fentry:neigh_update {
  printf("update: state=%d new_state=%d\n",
         args->neigh->nud_state, args->new);
}'

The numbers are NUD bitmask values (include/uapi/linux/neighbour.h): 1=INCOMPLETE, 2=REACHABLE, 4=STALE, 8=DELAY, 16=PROBE, 32=FAILED. In another terminal, flush and re-ping the gateway to drive a resolution. neigh_update() is the packet/netlink-driven path — it’s only ever called with NUD_STALE, NUD_REACHABLE, NUD_FAILED, or a user/ndisc-supplied state, so this hook catches transitions like INCOMPLETE→REACHABLE on an arriving reply:

# update: state=1 new_state=2   (INCOMPLETE -> REACHABLE, ARP reply received)

What this hook will never show is the internal STALE→DELAY, DELAY→PROBE, or timeout-driven INCOMPLETE→FAILED transitions: those are WRITE_ONCE(neigh->nud_state, ...) stores inside __neigh_event_send (neighbour.c:1249) and neigh_timer_handler — they don’t flow through neigh_update() at all. To watch those, attach an fentry on neigh_timer_handler instead, or trace the nud_state field directly.

These tie directly back to the state machine at the top of the chapter. (Swap %d for %x in the printf if you’d rather see the raw hex #define values.)

Probe gc thresholds

sysctl net.ipv4.neigh.default.gc_thresh1   # default 128
sysctl net.ipv4.neigh.default.gc_thresh2   # default 512
sysctl net.ipv4.neigh.default.gc_thresh3   # default 1024

# bump for high-fanout servers:
sudo sysctl -w net.ipv4.neigh.default.gc_thresh3=8192

# restore the default when you're done (this change is non-persistent —
# it's lost on reboot and not written to sysctl.conf):
sudo sysctl -w net.ipv4.neigh.default.gc_thresh3=1024

These are the very gc_thresh1/2/3 fields from arp_tbl (net/ipv4/arp.c:152) whose create-time check produces neighbor table overflow!.

What to break

Statically pin a wrong MAC

Warning: do not pin a wrong MAC for the gateway on the interface you’re SSH-ing in over — it drops your session before you can run the undo, locking you out. Pin a non-gateway LAN peer instead, or run this from the local console.

sudo ip neigh add 192.168.1.1 lladdr aa:bb:cc:00:00:00 dev eth0 nud permanent
ping -c 3 -W 1 192.168.1.1   # no replies — packets sent to a MAC nobody owns

# undo:
sudo ip neigh del 192.168.1.1 dev eth0

This shows the cache is authoritative until the kernel re-resolves. The -c 3 -W 1 is important: a bare ping runs until you Ctrl-C, and until you do, the del line never executes and the entry stays poisoned. (The nud permanent entry is also GC-exempt — it survives table pressure until you delete it.)

Watch FAILED state

To actually watch INCOMPLETE → FAILED you must (a) target an address that is directly connected — pick an unused IP inside your eth0 subnet, not an off-subnet one like 10.99.99.99 (those route via the gateway, so the kernel never ARPs for them — see the on-link vs via-gateway box) — and (b) send real traffic to kick off resolution. Note too that a bare ip neigh add IP dev eth0 with no lladdr/nud is rejected by modern iproute2 (“No link layer address given”) and creates no entry at all.

# pick an unused IP in your eth0 subnet, e.g. 192.168.1.250
sudo ip neigh flush 192.168.1.250 dev eth0 2>/dev/null
ping -c1 -W1 192.168.1.250 || true   # queues a packet -> kernel starts ARPing
sleep 8                              # past the INCOMPLETE probe budget: counter climbs from its UCAST_PROBES seed (3) to neigh_max_probes()=6, sending 3 multicast probes ~1s apart
ip neigh show 192.168.1.250
# 192.168.1.250 dev eth0  FAILED     (briefly INCOMPLETE first)
sudo ip neigh del 192.168.1.250 dev eth0   # cleanup

The 8-second wait is no accident: it’s past the INCOMPLETE probe budget. The counter is seeded at UCAST_PROBES (3) and increments once per multicast solicitation, so it sends 3 multicast probes spaced retrans_time (1 * HZ) apart (~3s of probing) before neigh_max_probes() = 6 trips and the timer flips the entry to FAILED. Eight seconds leaves a comfortable margin.

What to read in the kernel

• net/core/neighbour.c — the generic subsystem. Read neigh_lookup (line 625), ___neigh_create (line 646), neigh_update, neigh_timer_handler (line 1103 — the state machine engine), neigh_max_probes (line 1054 — the probe budget), and the GC create-time check (lines 507–516).
• net/ipv4/arp.c — ARP-specific protocol. arp_rcv (line 967), arp_send (line 323), arp_process (line 702), arp_solicit, and arp_tbl (line 152). ~1500 lines.
• include/net/neighbour.h — struct neighbour (line 140), struct neigh_hash_table (line 201), the nht field (line 244), the NUD_* state constants.
• include/uapi/linux/if_arp.h — struct arphdr (line 145), ARPOP_REQUEST/ARPOP_REPLY (lines 107–108).
• Documentation/networking/ip-sysctl.rst — the neigh.* sysctls (gc thresholds, probe counts, timers, and the jiffies-vs-*_ms aliases); pair it with the source itself.

Bullet Points

• The kernel measures time in jiffies (jiffies increments once per timer tick; HZ ticks/second). A duration is seconds * HZ jiffies — hence 30 * HZ, 1 * HZ in the source. neighbour.confirmed/updated are jiffies snapshots; staleness is a time_after(now, confirmed + reachable_time) comparison.
• An ARP packet is struct arphdr (hw/proto type, lengths, opcode) + SHA/SIP/THA/TIP. ARPOP_REQUEST (1) is broadcast with an empty THA; ARPOP_REPLY (2) is unicast with THA filled in. Gratuitous ARP = a request with TIP == SIP.
• The neighbour subsystem (ARP for IPv4, NDP for IPv6) lives at net/core/neighbour.c. The “table” is a resizable, RCU-protected hash table (neigh_hash_table) keyed per next-hop IP; lookups walk one bucket lock-free; entries are refcounted and RCU-freed (same free-at-zero model as Day 1’s sk_buff).
• A per-entry timer_list drives the state machine: neigh_timer_handler re-arms or transitions entries. neigh_max_probes() = 6 for INCOMPLETE, 3 for PROBE (ARP defaults), but because the counter is pre-seeded to UCAST_PROBES (3), an INCOMPLETE entry actually emits 3 multicast ARP requests before FAILED; the 3 unicast probes belong to PROBE-state revalidation of an entry that already has a (stale) MAC.
• Entries cycle through NONE → INCOMPLETE → REACHABLE → STALE → DELAY → PROBE → REACHABLE/FAILED.
• The kernel only ARPs the next hop: on-link destinations are their own next hop; off-subnet ones resolve the gateway (full FIB is Day 8).
• TX-side lookup is a lockless RCU hash lookup — ip_neigh_for_gw → __ipv4_neigh_lookup_noref in ip_finish_output2, falling back to __neigh_create on a miss (the exported neigh_lookup is for netlink/proc callers, not the per-packet path). Skbs queue on arp_queue while a neighbour is INCOMPLETE.
• GC thresholds (gc_thresh1/2/3, default 128/512/1024) cap entry count via a gc_entries counter checked at create time; exceeding gc_thresh3 after forced GC returns -ENOBUFS and logs neighbor table overflow!. NUD_PERMANENT entries are GC-exempt.
• Inspect: ip neigh show. Manipulate: ip neigh add/del/replace.

Check question

You set net.ipv4.neigh.default.gc_thresh3=128 on a server with 500 active client peers. What symptoms appear?

Tomorrow

Day 8: IP routing — the FIB. How the kernel decides where to send a packet (and therefore which next-hop IP the neighbour subsystem resolves), and why route lookups are so cheap on modern hardware.

Day 8 — IP routing: the FIB

Today’s mission: see how Linux decides where to send a packet, learn the data structure that powers route lookups, and understand how the result gets stapled onto the packet — and how ip route programs the table in the first place. We’ll teach four mechanisms the routing path leans on (the dst_entry handle, longest-prefix-match semantics, the LC-trie, and the rtnetlink config channel) so nothing in the path is a black box. Total time: ~110 minutes.

What “routing” means in Linux

Every packet that needs to go somewhere not-here requires a routing decision: which next-hop, which output interface, which source IP. The decision is fast — sub-microsecond on modern hardware, even against tables with hundreds of thousands of routes.

That speed comes from the FIB (Forwarding Information Base) and the LC-trie (level-compressed trie) data structure backing it. But before we touch the trie, we have to be honest about some things the rest of this chapter quietly assumes you already know: what it means for a route to “match,” what handle the lookup result is delivered through, how the trie itself works, and how a route got into the table at all. Days 1–7 never taught any of these. Two of them we teach up front (what “match” means, and the handle the result rides on); the trie itself and the write-side config channel we build up as we reach them — intuition first, then the concrete v7.1 struct.

Background 1: longest-prefix-match — what “the route matches” actually means

Picture your routing table as a stack of rules, each one saying “addresses that start with these bits go this way.” That “starts with these bits” idea is a prefix, and a route is a prefix plus a prefix length:

• 10.0.0.0/24 means “addresses whose top 24 bits equal 10.0.0.0.”
• 10.0.0.0/8 means “addresses whose top 8 bits equal 10.”
• 0.0.0.0/0 — the default route — has prefix length 0, so zero bits have to match. It matches everything.

Here’s the catch that makes routing interesting: a single destination address can match several routes at once. A packet to 10.0.0.5 matches all three of the routes above simultaneously. So which one wins?

The rule: the most specific match wins — the longest prefix length. 10.0.0.0/24 (24 bits) beats 10.0.0.0/8 (8 bits) beats 0.0.0.0/0 (0 bits). This is longest-prefix-match (LPM), and it’s what makes routing deterministic. It’s also why the default route is the last resort: 0.0.0.0/0 only wins when nothing more specific matched. (That’s exactly why the “break a default route” lab later in this chapter works — strip away the specific routes and the /0 catches the packet.)

The lookup result records which prefix won. That result is a struct fib_result (include/net/ip_fib.h:173):

struct fib_result {
    __be32          prefix;     /* the matched prefix, e.g. 10.0.0.0 */
    unsigned char   prefixlen;  /* its length, e.g. 24 — how the winner was chosen */
    unsigned char   nh_sel;
    unsigned char   type;       /* RTN_UNICAST, RTN_LOCAL, RTN_BROADCAST, ... */
    unsigned char   scope;
    struct fib_nh_common *nhc;  /* next-hop info */
    struct fib_info *fi;
    struct fib_table *table;
    /* ...tclassid, dscp, fa_head in v7.1... */
};

prefix and prefixlen are the matched prefix and its length — the winner of the LPM contest. The interesting field is nhc, the next-hop, a struct fib_nh_common (include/net/ip_fib.h:83):

struct fib_nh_common {
    struct net_device   *nhc_dev;     /* egress interface */
    int                  nhc_oif;
    u8                   nhc_gw_family;
    union {
        __be32          ipv4;         /* gateway IP, if any */
        struct in6_addr ipv6;
    } nhc_gw;
    /* ... */
};

It tells you the egress device (nhc_dev/nhc_oif) and the gateway IP (if any) (nhc_gw_family + nhc_gw). (The preferred source IP and the route MTU are not in fib_nh_common — they come from the enclosing fib_nh’s nh_saddr and from the route’s metrics on the dst, respectively.)

The type field is why the same machinery answers both “forward this” and “this is for us.” RTN_UNICAST (include/uapi/linux/rtnetlink.h:263) is a gateway or direct route; RTN_LOCAL (:264) means “accept locally”; RTN_BROADCAST (:265). The kernel keeps its own host IPs as RTN_LOCAL entries in a separate table — more on that below.

One more connection: LPM is why the FIB is a trie keyed on address bits and not a hash. A hash gives you exact-match — great for “is this exact key present?” But routing needs “what’s the longest prefix that is a leading substring of this address?” A bitwise trie answers that naturally: descend as far as the address bits keep agreeing, and the deepest matching prefix is the answer. That’s the structure we build up in Background 3.

Background 2: the dst_entry — how a route result rides on the packet

The lookup found a winner. Now what? The kernel doesn’t return a raw “go out eth0 via 192.168.1.1” answer and leave the caller to interpret it. Instead it builds a small object, staples it onto the skb, and lets the packet carry its own “what happens next” instructions. That object is the dst_entry (“destination entry”), and it’s the single most important handle in the forwarding path.

Intuition: a per-packet “next step” with two function pointers

Think of dst_entry as a tiny instruction card pinned to the packet. The card has two slots:

• input(skb) — what to do with the packet on the receive/forward side.
• output(net, sk, skb) — what to do on the transmit side.

This is the same function-pointer-as-vtable idea Day 3 used for proto_ops and Qdisc_ops: instead of a giant if/else over packet types, you store a function pointer and call through it. The dst_entry struct itself is new, but the dispatch pattern is one you already know.

Here’s the struct (include/net/dst.h:26):

struct dst_entry {
    union {
        struct net_device       *dev;
        struct net_device __rcu *dev_rcu;
    };
    struct dst_ops          *ops;                               /* :31 */
    /* ...metrics, expires, xfrm... */
    int  (*input)(struct sk_buff *);                            /* :39 */
    int  (*output)(struct net *net, struct sock *sk, struct sk_buff *skb); /* :40 */
    unsigned short flags;
    /* ... */
};

The route lookup doesn’t return a dst — it returns an rtable that embeds one

When the IPv4 route lookup finishes, it produces a struct rtable (route table entry, include/net/route.h:57):

struct rtable {
    struct dst_entry    dst;     /* <-- the very FIRST member */
    int                 rt_genid;
    unsigned int        rt_flags;
    __u16               rt_type;
    /* ...rt_gw4/rt_gw6, rt_iif, mtu... */
};

Look at the first member: struct dst_entry dst. Because it’s first, an rtable * and a dst_entry * point at the same address — an rtable is a dst with extra fields hanging off the end. To go the other way (recover the full rtable from a bare dst), the kernel uses a container_of helper (include/net/route.h:80):

#define dst_rtable(_ptr) container_of_const(_ptr, struct rtable, dst)

This is the classic Linux “embed the generic struct as the first member, recover the concrete one with container_of” trick. The routing code fills in rt->dst.input and rt->dst.output depending on what the lookup decided (net/ipv4/route.c):

• rt->dst.input = ip_local_deliver — packet is for us (:1668).
• rth->dst.input = ip_forward — transit traffic (:1894).
• rth->dst.input = ip_error — unreachable; rate-limits and sends ICMP (:2442).
• rt->dst.output = ip_output — the TX direction (:1666).

(ip_local_deliver lives at net/ipv4/ip_input.c:250; ip_forward at net/ipv4/ip_forward.c:83 — the same two handlers Day 2 mentioned without explaining.)

Stapling it on: skb_dst_set and dst_input

Two skb helpers connect the dst to the packet:

• skb_dst_set(skb, &rt->dst) (include/linux/skbuff.h:1217) staples the dst onto the skb. It’s stored in skb->_skb_refdst (skbuff.h:923) — a single unsigned long that holds the pointer plus a low “is-it-refcounted?” bit.
• skb_dst(skb) (skbuff.h:1159) reads it back.

And the entire “dispatch the next step” mechanism is one inline function, dst_input (include/net/dst.h:478):

static inline int dst_input(struct sk_buff *skb)
{
    return INDIRECT_CALL_INET(READ_ONCE(skb_dst(skb)->input),
                              ip6_input, ip_local_deliver, skb);
}

Strip the INDIRECT_CALL_INET wrapper (it’s a retpoline/speculation optimization that hints the two likely targets) and it reads: call skb_dst(skb)->input(skb). That’s it. The whole “the skb’s dst->input dispatches the next step” claim is this single jump through a function pointer that the route lookup pre-loaded.

The fast path: skip the lookup if a dst is already attached

The RX code first checks whether the skb already has a usable dst attached, and if so, skips the whole lookup. That check is skb_valid_dst (include/net/dst_metadata.h:93):

static inline bool skb_valid_dst(const struct sk_buff *skb)
{
    struct dst_entry *dst = skb_dst(skb);
    return dst && !(dst->flags & DST_METADATA);
}

“Has a dst, and it’s a real route (not a metadata placeholder used by tunnels).” If that’s true, the packet already knows where it’s going and the kernel jumps straight to dst_input.

The lookup pipeline

For an incoming packet, ip_rcv_finish (via ip_rcv_finish_core) first checks whether the skb already has a dst attached (skb_valid_dst → fast path, Background 2). If not, it calls ip_route_input_noref (net/ipv4/route.c:2546) which:

1. Calls fib_lookup → walks fib_rules to pick a table.
2. Calls fib_table_lookup (net/ipv4/fib_trie.c:1420) on that table — the LC-trie walk (Background 3).
3. Builds an rtable from the result, attaches it to the skb via skb_dst_set — loading rt->dst.input per Background 2.

The skb’s dst->input function pointer then dispatches the next step via dst_input, jumping to whichever handler the lookup loaded (ip_local_deliver / ip_forward / ip_error — Background 2). The one new detail worth pinning down here: ip_error handles route type RTN_UNREACHABLE (it rate-limits and sends ICMP Destination Unreachable), whereas TTL-exceeded is not dispatched here — it is detected inside ip_forward, which then emits ICMP Time Exceeded.

Anatomy of a lookup

The lookup key is a struct flowi4:

struct flowi4 {
    __be32  saddr;
    __be32  daddr;
    dscp_t  flowi4_dscp;
    __u32   flowi4_mark;
    int     flowi4_oif;
    int     flowi4_iif;
    __u8    flowi4_proto;
    /* ... ports, etc. */
};

(In v7.1 these fields are really #define aliases over an inner struct flowi_common __fl_common — e.g. flowi4_oif is __fl_common.flowic_oif. The flattened view above is a fair simplification; the trace lab below uses the real flp4->__fl_common.flowic_oif form.)

The result is the struct fib_result you met in Background 1 — prefix/prefixlen record which LPM winner was chosen, and nhc carries the egress device (nhc_dev) and the gateway (nhc_gw). (The preferred source IP and MTU live elsewhere — in the enclosing fib_nh’s nh_saddr and the route’s metrics on the dst — not in fib_nh_common.)

Background 3: the LC-trie — what it is and why it’s fast

The internet has ~1M IPv4 routes globally. A linear scan would be hopeless. The FIB uses an LC-trie (“level-compressed trie”) and the chapter keeps calling it “fast” — let’s earn that claim by building it up from scratch.

Start with a trie

A trie is a tree where you navigate by the digits of the key, not by comparing whole keys. For IP routing the key is the 32 bits of an IPv4 address, so the simplest version is a binary trie: each node looks at one bit of the address and branches left (0) or right (1). A route with prefix length L lives at depth L — you walked L bits to reach it.

The problem is depth. A /32 host route sits 32 levels deep, so a worst-case lookup is 32 memory dereferences — one per bit. On a million-route table that’s far too slow. This naive binary trie is the baseline the LC-trie improves on, with two compressions.

Compression #1: path compression (the first “C”)

Walk down a binary trie and you’ll often hit long runs where every node has exactly one child — a stretch of address space with only a single route underneath it. Spending one node (and one memory access) per bit through that run is pure waste.

Path compression collapses any chain of single-child internal nodes into one node that simply records how many bits to skip. The long single-child chains stop costing a node per bit.

Compression #2: level compression (the second “C”)

In the dense regions of the trie — where almost every branch is populated — you can do the opposite of skipping: fan out wider. Instead of branching on one bit per level, level compression replaces several binary levels with a single node that branches on multiple bits at once: a node consuming k bits has up to 2^k children and resolves k address bits in one memory access.

How v7.1 encodes both in one node

Both compressions live in a single struct, struct key_vector (net/ipv4/fib_trie.c:121):

struct key_vector {
    t_key key;
    unsigned char pos;    /* which bit position this node tests */
    unsigned char bits;   /* how many bits this node consumes */
    unsigned char slen;
    union {
        struct hlist_head leaf;                              /* if IS_LEAF */
        DECLARE_FLEX_ARRAY(struct key_vector __rcu *, tnode); /* if IS_TNODE — :130 */
    };
};

The two fields carry the whole scheme:

• pos is the bit position this node examines — path compression lives here, because pos can jump past skipped bits.
• bits is how many bits this node consumes in one step — level compression lives here. bits == 0 is a leaf (IS_LEAF, :119); bits > 0 is an internal tnode (IS_TNODE, :118). A node with bits = 4 has up to 16 children (the DECLARE_FLEX_ARRAY of child pointers is the multi-bit fan-out in memory) and consumes 4 address bits at once.

The key length is fixed by the address width: KEYLENGTH = 8 * sizeof(t_key) (:112) and typedef unsigned int t_key (:115), so KEYLENGTH = 32 for IPv4 — the 32 levels of the naive trie we started from. Internal nodes are wrapped in a struct tnode (:134) carrying child-occupancy bookkeeping (empty_children, full_children) and the parent pointer.

The payoff

On real internet tables, the dense regions get flattened (fewer levels via level compression) and the sparse single-child chains get skipped (path compression), so lookups average ~5–10 node visits regardless of table size — instead of up to 32. That’s the “fast” the chapter promised.

Finally, tie it back to LPM (Background 1): the walk descends to a leaf following the address bits, then checks the leaf’s fib_alias list to find the most specific prefix whose bits actually match the key. The trie narrows the candidates; the leaf check confirms the match length. Read net/ipv4/fib_trie.c (fib_table_lookup, line 1420) and Documentation/networking/fib_trie.rst for the gory details — both are well-commented.

How the walk actually works: each tnode owns a branching window

A plain binary trie examines one address bit per level. An LC-trie compresses both paths and levels: every internal key_vector (tnode) stores pos and bits, selecting a multi-bit window of the 32-bit host-order key and a child array of size 1 << bits.

fib_table_lookup begins with key = ntohl(flp->daddr) and uses:

struct key_vector {
    t_key key;
    unsigned char pos;   /* low-order suffix bits below this window */
    unsigned char bits;  /* width of this node's child index        */
    unsigned char slen;  /* longest suffix length below this node   */
    /* leaf hlist, or a (1 << bits)-entry child array                */
};

The hot expression is index = get_cindex(key, n) = (key ^ n->key) >> n->pos. The shift discards the low pos suffix bits, which are below this node’s branching window; it does not discard bits already examined on the way down. If all more-significant prefix and path-compressed bits agree with n->key, the remaining value fits in n->bits and directly indexes one child. A tnode with bits = 4 therefore selects among 16 children in one dereference. If index >= 1 << n->bits, some differing bit lies above the window, so this subtree cannot contain the key and lookup enters prefix backtracking.

The first descent aims at the most specific candidate. When index fits and bits == 0, it has reached a leaf whose key passed that step’s comparison. A leaf holds a sorted hlist of fib_alias routes; fa_slen is the number of low-order suffix bits excluded from that alias’s prefix, so prefixlen = 32 - fa_slen. The semantic scan also checks DSCP, route liveness/type/scope, output-interface constraints, and usable nexthops before filling fib_result.

Backtracking enumerates valid shorter-prefix candidates

If descent finds a missing child, a prefix mismatch, or no usable alias/nexthop, lookup does not restart at the root. It keeps a parent pn and child index cindex only where slen > pos says a shorter prefix may exist below that point.

At backtrace, a nonzero cindex is changed with cindex &= cindex - 1, clearing its least-significant set bit and selecting the next prefix-aligned predecessor branch in that child array. If cindex is already zero, the code ascends until an ancestor has another candidate, recomputes that ancestor’s child index with get_index, and continues. Prefix-matching mode then follows the first non-NULL child of a candidate subtree, while prefix_mismatch(key, n) and n->slen == n->pos prune branches that cannot contain a shorter match.

At a candidate leaf, index = key ^ n->key; an alias is eligible only when every differing bit fits inside its ignored suffix (index < 1 << fa_slen, with the 32-bit guard in the source). Thus longest-prefix behavior comes from the combination of deepest-first descent, aliases ordered by prefix length, and this structured backtrace toward shorter candidates — not from “the first leaf always wins” and not from blindly climbing one trie level per prefix bit.

Route insertion keeps the structure efficient by resizing tnodes: inflate widens dense branching windows and halve narrows sparse ones. Those operations change bits and child placement without changing the lookup contract.

You can see the resulting shape directly:

cat /proc/net/fib_trie

Each +-- line is an internal tnode printed as prefix/len bits full_children empty_children; each |-- line is a leaf with its routes. Watch a bits value of 2 or 3 on a busy line — that’s level compression fanning a node out to 4 or 8 children so the walk skips whole levels.

Read net/ipv4/fib_trie.c if you want the resize math; the lookup loop itself (lines 1420–1545) is the part worth reading top-to-bottom.

Multiple tables

Linux supports multiple routing tables. The defaults:

• local (255) — IPs on local interfaces; kernel-maintained.
• main (254) — user-added routes go here by default.
• default (253) — lowest priority; rarely used.
• custom — created with ip route add ... table N. Small examples often use IDs below 253, but the kernel carries table IDs as u32; larger IDs travel through netlink attributes such as RTA_TABLE (Background 4).

(These match RT_TABLE_LOCAL=255 / RT_TABLE_MAIN=254 / RT_TABLE_DEFAULT=253 in include/uapi/linux/rtnetlink.h:360.)

fib_rules decide which table to consult based on packet attributes: source IP, fwmark, OIF, IIF. Day 9 covers them.

Background 4: how a route gets into the FIB — the rtnetlink path

Every lab in this chapter uses ip route add/del/show, and the lookup side walks a trie those commands populated. But how does ip route add 10.99.0.0/16 dev lo actually become a leaf in the LC-trie? So far we’ve only looked at the read side. Here’s the write side.

ip route is a netlink client

ip route does not poke the kernel via ioctl or /proc. It opens an AF_NETLINK socket in the NETLINK_ROUTE family (“rtnetlink”) and sends a structured message:

• Adding a route → an RTM_NEWROUTE message (include/uapi/linux/rtnetlink.h:44, value 24).
• Deleting → RTM_DELROUTE (:46).
• ip route show → an RTM_GETROUTE dump (:48).

The message carries a struct rtmsg plus typed attributes (TLVs — type-length-value records): RTA_DST (the prefix), RTA_GATEWAY, RTA_OIF (egress interface), and RTA_TABLE (:385) — the very attribute the “Multiple tables” section mentioned for large table IDs.

Kernel side: per-message-type handlers that mutate the FIB

Each message type is registered to a handler function (net/ipv4/fib_frontend.c:1694):

static const struct rtnl_msg_handler fib_rtnl_msg_handlers[] __initconst = {
    {.protocol = PF_INET, .msgtype = RTM_NEWROUTE,
     .doit = inet_rtm_newroute, .flags = RTNL_FLAG_DOIT_PERNET},
    {.protocol = PF_INET, .msgtype = RTM_DELROUTE,
     .doit = inet_rtm_delroute, .flags = RTNL_FLAG_DOIT_PERNET},
    {.protocol = PF_INET, .msgtype = RTM_GETROUTE, .dumpit = inet_dump_fib, ...},
};

So RTM_NEWROUTE → inet_rtm_newroute (:910), RTM_DELROUTE → inet_rtm_delroute (:876), RTM_GETROUTE → inet_dump_fib (:1018, the dump that backs ip route show). inet_rtm_newroute parses the attributes and calls fib_table_insert (:930) — the function that actually grows the LC-trie the lookup side walks.

That closes the loop:

ip route add 10.99.0.0/16 dev lo
  → AF_NETLINK socket: RTM_NEWROUTE { rtmsg + RTA_DST/RTA_OIF/RTA_TABLE }
    → inet_rtm_newroute → fib_table_insert → new leaf in the LC-trie
ping 10.99.0.1
  → fib_table_lookup walks the trie → hits that freshly-inserted leaf

Why the local table fills itself

This also explains where the kernel’s own host-IP routes come from. When an interface gets an IP assigned, the kernel issues the same RTM_NEWROUTE internally — with type RTN_LOCAL — via fib_magic (net/ipv4/fib_frontend.c:1156: fib_magic(RTM_NEWROUTE, RTN_LOCAL, addr, 32, ...)). No userspace involved; the local table (255) is populated through the identical insert path your ip route add uses. We don’t need the deeper netlink framing (nlmsghdr, attribute parsing) here — the teaching point is just config arrives as rtnetlink messages dispatched to per-type handlers that mutate the FIB.

There are no Dumb Questions

Q: How does the kernel cache routes?

A: Modern Linux (since ~3.6) does NOT cache per-flow rtables. Instead, route lookups are cheap enough to do per-packet. The exception is per-CPU caches for outbound connect paths and the dst_cache mechanism that lwtunnels use. The pre-3.6 “rt_cache” was a security risk and is gone.

Q: Where does the kernel keep its own host IPs?

A: In the local FIB table (id 255). Kernel auto-inserts entries when interfaces get IPs assigned — via the internal fib_magic(RTM_NEWROUTE, RTN_LOCAL, ...) path from Background 4. ip route show table local lists them.

Q: How fast is a lookup, in practice?

A: ~50ns–150ns on modern x86_64 against a typical Linux server’s small FIB. Internet routers with 1M routes see ~200–500ns per lookup, dominated by memory access patterns and cache effects.

Today’s experiment

Inspect your routes

ip route show table main
ip route show table local
ip rule show

ip rule is the fib_rules list. A vanilla kernel installs three rules (local, main, default); some distros or host services add extra rules (e.g. a 220: from all lookup 220 entry), so you may see more. (Each ip route show you run is an RTM_GETROUTE dump handled by inet_dump_fib — Background 4.)

Watch a route lookup

sudo bpftrace -e '
fentry:fib_table_lookup {
  printf("lookup daddr=%s table_id=%d\n",
         ntop(args->flp->daddr),
         args->tb->tb_id);
}'

# in another terminal:
ping -c 1 8.8.8.8

You’ll see at least one fib_table_lookup per originated packet; exact counts vary with dst caching.

Add a route

sudo ip route add 10.99.0.0/16 dev lo
ip route show

# remove:
sudo ip route del 10.99.0.0/16

The add is silent on success; ip route show confirms it by listing the new prefix:

10.99.0.0/16 dev lo scope link

Under the hood, that ip route add sent an RTM_NEWROUTE message into inet_rtm_newroute → fib_table_insert, which inserted a fresh leaf into the main table’s LC-trie (Background 4). To connect this back to the FIB lookup, leave the fentry:fib_table_lookup trace from the previous section running, then in another terminal:

ping -c 1 10.99.0.1

The trace prints lookup daddr=10.99.0.1 table_id=254 — the LC-trie in the main table (254) now resolves the prefix you just added. Remember to run the ip route del above when you’re done so your host table is left clean.

Trace ip_route_output_flow

sudo bpftrace -e '
fentry:ip_route_output_flow {
  printf("out: daddr=%s oif=%d\n", ntop(args->flp4->daddr), args->flp4->__fl_common.flowic_oif);
}'

# in another terminal:
ping -c 1 8.8.8.8

This is the outbound counterpart to ip_route_input — it fires for every locally-originated packet. You’ll see one line per outbound route lookup:

out: daddr=8.8.8.8 oif=0

oif=0 means the caller did not pin an egress interface, so the lookup is free to choose one. (A box that talks to the network constantly will print background lookups too, e.g. DNS resolvers and the like.) Press Ctrl-C to stop.

What to break

Break a default route safely, inside a namespace

Do not replace your host’s real default route. Put the failure in a disposable namespace:

sudo ip netns add fibbreak
sudo ip -n fibbreak link set lo up
# onlink: tell the kernel to trust this gateway even though it is
# not on any subnet configured on lo (lo carries only 127.0.0.1/8).
sudo ip -n fibbreak route add default via 10.99.99.99 dev lo onlink
sudo ip -n fibbreak route show                  # default via 10.99.99.99 dev lo onlink
sudo ip netns exec fibbreak ping -c 1 8.8.8.8   # 100% packet loss, namespace only

sudo ip netns del fibbreak

With onlink the route installs, so the FIB lookup succeeds and resolves a next-hop — but 10.99.99.99 answers no ARP, so transmission has nowhere to go and the ping reports 100% packet loss. (Without onlink the kernel rejects the off-subnet gateway outright with Error: Nexthop has invalid gateway. and the route is never added at all.) Either way your host routing table never changes, and ip netns del tears down the namespace and its routes completely.

This is also LPM in miniature (Background 1): the only route present is 0.0.0.0/0, so a packet to 8.8.8.8 matches it because there was nothing more specific.

Inspect rt cache stats (legacy)

cat /proc/net/stat/rt_cache

The per-flow cache-hit columns (in_hit/out_hit) are zero because the per-flow rt_cache is gone — that’s the teaching point. The in_slow_tot/out_slow_tot/in_martian_src columns are just cumulative lookup counters, so they will be non-zero. The proc file remains for compatibility.

What to read in the kernel

• net/ipv4/route.c — ip_route_input_noref (line 2546), ip_route_output_flow (line 2929). Also the rt->dst.input/rt->dst.output assignments (lines 1666–1668, 1894, 2442).
• net/ipv4/fib_trie.c — fib_table_lookup (line 1420). The LC-trie implementation; struct key_vector (line 121), IS_TNODE/IS_LEAF (lines 118–119).
• net/ipv4/fib_frontend.c — netlink interface: inet_rtm_newroute (line 910), inet_rtm_delroute (line 876), inet_dump_fib (line 1018), the handler table (line 1694), fib_magic (definition at line 1099; the RTN_LOCAL call site is line 1156), and the fib_lookup wrapper.
• net/ipv4/fib_rules.c — fib_rules implementation.
• include/net/dst.h — struct dst_entry (line 26), dst_input (line 478).
• include/net/route.h — struct rtable (line 57), dst_rtable (line 80).
• include/net/flow.h — struct flowi4.
• include/net/ip_fib.h — struct fib_result (line 173), struct fib_nh_common (line 83).
• Documentation/networking/fib_trie.rst — LC-trie internals; Documentation/networking/ip-sysctl.rst — routing-related sysctls.

Bullet Points

• A route is a (prefix, prefixlen) pair. Many routes can match one address; the longest prefix wins (LPM). 0.0.0.0/0 (the default) matches everything and is the last resort.
• The lookup result is a struct fib_result: prefix/prefixlen record the LPM winner; nhc (a fib_nh_common) carries the egress dev and gateway IP — the preferred source IP (nh_saddr) and MTU live in the enclosing fib_nh/route metrics, not in nhc. type (RTN_UNICAST/RTN_LOCAL/RTN_BROADCAST) is why the same machinery handles “forward” and “for us.”
• A dst_entry is the per-packet “next step” handle, carrying input(skb) (RX/forward) and output(net,sk,skb) (TX) function pointers. The route lookup builds a struct rtable whose first member is a dst_entry (so rtable* == dst*; recover with dst_rtable). skb_dst_set staples it on; dst_input just calls skb_dst(skb)->input(skb) — that’s the entire dispatch.
• rt->dst.input is set to ip_local_deliver / ip_forward / ip_error. skb_valid_dst is the fast-path check that skips the lookup when a real dst is already attached.
• FIB is the kernel’s routing table; lookups go through fib_table_lookup against an LC-trie. A naive binary trie is 32 levels deep; path compression skips single-child chains and level compression uses multi-bit (2^bits-way) nodes, so lookups average ~5–10 node visits regardless of table size. struct key_vector{pos,bits} encodes both (bits==0 = leaf).
• The lookup key is struct flowi4; the result is struct fib_result.
• Three well-known tables: local (255), main (254), default (253). Custom table IDs are u32; low IDs are just convenient examples. fib_rules decide which table to consult.
• ip route is a netlink client. Add/del/show map to RTM_NEWROUTE/RTM_DELROUTE/RTM_GETROUTE → inet_rtm_newroute/inet_rtm_delroute/inet_dump_fib; inserts go through fib_table_insert. The kernel auto-populates the local table via the same path (fib_magic, RTN_LOCAL).
• No more flow-level rt_cache as of ~3.6 — lookups are cheap enough per-packet.

Check question

You add ip route add 10.0.0.0/24 via 192.168.1.1 dev eth0. A packet with daddr 10.0.0.5 enters. Walk the lookup.

Tomorrow

Day 9: multipath, policy routing, source-based routing. The fib_rules machinery — which decides which table each fib_table_lookup consults — gets serious.

Day 9 — Multipath, policy routing, source-based routing

Today’s mission: route packets based on something other than destination IP. Learn the four ideas the whole chapter leans on — packet marks, reverse-path filtering, why TCP hates reordering, and the nexthop object model — then use multiple FIB tables, policy rules, and ECMP with confidence. Budget ~130 minutes; if you already know fwmark, rp_filter, and DSCP, the four background boxes can be skimmed.

When destination-based routing isn’t enough

Yesterday’s lookup (Day 8) was a function of one input: the destination IP. You fed a struct flowi4 to fib_table_lookup, walked the LC-trie in the main table, and got back a struct fib_result with a next-hop. That’s the simple model and it covers ~95% of real-world routing — your laptop, most servers, most embedded devices. But there are hard cases where you genuinely need more inputs to make the decision:

• Source-based routing. Two tenants share a router; tenant A’s traffic must go through gateway A, tenant B through gateway B. The destination might be the same; the source IP is what differentiates.
• Mark-based routing. A firewall rule classified this packet as “VPN traffic” and stamped a mark on it. You want the route to follow that classification.
• Multipath (ECMP). You have two equal-cost paths to the same destination. Spread connections across them but keep each individual flow on a single path (so packets don’t reorder).
• Policy routing. “Traffic from this UID, on this incoming interface, with this DSCP, goes to that table.” The combinatorics of policy rules motivate multiple routing tables.

Linux solves all of these with the same machinery: fib_rules plus multiple FIB tables.

Four of these — the mark, reverse-path filter, DSCP byte, and flow-hash — lean on concepts you haven’t formally met (plus the nexthop object model the resilient-hashing section needs). They’re taught inline, intuition first, where the chapter first needs them — so every selector and every ip command reads as plain English when you reach it.

Background 1: what a packet mark actually is

The bullet list above, the fwmark selector, the entire “mark-based routing” section, and today’s Check question all hinge on “the mark.” Netfilter — the firewall subsystem that sets marks — isn’t taught until Day 20, so you’ve never formally met one. Let’s fix that now, because the idea is simple and you’ll lean on it all day.

A mark is a u32 scratch field that lives on the skb itself. Recall the sk_buff from Day 1: the universal packet container, a descriptor riding alongside the packet bytes. One of its fields is a 32-bit integer the kernel uses purely as an internal classification tag (include/linux/skbuff.h:1069):

union {
    __u32  mark;
    __u32  reserved_tailroom;
};

Two things to absorb about that field:

1. It travels with the packet inside the kernel and is never put on the wire. It is not part of the IP header, not an Ethernet field, nothing a remote host can see. It exists only while the skb is alive in your kernel. Think of it as a sticky note some earlier subsystem slapped on the packet so a later subsystem can read it.
2. Something must SET it before routing can match it. The mark starts at 0. It gets a non-zero value from one of two places:

   • An nftables/iptables rule — meta mark set 0x42 (nftables) or -j MARK --set-mark 0x42 (iptables). This is a firewall hook stamping the packet as it passes through.

   • The originating socket — a process calls setsockopt(fd, SOL_SOCKET, SO_MARK, ...), and every packet that socket sends is born with that mark. SO_MARK is option number 36 (include/uapi/asm-generic/socket.h:56):

     #define SO_MARK    36
     

The fwmark rule selector you’ll meet below only reads the mark. It never sets it. This is the single most common point of confusion, so say it out loud: a routing rule reads the mark; netfilter or a socket writes it.

How the mark reaches the route lookup

Here’s the connection back to Day 8. When the kernel is about to route a packet, it copies the skb’s mark into the lookup key — specifically into flowi4_mark, a field you already saw listed in struct flowi4 on Day 8. Now the rule engine has something to compare against. The generic rule matcher does exactly this (net/core/fib_rules.c:292):

if ((rule->mark ^ fl->flowi_mark) & rule->mark_mask)
    goto out;   /* mismatch — this rule does not apply */

That’s a masked compare, and it’s why fwmark selectors take a VALUE/MASK form. The XOR (rule->mark ^ fl->flowi_mark) is zero exactly where the bits agree; the & rule->mark_mask ignores any bits you don’t care about. So fwmark 0x42 matches when the masked bits equal 0x42, and fwmark 0x42/0xff says “only look at the low byte.” When you create a rule, iproute2 stuffs your value into rule->mark (net/core/fib_rules.c:622):

nlrule->mark = nla_get_u32(tb[FRA_FWMARK]);

Forward-reference honestly: how marks get set — the nftables/iptables rule syntax, which chain runs when, conntrack restoring a mark on reply traffic — is netfilter’s job, covered on Day 20. For today, treat the mark as a label some earlier hook stamped on the packet, and focus on how routing reacts to it.

fib_rules: which routing table to consult

fib_rules is a per-protocol decision pipeline. Each rule is a predicate (selectors that match packet attributes) plus a target (which routing table to look up in). At lookup time, the kernel walks rules in priority order and uses the first match. Implementation: net/core/fib_rules.c:313 — fib_rules_lookup.

The default rules

Run ip rule show on a clean system and you get exactly three rules:

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

• Priority 0 → table 255 (local). Holds the IPs of this host’s interfaces. Kernel auto-populates as you ip addr add. Always tried first so packets to ourselves resolve correctly.
• Priority 32766 → table 254 (main). Default destination for ip route add. User-added routes land here.
• Priority 32767 → table 253 (default). Lowest priority; rarely populated. Historical compatibility.

The lower the priority number, the earlier the rule is consulted. Custom rules slot in between.

Selectors: what each rule can match on

Each rule supports a set of selectors. The full list — see include/net/fib_rules.h:20 struct fib_rule and the per-protocol extension struct fib4_rule — includes:

• from PREFIX (src field): match by source IP/prefix.
• to PREFIX (dst field): match by destination (rarely useful — that’s what FIB lookup does anyway).
• iif IFNAME: match by input interface. Powerful for forwarding routers that need to apply different policy per arrival port.
• oif IFNAME: match by output interface. Less common; usually you want the route to select the OIF, not the other way around.
• tos VALUE / dsfield VALUE: match the DSCP class in the second byte of the IPv4 header (the top 6 bits — a QoS class set by applications or edge routers, “gold” vs “best-effort”; the low 2 bits are ECN, deliberately excluded from matching). You already saw flowi4_dscp listed in struct flowi4 on Day 8; this selector connects that field to the wire byte, so you can steer gold traffic onto one table and bulk traffic onto another. The two spellings look at different widths: dsfield matches the full 6-bit DSCP (mask 0xfc — flowi4_dscp is a dscp_t that already has the ECN bits cleared, so they never participate), while tos masks off the top three bits too and compares only the lower DSCP bits against INET_DSCP_LEGACY_TOS_MASK (0x1c) for backward compatibility — see fib_dscp_masked_match (include/net/ip_fib.h:441). At runtime it compiles to the same masked compare as the mark — (r->dscp ^ fl4->flowi4_dscp) & r->dscp_mask (net/ipv4/fib_rules.c:197).
• fwmark VALUE/MASK: match the packet mark (Background 1). Critical for firewall-driven policy.
• uidrange UID-UID: match by socket-owner UID. Lets you route a specific user’s traffic differently (VPN-by-user).
• ipproto PROTO, sport/dport RANGE: match by L4. Layer-violating but powerful.

Source-based routing — the canonical example

Two tenants: tenant A uses 10.99.0.0/24 and must egress via 192.168.99.1; tenant B uses 10.100.0.0/24 and goes through 192.168.100.1.

# Custom tables for each tenant:
sudo ip route add default via 192.168.99.1 table 100
sudo ip route add default via 192.168.100.1 table 200

# Rules:
sudo ip rule add from 10.99.0.0/24  lookup 100 priority 100
sudo ip rule add from 10.100.0.0/24 lookup 200 priority 200

Now any packet from a 10.99-prefix source consults table 100 first; from 10.100, table 200; everyone else falls through to main. The rule priorities (100 and 200) are rule walk order, not the table IDs — keep them straight.

Gotcha: Linux defaults to strict reverse-path filtering (net.ipv4.conf.all.rp_filter = 1 on many systems), and strict reverse-path filtering will silently drop the traffic you just so carefully steered. To understand why, you need to know what reverse-path filtering actually does — which is Background 2, immediately below. The fix is to relax it (rp_filter=2, loose mode) on the relevant interface; the reason that fix works will be obvious in two paragraphs.

Background 2: reverse-path filtering, the mechanism

Reverse-path filtering (rp_filter) is an anti-spoofing check. The intuition: when a packet shows up claiming to be from some source address, the kernel asks a skeptical question — “if I had to send a reply back to this sender, would I even route it out the interface this packet just came in on?” If the answer is no, the packet’s source address is probably forged, and the kernel drops it as a martian (an impossible packet).

Here’s the clever part: the kernel answers that question using the exact same FIB lookup machinery you learned on Day 8 — just run backwards. It takes the packet’s source address, treats it as if it were a destination, and does a routing lookup. The function is fib_validate_source (net/ipv4/fib_frontend.c:429), the entry point invoked on RX, which calls __fib_validate_source (net/ipv4/fib_frontend.c:345). That helper takes an int rpf argument selecting the mode, and the two modes diverge right here:

if (rpf == 1)        /* strict: net/ipv4/fib_frontend.c:405 */
    goto e_rpf;
...
if (rpf)             /* loose: net/ipv4/fib_frontend.c:417 */
    goto e_rpf;

The e_rpf: label (net/ipv4/fib_frontend.c:424) is where the martian-source drop is signalled — return -SKB_DROP_REASON_IP_RPFILTER; on line 425. (Don’t confuse this with the comment at line 447 in the separate fib_validate_source wrapper, whose “within the same container, it is regarded as a martian source” path returns -SKB_DROP_REASON_IP_LOCAL_SOURCE on line 451 — a different local-source check, not the rp_filter verdict.) The three modes:

• Strict (1): the reverse lookup must resolve back to the same interface the packet arrived on. Asymmetric paths are rejected.
• Loose (2): the source need only be reachable via any interface. Asymmetric paths survive.
• Off (0): no check at all.

Why this breaks source-based routing

Now the gotcha from the previous section explains itself. The reverse-path check runs a full fib_lookup — and once you’ve added any ip rule (exactly the scenario here), net->ipv4.fib_has_custom_rules is true, so that lookup does walk your custom rules (fib_lookup only short-circuits straight to main when no custom rules exist; include/net/ip_fib.h:374). The reason it still drops your traffic is subtler: the reverse lookup is built with source and destination swapped — fl4.daddr = src; fl4.saddr = dst (net/ipv4/fib_frontend.c:359-360). Your from <tenant-prefix> rule selects on the source field, but in the reverse direction the tenant’s source address is now the lookup’s destination, so that rule no longer selects your custom table. And even when a route is found, strict mode demands its output interface equal the arrival interface (if (rpf == 1) goto e_rpf, net/ipv4/fib_frontend.c:405); under asymmetric/policy routing it doesn’t, the kernel shouts “martian!”, and your carefully-routed packet is dropped before it ever reaches your rule.

Relaxing to loose mode (2) fixes it because loose mode only asks “is this source reachable somehow?” — which it is — instead of demanding the path be symmetric. That’s the whole reason the source-routing recipe tells you to set rp_filter=2 on the relevant interface. See Documentation/networking/ip-sysctl.rst for the per-interface sysctl details (note: the effective value is the max of conf.all.rp_filter and conf.<iface>.rp_filter).

Mark-based routing

This is the second canonical fib_rules recipe (the first was source-based routing above), and it pairs the routing side with netfilter — the mark from Background 1 in action.

# Rule: marked traffic uses table 200
sudo ip rule add fwmark 0x42 lookup 200 priority 200
sudo ip route add default via 192.168.42.1 table 200

# Mark anything destined for example.com:
sudo nft add rule inet filter output ip daddr 93.184.216.34 meta mark set 0x42

That meta mark set 0x42 is the netfilter side from Background 1 writing the mark; the ip rule add fwmark 0x42 is the routing side reading it. This is how policy routers, VPN clients (mwan3), and per-application VPN configs work.

Gotcha: this is the ordering subtlety from the mark box, now concrete — mark in OUTPUT after the route was bound at connect(), and that connection’s path is already fixed. To force re-routing on mark change, applications often combine setsockopt(SO_MARK) (Background 1 — the socket stamps every packet at birth, before routing) with route lookups, or use the suppress_prefixlength 0 rule action, which makes the kernel ignore a previously-cached route decision and re-run the rules.

Multiple FIB tables in detail

Routing table IDs are 32-bit values in modern netlink (RTA_TABLE). The familiar 253/254/255 IDs are well-known defaults, and examples often use small numbers like 100 or 200, but the kernel is not capped at 256 tables. Each table is a self-contained FIB and exists as soon as you add a route to it. To list a specific table:

ip route show table 100
ip route show table all   # everything everywhere

You name tables in /etc/iproute2/rt_tables (the canonical path; on some distros — e.g. Azure Linux — the file ships at /usr/share/iproute2/rt_tables and you may need to create /etc/iproute2/ first):

# echo "100  vpn"  >> /etc/iproute2/rt_tables
# now you can use 'vpn' as the table name
sudo ip route add default via 10.0.0.1 table vpn

Background 3: why packet reordering wrecks TCP

The next section’s central design driver — and the entire justification for resilient nexthop groups later — is the rule “keep one flow on one path.” The chapter is about to claim that splitting a connection across two paths makes “TCP throughput tank for seconds.” But TCP isn’t taught until Day 13+, so you’re being asked to accept the most important idea in the chapter on faith. Let’s not do that. Here’s just enough to make the rule self-evident.

First, define a “flow.” A flow is the set of packets sharing the same 5-tuple: source IP, destination IP, source port, destination port, and protocol. Every packet of a single TCP connection has the same 5-tuple. ECMP’s job is to hash a connection’s fixed fields so that every packet of one connection lands on the same next-hop — which is exactly what the multipath hasher does. By default (hash policy 0) the kernel hashes only the source/destination IP pair, so all connections between the same pair of hosts share one next-hop; enabling policy 1 adds the L4 ports for true per-5-tuple spreading. Either way a single connection always hashes the same, so its packets stay on one next-hop. You can see what “a flow” means to the kernel by reading fib_multipath_hash (net/ipv4/route.c:2066): in case 0 it hashes only the v4 source and destination addresses; in case 1 it adds the L4 ports. Those fields are the flow identity.

Now, why keeping a flow together matters. TCP has no direct “a packet was lost” signal — it infers loss from ordering. The receiver ACKs the highest in-order byte it has. If packets arrive out of order, the receiver keeps re-sending the same ACK number — duplicate ACKs. When the sender sees three duplicate ACKs, it concludes a packet was lost, triggers a fast retransmit, and cuts its congestion window (its sending rate). Throughput drops, then has to climb back up slowly.

Here’s the trap: nothing was actually lost. If you spread one connection’s packets across two paths of unequal latency, the packets do all arrive — just out of order, because the slow path delivers its share late. But out-of-order arrival generates exactly the duplicate-ACK pattern that looks like loss. TCP’s loss detector fires, the window collapses, and throughput tanks for seconds. Repeat every time the paths reorder, and a “working” multipath setup delivers terrible performance.

So the rule “same flow → same next-hop” is not about correctness — the packets would arrive either way. It’s about not fooling TCP’s loss detector. This is precisely why ECMP hashes per-flow instead of per-packet, and why rehashing flows onto new paths (which the next section is all about) is so damaging.

(The full duplicate-ACK / fast-retransmit / congestion-window story is Day 16. For today: reordering looks like loss to TCP, and looking like loss costs throughput.)

ECMP — multiple next-hops to one destination

When two paths to the same destination cost the same in routing-protocol terms, you want to use both. Linux’s solution is ECMP (Equal-Cost Multi-Path): a single route entry with multiple next-hops. The kernel hashes each flow’s fixed field-tuple (the “flow” you just met in Background 3) to pick one next-hop and keep that connection on it — the diagram shows the shape; the mechanics follow.

Configure with the multi-line form of ip route:

sudo ip route add default \
    nexthop via 192.168.1.1 weight 1 \
    nexthop via 192.168.2.1 weight 1

How the kernel picks one

For each new flow, the kernel computes a hash from packet fields and maps it onto the next-hops using the hash-threshold algorithm (RFC 2992), not a modulo. Each next-hop owns a contiguous slice of the 0..2^31-1 hash space sized by its weight: fib_rebalance (net/ipv4/fib_semantics.c:824) gives every next-hop a cumulative upper_bound = DIV_ROUND_CLOSEST_ULL((u64)w << 31, total) - 1, and fib_select_multipath (net/ipv4/fib_semantics.c:2164) picks the first next-hop whose hash <= nh_upper_bound. Because the hash is over the 5-tuple (Background 3), every packet of a connection produces the same hash and so lands in the same slice — that’s how ECMP keeps a flow together.

The hash function is configurable via net.ipv4.fib_multipath_hash_policy:

• 0 — L3 only: hash (src_ip, dst_ip). Same source-destination pair always uses one nexthop, even if it’s many connections. Poor balance for client-server workloads with few clients.
• 1 — L4: hash (src_ip, dst_ip, src_port, dst_port, proto). Different connections from the same client spread across nexthops. Useful when a few client/server pairs carry many connections, but it is an opt-in sysctl setting, not the kernel default.
• 2 — Inner L3: for tunneled traffic, hash on the inner IPs, not the outer. Used when a single tunnel carries many flows that should spread across paths.
• 3 — Custom: hash on a bitmask of fields you select via net.ipv4.fib_multipath_hash_fields. The mode itself is selected by fib_multipath_hash_policy (which accepts 0–3, extra2 = SYSCTL_THREE); when mode 3 is chosen, the field bitmask comes from fib_multipath_hash_fields (a mask, validated against fib_multipath_hash_fields_all_mask). Lets you cherry-pick exactly which L3/L4 fields enter the hash.

The kernel default is mode 0. Pick mode 1 when you want better balancing across many L4 flows between the same endpoints. Mode 2 is for overlay networks (VXLAN, GRE) where outer addresses are the same for many inner flows.

The reordering trade-off

ECMP guarantees a connection’s packets stay on the same path as long as the nexthop set is unchanged. The moment you add or remove a nexthop, fib_rebalance recomputes every slice boundary — so flows whose hash now falls in a shifted range jump to a different nexthop, mid-flight. (It’s not literally every flow under hash-threshold — flows whose hash still lands in an unshifted range stay put — but the churn is significant, since the boundaries move.) Those reassigned flows get reordered or routed differently — and now you know exactly what that costs: Background 3’s duplicate-ACK avalanche, fast-retransmit, and a window cut. TCP throughput tanks for seconds, not because anything was lost, but because reordering looks like loss.

Background 4: the nexthop object model

The fix for that boundary-shift problem (resilient hashing, next section) is about to switch from the inline nexthop via ... syntax you just used to a completely different way of writing routes:

sudo ip nexthop add id 10 ...
sudo ip nexthop add id 100 group 10/20 ...
sudo ip route add default nhid 100

Day 8 only ever showed next-hops embedded inline in a route (the fib_nh_common inside a fib_result). You’ve never seen a next-hop as a standalone, ID-addressable kernel object. Here’s the model.

Modern Linux (5.3+) lets a next-hop be a first-class object with its own id. Instead of baking the gateway/device into each route entry, you create the next-hop once, give it an id, and then point as many routes at it as you like via nhid. The object is struct nexthop (include/net/nexthop.h:142) — it carries an id and can be referenced by many routes.

A nexthop object can be a single next-hop OR a group. The group 10/20 form makes a group object whose members are other nexthop objects (id 10 and id 20). A group has a type:

• mpath — classic hash-threshold ECMP (what plain inline multipath gives you): weighted slices of the hash space, recomputed when the set changes.
• resilient — the bucket-table scheme (next section).

Why resilient hashing needs this API at all: the bucket-to-nexthop mapping table is state, and it has to live somewhere. In the old inline route format there’s no place to attach it — a route entry is just a prefix plus an embedded next-hop. A standalone group object has somewhere to keep that table. That’s the whole reason the new syntax exists.

Contrast cleanly with Day 8. Day 8’s fib_result.nhc was an inline next-hop pointer. With the object model, the route’s fib_info instead has an fi->nh pointing at a shared struct nexthop. You can see the kernel branch on exactly that in fib_select_multipath (net/ipv4/fib_semantics.c:2164):

struct fib_info *fi = res->fi;
...
if (unlikely(res->fi->nh)) {        /* a nexthop OBJECT, not an inline array */
    nexthop_path_fib_result(res, hash);
    return;
}
/* else: walk fi's inline nexthop array, the Day-8 form */

So res->fi->nh non-NULL means “this route points at a shared nexthop object”; NULL means “old-style inline next-hops.” Same fib_select_multipath, two source shapes.

Resilient nexthop groups (kernel 5.13+)

The fix for boundary-shift-on-change: resilient hashing. Plain ECMP recomputes the weighted hash-space slices whenever the set changes, so the boundaries move. Resilient instead inserts a fixed-size bucket table between the hash and the nexthops: nexthop_select_path_res indexes it with bucket_index = hash % res_table->num_nh_buckets (net/ipv4/nexthop.c:1479), and the bucket count (up to 65535) is constant regardless of how many nexthops exist. So the modulus never changes when you add or remove a nexthop — only the buckets that pointed at the removed nexthop get reassigned; every other bucket (and so every flow that wasn’t on the removed path) is undisturbed, confining Background 3’s penalty to just the affected flows.

sudo ip nexthop add id 10 via 192.168.1.1 dev eth0
sudo ip nexthop add id 20 via 192.168.2.1 dev eth0
sudo ip nexthop add id 100 group 10/20 type resilient buckets 65535 idle_timer 120
sudo ip route add default nhid 100

The three ip nexthop lines are the object model from Background 4: two member objects, then a resilient group object referencing them, then a route pointing at the group by nhid. The buckets 65535 sizes the bucket table — nexthop_res_table_alloc (net/ipv4/nexthop.c:561) allocates it sized by cfg->nh_grp_res_num_buckets (the buckets keyword), and the per-bucket bookkeeping uses struct nh_res_bucket (include/net/nexthop.h:74) inside a struct nh_res_table (include/net/nexthop.h:82). The idle_timer is how long a bucket can be idle before the kernel can re-balance it; the default is 120 seconds (net/ipv4/nexthop.c:20 — #define NH_RES_DEFAULT_IDLE_TIMER (120 * HZ)), which is the value cited above. unbalanced_timer is the soft deadline for forcing a rebalance.

Implementation: net/ipv4/nexthop.c:563 and surrounding (the num_nh_buckets/nh_res_table machinery). The background worker nh_res_table_upkeep (net/ipv4/nexthop.c:1841) is what walks the bucket table and migrates only the buckets that must move.

Use resilient hashing on production gateways with multiple uplinks. Plain ECMP is fine for symmetric, stable topologies (rare in practice).

Today’s experiment

# Source-based test on local interfaces. Save cleanup first.
cleanup() {
  sudo ip rule del from 10.99.0.0/24 lookup 99 priority 99 2>/dev/null || true
  sudo ip route del default via 127.0.0.1 table 99 2>/dev/null || true
  sudo ip route del 10.99.0.0/24 dev lo 2>/dev/null || true
  sudo ip addr del 10.99.0.5/32 dev lo 2>/dev/null || true
}
trap cleanup EXIT

# 10.99.0.5 must be a real local address — `ping -I` binds the socket to it,
# and the kernel rejects the bind if the address isn't assigned to an interface.
sudo ip addr add 10.99.0.5/32 dev lo
sudo ip rule add from 10.99.0.0/24 lookup 99 priority 99
sudo ip route add default via 127.0.0.1 table 99

# Trace which table is consulted
sudo bpftrace -e '
fentry:fib_table_lookup { printf("table_id=%d\n", args->tb->tb_id); }
' &
tracer=$!
sleep 2   # let bpftrace attach its fentry probes before we send traffic

# Send packet from 10.99 source — should hit table 99
ping -I 10.99.0.5 -c 1 8.8.8.8

# And from default — should hit main (254)
ping -c 1 8.8.8.8

sudo kill "$tracer"

For each ping you’ll first see one or more table_id=255 lines — that’s the always-first local table lookup (rule 0), which misses for an external destination like 8.8.8.8. The first ping then shows table_id=99 — proof the from 10.99.0.0/24 rule steered it into table 99 — while the second ping falls through to table_id=254 (main). The leading 255 lines are normal: fib_rules_lookup walks rule 0 first on every lookup, so don’t be surprised if 255 repeats.

Neither ping will actually get a reply from 8.8.8.8 — that’s expected. Table 99’s next-hop 127.0.0.1 is a deliberate dead-end used only to force a distinct table, and the route is loopback-bound. We only care about the table_id=... lines bpftrace prints, not the ping result.

There are no Dumb Questions

Q: If the mark never goes on the wire, how does the other end of a connection route reply traffic by mark?

A: It doesn’t see your mark. The mark is local to one kernel. On the reply side, that host’s own netfilter (often via conntrack restoring a saved mark) stamps its own mark on the reply skb. Each kernel marks independently. Day 20 covers the conntrack “save/restore mark” pattern that makes the two sides appear coordinated.

Q: I set a mark in the OUTPUT chain but my connection still takes the old path. Bug?

A: No — it’s the stickiness gotcha. The route was chosen when the socket bound, before your OUTPUT rule ran. The cleanest fix is to set the mark at the source via setsockopt(SO_MARK) (Background 1) so the packet is born marked, before routing. Marking later only affects connections routed after the mark exists.

Q: rp_filter dropped my traffic but there’s nothing in the firewall logs. Where did it go?

A: rp_filter is not a firewall rule — it’s the fib_validate_source martian check (Background 2), and it drops silently (you can count martians via net.ipv4.conf.<iface>.log_martians=1 to get kernel log lines). If source-based routing “just doesn’t work” with no visible drop, suspect strict rp_filter first and set it to 2 on the arrival interface.

Q: Why not just hash per-packet and get perfect balance?

A: Per-packet spraying reorders flows across paths of unequal latency, and TCP reads that as loss — see Background 3 (“why packet reordering wrecks TCP”). Per-flow hashing trades a little balance for not lying to TCP’s loss detector.

What to read in the kernel

• net/core/fib_rules.c:313 — fib_rules_lookup. The rule-walking dispatcher used by all protocols (IPv4, IPv6). Read top to bottom (~50 lines for the function itself). Notice how it iterates ops->rules_list in priority order, calls ops->match for each rule’s predicate, and ops->action once a match is found. The protocol-specific rule type plugs into this generic engine.

• net/ipv4/fib_rules.c — IPv4 specialization. The match callback fib4_rule_match checks src, dst, tos/dscp, fwmark, ipproto. Read it to see what each rule selector compiles to at runtime — it’s just a chain of masked compares against flowi4 fields (e.g. the dscp compare at line 197, the same form as the generic mark compare).

• net/ipv4/fib_frontend.c:429 — fib_validate_source (and __fib_validate_source at line 345). This is reverse-path filtering. Watch the rpf == 1 (strict, line 405) vs rpf (loose, line 417) split and the e_rpf: label (line 424) whose return -SKB_DROP_REASON_IP_RPFILTER; (line 425) is the martian drop. It’s a reverse FIB lookup — Day 8’s machinery run backwards on the source address (with src/dst swapped).

• net/ipv4/fib_semantics.c:2164 — fib_select_multipath. Picks one nexthop given a precomputed hash. Note the res->fi->nh branch at the top (nexthop object vs inline array, Background 4), then the loop that walks fib_info’s next-hop array comparing hash <= nh_upper_bound — each nexthop owns a weighted slice of the hash space (the upper bounds set by fib_rebalance at line 824), no modulo involved.

• net/ipv4/route.c:2066 — fib_multipath_hash. Read case 0 (v4 src/dst only) and case 1 (adds L4 ports) to see exactly what “a flow” means to the hasher (Background 3).

• include/net/ip_fib.h:556 — fib_multipath_hash_from_keys. The hash function for fib_multipath_hash_policy=0/1/2/3. Notice how it folds the chosen fields into a flow_keys struct, then runs flow_hash_from_keys_seed — a keyed siphash (net/core/flow_dissector.c:1803, the siphash() call inside __flow_hash_from_keys). Reading this answers “which exact bytes does the kernel hash for ECMP?”

• net/ipv4/nexthop.c:563 and surrounding — resilient nexthop groups. The num_nh_buckets/nh_res_table and bucket migration machinery. Look at nh_res_table_upkeep (line 1841) to see how the kernel decides which buckets to migrate when a nexthop is added/removed without disturbing others.

• include/net/fib_rules.h:20 — struct fib_rule. The selectors are the fields. Quick read; gives you the complete vocabulary of what’s matchable.

• Documentation/networking/ip-sysctl.rst — the official reference for the routing/rp_filter/multipath sysctls used here. One read; explains the knobs end-to-end.

Bullet Points

• A packet mark is a u32 scratch field on the skb (sk_buff.mark) that rides inside the kernel and never goes on the wire. Netfilter (meta mark set) or a socket (setsockopt(SO_MARK)) writes it; the fwmark rule selector only reads it, via a masked compare (rule->mark ^ flowi_mark) & mark_mask.
• fib_rules decide which routing table to consult. Default: local → main → default.
• Selectors: from, iif, oif, tos/dsfield (the IPv4 DSCP class → flowi4_dscp; ECN bits excluded), fwmark, uidrange, ipproto, sport/dport.
• Routing table IDs are u32. Custom tables via ip route add … table N. Names in /etc/iproute2/rt_tables.
• Source-based routing is the most common use case; mark-based for firewall integration.
• Reverse-path filtering (fib_validate_source) is an anti-spoofing reverse FIB lookup on the source address. Strict (rp_filter=1) demands the reply path return to the arrival interface and so silently drops asymmetric source-routed traffic; set loose (2) on the relevant interface.
• ECMP: multi-nexthop route, kernel hashes per-flow to pick one — by default the source/destination IP pair (hash policy 0); the full 5-tuple only with policy 1. Same flow → same nexthop, because reordering a flow across paths looks like loss to TCP (dup-ACKs → fast-retransmit → window cut), not because packets would be lost.
• Hash policy: 0 (L3 — kernel default), 1 (L4, opt-in for better flow spread), 2 (inner L3 for tunnels), 3 (custom fields via fib_multipath_hash_fields).
• Plain ECMP shifts the weighted hash-space boundaries when the nexthop set changes, moving the flows whose hash falls in a shifted range. The nexthop object model (ip nexthop add id …, referenced by nhid) lets a next-hop be a standalone/grouped object — the home for the resilient bucket table, which uses a fixed hash % num_buckets modulus (independent of nexthop count) and re-maps only the buckets of a removed nexthop. Use it for stable production gateways.

Check question

You add ip rule add from 192.168.1.0/24 lookup 100 priority 50 and ip rule add fwmark 0x42 lookup 200 priority 100. A packet has src 192.168.1.5 AND fwmark 0x42. Which table is consulted?

Tomorrow

Day 10: IPv6 specifics — NDP, autoconfig, extension headers.

Day 10 — IPv6 specifics: NDP, autoconfig, extension headers

Today’s mission: see what’s genuinely different about IPv6 from the IPv4 you’ve traced for nine days — the 40-byte base header and its nexthdr chain, the 128-bit address with its prefix/host split, the autoconfig flow, the neighbor discovery protocol that replaces ARP, and the extension-header chain that’s caused real CVEs. Total time: ~110 minutes.

Why IPv6 deserves its own day

Most of the kernel network stack is shared between IPv4 and IPv6: the socket layer, the routing infrastructure, netfilter, the device queue, qdiscs. But IPv6 has structurally different features that can’t be glossed over:

1. A different base header and an extensible header chain. Fixed 40 bytes, no options inside it, no checksum — and a nexthdr byte that can point at a chain of variable-length extension headers before you ever reach L4.
2. 128-bit addresses with built-in scope and a prefix/host split. Not “IPv4 with bigger numbers” — the address structure drives autoconfiguration.
3. Autoconfiguration. IPv6 hosts self-configure their addresses without DHCP, by listening to Router Advertisements.
4. NDP replaces ARP. Same idea (resolve next-hop link-layer address) but built on ICMPv6 with richer semantics, and — crucially — multicast instead of broadcast.
5. Extension headers. Variable-length, chained between the base IPv6 header and the L4 protocol. Powerful, fragile, and the source of multiple CVEs over the years.

Everything you learned Days 1–9 was IPv4-only: the sk_buff from Day 1, the RX path and protocol demux from Day 2, the neighbour/ARP subsystem from Day 7. IPv6 reuses an enormous amount of that machinery — so this chapter leans on it hard and only teaches what is genuinely new. We’ll build each new concept intuition-first, then point at the concrete v7.1 struct or function in your ~/code/linux checkout (line numbers from kernel 7.1).

The base header: 40 fixed bytes and a nexthdr byte

Everything today rests on one structure, so we open the struct before anything else. The natural point of contrast is the IPv4 header (struct iphdr, include/uapi/linux/ip.h:87), whose length varied: its 4-bit ihl field counted 32-bit words of header + options, so it could carry inline options. IPv6 throws that out. The base header is always exactly 40 bytes — no length field, no options inside it:

/* include/uapi/linux/ipv6.h:118 — simplified: real struct uses endian
 * bitfield guards + __struct_group(addrs,...) wrapping saddr/daddr */
struct ipv6hdr {
        __u8    priority:4,
                version:4;          /* version == 6 */
        __u8    flow_lbl[3];        /* 8-bit traffic class is split: priority
                                       nibble + top 4 bits of flow_lbl;
                                       low 20 bits of flow_lbl = flow label */
        __be16  payload_len;        /* length of everything AFTER these 40 bytes */
        __u8    nexthdr;            /* what comes next: an L4 proto OR an ext header */
        __u8    hop_limit;          /* IPv6's name for IPv4's TTL */
        struct in6_addr saddr;      /* 16 bytes */
        struct in6_addr daddr;      /* 16 bytes */
};

Read it field by field:

• version = 6, plus a 4-bit priority nibble (the top half of the 8-bit Traffic Class; its lower half is the top 4 bits of flow_lbl[3]) and a 20-bit flow label that occupies the low 20 bits of flow_lbl[3]. The flow label lets a router hash a flow without parsing L4.
• payload_len — the single biggest gotcha if you’re coming from IPv4. It is the length of everything after the 40-byte header, not including the header. IPv4’s tot_len includes the 20-byte header; IPv6’s payload_len does not. (This is why ip6_rcv_core trims the skb to 40 + payload_len, not just payload_len.)
• nexthdr — the protocol selector. For plain TCP traffic it’s just 6. But it can also name an extension header, which carries its own nexthdr, forming a chain. We’ll build that chain below.
• hop_limit — exactly TTL, renamed. Decremented per hop; packet dropped at zero.
• saddr / daddr — two 16-byte addresses (next section).

Three structural differences to burn in

Compared to the IPv4 header (struct iphdr, the one Day 2’s ip_rcv_core parsed), three things are gone or changed, and each one explains a design decision you’ll meet later today:

1. The header is fixed-length — there is no IHL field and no options inside the base header. That’s exactly why options had to move out into a separate extension-header chain. A fixed base header is cheap and predictable to parse; variable junk lives elsewhere.
2. There is no header checksum field at all. Grep proves it — include/uapi/linux/ipv6.h contains zero occurrences of the string check. IPv4 routers recompute the header checksum on every hop (because they decrement TTL); IPv6 routers don’t, because there’s nothing to recompute. Integrity is left to L4 checksums (which cover an IPv6 pseudo-header). The consequence: walking the nexthdr chain is the only per-hop parsing cost — there’s no checksum verify/update on the fast path.
3. hop_limit replaces TTL. Same mechanic, new name.

The nexthdr number space — the key to the whole chain

Here is the single insight that makes the extension-header “chain” make sense: nexthdr uses one shared one-byte protocol-number space, used identically by the base header and by every extension header. A value of 6 always means “TCP comes next,” whether it’s in the base header or seven extension headers deep. The kernel names them in include/net/ipv6.h:32:

#define NEXTHDR_HOP        0   /* Hop-by-Hop options    */
#define NEXTHDR_TCP        6   /* TCP                   */
#define NEXTHDR_ROUTING    43  /* Routing header (SRv6) */
#define NEXTHDR_FRAGMENT   44  /* Fragment header       */
#define NEXTHDR_ESP        50  /* IPsec ESP             */
#define NEXTHDR_AUTH       51  /* IPsec Authentication  */
#define NEXTHDR_NONE       59  /* nothing follows       */
#define NEXTHDR_DEST       60  /* Destination options   */
#define NEXTHDR_MOBILITY   135 /* Mobile IPv6           */

So parsing IPv6 is a loop: read nexthdr; if it names an extension header, parse that header (which is self-describing in length and carries the next nexthdr), and repeat; if it names an L4 protocol (TCP, UDP, ICMPv6…), stop — you’ve reached the payload. NEXTHDR_NONE (59) means “the chain ends here, no payload follows.”

ip6_rcv_core (net/ipv6/ip6_input.c:188) is the IPv6 twin of Day 2’s ip_rcv_core: it validates version == 6, uses payload_len to trim the skb to the right length, and then leaves nexthdr to be resolved by the extension-header and L4 handlers. ipv6_rcv (net/ipv6/ip6_input.c:344) is the ~10-line wrapper that calls ip6_rcv_core and then runs the netfilter PRE_ROUTING hook — structurally identical to IPv4’s ip_rcv.

IPv6 address anatomy: 128 bits, scopes, and the /64 split

The autoconfig and NDP stories below toss around fe80::/64, ::, ff02::1, “host portion,” “prefix” — none of which mean anything until you know what a 128-bit address actually is. Days 1–9 only ever handled the 32-bit IPv4 dotted-quad (__be32). IPv6 is four times wider and structured.

The struct: 16 bytes

/* include/uapi/linux/in6.h:33 */
struct in6_addr {
        union {
                __u8    u6_addr8[16];   /* 16 bytes  = 128 bits */
                __be16  u6_addr16[8];   /* 8 groups of 16 bits  */
                __be32  u6_addr32[4];   /* 4 words of 32 bits   */
        } in6_u;
};
#define s6_addr   in6_u.u6_addr8        /* the byte view */

Contrast with the bare 4-byte __be32 you’ve used everywhere up to Day 9. The union exists because different code wants different granularities — byte-at-a-time text formatting (s6_addr[16]), or word-at-a-time comparison (s6_addr32[4]); you’ll see both today.

Reading the text notation

Just enough to read the rest of the chapter. An address is eight groups of 16 bits, written in hex, colon-separated: 2001:0db8:0000:0000:0000:0000:0000:0001. Two shorthands:

• Drop leading zeros in each group: 2001:db8:0:0:0:0:0:1.
• One :: collapses a single run of all-zero groups — and you may use it only once (otherwise it’d be ambiguous): 2001:db8::1.

Now decode the chapter’s own recurring examples:

• :: — all 128 bits zero. This is the unspecified address, used as the source during Duplicate Address Detection (“I don’t have an address yet”).
• fe80::/64 — the link-local prefix (fe80:: followed by a 64-bit host portion).
• ff02::1, ff02::2 — link-local-scope multicast addresses (all-nodes, all-routers). More on multicast in the NDP section.

The /64 split — prefix half + host half

This is the structure the entire SLAAC/EUI-64 story depends on. A normal unicast IPv6 address is conceptually two 64-bit halves:

|<------ 64 bits: network prefix ------>|<------ 64 bits: interface ID ------>|
   fe80::  (link-local)  OR  from an RA       the "host portion" you keep hearing about

• The upper 64 bits are the network prefix — fe80:: for a link-local address, or a globally-routable prefix learned from a Router Advertisement.
• The lower 64 bits are the interface identifier, a.k.a. the “host portion.” EUI-64, stable_secret, and tempaddr (all in the autoconfig section) are simply three different ways to fill in that lower 64 bits.

Scopes are first-class

In IPv4, “scope” was mostly convention. In IPv6 it’s structural, and the chapter relies on it:

• Link-local (fe80::/10): never routed off the link. Every interface always has one, generated the moment the link comes up — before any router is heard from. This is what makes DAD and NDP possible before you have a global address.
• Global (routable): assigned from prefixes a router advertises.
• Multicast (ff00::/8): one-to-many; the substrate NDP runs on.

Because a link-local address exists immediately, the kernel can do neighbor discovery and address autoconfig over it from the very first millisecond of link-up.

Where the code starts

addrconf_addr_gen (net/ipv6/addrconf.c:3417) is where concept meets code. It seeds the link-local base and then branches on the per-interface generation mode:

/* paraphrased from addrconf_addr_gen() */
ipv6_addr_set(&addr, htonl(0xFE800000), 0, 0, 0);  /* fe80:: ... upper half */
switch (idev->cnf.addr_gen_mode) {                 /* ... then fill lower 64 bits */
case IN6_ADDR_GEN_MODE_EUI64:        /* from the MAC */
case IN6_ADDR_GEN_MODE_STABLE_PRIVACY:
case IN6_ADDR_GEN_MODE_RANDOM:
case IN6_ADDR_GEN_MODE_NONE:
}

The four modes are an enum (include/uapi/linux/if_link.h:459), and their values are exactly the 0/1/2/3 you set via sysctl:

enum in6_addr_gen_mode {
        IN6_ADDR_GEN_MODE_EUI64,            /* 0 */
        IN6_ADDR_GEN_MODE_NONE,             /* 1 */
        IN6_ADDR_GEN_MODE_STABLE_PRIVACY,   /* 2 */
        IN6_ADDR_GEN_MODE_RANDOM,           /* 3 */
};

There are no Dumb Questions

Q: If a link-local address is auto-generated the instant the link comes up — before any router is heard from — why do I still need a Router Advertisement at all?

A: Because link-local is non-routable. fe80::/10 never leaves the link; a router will not forward it. It’s enough to talk to neighbors on the same segment (and to run DAD and NDP), but to reach anything off-link you need a globally-routable prefix, and the only way a host learns one via autoconfig is from an RA. Link-local gets you bootstrapped; the RA gets you on the internet.

Q: Why is the split always /64? Why not some other prefix/host boundary?

A: Because SLAAC and EUI-64 assume a 64-bit interface identifier. The whole “append the host portion to the prefix” trick — EUI-64 stuffing fffe into a 48-bit MAC, stable-privacy hashing into 64 bits — produces a 64-bit lower half by construction. A prefix longer than /64 wouldn’t leave room for it, so SLAAC simply won’t run on non-/64 prefixes. The /64 boundary is baked into the autoconfig machinery, not just a convention.

Autoconfiguration

The first time you bring up an IPv6-enabled interface, the kernel runs through this sequence — most of it before any user-supplied configuration:

Step 1: assign a link-local address

Every IPv6 interface gets at least a link-local address in fe80::/64. As we just saw, that’s fe80:: (upper 64 bits) plus a host portion (lower 64 bits) derived from one of:

• EUI-64: insert fffe between the two halves of the MAC address (and flip the U/L bit). For MAC aa:bb:cc:dd:ee:ff you get fe80::a8bb:ccff:fedd:eeff. Predictable but privacy-leaking — the MAC stays the same across networks.
• stable_privacy (RFC 7217, configured via the stable_secret sysctl): hash a per-interface seed with a cryptographic RNG. Different every fresh boot or interface reset. This is the same mechanism addr_gen_mode=2 selects below.
• tempaddr (RFC 4941): privacy extensions; periodically rotate the host portion.

Sysctls that matter:

• net.ipv6.conf.<dev>.addr_gen_mode: 0 = EUI-64, 1 = none, 2 = stable_privacy, 3 = random (initializes a random secret, then generates via the stable_privacy algorithm).
• net.ipv6.conf.<dev>.use_tempaddr: 0 = no privacy addrs, 1 = use them but prefer stable, 2 = prefer privacy.

Implementation: net/ipv6/addrconf.c:3417 addrconf_addr_gen — branches on addr_gen_mode and calls the right generator (the function we read above).

Step 2: Duplicate Address Detection (DAD)

Before committing an address, the host sends an ICMPv6 Neighbor Solicitation for it — with src :: (the unspecified address you now recognize), telling everyone “I’m tentatively claiming this.” If anyone replies, the address is in use; the kernel marks it dadfailed and refuses to use it. If nobody replies after dad_transmits retries (default 1) at intervals of RetransTimer (default 1 s), the address moves from tentative to preferred.

Where does that NS get sent, though? Not to a broadcast — to the target’s solicited-node multicast address (a per-address group derived from the address’s low 24 bits, built in the NDP section below). That’s the mechanism that makes DAD (and ARP-style lookup) cheap. For now:

ip -6 addr show dev eth0     # look for 'tentative' or 'dadfailed' flags

DAD is mandatory by spec; you can disable it with net.ipv6.conf.<dev>.accept_dad=0 for niche cases (link with no neighbors yet) but never on a shared LAN.

Step 3: Listen for Router Advertisements

ICMPv6 type 134 — emitted periodically by routers (the kernel itself doesn’t originate RAs; that’s a userspace daemon like radvd, whose MaxRtrAdvInterval defaults to 600 s — RFC 4861 permits 4 s–1800 s; unsolicited RAs are actually emitted at random intervals between MinRtrAdvInterval (~200 s by default) and MaxRtrAdvInterval (600 s)), or on demand in response to a Router Solicitation (type 133) from hosts that just came up. RAs carry, among other things:

• A list of prefixes with on-link/auto-config flags.
• Default route (the router itself).
• MTU.
• M flag (“Managed” — use DHCPv6 for addresses).
• A flag (“Autonomous” — do SLAAC from this prefix).
• O flag (“Other” — use DHCPv6 for non-address config like DNS).
• RDNSS option — recursive DNS server (RFC 8106).

If A=1 in a prefix, the host runs SLAAC: take the announced 64-bit prefix, append its 64-bit host portion (Step 1’s interface identifier), run DAD on the result, and install the address. If M=1, the kernel kicks off a DHCPv6 client (handled by userspace — dhclient -6, wpa_supplicant, NetworkManager, etc.). Both can be true simultaneously.

Step 4: NDP everywhere else

After address setup, NDP keeps neighbor state fresh. Same plumbing as Day 7’s neighbour subsystem — there’s a struct neigh_table nd_tbl at net/ipv6/ndisc.c:109 that’s structurally identical to arp_tbl. Same NUD states (REACHABLE, STALE, etc.), same gc thresholds, same ip neigh show. We do not re-teach the neighbour subsystem here — flip back to Day 7 for NUD and nd_tbl internals.

NDP — Neighbor Discovery Protocol in detail

NDP runs over ICMPv6. Before the message table, two pieces of new background: how an ICMPv6 packet even reaches the NDP handlers, and why NDP’s multicast is cheaper than ARP’s broadcast.

How an inbound packet reaches ndisc_rcv (ICMPv6 as NDP’s carrier)

NDP messages aren’t a new EtherType the way ARP was (Day 7). They are ICMPv6 messages — IPPROTO_ICMPV6 = 58, which sits at the end of the nexthdr chain like any other L4 protocol. So the extension-header walk from earlier must complete before NDP is even seen.

The dispatch mechanism is the IPv6 twin of Day 2’s registered-handler demux, so we don’t re-teach it: inet6_protos[] is the IPv6 twin of Day 2’s inet_protos[] — an RCU-protected, MAX_INET_PROTOS-sized array of struct inet6_protocol, indexed by protocol number (net/ipv6/protocol.c:25). ICMPv6 registers itself at slot 58 (net/ipv6/icmp.c:96):

static const struct inet6_protocol icmpv6_protocol = {
        .handler = icmpv6_rcv,        /* net/ipv6/icmp.c:1101 */
        ...
};

What is new is the second-level demux inside ICMPv6. Unlike IPv4’s icmp_rcv (which carries only echo/error), ICMPv6 does double duty — it carries echo/error and the entire neighbor-discovery control plane and (separately) MLD. So icmpv6_rcv switches on the ICMPv6 type byte, and for the five NDP types (133–137) hands off to ndisc_rcv (net/ipv6/icmp.c:1188):

/* icmpv6_rcv(), net/ipv6/icmp.c */
case NDISC_ROUTER_SOLICITATION:
case NDISC_ROUTER_ADVERTISEMENT:
case NDISC_NEIGHBOUR_SOLICITATION:
case NDISC_NEIGHBOUR_ADVERTISEMENT:
case NDISC_REDIRECT:
        reason = ndisc_rcv(skb);

ndisc_rcv (net/ipv6/ndisc.c:1805) then switches again on the type to reach the concrete handler — ndisc_recv_ns for a Neighbor Solicitation (net/ipv6/ndisc.c:1831), ndisc_recv_na for an advertisement (net/ipv6/ndisc.c:1836), and so on. That two-step switch is the missing link between “NDP runs over ICMPv6” and the handler names below. In IPv6 there is no ARP EtherType and no separate ARP handler — it all funnels through this one inet6_protos slot.

The five message types

(Type constants: include/net/ndisc.h:9 — 133 RS, 134 RA, 135 NS, 136 NA, 137 Redirect.)

Why NS is cheaper than an ARP broadcast: solicited-node multicast

Day 7’s ARP resolves a neighbor by broadcasting at L2 — every NIC on the segment receives the frame, processes it, and (almost all) discard it. NDP does better with multicast, and this is the mechanism the comparison table’s “multicast-based, saves bandwidth” line actually refers to.

Minimal multicast model:

• ff00::/8 is the entire multicast address space. The second nibble encodes scope — ff02:: is link-local scope (the only scope NDP needs).
• Two you meet immediately: ff02::1 (all-nodes) and ff02::2 (all-routers — where Router Solicitations go).

The clever bit is the solicited-node multicast address. For any target unicast address, you take its low 24 bits and append them to the well-known prefix ff02::1:ff00:0/104, producing ff02::1:ffXX:XXXX. A host joins the solicited-node group for each of its own addresses. So when you want to resolve (or DAD-probe) a specific address, you send the NS to that address’s solicited-node group — and (almost always) only the target is subscribed, so only the target’s NIC is interrupted. Every other host’s hardware multicast filter drops the frame without waking the CPU. Same goal as an ARP broadcast, far fewer interrupts, and L2 switches can snoop the groups.

The kernel computes it with one helper (include/net/addrconf.h:484):

static inline void addrconf_addr_solict_mult(const struct in6_addr *addr,
                                             struct in6_addr *solicited)
{
        ipv6_addr_set(solicited,
                      htonl(0xFF020000), 0,         /* ff02::         */
                      htonl(0x1),                   /* ...1:          */
                      htonl(0xFF000000) | addr->s6_addr32[3]); /* ff + low 24 bits */
}

This ties Steps 2 and 3 together mechanically. DAD (Step 2) sends an NS for the tentative address, from src ::, to that address’s own solicited-node multicast group — computed by exactly this helper. The NS senders call it: net/ipv6/ndisc.c:382 and net/ipv6/ndisc.c:395 both use addrconf_addr_solict_mult to pick the destination. And ndisc_send_na (net/ipv6/ndisc.c:524) carries the solicited/override/router flags the table references. The NA installs a neigh entry — same nd_tbl/NUD machinery as Day 7.

Implementation entry points:

• ndisc_recv_ns — incoming neighbor solicitation.
• ndisc_recv_na — incoming advertisement.
• ndisc_send_na, ndisc_send_ns — outgoing.

net/ipv6/ndisc.c is ~2000 lines but most of it is option parsing.

Compared to ARP, NDP is also richer in other ways:

• Authenticated with SEND (RFC 3971), though deployment is rare.
• Detects unreachability actively (bidirectional reachability checks, not just on-demand).
• Carries options for MTU, source link-layer address, prefix info, route info.

Extension headers — power and pitfall

We met the nexthdr chain at the top of the day. Now we walk it. The base header is 40 bytes; its nexthdr may point at an extension header, which carries its own nexthdr, forming a chain that ends at an L4 protocol (or NEXTHDR_NONE):

The headers in the chain

• Hop-by-Hop Options (0) — every router on the path must process. Used by Jumbograms (>64KB), MLD, RPL. Must be first if present.
• Routing (43) — Source Routing Header (SRH) for SRv6, RPL.
• Fragment (44) — IPv6 fragmentation (only by source; routers don’t fragment).
• Destination Options (60) — end-host options.
• Authentication Header (51) / ESP (50) — IPsec.
• Mobility (next-header 135 — distinct from the ICMPv6 message type 135, Neighbor Solicitation, above; different namespaces) — Mobile IPv6.

Each extension is parsed by a per-protocol handler registered in inet6_protos[] (the same table ICMPv6 lives in). The Routing handler is ipv6_rthdr_rcv (net/ipv6/exthdrs.c:658), Destination is ipv6_destopt_rcv (net/ipv6/exthdrs.c:299), etc.

How an option is encoded: the TLV format

The Hop-by-Hop and Destination Options headers don’t carry fixed fields — they carry a list of TLV (Type-Length-Value) records, and you need this layout to understand both the Check question and the CVE class below.

Each option is three parts:

+--------+--------+--------------------------+
| Type   | Length | Value (Length bytes)     |
| 1 byte | 1 byte | <-- exactly Length -->   |
+--------+--------+--------------------------+

• Type (1 byte) — which option this is.
• Length (1 byte) — the length of the Value that follows.
• Value (Length bytes) — the option data.

A Hop-by-Hop or Destination Options header is just a chain of these TLVs, padded to 8-byte alignment with two special options: Pad1 (a single zero byte) and PadN (a TLV whose value is N zero bytes). This is the classic “length-prefixed-buffer parser” pattern — and the Length byte is attacker-controlled, which is the seed of the whole CVE story.

The two high bits of the Type byte: unknown-option actions

This is the exact mechanism the Check question hinges on. The top 2 bits of the Type byte tell a node what it must do when it doesn’t recognize the option:

(A third bit, 0x20, separately flags whether the option may change en route — relevant to the Authentication Header.)

The kernel implements precisely this in ip6_tlvopt_unknown (net/ipv6/exthdrs.c:65), which ip6_parse_tlv (net/ipv6/exthdrs.c:114) calls for each unrecognized option. It dispatches on those two bits (switch at net/ipv6/exthdrs.c:79):

switch ((skb_network_header(skb)[optoff] & 0xC0) >> 6) {
case 0: /* ignore */                 return true;   /* skip, keep parsing */
case 1: /* drop packet */            break;         /* silent drop        */
case 3: /* ICMP unless multicast */  ... fallthrough;
case 2: /* send ICMP, always drop */ icmpv6_param_prob_reason(...); return false;
}

ip6_tlvopt_unknown is invoked from ip6_parse_tlv’s option loop (net/ipv6/exthdrs.c:195 and :211), which handles Destination Options via ipv6_destopt_rcv (net/ipv6/exthdrs.c:329) and Hop-by-Hop at net/ipv6/exthdrs.c:1084; a hopbyhop bool parameter distinguishes the two callers. The Jumbo Payload option’s parser, ipv6_hop_jumbo (net/ipv6/exthdrs.c:996), is dispatched from the Hop-by-Hop option table — a tiny but instructive length-field parser, and historically a buggy one.

Why this is dangerous

Extension-header parsers are length-prefixed-buffer parsers in C, deep in the network stack, fed an attacker-controlled Length byte per TLV. A parser that trusts that Length without bounding it against the header’s own declared length is the recurring bug class. They’ve been a recurring source of CVEs:

• 2026 (kernel 7.1): SRv6 RPL OOB write — ipv6_rpl_srh_rcv (net/ipv6/exthdrs.c:491) could push a recompressed SRH that exceeded headroom, causing skb_mac_header_rebuild (called at net/ipv6/exthdrs.c:615) to underflow mac_header to ~65530 and memmove to write 14 bytes ~64KiB past skb->head. Fix in commit 9e6bf146b559.
• 2022: SRv6 Segment Routing Header out-of-bounds read when setting HMAC data (CVE-2022-48687, “ipv6: sr: fix out-of-bounds read when setting HMAC data”).
• Various jumbogram bugs in ipv6_hop_jumbo (net/ipv6/exthdrs.c:996).

The pattern: an attacker controls extension-header lengths, and a parser miscomputes how much memory to allocate or how many bytes are valid. (That 2026 SRv6 bug is a skb-geometry bug — mac_header underflowing, memmove past skb->head. That’s the head/data/tail/end + headroom geometry from Day 1’s sk_buff chapter; refer back there rather than re-deriving it.) If you write an extension header parser, treat it like a fuzzing target from day one.

Skipping the chain

To get to L4, code calls ipv6_skip_exthdr(skb, start, &nexthdr, &frag_off) — walks the chain until reaching a non-extension nexthdr and returns the offset. The signature (net/ipv6/exthdrs_core.c:72):

int ipv6_skip_exthdr(const struct sk_buff *skb, int start,
                     u8 *nexthdrp, __be16 *frag_offp);

Quirks:

• Some L4 lookups need to skip all extension headers; others (like conntrack) want to inspect specific ones.
• A malformed chain (loop, oversized) causes ipv6_skip_exthdr to return -1, and the kernel drops the packet.
• It takes frag_off because a Fragment header tells you the packet is part of a larger original — the caller may need to defer processing.

Today’s experiment

# See your IPv6 addresses and their states
ip -6 addr show
# Look for: 'global', 'mngtmpaddr', 'dynamic', 'tentative', 'deprecated'

# Watch DAD on bring-up
sudo bpftrace -e 'fentry:ndisc_send_ns {
  printf("NS sent target=%s\n", ntop(args->solicit->in6_u.u6_addr8));
}' &
sudo ip link set eth0 down && sleep 1 && sudo ip link set eth0 up

# Watch NDP traffic globally
sudo tcpdump -i eth0 -nn icmp6 and not host ::

# Inspect autoconfiguration sysctls for one interface
sysctl net.ipv6.conf.eth0 | grep -E "addr_gen_mode|accept_ra|use_tempaddr|dad_transmits"

# Trace extension-header parsing
sudo bpftrace -e 'fentry:ipv6_skip_exthdr { printf("skip nexthdr=%d start=%d\n", *args->nexthdrp, args->start); }'

What to read in the kernel

• net/ipv6/ip6_input.c:188 — ip6_rcv_core. The IPv6 receive core logic (~145 lines, ~80 of real logic). ipv6_rcv (line 344) is a ~10-line wrapper that calls ip6_rcv_core, then runs the netfilter PRE_ROUTING hook. Notice how it parses the base header, validates version=6, trims to payload_len, and dispatches via the registered inet6_protos[] table — same pattern as IPv4’s ip_rcv but with the extension-header parser as the first handler.

• net/ipv6/addrconf.c — autoconf state machine. ~7600 lines but you only need a few entry points:

  • addrconf_dad_start (search for the function; no fixed line) — kicks off DAD.
  • addrconf_rs_timer — periodic RS solicitation when no router heard from.
  • addrconf_prefix_rcv — handle a prefix from RA: install address, run DAD on it.
  • addrconf_addr_gen (line 3417) — host-portion generation.

  Read the top of the file’s comments first; the model is a state machine per inet6_dev.

• net/ipv6/ndisc.c:109 — nd_tbl, the neighbour-table instance. The struct is identical to IPv4’s arp_tbl — confirms how generic the neighbour subsystem is. Around it, ndisc_recv_ns, ndisc_recv_na, ndisc_send_na, ndisc_send_ns are the protocol handlers; ndisc_rcv (line 1805) is the type-switch they hang off.

• net/ipv6/exthdrs.c — extension-header parsers.

  • ipv6_rthdr_rcv (line 658): Routing header. Read this to understand SRv6 — the most actively-developed extension. Also where most CVEs have been.
  • ipv6_destopt_rcv (line 299): Destination Options. Simpler; good warm-up. Calls ip6_parse_tlv.
  • ip6_parse_tlv (line 114): the generic TLV walker, with the unknown-option action dispatch at line 79.
  • ipv6_hop_jumbo (line 996): the parser for the Jumbo Payload option in HOPOPT. Tiny but instructive.

• include/uapi/linux/in6.h and include/net/ipv6.h — the canonical structs (struct ipv6hdr, struct in6_addr, the NEXTHDR_* constants).

• Documentation/networking/ipv6.rst — overview. Light but has pointers to RFC numbers.

• RFCs to skim: 8200 (IPv6 spec), 4861 (NDP), 4862 (SLAAC), 4941 (privacy extensions), 7136 (modified EUI-64), 8754 (SRv6).

Bullet Points

• The base header is fixed at 40 bytes: version=6, a priority nibble plus flow_lbl[3] (together holding the 8-bit traffic class — priority + top 4 bits of flow_lbl — and the 20-bit flow label in the low 20 bits of flow_lbl), payload_len (length after the header — unlike IPv4 tot_len), nexthdr, hop_limit (TTL renamed), two 16-byte addresses. No IHL, no header checksum — options moved to the extension-header chain.
• nexthdr is one shared protocol-number space (6=TCP, 0=HopByHop, 43=Routing, 44=Fragment, 59=None, 60=DestOpts): parse it as a loop until you hit an L4 protocol.
• An IPv6 address is 128 bits (struct in6_addr, 16 bytes), split into a 64-bit prefix + 64-bit host portion. :: = unspecified, fe80::/64 = link-local (always present), ff00::/8 = multicast.
• IPv6 hosts auto-configure: link-local (fe80::/64) immediately, global addresses from RAs.
• Address generation modes (addr_gen_mode): EUI-64 (0, kernel default), none (1), stable_privacy (2), random (3).
• DAD is mandatory: send NS for your tentative address to its solicited-node multicast group (ff02::1:ffXX:XXXX, low 24 bits of the target), from src ::, wait, then commit.
• Solicited-node multicast is why NDP beats ARP: only the target host is subscribed, so only its NIC wakes — vs an ARP L2 broadcast every host processes.
• ICMPv6 (proto 58) carries NDP. inet6_protos[] (Day 2’s inet_protos[] twin) → icmpv6_rcv → (type switch) → ndisc_rcv → ndisc_recv_ns/_na/…
• NDP = ICMPv6 messages 133–137: RS, RA, NS, NA, Redirect. Replaces ARP. The neighbour subsystem (nd_tbl) is structurally identical to IPv4’s arp_tbl (Day 7).
• Extension headers chain between base IPv6 header and L4. Walked by ipv6_skip_exthdr. Important ones: HopByHop (0), Routing (43, SRv6), Fragment (44), DestOpts (60), AH/ESP (51/50).
• HopByHop/DestOpts options are TLV-encoded (Type, Length, Value). The top 2 bits of Type select the unknown-option action: skip / drop / drop+ICMP / drop+ICMP-unless-multicast (ip6_parse_tlv).
• Extension-header parsers trust an attacker-controlled Length byte — a recurring CVE source. Treat as a fuzz-prone area.

Check question

A node receives an IPv6 packet whose first extension header is nexthdr=0 (Hop-by-Hop Options). What action is the kernel obligated to take, even if the packet is for a peer (not for us)?

Tomorrow

Day 11: the bridge subsystem. Linux as a software L2 switch.

Day 11 — The bridge subsystem

Today’s mission: create a software Linux bridge, attach two interfaces to it, and watch the FDB learn MACs as traffic flows. Along the way you’ll learn the four mechanisms the bridge is built on — the rx_handler hook that hijacks a NIC’s frames, the RCU-protected hash table that keeps the FDB lock-free for readers, why L2 loops are catastrophic (and what spanning tree does about it), and how br_netfilter drags bridged frames through iptables — plus how VLAN-aware bridges deliver real switch behavior. Total time: ~110 minutes.

What a Linux bridge is, and why you have one

A Linux bridge is a software L2 switch. It looks like a netdev (you can give it an IP, ping it, configure routes through it) but its job is to forward Ethernet frames between member ports based on destination MAC. Like any switch, it learns which MAC lives behind which port and stores that mapping in a table called the FDB (forwarding database). The implementation lives in net/bridge/ (~29k lines of top-level .c).

You’re using bridges constantly even if you’ve never explicitly created one:

• Container runtimes. Docker creates docker0 by default; Podman creates cni-podman0; CNI plugins create cni0/cbr0. Each one is a Linux bridge that wires container veths to the host network.
• VMs. libvirt’s “default” network is a bridge (virbr0); bridge=br0 mode in qemu attaches the VM’s TAP (a software NIC that hands frames to a userspace process like qemu) directly to a host bridge.
• Multi-NIC servers acting as switches. A box with multiple Ethernet NICs can bridge them with one ip link add br0 type bridge plus ip link set <if> master br0 per port.

The bridge is the fastest L2-forwarding plane in the kernel — much faster than the IP routing path because, switching purely by destination MAC, it skips everything L3 does: no L3/FIB lookup, no TTL decrement and header-checksum rewrite, and (by default — see br_netfilter) no netfilter traversal or decapsulation.

This chapter leans on four pieces of machinery that no earlier day taught you: the rx_handler hook that diverts a slave NIC’s frames into bridge code; the RCU-protected rhashtable that backs the forwarding database; the broadcast-storm / spanning-tree problem that explains why switches need STP at all; and the netfilter chains that br_netfilter exposes. We’ll teach each one — intuition first, then the concrete v7.1 struct or function — right where the bridge depends on it.

Two things from earlier days are load-bearing today and will not be re-taught:

• The RX path. Recall from Day 2 — the NIC’s NAPI poll builds an skb, then __netif_receive_skb_core walks the ptype_base hash to find the L3 handler (ip_rcv for IPv4). Today’s whole story is about a hook that fires before that demux.
• The CPU cache line. Recall from Day 1 — the CPU reads memory in 64-byte lines, so write-heavy fields are kept off read-mostly cache lines to avoid false sharing. We’ll meet that idea once more, in the FDB entry.

Anatomy

A bridge is two structs working together:

• struct net_bridge (net/bridge/br_private.h:495) — the bridge itself. Holds the FDB (forwarding database) — the MAC-address-to-port table that is the heart of any L2 switch — plus the port list, VLAN config, IGMP-snooping state, MST/STP state. One of these per brN netdev.
• struct net_bridge_port (net/bridge/br_private.h:387) — one per slave interface. Holds the dev pointer (the underlying netdev), the port number, STP state (DISABLED, LEARNING, FORWARDING, …), and per-port flags (BPDU guard, root guard, hairpin, etc.).

When you do ip link set eth0 master br0, the kernel attaches a net_bridge_port to eth0, registers br_handle_frame as eth0’s rx_handler, and the slave’s frames now flow into bridge logic instead of the normal stack. That sentence hides the single most important mechanism in the chapter — so let’s open it up.

Background 1: the rx_handler — how a slave NIC’s frames get hijacked

Recall the RX path from Day 2: a frame is DMAed in, NAPI builds an skb, and __netif_receive_skb_core indexes ptype_base by skb->protocol to find the one L3 handler (ip_rcv, the IPv6 handler, ARP, …). That’s the “normal stack.” A bridge has to get its hands on the frame before that L3 demux runs — otherwise a frame destined for some other host would be delivered to this host’s IP stack instead of being switched out another port.

The kernel’s hook for exactly this is the rx_handler.

One function pointer per netdev

Every struct net_device carries a single, optional rx_handler slot (include/linux/netdevice.h:2189):

rx_handler_func_t __rcu  *rx_handler;
void __rcu               *rx_handler_data;

When that pointer is set, __netif_receive_skb_core calls it before the ptype_base packet-type walk that Day 2 traced. So the rx_handler sits at the exact spot in the RX path where the L3 demux would otherwise run, and gets first refusal on the frame. There is only one rx_handler per device — registering a second returns -EBUSY (include/linux/netdevice.h:461-477). The bridge, bonding, macvlan, and OVS all compete for this same plug-in slot; a “join a master” device is one that has installed an rx_handler.

Its return value decides the frame’s fate

The handler tells the receive path what to do with the skb via an enum rx_handler_result (documented at include/linux/netdevice.h:461-477). The two that matter:

• RX_HANDLER_CONSUMED — “I took the skb; do not process it further.” The frame never reaches ptype_base, never reaches ip_rcv on the slave NIC. The bridge returns this for every data frame — including one addressed to the bridge’s own IP. That last case is the subtle one: br_handle_frame still returns CONSUMED, but br_handle_frame_finish notices the destination is a BR_FDB_LOCAL entry and calls br_pass_frame_up(), which re-injects the skb onto the bridge netdev (skb->dev = br0) via netif_receive_skb (net/bridge/br_input.c:218-220). So “pinging br0” works by re-injection on br0, not by passing the frame through on the slave port.
• RX_HANDLER_PASS — “Do nothing; proceed as if no handler ran.” The bridge uses this only for the genuine pass-through cases: PACKET_LOOPBACK, and locally-consumed link-local control frames (STP BPDUs, LLDP) handled via __br_handle_local_finish. It is not how a frame reaches the bridge’s own IP.

(There are two more — RX_HANDLER_ANOTHER, re-loop because skb->dev changed, and RX_HANDLER_EXACT, force exact delivery — that the bridge doesn’t lean on today.)

Where the bridge installs it

Back in br_add_if (the code path behind ip link set eth0 master br0), the bridge registers its handler (net/bridge/br_if.c:613):

err = netdev_rx_handler_register(dev, br_get_rx_handler(dev), p);

Three things happen in that one call:

1. br_get_rx_handler(dev) (net/bridge/br_input.c:463) returns br_handle_frame for an ordinary port (a special dummy is used for DSA switch ports).
2. netdev_rx_handler_register plants br_handle_frame into dev->rx_handler.
3. The third argument — p, the net_bridge_port — is stashed as rx_handler_data, so when a frame later arrives, br_handle_frame can recover which port and which bridge it belongs to.

br_handle_frame itself is at net/bridge/br_input.c:339:

static rx_handler_result_t br_handle_frame(struct sk_buff **pskb)

So the picture is: a frame on a slave NIC reaches __netif_receive_skb_core, which checks dev->rx_handler. It’s set, so br_handle_frame runs. For any data frame — switched or destined to the bridge’s own IP — it returns RX_HANDLER_CONSUMED, so the frame is gone from the slave NIC’s perspective. A frame addressed to the bridge’s own IP still reaches ip_rcv, but by re-injection on br0 (via br_pass_frame_up → netif_receive_skb), not by falling through the slave’s ptype_base demux. RX_HANDLER_PASS is reserved for loopback and locally-handled link-local control frames.

The forwarding decision

For every frame that lands on a bridge port, the kernel runs the same compact decision tree:

1. Drop or forward control frames. STP BPDUs (Bridge Protocol Data Units — Spanning Tree control frames, covered in Background 3; destined to 01:80:c2:00:00:00) get special handling; non-bridge multicast may be flooded or snooped.
2. Learn the source. Insert/refresh (src MAC, vid, port) in the FDB — the forwarding database, whose internals are Background 2 just below — via br_fdb_update (net/bridge/br_fdb.c:972). Aging timers tick.
3. Look up the destination. br_fdb_find_rcu (net/bridge/br_fdb.c:263) walks the hash table.
4. Decide:
   • Hit, same port → drop (don’t loop the frame back to where it came from).
   • Hit, different port → br_forward (net/bridge/br_forward.c:144) — send out that one port.
   • Miss → br_flood (net/bridge/br_forward.c:201) — send to every port except the input port.
   • Broadcast/multicast — flood, possibly filtered by IGMP/MLD snooping.

The “miss → flood” rule is what makes bridges self-learning. Initial frames flood; replies teach the FDB the source’s port; subsequent frames are unicast.

The actual entry point is br_handle_frame (net/bridge/br_input.c:339), which is registered as the rx_handler when an interface joins a bridge. Following the path br_handle_frame → br_handle_frame_finish (net/bridge/br_input.c:76) gives you the full L2 forwarding code in ~250 lines.

Background 2: the FDB is an RCU-protected rhashtable

Steps 2 and 3 above both hit the FDB — one write, one read — on every single frame. At line rate that’s millions of lookups per second, so the data structure underneath has to let readers run without ever blocking each other or the writers. Two kernel mechanisms make that possible, and neither was introduced earlier, so let’s build them up.

rhashtable: the kernel’s resizable hash table

rhashtable is the kernel’s generic, automatically-resizing hash table. The FDB configures one with a small parameter block (net/bridge/br_fdb.c:27):

static const struct rhashtable_params br_fdb_rht_params = {
    .head_offset = offsetof(struct net_bridge_fdb_entry, rhnode),
    .key_offset  = offsetof(struct net_bridge_fdb_entry, key),
    .key_len     = sizeof(struct net_bridge_fdb_key),
    .automatic_shrinking = true,
};

The key is not just the MAC — it’s the {MAC, vlan_id} pair (net/bridge/br_private.h:286):

struct net_bridge_fdb_key {
    mac_addr addr;
    u16      vlan_id;
};

That single fact explains a behavior you’ll see in the VLAN lab below: the same MAC address in two different VLANs is two separate FDB entries, because the VID is part of the key. A lookup is rhashtable_lookup(tbl, &key, br_fdb_rht_params) (net/bridge/br_fdb.c:216).

RCU in one paragraph (it’s new here)

RCU (Read-Copy-Update) is a synchronization scheme tuned for read-mostly data. Readers take no lock at all: they call rcu_read_lock(), walk the structure, and call rcu_read_unlock(). A writer that wants to delete an entry first unlinks it so new readers can’t find it, then defers actually freeing the memory until every reader that might still hold a pointer has finished (the “grace period”). The payoff is enormous for the FDB: the per-frame forwarding hot path looks up destinations with zero lock contention, even while another CPU is busy learning new MACs.

This is what the _rcu suffix on br_fdb_find_rcu announces by convention: “you must call me inside an RCU read-side section, and the pointer I return is only valid until rcu_read_unlock().” So:

• Readers — the forwarding path — call br_fdb_find_rcu (net/bridge/br_fdb.c:263) → rhashtable_lookup, no lock per frame.
• Writers — the learning path — call br_fdb_update (net/bridge/br_fdb.c:972). Crucially, the common per-frame case (an already-known source) refreshes the entry’s updated/dst fields lock-free via WRITE_ONCE() — no lock at all. br->hash_lock (net/bridge/br_private.h:497) is taken only on the slow path that inserts a brand-new entry (fdb_create) or deletes one.

One callback to Day 1’s cache line

That updated timestamp is touched on every learning frame — it’s write-heavy. The FDB lookup fields (key, dst) are read-mostly. If they shared a 64-byte cache line, every aging-timestamp write would invalidate the line that lookups are reading — false sharing (recall Day 1). So the kernel deliberately puts updated on its own line (net/bridge/br_private.h):

/* write-heavy members should not affect lookups */
unsigned long  updated ____cacheline_aligned_in_smp;

That ____cacheline_aligned_in_smp is the same cache-line lesson from Day 1, applied to keep write-heavy aging off the read-mostly lookup path.

Put together: readers are lockless. A writer makes a new entry visible atomically through the rhashtable insert (hlist_add_head_rcu), and a deleted entry’s memory is reclaimed only after an RCU grace period — so a concurrent reader mid-traversal always sees a consistent, whole entry, never a torn or freed one. The per-frame refresh of an existing source’s updated/dst fields is a lock-free in-place WRITE_ONCE(), taking no lock and forcing no grace period. That’s why FDB lookups scale.

VLAN-aware bridges

By default a bridge ignores VLAN tags — it’s just an L2 switch. Toggle VLAN-awareness:

sudo ip link set br0 type bridge vlan_filtering 1

Now the bridge implements proper 802.1Q switching:

• Each port has a list of allowed VLANs (egress: tagged or untagged) and a PVID (default VLAN for untagged ingress).
• Frames are forwarded only between ports that share a VLAN.
• The FDB is keyed on (MAC, vid) not just MAC — same MAC in different VLANs is two separate entries (that’s the net_bridge_fdb_key from Background 2).