Delivering multi-gigabit throughput in userspace VPN gateways demands rethinking how network packets transition between physical network interface cards (NICs) and application runtimes. Traditionally, software-defined tunnels running in Go, Rust, or C interface with the operating system via virtual TUN/TAP devices (/dev/net/tun). While TUN devices offer broad compatibility, they impose severe architectural penalties: every packet incurs kernel-to-userspace context switches, intermediate struct sk_buff metadata allocation, and memory copying via copy_to_user().

At 10 Gbps and 40 Gbps line rates, these overheads saturate CPU memory buses and thrash processor caches, capping single-core throughput at approximately 1.5 million packets per second (Mpps). While the Data Plane Development Kit (DPDK) circumvents these bottlenecks by completely monopolizing the network hardware, DPDK removes the NIC from the operating system entirely, rendering standard Linux tooling (iproute2, nftables, tcpdump) and non-tunnel services inoperable.

Linux Address Family XDP (AF_XDP, also known as XSK) bridges this architectural divide. By pairing eBPF packet redirection at the network driver layer with user-allocated memory pools (UMEM), AF_XDP delivers hardware zero-copy packet streaming directly into userspace applications while preserving the standard Linux operating system environment.

The Architectural Tax: TUN/TAP vs AF_XDP Datapaths

To understand the performance divergence, consider the lifecycle of an encrypted VPN packet across both datapath paradigms:

[Traditional TUN/TAP Datapath]
NIC RX Ring ──> alloc_skb() ──> Kernel IP Stack / Netfilter ──> TUN Driver
                                                                    │
                 Context Switch + Memory Copy (copy_to_user) <──────┘
                                    │
                                    ▼
                         Userspace VPN Application (Decrypt)

─────────────────────────────────────────────────────────────────────────

[AF_XDP Zero-Copy Datapath]
NIC RX Ring ──> Direct Memory Access (DMA) ──> Pre-allocated UMEM Chunk
                     │
         XDP Filter: bpf_redirect_map()
                     │
                     ▼ (Zero Copy, Lockless Descriptor Handover)
           Userspace VPN Engine (Direct In-Memory Decryption)

In the AF_XDP pipeline, the operating system kernel does not allocate struct sk_buff containers. Instead, the NIC hardware DMA controller streams raw Ethernet frames directly into pre-registered userspace memory chunks (UMEM). An eBPF program running at the driver hook point (BPF_PROG_TYPE_XDP) inspects the packet header. If the packet targets the VPN service (matching the designated UDP port or cryptographic handshake signature), the program executes bpf_redirect_map(&xsk_map, queue_id, 0). Non-VPN traffic returns XDP_PASS, flowing seamlessly into the regular Linux IP stack.

UMEM Architecture and the Four Lockless Circular Rings

The core engine of AF_XDP is the UMEM—a contiguous region of virtual memory registered with the kernel via the setsockopt(..., SOL_XDP, XDP_UMEM_REG, ...) system call. Memory within UMEM is subdivided into fixed-size frames (typically 2048 or 4096 bytes).

Communication between userspace and the kernel driver occurs through four lockless, single-producer single-consumer (SPSC) circular ring queues:

  1. Fill Ring (Userspace Produces -> Kernel Consumes): Userspace places empty UMEM frame physical addresses into this ring, providing the NIC driver with buffers ready to receive incoming packets.
  2. Rx Ring (Kernel Produces -> Userspace Consumes): When packets arrive, the NIC driver deposits frame descriptors (address offset, packet length) into this ring for userspace processing.
  3. Tx Ring (Userspace Produces -> Kernel Consumes): When transmitting outgoing encrypted packets, userspace queues populated frame descriptors into the Tx ring for wire transmission.
  4. Completion Ring (Kernel Produces -> Userspace Consumes): Once the physical NIC transmits a frame over the wire, the driver writes the frame address into the Completion ring, returning the buffer to userspace for immediate recycling.

Because these rings operate via atomic pointer increments without mutexes or spinlocks, CPU cores process packets in batches of 32 or 64 without thread contention.

Kernel Implementation: Driver-Level XDP Redirection

The in-kernel component requires a minimal eBPF classifier attached to the physical interface:

#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <linux/udp.h>
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_XSKMAP);
    __uint(max_entries, 64);
    __type(key, __u32);
    __type(value, __u32);
} qidconf_map SEC(".maps");

SEC("xdp")
int xdp_vpn_redirect(struct xdp_md *ctx)
{
    void *data = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;
    
    /* Boundary check: minimum Ethernet header */
    if (data + sizeof(struct ethhdr) > data_end)
        return XDP_PASS;
        
    struct ethhdr *eth = data;
    if (eth->h_proto != __constant_htons(ETH_P_IP))
        return XDP_PASS;
        
    struct iphdr *iph = data + sizeof(struct ethhdr);
    if ((void *)(iph + 1) > data_end || iph->protocol != IPPROTO_UDP)
        return XDP_PASS;
        
    struct udphdr *udph = (void *)iph + (iph->ihl * 4);
    if ((void *)(udph + 1) > data_end)
        return XDP_PASS;
        
    /* Direct VPN UDP listening port to AF_XDP socket map */
    if (udph->dest == __constant_htons(51820)) {
        return bpf_redirect_map(&qidconf_map, ctx->rx_queue_index, XDP_PASS);
    }
    
    return XDP_PASS;
}

char _license[] SEC("license") = "GPL";

This program executes in nanoseconds directly inside the driver interrupt context. Validated VPN datagrams stream directly into userspace rings, while all SSH, management, and BGP routing packets continue undisturbed through the Linux network stack.

Userspace Engine: UMEM Registration and Zero-Copy Polling

In the userspace runtime, setting up an AF_XDP worker involves allocating memory through posix_memalign(), configuring UMEM parameters, and binding the socket with zero-copy flags:

#include <xdp/xsk.h>
#include <stdlib.h>
#include <sys/socket.h>

#define NUM_FRAMES 4096
#define FRAME_SIZE 2048

struct xsk_umem_info {
    struct xsk_ring_prod fq;
    struct xsk_ring_cons cq;
    struct xsk_umem *umem;
    void *buffer;
};

int init_af_xdp_socket(struct xsk_umem_info *umem_info, const char *ifname, int queue_id)
{
    struct xsk_socket_config xsk_cfg;
    struct xsk_umem_config umem_cfg = {
        .fill_size = NUM_FRAMES,
        .comp_size = NUM_FRAMES,
        .frame_size = FRAME_SIZE,
        .frame_headroom = XDP_PACKET_HEADROOM,
        .flags = 0
    };
    
    /* Allocate aligned contiguous memory buffer */
    posix_memalign(&umem_info->buffer, getpagesize(), NUM_FRAMES * FRAME_SIZE);
    
    /* Register UMEM memory pool with the kernel */
    xsk_umem__create(&umem_info->umem, umem_info->buffer, 
                     NUM_FRAMES * FRAME_SIZE, &umem_info->fq, 
                     &umem_info->cq, &umem_cfg);
                     
    xsk_cfg.rx_size = NUM_FRAMES;
    xsk_cfg.tx_size = NUM_FRAMES;
    xsk_cfg.libbpf_flags = 0;
    xsk_cfg.xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST | XDP_FLAGS_DRV_MODE;
    xsk_cfg.bind_flags = XDP_ZEROCOPY | XDP_USE_NEED_WAKEUP;
    
    return 0;
}

Using XDP_USE_NEED_WAKEUP, the application minimizes unnecessary system calls. The kernel only enters the driver poll path when userspace explicitly signals that the hardware queue requires replenishing.

Production Impact in RiceGuard Anycast Edge

RiceGuard employs AF_XDP across its Anycast gateway clusters to maintain deterministic performance under dense multi-tenant traffic loads:

  • Massive Packet Rate Density: RiceGuard processes over 24 million packets per second (Mpps) per dual-socket node, representing an order-of-magnitude leap over legacy TUN interfaces.
  • Microsecond Tail Latency: By eliminating kernel socket lock contention and memory copies, packet traversal latency across the gateway stays consistently below 1.2 microseconds.
  • Non-Invasive Operational Topology: Because the underlying physical interface remains a first-class Linux network adapter, network monitoring tools and Anycast BGP routing daemons operate seamlessly alongside the accelerated VPN dataplane.

By harnessing AF_XDP and zero-copy UMEM memory rings, WireGuard and RiceGuard architectures push userspace packet processing to wire speed without sacrificing Linux kernel integration.