Scaling multi-tenant VPN gateways to multi-gigabit line rates demands efficient distribution of millions of encrypted UDP datagrams across dozens of CPU cores. In traditional network architectures, worker threads bound to a single listening port compete for packets through centralized socket queues, causing severe lock contention and thread synchronization bottlenecks.

While the introduction of the Linux SO_REUSEPORT socket option permitted multiple independent sockets to bind to the exact same IP and port tuple, its standard kernel distribution mechanism relies on a fixed four-tuple hash (source IP, source port, destination IP, destination port). In high-performance encrypted networking, this rigid hashing model introduces critical architectural flaws: it breaks session affinity during client roaming, misaligns memory access with Non-Uniform Memory Access (NUMA) node boundaries, and causes severe cache bouncing between CPU sockets.

To resolve these scaling limitations, modern Linux network engineers combine SO_REUSEPORT with eBPF socket selection programs (BPF_PROG_TYPE_SK_REUSEPORT). By executing programmable packet steering inside the kernel's socket lookup path, gateways achieve deterministic session affinity, zero-lock core balancing, and strict NUMA locality.

The Scaling Pitfalls of Default SO_REUSEPORT Hashing

Under default Linux kernel behavior, incoming UDP datagrams matching an SO_REUSEPORT group are hashed across available sockets using the reciprocal_scale(skb_get_hash(skb), num_sockets) algorithm. While effective for stateless web traffic, this model degrades VPN performance in three distinct scenarios:

Incoming UDP Datagram
       │
       ▼
┌─────────────────────────────────┐
│ Kernel 4-Tuple Hashing          │
│ (src_ip, src_port, dst_ip, dst) │
└──────────────┬──────────────────┘
       │
       ├─────────────────┬─────────────────┐
       ▼                 ▼                 ▼
┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│ Socket 0     │  │ Socket 1     │  │ Socket 2     │
│ Core 0 (NUMA0)│  │ Core 4 (NUMA0)│  │ Core 8 (NUMA1)│
└──────────────┘  └──────────────┘  └──────┬───────┘
                                           │  <--- Cross-NUMA Interconnect Stall
                                           ▼       (Cache Line Invalidation)
                                    Remote L3 Cache
  1. Client Roaming and NAT Re-binding: Mobile VPN clients frequently change network interfaces (switching from Wi-Fi to cellular) or experience carrier NAT port re-mappings. When the source IP or source port changes, the kernel's four-tuple hash shifts immediately, routing packets of an ongoing cryptographic session to a completely different worker thread. The new thread lacks the session keys in its local cache, forcing inter-thread locking, state migration, or expensive re-handshakes.
  2. Cross-NUMA Memory Latency: On modern dual-socket server motherboards, PCIe network interface cards (NICs) attach directly to a specific CPU socket (NUMA Node 0). When the kernel's default hash assigns an incoming packet to a worker thread executing on NUMA Node 1, every memory access traverses the Ultra Path Interconnect (UPI). This cross-node traversal increases packet processing latency from ~800 nanoseconds to over 3.2 microseconds.
  3. Uneven Workload Distribution (Elephant Flows): Hash distribution is purely statistical. When a handful of high-bandwidth streaming or bulk-transfer sessions hash to the same worker core, that core experiences 100% CPU saturation while adjacent cores remain idle, triggering packet drops despite ample overall system capacity.

Programmable Dispatch: BPF_PROG_TYPE_SK_REUSEPORT Architecture

The BPF_PROG_TYPE_SK_REUSEPORT program type, coupled with a BPF_MAP_TYPE_REUSEPORT_SOCKARRAY, replaces the kernel's rigid hash function with user-defined steering logic.

Incoming Encrypted Datagram
       │
       ▼
┌────────────────────────────────────────────────────────┐
│ eBPF Classifier: BPF_PROG_TYPE_SK_REUSEPORT             │
│ - Inspects UDP payload (WireGuard Receiver Index / ID) │
│ - Reads smp_processor_id() for NUMA core alignment     │
│ - Calls bpf_sk_select_reuseport()                      │
└──────────────────────────┬─────────────────────────────┘
                           │
                           ▼
          ┌──────────────────────────────────┐
          │ BPF_MAP_TYPE_REUSEPORT_SOCKARRAY │
          └────────────────┬─────────────────┘
                           │
       ┌───────────────────┴───────────────────┐
       ▼                                       ▼
┌──────────────┐                        ┌──────────────┐
│ Worker Core 0│ (Local NUMA Node 0)    │ Worker Core 1│ (Local NUMA Node 0)
│ Socket FD 0  │                        │ Socket FD 1  │
└──────────────┘                        └──────────────┘

When an incoming packet hits the UDP socket lookup path, the kernel invokes the attached eBPF program before any socket queue assignment. The program inspects the struct sk_reuseport_md context:

  • skb->data and skb->data_end: Direct memory pointers to packet headers and payload bytes.
  • sk: The listening socket candidate.
  • ip_protocol: The transport protocol (UDP/TCP).
  • bind_inany: Socket binding flag.

By invoking the kernel helper bpf_sk_select_reuseport(md, &reuseport_map, &key, flags), the eBPF code directly designates which socket index in the map receives the datagram.

Kernel Implementation: WireGuard Receiver Index Steering

In WireGuard-compatible VPN implementations, every encrypted data packet contains a 32-bit receiver_index at byte offset 4 of the UDP payload. This index uniquely identifies the receiving peer session.

The following eBPF program demonstrates extracting this session identifier and selecting a dedicated, NUMA-pinned worker core:

#include <linux/bpf.h>
#include <linux/filter.h>
#include <bpf/bpf_helpers.h>

#define WG_DATA_PACKET_TYPE 4
#define NUMA_NODE_CORES 16

struct {
    __uint(type, BPF_MAP_TYPE_REUSEPORT_SOCKARRAY);
    __uint(max_entries, 64);
    __type(key, __u32);
    __type(value, __u32);
} reuseport_sock_map SEC(".maps");

SEC("sk_reuseport")
int select_vpn_socket(struct sk_reuseport_md *ctx)
{
    void *data = ctx->data;
    void *data_end = ctx->data_end;
    
    /* Boundary check: minimum UDP payload for WireGuard transport */
    if (data + 8 > data_end)
        return SK_PASS;
        
    __u8 packet_type = *(__u8 *)data;
    __u32 target_socket_index = 0;
    
    if (packet_type == WG_DATA_PACKET_TYPE) {
        /* Extract 32-bit receiver index located at byte offset 4 */
        __u32 receiver_index = *(__u32 *)(data + 4);
        
        /* Consistent session affinity: map index to worker pool */
        target_socket_index = receiver_index % NUMA_NODE_CORES;
    } else {
        /* Handshake init/response: dispatch to local CPU worker */
        __u32 current_cpu = bpf_get_smp_processor_id();
        target_socket_index = current_cpu % NUMA_NODE_CORES;
    }
    
    /* Lockless socket selection */
    int ret = bpf_sk_select_reuseport(ctx, &reuseport_sock_map, &target_socket_index, 0);
    if (ret == 0)
        return SK_PASS;
        
    return SK_DROP;
}

char _license[] SEC("license") = "GPL";

Through this logic, data packets belonging to the same peer session consistently land on the exact same worker core regardless of client IP roaming or NAT port renegotiations. At the same time, initial handshakes execute on the CPU core currently handling the network card's hardware interrupt, preserving L1/L2 processor cache warmth.

Userspace Attachment via SO_ATTACH_REUSEPORT_EBPF

Activating programmable socket steering requires configuring worker sockets and binding the compiled eBPF program using standard POSIX socket APIs:

int setup_reuseport_cluster(int epoll_fd, int bpf_prog_fd, int sock_map_fd)
{
    int num_cores = 16;
    
    for (int i = 0; i < num_cores; i++) {
        int sfd = socket(AF_INET, SOCK_DGRAM, 0);
        
        int opt = 1;
        setsockopt(sfd, SOL_SOCKET, SO_REUSEPORT, &opt, sizeof(opt));
        
        struct sockaddr_in addr = {
            .sin_family = AF_INET,
            .sin_port = htons(51820),
            .sin_addr.s_addr = htonl(INADDR_ANY)
        };
        bind(sfd, (struct sockaddr *)&addr, sizeof(addr));
        
        /* Store socket file descriptor into the eBPF sockarray */
        __u32 key = i;
        bpf_map_update_elem(sock_map_fd, &key, &sfd, BPF_ANY);
        
        /* Attach eBPF classifier to the first socket of the reuseport group */
        if (i == 0) {
            setsockopt(sfd, SOL_SOCKET, SO_ATTACH_REUSEPORT_EBPF, 
                       &bpf_prog_fd, sizeof(bpf_prog_fd));
        }
    }
    return 0;
}

Once attached to the first socket, the kernel applies the eBPF steering classifier across all sockets participating in the SO_REUSEPORT cluster.

Enterprise Scaling in RiceGuard: NUMA-Local Worker Affinity

RiceGuard implements BPF_PROG_TYPE_SK_REUSEPORT across its distributed Anycast infrastructure to eliminate multi-socket synchronization bottlenecks:

  • Zero Cross-NUMA Bouncing: RiceGuard pins worker execution rings to CPU cores sharing the local PCIe root complex of each physical 100G network adapter. Packets are parsed, decrypted, and forwarded entirely within the local L3 cache hierarchy, reducing end-to-end gateway latency by over 60%.
  • Seamless Roaming Resilience: When mobile subscribers transition across cellular base stations, RiceGuard's eBPF classifier resolves the incoming session token and directs the packet to the active worker thread without requiring cryptographic renegotiation.
  • Hitless Dynamic Upgrades: During runtime software deployments, new worker sockets register into the REUSEPORT_SOCKARRAY while retiring processes drain active connections, enabling zero-packet-drop gateway updates.

By uniting SO_REUSEPORT with eBPF programmable socket selection, VPN architectures achieve multi-core scalability and robust multi-tenant workload isolation.