High-throughput VPN gateways processing tens of thousands of concurrent encrypted tunnels often encounter an unexpected bottleneck: severe CPU core imbalance. A multi-core server with 32 or 64 cores may show aggregate CPU usage below 25%, yet drop incoming datagrams during peak load. Inspecting system metrics typically reveals CPU core 0 running at 100% software interrupt (ksoftirqd/0 and NET_RX_SOFTIRQ), while adjacent cores remain idle.

This processing imbalance stems from how modern network hardware and tunnel encapsulations interact with the operating system. When hardware Receive Side Scaling (RSS) cannot distribute encapsulated tunnel datagrams evenly, Linux provides two software acceleration mechanisms: Receive Packet Steering (RPS) and Receive Flow Steering (RFS). Configuring these mechanisms correctly enables VPN operators to scale packet processing across all available cores while preserving processor cache locality.

The Tunnel Bottleneck: Single-Queue Collapse in VPN Ingress

Physical Network Interface Cards (NICs) distribute incoming frames across hardware receive queues using Receive Side Scaling. The NIC computes a Toeplitz hash of the packet headers—usually a 4-tuple comprising source IP, destination IP, source port, and destination port—to assign each flow to an RX ring buffer. Each RX ring triggers a dedicated MSI-X hardware interrupt mapped to a specific CPU core.

Hardware RSS Limitations with Encapsulated VPN Traffic:
[ Remote Clients ] ---> [ Outer WireGuard / IPsec / RiceGuard UDP Packets ]
                               |
                               v
                     [ Single Target Port (e.g. UDP 51820) ]
                               |
                   [ Ingress NIC Toeplitz Hash ]
                               |
       =================================================
       All flows hash to RX Queue 0 (Single MSI-X IRQ)
       =================================================
                               |
                               v
       [ CPU Core 0: 100% ksoftirqd / NET_RX_SOFTIRQ ] ---> Ring Buffer Overrun!
       [ CPU Cores 1-31: 0% SoftIRQ Load ]

When thousands of users connect to a VPN concentrator through protocol tunnels such as WireGuard, IPsec, or custom datagram protocols, the outer packet stream terminates at a single destination port on the gateway. If multiple client streams arrive from identical intermediate carrier NAT gateways, the 4-tuple hash collapses onto a small subset of hardware queues.

Virtual tunnel interfaces (tun or tap) also lack multi-queue hardware distribution by default. When the gateway decrypts incoming payloads and injects inner IP packets into the kernel networking stack, softirq processing concentrates heavily on the CPU core executing the decryption thread. Unserviced ring buffers fill up, tail latency spikes, and silent packet loss degrades real-time streaming and gaming sessions.

Receive Packet Steering: Software Multi-Core Distribution

Receive Packet Steering distributes packet processing load across an administrator-defined mask of CPU cores after the initial hardware interrupt completes. Unlike hardware RSS, which operates entirely within NIC silicon, RPS executes in software during the driver's NAPI receive poll cycle (netif_receive_skb).

RPS Software Distribution Architecture:
[ Ingress Packet on RX Queue 0 ]
              |
              v
     [ NAPI Poll Routine ]
              |
     [ rps_cpus CPU Bitmask: e.g., 0x0000ffff (Cores 0-15) ]
              |
     [ Compute jhash over 4-tuple: hash % active_cpus ]
              |
              v
 [ Enqueue sk_buff to Target CPU's input_pkt_queue ]
              |
    [ Inter-Processor Interrupt (IPI): trigger NET_RX on Target Core ]
              |
     +--------+--------+--------+--------+
     |        |        |        |        |
     v        v        v        v        v
  [Core 0] [Core 1] [Core 2] [Core 3] [Core 4] ... [Core 15]

When a packet arrives, the driver determines whether RPS is enabled on that interface queue by checking sysfs. If enabled, the kernel hashes the packet's network and transport headers using Jenkins hash (jhash). The resulting hash maps to one of the CPUs enabled in the queue's rps_cpus bitmask.

The receiving core places the sk_buff descriptor into the target CPU's per-processor backlog queue (input_pkt_queue) and triggers an Inter-Processor Interrupt (IPI). The target CPU then processes the packet through the upper protocol layers: firewall filters, routing tables, and socket delivery.

Receive Flow Steering: Cache Locality and Flow Affinity

While RPS successfully spreads CPU load across multiple cores, it chooses target CPUs based solely on the packet hash. It does not know where the user-space application thread reading from the socket resides.

If core 4 processes the incoming network packet through the kernel stack, but the VPN daemon worker thread runs on core 12, the payload data must travel across CPU cache boundaries. This cross-core data migration causes CPU L1/L2 cache misses, invalidates cache lines, and degrades memory throughput.

Receive Flow Steering solves this inefficiency by directing packet processing directly to the CPU core where the application thread is actively polling or reading from the socket.

RFS Flow Affinity Mechanism:
1. User-Space Thread on Core 3 calls recvmsg(sock_fd)
   ---> Kernel updates Global Flow Table (rps_sock_flow_table):
        [ Flow Hash #A18C -> Running on Core 3 ]

2. Next Packet for Flow #A18C arrives on NIC Queue
   ---> Driver queries rps_sock_flow_table:
        Target CPU = Core 3 (Where application memory is hot!)

3. sk_buff enqueued directly to Core 3's input_pkt_queue
   ---> SoftIRQ and Application execute on same L1/L2 Cache Hierarchy!

RFS coordinates two internal tables:

  1. Global Socket Flow Table (rps_sock_flow_table): Populated automatically whenever an application thread invokes recvmsg(2), read(2), or epoll_wait(7) on a socket. The kernel records the flow hash and the CPU core ID of the calling thread.
  2. Per-Queue Flow Table (rps_flow_table): Maintained on each network interface receive queue. It stores the CPU ID that processed the most recent packet for each flow.

Before routing a packet to the CPU indicated by the global socket table, RFS compares the entry against the per-queue table. If the application migrated to a new core, RFS only shifts packet processing after all pending backlog packets on the previous core have cleared. This safeguard prevents out-of-order packet delivery.

Practical Implementation: Configuring RPS and RFS on Gateway Nodes

To illustrate deployment on a 16-core Linux VPN gateway with an external interface eth0 and a WireGuard interface wg0, consider the following tuning procedure.

Step 1: Assign RPS CPU Bitmasks

First, allocate CPU cores for software packet steering. On a system with 16 cores (CPUs 0 through 15), a bitmask of 0xffff enables all cores. For dedicated setups, reserve CPUs 0 and 1 for hardware interrupts and system tasks, dedicating CPUs 2 through 15 (bitmask 0xfffc) to packet processing:

# Enable RPS across CPUs 2-15 on all RX queues of eth0
for rx_queue in /sys/class/net/eth0/queues/rx-*; do
    echo "fffc" > "$rx_queue/rps_cpus"
done

# Enable RPS on virtual tunnel device
echo "fffc" > /sys/class/net/wg0/queues/rx-0/rps_cpus

Step 2: Configure Global and Per-Queue RFS Tables

Next, size the flow tables based on concurrent active connections. For a gateway handling up to 32,768 simultaneous client sessions, round up to the nearest power of two (65,536):

# Set global socket flow table entries
sysctl -w net.core.rps_sock_flow_entries=65536

# Allocate flow entries across each physical RX queue (e.g., 4 queues: 65536 / 4 = 16384)
for rx_queue in /sys/class/net/eth0/queues/rx-*; do
    echo 16384 > "$rx_queue/rps_flow_cnt"
done

Step 3: Expand the Input Packet Backlog

Under high-velocity packet bursts, the target CPU's backlog queue can overflow if SoftIRQ execution lags behind incoming traffic. Increase netdev_max_backlog to prevent drop counts in /proc/net/softnet_stat:

# Increase maximum backlogged packets per core from 1000 to 10000
sysctl -w net.core.netdev_max_backlog=10000

# Increase budget for each NAPI polling cycle
sysctl -w net.core.netdev_budget=600
sysctl -w net.core.netdev_budget_usecs=4000

Architectural Integration in RiceGuard VPN

RiceGuard, the proprietary Layer-3 communication engine developed by VPNRice, integrates directly with kernel-level packet steering primitives. Rather than relying on default operating system scheduling, RiceGuard coordinates its thread pools with Linux network topology:

  1. NUMA-Aware Core Pinning: RiceGuard isolates hardware interrupt handling, cryptographic workers, and socket I/O loops onto dedicated NUMA nodes. When ingress traffic arrives on a physical NIC connected to NUMA node 0, RiceGuard configures rps_cpus exclusively across CPU cores sharing local L3 cache with that PCI Express bus.
  2. Synchronized Flow Registration: During high-concurrency connection ramp-ups, RiceGuard socket handlers register affinity hints with the kernel socket table, maintaining strict CPU locality between user-space tunnel processors and in-kernel SoftIRQ routines.
  3. Deterministic Latency Benchmarks: In internal stress tests on 10 Gbps transit links carrying 25,000 active client streams, enabling synchronized RPS and RFS lowered 99th percentile (p99) transit latency from 8.4 ms to 1.9 ms, while eliminating SoftIRQ throttling drops.

Frequently Asked Questions

What is the difference between RSS, RPS, and RFS?

RSS is a hardware feature of the network adapter that uses hardware queues and MSI-X interrupts to distribute packets across CPU cores. RPS is a software implementation that balances packet receive processing across multiple cores after the initial interrupt. RFS builds on RPS by steering packets specifically to the CPU core executing the application thread that owns the destination socket.

Does enabling RPS increase CPU utilization?

RPS introduces a modest overhead of approximately 1% to 2% CPU usage due to Jenkins hashing and Inter-Processor Interrupts. However, by preventing single-core saturation and distributing work evenly, it substantially raises the server's overall throughput ceiling.

Can RPS and RFS be applied to virtual tun/tap interfaces?

Yes. Virtual tun and tap interfaces benefit significantly from RPS. Because virtual tunnel devices usually have only a single queue, enabling rps_cpus on rx-0 distributes decrypted packet processing across the remaining CPU cores, preventing tunnel bottlenecks.