The Multi-Tenant Hazard: Noisy Neighbors and Kernel Depletion

In enterprise remote-access infrastructures, cloud edge nodes, and carrier-grade tunnel concentrators, hosting multiple tenant workloads on shared bare-metal bare servers represents standard operational practice. A single physical host often orchestrates hundreds of concurrent client tunnels, microsegmentation proxies, and containerized cryptographic gateway daemons.

However, running multi-tenant VPN architectures without strict, low-level resource sandboxing introduces severe systemic risks:

  1. Kernel Socket Memory Depletion: Network sockets allocate kernel memory through the slab allocator (skbuff_head_cache). A misconfigured or abusive customer instance experiencing an internal packet loop or high-volume DDoS can consume gigabytes of unswappable kernel buffer memory (sk_rmem_alloc). Under legacy isolation, this triggers global Out-Of-Memory (OOM) panics that crash unrelated customer tunnels.
  2. CPU Starvation via Cryptographic Hogging: Symmetric decryption routines (such as ChaCha20-Poly1305 and AES-256-GCM) are heavily CPU-intensive. When an aggressive tenant saturates all shared CPU threads with unthrottled cryptographic handshakes, packet forwarding for adjacent tenants stalls, introducing unpredictable multi-millisecond latency spikes.
  3. Network Ingress Buffer Contention: Sockets competing for interface hardware queues can fill transmission buffers, causing random tail-drops across the entire host network interface card (NIC).

While legacy Linux cgroups v1 offered rudimentary controls, its fractured multi-hierarchy design suffered from uncoordinated resource attribution, inability to track kernel memory allocations, and high synchronization overhead. The modern Linux Control Groups v2 (cgroups v2) unified architecture provides the deterministic containment necessary for enterprise multi-tenant operations.

Inside cgroups v2: The Unified Hierarchy Architecture

Introduced and refined across Linux kernels 4.5 through 6.x, cgroups v2 replaces the independent per-subsystem trees of v1 with a single unified process hierarchy rooted at /sys/fs/cgroup.

+-------------------------------------------------------------------------+
|                  Unified cgroups v2 Root (/sys/fs/cgroup)               |
+-------------------------------------------------------------------------+
                                    |
      +-----------------------------+-----------------------------+
      | cgroup.subtree_control = "+cpu +memory +io"               |
      v                                                           v
+-------------------------------+               +-------------------------------+
| Tenant Slice: Alpha           |               | Tenant Slice: Beta            |
| /sys/fs/cgroup/vpn/tenant_a   |               | /sys/fs/cgroup/vpn/tenant_b   |
+-------------------------------+               +-------------------------------+
| - cpu.max = 200000 100000     |               | - cpu.max = 100000 100000     |
|   (Cap: 2 CPU cores)          |               |   (Cap: 1 CPU core)           |
| - memory.high = 2G            |               | - memory.high = 1G            |
| - memory.max = 3G             |               | - memory.max = 1.5G           |
| - eBPF cgroup_skb ingress     |               | - eBPF cgroup_skb ingress     |
| - eBPF cgroup_skb egress      |               | - eBPF cgroup_skb egress      |
+-------------------------------+               +-------------------------------+
       |                                               |
       v Attached Processes                            v Attached Processes
+-------------------------------+               +-------------------------------+
|  tun_worker_a (PID 14210)     |               |  tun_worker_b (PID 14502)     |
|  crypto_a     (PID 14211)     |               |  crypto_b     (PID 14503)     |
+-------------------------------+               +-------------------------------+

The "No Internal Processes" Rule

A fundamental architectural improvement in cgroups v2 is the strict operational principle that non-root control groups cannot simultaneously host both member processes and child control groups. Only leaf nodes can contain active process IDs (cgroup.procs). This eliminates ambiguous resource attribution and guarantees deterministic hierarchical scheduling across all subsystems.

To enable resource controllers for child slices, the parent node writes controller names to cgroup.subtree_control:

# Enable CPU, memory, and I/O controllers for VPN tenant slices
echo "+cpu +memory +io" > /sys/fs/cgroup/vpn/cgroup.subtree_control

Socket Memory Accounting and Reclaim Dynamics

A critical breakthrough in cgroups v2 is complete kernel memory integration. In cgroups v1, network socket buffers (struct sock and struct sk_buff) were largely invisible to memory limits unless complex auxiliary patches were applied. In cgroups v2, all socket allocations initiated by processes inside a cgroup are accounted directly against that group's memory footprint.

Proactive Control via memory.high vs memory.max

The memory controller provides a staged defense against runaway resource consumption:

  • memory.min (Hard Guarantee): Memory protected below this boundary cannot be reclaimed under global memory pressure.
  • memory.low (Best-Effort Protection): Memory below this threshold is preserved from reclamation unless no alternative reclaim targets exist.
  • memory.high (Throttling Threshold): When a tenant's aggregate memory usage (including active UDP/TCP tunnel buffers) exceeds memory.high, the kernel does not invoke the destructive OOM killer. Instead, the kernel throttles the offending processes on return from system calls and aggressively forces asynchronous page reclaim.
  • memory.max (Hard Upper Ceiling): If memory exceeds this absolute limit and reclaim fails, only processes within that specific tenant slice are targeted by the OOM killer, leaving the rest of the gateway entirely undisturbed.
# Configure defensive boundaries for a 2 GB VPN gateway slice
mkdir -p /sys/fs/cgroup/vpn/tenant_alpha
echo "512M" > /sys/fs/cgroup/vpn/tenant_alpha/memory.min
echo "2G"   > /sys/fs/cgroup/vpn/tenant_alpha/memory.high
echo "3G"   > /sys/fs/cgroup/vpn/tenant_alpha/memory.max

Network Throttling with eBPF cgroup_skb Programs

In legacy systems, shaping network traffic between co-located services required complicated traffic control (tc) disciplines attached to software bridge interfaces. In modern Linux environments, cgroups v2 couples directly with eBPF via the BPF_PROG_TYPE_CGROUP_SKB program type.

Network administrators can attach eBPF filters directly to any cgroup directory using BPF_CGROUP_INET_INGRESS and BPF_CGROUP_INET_EGRESS hooks. Every socket created by a process within that cgroup automatically inherits the packet filters, regardless of which network interface or namespace the socket communicates through.

// Example eBPF cgroup_skb egress rate limiter snippet
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __type(key, __u32);
    __type(value, __u64); // Token bucket tracking
    __uint(max_entries, 1);
} token_bucket SEC(".maps");

SEC("cgroup_skb/egress")
int throttle_tenant_egress(struct __sk_buff *skb)
{
    __u32 key = 0;
    __u64 *tokens = bpf_map_lookup_elem(&token_bucket, &key);
    if (!tokens)
        return 1; // Pass packet if map lookup fails

    // Check packet length against token balance
    if (skb->len > *tokens) {
        // Exceeds allocated bandwidth quota; drop packet cleanly
        return 0;
    }

    *tokens -= skb->len;
    return 1; // Permit transmission
}

char _license[] SEC("license") = "GPL";

Attaching this program to a tenant's cgroup node enforces line-rate bandwidth throttling inside the kernel datapath with zero userspace context switching.

Hardening RiceGuard Multi-Tenant Slices

In VPNRice's carrier-grade distributed gateways, the proprietary RiceGuard engine uses cgroups v2 to deliver isolated, multi-tenant network partitioning across shared enterprise clusters.

Physical Host Resources (64 Cores, 256 GB RAM, Dual 100GbE NICs)
                               |
                               v
+-------------------------------------------------------------------+
|               RiceGuard cgroups v2 Partition Manager              |
+-------------------------------------------------------------------+
       |                                             |
       v Slice Alpha (Gold Tier)                     v Slice Beta (Silver Tier)
+-----------------------------+               +-----------------------------+
| - cpu.weight: 200           |               | - cpu.weight: 50            |
| - cpu.max: 800000 100000    |               | - cpu.max: 200000 100000    |
| - memory.min: 4G            |               | - memory.min: 1G            |
| - memory.high: 16G          |               | - memory.high: 4G           |
| - eBPF Ingress Rate: 10Gbps |               | - eBPF Ingress Rate: 2Gbps  |
+-----------------------------+               +-----------------------------+
       |                                             |
       v                                             v
  Zero Noisy Neighbor Spill                     Zero Noisy Neighbor Spill

1. Completely Isolated CPU Scheduling

RiceGuard orchestrates multi-tenant workloads by combining cpu.weight (proportional fair-share weighting under contention) with cpu.max (hard CFS bandwidth enforcement). High-priority enterprise customer slices receive elevated proportional weights, guaranteeing that cryptographic worker pools always obtain necessary CPU cycles even during sudden platform-wide traffic spikes.

2. Lockless Memory Pressure Instrumentation

Rather than relying on periodic userspace monitoring tools, RiceGuard integrates with Linux Pressure Stall Information (PSI) within each cgroup (/sys/fs/cgroup/vpn/<tenant>/memory.pressure). When memory stalls indicate nascent queue saturation, the engine automatically signals clients to pace UDP window growth, neutralizing queue buildup before packet loss can occur.

3. Namespace and Network Slice Isolation

RiceGuard pairs cgroups v2 containment with dedicated Linux network namespaces. Sockets allocated by tenant workers are restricted to tenant-specific VRF or namespace routing tables. Coupled with eBPF cgroup egress filters, this architecture ensures that unencrypted packets cannot escape tenant boundaries, providing impenetrable privacy, guaranteed quality of service, and rock-solid platform reliability under extreme multi-tenant loads.