TCP BBR vs CUBIC: Maximizing VPN Throughput and Eliminating Bufferbloat

A ubiquitous frustration among remote professionals and network engineers is the "long-distance throughput cliff." A client provisioned with a 1 Gbps symmetric fiber connection initiates an encrypted tunnel across international transits—for example, connecting from Tokyo to Frankfurt with a 220 ms Round-Trip Time (RTT). Despite negligible CPU load and plentiful bandwidth at both endpoints, large file transfers and single-stream downloads collapse from hundreds of megabits per second down to an abysmal 12 to 18 Mbps.

When network administrators encounter this bottleneck, their first instinct is often to blame cryptographic cipher overhead or intermediate Internet Service Provider (ISP) traffic shaping. In reality, the culprit is almost always the transport layer's legacy congestion control algorithm: TCP CUBIC.

For decades, the global internet has relied on loss-based congestion control architectures. However, inside high Bandwidth-Delay Product (BDP) paths and encapsulated VPN tunnels, loss-based paradigms break down catastrophically. The emergence of BBR (Bottleneck Bandwidth and Round-trip propagation time), originally designed by Van Jacobson, Neal Cardwell, and the Google networking research team, has redefined transport engineering by swapping loss detection for precise physical modeling.

[Traditional Loss-Based CUBIC Dynamic]
Window Growth ──> Fills Router Buffer ──> Buffer Overflows (Tail Drop) ──> Packet Loss ──> Window Halved (-30%)
                     ▲                                                                           │
                     └──────────────────── Aggressive Sawtooth Rebuild ──────────────────────────┘
                             Result: Severe Bufferbloat, High Jitter, Collapsed Throughput

─────────────────────────────────────────────────────────────────────────────────────────────────────────────

[Model-Based TCP BBR Dynamic]
Continuously Measures: Bottleneck Bandwidth (BtlBw) & Minimum RTT (RTprop)
Paces Packets at Exact Link Delivery Rate ──> Router Queues Remain Empty ──> Minimal Inflight Data
                             Result: Maximum Throughput, Zero Bufferbloat, Deterministic Latency

The Mathematical Flaw of Loss-Based Control: CUBIC (RFC 9438)

To understand why traditional VPNs suffer across distance, one must examine how TCP CUBIC (standardized in RFC 9438) determines transmission rates.

Under the classic additive-increase/multiplicative-decrease (AIMD) framework and CUBIC's cubic window growth function, an endpoint assumes that packet loss is the sole indicator of network congestion. CUBIC probes for available capacity by continuously expanding its congestion window ($cwnd$) until a router along the route runs out of buffer memory and discards a packet.

Upon detecting a dropped packet via triple-duplicate ACKs or Selective Acknowledgment (SACK), CUBIC executes a sharp multiplicative penalty:

$$W_{\max} = \text{Current Flight Size}$$ $$cwnd_{\text{reduced}} = W_{\max} \times \beta \quad (\text{where } \beta = 0.7)$$

The algorithm slashes its transmission window by 30%, regardless of whether the packet drop was caused by genuine router saturation or transient wireless noise.

In 1997, Matthew Mathis established the fundamental throughput ceiling for loss-based TCP over IP paths, known as the Mathis Formula:

$$\text{Throughput} \le \frac{\text{MSS}}{\text{RTT}} \times \frac{C}{\sqrt{p}}$$

Where $\text{MSS}$ is the Maximum Segment Size, $\text{RTT}$ is the Round-Trip Time, $p$ is the packet loss probability, and $C$ is a constant approximately equal to $\sqrt{3/2} \approx 1.22$.

Consider the empirical reality of an international VPN tunnel:

• $\text{RTT} = 160\text{ ms } (0.160\text{ s})$
• $\text{MSS} = 1380\text{ bytes } (11,040\text{ bits})$
• Background packet drop rate $p = 1% \ (0.01)$, typical across public transits or home Wi-Fi networks:

$$\text{Throughput} \le \frac{11,040}{0.160} \times \frac{1.22}{\sqrt{0.01}} = 69,000 \times 12.2 \approx 841,800\text{ bps} \approx 6.7\text{ Mbps}$$

Despite operating over a physical 1 Gbps fiber link, a mere 1% stochastic packet drop forces CUBIC to throttle the transmission to under 7 Mbps. Because CUBIC requires multiple round-trips to restore its cubic curve, high latency compounds the penalty, ensuring the connection never recovers before the next random drop occurs.

The Scourge of Bufferbloat in Encrypted Tunnels

Loss-based congestion control does not merely destroy throughput over lossy paths; it actively degrades real-time traffic through Bufferbloat.

Because CUBIC only recognizes congestion after a buffer overflows, it relentlessly pumps unacknowledged packets into intermediate routing equipment. Network switches and consumer broadband modems, equipped with oversized FIFO buffers designed to prevent packet loss, store these excess datagrams in memory queues.

This creates massive artificial latency:

1. When an encrypted tunnel carries a high-bandwidth background stream (such as a database synchronization or file backup), CUBIC fills the bottleneck buffer to 100% capacity.
2. Latency-sensitive packets—such as WireGuard keepalives, SSH keystrokes, DNS queries, and real-time VoIP audio—are trapped behind thousands of queued bulk packets.
3. Physical round-trip latency surges from a baseline of 25 ms to over 400 ms.
4. The user experiences severe packet jitter, stuttering video calls, and responsive freezes, even though physical bandwidth remains unexhausted.

The BBR Architecture: Operating at the Kleinrock Bound

In 2016, researchers Van Jacobson, Neal Cardwell, Yuchung Cheng, and Matt Mathis introduced BBR (Bottleneck Bandwidth and RTT) to eliminate this forty-year-old compromise.

BBR departs completely from loss-based signaling. Instead of treating packet loss as congestion, BBR treats congestion as a physical state characterized by two independent properties:

1. Bottleneck Bandwidth ($\text{BtlBw}$): The maximum physical delivery rate of the slowest link along the end-to-end path.
2. Round-Trip Propagation Time ($\text{RTprop}$): The true physical latency of the path, determined purely by the speed of light in fiber and hardware switching overhead, without queueing delay.

Leonard Kleinrock demonstrated in 1979 that the optimal operating point for any data network occurs when the volume of inflight data exactly equals the Bandwidth-Delay Product (BDP):

$$\text{BDP} = \text{BtlBw} \times \text{RTprop}$$

Latency / Throughput
       │                       Kleinrock Optimal Operating Point
       │                                     ▼
Throughput ─────────────────────────┐────────────────────────── (Link Saturated at BtlBw)
       │                           ╱│
       │                          ╱ │
       │                         ╱  │
RTT    │                        ╱   │  Queue Accumulates ───────> (Bufferbloat Begins!)
       │                       ╱    │  ┌────────────────────────
Baseline RTprop ──────────────┴─────┴──┴────────────────────────
       └────────────────────────────────────────────────────────
                                 Inflight Data Volume

At the Kleinrock point:

• Delivery rate is maximized ($\text{Throughput} = \text{BtlBw}$).
• Round-trip time is minimized ($\text{RTT} = \text{RTprop}$).
• Queue length at the bottleneck router is zero, eliminating bufferbloat entirely.

To maintain this equilibrium without explicit router signaling, BBR implements a state machine governed by continuous active measurement:

• STARTUP: Exponentially ramps up transmission rate to discover the link's maximum delivery bandwidth.
• DRAIN: Temporarily reduces the sending rate to purge any transient queues created during STARTUP.
• PROBE_BW: Cycles the transmission pacing rate through a deterministic gain pattern (typically 1.25x for one round trip, 0.75x for one round trip, followed by six round trips at 1.0x). This probes for newly available capacity while actively draining queues.
• PROBE_RTT: Drops inflight packets to 4 segments once every 10 seconds for approximately 200 ms, allowing the sender to sample the true unloaded propagation time ($\text{RTprop}$).

Because BBR paces packets using kernel rate-limiting rather than unconstrained packet bursts, it transmits datagrams smoothly into the network fabric, preventing queue buildup at intermediate hops.

From BBRv1 to BBRv3: Maturing the Model

While initial deployments of BBRv1 delivered historic throughput gains across transcontinental routes, early enterprise evaluations identified two operational shortcomings:

1. Queue Squeeze against CUBIC: In deep-buffer environments, loss-based CUBIC flows persistently fill buffers until drops occur. BBRv1, striving to keep queues minimal, was occasionally starved of its fair-share bandwidth when competing directly with aggressive CUBIC connections.
2. Loss Inefficiency in Shallow Buffers: Under heavy random packet loss (exceeding 15% to 20%), BBRv1 occasionally maintained excessively high inflight data, increasing retransmission overhead.

The Internet Engineering Task Force (IETF) and Google addressed these challenges with BBRv2 and the recently deployed BBRv3:

• Explicit Congestion Notification (ECN) Integration: BBRv3 parses RFC 3168 ECN marks (Congestion Experienced - CE) to measure router queuing before packets are dropped.
• Dynamic Inflight Bounds: BBRv3 caps inflight data based on the observed packet loss rate, scaling down pacing gracefully to coexist equitably with legacy CUBIC traffic.
• Rapid RTT Discovery: Shortened PROBE_RTT phases minimize throughput dips while preserving precise propagation metrics.

Optimizing Encrypted Tunnel Performance with BBR

A widespread misconception in virtual networking is that BBR must be negotiated end-to-end across every hop. In reality, TCP congestion control is an endpoint decision.

In Layer-3 VPN topologies like WireGuard or RiceGuard, user applications generate inner TCP streams encapsulated within outer UDP datagrams. When the host operating system or the VPN gateway uses BBR for TCP connections, the rate-limiting and pacing decisions govern how quickly data enters the tunnel.

Linux Kernel Configuration for BBR

To enable BBR on modern Linux servers and client environments, Fair Queueing (fq) packet scheduling must be paired with the congestion module:

# Verify current available congestion control modules
sysctl net.ipv4.tcp_available_congestion_control

# Enable Fair Queueing and activate BBR in runtime
sudo sysctl -w "net.core.default_qdisc = fq"
sudo sysctl -w "net.ipv4.tcp_congestion_control = bbr"

# Optimize socket send buffer limits for high-speed high-latency links
sudo sysctl -w "net.ipv4.tcp_notsent_lowat = 16384"

To persist these parameters across system reboots, append the directives to /etc/sysctl.d/99-bbr-optimization.conf:

net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_notsent_lowat = 16384
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432

The parameter tcp_notsent_lowat = 16384 is particularly vital for VPN endpoints. By limiting unacknowledged data buffered inside local kernel sockets to 16 KB, the system avoids generating internal bufferbloat before packets ever reach the network interface.

Inspecting Live Socket Telemetry

Network engineers can verify BBR pacing parameters in real time using the Linux socket statistics utility:

# Inspect active TCP socket telemetry including BBR pacing state
ss -tin '( dport = :443 or sport = :443 )'

Output inspection confirms active model metrics:

ESTAB 0 0 10.0.0.2:48392 198.51.100.1:443
  bbr:(bw:182.4Mbps,mrtt:158.21,pacing_rate:228.0Mbps,cwnd:240)
  cubic:(...)

The output displays the measured bottleneck bandwidth (bw:182.4Mbps), the minimum round-trip time (mrtt:158.21), and the hardware pacing rate calculated by the kernel scheduler.

How VPNRice and RiceGuard Leverage BBR Pacing

High-throughput security networks require more than raw bandwidth; they demand intelligent traffic shaping that shields users from the structural flaws of the public internet. VPNRice integrates model-based pacing directly into the RiceGuard transport architecture:

• Edge Pacing across Anycast Gateways: All VPNRice ingress nodes execute tuned BBR schedulers on transit interfaces. By pacing traffic at the edge, data flows smoothly through international backhauls without triggering intermediate router buffer overflows.
• Zero-Loss Tunnel Traversal: When roaming over volatile Wi-Fi or cellular connections where transient packet drops occur naturally, RiceGuard prevents the artificial 70% throughput collapse enforced by legacy CUBIC implementations.
• Elimination of Gaming and Streaming Jitter: By keeping router queues essentially empty at the Kleinrock optimum, VPNRice maintains rock-solid, single-digit jitter even while concurrent gigabit downloads saturate the tunnel.