Transport Layer Security (TLS 1.3) serves as the cryptographic backbone of modern internet communications. In virtual private network (VPN) architectures, edge relays, and encapsulated privacy tunnels (such as DNS-over-HTTPS or TLS-wrapped datagram multiplexers), transport encryption safeguards data against inspection and tampering.

However, standard userspace TLS implementations—including OpenSSL, BoringSSL, and Rustls—suffer from structural performance bottlenecks when deployed on high-throughput network gateways. In traditional setups, every packet moving through a secure relay must be copied between kernel socket buffers and userspace memory across multiple context switches.

As edge gateways scale to 40GbE, 100GbE, and multi-gigabit workloads, these repeated memory copies and cryptographic operations saturate CPU caches, trigger memory bus contention, and cap throughput.

To eliminate userspace memory copying and offload symmetric cipher operations directly to hardware, the Linux kernel introduced Kernel TLS (kTLS, CONFIG_TLS), paired with SmartNIC hardware offloading.

+--------------------------------------------------------------------------+
|          DATA TRANSFER COMPARISON: USERSPACE TLS VS. LINUX KTLS          |
+--------------------------------------------------------------------------+
|                                                                          |
|  [ TRADITIONAL USERSPACE TLS RELAY ]                                     |
|                                                                          |
|      NIC Ingress ──> sk_buff ──(Copy 1)──> Userspace Buffer (RAM)        |
|                                                  │                       |
|                                      CPU Decrypt & Encrypt               |
|                                      (L1/L2 Cache Thrashing)             |
|                                                  v                       |
|      NIC Egress  <── sk_buff <──(Copy 2)── Userspace Buffer (RAM)        |
|      (Requires 4 Copies & 2 Context Switches Per Stream Transfer)        |
|                                                                          |
|  ----------------------------------------------------------------------  |
|                                                                          |
|  [ LINUX KTLS ZERO-COPY HARDWARE OFFLOAD RELAY ]                         |
|                                                                          |
|      NIC Ingress (DMA)                                                   |
|           │                                                              |
|           v (Kernel Page Buffer / sk_buff)                               |
|      +-------------------------------------------------------------+     |
|      |               LINUX KERNEL NETWORKING STACK                 |     |
|      |                                                             |     |
|      |   kTLS Engine (CONFIG_TLS) / eBPF Sockmap Redirection       |     |
|      |   - Zero Userspace Memory Copies                            |     |
|      |   - Direct pipe/splice() kernel forwarding                  |     |
|      +-------------------------------------------------------------+     |
|           │                                                              |
|           v (Zero-Copy DMA Egress)                                       |
|      NIC Hardware Crypto Engine (Mellanox / Intel Inline Offload)        |
|           │                                                              |
|           v                                                              |
|      Encrypted Wire Output (Line-Rate 100GbE with Zero Host CPU Overhead)|
|                                                                          |
+--------------------------------------------------------------------------+

The Userspace Overhead Bottleneck

In conventional network daemon architectures, relaying encrypted traffic over TLS incurs substantial overhead. For an edge relay receiving encrypted traffic and forwarding it across a secure tunnel:

  1. System Call Frequency: The process invokes read() or SSL_read(), triggering context transitions from userspace to kernel space.
  2. Double Memory Copying: Ingress network frames in kernel ring buffers must be copied into userspace heap allocations. Following processing, write() or SSL_write() copies modified payloads back into outgoing kernel socket buffers.
  3. Cache Line Invalidation: Continuous symmetric cryptographic operations (AES-GCM or ChaCha20-Poly1305) pollute L1 and L2 processor caches, causing memory pipeline stalls when processing millions of packets per second.

On 100GbE server interfaces, this architectural limitation restricts single-core network throughput to approximately 18–25 Gbps, regardless of hardware specifications.

Linux kTLS Architecture: Separating Handshake and Record Layers

Kernel TLS addresses this bottleneck through clean separation of responsibilities:

  • Control Plane in Userspace: Complex, stateful tasks—such as X.509 certificate validation, ALPN negotiation, and asymmetric key exchange (ECDHE / Post-Quantum ML-KEM)—remain safely inside userspace libraries (OpenSSL or GnuTLS).
  • Data Plane in the Kernel: Once the symmetric session keys are negotiated, userspace transfers the cipher parameters into the kernel socket layer via setsockopt().

From that point forward, the Linux kernel manages TLS record framing, sequence numbering, and encryption/decryption natively during socket I/O operations (tcp_sendmsg and tcp_recvmsg).

+--------------------------------------------------------------------------+
|                  KTLS KERNEL INITIALIZATION WORKFLOW                     |
+--------------------------------------------------------------------------+
|                                                                          |
|    1. Establish TCP Handshake (Standard Socket)                          |
|                                                                          |
|    2. Perform Userspace TLS 1.3 Handshake (OpenSSL / BoringSSL)          |
|       - Negotiate Cipher: TLS_AES_256_GCM_SHA384                         |
|       - Derive Symmetric Traffic Keys & IVs                              |
|                                                                          |
|    3. Activate kTLS via setsockopt():                                    |
|       setsockopt(fd, SOL_TCP, TCP_ULP, "tls", sizeof("tls"));            |
|                                                                          |
|    4. Push Symmetric Keys into Kernel:                                   |
|       setsockopt(fd, SOL_TLS, TLS_TX, &crypto_info, sizeof(info));       |
|       setsockopt(fd, SOL_TLS, TLS_RX, &crypto_info, sizeof(info));       |
|                                                                          |
|    5. Direct Kernel I/O:                                                 |
|       Kernel automatically frames, encrypts, and authenticates records   |
|       Userspace uses splice(), sendfile(), and MSG_ZEROCOPY seamlessly   |
|                                                                          |
+--------------------------------------------------------------------------+

Enabling kTLS in Modern C / Rust Applications

Below is an implementation demonstrating how an edge relay activates kTLS on an active file descriptor using OpenSSL 3.0+:

#include <sys/socket.h>
#include <netinet/tcp.h>
#include <linux/tls.h>
#include <openssl/ssl.h>

int enable_ktls_session(SSL *ssl, int socket_fd) {
    /* Step 1: Instruct OpenSSL to use kernel TLS */
    SSL_set_options(ssl, SSL_OP_ENABLE_KTLS);

    /* Step 2: Complete the standard TLS 1.3 cryptographic handshake */
    if (SSL_accept(ssl) <= 0) {
        return -1; /* Handshake failure */
    }

    /* Step 3: Verify if kTLS was successfully enabled on TX and RX */
    if (BIO_get_ktls_send(SSL_get_wbio(ssl))) {
        /* Kernel is actively handling outbound TLS framing and encryption */
    }
    if (BIO_get_ktls_recv(SSL_get_rbio(ssl))) {
        /* Kernel is actively handling inbound TLS decryption and verification */
    }

    return 0;
}

Zero-Copy Forwarding via splice() and sendfile()

The primary architectural advantage of kTLS is unlocking kernel zero-copy primitives for encrypted sockets.

In userspace TLS, system calls like splice() cannot be used because plaintext data must be encrypted before reaching the socket buffer. Because kTLS handles encryption inside the kernel's network layer, applications can stream data directly between file descriptors without copying data into userspace:

/* Zero-copy forwarding between an inbound socket and outbound tunnel */
int pipefd[2];
pipe(pipefd);

while (1) {
    /* Splice from decrypted inbound socket into in-kernel pipe buffer */
    ssize_t bytes = splice(inbound_fd, NULL, pipefd[1], NULL, 65536, SPLICE_F_MOVE);
    if (bytes <= 0) break;

    /* Splice from pipe buffer into outbound kTLS-encrypted socket */
    splice(pipefd[0], NULL, outbound_fd, NULL, bytes, SPLICE_F_MOVE);
}

Through splice(), network packets transition from the inbound ring buffer to the outbound network interface without crossing the userspace memory boundary.

Hardware Offload: Shifting Cryptography to SmartNIC Silicon

While software kTLS running on host CPUs provides significant speedups, high-end enterprise network interfaces support kTLS Hardware Device Offload (supported by NVIDIA Mellanox ConnectX-6 Dx / BlueField and Intel Ethernet 800 Series).

The Three Modes of kTLS Operation

The Linux kernel supports three distinct operational tiers for kTLS:

+--------------------------------------------------------------------------+
|                      KTLS OPERATIONAL TIERS COMPARED                     |
+--------------------------------------------------------------------------+
| Mode                | Processing Engine     | CPU Load | 100GbE Feasible |
+---------------------+-----------------------+----------+-----------------+
| Userspace Software  | Host CPU (Userspace)  | 100%     | No (Bottleneck) |
| kTLS Software (SW)  | Host CPU (Kernel ASM) | 35-50%   | Yes (Multi-Core)|
| kTLS Device Offload | SmartNIC Silicon ASIC | <2%      | Yes (Full Line) |
+--------------------------------------------------------------------------+

When device offload is active:

  1. The host kernel attaches crypto context tags to the sk_buff descriptor rather than encrypting payload memory.
  2. The network interface's hardware DMA engine pulls plaintext directly from host RAM.
  3. Dedicated hardware AES engines on the physical NIC encrypt data at full line rate immediately before physical serialization onto optical fiber.

To verify and activate hardware kTLS offload on supported interfaces:

# Check current hardware TLS offload capabilities
ethtool -k eth0 | grep tls-hw

# Enable transmit and receive hardware TLS offload
ethtool -K eth0 tls-hw-tx-offload on
ethtool -K eth0 tls-hw-rx-offload on

Integration with eBPF Sockmap for In-Kernel Relay Routing

For maximum forwarding efficiency, kTLS integrates directly with eBPF sockmaps (BPF_MAP_TYPE_SOCKMAP and BPF_PROG_TYPE_SK_MSG).

When an eBPF verdict program is attached to a kTLS socket, the kernel decrypts incoming records into the socket receive queue. The eBPF program parses application-layer headers (such as HTTP/2 framing or proxy headers) and executes bpf_msg_redirect_map() to reroute the packet to an egress tunnel socket without waking userspace worker threads:

Ingress NIC ──> kTLS Decrypt ──> eBPF sk_msg Filter ──> kTLS Encrypt ──> Egress NIC
               [ 100% Executed In-Kernel Without Context Switches ]

Engineering Synergies with RiceGuard and VPNRice

Within VPNRice's global distributed edge fabric, kTLS and hardware offload deliver fundamental advantages across demanding network workloads:

  1. High-Throughput Edge Termination: Edge nodes terminating encapsulated DoH (DNS-over-HTTPS) and DoT (DNS-over-TLS) services utilize kTLS to process hundreds of thousands of concurrent encrypted queries per node, maintaining sub-millisecond response latency.
  2. Thermal and Energy Efficiency: Offloading symmetric AES-GCM operations to NIC hardware reduces gateway CPU core utilization by up to 80%, lowering server power consumption and eliminating thermal throttling on dense server clusters.
  3. Strict Zero-Copy Data Protection: Because decrypted network payloads are redirected directly between kernel socket queues, unencrypted customer traffic is never retained in userspace process memory heaps, mitigating exposure to memory-dump inspection vulnerabilities.

Operational Considerations and Deployment Checklist

System administrators deploying kTLS in high-load production environments should observe the following guidelines:

  • Kernel Compatibility: Ensure edge systems operate Linux kernel 5.15 LTS or newer for mature TLS 1.3 support, and kernel 6.1+ for hardware device offload stability.
  • ULP Permissions: Verify that unprivileged user policies allow User Level Protocols (ULP) socket configuration, or run network daemons with CAP_NET_ADMIN.
  • Cipher Negotiation: Restrict negotiated ciphers to TLS_AES_128_GCM_SHA256 or TLS_AES_256_GCM_SHA384, as hardware NIC offloading engines generally target AES-GCM rather than ChaCha20.

The synergy of Linux kTLS, zero-copy forwarding, and SmartNIC hardware offload redefines high-performance secure networking, delivering uncompromising privacy and wire-speed throughput across modern enterprise VPN gateways.