High-throughput VPN gateways frequently maintain TCP-based ingress entry points alongside UDP tunnels. Operational environments rely on TCP for fallback encapsulation when restrictive enterprise networks filter UDP packets, as well as for control planes, TLS-based transport mechanisms, and user management APIs. When public-facing nodes expose open TCP listening sockets on port 443, they inevitably encounter distributed TCP SYN flood attacks.

While modern hardware easily pushes millions of packets per second, traditional Linux connection tracking architectures struggle under brute-force connection floods. If a firewall tracks every incoming packet before verifying client legitimacy, internal state tables collapse long before network interfaces reach saturation. Solving this architectural vulnerability requires eliminating connection state during initial handshakes using Linux nftables SYNPROXY.

Understanding how SYNPROXY maintains service availability requires analyzing how the Linux Netfilter stack handles connection tracking under high packet rates.

Incoming SYN Flood Threat:
Attacker SYN Flood ──> NIC Ingress ──> Netfilter PREROUTING ──> nf_conntrack alloc (struct nf_conn)
                                                                           │
                                                                 TABLE EXHAUSTION (nf_conntrack: table full)
                                                                           │
                                                                    PACKET LOSS (All users dropped)

─────────────────────────────────────────────────────────────────────────────────────────────────────────────

nftables SYNPROXY Stateless Mitigation:
Attacker SYN Flood ──> PREROUTING (notrack) ──> INPUT (SYNPROXY) ──> Stateless SYN-ACK (Syncookie)
                                                                           │
                                                            (No Memory / No State Allocation)
                                                                           │
Legitimate Client ACK ─────────────────────────────────────────> Verify Cookie ──> Establish Conntrack ──> VPN Daemon

The Conntrack Bottleneck: Why Socket Syncookies Fall Short

System administrators commonly assume that enabling kernel TCP syncookies via net.ipv4.tcp_syncookies = 1 provides complete defense against SYN flooding. While host syncookies protect the local socket listen queue (listen(backlog)), they do not shield the Netfilter connection tracking subsystem (nf_conntrack).

In standard Linux network packet flows, the Netfilter connection tracker hooks into the packet path inside NF_INET_PRE_ROUTING, well ahead of the local transport stack (NF_INET_LOCAL_IN). When an incoming TCP packet with the SYN flag set reaches the firewall, nf_conntrack allocates a tracking structure (struct nf_conn) and inserts it into the global hash table in the SYN_RECV state:

Packet Arrival ──> nf_conntrack_in() ──> init_conntrack() ──> kmem_cache_alloc(nf_conntrack_cachep)

Each tracking record consumes approximately 320 to 384 bytes of kernel memory slab space. On a host configured with a default nf_conntrack_max limit of 262,144 entries, an adversary firing 300,000 spoofed SYN packets per second will exhaust the connection table in less than 900 milliseconds.

When the conntrack table fills:

  1. Subsequent incoming packets trigger net_ratelimit: nf_conntrack: table full, dropping packet.
  2. Existing active VPN tunnels and legitimate new handshakes are dropped unconditionally.
  3. System CPU cycles spike as kernel softirq routines walk saturated hash buckets.

Because standard socket syncookies operate inside tcp_v4_rcv()—after nf_conntrack has already attempted and failed to allocate memory—they never execute. Protecting ingress gateways requires moving handshake validation in front of connection tracking.

Anatomy of Netfilter SYNPROXY Architecture

The Netfilter synproxy engine, implemented in kernel space within net/netfilter/nf_synproxy_core.c and exposed through nftables, functions as an in-kernel stateless reverse proxy. It splits connection establishment into two discrete phases: an untracked outer handshake with the external client, followed by an inner handshake with the local listening daemon.

External Client                         nftables SYNPROXY                        Local VPN Daemon
      │                                         │                                       │
      │─── 1. TCP SYN [MSS, WScale, SACK] ─────>│ (Untracked, zero memory allocated)    │
      │                                         │                                       │
      │<── 2. Stateless SYN-ACK [Cookie] ───────│ (Cryptographic hash of 4-tuple & ISN) │
      │                                         │                                       │
      │─── 3. TCP ACK [Cookie + 1] ────────────>│                                       │
      │                                         │─── 4. Local TCP SYN ─────────────────>│
      │                                         │<── 5. Local SYN-ACK [Local ISN] ──────│
      │                                         │─── 6. Local ACK ─────────────────────>│
      │                                         │                                       │
      │═════════════════════════════════════════╪═══════════════════════════════════════│
      │              Transparent Data Phase with Hardware Delta Rewriting               │
      │─── 7. Application TLS / Data ──────────>│─── 8. Rewrite SEQ/ACK (delta) ───────>│
      │<── 10. Rewrite SEQ/ACK (delta) ─────────│<── 9. Response Data ──────────────────│

The protocol flow operates through precise stages:

Stage 1: Untracked Ingress

Incoming SYN packets targeting the protected VPN listening port are flagged with notrack in the raw PREROUTING table. This prevents nf_conntrack from allocating a struct nf_conn object.

The SYN reaches the SYNPROXY engine in the INPUT chain. Rather than forwarding the frame to socket queues, SYNPROXY computes a cryptographic 32-bit Initial Sequence Number (ISN) based on:

  • Source IP and destination IP addresses
  • Source port and destination port numbers
  • Client initial sequence number
  • A kernel secret key refreshed every 60 seconds
  • Encoded TCP options (Maximum Segment Size index, Window Scaling factor, SACK permission)

SYNPROXY constructs a synthesized TCP SYN-ACK packet carrying this syncookie and transmits it directly out of the egress interface. The kernel retains zero state, zero socket objects, and zero timer structures. If the packet was part of a spoofed flood, the exchange terminates here without consuming host resources.

Stage 3: ACK Validation and State Creation

When a genuine client receives the SYN-ACK, it responds with a standard ACK carrying sequence number Cookie + 1. SYNPROXY intercepts this packet, recalculates the cryptographic cookie formula, and verifies its validity.

Once cryptographic proof verifies client authenticity, SYNPROXY initiates connection tracking:

  • A real struct nf_conn entry is instantiated in nf_conntrack.
  • The connection state transitions directly to ESTABLISHED.
  • Negotiated options (TCP timestamps, selective acknowledgment, window scale) are restored from the decoded cookie.

Stage 4: Local Handshake and Delta Tracking

Because the local VPN daemon (such as OpenVPN, a TLS proxy, or a STUN coordinator) expects a standard 3-way handshake, SYNPROXY creates a synthetic local SYN packet directed at the local listening socket.

The daemon responds with a local SYN-ACK containing its own generated sequence number (ISN_server). SYNPROXY absorbs this packet, calculates the numerical sequence number delta:

$$\Delta = ext{ISN}_{ ext{server}} - ext{Cookie}$$

From this moment onward, SYNPROXY remains in the packet path, modifying TCP header fields at wire speed:

  • Inbound client packets: increments sequence numbers by $\Delta$.
  • Outbound server responses: decrements acknowledgment numbers by $\Delta$.

This translation occurs completely in the kernel datapath, delivering full protocol transparency to application code.

Production Implementation: Configuring nftables SYNPROXY

Deploying SYNPROXY requires configuring nftables with appropriate table priorities and rule structures. The configuration defines an untracked raw prerouting hook and an input inspection policy.

Below is an enterprise production configuration targeting VPN TCP services on port 443:

#!/usr/sbin/nft -f

flush ruleset

table inet vpn_filter {
    # Define synproxy parameters matching target server capabilities
    synproxy synproxy_vpn {
        mss 1460
        wscale 7
        timestamp sack-perm
    }

    chain raw_prerouting {
        type filter hook prerouting priority raw; policy accept;

        # Exclude from connection tracking for initial TCP SYN packets on port 443
        tcp dport 443 tcp flags syn notrack
    }

    chain filter_input {
        type filter hook input priority filter; policy drop;

        # Allow loopback traffic
        iif "lo" accept

        # Accept established and related connections
        ct state established,related accept

        # Drop invalid packet states
        ct state invalid drop

        # Route incoming TCP SYN/ACK to SYNPROXY validation engine
        tcp dport 443 synproxy config synproxy_vpn

        # Standard accept for validated connections passing to local daemon
        tcp dport 443 ct state new accept
    }

    chain filter_forward {
        type filter hook forward priority filter; policy drop;
        # Forwarding rules for tunnel subnets
    }

    chain filter_output {
        type filter hook output priority filter; policy accept;
    }
}

To load and enforce the configuration:

sudo nft -f /etc/nftables.conf

Verify that the synproxy object and associated chains loaded correctly:

sudo nft list table inet vpn_filter

Kernel Parameter Tuning and Performance Verification

For SYNPROXY to function predictably, supporting Linux networking sysctl options must be aligned with packet handling requirements.

Update /etc/sysctl.d/99-synproxy.conf:

# Enable kernel syncookies
net.ipv4.tcp_syncookies = 1

# Enable TCP timestamps (required for encoding window scaling and SACK inside cookies)
net.ipv4.tcp_timestamps = 1

# Disable loose conntrack pickup to prevent spoofed ACK state insertion
net.netfilter.nf_conntrack_tcp_loose = 0

# Expand conntrack hash buckets and limits for validated post-handshake sessions
net.netfilter.nf_conntrack_max = 2097152
net.netfilter.nf_conntrack_buckets = 524288

# Optimize conntrack TCP timeout thresholds
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 10
net.netfilter.nf_conntrack_tcp_timeout_established = 3600

Apply settings dynamically:

sudo sysctl --system

Inspecting Runtime Statistics

You can monitor SYNPROXY packet interception and validation directly from the command line:

# Monitor conntrack entry counts
cat /proc/sys/net/netfilter/nf_conntrack_count

# View SYNPROXY operational counters
cat /proc/net/stat/synproxy

The output file displays columnar metrics tracking:

  • entries: Number of active SYNPROXY tracked sessions.
  • syn_received: Total incoming SYN packets intercepted statelessly.
  • cookie_invalid: Malformed or forged ACKs rejected.
  • cookie_valid: Successful 3-way handshakes transitioned into active conntrack entries.

Architectural Synergy with RiceGuard Edge Gateways

Modern security architectures mandate defense-in-depth across every network protocol layer. While the core RiceGuard protocol operates over UDP with native cryptographic cookie verification, enterprise gateway deployments expose dual-stack TCP interfaces for legacy environment traversal, WebSocket encapsulation, and DNS over HTTPS (DoH) lookups.

By placing nftables SYNPROXY directly at the bare-metal kernel boundary, RiceGuard infrastructure decouples ingress survivability from connection state tracking. Volumetric floods targeting TCP ingress points are neutralized statelessly at the network interface layer, ensuring that memory structures, scheduling queues, and CPU caches remain reserved for encrypted tunnel traffic.

Configuring stateless packet filtering provides resilient infrastructure capable of withstanding severe network hostility while maintaining dependable throughput for connected users.

Learn more about cross-platform protection and multi-layered encryption at VPNRice.