Production Architecture for Sovereign On-Premise LLM Inference & Air-Gapped RAG

Prototyping generative workflows against commercial multi-tenant LLM APIs takes hours; clearing those same external endpoints through central-bank operational risk committees, defense accreditation boards, or national healthcare data-protection regulators is frequently impossible. When every inference prompt contains raw credit dossiers, clinical histories, subscriber call-detail metadata, or proprietary source repositories, routing cleartext tokens across jurisdictional borders to opaque third-party inference black boxes violates fundamental statutory residency mandates.

Transitioning from metered cloud APIs to sovereign, air-gapped GPU infrastructure requires rigorous systems engineering across High-Bandwidth Memory (HBM) allocation, interconnect topology, and cryptographic retrieval access control. This blueprint details the production architecture our team deploys for 70B to 405B parameter foundation models operating inside zero-egress enterprise enclaves.

1. Why Regulated Institutions Exit Third-Party Token APIs

Across commercial banking, defense agencies, healthcare networks, and Tier-1 telecommunications carriers, four structural constraints drive the migration onto dedicated sovereign GPU clusters:

• Zero-Egress Statutory Data Residency: Even when commercial API vendors offer contractual "zero-data-retention" addenda, TLS termination occurs outside the institution's physical perimeter, and transient prompt buffers traverse shared host RAM, crash-dump pipelines, or automated abuse classifiers. Air-gapped clusters guarantee mathematically verifiable zero WAN egress at the switch and Kubernetes CNI layers.
• Cryptographic Checkpoint Immutability: Regulated underwriting, AML triage, and clinical summarization pipelines require deterministic auditability. Upstream API providers routinely swap quantized sub-versions or safety post-filters behind stable endpoint aliases, silently invalidating regression benchmarks. Sovereign deployments pin immutable SHA-256 SafeTensors digests indefinitely.
• Deterministic Tail Latency (TTFT & ITL): Multi-tenant API gateways enforce token-per-minute quotas (HTTP 429) and exhibit multi-second Time-To-First-Token (TTFT) variance during global traffic peaks. Dedicated GPU pools reserve guaranteed HBM capacity for mission-critical queues.
• Amortized TCO Crossover at Sustained Load: Once enterprise pipelines exceed ~35–45% sustained concurrency (roughly 1.2 billion tokens/day across batch document extraction and interactive copilot sessions), owned or single-tenant bare-metal 8× H100/H200 nodes amortized over 36 months reduce effective inference cost per million tokens by 60–75%.

2. Exact VRAM & KV-Cache Capacity Mathematics

Under-sizing GPU memory is the most expensive procurement mistake in private AI infrastructure. Unlike training, inference memory consumption is split between a fixed static footprint (model weights plus CUDA/NCCL runtime workspaces) and an elastic dynamic footprint (the autoregressive Key-Value attention cache) that scales linearly with both sequence length and batch concurrency.

Static Parameter Footprint and Quantization Regimes

For a dense Transformer of N_params parameters quantized to B_w bytes per parameter, static HBM consumption including ~8% overhead for PyTorch CUDA contexts, NCCL ring communication buffers, and temporary prefill activation tensors is:

M_weights = N_params × B_w × 1.08

Taking Llama-3.3-70B-Instruct (70.6B parameters, L = 80 layers, hidden dimension d_model = 8192) as our canonical enterprise baseline:

• BF16 / FP16 Unquantized (2.0 bytes/param): 141.2 GB raw weights → 152.5 GB with runtime workspace. Fits across 2× 80 GB GPUs with almost zero room left for KV-cache, forcing a minimum 4× 80 GB topology.
• Native Hopper/Blackwell FP8 (W8A8 E4M3, 1.0 byte/param): 70.6 GB raw weights → 76.2 GB total static footprint, preserving >99.4% of BF16 benchmark accuracy via hardware FP8 Tensor Cores while freeing over half of a 2× 80 GB NVLink domain for active KV-cache.
• INT4 AWQ (Activation-Aware Weight Quantization, ~0.55 bytes/param including FP16 group-128 scales): ~38.8 GB static footprint, enabling high-concurrency deployment on cost-efficient 2× L40S (96 GB) PCIe pairs where slight perplexity trade-offs on dense math/code tasks are acceptable.

Grouped-Query Attention (GQA) KV-Cache & PagedAttention

Modern 70B+ architectures replace classic Multi-Head Attention (MHA) with Grouped-Query Attention (GQA), sharing a reduced number of Key and Value heads (n_kv_heads = 8) across n_heads = 64 Query heads (with head dimension d_head = 128). This compresses the KV-cache by an exact factor of 8×. The total KV-cache footprint across C_batch concurrent streams of context length S_ctx stored at precision B_kv bytes is:

M_KV = 2 (K & V) × L × n_kv_heads × d_head × B_kv × S_ctx × C_batch

# Worked Example: Llama-3.3-70B (L=80, n_kv_heads=8, d_head=128)
Per-Token Elements = 2 × 80 × 8 × 128 = 163,840 values/token
• FP16 KV-Cache (B_kv = 2): 327,680 bytes ≈ 320.0 KB / token
• FP8  KV-Cache (B_kv = 1): 163,840 bytes ≈ 160.0 KB / token

# At 32 concurrent RAG streams × 32,768 tokens (1,048,576 active tokens):
• FP16 KV Pool = 1,048,576 × 320 KB ≈ 320.0 GB HBM
• FP8  KV Pool = 1,048,576 × 160 KB ≈ 160.0 GB HBM

Notice that at 32k context across 32 concurrent analysts, the dynamic KV-cache (160–320 GB) dwarfs the 70B model weights themselves (76.2 GB in FP8). You can model custom context-window, batch-size, and accelerator combinations interactively in our Sovereign LLM VRAM, KV-Cache & GPU Cluster Sizer.

Legacy serving engines pre-allocated contiguous HBM slabs sized to max_model_len for every incoming request, wasting 60% to 80% of VRAM to internal over-reservation and external fragmentation. Modern production engines (vLLM and TensorRT-LLM) implement PagedAttention, partitioning HBM into fixed 16-token physical KV blocks mapped via an OS-style virtual block table. PagedAttention cuts KV fragmentation waste to under 4% and enables zero-copy Automatic Prefix Caching (APC): shared 4,000-token enterprise system instructions or regulatory policy rubrics hash to identical physical KV blocks across hundreds of users, eliminating redundant prefill compute.

Prefill (Compute-Bound) vs. Decode (Memory-Bandwidth-Bound)

Sizing accelerators solely by VRAM capacity ignores the two distinct physical regimes of LLM execution. On an NVIDIA H100 SXM5 (1,979 TFLOPS dense FP8 compute vs. 3,350 GB/s HBM3 bandwidth), the roofline arithmetic-intensity ridge point sits near ~590 FLOPs/byte:

1. Prefill Phase (Compute-Bound): Ingesting a 16,384-token RAG prompt executes large parallel matrix-matrix multiplications (GEMM). Arithmetic intensity far exceeds 590 FLOPs/byte, saturating Tensor Core TFLOPS. Time-To-First-Token (TTFT) scales with raw compute throughput.
2. Autoregressive Decode Phase (Memory-Bandwidth-Bound): Emitting each subsequent output token executes narrow matrix-vector products (GEMV at modest batch sizes), forcing every GPU in the Tensor-Parallel group to stream the entire 70.6 GB FP8 parameter set from HBM into SRAM for every single token step. Inter-Token Latency (ITL) is therefore governed strictly by aggregate HBM bandwidth and effective Model Bandwidth Utilization (MBU ≈ 60–68%).

3. Serving Stack: NVLink Tensor Parallelism vs. RoCEv2 Pipeline Parallelism

Partitioning a large model across multiple GPUs requires matching the parallelism strategy to the physical interconnect hierarchy. Tensor Parallelism (TP) shards individual attention and MLP weight matrices column-wise and row-wise across GPUs, requiring two synchronous AllReduce collective barriers per Transformer layer—meaning 160 AllReduce synchronization points for every single generated token on an 80-layer 70B model.

Inside a single 8-GPU HGX chassis equipped with fourth-generation NVLink and NVSwitch (900 GB/s bidirectional bandwidth per GPU), AllReduce latency takes mere microseconds, making TP=2, TP=4, and TP=8 highly efficient (always selecting power-of-two degrees that divide n_kv_heads = 8 evenly). Conversely, stretching Tensor Parallelism across separate physical servers over Ethernet or InfiniBand introduces network serialization stalls at every layer, collapsing GPU utilization below 25%. When a model exceeds a single chassis (such as Llama-3.1-405B in BF16 across two 8× H100 servers), keep TP=8 strictly intra-node over NVSwitch and use Pipeline Parallelism (PP=2) inter-node over lossless 400GbE RoCEv2 or NDR InfiniBand, which transmits only compact inter-layer activation tensors at stage boundaries.

Continuous Batching, Chunked Prefill, and Air-Gapped Kubernetes Deployment

To prevent a newly submitted 48,000-token legal contract prefill from monopolizing GPU SMs and freezing token generation for 30 concurrent interactive users, we enable iteration-level continuous batching paired with chunked prefill (--enable-chunked-prefill --max-num-batched-tokens 8192). Long prompts are sliced into bounded token chunks and piggybacked onto ongoing decode steps, holding p95 Inter-Token Latency reliably below 35 ms.

Below is our hardened Kubernetes StatefulSet and zero-egress NetworkPolicy for serving a 70B FP8 checkpoint on a 4-GPU NVLink slice with memory-backed /dev/shm for NCCL shared-memory rings and strict offline enforcement:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vllm-llama33-70b-fp8
  namespace: sovereign-ai
spec:
  serviceName: vllm-llama33-70b
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: vllm-llama33-70b
  template:
    metadata:
      labels:
        app.kubernetes.io/name: vllm-llama33-70b
    spec:
      runtimeClassName: nvidia
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile:
          type: RuntimeDefault
      initContainers:
        - name: verify-cosign-digest
          image: registry.enclave.local/secops/cosign-verifier:v2.4.1
          command:
            - /bin/sh
            - -c
            - |
              cosign verify --key /etc/pki/model-signing.pub \
                --offline registry.enclave.local/models/llama-3.3-70b-fp8@sha256:4e81d7a0c29b31f5e6a87d921c440b8f91e23c674a98f120b67d5e43a091c8b2
          volumeMounts:
            - name: signing-pubkey
              mountPath: /etc/pki
              readOnly: true
      containers:
        - name: vllm-engine
          image: registry.enclave.local/ai/vllm-openai:v0.7.3-cuda12.4
          env:
            - name: HF_HUB_OFFLINE
              value: "1"
            - name: TRANSFORMERS_OFFLINE
              value: "1"
            - name: VLLM_NO_USAGE_STATS
              value: "1"
            - name: NCCL_P2P_LEVEL
              value: "NVL"
          args:
            - "--model=/models/llama-3.3-70b-instruct-fp8"
            - "--served-model-name=sovereign-70b"
            - "--load-format=safetensors"
            - "--tensor-parallel-size=4"
            - "--quantization=fp8"
            - "--kv-cache-dtype=fp8"
            - "--gpu-memory-utilization=0.90"
            - "--max-model-len=65536"
            - "--enable-prefix-caching"
            - "--enable-chunked-prefill"
            - "--max-num-batched-tokens=8192"
            - "--disable-log-requests"
          ports:
            - name: http-internal
              containerPort: 8000
          resources:
            requests:
              cpu: "16"
              memory: "128Gi"
              nvidia.com/gpu: "4"
            limits:
              cpu: "32"
              memory: "128Gi"
              nvidia.com/gpu: "4"
          volumeMounts:
            - name: dshm
              mountPath: /dev/shm
            - name: model-weights
              mountPath: /models
              readOnly: true
      volumes:
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 64Gi
        - name: model-weights
          persistentVolumeClaim:
            claimName: pvc-nvme-models-ro
        - name: signing-pubkey
          configMap:
            name: enclave-model-signing-key
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vllm-zero-egress
  namespace: sovereign-ai
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: vllm-llama33-70b
  policyTypes: ["Ingress", "Egress"]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: rag-orchestrator
      ports:
        - protocol: TCP
          port: 8000
  egress: [] # Strict hardware-enforced zero outbound network access

4. Air-Gapped RAG Security: Vector RBAC, Prompt Blast-Radius & Artifact Integrity

Deploying an air-gapped LLM solves external leakage but introduces a severe internal threat vector if Retrieval-Augmented Generation (RAG) indexes are built naively: cross-role privilege escalation. When engineering teams ingest SharePoint libraries, Confluence spaces, M&A data rooms, and core banking memos into a single unfiltered vector collection, any authenticated junior employee can query semantic neighbors that surface executive board minutes or confidential payroll figures.

Cryptographic ACL Propagation into Pre-Filtered Vector Queries

Access control cannot be delegated to LLM prompt instructions or post-retrieval application heuristics (post-filtering top-k results causes empty result sets whenever a user lacks permission for the globally closest vectors). Instead, our ingestion pipeline extracts explicit Identity Provider group SIDs and security classification tiers from source repositories, indexing them directly into Qdrant keyword payload indexes (or PostgreSQL pgvector tables protected by native Row-Level Security).

At query time, the RAG orchestrator extracts verified group memberships directly from the caller's SPIFFE mTLS peer certificate and OIDC JWT (never from mutable request body fields—see Zero-Trust Infrastructure Governance) and injects an immutable HNSW pre-filter:

# Mandatory Server-Side RBAC Vector Pre-Filtering (Python / Qdrant Internal mTLS Client)
from qdrant_client import QdrantClient, models

def retrieve_authorized_chunks(
    qdrant: QdrantClient,
    query_dense_vec: list[float],
    verified_jwt_claims: dict,
    top_k: int = 8,
) -> list[dict]:
    # Group principals & clearance extracted strictly from signature-verified OIDC JWT
    caller_tenant = verified_jwt_claims["tenant_id"]
    caller_groups = verified_jwt_claims["groups"]          # e.g. ["risk-emea", "corp-credit-l2"]
    max_clearance = int(verified_jwt_claims["clearance"])  # e.g. 2 (Confidential)

    rbac_filter = models.Filter(
        must=[
            models.FieldCondition(
                key="tenant_id",
                match=models.MatchValue(value=caller_tenant),
            ),
            models.FieldCondition(
                key="acl_groups",
                match=models.MatchAny(any=caller_groups),
            ),
            models.FieldCondition(
                key="classification_level",
                range=models.Range(lte=max_clearance),
            ),
        ]
    )

    # Filterable HNSW evaluates similarity ONLY across ACL-permitted graph nodes
    hits = qdrant.query_points(
        collection_name="regulated_corpus_v3",
        query=query_dense_vec,
        query_filter=rbac_filter,
        limit=top_k,
        with_payload=["doc_uri", "chunk_sha256", "effective_date", "text"],
        search_params=models.SearchParams(hnsw_ef=128, exact=False),
    ).points

    return [pt.payload for pt in hits]

Prompt-Injection Blast-Radius Containment

Even within authorized documents, untrusted external text (such as ingested customer emails, vendor PDFs, or SWIFT remittance free-text fields) may harbor indirect prompt-injection payloads designed to hijack tool-calling agents. Because probabilistic guardrail classifiers can always be bypassed by adversarial encodings, we contain blast radius deterministically at the systems boundary:

1. Zero Ambient Credentials in Inference Pods: As shown in our Kubernetes manifest (automountServiceAccountToken: false and empty egress rules), the LLM process itself has zero network path to databases, internal APIs, or the Kubernetes API server.
2. Finite-State Grammar Constrained Decoding: Tool invocations are enforced via vLLM's guided JSON-schema finite-state machine (guided_json), stripping arbitrary free-form shell or SQL string generation.
3. Scoped Capability Tokens & Dual Control: Downstream tool executors run under the human caller's short-lived OAuth2 On-Behalf-Of (OBO) token—never a privileged service account—and any state-mutating financial operation (such as posting an adjustment entry in our Double-Entry Financial Ledger) stages an immutable draft requiring out-of-band cryptographic approval by a second authorized officer.

5. MIG Partitioning, RoCEv2 Tuning & Production GPU SLOs

A complete RAG pipeline runs more than the primary 70B or 405B generator: it also requires dense/sparse hybrid embedding models (e.g., BGE-M3), cross-encoder rerankers (BGE-Reranker-v2-M3), and OCR layout parsers. Co-locating those auxiliary models on the same physical GPUs as the main TP=4 vLLM engine via naive CUDA time-slicing causes L2 cache thrashing and erratic AllReduce tail jitter. Instead, we partition a dedicated auxiliary H100/A100 accelerator using hardware NVIDIA Multi-Instance GPU (MIG) slices—isolating SMs, HBM memory controllers, and crossbar paths—while tuning host Linux sysctls for lossless GPUDirect RDMA over RoCEv2 on multi-node clusters:

# 1. Partition GPU 0 on Auxiliary Worker Node into Isolated Hardware MIG Instances
nvidia-smi -i 0 -mig 1
# Create two 2g.20gb slices (Embedder + Reranker) and three 1g.10gb slices (OCR / Guardrails)
nvidia-smi mig -i 0 -cgi 14,14,19,19,19 -C

# 2. /etc/sysctl.d/99-sovereign-rocev2-gpu.conf — Lossless 400GbE RoCEv2 & PCIe/NUMA Tuning
# Prevent ARP flux across multi-homed ConnectX-7 RDMA NICs
net.ipv4.conf.all.arp_ignore = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.all.rp_filter = 0

# Disable automatic NUMA page migration jitter on pinned GPU/RDMA host buffers
kernel.numa_balancing = 0
vm.swappiness = 0
vm.max_map_count = 1048576

# Enable ECN marking for DCQCN congestion control alongside switch PFC on Priority 3
net.ipv4.tcp_ecn = 1

Finally, production observability pairs NVIDIA DCGM Exporter hardware telemetry with vLLM's Prometheus endpoint. We configure automated node cordoning via Kubernetes Node Problem Detector whenever DCGM detects uncorrectable double-bit ECC memory errors (DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0) or fatal NVLink Xid faults (Xid 74, 79, 94), draining inference traffic cleanly across surviving replicas—or across regional GPU clusters federated via the patterns in Designing Deterministic Active-Active Kubernetes Clusters. Service-Level Objectives alert on vllm:gpu_cache_usage_perc > 0.88 and rate(vllm:num_preemptions_total[5m]) > 0, signaling KV-cache saturation well before user-visible latency degrades.

Related Engineering Resources