Self-Hosted AI on Kubernetes: Deploying vLLM at Production Grade Without Giving Your Data to OpenAI

There are two dominant stories about self-hosted large language models. The first comes from the homelab crowd: "It is easy. Install Ollama, pull a model, done." The second comes from hyperscaler marketing: "You need a $2M GPU cluster and a team of ML engineers before you can think about running your own inference." Both are wrong in the same direction. They describe the tails of the distribution and ignore the middle, which is where the actual decision lives for most engineering organizations.

The real question is narrower and more tractable: what does a production-grade self-hosted LLM deployment look like on Kubernetes, one that handles request routing, autoscaling, observability, model versioning, and audit logging correctly? Not a proof of concept, not a demo for the board, not a research environment. Production. The answer, in 2026, is that vLLM combined with standard Kubernetes primitives and a thin inference gateway gets you there at a cost point that makes cloud API pricing economically irrational beyond moderate volume.

This article walks through the architecture in enough detail that you can validate the decisions independently. There is math. There are manifests. There are honest trade-offs. If you are at a regulated company that has been told to "get AI" but cannot send patient records or transaction data to OpenAI, this is the engineering path.

The Two Lies About Self-Hosted AI: "It's Easy" and "It's Impossible"

The homelab narrative is seductive because Ollama genuinely is easy. You pull a four-bit quantized 7B model, it runs on a consumer GPU in under five minutes, and the inference quality on simple tasks is good enough to be impressive. What it does not give you: batched throughput, GPU memory management under concurrent load, model versioning with rollback, structured audit logging, authentication, rate limiting, or any of the operational guarantees a production system requires. Ollama is a personal tool. That is not a criticism. It is a scope statement.

The enterprise narrative fails in the opposite direction. Hyperscaler sales material routinely implies that self-hosted AI requires H100 clusters and dedicated ML Ops teams. This is true for frontier model training and cutting-edge research. It is not true for inference workloads running open-weight models at task-specific scale. Two A100 80GB cards in tensor-parallel configuration handle Llama-3.1-70B in eight-bit quantization with enough throughput for most enterprise internal tooling; a single A100 80GB suffices at four-bit quantization. The hardware exists, the software stack is mature, and financial-services data residency requirements and the compliance pressures described below are driving a measurable shift toward on-premises inference infrastructure.

Why the EU AI Act Makes Cloud LLM APIs a Compliance Liability for High-Risk Workloads

The EU AI Act's broad applicability provisions entered enforcement in August 2026. Article 12 imposes a specific technical requirement on high-risk AI systems: you must log inputs, outputs, model version, user identity, and risk classification in a manner that is auditable for the duration prescribed by sectoral law. For financial services under DORA, that retention period extends to ten years. For healthcare under the MDR intersection, it extends to the lifetime of the device.

The structural problem with cloud LLM APIs is not that vendors refuse to log. Several offer logging options. The problem is where the logs live. When you send a prompt to a cloud API, the input transits and is processed on infrastructure owned by a US-parent company subject to CLOUD Act jurisdiction. The completion and associated metadata are generated on that same infrastructure. Even if you receive a copy of the log, the source-of-truth record sits outside your legal perimeter. An EU supervisory authority conducting an Article 12 audit expects to verify the completeness and integrity of the log chain from inside your organization's control. A pointer to an API provider's log export is not that.

This is not a theoretical concern. The EU AI Act compliance architecture for a regulated healthcare SaaS we helped instrument required exactly this: an in-cluster audit store with append-only semantics, cryptographic log integrity verification, and an export path available only to authorized auditors. None of that is achievable when the inference endpoint is a third-party API. Self-hosted inference is not just a cost optimization at this point. For certain workload categories in regulated industries, it is the only architecture that satisfies the legal requirement.

Hardware Sizing Reality: VRAM, Quantization, and What You Actually Need

VRAM is the binding constraint in LLM inference, and the math is straightforward once you strip away the marketing. A model's minimum VRAM footprint is approximately (parameters * bytes_per_parameter) * 1.2, where the 1.2 factor accounts for the KV cache and runtime overhead at modest batch sizes. At full precision (float16), a 7B model needs roughly 14GB. At four-bit quantization (GGUF Q4_K_M or AWQ), that drops to approximately 4.5GB. At eight-bit, it lands around 7GB. For the full hardware buy-decision beneath the serving software — VRAM math per model size and quantization, consumer-versus-datacenter economics, and throughput-versus-latency capacity planning — see the dedicated guide to sizing the iron.

For a 70B model — which covers the quality range needed for most enterprise code assistance, document analysis, and reasoning tasks — the numbers are: float16 requires ~140GB VRAM, which means two A100 80GBs in tensor-parallel configuration or a single H100 NVL 94GB with headroom. Eight-bit quantization brings this to ~75GB of weight storage alone, leaving limited KV cache headroom on a single A100 80GB — two A100 80GBs in tensor-parallel is the comfortable configuration at this precision. Four-bit quantization reaches ~38GB for the weights, which fits on a single A6000 48GB or two consumer-grade GPUs with 24GB each, with meaningful KV cache budget remaining. The quality trade-off at four-bit on a well-quantized open-weight model is measurable but typically acceptable for enterprise task-specific workloads.

NVIDIA's RTX Spark, announced at Computex 2026, represents a significant inflection for the on-premises case: a Blackwell GPU with 128GB unified memory in a single device, enabling 120B+ parameter models locally without tensor parallelism. At the time of writing, this sits in the prosumer segment at a cost basis that makes single-device 100B+ inference viable for teams that previously assumed they needed a multi-GPU server. For Kubernetes deployment, it slots into the standard GPU device plugin model as a single allocatable unit.

#!/usr/bin/env bash
# Usage: ./vram-budget.sh <param_billions> <quant_bits> <max_concurrent> <ctx_len>
PARAMS=${1}   # e.g. 70
BITS=${2}     # e.g. 8  (weight quantization: 16=fp16, 8=int8, 4=int4/AWQ)
CONC=${3}     # e.g. 64
CTX=${4}      # e.g. 2048

# Weight memory: parameters * bytes-per-weight * 1.1 overhead factor
BYTES_PER_PARAM=$(echo "scale=4; $BITS / 8" | bc)
MODEL_GB=$(echo "scale=1; $PARAMS * $BYTES_PER_PARAM * 1.1" | bc)

# KV cache: 2 * num_heads * head_dim * ctx_len * bytes_per_kv_elem * num_layers
# NOTE: KV cache elements are stored in BF16 (2 bytes) by default in vLLM
# regardless of weight quantization — do NOT use BYTES_PER_PARAM here.
# Approximate for a 70B Llama-3 class model (80 layers, 64 heads, head_dim 128)
KV_BYTES_PER_ELEM=2  # BF16 always
KV_PER_TOKEN_GB=$(echo "scale=8; 2 * 64 * 128 * 80 * $KV_BYTES_PER_ELEM / 1073741824" | bc)
KV_TOTAL_GB=$(echo "scale=1; $KV_PER_TOKEN_GB * $CTX * $CONC" | bc)

TOTAL_GB=$(echo "scale=1; $MODEL_GB + $KV_TOTAL_GB" | bc)

echo "Model weights (${BITS}-bit):  ${MODEL_GB} GB"
echo "KV cache (BF16, ${CONC} concurrent x ${CTX} ctx):  ${KV_TOTAL_GB} GB"
echo "Total required:  ${TOTAL_GB} GB VRAM"

vLLM on Kubernetes: Architecture, GPU Resource Scheduling, and the Production Deployment Model

vLLM is the correct inference engine choice for production Kubernetes deployments as of 2026. Its PagedAttention algorithm manages GPU memory with the same logic a virtual memory system uses for RAM: it allocates KV cache in fixed-size pages, reclaims unused pages, and allows multiple requests to share cache pages when their prefixes match. The practical effect is 2-4x higher throughput versus a naive implementation at the same VRAM budget, with a preemption model that degrades gracefully under overload rather than OOM-killing.

The Kubernetes integration relies on the NVIDIA device plugin, which exposes GPUs as nvidia.com/gpu extended resources. Each vLLM pod requests whole GPU units; fractional GPU allocation is technically possible with MIG, time-slicing, or DRA on A100/H100 hardware and becomes worth the operational complexity precisely when a node runs mixed inference and embedding workloads rather than one dedicated model. The standard pattern is: one vLLM pod per GPU (or per GPU pair for tensor parallelism), scheduled via node selectors and tolerations onto a dedicated GPU node pool.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-70b
  namespace: ai-inference
  labels:
    app: vllm
    model: llama3-70b-instruct
    model-version: "3.1.0"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm
      model: llama3-70b-instruct
  template:
    metadata:
      labels:
        app: vllm
        model: llama3-70b-instruct
        model-version: "3.1.0"
      annotations:
        # Scraped by Prometheus via port 8000 /metrics
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      # Schedule only onto GPU-capable nodes in the inference pool
      nodeSelector:
        node-role: gpu-inference
        nvidia.com/gpu.product: "A100-SXM4-80GB"
      tolerations:
        - key: "gpu-inference"
          operator: "Exists"
          effect: "NoSchedule"
      # Pull model weights from shared PVC; never bundle in the image
      volumes:
        - name: model-weights
          persistentVolumeClaim:
            claimName: llm-model-store
            readOnly: true
        - name: hf-cache
          emptyDir: {}
      initContainers:
        # Verify model digest before each start — prevents silent weight corruption
        - name: verify-model-digest
          image: alpine:3.21
          command:
            - sh
            - -c
            - |
              EXPECTED="sha256:a1b2c3d4..."   # pin to known-good weights
              ACTUAL=$(sha256sum /models/llama3-70b/config.json | cut -d' ' -f1)
              if [ "sha256:$ACTUAL" != "$EXPECTED" ]; then
                echo "Model digest mismatch — aborting" >&2
                exit 1
              fi
          volumeMounts:
            - name: model-weights
              mountPath: /models
              readOnly: true
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.6.4
          args:
            - "--model=/models/llama3-70b"
            - "--served-model-name=llama3-70b-instruct"
            - "--tokenizer=/models/llama3-70b"
            - "--dtype=bfloat16"
            - "--quantization=awq"
            - "--max-model-len=8192"
            - "--gpu-memory-utilization=0.90"
            - "--enable-chunked-prefill"
            - "--max-num-batched-tokens=8192"
            - "--tensor-parallel-size=1"
            - "--port=8000"
            - "--trust-remote-code=false"
          ports:
            - containerPort: 8000
              name: http
          resources:
            requests:
              cpu: "8"
              memory: "32Gi"
              nvidia.com/gpu: "1"
            limits:
              cpu: "16"
              memory: "64Gi"
              nvidia.com/gpu: "1"
          volumeMounts:
            - name: model-weights
              mountPath: /models
              readOnly: true
            - name: hf-cache
              mountPath: /root/.cache/huggingface
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 90
            periodSeconds: 10
            failureThreshold: 6
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 30
            failureThreshold: 3
          env:
            - name: VLLM_LOGGING_LEVEL
              value: "WARNING"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://otel-collector.observability:4317"
            - name: OTEL_SERVICE_NAME
              value: "vllm-llama3-70b"

Several decisions in this manifest are worth calling out explicitly. The --trust-remote-code=false flag is not optional for regulated environments: it prevents arbitrary Python execution embedded in model configs from running on your inference nodes. The --quantization=awq flag selects AWQ quantization; vLLM auto-selects the Marlin-optimized AWQ kernel when the underlying hardware supports it, delivering better throughput than GPTQ at equivalent quality for the Llama-3 model family. The initContainer that verifies the model weight digest is the kind of paranoia that looks unnecessary until it catches a corrupted or tampered model weight in production — at which point it pays for itself many times over.

The model weights live on a PVC, not inside the container image. This is load-bearing. A 70B model at four-bit quantization is roughly 38GB of weight data; at eight-bit it is roughly 75GB. Either way, bundling weights in a Docker image defeats layer caching, makes pulls absurdly slow, and prevents sharing across deployments. The PVC pattern allows multiple pods to mount the same weights read-only, and lets you version the PVC independently of the inference server image.

Click to zoom

The Inference Gateway: Rate Limiting, Authentication, and RAG Integration

vLLM's built-in HTTP server exposes an OpenAI-compatible API. That is convenient for client compatibility but it is not a production-ready gateway. It has no authentication, no per-user or per-service rate limiting, no request enrichment for audit logging, and no circuit breaking. The inference gateway layer fills that gap.

Envoy proxy with the external authorization filter is the right choice here. It intercepts every request, validates a JWT issued by your identity provider, and passes user identity downstream as request headers that vLLM passes through to its log output. Rate limiting is implemented via Envoy's local rate limit filter with Redis-backed global limits for multi-replica setups. The gateway also handles context enrichment for RAG: for workloads that use retrieval-augmented generation, the gateway or a sidecar calls pgvector before forwarding to vLLM, injecting the retrieved chunks into the prompt template. This keeps RAG logic out of the vLLM server itself, which should stay stateless.

pgvector running inside the cluster as a PostgreSQL extension is the practical choice for regulated environments. Managed vector databases (Pinecone, Weaviate Cloud, etc.) reintroduce the same data-residency problem that motivated self-hosted inference in the first place: your retrieval corpus, which may contain sensitive documents, transits to a third-party SaaS. A PostgreSQL pod with pgvector satisfies the same retrieval workload for most enterprise RAG use cases, runs inside your perimeter, and benefits from the same backup and access-control infrastructure you already operate.

# envoy-ratelimit-config.yaml — mounted as ConfigMap into the Envoy gateway pod
static_resources:
  listeners:
    - name: ingress
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 8080
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ai_gateway
                http_filters:
                  # 1. JWT validation — extracts sub, email, roles
                  - name: envoy.filters.http.jwt_authn
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
                      providers:
                        internal_idp:
                          issuer: "https://auth.internal.example.com"
                          audiences:
                            - "ai-inference"
                          remote_jwks:
                            http_uri:
                              uri: "https://auth.internal.example.com/.well-known/jwks.json"
                              cluster: auth_cluster
                              timeout: 5s
                          forward_payload_header: "x-jwt-payload"
                  # 2. Per-user rate limit: 60 req/min; global: 500 req/min
                  - name: envoy.filters.http.local_ratelimit
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
                      stat_prefix: ai_ratelimit
                      token_bucket:
                        max_tokens: 60
                        tokens_per_fill: 60
                        fill_interval: 60s
                      filter_enabled:
                        default_value:
                          numerator: 100
                          denominator: HUNDRED
                      response_headers_to_add:
                        - header:
                            key: X-RateLimit-Limit
                            value: "60"
                  # 3. Audit enrichment — injects request_id for log correlation
                  - name: envoy.filters.http.lua
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
                      default_source_code:
                        inline_string: |
                          function envoy_on_request(handle)
                            local req_id = handle:headers():get("x-request-id")
                            handle:headers():add("x-audit-request-id", req_id)
                            handle:headers():add("x-audit-ts", tostring(os.time()))
                          end
                  - name: envoy.filters.http.router

Model Versioning and A/B Testing Without the MLflow Complexity Tax

Model versioning is one of the areas where teams reach for MLflow and end up running a complex experiment-tracking platform for a problem that is really a Kubernetes deployment problem. The models are weights on a PVC. The versions are labeled PVCs and Deployment manifests. The A/B split is a Kubernetes Service with traffic weights — exactly the same pattern you would use for a canary deployment of any application.

The concrete pattern: each model version gets its own Deployment with a model-version label. The Service selector matches app: vllm without pinning to a specific version. Traffic weighting between the current stable model and a candidate is handled by the inference gateway routing rules, which can split by percentage or by a specific request header (useful for internal dogfooding before general rollout). The vLLM --served-model-name argument controls what name the model advertises to clients; using a stable alias (llama3-70b-instruct) decoupled from the specific weight version means clients do not need to change when you promote a new checkpoint.

Rollback is a kubectl rollout undo or a change to the gateway routing rule. There is no MLflow state to clean up, no experiment database to maintain, no pipeline artifact store to manage. For teams that need richer experiment tracking for model evaluation, those tools remain valuable. For production routing decisions, they are overkill. Optionality here means using the simplest infrastructure that gives you the control you need, rather than importing an entire MLOps platform because the concept sounds sophisticated.

Observability: What to Instrument, What to Log, and Why the Audit Trail Matters

Observability for LLM inference has three distinct layers: operational metrics (is the system healthy?), performance metrics (is it fast enough?), and audit logs (what exactly ran, for whom, and what did it produce?). The first two are standard OpenTelemetry work. The third is distinct and compliance-critical.

vLLM exposes a Prometheus-compatible /metrics endpoint with the key operational signals: vllm:num_requests_running (current active batch), vllm:gpu_cache_usage_perc (KV cache utilization — if this reaches 100% you are blocking or preempting), vllm:time_to_first_token_seconds (histogram), and vllm:request_success_total / vllm:request_failure_total. These wire directly into a standard Prometheus scrape configuration via the pod annotations shown in the Deployment manifest above. Loki captures structured logs from both vLLM and the gateway.

The audit log is a different beast. Every inference request to a high-risk AI system under EU AI Act Article 12 requires a durable, integrity-protected record that includes: the input prompt (or a cryptographic digest if the prompt is too large), the model output, the model version and configuration hash, the user identity (from the JWT sub claim), the risk classification of the workload, and the timestamp with microsecond precision. This record must be immutable, retained for the period prescribed by sectoral law, and accessible only to authorized auditors.

The implementation is a structured event emitted by the gateway's Lua enrichment filter, collected by OpenTelemetry, and written to an in-cluster audit store configured with append-only semantics. The audit store can be as simple as a PostgreSQL instance with an append-only table enforced by row-level security, or a Loki instance with immutability rules enabled. What it cannot be is a third-party SaaS or a cloud-managed service, because the legal perimeter requirement is absolute: the record must stay inside your organization's infrastructure for the retention period. See the sovereignty thesis for the broader framing of why infrastructure ownership is the prerequisite for this class of compliance.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

processors:
  # Extract audit-relevant attributes from inference spans
  attributes/audit:
    actions:
      - key: audit.model_version
        from_attribute: model.version
        action: insert
      - key: audit.user_id
        from_attribute: http.request.header.x-jwt-sub
        action: insert
      - key: audit.risk_class
        from_attribute: http.request.header.x-risk-class
        action: insert
      - key: audit.request_id
        from_attribute: http.request.header.x-audit-request-id
        action: insert
  # Batch for efficiency
  batch:
    timeout: 1s
    send_batch_size: 512

exporters:
  # Operational metrics → Prometheus
  prometheus:
    endpoint: "0.0.0.0:9090"
    namespace: "vllm"

  # Logs → Loki (operational)
  loki:
    endpoint: "http://loki.observability:3100/loki/api/v1/push"

  # Audit trail → append-only PostgreSQL store inside the legal perimeter
  otlphttp/audit:
    endpoint: "http://audit-store.ai-compliance:8080/v1/logs"
    headers:
      x-audit-sink: "true"
    tls:
      insecure: false
      ca_file: /etc/ssl/audit-store-ca.crt

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes/audit, batch]
      exporters: [otlphttp/audit]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [attributes/audit, batch]
      exporters: [loki, otlphttp/audit]

Click to zoom

Connecting to the EU AI Act Article 12 Audit Logging Requirement

Article 12 of the EU AI Act requires that providers of high-risk AI systems "ensure that the AI system is designed and developed with capabilities enabling the automatic recording of events relevant for identifying risks to health or safety or fundamental rights." The recital clarifies that these logs must support post-hoc investigation, meaning the record must be complete enough to reconstruct what the model was asked and what it answered, in context, at a later date.

The practical implementation requirements that follow from this language are: input logging (either full prompt or a verifiable digest plus a stored copy), output logging (the full completion, since the risk may be in the model's answer, not the question), model version pinning (so the log references a specific, reproducible model configuration rather than just a model name), user identity binding (the log must connect to a natural or organizational person who can be contacted for investigation), and retention with integrity guarantees (the log cannot be modified or deleted during the retention period).

The architecture described here satisfies all five requirements within a single legal perimeter. The OTel pipeline captures input and output at the gateway layer, where the full prompt and response are available before any filtering or truncation. The model-version label on the Deployment manifest, combined with a configuration hash in the audit event, pins the model configuration. The JWT sub claim provides user identity. The append-only audit store with restricted delete permissions satisfies the integrity requirement. And critically, none of this touches infrastructure outside the organization's control.

The Cost Crossover: At What Volume Does Self-Hosted Beat Cloud APIs

The economic argument for self-hosted inference is straightforward once you work the numbers honestly. Cloud API pricing is per-token with no fixed cost. On-premises infrastructure has high fixed cost (hardware capital or colo lease) and near-zero marginal cost per request. The crossover point depends on your request volume, your model quality requirements, and your amortization period.

For this analysis, the baseline is a Llama-3.1-8B-Instruct class model — competitive quality with GPT-4o-mini on most enterprise task-specific workloads, which is where the volume actually lives. Cloud API cost is $0.30 per million tokens blended (input + output at current GPT-4o-mini pricing). Hyperscaler GPU is $2.00/hr for an A100 on-demand instance. Colocation amortized is $700/month for a server carrying a single A100 (3-year hardware amortization on a $25K server at $500/mo colo).

At 10,000 requests per month averaging 1,000 tokens per request, cloud API costs $3. Colo fixed cost is $700 regardless of volume — obviously not justified. At 100,000 requests per month (100M tokens), cloud API is $30. Colo is still $700. At 1,000,000 requests per month, cloud API is $300 and colo is $700 plus about $24 in power — not yet a crossover, but close. At 10,000,000 requests per month, cloud API is $3,000 and colo is $80 all-in. The hyperscaler GPU instance is even more expensive than cloud API at that volume because you pay for idle time.

The crossover between cloud API and amortized colo lands at approximately 2.3 million requests per month at the token volumes above — around $700/mo in API cost. For an internal tooling workload that serves 50 engineers each making 100 AI-assisted queries per workday, that is roughly 100,000 requests per month, well below crossover. For a customer-facing product with 10,000 active users making 10 AI requests per day, that is 3 million requests per month, firmly in colo-wins territory. The break-even is not particularly high.

Click to zoom

This is also where the optionality argument has concrete teeth. Once you own the inference hardware and are running at scale, switching from Llama-3.1 to a future open-weight model is a PVC swap and a Deployment update. Switching cloud API providers requires contract negotiation, API adapter changes, and potential re-evaluation of data processing agreements. Owning the inference layer is owning the exit ramp from every future foundation model negotiation.

What Production-Grade Actually Requires: The Non-Negotiable List

"Production-grade" is often used as a vague quality signal. For self-hosted LLM inference, it has a specific meaning. A deployment is production-grade when it can handle all of the following without human intervention: a single GPU pod crash (the other replica keeps serving while Kubernetes reschedules), a model weight corruption (the initContainer catches it and blocks the pod start before it serves bad output), a traffic spike beyond the rate limit (the gateway returns 429 rather than OOM-killing the inference pod), a GPU memory exhaustion event (vLLM's preemption model degrades gracefully to sequential processing rather than crashing), and an audit request from a regulator (the export path produces a complete, integrity-verified log without requiring any operational changes to the running system).

None of these requirements are exotic. They are the same properties you would require of any production database or API service. The difference is that the LLM inference stack is younger and the defaults are less hardened. The Deployment manifest and gateway configuration in this article encode most of these requirements directly. The audit log architecture handles the compliance dimension. What remains is operational maturity: runbooks, alerts on the vLLM metrics, load testing before production, and GPU node monitoring for memory bandwidth saturation.

The capabilities we bring to this problem include GPU infrastructure design, vLLM production configuration, inference gateway integration, audit logging architecture, and EU AI Act compliance validation. If you are at a regulated company building toward sovereign AI infrastructure, the path is tractable. It requires engineering rigor, not a $2M cluster. The Kubernetes primitives you already operate are the foundation. vLLM is the inference engine. The audit log architecture is the compliance proof. Everything else is configuration.