The Sovereign AI Gateway: LiteLLM, MCP, and Agent Data Residency

A conversation repeats in every engineering organisation that has taken AI seriously. It starts well: we cannot send customer data to a third-party model, so we will host our own. The team stands up vLLM on GPUs it controls, gets a Llama or Qwen model answering on an internal endpoint, and the residency question looks answered. Then the second wave arrives — agents. Something that does not merely answer but acts: reads a ticket, queries a database, opens a pull request. The agent needs tools, so somebody wires it to an MCP server, and the diagram everyone was proud of quietly grows an arrow that leaves the building. The model is on-prem. The tool-call is not. Nobody drew that arrow, so nobody argued about it.

This article is about the layer that governs both — the control plane above the model. It is deliberately not about serving a model well; we covered that in self-hosted LLM inference on Kubernetes with vLLM, and everything there still holds. This is what sits on top once there is more than one model, more than one team, and agents making tool-calls on your behalf: routing across backends, policy in one place, masking what must never leave, an audit trail a regulator will accept. That layer now has a competent open-source answer, and its shape is worth getting right early — retrofitting is expensive.

Model residency is not agent residency

Start with the distinction the rest of this rests on. Self-hosting inference makes a claim about one flow: the prompt goes to a GPU you own, the tokens come back, no third party saw either. Worth making, worth auditing — but it is a claim about the completion, and an agent is not a completion. An agent is a loop: reason, choose a tool, call it with arguments, read the result, reason again. Each tool-call is its own network event, with its own destination, jurisdiction, and payload.

The payload is what gets missed. People picture a tool-call as a control message — a small structured instruction, plumbing. It is not. The arguments are data, usually your most specific data, because the point of the call is to act on something real. search_customers(email: "…") carries an identifier; create_ticket(body: "…") carries whatever the customer wrote; query_db(sql: "…") carries your schema and often your filters. If the MCP server on the other end is hosted by someone else, you have exported all of it — through a path your model-residency architecture never contemplated, because it was drawn as an arrow to a tool rather than to a vendor.

MCP is worth understanding rather than dismissing, because it won. Anthropic donated the Model Context Protocol to the Linux Foundation's Agentic AI Foundation on 9 December 2025 — a directed fund co-founded with Block and OpenAI, with Google, Microsoft, AWS, Cloudflare, and Bloomberg among the named supporters. That announcement put the ecosystem at more than 10,000 active public MCP servers and 97M+ monthly SDK downloads across Python and TypeScript. A protocol with that adoption and governance is not a fad to wait out. It is the integration surface your agents will use — which makes it the surface needing a boundary.

What a gateway actually is

The word gateway is overloaded, so be precise. The vLLM article describes an inference gateway in the traffic sense — an Envoy in front of a model server doing auth, rate-limiting, and L7 routing to pods. A data-plane concern; it belongs in the cluster. This is a control plane: a service owning the semantics of an AI call across every model and tool, and therefore the one place policy is decided rather than merely enforced.

LiteLLM is the open-source implementation most teams land on — a proxy presenting one OpenAI-compatible endpoint that speaks to 100+ LLM APIs behind it (self-hosted vLLM, Bedrock, Azure, Vertex, Anthropic), with cost tracking, guardrails, load-balancing, and logging built in. The core is free to self-host as a container; an enterprise tier adds SSO, RBAC, and audit-log features under a commercial licence. Version v1.92.0 shipped on 12 July 2026, at a cadence fast enough that pinning a digest matters. Its relevance is structural: the gateway is the only component that sees every AI request your organisation makes, which makes it the only place a policy can be true rather than aspirational.

Click to zoom

The configuration is plain YAML — a virtue, because it reviews, diffs, and lives in Git like code. Pointing the gateway at your own vLLM deployments uses the hosted_vllm/ prefix, and deployments sharing a model_name become a load-balanced pool automatically.

# config.yaml — models are declarations, not code paths
model_list:
  # Two vLLM replicas, same weights → one logical model.
  - model_name: internal-chat
    litellm_params:
      model: hosted_vllm/meta-llama/Llama-3.3-70B-Instruct
      api_base: http://vllm-llama.ai-serving.svc.cluster.local:8000/v1
      api_key: os.environ/VLLM_INTERNAL_KEY
      rpm: 900
  - model_name: internal-chat
    litellm_params:
      model: hosted_vllm/meta-llama/Llama-3.3-70B-Instruct
      api_base: http://vllm-llama-b.ai-serving.svc.cluster.local:8000/v1
      api_key: os.environ/VLLM_INTERNAL_KEY
      rpm: 900

  - model_name: internal-code
    litellm_params:
      model: hosted_vllm/Qwen/Qwen2.5-Coder-32B-Instruct
      api_base: http://vllm-qwen.ai-serving.svc.cluster.local:8000/v1

router_settings:
  routing_strategy: usage-based-routing-v2   # simple-shuffle | least-busy |
                                            # latency-based-routing | cost-based-routing
  num_retries: 3
  timeout: 60
  cooldown_time: 30
  allowed_fails: 3
  enable_pre_call_checks: true

Notice what this buys beyond convenience. Callers no longer name infrastructure — they name a capability. internal-chat is an interface; the deployment behind it is an implementation detail you change without touching a client. What frees you to swap providers also frees you to swap your own GPUs.

Virtual keys: the accountability layer

The second thing the gateway centralises is identity, and this is where homegrown proxies quietly fail. Without one, every team that wants AI needs a provider credential, and it ends up in a .env, a CI secret, a notebook, and eventually a Slack message. You cannot revoke what you cannot enumerate, nor attribute a spend spike or a data incident to a caller sharing a key with eleven others.

The gateway inverts this. Upstream provider credentials exist exactly once, injected from your secret store — where secrets management with External Secrets and Vault stops being generic advice and becomes load-bearing, because the gateway is a concentrated credential target whose keys should be short-lived and rotated on your schedule. Downstream, it issues its own virtual keys: one per team, service, or environment, each with a budget, rate limit, and model allow-list. Revoking access becomes an API call rather than an archaeology project, and every request carries an identity into the audit log.

Guardrails: masking before the boundary, not after

Most organisations will not reach zero external calls, and pretending otherwise produces architectures people route around: there is always a frontier model better at one task, an inherited integration, a team with a deadline. The sovereign answer is not prohibition — it is that the exception is explicit, gated, logged, and stripped. LiteLLM's guardrail hooks run at defined points in the request lifecycle, and the placement is the whole design decision. Presidio is the common open-source detector, self-hosted alongside the proxy so the analysis does not leave either — worth checking in any masking product you evaluate, because a PII detector that phones home is a PII exfiltrator with good branding.

guardrails:
  - guardrail_name: "presidio-mask-outbound"
    litellm_params:
      guardrail: presidio
      mode: "pre_call"        # pre_call | post_call | during_call | logging_only
      pii_entities_config:
        EMAIL_ADDRESS: "MASK"
        CREDIT_CARD: "BLOCK"  # MASK degrades the prompt; BLOCK refuses the call
        IBAN_CODE: "BLOCK"
        PHONE_NUMBER: "MASK"
      default_on: true

# Analyzer + anonymizer run in-cluster; the detection itself never leaves.
# PRESIDIO_ANALYZER_API_BASE=http://presidio-analyzer.ai-gw.svc:5002
# PRESIDIO_ANONYMIZER_API_BASE=http://presidio-anonymizer.ai-gw.svc:5001

Be honest about the limits. Statistical PII detection is recall-limited: it will not catch an internal customer identifier that looks like any other integer, nor context identifying only in combination. A guardrail is a strong second line and a weak first one. The first line is that the request never needed to leave — a routing decision, made by the same gateway, one layer up. This is the claim we made for regulated healthcare in HIPAA-compliant LLMs: on-prem PHI inference without the BAA gap: the de-identification boundary belongs at the gateway, not the model server (too late) and not in every application (too many places). What changes here is scope — build it once and it serves your HIPAA, GDPR, and DPDP cases through one code path, instead of three vertical implementations that drift apart within a year.

The MCP gateway: governing the tool-call

Now the genuinely new part, and the reason this article exists. LiteLLM's MCP gateway gives you one fixed endpoint for all MCP tools, with access controlled by key, team, or organisation. It is the model gateway's idea applied to a different verb: instead of each agent holding its own list of MCP servers, the agent connects to your gateway, which decides what tools exist, which arguments are acceptable, and where the call terminates.

Click to zoom

The configuration mirrors the model list — that is the point. One file, one review, one place where an external destination gets justified in a pull request rather than a library call buried in an agent's source.

mcp_servers:
  # Internal MCP servers — in-cluster, no egress.
  internal_tickets:
    url: "http://mcp-tickets.ai-tools.svc.cluster.local:8080/mcp"
    transport: "http"
    allowed_tools: ["search_tickets", "create_ticket"]   # deny-by-default

  internal_warehouse:
    url: "http://mcp-warehouse.ai-tools.svc.cluster.local:8080/mcp"
    transport: "http"
    allowed_tools: ["run_saved_query"]   # NOT run_sql — verbs, not a shell
    auth_type: "bearer_token"
    auth_value: os.environ/MCP_WAREHOUSE_TOKEN

  # An external server, if you must: named, scoped, reviewable.
  vendor_docs:
    url: "https://mcp.example-vendor.com/mcp"
    transport: "http"
    spec_version: "2025-06-18"
    allowed_tools: ["search_docs"]   # read-only, no customer data

Three properties matter more than the syntax. First, the allow-list is where you decide what an agent is allowed to want: exposing run_saved_query rather than run_sql is the difference between a tool whose blast radius you enumerated and one whose blast radius is your database. A non-deterministic caller gets verbs, not a shell. Second, the tool-call becomes attributable — the virtual key identifying a model request identifies a tool request, so "which team's agent read that table at 03:00" has an answer. Third, versioning is central rather than per-agent: LiteLLM pins the MCP spec version per server (spec_version, default 2025-06-18), and as of v1.80.18 the gateway's own MCP protocol version is 2025-11-25. When the protocol moves — and one at this velocity will — you upgrade a gateway, not a fleet of agents.

It also makes the developer-tooling story coherent. The self-hosted coding assistants piece puts Continue and Tabby behind a single-purpose proxy for completion traffic. Once those assistants gain tools — read this repo, run this test, open this ticket — that proxy becomes one of several, each with its own auth story, none with a shared log. The MCP gateway extends the shared enforcement point to the tool half of the traffic.

The audit trail you were going to need anyway

The regulatory dividend falls out of the architecture for free, which is the mark of a good boundary: every model call and tool-call transits one process with an identity attached, making the gateway the natural emission point for the record. It is the mechanism behind an obligation we have described elsewhere in the abstract. The EU AI Act for high-risk AI systems requires automatic recording of events over a system's lifetime — and for an agent, "the system" plainly includes what it did, not merely what it said. Per-application logging cannot satisfy that coherently; the events live in as many formats as you have applications. One gateway emitting one structured record per call — caller, model, tool, timestamp, latency, tokens, guardrail verdict, destination — maps onto the duty directly. The same record answers the NIS2, DORA, and EU AI Act convergence question about third-party dependency: the log of which calls took the external path is your ICT third-party exposure for AI, measured rather than asserted.

Two cautions. The log now holds prompts and tool arguments, so it inherits the sensitivity of the traffic it describes: it is a PII store, with the retention, encryption, and access control of one. mode: logging_only exists for exactly this — mask what reaches the sinks even when the call was internal and needed raw values. And treat metrics and logs as different products: latency and spend series live for years cheaply; request bodies should probably not live for months.

Failure modes: what a gateway costs you

An honest architecture piece names the bill. Centralising every AI call buys enforcement and creates concentration; both are real. None of what follows is a reason not to build it — they are reasons to build it deliberately, at two models or two teams, rather than at nine when the retrofit means touching every caller at once.

• It is a single point of failure by construction. Every agent stops when the gateway stops. Several replicas across nodes, stateless, Postgres on real HA — and test the failover. A gateway never failed over in business hours is a hypothesis.
• It adds a hop. Single-digit-millisecond overhead is irrelevant for chat and very relevant for an agent loop making forty sequential tool-calls. Measure the loop, not the request.
• It concentrates credentials. Every upstream key now lives in one place. That beats eleven places only if the one place is hardened: short-lived injected secrets, no key material in the image, tight network policy.
• Config drift is the quiet failure. config.yaml is now your AI security policy. If it can be edited outside review — via the admin UI, at 02:00 — the allow-lists are decorative. Reconcile from Git; treat UI edits as incidents.
• The database is the state. Virtual keys, budgets, and spend live in Postgres. Put it in the backup and DR plan, or the gateway becomes a confident way to lose your access-control model.

The exit ramp

Every component we recommend must answer the same question: how do you leave? A gateway governing all AI traffic is exactly the chokepoint that becomes lock-in if you are careless — so design the exit alongside the entrance. The structural protection is that neither interface is proprietary: clients speak the OpenAI-compatible API every serious gateway implements, and tools speak MCP, now governed by a Linux Foundation directed fund rather than a vendor. Replacing the gateway means standing up something else that speaks those two protocols and re-pointing a base URL — your callers do not change. That is why you insist on standard protocols at the boundary even when a proprietary SDK is more ergonomic on day one.

1. Keep the config in Git, not the UI. config.yaml is a portable declaration of your models, routes, guardrails, and tools — and the migration document if you ever move. Pin and mirror the images too, so a fast-moving upstream cannot break you on its schedule.
2. Never let a client name a vendor. If application code says internal-chat, you can move it to another backend, region, or gateway. If it names a provider's model string, you have hard-coded a supplier into your source tree.
3. Export the audit log to a sink you own. Structured records in your own observability stack outlive the tool that produced them. Evidence trapped in a vendor's dashboard is evidence you are renting.
4. Watch the licence line. The core is open source; parts of the enterprise tier are not. Know which side each feature you depend on sits — a commercial tier is a legitimate choice, but only when it is a choice rather than a discovery.

The long game

Step back from the YAML. The models you serve today will be obsolete within eighteen months — the one confident prediction available in this field. Weights improve, licences change, a 32B model does what a 70B did last year. If your architecture's identity is bound to a model, it has a model's shelf life.

The control plane is the durable part. Who may call what, with which data, under which policy, recorded where — those questions outlive every particular answer, and auditors, regulators, and your own incident reviews will ask them in 2030 exactly as today. A team that built the gateway early swaps models as a config change, carrying policy, keys, budgets, and audit history intact across every generation. A team that wired applications straight to whatever was good that quarter rebuilds the governance each time, badly, under deadline. Interfaces outlive implementations, and the organisations that survive transitions cheaply owned their interfaces. Sovereignty is not the GPU. It is the ability to change the GPU, the model, the vendor, and the protocol version without renegotiating your relationship with your own data. Build the boundary while it is still a config file.