LiteLLM Alternatives: Where Your LLM Gateway Should Live

Most comparisons rank gateways on request overhead. That is the least interesting axis. A gateway holds every provider key you own, sits in front of every model call, and either enforces your spend limits or quietly does not. The questions that matter are operational: what happens to a request when the gateway's state store is unreachable, whether a budget is counted once or once per replica, which audit features sit behind a licence line, and how much of your configuration survives a move. This article answers those from each project's own documentation and source, as of 25 October 2026.

Why people are looking: March 2026 and the gateway as blast radius

Search interest in LiteLLM alternatives has an obvious trigger. According to LiteLLM's own security update, malicious releases 1.82.7 and 1.82.8 were live on PyPI on 24 March 2026 from 10:39 UTC for about 40 minutes before PyPI quarantined them. Version 1.82.8 contained a litellm_init.pth file and a malicious payload in the gateway's proxy_server.py. The post says the compromised versions appear to have included a credential stealer designed to encrypt and exfiltrate data with a POST request to models.litellm.cloud, which is not a BerriAI domain. LiteLLM believes the compromise started from the Trivy dependency in its CI security-scanning workflow. Bitsight, a security vendor and a secondary source, describes the payload as three stages: a credential harvester, a Kubernetes lateral-movement toolkit and a persistent backdoor.

Do not check your exposure against the 40 minutes alone. LiteLLM's affected-user test is wider. By its own list, you may be affected if any of these is true:

• You installed or upgraded LiteLLM with pip on 24 March 2026 between 10:39 UTC and 16:00 UTC.
• You ran pip install litellm without pinning a version and received v1.82.7 or v1.82.8.
• You built a Docker image of your own in that window that ran pip install litellm without a pinned version.
• A dependency in your project pulled LiteLLM in as a transitive, unpinned dependency, for example through an agent framework, an MCP server or an LLM orchestration tool.

The same post lists who was not affected: users of the official gateway image ghcr.io/berriai/litellm, whose deployment path pins dependencies in requirements.txt and does not rely on the compromised PyPI packages; LiteLLM Cloud users; anyone on v1.82.6 or earlier who did not upgrade during the window; and anyone who installed from the GitHub source, which was not compromised. A clean release, v1.83.0, followed on 30 March, and from v1.83.0-nightly every LiteLLM image published to GHCR is signed with cosign.

The lesson is not that LiteLLM is uniquely unsafe. It is that a gateway is the worst possible place to run unpinned code, because it is the one process that legitimately holds every credential. That argues for pinning, digest references and signature checks on whichever gateway you pick, which is the same discipline set out in Stribog's piece on SBOMs, Sigstore and admission control. It does not, on its own, argue for leaving.

#!/usr/bin/env bash
set -euo pipefail

# Replace with the LiteLLM release you deploy. v1.83.0-nightly is the
# first signed image; check the tag resolves on GHCR before you pin it.
TAG="v1.83.0-nightly"
IMAGE="ghcr.io/berriai/litellm"
# Public key pinned to a commit, not a mutable branch or tag.
KEY="https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub"

# Requires cosign and crane (go-containerregistry).
# Resolve the tag once, then verify and deploy the digest, never the tag.
DIGEST="$(crane digest "${IMAGE}:${TAG}")"
cosign verify --key "${KEY}" "${IMAGE}@${DIGEST}"

echo "Deploy: ${IMAGE}@${DIGEST}"

Two places a gateway can live

If you need the basics of what a gateway does — one endpoint, virtual keys, routing, guardrails at the tool-call boundary — Stribog's walkthrough of LiteLLM as a model and MCP gateway covers them. Here the question is narrower: which kind of thing are you adding to your estate?

Application gateways

An application gateway is a service you deploy beside your apps, with its own process, its own state store and its own release train.

• LiteLLM is Python. Its repository is MIT-licensed except an enterprise/ directory under its own licence. Virtual keys, teams and spend live in Postgres; once you run more than one instance, Redis shares rate-limit counters, router state and the response cache. Its production guide asks for 1 vCPU and 4Gi of memory per worker, as both requests and limits, so a container running 8 workers wants 8 vCPU and 32Gi.
• Bifrost is Go and Apache-2.0, started with docker run -p 8080:8080 maximhq/bifrost. It stores configuration in SQLite or PostgreSQL.
• Portkey's gateway is TypeScript and MIT, started locally with npx @portkey-ai/gateway. Palo Alto Networks closed its acquisition of Portkey on 29 May 2026, and Portkey's documentation now carries the banner that Portkey is Prisma AIRS AI Gateway: that is the hosted and enterprise product. The MIT gateway on GitHub is the self-hostable artifact, and everything in this article about Portkey refers to it.

Each project publishes a speed figure, and each is a vendor benchmark under its own conditions. LiteLLM reports 1,170 requests per second with 8 ms p95 gateway overhead across four instances in one of its tests; Bifrost's README reports 11 µs of added overhead per request in sustained 5,000 RPS benchmarks on a t3.xlarge, and 59 µs on a t3.medium. They are not a ranking. For most teams, the model's own latency dwarfs either number.

Data-plane gateways

A data-plane gateway is not a new service on the request path. It is an extension to a proxy you already run. The clearest example is Agent Router, which was Envoy AI Gateway until this month. Its README says it is now an Agentic AI Foundation project with the same code, maintainers and Apache 2.0 licence; v1.2.0, released on 7 October 2026, is the first release under the new name. The controller configures an External Processor and adjusts Envoy's xDS through Envoy Gateway's extension-server mechanism, so routing, retries and priority fallback run in Envoy itself, with the AI-specific work in an ext_proc filter beside it. If your ingress already runs Envoy Gateway — the bare-metal pattern in Stribog's MetalLB versus kube-vip piece — this is the gateway that adds the fewest new moving parts.

Kong and Apache APISIX sit on the same side of the line, as AI plugins on an API gateway. APISIX's ai-proxy-multi extends ai-proxy with load balancing, retries, fallbacks and health checks. Kong's ai-proxy-advanced, which carries load balancing, retries and failover, and its ai-rate-limiting-advanced are both only available in Kong's AI Gateway Enterprise offering. Check that licence before shortlisting Kong for self-hosted routing.

Click to zoom

Fallback and routing, configured

The same job — the primary provider fails, so send the request to a secondary — looks different in each placement. In Agent Router, you declare prioritised backends on an AIGatewayRoute, and fallback is triggered by retry policies set with Envoy Gateway's BackendTrafficPolicy, attached to the HTTPRoute the controller generates with the same name. The project's provider-fallback doc introduces the feature as covering network errors, 5xx responses and other health-check failures; the example policy below is narrower, and that is worth reading closely.

apiVersion: aigateway.envoyproxy.io/v1beta1
kind: AIGatewayRoute
metadata:
  name: chat-fallback
  namespace: default
spec:
  parentRefs:
    - name: ai-gateway
      kind: Gateway
      group: gateway.networking.k8s.io
  rules:
    - matches:
        - headers:
            - type: Exact
              name: x-ai-eg-model
              value: chat-default # an alias you own, not a vendor model name
      backendRefs:
        - name: openai-primary # AIServiceBackend, priority 0
          priority: 0
          modelNameOverride: gpt-4o-mini
        - name: vllm-fallback # AIServiceBackend, priority 1
          priority: 1
          modelNameOverride: llama-3.1-8b-instruct
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: chat-fallback
  namespace: default
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: chat-fallback # generated with the AIGatewayRoute's name
  retry:
    numAttemptsPerPriority: 1
    numRetries: 5
    perRetry:
      backOff:
        baseInterval: 100ms
        maxInterval: 10s
      timeout: 30s
    retryOn:
      httpStatusCodes:
        - 500
      triggers:
        - connect-failure
        - retriable-status-codes

Two details carry the weight. numAttemptsPerPriority: 1 means a failing primary is not retried on itself before the request moves on; once every priority has been tried, the exclusion resets, so remaining retries alternate between them. And retryOn lists HTTP 500 plus connection failures, so a 429 from a rate-limited provider or a 503 from an overloaded one will not fail over unless you add those codes. The request carries one model name, here an alias; modelNameOverride on each backendRef rewrites it to the name that backend actually serves. Agent Router's supported-providers page puts self-hosted servers at "depending on the API schema spoken", so a vLLM deployment exposing the OpenAI-compatible API is the natural fallback target.

The Portkey gateway does the same job as JSON, sent per request in the x-portkey-config header or attached by your client. Fallback triggers on any non-2xx status by default; on_status_codes narrows it. Retries take attempts and their own on_status_codes, and when that list is set, retries fire only on those codes rather than the defaults.

{
  "strategy": { "mode": "fallback", "on_status_codes": [429, 503] },
  "targets": [
    {
      "provider": "openai",
      "api_key": "REPLACE_WITH_OPENAI_KEY",
      "override_params": { "model": "gpt-4o" },
      "retry": { "attempts": 2, "on_status_codes": [429] }
    },
    {
      "provider": "anthropic",
      "api_key": "REPLACE_WITH_ANTHROPIC_KEY",
      "override_params": { "model": "claude-3-5-sonnet-20241022" }
    }
  ]
}

Bifrost and LiteLLM take a third and fourth shape. In Bifrost, each fallback provider gets its own full retry budget. LiteLLM's router cools a deployment immediately on a 429 for cooldown_time (5 seconds by default), and also immediately on a 401, 404 or 408 and when more than half of the current minute's calls fail; allowed_fails: 3 is a router_settings default, not the only trigger. It also has a separate context_window_fallbacks list for context-window-exceeded errors, which it maps across providers so a long prompt can move to a model with a bigger window.

When the gateway is down

Every gateway fails. What matters is that it fails in the direction you chose, and that you know which direction that is before the pager does. LiteLLM is the best-documented case. Its production guide splits database trouble into distinct cases once you set allow_requests_on_db_unavailable, and they should be read separately:

• Database unreachable (connection refused or reset): requests proceed with a restricted INTERNAL_USER fallback identity, never admin. The docs warn to enable this only when the gateway runs inside a VPC that cannot be reached from the public internet.
• Database reachable but the query fails: the key cannot be verified and the request gets a 401. That path stays fail-closed.
• Independently of the flag, virtual keys already in the in-memory auth cache keep authenticating during an outage until user_api_key_cache_ttl expires, which defaults to 60 seconds.
• Without Redis, each instance enforces rate limits on its own, so a limit you meant for the fleet is enforced once per replica.

model_list:
  - model_name: chat-default
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  # Fail open on DB connection loss, as a restricted INTERNAL_USER.
  # Only inside a VPC unreachable from the public internet.
  allow_requests_on_db_unavailable: true

router_settings:
  # Documented defaults, written out so a reviewer can see them.
  allowed_fails: 3
  cooldown_time: 5
  # Shared counters and router state across instances.
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  redis_password: os.environ/REDIS_PASSWORD

Agent Router fails the other way, visibly in the source. In v1.2.0, post_translate_modify.go inserts the AI Gateway's ext_proc filter with FailureModeAllow: false and a 10-second message timeout. If the processor is unreachable or slow, the request on that AI-Gateway-generated route fails rather than passing through unprocessed. That is a property of that filter on those routes, not a fail-closed policy for every listener on the proxy. It is also the right default for a component that does request processing and token accounting: a gateway that silently skips usage extraction during an incident is a gateway whose spend numbers you can no longer trust.

Bifrost's docs say where its configuration lives — SQLite or PostgreSQL — but the pages cited here do not describe request behaviour while that store is unavailable. Treat that as a test you run yourself before production, not as a property you can quote. The same applies to the Portkey gateway. The general rule: fail-closed protects budgets and audit trails at the cost of availability, fail-open protects availability at the cost of both. Pick one per route, write it down, and rehearse it.

Budgets, logs and what an auditor is shown

A budget is only as strong as the counter behind it. LiteLLM's counters are per instance until Redis is shared, as above. Agent Router extracts token usage from responses in the OpenAI schema for usage-based rate limiting, and the llmRequestCosts you define are scoped exclusively to the AIGatewayRoute that defines them, so a cost limit on one route says nothing about traffic on another. Bifrost documents virtual keys that carry access permissions, budgets and rate limits among its open-source features.

The licence line is where audit questions get expensive, so map it per project from the vendor's own words:

• LiteLLM: SSO is free for up to 5 users, beyond which an Enterprise licence is required. The Enterprise tier adds SSO and SCIM, audit logs, org and team admins, secret managers, IP allowlists and per-team logging on top of the open gateway.
• Bifrost: the README says, in full, that "in addition to private networking, custom security controls, and governance, enterprise deployments unlock advanced capabilities including adaptive load balancing, clustering, guardrails, MCP gateway, and other features designed for enterprise-grade scale and reliability." Its public MCP docs describe Bifrost acting as an MCP client and optionally as an MCP server, and document a Gateway Mode, but they do not settle which side of the licence line the MCP gateway sits on. Assume the README's placement until the vendor says otherwise in writing.
• Portkey: the README marks some features as available only in hosted and enterprise versions, and says its core enterprise gateway is merging into open source with the 2.0 release, which is a pre-release branch today. Check the exact feature you need against the branch you would deploy.
• Agent Router: Apache 2.0 throughout, with MCP routing through an MCPRoute API.
• Kong: the advanced AI proxy and AI rate-limiting plugins are Enterprise-only, as above.

The practical test for an auditor is simple to state: can you show who called which model, under which key, at what cost, without buying a tier? For a regulated team, the answer per project is a procurement input, and it belongs in the decision record before the gateway goes live, not after the first evidence request.

The exit ramp

Every gateway here speaks some form of the OpenAI-compatible API to callers. That is the portability you keep, and it only holds if you protect it:

1. Own the model aliases. Callers ask for chat-default, not a provider's model ID. The gateway maps aliases to providers; changing gateways means rebuilding that map, not touching application code.
2. Keep provider keys in your secret store, injected into the gateway at deploy time. A gateway that becomes the only holder of your keys has made itself hard to leave.
3. Issue gateway keys per service, so a migration can move one service at a time behind a second gateway instead of one weekend cut-over.
4. Export spend and request logs to your own store on a schedule, so history survives the gateway's database.

Routing config does not port. A LiteLLM router block, a Bifrost fallback list, a Portkey JSON config and an Agent Router route with its traffic policy express the same intent in four incompatible shapes, and retry semantics differ underneath them. Budget that translation as real engineering, with a test suite that injects 429s, 5xx and connection failures and asserts where each request ends up. The Envoy AI Gateway rename is a useful reminder of what you actually depend on: the project's name changed this month, but its AIGatewayRoute, AIServiceBackend and BackendSecurityPolicy resources and the aigateway.envoyproxy.io API group stayed as they were. Pin to APIs and resource kinds, not to brands.

The serving layer behind the gateway should be equally replaceable. Stribog's comparison of vLLM and Ollama argues for drawing the concurrency line between them, and a gateway is what makes running both invisible to callers.

The long game

Model choice is increasingly deferred to the routing layer, which makes the gateway the most load-bearing piece of AI infrastructure most teams own. Pick it the way you would pick any decade-scale dependency. Who maintains it, and under which licence, today and after the next acquisition? How is it released, and are the artifacts signed and pinnable by digest? Does it fail in the direction you chose, and can you show that in a rehearsal rather than a document? Where does it sit in the wider decision order for your on-premise AI stack?

If you already run Envoy Gateway and want the fewest new stateful parts, Agent Router fits. If you need virtual keys, budgets and a UI today and can run Postgres and Redis properly, LiteLLM or Bifrost fits, with the licence line read carefully. If you value a small stateless process, the MIT Portkey gateway fits, with its hosted product's new owner kept clearly separate. Whichever you choose, keep callers on aliases you own, keys in a store you control, and retries in exactly one layer. That is what keeps the next migration a config change.