vLLM vs Ollama: Run Both, and Draw the Concurrency Line

Search for this comparison and you get a wall of throughput multipliers. They disagree, none is reproducible from a primary source, and none describes the GPU in your rack. This article adds no number to that pile. It quotes what each project documents about allocating memory, and hands you the instruments that show which side of the line you are on.

Defaults change between releases, so everything below is pinned to Ollama v0.32.15 (19 August 2026) and vLLM v0.27.1 (11 August 2026), read against the vLLM stable documentation on 22 August 2026. Re-check any default you depend on.

The Question the Benchmarks Answer Is the Wrong One

The head-to-head framing assumes both projects compete for the same job. Read their own material and that falls apart. Ollama's README lists exactly one supported backend, llama.cpp, and its documentation is organised around a person at a machine: pull a model, run it, set num_ctx interactively. vLLM's README leads with continuous batching, chunked prefill, prefix caching, and tensor, pipeline, data and expert parallelism for distributed inference. Two different questions.

That difference has an organisational consequence, which is the real reason to write the comparison down. Ollama is superb at what it is for, and that is how it ends up terminating a production endpoint: someone had it working locally, someone else pointed a service at it, eighteen months later it is load-bearing. The useful question is not which is faster, but when a runtime built for one operator stops being right behind a shared address.

Two Memory Models Sharing One GPU

Every throughput chart is downstream of one architectural difference: how each runtime reserves the memory holding attention state. Understand it and you no longer need the charts.

Ollama's FAQ is unusually direct. Under OLLAMA_NUM_PARALLEL — "the maximum number of parallel requests each model will process at the same time, default 1" — it states: "Required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH." The worked example that follows is the key sentence: "a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation."

Read that as an allocation statement, not a performance one. You do not get four cheap 2K conversations; you get an 8K allocation, sized up front, held whether or not four callers exist. Note what the passage does not say: it never uses the phrase "KV cache", and publishes no bytes-per-token figure. Anyone offering a closed-form VRAM formula for Ollama invented the constant.

vLLM starts from the opposite premise. The PagedAttention paper (SOSP 2023) describes "an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems," achieving "near-zero waste in KV cache memory" and "flexible sharing of KV cache within and across requests." Blocks are handed out as tokens are produced: an idle client holds no reservation, and two requests sharing a system prompt share the blocks holding it.

One number from that paper circulates without its context, so state it precisely: the authors report vLLM improving "the throughput of popular LLMs by 2-4× with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca." A 2023 result against those two baselines — not a measurement of Ollama, and never intended as one. The mechanism transfers; the multiplier does not.

Click to zoom

Computing Your Own Crossover, Then Measuring It

Here most comparison articles reach for a benchmark; this one reaches for a terminal. Scale the context Ollama will allocate by the FAQ's product — arithmetic on a documented sentence, not a prediction of gigabytes. Four slots at 8192 tokens is a 32K allocation. Hold that number, not 8192.

Ollama's documentation makes the case for measuring rather than calculating, because it contradicts itself on the starting value. The FAQ says "By default, Ollama uses a context window size of 4096 tokens." The context-length page says Ollama "defaults to the following context lengths based on VRAM: < 24 GiB VRAM: 4k context; 24-48 GiB VRAM: 32k context; >= 48 GiB VRAM: 256k context." Both are live; neither supersedes the other.

That same page tells you how: "avoid offloading the model to CPU. Verify the split under PROCESSOR using ollama ps." 100% GPU means the allocation fit. A split such as 48%/52% CPU/GPU means it did not — and nothing errored. Throughput fell, silently.

Write the arithmetic where the machine can see it. A shared Ollama host should not run on defaults nobody chose:

# /etc/systemd/system/ollama.service.d/override.conf
# systemctl daemon-reload && systemctl restart ollama

[Service]
# Default bind is 127.0.0.1:11434. Changing it adds no authentication.
Environment="OLLAMA_HOST=0.0.0.0:11434"

# FAQ: "Required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH".
# The product sizes the allocated context: 4 x 8192 = 32K.
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_CONTEXT_LENGTH=8192"

# Default "3 * the number of GPUs or 3 for CPU inference": three resident
# models on one card, and ALL new requests queue when a fourth will not fit.
Environment="OLLAMA_MAX_LOADED_MODELS=1"

# Default 512 hides saturation behind latency; 64 returns the documented 503
# while the caller can react.
Environment="OLLAMA_MAX_QUEUE=64"

# Models unload after 5 minutes; a negative value pins them. The keep_alive
# API parameter overrides this per request.
Environment="OLLAMA_KEEP_ALIVE=-1"

# Forced on because K/V cache quantization requires it. Default type is f16;
# q8_0 uses about half.
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

Then measure. This benchmarks nothing — it drives concurrency at both endpoints and reads what each runtime reports about its own memory under load.

#!/usr/bin/env bash
# Locate the crossover on YOUR GPU. Both endpoints must already be running.
# Measures allocation and live scheduler state, not throughput.
set -euo pipefail

OLLAMA="${OLLAMA:-http://localhost:11434}"
VLLM="${VLLM:-http://localhost:8000}"
MODEL="${MODEL:-llama3.2}"
VLLM_MODEL="${VLLM_MODEL:-team-default}"
N="${N:-8}"

body() { printf '{"model":"%s","messages":[{"role":"user","content":"Count to 500."}]}' "$1"; }

# 1. Pin the model and read what Ollama reserved AT LOAD TIME: PROCESSOR is a
#    fit signal, fixed at load, not a live one.
curl -sS "${OLLAMA}/api/generate" \
  -d "{\"model\":\"${MODEL}\",\"prompt\":\"warm\",\"keep_alive\":-1}" >/dev/null
ollama ps

# 2. Load BOTH endpoints concurrently. Ollama queues past its parallel slots
#    up to OLLAMA_MAX_QUEUE, then answers 503.
for i in $(seq 1 "${N}"); do
  curl -sS -o /dev/null -w "ollama ${i}: HTTP %{http_code}\n" \
    "${OLLAMA}/v1/chat/completions" -H "Content-Type: application/json" \
    -d "$(body "${MODEL}")" &
  curl -sS -o /dev/null -w "vllm ${i}: HTTP %{http_code}\n" \
    "${VLLM}/v1/chat/completions" -H "Content-Type: application/json" \
    -H "Authorization: Bearer ${VLLM_API_KEY:-none}" \
    -d "$(body "${VLLM_MODEL}")" &
done

# 3. Scrape WHILE those requests are in flight: these gauges are live scheduler
#    state and read zero on an idle server.
#    waiting up, kv near 1  -> the block pool is the constraint
#    waiting up, kv low     -> look at max-num-seqs, not at VRAM
for _ in 1 2 3; do
  sleep 2
  curl -sS "${VLLM}/metrics" | grep -E \
    "^vllm:(num_requests_running|num_requests_waiting|kv_cache_usage_perc)" || true
done
wait

The Cluster-Side Configuration That Survives Contact

Once measurement says a shared endpoint belongs on vLLM, four flags carry most of the risk. All four are in the stable engine-argument reference, and all four have bitten somebody.

• --gpu-memory-utilization — "the fraction of GPU memory to be used for the model executor... If unspecified, will use the default value of 0.92." The next sentence is the trap: it is "a per-instance limit... It does not matter if you have another vLLM instance running on the same GPU." Two instances at the default both try to take 92% of the card. Older articles still quote 0.9; the stable docs say 0.92 today.
• --max-model-len — accepts -1 or auto to "automatically choose the maximum model length that fits in GPU memory". Convenient on a workstation, a liability on a shared endpoint: the context clients rely on should be your decision, not a property of whichever card the pod landed on.
• --max-num-seqs — the "maximum number of sequences to be processed in a single iteration." The documentation publishes no numeric default, noting it is mainly for testing convenience. Set it from the two gauges above.
• --served-model-name — "the model name(s) used in the API... The model name in the model field of a response will be the first name in this list." Cheap optionality: clients address team-default while you swap the weights underneath.

Two more defaults surprise people. --tensor-parallel-size defaults to 1, so a four-GPU node runs on one card until you say otherwise. And tensor-parallel inference needs host shared memory — in Kubernetes an emptyDir at /dev/shm, as the project's own manifest shows alongside liveness and readiness probes on /health.

# A shared vLLM endpoint, reduced to the memory and authentication argument.
# Node affinity, PVC layout, GPU scheduling and the Service belong to the full
# manifest. Namespace inference and Secret vllm-api-key must already exist;
# gated weights also need HF_TOKEN.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-shared
  namespace: inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-shared
  template:
    metadata:
      labels:
        app: vllm-shared
    spec:
      volumes:
        # Host shared memory, needed for tensor-parallel inference.
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: "2Gi"
      containers:
        - name: vllm
          # Upstream uses :latest. Not a version pin.
          image: vllm/vllm-openai:latest
          command: ["/bin/sh", "-c"]
          args:
            - >
              vllm serve mistralai/Mistral-7B-Instruct-v0.3
              --served-model-name team-default
              --gpu-memory-utilization 0.85
              --max-model-len 8192
              --max-num-seqs 32
              --api-key "$(VLLM_API_KEY)"
          env:
            - name: VLLM_API_KEY
              valueFrom:
                secretKeyRef:
                  name: vllm-api-key
                  key: key
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: "1"
            requests:
              nvidia.com/gpu: "1"
          volumeMounts:
            - name: shm
              mountPath: /dev/shm
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 5

The complete manifest — node affinity, GPU scheduling, storage, regulatory framing — is in our production vLLM on Kubernetes guide, which assumes the runtime decision this article makes. vLLM also documents Helm, KServe, llm-d, KubeRay and AIBrix: the native manifest is a starting point, not the only road.

Failure Modes on Both Sides of the Line

Each degrades in a characteristic way, and neither degrades loudly. Knowing the shape in advance is the whole value.

How Ollama Fails

The first is silent context multiplication. Someone raises OLLAMA_NUM_PARALLEL to improve concurrency, the allocated context grows by that factor, the model no longer fits, and the split moves to CPU. There is no error. ollama ps reports it under PROCESSOR; nothing else does.

The second is the queue. Requests beyond the parallel slots queue up to OLLAMA_MAX_QUEUE, whose "default is 512", and past that the server "will respond with a 503 error indicating the server is overloaded." A queue that deep converts a capacity problem into a latency mystery for any caller with a thirty-second timeout. Set it shallow enough that saturation stays visible.

The third is model thrash. OLLAMA_MAX_LOADED_MODELS defaults to "3 * the number of GPUs or 3 for CPU inference", and the documentation is explicit: "If there is insufficient available memory to load a new model... all new requests will be queued until the new model can be loaded." All new requests — not only those for the incoming model. One person trying an unfamiliar model stalls the endpoint for everybody.

How vLLM Fails

vLLM's characteristic failure is the utilisation ceiling meeting a neighbour. Because --gpu-memory-utilization is per-instance, co-locating two vLLM pods on one card without lowering both is an out-of-memory event at startup. Sharing GPUs deliberately is a different discipline: MIG partitions or time-slicing give the scheduler an allocation it understands, which a per-instance fraction does not.

The subtler one is pressure on the block pool. When it fills, requests wait — graceful, and invisible if you watch only latency percentiles. vllm:num_requests_waiting climbing while vllm:kv_cache_usage_perc sits near 1 is the signal; without /metrics you diagnose it as "the model got slower". For a repeatable curve, use vllm bench serve.

The Endpoint You Just Put on the Network

Here the two projects diverge most sharply, and it is easiest to overstate. Take it strictly from what each documents.

Ollama's authentication page is unambiguous: "No authentication is required when accessing Ollama's API locally via http://localhost:11434." Authentication exists for a different purpose — "running cloud models via ollama.com, publishing models, downloading private models." Its own OpenAI-compatibility example puts it plainer than commentary could, annotating api_key='ollama', # required but ignored. And the FAQ notes that "Ollama binds 127.0.0.1 port 11434 by default", changed with OLLAMA_HOST.

Put those together: the bind default is the only thing between an unauthenticated inference endpoint and your network, and it is one environment variable deep. Setting OLLAMA_HOST to a routable address does not add authentication; it removes the last control. Be precise: the documentation says the server does not require a credential, not that it cannot be protected. The answer is a reverse proxy that terminates TLS and enforces identity.

vLLM differs in kind, not degree. Per the CLI reference, --api-key means "the server will require one of these keys to be presented in the header," while --ssl-keyfile and --ssl-certfile terminate TLS at the process. An unflagged vLLM is just as open — but the control lives in the runtime, is set in a manifest, and is evidenced to an auditor without also proving a proxy is always in the path.

For anything multi-tenant neither answer is complete alone. A gateway — LiteLLM as a policy and residency boundary — is where per-team keys, rate limits and audit logging belong: one place, not two runtimes to keep in agreement.

Click to zoom

Exit Ramps: Keeping Both Replaceable

Running two runtimes is defensible only if leaving either is cheap. Split the question; the halves have different answers.

The request path is genuinely portable. Ollama "provides compatibility with parts of the OpenAI API" and serves /v1/chat/completions; vLLM's README lists an OpenAI-compatible API server plus an Anthropic Messages API. One client reaches either:

"""One client, two runtimes: only base_url, the credential and the model name
change. The request path is portable; the model artefact path is not.
"""

import os

from openai import OpenAI

# Workstation. Ollama's own example annotates the key "required but
# ignored".
laptop = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

# Cluster. --api-key enforces the key; --served-model-name decouples the
# name clients send from the weights.
cluster = OpenAI(
    base_url="https://inference.internal.example.com/v1",
    api_key=os.environ["VLLM_API_KEY"],
)

for client, model in ((laptop, "llama3.2"), (cluster, "team-default")):
    reply = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "Summarise this changelog."}],
    )
    print(reply.choices[0].message.content)

The artefact path is where the asymmetry lives, and it runs opposite to the direction most people assume. Toward Ollama is easy: it documents importing Safetensors weights via a Modelfile whose FROM points at the weights directory, and importing "a GGUF based model or adapter" directly. Moving a GGUF toward vLLM is the awkward direction. vLLM's own page warns that "GGUF support in vLLM is highly experimental and under-optimized at the moment, it might be incompatible with other features," and that it "has migrated to OOT vllm-gguf-plugin" — install the plugin or it will not serve.

So "just point vLLM at the GGUF you already pulled" is not the free move it is described as. Plan the promotion path in the format vLLM serves natively — its README leads with Hugging Face model integration — and keep the quantised local copy as a derivative, not the source of truth. A five-minute decision at project start; a fortnight if deferred. If hardware binds, GPU and VRAM sizing is the prior question and a second-source GPU vendor the deeper ramp.

Be honest that the both-tools answer is not free: two supply chains, two CVE feeds, two upgrade cadences, two sets of defaults. Licensing stays simple — Ollama is MIT, vLLM is Apache-2.0 — but the overhead is real. Worth paying, because the alternative is developers queuing for cluster GPUs, or a laptop tool holding a production endpoint.

The Long Game: A Laptop Convenience Is Not a Platform Commitment

The failure this article exists to prevent has a recognisable shape. A developer installs Ollama because it is the fastest way to get a model answering questions. Someone points a prototype at it. The prototype ships. OLLAMA_HOST becomes 0.0.0.0 so a colleague can reach it. Eighteen months later an unauthenticated endpoint runs on a workstation under a desk, with a static context allocation nobody sized, serving traffic a business depends on.

No one decided that. Every step was small and reasonable. That is what makes it a governance problem rather than a technical one: the fix is an afternoon, and it never happens because nobody owns the question.

So own it explicitly. Write down which runtimes may terminate a shared endpoint, and name the ones that may not — including the excellent development tool. Attach a measurement rather than an opinion: an endpoint addressed by more than one caller gets /metrics scraped, and a runtime that can require a credential. Short enough for a wiki, specific enough to settle an argument.

This is what sovereignty looks like in practice, and it is unglamorous. Both projects are open source and no pricing change takes either away. The risk in self-hosted inference is no longer a vendor — it is drift: defaults nobody chose, allocations nobody sized, endpoints nobody authenticated. Both runtimes publish enough to make each a decision instead, at the price of one afternoon and one dashboard. Developers keep the tool that made this easy — including for self-hosted coding assistants, where the same question repeats one layer up.