Sizing the Iron: GPU and VRAM Planning for Self-Hosted LLM Inference

For most of the last three years, the first question in a self-hosted-LLM evaluation was "which model?" In 2026 it is "what hardware do I actually need?" — because open-weight models finally reached sizes that run on iron a normal engineering budget can buy, and the moment on-prem inference became viable, the capacity-planning question moved to the front of the queue. Procurement wants a number. Finance wants a depreciation schedule. The platform team wants to not over-buy an H100 cluster for a workload a pair of workstation cards would serve. All three are asking the same thing: how do you turn a model and a service-level objective into a defensible bill of materials?

This is a different problem from the two adjacent ones we have already written about, and it is worth drawing the boundary clearly. It is not the serving-software problem — how to run vLLM in production, schedule GPUs on Kubernetes, and put an audited gateway in front — which we cover in the production vLLM deployment guide. And it is not the utilization problem — how to slice GPUs you already own across many workloads with MIG and time-slicing. This is the buy-decision underneath both: the VRAM arithmetic, the consumer-versus-datacenter economics, and the throughput-versus-latency trade that together tell you which cards to put in the rack. Get this tier right and the software tiers above it have a stable foundation; get it wrong and no amount of clever serving recovers the money.

The good news for anyone who dreads hardware capacity planning: the math is deterministic. Unlike model quality, which is a moving benchmark, VRAM consumption is arithmetic you can do on a napkin, and GPU throughput is bounded by two physical quantities — memory bandwidth and compute — that vendors publish. What follows is that arithmetic, the economic trade behind consumer and datacenter silicon, and a sizing method that produces a bill of materials a CFO will sign and an engineer will not have to apologize for in six months.

The VRAM Budget: Three Numbers, Not One

The single most common sizing error is treating "model size" as "VRAM required." A 70B model does not need 70GB; the parameter count is only the first of three terms. Total effective VRAM is the sum of the weights, the KV cache, and a fixed overhead for activations, the CUDA context, and memory fragmentation. Miss any term and you either over-provision (wasted capital) or watch the server out-of-memory at exactly the concurrency you built it for.

• Weights — parameters × bytes-per-weight. This is the term quantization moves: bf16 is 2 bytes, int8 is 1 byte, four-bit is 0.5 bytes per parameter. A 70B model is ~140GB at bf16, ~70GB at int8, ~35GB at four-bit.
• KV cache — the running memory of the attention mechanism, proportional to concurrency × context length. Critically, it is stored in BF16 (2 bytes per element) regardless of how aggressively you quantized the weights, so it does not shrink when the weights do. At high concurrency it can rival the weights in size.
• Overhead — activations, the CUDA context, and allocator fragmentation. Budget a 1.1–1.2× multiplier on the running total; it is not optional and it is where naive estimates OOM.

The KV-cache term is the one that surprises people, because it scales with how the model is used, not just which model it is. Serving one user at 2K context is cheap; serving a hundred concurrent users at 8K context can add tens of gigabytes on top of the weights. The production deployment guide works the exact KV-cache formula at a target concurrency — for hardware sizing, the operational rule is simpler: size for the KV cache at your peak concurrency and longest supported context, because that is the moment the card either holds or fails.

Click to zoom

A small estimator makes the trade concrete and belongs in your capacity-planning repo next to the serving manifests. The function below returns the three terms and a recommended tier; run it for your model, precision, concurrency, and context before anyone raises a purchase order.

def vram_budget_gb(params_b, weight_bits, concurrent, ctx_len,
                   n_layers=80, n_kv_heads=8, head_dim=128, overhead=1.15):
    """Effective VRAM for a decoder LLM at a target concurrency.
    Defaults model a 70B-class model with grouped-query attention (GQA)."""
    # 1) Weights: params * bytes-per-weight
    weights_gb = params_b * (weight_bits / 8)

    # 2) KV cache: 2 (K and V) * layers * kv_heads * head_dim * bytes
    #    KV is stored BF16 (2 bytes) regardless of weight quantization.
    kv_bytes_per_token = 2 * n_layers * n_kv_heads * head_dim * 2
    kv_gb = kv_bytes_per_token * ctx_len * concurrent / (1024 ** 3)

    # 3) Overhead: activations + CUDA context + fragmentation
    total_gb = (weights_gb + kv_gb) * overhead

    tier = next(t for t, cap in [("24GB consumer", 24), ("48GB prosumer", 48),
                                 ("80GB datacenter", 80), ("2x80GB tensor-parallel", 160)]
                if total_gb <= cap)
    return {"weights": round(weights_gb, 1), "kv_cache": round(kv_gb, 1),
            "total_effective": round(total_gb, 1), "tier": tier}

# 70B at four-bit, 32 concurrent users, 8K context:
print(vram_budget_gb(70, 4, 32, 8192))
# -> weights 35.0, kv_cache ~10.7, total_effective ~52.6, tier '80GB datacenter'

Quantization: The Lever That Moves the Whole Budget

Quantization is the single most powerful knob in hardware sizing because it is the only one that changes the weight term without changing the model you chose. Dropping from bf16 to four-bit cuts the weight footprint by four, which is frequently the difference between one card and two, or between a consumer card and a datacenter one. That is not a rounding-error saving; it is a tier change, and a tier change is a budget change.

• bf16 / fp16 (2 bytes) — full-quality baseline, the reference every benchmark is measured against. Use it when accuracy is non-negotiable and VRAM is not the constraint.
• int8 (1 byte) — halves the weights with negligible quality loss on most tasks; a safe default for production where you want headroom without measurable degradation.
• four-bit — AWQ / GPTQ (0.5 bytes) — quarters the weights. Modern activation-aware methods (AWQ) and calibrated quantization (GPTQ) keep quality loss small on most workloads, though it is model- and task-dependent and must be measured, not assumed.

The optionality angle matters here too. Choose a quantization format supported by the mainstream open-source serving stack — AWQ and GPTQ are both first-class in the engines you will actually run — rather than a vendor-proprietary format that only one runtime loads. The format you quantize into is a dependency like any other; picking an open one keeps the model portable across serving engines and GPUs, which is exactly the anti-lock-in discipline that makes the hardware investment safe.

How Much GPU for a 70B Model?

This is the question every evaluation asks by name, so here is the direct answer, precision by precision, with the KV cache in mind. A 70B-class model is the current sweet spot for a capable general model you can actually own, so it is the right worked example.

• bf16 (~140GB weights) — needs two 80GB datacenter cards (A100/H100) in tensor-parallel, or more once you add KV cache for real concurrency. This is the full-quality, highest-cost path.
• int8 (~70GB weights) — fits a single 80GB card for the weights, but the KV cache at meaningful concurrency pushes you toward a second card or a reduced context. Comfortable on a tensor-parallel pair.
• four-bit (~35GB weights) — fits a single 48GB prosumer card for the weights, with the KV cache determining how much concurrency and context you can serve before you need 80GB or a second card.

Notice the pattern: at every precision, the weights set the floor and the KV cache sets the ceiling on how many users you serve. A 70B model at four-bit "fits" on a 48GB card the way a car "fits" four passengers — technically true, until you add luggage. For a low-concurrency internal tool, that single 48GB card is genuinely enough. For a shared production service at dozens of concurrent requests and long context, you are on an 80GB card or a tensor-parallel pair, and the honest bill of materials reflects the workload, not the marketing-friendly minimum.

Consumer vs. Datacenter GPUs: The Economics

Once you know the VRAM budget, the next decision is which class of silicon fills it — and this is where the CFO-legible trade lives. Consumer and prosumer cards (RTX 4090, RTX 6000 Ada, A5000/A6000) offer dramatically more VRAM per dollar than datacenter cards (A100, H100). The datacenter premium buys four things that matter under sustained, multi-GPU, always-on load: much higher memory bandwidth (HBM versus GDDR), NVLink for fast multi-GPU tensor parallelism, ECC memory for correctness, and MIG for hardware partitioning. You are not paying more for the same thing; you are paying for a different operating envelope.

Click to zoom

For a developer inner-loop assistant or a low-concurrency internal service, consumer cards are the economically correct choice — a point we make in detail in the self-hosted coding-assistant guide, where a single 24GB card comfortably serves a whole squad. For a production inference service under sustained concurrent load, the datacenter card's memory bandwidth and NVLink usually win on throughput-per-watt and throughput-per-rack-unit, even at the higher sticker price, because you are running the card hot around the clock and density and power then dominate the total cost.

Throughput vs. Latency: Sizing for the Workload You Have

VRAM tells you whether a model fits; it says nothing about how fast it runs. Inference performance splits into two phases with different bottlenecks, and sizing for the wrong one wastes money. The prefill phase — processing the prompt to produce the first token — is compute-bound: it does a lot of matrix multiplication in parallel and is limited by the card's raw FLOPs. The decode phase — generating each subsequent token one at a time — is memory-bandwidth-bound: every token requires streaming the entire model's weights out of VRAM, so tokens-per-second is governed by how fast the card can read its own memory, not by how many FLOPs it has.

This distinction determines what you optimize. If your workload is chat — short prompts, long generations — decode dominates and memory bandwidth is the number that matters. If it is document analysis or RAG — long prompts, short answers — prefill dominates and raw compute matters more. If it is a high-QPS API serving many users, aggregate throughput matters and you want a serving engine that batches aggressively across the fleet, which is why continuous batching (vLLM's PagedAttention) can lift throughput several-fold at the same VRAM budget. Latency-per-user and throughput-across-users are different objectives, and the same card can be sized for either depending on how you batch.

Memory Bandwidth Is the Hidden Ceiling

Because decode is memory-bound, memory bandwidth is the specification that most directly predicts single-stream generation speed — and it is the one buyers most often overlook in favor of VRAM capacity or FLOPs. The back-of-the-envelope upper bound is blunt but useful: the maximum tokens-per-second for a single sequence is roughly the memory bandwidth divided by the number of bytes read per token, which is approximately the model's size in memory. Real throughput lands well below this ceiling, but the ceiling itself tells you which card is faster before you benchmark anything.

• RTX 4090 — 24GB GDDR6X, ~1.0 TB/s bandwidth. Excellent single-stream speed per dollar; the reason it is the default developer card.
• RTX 6000 Ada / A6000 — 48GB GDDR6 with ECC, ~0.96 / ~0.77 TB/s. More VRAM, datacenter-licensed, similar bandwidth class to the 4090.
• A100 80GB — HBM2e, ~2.0 TB/s. Roughly double the bandwidth of a top consumer card, which is much of what the price buys.
• H100 80GB — HBM3, up to ~3.35 TB/s (SXM). The bandwidth leader, and correspondingly the throughput leader for memory-bound decode.

Read that list as a decode-speed ranking, because for token generation it very nearly is one. A four-bit 35GB model on an A100 at 2 TB/s has a single-stream ceiling near 57 tokens per second (2000 ÷ 35), and a real-world rate perhaps half that; the same model on a card with half the bandwidth generates at roughly half the speed regardless of its FLOPs. When someone proposes a card because of its compute numbers, ask what its memory bandwidth is — for the decode phase that dominates chat, bandwidth is the honest predictor. Instrument it in production the way you would any saturating resource, with GPU and memory-bandwidth metrics on your observability stack so utilization guides the next purchase rather than guesswork.

Headroom, Multi-GPU, and the Interconnect Tax

Two sizing mistakes hide at the margins: provisioning to exactly the computed VRAM, and assuming multi-GPU scaling is free. The first is a reliability trap — run a card at 99% VRAM and the first unusually long context or concurrency spike OOM-kills the server. Provision to roughly 85–90% utilization (vLLM's --gpu-memory-utilization defaults near this for a reason) and keep genuine headroom for the traffic you did not model. VRAM you never touch is cheaper than an outage at peak.

The second is the interconnect tax. When a model does not fit on one card, tensor parallelism splits it across several, and every token then requires the cards to exchange activations. Over NVLink — the datacenter interconnect on A100/H100 — that exchange is fast and tensor parallelism scales well. Over PCIe alone, the same exchange is far slower, and two consumer cards in tensor-parallel can deliver disappointing throughput because they spend their time talking to each other rather than generating. This is a concrete reason the datacenter premium can pay for itself at multi-GPU scale: you are buying the interconnect as much as the card. If a model fits on one card, prefer that to two smaller cards; if it genuinely needs multiple, the quality of the link between them is a first-order sizing input, not a detail.

The Capex Case a CFO Will Sign

Hardware sizing is not finished until it is an economic argument, because the objection to owning iron is never technical — it is "why not just rent it?" The honest answer is a total-cost comparison over the depreciation life of the card, and at steady utilization it favors ownership by a wide margin. A datacenter GPU amortized over three years against its all-in cost — card, power, cooling, rack, and operations — undercuts on-demand cloud GPU rental per hour of actual use once utilization is even moderately high, and undercuts per-token cloud API pricing dramatically at volume. The cloud-repatriation playbook works this economics in full; the sizing-specific point is that the crossover is a function of utilization, and inference workloads that run continuously sit well past it.

The number that makes the case, though, is amortization across models. A rented GPU-hour serves one workload for one hour. An owned card serves every model you deploy on it for its entire service life — a coding assistant, a document-analysis model, a classifier, and whatever open-weight model ships next quarter, all dividing the same fixed cost. That is the lever that turns a large capital number into a small per-workload one, and it is worth running through a TCO model with your real utilization and workload mix before the decision, so the capex case rests on your numbers rather than a vendor's. The card is a fixed asset that gets cheaper per unit of work every time you find another job for it.

Buying Iron Without Buying Lock-In

Owning hardware sounds like the opposite of optionality — capital committed, a fixed asset on the books — but bought correctly it is the more optional path, and this is the sovereignty argument made concrete in silicon. Commodity GPUs speaking CUDA, running open-source serving engines, loading open-weight models in open quantization formats, compose into a stack where every layer is replaceable. The model is a file swap. The serving engine is a container swap. The card, when a better one ships or the workload outgrows it, has a deep secondary market and a residual value — because you bought a general-purpose accelerator, not a locked appliance.

The anti-pattern is the turnkey inference appliance: a vendor-integrated box with proprietary firmware, a proprietary model format, and a support contract that is really a subscription. It demos well and it quietly reintroduces exactly the lock-in you self-hosted to escape — you cannot swap the model, cannot move the weights to a different card, and cannot exit without a forklift. The same optionality discipline that governs software choices governs hardware: prefer the commodity component talking a standard protocol over the integrated product that captures you. A rack of standard GPUs is an asset you control; an appliance is a contract you rent with extra steps.

The Long Game: Iron Outlasts Models

The deepest sizing principle is a mismatch of timescales. Open-weight models turn over every few months; a well-chosen GPU depreciates over three to five years. That asymmetry is not a problem to be solved but a strategy to be exploited: you are not buying hardware for the model in front of you, you are buying a unit of inference capacity — so many gigabytes of VRAM at so much bandwidth — that will serve a dozen models over its life. Size for the workload class (the concurrency, the context, the latency budget your service needs) rather than the specific weights, and the hardware stays correct as the models churn beneath it.

This is where hardware sovereignty stops being a slogan and becomes a balance-sheet fact. The team that owns its inference capacity absorbs the model churn as a free upgrade — each better open-weight release is a config change on hardware already paid for. The team renting inference pays the churn as a recurring cost forever, and pays it to a provider whose pricing it does not control. For regulated workloads the case is stronger still: when data residency is a legal requirement, not a preference, owning the iron is the only architecture that makes the residency guarantee a property of the network rather than a clause in a contract. The capacity plan and the compliance posture turn out to be the same document.