On-Premise AI: The Decision Order for the Whole Stack

The build order is physical and obvious. Silicon arrives and gets racked. A serving engine goes on top of whatever arrived. Sharing appears when the second team complains about queueing, a vector store when answers turn out wrong, a gateway when finance asks who spent what, attestation only if a regulator names it. Each step answers the one before — and together they fix the hardest constraints in a purchase order written before anyone said what the platform is for.

The sequencing error that makes on-premise AI expensive

There is no rack unit for a workload contract. Nothing in the build order forces anyone to write down the data class, the latency the business accepts, the concurrency it must survive, or the residency boundary. So it is not written, and every layer below gets decided backwards from hardware already bought.

The build order runs floor upward; the decision order starts above the stack, at the workload — the only thing that can say whether the lower layers are needed at all and in what shape. What follows is one pass down that order, each section linking to the deep treatment of its layer. Each layer gets a gate question and an artifact that closes it; skip the artifact and buying the next layer only makes the missing decision costlier to reverse.

Click to zoom

Layer 0: write the workload contract before you buy silicon

Gate: can you state in a file what data class this platform handles, what latency it owes, what concurrency it absorbs, and where the residency boundary sits? Artifact: a committed workload contract every later layer reads.

This is not a planning document. It is a small machine-readable file beside the manifests that converts arguments into lookups: concurrency becomes a number a serving flag consumes, context length an input to a VRAM calculation, residency a boolean a gateway policy enforces. Write it as plain data with an owner and a date — versioned, reviewed like code, cited by the layers below.

# workload-contract/clinical-summariser.yaml
#
# Plain YAML your own tooling reads: no CRD, no operator, hence no apiVersion
# and no kind. Values illustrative — fill from your own classification.
name: clinical-summariser
owner: platform-ai
reviewed_on: 2026-08-30

data:
  class: special-category-personal-data
  may_leave_residency_boundary: false
  residency_boundary: tenant-owned-racks-eu

latency:
  time_to_first_token_p95_ms: 800
  end_to_end_p95_ms: 6000

load:
  peak_concurrent_requests: 40
  sustained_concurrent_requests: 12

tokens:
  prompt_p95: 6000
  output_p95: 700
  max_context: 16000

exit:
  api_surface: openai-compatible
  weights_mirrored_inside_boundary: true
  second_source_engine_evaluated: true

Layer 1: silicon, and the two numbers that actually size it

Gate: does the VRAM working set fit, and at what memory bandwidth? Artifact: a sizing calculation someone else can reproduce from the contract.

Two numbers decide this layer, and neither is the headline FLOPS figure. The first is the VRAM working set: weights at your chosen precision plus the key-value cache, which grows with context length multiplied by concurrent sequences. Both inputs are in the contract: arithmetic once Layer 0 exists, guesswork before it. The full sizing method beats rounding up to the largest card in the quote.

The second is memory bandwidth. Token-by-token decoding is dominated by moving weights out of memory, not arithmetic on them, so two cards with similar compute figures and different memory subsystems produce visibly different interactive latency. A card chosen on compute alone optimises the half users do not feel.

Second-sourcing belongs here too, as a checkable answer rather than a slogan: does the engine you pick publish a supported build for a second vendor's runtime? Treating an alternative GPU stack as a real second source means testing that while you still have leverage.

Layer 2: serving, where the API surface is the real decision

Gate: which API surface does this engine expose to callers? Artifact: a pinned endpoint contract that applications bind to instead of binding to the engine.

Engines get replaced. Endpoints do not, because every application, notebook and evaluation harness is wired to them — which makes the API surface the durable asset and the engine an implementation detail.

vLLM is the common answer for production concurrency, and the reason is as much interface as throughput: its OpenAI-compatible server covers the Completions, Chat Completions, Chat Completions batch, Responses and Embeddings APIs at their standard /v1 paths. Anything written against that surface ports to any other implementation. The engine comparison is worked through separately, as is running it on Kubernetes properly.

One flag matters more than it looks, because it shapes Layer 4. vllm serve carries `--enable-lora`, which turns on handling of LoRA adapters: adapter serving is a server-level toggle, not a second deployment. Decide it here and fine-tuning later is a weights question, not an infrastructure one.

#!/usr/bin/env bash
# Layer 0 supplies these two directly:
#   --max-model-len <- tokens.max_context            (16000)
#   --max-num-seqs  <- load.peak_concurrent_requests (40)
# --tensor-parallel-size is Layer 1 topology: the GPU count for THIS host.
#   Set 1 (or drop it) on a single card; 2 here fails at startup.
# --gpu-memory-utilization is headroom, --enable-lora the Layer 4 toggle.
set -euo pipefail

vllm serve Qwen/Qwen3-32B \
  --served-model-name clinical-summariser \
  --max-model-len 16k \
  --max-num-seqs 40 \
  --gpu-memory-utilization 0.90 \
  --tensor-parallel-size 2 \
  --enable-lora \
  --max-lora-rank 16

Layer 3: partitioning, from MIG profiles to DRA

Gate: hardware isolation, or scheduler-level allocation? Artifact: a claim describing the device you need rather than naming the one you were given.

Multi-Instance GPU partitions the card itself, and its arithmetic has a trap. NVIDIA defines an SM slice as roughly one seventh of the GPU's streaming multiprocessors in MIG mode — but memory is cut on a different denominator. On B200 the `MIG 1g.23gb` profile takes 1/8 of GPU memory and 1/7 of the SMs. The same profile is "an eighth of a card" or "a seventh" depending which resource you count: size against whichever your workload exhausts first. The comparison against time-slicing covers when isolation earns its rigidity.

The scheduler side moved. Dynamic Resource Allocation is enabled by default from Kubernetes 1.34, with the stable `v1` API as the default version rather than the v1beta1 and v1beta2 versions that needed opt-in, and Kubernetes documents it as stable since v1.35 with the `DynamicResourceAllocation` gate locked: on 1.35 or later there is nothing to enable and a value set for that gate is ignored without error. Check the server version first: YAML idiomatic on 1.34 fails closed on 1.33.

DRA changes the direction of the request: a workload specifies the properties of the devices it needs and leaves the scheduler to allocate actual devices. That is optionality as much as utilisation: a claim against declared attributes survives a hardware refresh that a pinned card name does not.

# REQUIRES KUBERNETES 1.34+. From 1.34 DRA is on by default and
# resource.k8s.io/v1 is the default API version; on 1.35+ the
# DynamicResourceAllocation gate is locked. Pre-1.34, these fail closed.
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: inference-gpu
spec:
  selectors:
    - cel:
        expression: |-
          device.driver == "gpu.example.com"
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: clinical-summariser-gpu
  namespace: inference
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: inference-gpu
            # Describe the device; do not name it. Attribute keys are
            # driver-specific: kubectl get resourceslices
            selectors:
              - cel:
                  expression: |-
                    device.attributes["gpu.example.com"].memoryGiB >= 80
---
apiVersion: v1
kind: Pod
metadata:
  name: clinical-summariser
  namespace: inference
spec:
  containers:
    - name: vllm
      # Pin by digest from a registry you mirror; tag shown for readability.
      image: vllm/vllm-openai:v0.28.0
      resources:
        claims:
          - name: gpu
  resourceClaims:
    - name: gpu
      resourceClaimTemplateName: clinical-summariser-gpu

Layer 4: retrieval before adaptation, always in that order

Gate: has retrieval been measured before adaptation is considered? Artifact: a labelled evaluation set with a retrieval score on it.

The most expensive detour in on-premise AI starts with a reasonable-sounding sentence: the model does not know our domain, so fine-tune it. Often the failing answers are not about style or format but about facts, and the facts are wrong because the right document never reached the context window. The two fix different faults. Adaptation changes how a model behaves: tone, structure, schema adherence, vocabulary. It cannot keep a corpus current — a fine-tune ships a snapshot, so a corpus that changes weekly is stale the day the model lands, at the cost of a retraining pipeline you operate permanently.

The stop condition is cheap and almost nobody applies it. Take fifty questions the system answers wrong and record whether the passage containing the correct answer reached the context at all. If it usually did not, this is a retrieval problem, and the fix belongs in chunking, embeddings and the vector store. If it usually did and the answer was still wrong, you have a genuine adaptation case: QLoRA on infrastructure you own, served through the adapter toggle enabled at Layer 2.

Layer 5: the gateway is a control plane, not a proxy

Gate: are residency, budget and audit enforced at one choke point? Artifact: a routing policy a reviewer can read end to end.

A gateway usually arrives as plumbing — somewhere to hold API keys and even out retries — which is why it arrives late and underpowered. Its real job is the one place a rule about data is stated once and enforced for every caller — including ones written next year by people who never read the policy.

Three properties deserve deliberate design. Residency: a model entry that must never receive regulated payloads should be unreachable for that class of request, not discouraged in a wiki. Budget: per-team spend and rate limits belong where requests are counted. Audit: one log in one schema is the difference between answering "what has this system seen?" in an afternoon and never. A self-hosted gateway there makes residency testable rather than aspirational.

# EXCERPT — placeholders must be replaced before use.
model_list:
  - model_name: clinical-summariser      # only route cleared for
    litellm_params:                      # special-category data
      model: openai/clinical-summariser
      api_base: <INTERNAL_VLLM_ENDPOINT> # inside the residency boundary
      api_key: <VLLM_API_KEY_REF>
  - model_name: general-drafting
    litellm_params:
      model: openai/general-drafting
      api_base: <INTERNAL_VLLM_ENDPOINT>
      api_key: <VLLM_API_KEY_REF>

general_settings:
  master_key: <MASTER_KEY_REF>
  database_url: <POSTGRES_DSN_REF>

litellm_settings:
  # One log, one schema: answers "what has this system seen?"
  success_callback: ["<AUDIT_SINK>"]
  failure_callback: ["<AUDIT_SINK>"]
  # Ceiling only. Routing and per-team spend live on the key/team object:
  # models=[...] plus its own max_budget.
  upperbound_key_generate_params:
    max_budget: <MONTHLY_CEILING>
    budget_duration: 30d

Layer 6: attestation, and the vendor call inside the perimeter

Gate: can this hardware be attested without a call to a vendor-operated service? Artifact: an attestation report with its verification path recorded as evidence.

NVIDIA defines attestation as cryptographically verifying claims about hardware and software to establish trust between parties — independent confirmation that systems are authentic, unmodified and operating as intended. Valuable for a platform whose premise is that regulated data stays inside a boundary you control, and also the easiest place to hollow that premise out unnoticed.

NVIDIA's Attestation Suite is composed of three named services: the Remote Attestation Service (NRAS), the Reference Integrity Manifest (RIM) Service and the NVIDIA OCSP Service. Its documentation index groups those three under a Cloud Services heading — a reason to ask the question, not an answer to it, since the same index also lists local Client Tools. Do not read "cloud-only" out of a navigation label — or assume the opposite. Establish which verification path your deployment uses, whether it crosses your residency boundary, and what happens if it is unreachable in an incident. The broader confidential-computing picture covers what a TEE does and does not give you.

Click to zoom

What the amended AI Act dates change about sequencing

The Digital Omnibus on AI, Regulation (EU) 2026/1744, moved the high-risk application dates; those dates and what they mean for a build plan are covered in full. In short: Chapter III Sections 1, 2 and 3, with the exception of Article 6(5), now apply from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I. The stated reason is delayed standards, specifications, guidance and national competent authorities — not a reduction in the obligations.

Two precision points are routinely got wrong. First, 2 August 2030 is neither new nor a general public-authority application date. Regulation (EU) 2024/1689 Article 111(2) already carried that sentence, and it binds only providers and deployers of high-risk AI systems intended to be used by public authorities. The rest of that paragraph is the grandfathering rule: already-on-market high-risk systems come into scope only on a significant design change, and do not inherit the 2030 date. 2026/1744 replaces the paragraph, changing only the grandfathering trigger and restating the 2030 sentence unchanged. A new high-risk system does not get to plan against it. Second, the amended Article 113 point (a) keeps Chapters I and II applying from 2 February 2025 and sets 2 December 2026 only for the newly inserted Article 5(1) points (ba) and (bb) and Article 5(1a) and (1b) — the original Article 5 prohibitions are already in force.

The sequencing consequence is narrow: none of this changes the decision order. A later date is runway for building the evidence that order produces, not a reason to defer Layer 0.

Failure modes: what breaks when a layer's gate is skipped

Each skipped gate has a characteristic end-state, and each end-state a measurement that catches it while it is still cheap. These follow from the mechanics above, not any deployment sample.

• The idle cluster. Layer 0 skipped, silicon bought against a hoped-for workload. Detection: allocation versus utilisation, tracked separately per GPU — fully allocated and lightly utilised is a Layer 0 failure in a capacity costume.
• The unpartitionable GPU. Layer 3 decided after the fact: the card is in the wrong mode for the profile you need, or claims name devices that no longer exist. Detection: pending pods citing an unsatisfiable device request, and any manifest pinning a card name.
• The fine-tune that could not fix a retrieval gap. Layer 4 out of order: a retraining pipeline is in production, costs recur, and answers are still wrong about anything changed since the snapshot. Detection: the fifty-question audit, run before any weights are touched.
• The ungoverned gateway. Layer 5 added as plumbing: applications hold their own keys and call the engine directly, so residency is only a convention. Detection: count the client identities the serving engine sees; more than the gateway's routes means the choke point is not one.

The pattern is the same in all four: the symptom shows where the money is visible, the cause sits at a layer never gated, and fixing the symptom buys more of it.

Exit ramps and the long game

Sovereignty that cannot be exercised is a preference. What makes a layer replaceable is a checkable artifact, not an appeal to open source.

• Silicon. A dated evaluation run on a second vendor's runtime, not a claim that one exists.
• Serving. An application speaking only the OpenAI-compatible endpoint set, so the engine swaps in a config change.
• Partitioning. Claims against declared device properties, which survive a refresh that pinned names do not.
• Retrieval and adaptation. Embeddings you can regenerate, adapter weights you hold.
• Gateway. A routing policy and audit schema that are yours, so replacing the gateway does not replace the evidence.
• Attestation. A verification path you have tested, including what happens when the far end is unavailable.

Model licensing is where optionality is most often assumed rather than checked. Meta's community licences are not unconditional open-source grants, and their commercial condition is widely mis-stated as a growth threshold. It is not. Under the Llama 3.3 Community License, and identically under the Llama 4 Community License whose version effective date is 5 April 2025, the test is a snapshot: if, on that model version's release date, the licensee's monthly active users in the preceding calendar month exceeded 700 million, they must request a licence from Meta, granted at Meta's sole discretion. Crossing that number later is not the trigger.

Nor is that clause the whole licence: the agreement also requires a "Built with Llama" attribution, prefixes any distributed model you improve with "Llama", mandates a notice file on redistribution, incorporates an Acceptable Use Policy by reference, and names a different Meta entity as counterparty depending where you are established. Read the notices for the exact weights you deploy. By contrast, Qwen3-32B is published under Apache-2.0, with no user-count condition at all.

"""Layer 2's exit ramp, exercised through Layer 5.

No vendor SDK, no engine-specific call. Point INFERENCE_BASE_URL at the
gateway, a vLLM pod, or any other implementation of the same endpoints:
this file does not change.
"""

import os

from openai import OpenAI

client = OpenAI(
    # e.g. http://gateway.inference.svc:4000/v1
    base_url=os.environ["INFERENCE_BASE_URL"],
    api_key=os.environ["INFERENCE_API_KEY"],
)

response = client.chat.completions.create(
    model=os.environ.get("INFERENCE_MODEL", "clinical-summariser"),
    messages=[
        {"role": "system", "content": "Summarise. Use only supplied text."},
        {"role": "user", "content": "..."},
    ],
    max_tokens=700,
)

print(response.choices[0].message.content)

The long game is not that on-premise is cheaper — that depends on utilisation. It is that a platform decided in this order absorbs change. A model family gets a worse licence and you move, because the application binds to an endpoint. A regulator names a date and the artifacts exist, because each gate produced one. A card generation ends and the claims still resolve, because they describe properties rather than part numbers. That is sovereignty engineered rather than asserted: a stack whose layers replace one at a time.