QLoRA Infrastructure: Fine-Tuning You Actually Own

This looks like a build-versus-buy question and is not. Every other tier of a self-hosted AI stack can be rented and later reclaimed — inference moves, gateways move, vector stores move. Fine-tuning is different: the thing you hand over is not compute. It is the corpus.

The Tier You Cannot Rent Without Giving Up the Asset

Start with the question that precedes any fine-tuning project: knowledge problem, or behaviour problem? If the model needs facts that change — prices, policies, ticket state — the answer is retrieval, and an on-prem RAG tier beats a fine-tune you re-run every time the facts move. Fine-tuning earns its keep on the other axis — house vocabulary, output structure, refusal behaviour, register — which context stuffing never makes consistent.

On that axis the infrastructure question follows immediately, and it is not about GPUs. Training means presenting the model with the material that defines the behaviour: transcripts, resolved tickets, internal prose, annotated decisions. That corpus holds, unredacted, most of what your compliance function spends the year defending — and a managed service takes custody of it by construction, because it cannot train without a copy.

Custody does not reverse. Migrate off, delete the project, collect the attestation: the disclosure has still happened. The adapter may be portable; the upload is not. That asymmetry, not the GPU rate, is why this tier belongs on infrastructure you control.

What QLoRA Actually Changed About the Hardware Bill

The economics that once made that argument academic no longer hold. Full fine-tuning updates every weight and carries optimizer state for each. LoRA broke half of that: freeze the base, train a pair of small low-rank matrices instead. The original LoRA paper reports it "can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times" with "no additional inference latency" versus adapter approaches — measured against GPT-3 175B with Adam, so read the ratios as the shape of the saving, not a figure for your model.

QLoRA broke the other half by quantizing the frozen base. The 2023 paper reports it "reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance" — a result on that year's stack, not a guarantee about yours. Three mechanisms do the work: 4-bit NormalFloat, which the authors call information-theoretically optimal for normally distributed weights; double quantization, which quantizes the quantization constants; and paged optimizers, which absorb the memory spikes that end a run.

For sizing, Unsloth publishes a VRAM table listing 6 GB for 4-bit QLoRA at 8B against 22 GB for 16-bit LoRA, and 41 GB against 164 GB at 70B, with a minimum CUDA capability of 7.0. Read those as vendor-published floors, not measurements: the same docs call them "the absolute minimum", and the project's issue tracker carries reports of runs consuming considerably more. They establish that an 8B fine-tune is a single-owned-GPU problem — they are not a procurement number. Benchmark your own sequence length, batch size and checkpointing, as you would when sizing an inference tier.

The library knobs fit on one screen. PEFT's LoraConfig documents r as the LoRA attention dimension — the rank — lora_alpha as the scaling factor, and target_modules: all-linear as targeting every linear module, the output layer excluded for a PreTrainedModel:

import torch
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

BASE = "Qwen/Qwen3-8B"  # Apache-2.0, so the licence follows nothing downstream

quant = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # 4-bit NormalFloat, per the QLoRA paper
    bnb_4bit_use_double_quant=True,     # quantize the quantization constants
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=quant)
model = prepare_model_for_kbit_training(model)

lora = LoraConfig(
    r=32,                          # rank: the adapter's capacity
    lora_alpha=16,                 # scaling factor applied to the update
    lora_dropout=0.05,
    target_modules="all-linear",   # output layer excluded on a PreTrainedModel
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora)

trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"trainable {trainable:,} / {total:,} = {100 * trainable / total:.4f}%")

The Training Run as a Kubernetes Job, Not a Notebook

Almost every QLoRA tutorial is a notebook — a fine way to learn the technique and a poor way to own it. A notebook records no dataset revision, has no queue behind it, and leaves no artifact anyone can reproduce. What follows is the same technique with the three properties that make it infrastructure: a version-controlled config, a quota-governed Job, an addressable artifact.

Click to zoom

The config comes first, because it is what makes the run repeatable. Axolotl's config reference documents adapter as the key selecting a built-in or plugin adapter method — left blank, it trains all parameters of the original model — and load_in_4bit as the bitsandbytes 4-bit switch. For QLoRA the value is the literal string qlora. The key that trips people up is lora_model_dir: it points at an already-trained LoRA, so it stays empty.

# Base model choice is a licensing decision before it is a quality one.
# Qwen3-8B is Apache-2.0: nothing in the licence follows the adapter downstream.
base_model: Qwen/Qwen3-8B

load_in_8bit: false
load_in_4bit: true          # bitsandbytes 4-bit — the Q in QLoRA

datasets:
  - path: /data/corpus/support-triage-2026-07.jsonl
    type: chat_template

dataset_prepared_path: /data/prepared
val_set_size: 0.05
output_dir: /artifacts/qlora-support-triage

adapter: qlora              # the adapter METHOD, not a filesystem path
lora_model_dir:             # empty on a fresh run; set only to resume or test

sequence_len: 4096
sample_packing: true

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true    # target all linear modules

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 3
optimizer: paged_adamw_32bit
learning_rate: 0.0002
bf16: auto
gradient_checkpointing: true

warmup_steps: 20
saves_per_epoch: 1

Then the run. GPUs are the scarcest thing in the estate and training jobs the greediest consumers, so admission must be governed or the first long fine-tune starves everything else. Kueue is the Kubernetes-native answer: it "manages quotas and how jobs consume them" and decides "when a job should wait, when a job should be admitted to start" and when it should be preempted. Read the boundary carefully — Kueue deliberately does not do pod-to-node scheduling or autoscaling, which stay with kube-scheduler and cluster-autoscaler. It offers StrictFIFO and BestEffortFIFO strategies, Fair Sharing across cohorts, and all-or-nothing with ready Pods.

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: gpu-a100-80g
spec:
  nodeLabels:
    nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: training-queue
spec:
  namespaceSelector: {}
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu"]
      flavors:
        - name: gpu-a100-80g
          resources:
            - name: "nvidia.com/gpu"
              nominalQuota: 4
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  name: ml-team-queue
  namespace: ml-training
spec:
  clusterQueue: training-queue
---
apiVersion: batch/v1
kind: Job
metadata:
  generateName: qlora-support-triage-
  namespace: ml-training
  labels:
    kueue.x-k8s.io/queue-name: ml-team-queue
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: train
          image: registry.internal.example/ml/axolotl:2026.07
          command: ["axolotl", "train", "/config/qlora-support-triage.yaml"]
          resources:
            limits:
              nvidia.com/gpu: 1
              cpu: "16"
              memory: 128Gi
          volumeMounts:
            - { name: corpus, mountPath: /data/corpus, readOnly: true }
            - { name: config, mountPath: /config, readOnly: true }
            - { name: artifacts, mountPath: /artifacts }
      volumes:
        - name: corpus
          persistentVolumeClaim:
            claimName: support-corpus
            readOnly: true
        - name: config
          configMap:
            name: qlora-support-triage
        - name: artifacts
          persistentVolumeClaim:
            claimName: adapter-artifacts

Two details carry the pattern. The corpus is mounted read-only, so a run cannot mutate the asset it reads. And you do not set suspend yourself: Kueue's Job integration manages suspension by webhook. A training run also wants a whole GPU — the partitioning that suits mixed inference is the wrong tool here.

Adapters Are Build Artifacts: Registry, Digest, Provenance

When the Job finishes you hold two small files — adapter_model.safetensors and adapter_config.json — on a PVC. In most organisations that is where it stops, and months later nobody can say which base checkpoint production was trained against. The adapter is a build output; treat it like one.

Adapters are tens of megabytes, so the registry you already run for images can hold them as OCI artifacts: no new storage tier, no new access-control model, no new backup path. If you run Harbor or Zot you already have replication, retention, RBAC and an audit trail — apply them to model artifacts and provenance becomes infrastructure rather than convention.

#!/usr/bin/env bash
set -euo pipefail

REGISTRY="registry.internal.example"
REPO="ml/adapters/support-triage"
TAG="v4"
ART="/artifacts/qlora-support-triage"

# The base-model revision makes the adapter reproducible: record the exact
# commit the run trained against, not a moving branch name.
BASE_MODEL="Qwen/Qwen3-8B"
BASE_REVISION="$(cat "${ART}/base_model_revision.txt")"
CONFIG_HASH="$(sha256sum /config/qlora-support-triage.yaml | cut -d' ' -f1)"
DATASET_REV="support-triage-2026-07"

oras push "${REGISTRY}/${REPO}:${TAG}" \
  --artifact-type application/vnd.stribog.lora.adapter.v1 \
  --annotation "ai.base-model=${BASE_MODEL}" \
  --annotation "ai.base-model-revision=${BASE_REVISION}" \
  --annotation "ai.training-config-sha256=${CONFIG_HASH}" \
  --annotation "ai.dataset-revision=${DATASET_REV}" \
  --annotation "ai.adapter-rank=32" \
  "${ART}/adapter_model.safetensors:application/octet-stream" \
  "${ART}/adapter_config.json:application/json"

# Consume by digest, never by tag: a tag is a pointer someone can move.
DIGEST="$(oras manifest fetch --descriptor "${REGISTRY}/${REPO}:${TAG}" \
  | python3 -c 'import json,sys; print(json.load(sys.stdin)["digest"])')"

oras pull "${REGISTRY}/${REPO}@${DIGEST}" --output /srv/adapters/support-triage-v4
oras manifest fetch "${REGISTRY}/${REPO}@${DIGEST}" | python3 -m json.tool

This is a pattern we recommend, not a ratified standard — no specification says a LoRA adapter must be an OCI artifact with these annotation keys. Adopt the reasoning, not the names: whatever you need to reconstruct the adapter belongs in the manifest, and what you deploy is a digest.

Serving: Many Adapters, One Base Model, One GPU

The serving economics make per-team customization viable. Because LoRA leaves the base weights untouched, one loaded base model serves many adapters at once — the marginal cost of the fourth fine-tune is a slot, not another GPU. If you already run vLLM in production, this is a flag change, not a new deployment.

Click to zoom

The flag semantics repay precision, because the names invite the wrong mental model. vLLM's LoRA feature docs cover --enable-lora, --lora-modules, and --max-lora-rank, which should be set to the maximum rank across your adapter set because oversizing wastes memory. The other two live in the LoRAConfig API reference: max_loras is the maximum number of LoRAs in a single batch, not a ceiling on registrations, and max_cpu_loras is how many are held in CPU memory, which must be greater than or equal to max_loras or vLLM raises a ValueError. max_lora_rank defaults to 16.

#!/usr/bin/env bash
set -euo pipefail

# Verified against vLLM 0.26.0.
#   --max-lora-rank   per-adapter rank ceiling (default 16); oversizing wastes VRAM
#   --max-loras       max LoRAs in a single BATCH, not the number registered
#   --max-cpu-loras   host-memory pool; must be >= --max-loras or vLLM ValueErrors
export VLLM_ALLOW_RUNTIME_LORA_UPDATING=True

vllm serve Qwen/Qwen3-8B \
  --enable-lora \
  --max-lora-rank 32 \
  --max-loras 4 \
  --max-cpu-loras 16 \
  --lora-modules \
    support-triage=/srv/adapters/support-triage-v4 \
    contract-clause=/srv/adapters/contract-clause-v2

# Register a freshly promoted adapter without restarting the server.
curl -sS -X POST http://localhost:8000/v1/load_lora_adapter \
  -H 'Content-Type: application/json' \
  -d '{"lora_name": "release-notes", "lora_path": "/srv/adapters/release-notes-v1"}'

# Route a request to one adapter by naming it as the model.
curl -sS http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "support-triage", "prompt": "Triage: card declined at checkout"}'

# Reclaim the slot with POST /v1/unload_lora_adapter when the adapter retires.

The runtime load endpoint deserves a moment before you enable it estate-wide. VLLM_ALLOW_RUNTIME_LORA_UPDATING turns an inference server into something that loads weights from a path for whoever can reach it — an authorization decision, not a convenience flag. Keep it off the tenant-facing listener; a self-hosted model gateway is one place to enforce that split.

Failure Modes That Actually Bite

The technique is well-trodden; the interesting failures are operational. Five recur:

• Rank inflation. Rank is capacity, and more is not better — a higher r costs VRAM in every serving slot and overfits a small corpus more easily. It is load-bearing across tenants: --max-lora-rank must fit the largest adapter in the set, so one team's rank-128 experiment taxes everyone's memory.
• Chat-template mismatch. The most common cause of a fine-tune that trained beautifully and serves badly. If the template that formatted the training examples differs from the one the serving stack applies, the model sees a prompt shape it never learned. Pin it with the adapter.
• Base drift. An adapter is a delta against specific weights. Change the base checkpoint, or the quantization it loads under, and you apply that delta to something else — the result may look fine and be measurably worse. That is why the base revision belongs in the artifact.
• Silent eval regression. Held-out loss falling proves the model learned your corpus, not that it beat the alternative. Compare against the unmodified base at every promotion; the fine-tune that sounds more in-house while answering slightly worse is the one that survives.
• Startup misconfiguration. Setting --max-cpu-loras below --max-loras is not a degraded mode — vLLM raises a ValueError. Catch it in a smoke test: it fails at boot, taking the replica with it.

Does Fine-Tuning Make You a GPAI Provider?

For EU teams this is the question that stalls the project, and it is answerable from primary sources. The Commission's guidelines on the scope of obligations for providers of general-purpose AI models were published on 18 July 2025, ahead of entry into application on 2 August 2025.

The guidelines treat fine-tuning as "one way of 'modifying' a general-purpose AI model", and note that who controls the weights may be an important factor in deciding who the modifier is — itself an argument for holding your own. The governing test is qualitative: a downstream modifier becomes the provider of the modified model "only if the modification leads to a significant change in the model's generality, capabilities, or systemic risk". Alongside it sits an explicitly *indicative* criterion — training compute for the modification exceeding a third of the original model's. Where the original figure is unknown, the fallback is a third of 10^25 FLOP for systemic-risk models and a third of 10^23 FLOP otherwise.

That criterion is computable, because Annex A.2.2 of the same document supplies the formula: training compute may be approximated as C ≈ 6 · P · D, where P is total parameters and D total training tokens. Apply it to a fine-tune of the same model — our arithmetic, not the Commission's — and P cancels, so the threshold collapses to a third of the base model's pretraining token budget. Qwen3 was pre-trained on 36 trillion tokens, putting the line near 12 trillion. A 20-million-token domain corpus sits about six orders of magnitude below it (20e6 / 12e12 ≈ 1.7e-6) — and the assumption is deliberately unkind to us, charging the QLoRA run a full dense pass it never incurs.

Run the calculation anyway, because it shows what is at stake if you cross. A modification meeting the threshold brings the Article 53(1)(a) and (b) transparency obligations, and the Commission notes it can be expected to have used a significant amount of data, which may be relevant to the Article 53(1)(c) copyright policy and 53(1)(d) training-content summary obligations. Documentation duties — cheap when the pipeline already emits provenance, ruinous when it does not.

Exit Ramps: What Stays Portable When the Base Model Moves

Be honest about which artifact is portable. An adapter is bound to the weights it trained against and to that base model's licence, which makes it the *least* durable thing the pipeline produces. The durable assets are the corpus, the training config and the eval set: with those, reproducing the adapter against a different base is a scheduled Job. Without them it is a research project.

Licence terms are why this is not hypothetical. All Qwen3 models are publicly accessible under Apache 2.0, so nothing propagates to what you build. Compare the Llama 3.3 Community License: display "Built with Llama", prefix derivative model names with "Llama", and above 700 million monthly active users request a licence Meta "may grant to you in its sole discretion". Workable terms that follow your derivative's distribution. A permissive base is the cheapest exit-ramp decision here, and it is made once.

Price the exit as you would any other lock-in: moving base models costs one retrain plus one eval cycle per adapter, in GPU-hours and engineer-days — a number you can write down today. Then rehearse it. Re-run one adapter against a newer base each quarter; it costs a Job, and turns the exit from an assertion into something you have done.

The Long Game: The Dataset Is the Compounding Asset

Base models will keep improving and the adapter you ship this quarter will be obsolete within a year. That is fine — it was always the disposable half. What compounds is the curated corpus, the eval set that encodes what "good" means in your domain, and the ability to run the loop. Each is worth more after five years than on the day it was built.

So the durable question is not which model to fine-tune. It is whether, in five years, you still hold the material that makes customization possible and the pipeline that makes it routine. Build the boundary first, keep every artifact inside it addressable by digest, and the model generation becomes an implementation detail — which is what it should have been.