AMD ROCm as a Second Source for GPU Inference

Most risk registers carry a row for the cloud provider and one for the database, and nothing for the accelerator. Defensible while GPUs were a line item; not once allocation decides whether a roadmap ships.

Why the GPU Vendor Became a Single Point of Failure

The exposure reads better as a scheduling question than a pricing one. If one vendor's supply slips a quarter, which workloads stop? In our engagements that list is short — an internal assistant, a nightly scoring job, a retrieval tier — and naming it is most of the work. Written down it becomes a number, the same way any other lock-in is.

Qualifying a second vendor is not free, but the cost is not where people expect. Not the model: weights are safetensors and a serving stack loads them either way. Not really the framework. It sits where the stack stops being portable — fewer such places than the debate suggests.

Click to zoom

Reading AMD's Compatibility Matrix Without Misreading It

AMD's compatibility matrix contains a trap that will cost you an OS rollout. As of ROCm 7.14.0 it is one unified matrix covering Instinct, Radeon and Ryzen, grouping Instinct parts into four series: MI350 (CDNA 4, gfx950: MI355X, MI350X, MI350P), MI300 (CDNA 3, gfx942: MI325X, MI300X, MI300A), MI200 (CDNA 2, gfx90a: MI250X, MI250, MI210) and MI100 (CDNA, gfx908).

It renders as a flat table with one column per series. It is not one. Every cell carries a data-show-cond attribute in the page source naming the exact parts it applies to, and rows often have fewer cells than columns — so reading it without setting the part selector assigns a cell to whichever series sits at that index.

The Ubuntu row demonstrates it. One cell lists Ubuntu 26.04 (GA kernel 7.0), 24.04.4 (6.8) and 22.04.5 (5.15), conditioned on nine parts: MI355X, MI350X, MI325X, MI300X, MI300A, MI250X, MI250, MI210 and MI100. A second lists only 26.04 and 24.04.4, conditioned on MI350P alone. So MI300X is listed for 22.04.5 and MI350P is not — though MI350P shares a series with MI355X and MI350X, which are.

The other OS rows split differently again, which makes per-series reasoning unsafe rather than merely imprecise. RHEL 8.10 covers MI355X, MI350X, MI300X, MI300A, MI250X, MI250, MI210 and MI100 — not MI325X or MI350P. Debian 13 covers MI355X, MI350X, MI325X, MI300X and MI350P; MI300A, MI250X and MI250 get Debian 12 only; MI210 and MI100 get no Debian row. Rocky Linux 9 is listed for MI300X and MI300A alone.

The standalone ROCm on Radeon and Ryzen documentation covers releases only through ROCm 7.2.1 and says why: documentation unified across all supported hardware from ROCm Core SDK 7.13.0. A retired doc set, not a second shipping track.

And in the unified framework table, the MIGraphX and ONNX Runtime rows are conditioned on gfx950 and gfx942 only — the Instinct MI350 and MI300 targets — while the vLLM and PyTorch rows also reach Radeon targets. Inference tooling narrows as you move off Instinct: why a consumer card is not a drop-in second source for production serving.

The Driver Line Is Its Own Product

The seam that surprises teams most is the kernel driver, because few operators treat it as a separate artefact. AMD distributes amdgpu separately from the ROCm stack, under its own /amdgpu/ location at repo.radeon.com, with its own docs and version line: the amdgpu docs state that AMD GPU Driver 31.40.1 is compatible with ROCm 7.14.0.

Two independently scheduled upgrades is a feature if you plan for it and an incident if you do not. The driver is a reboot-class change gated by kernel policy; the runtime is a container image. Fusing them makes every runtime bump inherit a maintenance window.

How much skew is supported has recently moved. The live 7.14.0 matrix enumerates fourteen amdgpu versions — 31.40.1, 31.40.0, 31.30.0, 31.20.0, 31.10.0, 30.30.3, 30.30.2, 30.30.1, 30.30.0, 30.20.1, 30.20.0, 30.10.2, 30.10.1 and 30.10.0 — and unlike the OS rows it carries no per-part condition: the identical list prints for every Instinct part and repeats in the Radeon table.

A cleaner-sounding rule is quoted widely: from ROCm 6.4.0, driver and user space compatible up to a year apart, plus or minus two releases before that. It is real, but as of 7 August 2026 it survives only on the user/kernel space support matrix page — which carries a "This page has moved!" banner dated 15 July 2026 and a table stopping at ROCm 7.2.x. Dated evidence of how AMD has expressed the window, not a current guarantee. AMD also notes tools such as AMD SMI can have tighter driver dependencies, reading telemetry through the kernel directly.

#!/usr/bin/env bash
set -euo pipefail

# 1. Is the kernel module loaded, and at which version?
lsmod | grep -q '^amdgpu' || { echo "amdgpu module not loaded"; exit 1; }
modinfo amdgpu | awk -F': *' '/^version/ { print "amdgpu module: " $2 }'

# 2. Match this against the matrix row for YOUR PART, not your series.
. /etc/os-release && echo "os: $PRETTY_NAME   kernel: $(uname -r)"

# 3. What the driver presents. Under CPX one physical GPU enumerates as
#    several smaller ones — ground truth for partition mode, not the PO.
amd-smi list

# 4. Device nodes and the groups AMD's install page asks for.
stat -c '%n group=%G mode=%a' /dev/kfd /dev/dri/renderD*
id -nG "$USER" | tr ' ' '\n' | grep -Ex 'render|video' \
  || echo "missing: sudo usermod -a -G render,video \$LOGNAME"

That last check is AMD's documented host requirement: the installation page instructs sudo usermod -a -G render,video $LOGNAME, with a udev rule as the alternative. Both groups appear for a reason — video traditionally handles video device access, render manages GPU access through DRM render nodes.

Scheduling AMD GPUs on Kubernetes

The AMD GPU Operator is the supported path; its 1.5.1 documentation states the envelope: Kubernetes 1.29–1.36 on Ubuntu 22.04 and 24.04 LTS, 1.29–1.36 on Debian 12 with driver management explicitly not supported, OpenShift 4.16–4.22 on RHCOS, Helm v3.2.0+. If you would rather not run an operator, the device plugin alone deploys with kubectl create -f k8s-ds-amdgpu-dp.yaml.

The DeviceConfig custom resource holds the driver decision, and it is one boolean. spec.driver.enable: false uses the inbox or pre-installed driver; true installs the out-of-tree module, which also requires blacklist: true, a reboot, and the node labeller. If your nodes come from an immutable OS or golden image, the inbox path keeps the driver inside the artefact you already sign.

apiVersion: amd.com/v1alpha1
kind: DeviceConfig
metadata:
  name: instinct-inference
  namespace: kube-amd-gpu     # where the AMD GPU Operator runs
spec:
  driver:
    # false -> inbox / pre-installed driver, skip installation.
    # true  -> out-of-tree module; also needs blacklist: true, a reboot, the
    #          node labeller, and version: "30.20.1" (ROCm 7.1 scheme).
    enable: false

  devicePlugin:
    enableDevicePlugin: true  # cannot be enabled alongside the DRA driver
    # Docs default both images to :latest; pin what you tested.
    devicePluginImage: rocm/k8s-device-plugin:latest
    nodeLabellerImage: rocm/k8s-device-plugin:labeller-latest
    devicePluginArguments:
      # The resource NAME pods request. single -> amd.com/gpu everywhere;
      # mixed -> amd.com/<partition style>, and every limits key must change.
      resource_naming_strategy: single

  metricsExporter:
    enable: true              # defaults to false
    serviceType: "ClusterIP"

  selector:
    feature.node.kubernetes.io/amd-gpu: "true"

Now the field that quietly rewrites your manifests. AMD's device plugin documentation defines two naming strategies that fail in opposite directions. Under single, every GPU — whole or partitioned — is advertised as amd.com/gpu: 8 unpartitioned GPUs report amd.com/gpu: 8, the same node under CPX-NPS4 reports amd.com/gpu: 64. The name holds, the count moves — so replica math, quotas and count-derived autoscaler targets shift under an unchanged manifest. AMD documents single as supported on homogeneous but not heterogeneous nodes.

Under mixed, the name matches the partition style: that node reports amd.com/cpx_nps4: 64, and a heterogeneous node with 5 GPUs in SPX-NPS1 and 3 in CPX-NPS1 reports amd.com/spx_nps1: 5 alongside amd.com/cpx_nps1: 24. Anything hardcoding amd.com/gpu stops scheduling there. mixed is opt-in — absent resource_naming_strategy, the plugin defaults internally to single. Either way, a pod's limits key must name what the node actually advertises; the manifests here set single, so they request amd.com/gpu.

Partition mode is therefore a scheduling decision before a performance one — and AMD's own documentation sets disagree about the modes available. The workload optimization guide lists four compute modes (SPX, DPX, QPX, CPX) yielding on MI300X 8 XCDs / 192 GB, 4 / 96 GB, 2 / 48 GB and 1 / 24 GB per partition, each paired with an NPS memory mode, and names QPX + NPS4 most efficient for MI300X and MI325X, DPX + NPS2 for MI350X and MI355X. The amdgpu driver's MI300X partitioning overview documents three, with no QPX. Both read the same day; confirm with amd-smi.

Two constraints survive the disagreement. In CPX mode on MI300X, amd-smi reports 8 GPUs of 38 compute units and 24 GB HBM each — the figure a model must fit, not the 192 GB on the box. And memory partitions must be less than or equal to compute partitions. If you reason about MIG and time-slicing already, same discipline, different vocabulary.

Track a third path rather than adopting it: the Operator supports Dynamic Resource Allocation instead of the device plugin — scheduler-driven allocation, fine-grained selection, GPU sharing. The two cannot be enabled on one DeviceConfig, and the Operator's DRA driver requires Kubernetes 1.32 or later, with the DynamicResourceAllocation feature gate enabled on 1.32 and 1.33.

Serving: vLLM on ROCm and the Documented Gaps

If you already run vLLM in production, the serving tier is the least disruptive part. vLLM's GPU installation docs list ROCm support for MI200s (gfx90a), MI300 (gfx942), MI350 (gfx950), Radeon RX 7900 (gfx1100/gfx1101), RX 9000 (gfx1200/gfx1201) and Ryzen AI MAX / AI 300 (gfx1151/gfx1150), requiring ROCm 6.3+ — MI350 needs 7.0+, Ryzen AI MAX / AI 300 needs 7.0.2+.

Version pinning needs care, because two current documents are not the same statement. AMD's ROCm 7.14.0 AI-ecosystem table names vLLM 0.23 (Python 3.14, requiring PyTorch 2.11.0) alongside SGLang 0.5.13, PyTorch 2.12.0/2.11.0/2.10.0 and JAX 0.10.0/0.9.1. docs.vllm.ai/en/stable/ tracks vLLM's stable release on vLLM's own schedule. Pin against AMD's table for the ROCm stack, read vLLM's docs for feature behaviour; both were read on 7 August 2026.

The container surface is the visible difference. vLLM documents the ROCm docker run path with image vllm/vllm-openai-rocm and the flags --device /dev/kfd, --device /dev/dri, --group-add=video, --cap-add=SYS_PTRACE, --security-opt seccomp=unconfined and --ipc=host. Under Kubernetes with the Operator installed, do not translate that literally — the device plugin is the allocation path. What the pod spec still needs is the shared memory --ipc=host covers — and, if a scheduled container still sees no /dev/kfd, the video/render GID through securityContext.supplementalGroups.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference
  namespace: serving
spec:
  replicas: 2
  selector:
    matchLabels: { app: inference }
  template:
    metadata:
      labels: { app: inference }
    spec:
      containers:
        - name: vllm
          # Pin the tag you tested against your ROCm version.
          image: vllm/vllm-openai-rocm:pin-the-tag-you-tested
          args: ["--model=Qwen/Qwen3-8B", "--port=8000"]
          resources:
            limits:
              # Matches resource_naming_strategy: single above, so the count
              # moves with partitioning. Under mixed: amd.com/cpx_nps4, ...
              amd.com/gpu: 1
          ports:
            - { name: http, containerPort: 8000 }
          readinessProbe:
            httpGet: { path: /health, port: http }
            initialDelaySeconds: 60
          volumeMounts:
            - { name: dshm, mountPath: /dev/shm }
      volumes:
        # Pod-spec equivalent of docker's --ipc=host: the default 64Mi
        # /dev/shm throttles multi-process serving.
        - name: dshm
          emptyDir: { medium: Memory, sizeLimit: 16Gi }

Then the gaps, read off vLLM's own Feature-by-Hardware matrix rather than anyone's claim. In the column labelled AMD, encoder-decoder models and async output are marked unsupported. Chunked prefill, automatic prefix caching, LoRA, speculative decoding, CUDA graph, pooling, multimodal, logprobs, multi-step, best-of and beam-search are supported. A short exclusion list — but a rolling document. Check it against the release you intend to run.

Practically: if your serving tier is decoder-only text or multimodal generation with LoRA adapters — most of them, including anything built on adapters you fine-tuned yourself — nothing on that list touches you. If you run encoder-decoder models it does, and that workload stays put.

Observability and Audit Parity

A second source instrumented differently is a blind spot with GPUs in it. The reliable failure in a mixed fleet is not hardware — it is the AMD half landing on a dashboard nobody watches.

AMD's Device Metrics Exporter exposes GPU and NIC telemetry in Prometheus format — temperature, utilization, memory usage, power consumption — so it scrapes into the stack already collecting dcgm-exporter. Enable it in the same DeviceConfig as the device plugin, as above, and it ships with the rollout rather than as a follow-up ticket. Its own DRA GPU claim support is a separate Beta line at Kubernetes 1.34+ — an exporter feature, not the floor for DRA scheduling.

Do the alerting work at the point of scrape. Metric names differ between the exporters, so normalise into your own recording rules and alert against those — vendor-agnostic in name, vendor-specific only in expression. On a Prometheus stack you own that is a recording-rule change, not a second estate.

The audit angle repays five minutes. A GPU node's real configuration is three facts — amdgpu version, ROCm runtime version, current partition mode — and none appear in any Kubernetes object. Emit all three as node labels or exporter metadata and a reviewer can answer "what was this node running during the incident?" from the metrics store rather than from memory.

Designing the Exit Ramp in Both Directions

The point of a second source is not to move; it is to be able to, either direction, without a project. That property is designed in, and lost in exactly three places.

First, the resource name. A literal nvidia.com/gpu in a base manifest is a vendor commitment written where nobody reviews it — as is a replica count derived from whole-GPU assumptions. Second, the container device surface. Third, application code branching on a vendor rather than a capability.

The third is cheapest, because PyTorch solved it: PyTorch for HIP reuses the existing torch.cuda interfaces, so the vendor check is a version attribute, not a different API.

import torch


def accelerator_report() -> None:
    if not torch.cuda.is_available():
        print("no accelerator visible to this container")
        return

    # PyTorch for HIP reuses torch.cuda; the only place a vendor is named.
    if torch.version.hip:
        vendor, build = "AMD ROCm", torch.version.hip
    elif torch.version.cuda:
        vendor, build = "NVIDIA CUDA", torch.version.cuda
    else:
        vendor, build = "unknown", "unknown"

    print(f"vendor={vendor} build={build} devices={torch.cuda.device_count()}")

    for i in range(torch.cuda.device_count()):
        props = torch.cuda.get_device_properties(i)
        free, total = torch.cuda.mem_get_info(i)
        # ROCm exposes the gfx target; the fallback keeps this CUDA-safe.
        arch = getattr(props, "gcnArchName", f"sm_{props.major}{props.minor}")
        print(f"  [{i}] {props.name} arch={arch} "
              f"free={free / 1e9:.1f}GB of {total / 1e9:.1f}GB")


accelerator_report()

Note what it reports: memory per visible device. Under CPX that is 24 GB, not 192 GB — so it doubles as the check that your VRAM sizing survives a repartition.

For the first two, keep one vendor-neutral base and push every vendor-shaped line into an overlay: the base names no vendor — not in a resource key, an image, or a node selector.

# overlays/amd/kustomization.yaml
resources:
  - ../../base
images:
  - name: vllm/vllm-openai
    newName: vllm/vllm-openai-rocm
    # newTag: pin the tag you tested against your ROCm version
patches:
  - path: gpu-resource.yaml
    target:
      kind: Deployment
      name: inference
---
# overlays/amd/gpu-resource.yaml — under mixed naming this key becomes
# amd.com~1cpx_nps4, which is exactly why it lives in an overlay.
- op: remove
  path: /spec/template/spec/containers/0/resources/limits/nvidia.com~1gpu
- op: add
  path: /spec/template/spec/containers/0/resources/limits/amd.com~1gpu
  value: 1

Click to zoom

Run that flow per workload, and notice that "stay on NVIDIA" is a valid terminal state. The deliverable is not a migration. It is a written list: which workloads could move next quarter, which need a manifest change first, and which are pinned, each now carrying a reason rather than an assumption.

The Long Game: Second-Sourcing as a Standing Capability

Everything above has a shelf life. AMD is reorganising its documentation: pages canonical in June are moved-page stubs today, the Radeon and Ryzen set froze at 7.2.1, and two AMD pages disagree on how many compute-partition modes an MI300X exposes. Not a reason to wait — a reason to treat the check as recurring work.

Four pages, re-read at each ROCm release: the unified compatibility matrix (your exact part and OS, plus the enumerated driver list), the amdgpu driver's compatibility page, the GPU Operator's platform matrix and device-plugin semantics, and vLLM's Feature-by-Hardware matrix. An hour a quarter — the difference between knowing your second source works and believing it does.

Then rehearse it, because a capability never exercised is a claim. Keep one real workload on the second vendor permanently — not a benchmark, a workload with users. It costs a node, and converts assumption into observation on hardware you own. The team that has served production traffic on both vendors can second-source in a week. The team with a plan cannot.

That is the whole argument, and it is older than this hardware. Sovereignty at the compute layer is not owning a particular accelerator. It is holding the ability to change your mind about one — and the seams are cheapest to engineer before the quarter you need them.