Slurm vs Kubernetes for GPU Work: Batch Queue or Service Fleet

GPU capacity bought for training rarely stays a training cluster. Once a model ships, the same nodes are asked to serve it, and the scheduler chosen for one job shape turns out to be wrong for the other. That is the real question behind most Slurm-versus-Kubernetes searches, and feature grids do not answer it.

This article maps each Slurm primitive to its Kubernetes counterpart, says where the counterpart is weaker or still beta, and covers the axes vendor explainers skip: upgrade order and accounting downtime, what each system shows an auditor, and who now owns Slurm's main developer.

Short answer: different layers, one pool of GPUs

The cleanest statement of the split comes from SchedMD's own slurm-bridge README: Kubernetes "excels at scheduling workloads that run for an indefinite amount of time, with potentially vague resource requirements, on a single node, with loose policy", while Slurm "excels at quickly scheduling workloads that run for a finite amount of time, with well-defined resource requirements and topology, on multiple nodes, with strict policy, and a known resource pool."

Read that as a description of two workloads, not two products. A multi-node pre-training run is finite, topology-sensitive and policy-bound. An inference endpoint behind a vLLM deployment is indefinite, scales with traffic and must be restarted when it dies. One pool of hardware, two job shapes — and the scheduler you pick decides which shape is a first-class citizen.

Click to zoom

What Slurm actually does with a GPU job

Slurm's overview lists three functions: it "allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time", provides a framework for starting and monitoring work on the allocated nodes, and "arbitrates contention for resources by managing a queue of pending work." Every other behaviour follows from those three. A job asks for nodes and a duration, waits in a queue, and receives an allocation it holds until it finishes or its time runs out.

GPUs enter through generic resources (GRES). The GPU plugin sets CUDA_VISIBLE_DEVICES "for each job step to determine which GPUs are available for its use on each node", so a process sees only what it was granted. With AutoDetect=nvml in gres.conf, configuration details are filled in for any detected GPU, "though the Gres= line in slurm.conf is still required in order to tell slurmctld how many GRES to expect." MIG instances are treated as individual GPUs with cgroup isolation, but Slurm "expects MIG devices to already be partitioned, and does not support dynamic MIG partitioning" — partition first, as the MIG and time-slicing guide describes, then let Slurm count slices.

# excerpt — not a complete slurm.conf; site-specific lines elided as "# ..."
# --- slurm.conf ---
SelectType=select/cons_tres
GresTypes=gpu
PriorityType=priority/multifactor
PriorityDecayHalfLife=7-0
# ...
NodeName=gpu[01-04] CPUs=128 RealMemory=1000000 Gres=gpu:8 State=UNKNOWN
PartitionName=train Nodes=gpu[01-04] Default=YES MaxTime=2-00:00:00 State=UP

# --- gres.conf ---
AutoDetect=nvml

The submission side is a shell script. The fields that matter are the ones the scheduler arbitrates on: node count, GPUs per node, a time limit, and the account and QOS that fair share and preemption key on.

#!/bin/bash
#SBATCH --job-name=pretrain-run
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --gres=gpu:8
#SBATCH --cpus-per-task=8
#SBATCH --time=12:00:00
#SBATCH --account=ml-research
#SBATCH --qos=normal
#SBATCH --output=%x-%j.out

set -euo pipefail

# Bind each task to one GPU; without --gpu-bind every task sees all eight.
# --cpu-bind=cores is best-effort: when the task count differs from the
# allocated core count, the man page notes binding can be sub-optimal.
srun --cpu-bind=cores --gpu-bind=single:1 python train.py --config configs/run.yaml

Notice what the script does not say: which nodes, which GPUs, or when. Those are the scheduler's decisions, made against everything else in the queue. The researcher states a shape and a budget; slurmctld owns the rest. That division of labour is why Slurm sites tend to run as shared facilities with a published policy — the policy is the product, and the job script is a request against it.

What Kubernetes does, and what it needs Kueue for

Kubernetes places pods. kube-scheduler takes each pending pod, filters nodes that fit its requests — nvidia.com/gpu from a device plugin, or a claim under Dynamic Resource Allocation — and binds it. There is no native queue of jobs competing for a fixed budget, and no notion that a workload should finish by a deadline. For services that is the right model; for a queue of training runs it is missing a layer.

Kueue is that layer. It "manages quotas and how jobs consume them" and decides when a job should wait, be admitted, or be preempted. Its scope is deliberately narrow: "Kueue does not replace any existing Kubernetes components", and pod-to-node scheduling stays with kube-scheduler. Admission means the pods may be created; placement is still someone else's job. The QLoRA infrastructure article uses it for exactly this: one governed fine-tuning Job on owned GPUs.

Serving workloads sit awkwardly in that model. Kueue "does not yet support managing a Deployment as a single Workload"; instead "every Pod from a Deployment is represented as a single independent Plain Pod." Quota applies per replica, not to the service as a unit — workable for scaling inference, but not the same contract a batch job gets. Volcano and YuniKorn are alternative Kubernetes batch schedulers; this comparison stays with Kueue, whose documentation states that it leaves kube-scheduler in place.

The practical consequence is that the platform team, not the researcher, writes the queueing policy — as Kubernetes objects. That is a strength where policy already flows through Git review and admission control, and a burden where the people who understand GPU contention do not also maintain the cluster. Who will own the ClusterQueue definitions is a staffing decision worth making before the first queue exists.

apiVersion: kueue.x-k8s.io/v1beta2
kind: Topology
metadata:
  name: gpu-topology
spec:
  levels:
  - nodeLabel: "topology.example.com/block"
  - nodeLabel: "topology.example.com/rack"
  - nodeLabel: "kubernetes.io/hostname"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: gpu-node
spec:
  nodeLabels:
    node.example.com/pool: "gpu"
  topologyName: gpu-topology
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: training
spec:
  namespaceSelector: {}
  cohortName: gpu-pool
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: gpu-node
      resources:
      - name: "cpu"
        nominalQuota: 256
      - name: "memory"
        nominalQuota: 2048Gi
      - name: "nvidia.com/gpu"
        nominalQuota: 16
        borrowingLimit: 8
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: inference
spec:
  namespaceSelector: {}
  cohortName: gpu-pool
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: gpu-node
      resources:
      - name: "cpu"
        nominalQuota: 256
      - name: "memory"
        nominalQuota: 2048Gi
      - name: "nvidia.com/gpu"
        nominalQuota: 16
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  name: training
  namespace: ml-training
spec:
  clusterQueue: training

Gang scheduling means two different things

The same phrase names two unrelated mechanisms, and a comparison that does not separate them compares nothing.

In Slurm, gang scheduling is timeslicing: two or more jobs "are allocated to the same resources in the same partition and these jobs are alternately suspended to let one job at a time have dedicated access to the resources for a configured period of time." Its constraint is memory — "all jobs must be able to fit into memory at the same time", which rules out timeslicing jobs that each need most of a node's memory. Slurm's multi-node start does not need this feature at all: a job is allocated its nodes for a duration, as the overview describes, and runs on that allocation.

In Kubernetes, gang scheduling means all-or-nothing placement: if the cluster cannot fit the whole group, or a minimum given by minCount, "none of the Pods are bound to a node." The problem it fixes is stated plainly by the project: without it, "a Job might be partially scheduled, consuming resources without being able to run, leading to resource wastage and potential deadlocks." It is "Beta since Kubernetes v1.37; disabled by default", and needs the GenericWorkload feature gate on all relevant components plus the scheduling.k8s.io/v1beta1 API group.

The 1.37 shape is two objects: a Workload in scheduling.k8s.io/v1beta1 holds the gang policy as a long-lived template, a PodGroup carries per-instance runtime state, and pods reference it through spec.schedulingGroup.podGroupName. The Job controller is currently the only built-in controller that follows this pattern. When the feature first arrived in v1.35, a group only partly placed within a 5-minute timeout was rejected entirely and returned to the queue.

Kueue offers a third, older answer: waitForPodsReady, an admission-level timeout rather than a placement guarantee. It defaults to 30 minutes; if an admitted workload's pods are not all ready by then, "the Workload's admission is cancelled, the corresponding Job is suspended, and the Workload is requeued."

Queueing and fair share side by side

Slurm's queue is two mechanisms working together. Priority decides order: since Slurm 19.05 the fair-share factor of the multifactor plugin defaults to Fair Tree, and PriorityDecayHalfLife sets how long past usage counts against an account — "the larger the number, the longer past usage affects fair-share", with a default of 7 days. Backfill decides utilisation: it is loaded by default and "will start lower priority jobs if doing so does not delay the expected start time of any higher priority jobs."

Kueue maps the same ideas onto ClusterQueues. Each queue has a nominalQuota; queues sharing a cohortName lend each other capacity, and "the borrowable resources are the unused nominal quota of all the ClusterQueues in the cohort." borrowingLimit caps how far one queue can reach. Fair Sharing, stable since Kueue v0.7, "uses preemptions to achieve an equal or weighted share of the borrowable resources between the tenants of a cohort."

The difference is in what is shared. Fair Tree weighs decayed historical usage against an account hierarchy. Kueue fair sharing balances borrowing of currently unused quota, enforced by preemption. Neither is wrong; they encode different promises, and a team moving from one to the other should expect the queue to behave differently under contention. Under classic preemption, with the ClusterQueue manifests above applied on their own, training can borrow up to eight GPUs of inference's unused nominal quota, and a pending inference workload that fits within its own nominal quota can reclaim them by preempting cohort usage above nominal, irrespective of priority. That reading does not survive unchanged once the fair-sharing Configuration below is applied. Either way, preemption suspends the training Job, and the Job loses its progress unless it checkpoints.

# Merge into the existing controller Configuration; keep its integrations.frameworks list.
apiVersion: config.kueue.x-k8s.io/v1beta2
kind: Configuration
waitForPodsReady:
  timeout: 10m
  recoveryTimeout: 5m
  blockAdmission: true
  requeuingStrategy:
    timestamp: Eviction
    backoffLimitCount: 5
    backoffBaseSeconds: 60
    backoffMaxSeconds: 3600
fairSharing:
  preemptionStrategies: [LessThanOrEqualToFinalShare, LessThanInitialShare]

Multi-node jobs also care where pods land. Kueue's Topology-Aware Scheduling (beta since v0.14, enabled by default) annotation kueue.x-k8s.io/podset-required-topology requires all pods of a PodSet on nodes within one topology domain, such as a rack or a block:

apiVersion: batch/v1
kind: Job
metadata:
  name: pretrain-run
  namespace: ml-training
  labels:
    kueue.x-k8s.io/queue-name: training
spec:
  parallelism: 2
  completions: 2
  completionMode: Indexed
  template:
    metadata:
      annotations:
        kueue.x-k8s.io/podset-required-topology: "topology.example.com/rack"
    spec:
      restartPolicy: Never
      containers:
      - name: trainer
        image: registry.example.com/ml/trainer:1.0.0
        command: ["python", "train.py", "--config", "configs/run.yaml"]
        resources:
          requests:
            cpu: "64"
            memory: 512Gi
            nvidia.com/gpu: 8
          limits:
            nvidia.com/gpu: 8

Running both: Slinky's slurm-operator and slurm-bridge

Slinky is SchedMD's set of projects for Slurm and Kubernetes interoperability, and it offers two shapes. slurm-operator runs Slurm on Kubernetes — it will "manage and scale Slurm clusters on Kubernetes as pods." slurm-bridge runs "Slurm as a Kubernetes scheduler", so both kinds of workload are scheduled by Slurm.

The operator keeps Slurm semantics for researchers while Kubernetes owns the node lifecycle. Its README states that Slurm nodes "will be marked as drain before their eventual termination pending scale-in or upgrade", and that NodeSets can be scaled to zero replicas — the useful property when the same hardware must be returned to an inference fleet.

The bridge goes the other way. Workloads can be submitted "as a Pod, PodGroup, Job, JobSet, or LeaderWorkerSet" from Kubernetes, or with salloc or sbatch from Slurm. Two defaults in the default-branch README deserve attention before anything is designed around it (the v1.2.3 tag states that exclusive, whole-node allocations are made for each pod, and its limitations list also supports only two named DRA drivers and notes that native CPU requests do not constrain a container to the CPU set Slurm allocated): "bridge jobs use exclusive, whole-node allocations by default", and on hybrid nodes "the hardware is shared over time: a physical node must not run native Slurm user workloads and Slurm-bridge-managed Kubernetes user workloads simultaneously." The same README continues: "Kubernetes and Slurm system daemons are expected to remain co-located on those nodes." The kubelet and slurmd stay side by side; only the user workloads are divided over time.

Failure modes and upgrades

• Partial gangs. Without all-or-nothing placement, half a training job can hold GPUs it cannot use. On Kubernetes, that is the default unless the Workload API gate is enabled, or Kueue's waitForPodsReady is configured to suspend and requeue a job whose pods are not all ready in time.
• Admission deadlock. Two large jobs each admitted and each waiting for capacity the other holds. Kueue's blockAdmission admits workloads "sequentially to prevent deadlock situations" — at the cost of admission throughput.
• Accounting downtime. Slurm's upgrade guide warns that slurmdbd upgrades "may require significant downtime": on large accounting databases the upgraded daemon may be unresponsive for tens of minutes while it migrates the schema, and sacct and sacctmgr do not work while slurmdbd is down.
• Upgrade order. Slurm daemons go slurmdbd, then slurmctld, then slurmd (preferably in the same window as slurmctld), then login nodes and client commands. Slurm 26.05 supports upgrading directly from 25.11, 25.05 and 24.11; skipping further means stepping through releases.
• Slinky CRD churn. Per the slurm-operator README, a CRD version change (e.g. v1beta1 to v1beta2) may remove deprecated fields; older versions are converted by the Kubernetes API to the stored version and will still work, but upgrade the slurm-operator-crds chart before slurm-operator, or both together with crds.enabled=true.
• Version pinning. slurm-bridge's default-branch README carries a matrix in which only a 1.3.X row lists Kubernetes 1.37, and every row requires Slurm 25.11. As of 2026-10-20 the newest git tag is v1.2.3, whose README states only minimums: Kubernetes v1.35 and Slurm 25.11. Pin to a tag and read that tag's README, not the default branch.
• Beta gates. Kubernetes 1.37 (patch 1.37.1 at the time of writing) ships gang scheduling as beta behind a gate. A beta API can still change shape between minors; pin manifests to the version you tested.

What an auditor sees

Who used which GPUs, for how long, under what policy, is an audit question for any shared pool and a chargeback question for every finance team. Slurm answers it in one place: slurmdbd records usage per job, account and QOS; sacct reports what ran and sshare reports fair-share standing. The policy itself — accounts, QOS limits, priority weights — lives in slurm.conf and the accounting database, both versionable and inspectable.

Kubernetes spreads the evidence across objects. ClusterQueues and LocalQueues are declarative policy and belong in Git; Kueue Workload objects carry admission, eviction and requeue conditions for each job. Turning that into durable GPU-hour attribution is your work: export it to a time-series or log store before finished Jobs and their Workload objects are cleaned up. Plan the retention explicitly — an audit trail that disappears with its Job is not one.

In both systems the evidence an auditor trusts most is the change history of the policy, not a snapshot of it. A quota raised for one team, a QOS that may preempt another, a borrowing limit removed during a deadline — each is a decision with a cost to someone. Keep slurm.conf and the Kueue manifests in version control, change them only through review, and the question "who was allowed to take these GPUs, and when was that decided?" has an answer that does not depend on anyone's memory.

Exit ramps and the long game

Portability differs in kind. An sbatch script is a shell script with a header: it moves between Slurm sites with edits to account and partition names. Kubernetes manifests move between clusters, but carry add-on APIs — Kueue labels, topology annotations, beta scheduling objects — that only work where those add-ons are installed. Keep training code agnostic to both and push scheduler knowledge into thin wrappers, so the launch layer can change without touching the model.

Governance changed in December 2025, when NVIDIA acquired SchedMD, "the leading developer of Slurm", and stated it "will continue to develop and distribute Slurm as open-source, vendor-neutral software." NVIDIA's announcement also states that Slurm is used in more than half of the top 10 and top 100 TOP500 systems — the acquirer's own figure. The pledge is a commitment to watch, not evidence; for a fleet that might also run AMD GPUs as a second source, vendor neutrality of the scheduler is a property worth testing each release.

Cadence matters too. Slurm moved to a six-month major cycle with 24.05, version numbers encode year and month, and supported upgrade windows are published. Kubernetes moves faster, and its batch features move fastest. Sizing the pool in the first place is covered in the GPU and VRAM sizing guide; where the scheduler sits in the wider stack is in the on-premise AI decision guide.

Decision table and FAQ

• Mostly multi-node training, research users, strict accounting: Slurm, with an honest time-limit culture so backfill works.
• Mostly inference and services, platform team already on Kubernetes: Kubernetes with Kueue for the batch share — and do not assume gang placement on a default 1.37 cluster without enabling GenericWorkload on all relevant components.
• Training capacity that must flip to serving: slurm-operator to run Slurm inside Kubernetes with drain-before-scale-in, or slurm-bridge if Slurm should make placement decisions — remembering the default-branch README (not the v1.2.3 tag) says hybrid nodes are shared over time, not simultaneously — read the README of the tag you install.
• Small team, one on-call rota: run one control plane well rather than two badly; add the other only when a workload class is starved.