Two-Tier Autoscaling on Bare Metal: KEDA Pods, Descheduler Bin-Packing

Autoscaling on a hyperscaler is two moves that feel like one. The HorizontalPodAutoscaler adds pods when CPU climbs, and when the pods no longer fit, Karpenter or the Cluster Autoscaler calls the provider's API and a new instance materializes minutes later. Capacity is a credit card: you spend it when you need it and stop paying when you scale in. Move that workload into your own rack and the second move evaporates. There is no API that conjures a server; your node pool is the machines you racked, and it does not grow because a pod is Pending. The elasticity you took for granted was never a Kubernetes feature — it was a billing relationship with someone else's data center.

This is the moment self-hosted platforms discover that autoscaling has to be re-derived from first principles. You cannot provision nodes on demand, so the goal shifts from "buy capacity when busy" to "use the fixed capacity you have as densely as possible, and free whole machines when you can." That reframing produces a two-tier design with no exact cloud equivalent: KEDA drives pod-level elasticity — including the scale-to-zero that idle GPUs make non-negotiable — and the Kubernetes descheduler drives node-level elasticity by bin-packing pods back together so emptied machines can be powered down or held in reserve. This article wires both tiers together for production, and is honest about where the seams are.

It is a companion to the cloud repatriation playbook — repatriating compute is the first half; getting elastic behavior out of inelastic hardware is the half that decides whether the economics work. It assumes the self-hosted observability stack is already in place, because both tiers are only as good as the metrics that drive them.

Why Cloud Autoscaling Doesn't Port to Bare Metal

Karpenter is a superb piece of engineering aimed at a problem you no longer have. Its job is to watch for unschedulable pods and provision right-sized cloud instances directly from the provider's compute API — choosing instance types, spot versus on-demand, and zones, then terminating them when they drain. Every one of those verbs is a call into AWS, Azure, or GCP, and on bare metal there is nothing on the other end of that call. Your capacity is a countable set of physical machines with a fixed aggregate of CPU, memory, and GPUs, and the only ways to change it are a purchase order and a rack visit. Provisioner-style autoscalers are therefore not "harder" on-premises; they are category-inapplicable.

What remains, and what you fully control, is placement and pod count. Bare-metal autoscaling is the disciplined management of two independent axes on a constant node pool: how many pod replicas exist (which should track real demand, not a proxy like CPU), and how tightly they pack onto nodes (which decides how much of your fixed hardware is stranded as fragmentation). Cloud lets you buy your way out of bad packing; you cannot, so you must solve both — and doing so deliberately is what makes a fixed pool behave elastically. The diagram below contrasts the inapplicable cloud model with its two-tier replacement.

Click to zoom

Tier One: KEDA and the 0↔1 Problem HPA Can't Solve

The built-in HorizontalPodAutoscaler has a hard floor of one replica and a native vocabulary of CPU and memory — the wrong instrument for event-driven work. A consumer draining a Kafka topic, a batch of image renders, or an LLM inference server does not get busy because its CPU rose; it gets busy because work arrived, and CPU only climbs *after* the backlog has formed. KEDA, a CNCF-graduated project since 2023 with more than sixty built-in scalers, closes both gaps: it scales on the arrival signal itself — Kafka consumer lag, RabbitMQ queue length, a Prometheus query, a cron window — and it scales to and from zero, which the HPA cannot do alone.

The mechanism explains every quirk that follows. KEDA does not replace the HPA; it feeds it. From one replica upward, KEDA runs a metrics adapter and a generated HPA does the arithmetic exactly as it always has. The boundary at zero is the special case: the KEDA operator itself handles the 0 → 1 activation and 1 → 0 deactivation, because an HPA structurally cannot reason about zero. So a ScaledObject is two controllers cooperating — the operator as the on/off switch, the HPA as the volume knob. A Kafka-lag ScaledObject for a self-managed event backbone looks like this:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: orders-consumer
  namespace: ingest
spec:
  scaleTargetRef:
    name: orders-consumer          # the Deployment to scale
  minReplicaCount: 0               # scale-to-zero: no idle consumers
  maxReplicaCount: 40
  cooldownPeriod: 120              # wait 120s after last active trigger before 1 -> 0
  pollingInterval: 15             # how often KEDA evaluates the trigger
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: redpanda.ingest.svc:9093
        consumerGroup: orders
        topic: orders
        lagThreshold: "500"        # target: ~500 lagging msgs per replica (the HPA knob)
        activationLagThreshold: "50"  # 0 -> 1 only once lag exceeds 50 (the operator knob)

Two fields carry the whole scale-to-zero contract, and conflating them is the most common KEDA mistake. lagThreshold is the HPA's target: with a value of 500 and 4,000 lagging messages, the HPA wants eight replicas. activationLagThreshold is a separate gate the operator checks first, and it wins ties — set it to 50 and the deployment stays at zero until lag crosses 50, even though the HPA's math would already justify a replica. That split encodes the exact policy expensive workloads need: do not wake for trivial trickles, but once you wake, scale aggressively.

Scale-to-Zero on Expensive GPUs: The Sharpest Case

The economics of scale-to-zero are marginal for a stateless web service and decisive for a GPU. A single owned H100-class accelerator is tens of thousands of dollars of capital, and an inference pod holding one idle overnight is that capital earning nothing while still drawing power and heat. You cannot delete the machine as you would a cloud instance, but you can free the GPU — evict the idle pod so the card is available for a different job, and if nothing needs it, let the node fall idle for the descheduler and power management to handle. This is why the CNCF community spent early 2026 building KEDA external scalers for GPU-backed inference: the idle-GPU bill is the sharpest current driver of event-driven autoscaling.

A self-hosted vLLM inference server is the canonical target. Drive it from a queue metric, keep the floor at zero, and set a cooldown long enough to absorb bursts without thrashing a slow-loading model:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-mistral
  namespace: inference
spec:
  scaleTargetRef:
    name: vllm-mistral
  minReplicaCount: 0               # release the GPU when the queue is empty
  maxReplicaCount: 6               # bounded by the GPUs you physically own
  cooldownPeriod: 600              # models are slow to load; do not flap the GPU
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.observability.svc:9090
        query: sum(vllm:num_requests_waiting{service="vllm-mistral"})
        threshold: "8"                 # ~8 queued requests per replica
        activationThreshold: "1"       # wake on the very first queued request

Scale-to-zero pairs with — rather than replaces — GPU partitioning. Where several small models share one card, MIG and time-slicing raise utilization while pods run; KEDA decides whether any pod runs at all. Partition to pack concurrent models onto a card, then scale the whole card to zero when its queues drain — different granularities, cleanly composed.

ScaledJob: Per-Event Batch Without a Standing Consumer

A ScaledObject scales a long-lived Deployment. For discrete, run-to-completion work — a render, an ETL shard, a CI build — KEDA offers ScaledJob, which spawns a Kubernetes Job per unit of pending work and lets it exit cleanly, with no consumer idling between bursts. It is the same pattern that powers scale-to-zero self-hosted GitHub Actions runners: a queued job materializes a runner pod, the pod does one job and terminates, and the fleet returns to zero. On a fixed pool that matters twice — you never hold a machine warm for a queue that may stay empty for hours, and each Job carries its own resource request so the scheduler packs it densely from the start.

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: render-worker
  namespace: batch
spec:
  jobTargetRef:
    template:
      spec:
        containers:
          - name: render
            image: registry.internal/render:v3
        restartPolicy: Never
  minReplicaCount: 0
  maxReplicaCount: 50               # cap concurrent Jobs against pool capacity
  pollingInterval: 20
  triggers:
    - type: rabbitmq
      metadata:
        queueName: renders
        mode: QueueLength
        value: "1"                  # one Job per queued render

Tier Two: The Fragmentation KEDA Leaves Behind

Aggressive pod-level autoscaling has a side effect that is invisible on the cloud and expensive on bare metal. When KEDA scales a workload from two replicas to twenty during a burst, the scheduler places those pods wherever there is room at that instant — spread across every node. When the burst passes and KEDA scales back to two, the survivors do not migrate; Kubernetes never reschedules a running pod for packing reasons on its own. You are left with a sparse scatter — a handful of pods thinly distributed across many nodes, every node still powered on, none empty enough to reclaim. That is fragmentation, and on a fixed pool it is pure waste: you pay the full power and cooling bill for a cluster running at a fraction of its density.

The cloud hides this because Karpenter's consolidation continuously deletes underused nodes and reschedules their pods, so fragmentation self-heals by shrinking the fleet. Your fleet cannot shrink, so the equivalent win is to *concentrate* the pods and let whole machines fall idle deliberately — precisely the Kubernetes descheduler's job. It does not schedule pods (the kube-scheduler does that); it evicts pods that are now poorly placed, so the scheduler can place them better next cycle. Paired with a bin-packing scheduler policy, eviction plus rescheduling becomes a compaction pass that turns scattered survivors back into a dense core and a set of reclaimable nodes.

Click to zoom

Descheduler + MostAllocated: Bin-Packing to Reclaim Nodes

Bin-packing on a fixed pool is a two-component contract, and running either half alone is a classic misconfiguration. The kube-scheduler decides *where pods land*, and by default uses LeastAllocated scoring, which spreads pods for resilience — the opposite of packing. The descheduler decides *which running pods to evict* for being poorly placed. They must agree on the goal: switch the scheduler's NodeResourcesFit plugin to MostAllocated scoring so it prefers the fullest node that still fits, and run the descheduler's HighNodeUtilization profile so it evicts pods off under-used nodes. The descheduler's documentation is explicit that HighNodeUtilization only behaves correctly with MostAllocated scheduling — otherwise the descheduler evicts a pod and the scheduler immediately spreads it back, and you have built a thrash loop.

# 1) kube-scheduler: pack pods onto the fullest node that fits.
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
    pluginConfig:
      - name: NodeResourcesFit
        args:
          scoringStrategy:
            type: MostAllocated       # bin-pack (default is LeastAllocated = spread)
---
# 2) descheduler: evict pods off under-used nodes so the scheduler repacks them.
apiVersion: descheduler/v1alpha2
kind: DeschedulerPolicy
profiles:
  - name: consolidate
    pluginConfigs:
      - name: HighNodeUtilization
        args:
          thresholds:
            cpu: 40                   # a node under 40% cpu AND memory is "under-used"
            memory: 40
      - name: DefaultEvictor
        args:
          evictSystemCriticalPods: false
          nodeFit: true               # only evict if the pod can land elsewhere
    plugins:
      balance:
        enabled: [HighNodeUtilization]

Run the descheduler as a CronJob on a cadence matching your workload's rhythm — every few minutes for spiky traffic, hourly for steadier fleets — not a tight loop, because every eviction is a real disruption. The DefaultEvictor is where safety lives: honor PodDisruptionBudgets, refuse system-critical pods, and keep nodeFit: true so no pod is evicted that cannot be placed better. Descheduling then becomes a governed, reviewable policy in Git — the platform-engineering golden-path posture, where the whole cluster's packing behavior is one auditable object, not tribal knowledge.

Closing the Loop: Scale, Pack, Reclaim, Repeat

The two tiers become a system when they feed each other on a shared substrate of metrics. KEDA reacts in seconds — pod counts up on a burst, down to zero where safe when it passes — and that scale-down produces the fragmentation the descheduler compacts on its cadence, surfacing idle nodes to power down or hold as headroom. That instrumentation is the loop's control plane, not an afterthought: KEDA's scaling decisions, HPA target ratios, per-node allocation, and eviction counts belong on the dashboards your OpenTelemetry and Prometheus stack already serves. The failure modes are only visible in aggregate — a scale-to-zero that never reaches zero, a descheduler evicting pods that reschedule to the same node, a cooldown so short the GPU flaps — and none throws an error; each is a flat line where you expected a sawtooth. Nothing provisions hardware; the loop extracts elastic behavior from a constant pool.

Failure Modes and the Guardrails That Contain Them

Both tiers fail quietly until the failure is expensive, and each has a specific guardrail. Eviction thrash is the signature descheduler failure: evict a pod for poor placement, watch the scheduler put it right back because the two disagree on strategy. The fix is the MostAllocated/HighNodeUtilization pairing above plus nodeFit: true; without both, do not enable descheduling. Cold-start SLO breaches are the signature KEDA failure — a latency-critical service scaled to zero missing its budget on wake-up; the guardrail is workload triage, scale-to-zero for tolerant traffic and a warm floor for synchronous user paths.

Two more deserve naming. Availability erosion from over-eager eviction happens when the descheduler moves too many replicas of one service at once; PodDisruptionBudgets are the hard backstop, and the descheduler honors them — so every workload that matters must ship one, or the guardrail is not armed. And metric-source coupling is subtle: a KEDA trigger on a Prometheus query makes your autoscaler depend on your monitoring stack, so a Prometheus outage can freeze scaling. Run that dependency deliberately — an HA metrics backend, a sane ScaledObject fallback, and alerting on the scaler's own health — rather than discovering it mid-incident.

The Exit Ramp: Autoscaling You Own, Not Rent

The optionality argument here is unusually clean, because the cloud alternative is unusually locked-in. Karpenter is excellent and inseparable from the provider whose API it drives — its provisioning logic is AWS-shaped, and adopting it deeply couples your scaling to one vendor's compute plane. KEDA and the descheduler are the opposite: plain Kubernetes controllers driving standard objects — ScaledObject, ScaledJob, DeschedulerPolicy, a scheduler profile — that run identically on bare metal, on any managed cluster, or across a hybrid. Your autoscaling policy becomes portable infrastructure-as-code, not a dependency on one datacenter's API; burst into a cloud for a launch and the same ScaledObject works unchanged, because nothing in the design assumes where the nodes came from.

That portability is what optionality means in practice — not the absence of tools, but the absence of tools you cannot leave. You are not renting elasticity from a vendor's spare capacity; you are engineering it into infrastructure you own, with primitives you can read.

The Long Game: Elasticity Without Elastic Infrastructure

The reflex belief is that owning hardware means giving up elasticity — that a fixed pool is a static pool. The two-tier model refutes it. Elasticity was never a property of the hardware; it was a property of how densely and how responsively you use it. KEDA gives you responsiveness in seconds against the signals that actually predict load; the descheduler gives you density by continuously reclaiming the fragmentation that responsiveness creates. Between them, a fixed set of machines you own behaves like a pool that breathes — expanding into a burst, contracting to zero on idle GPUs, compacting back to a dense core with nodes to spare — all without a single call to a provisioning API.

That is the durable position. The repatriation case is usually made on cost, and that case is real — you stop paying cloud margins on capacity you already bought. But the compounding return is a scaling system your team understands end to end and can operate for a decade: two open-source controllers, a handful of typed objects in Git, and a feedback loop you can draw on a whiteboard. Elastic infrastructure you rent is elasticity you can lose in a pricing email; elastic behavior you engineer into hardware you own is elasticity you keep — which, on a long enough horizon, is the only kind worth building on.