KubeVirt: The VMware Exit for VMs You Cannot Containerise

The migration goes well. The VMs boot, the applications answer, the smoke tests are green. Six weeks later a node needs a kernel patch, kubectl drain is issued, and it never finishes: the VMs cannot leave their node. Nothing failed. The estate was pinned at creation time by an access mode nobody chose deliberately, and the first drain is when you find out.

The VMs That Were Never Going to Be Containers

Every estate that has run a containerisation programme for more than three years has a residue. It is not a backlog and it will not burn down: each remaining workload is there for a structural reason.

• Vendor appliances. Shipped as an OVA, supported only as an OVA. Repackaging voids the contract you bought it for.
• The Windows estate. A desktop session, a signed kernel driver, an agent, or an installer assuming a persistent C: drive.
• Licence-pinned databases. Where the licence attaches to a core count, a socket count or a named hypervisor clause, re-architecting means re-negotiating.
• Kernel-coupled software. Real-time, telecom and industrial stacks older than the container abstraction, pinned to a kernel you do not choose.

Containers were never the goal; owning the substrate was. Repatriation moves the containerised tier home and leaves this residue behind, still on a hypervisor whose commercial terms you no longer control. KubeVirt's proposition is that the residue runs on the same Kubernetes API as everything else — one scheduler, one RBAC boundary, one control plane — not a second, separately-licensed platform.

The project ships on a predictable cadence. KubeVirt v1.9.0 was released on 22 July 2026, built for Kubernetes v1.36 and additionally supported for the previous two versions; v1.8.0 landed on 25 March 2026 aligned with Kubernetes v1.35. Plan around that rhythm: your virtualisation layer now moves at Kubernetes' pace.

What the Broadcom Repricing Actually Changes

Most virtualisation-exit conversations start with a renewal quote and never get past it. Split the bill into contractual and architectural; only one is yours to fix.

The contractual part is published and countable. Broadcom's core-counting guidance for VMware Cloud Foundation and vSphere Foundation states it plainly: you must license a minimum of 16 physical cores for each CPU in your ESXi hosts, even if a CPU has fewer. Run their worked example against your inventory: a host with two 8-core CPUs plus a host with two 24-core CPUs — 64 physical cores — convert to 80 licensable cores. Small-socket hosts and edge nodes bill as though they were larger.

Beyond the published rule, treat figures carefully. In March 2025 The Register reported that VMware distributor Arrow told partners the minimum software subscription was set to jump from 16 to 72 cores — a distributor's statement reported by the press, not published Broadcom policy, and it should be quoted that way in a steering paper. The 16-core minimum alone carries the argument: a model that prices the shape of your hardware rather than its use cannot be optimised around. That is an exit cost worth computing rather than absorbing.

Live Migration Is a Storage Decision, Not a Virtualisation One

This is what surprises experienced virtualisation engineers: on vSphere the equivalent capability belongs to the cluster. On KubeVirt it belongs to each individual disk.

KubeVirt's limitation is unambiguous: virtual machines using a PersistentVolumeClaim must have a shared ReadWriteMany access mode to be live migrated. At VMI start KubeVirt computes migratability and records a LiveMigratable condition — a calculation the docs say is *largely based on the access mode of the VMI volumes* — and migration requests for a non-LiveMigratable VMI are rejected. Alongside it sits a method: BlockMigration means some disks require copying, LiveMigration means only instance memory is copied.

Click to zoom

So migratability is auditable *before* a maintenance window rather than discovered during one. Read the condition and method off every VMI, print the access mode of the failing claims, exit non-zero, and gate the runbook on it.

#!/usr/bin/env bash
# Refuses a maintenance window while any VMI is pinned to its node.
set -euo pipefail

pinned=0

while read -r ns name migratable method; do
  [ "$migratable" = "True" ] && continue
  pinned=1
  printf 'PINNED  %s/%s  (LiveMigratable=%s, method=%s)\n' \
    "$ns" "$name" "$migratable" "$method"

  # Access mode and volume mode of every claim the VMI uses.
  kubectl get vmi -n "$ns" "$name" -o json |
    jq -r '.spec.volumes[]? | (.persistentVolumeClaim.claimName // .dataVolume.name // empty)' |
    sort -u |
    while read -r pvc; do
      printf '        pvc=%-28s access=%-14s volumeMode=%s\n' "$pvc" \
        "$(kubectl get pvc -n "$ns" "$pvc" -o jsonpath='{.spec.accessModes[*]}')" \
        "$(kubectl get pvc -n "$ns" "$pvc" -o jsonpath='{.spec.volumeMode}')"
    done
done < <(
  kubectl get vmi --all-namespaces -o json | jq -r '
    .items[]
    | [ .metadata.namespace,
        .metadata.name,
        (first((.status.conditions // [])[]
               | select(.type == "LiveMigratable") | .status) // "Unknown"),
        (.status.migrationMethod // "Unknown")
      ] | @tsv'
)

if [ "$pinned" -ne 0 ]; then
  echo "Maintenance window refused: the VMs above cannot be drained." >&2
  exit 1
fi

echo "Every VMI reports LiveMigratable=True."

Which backend supplies RWX Block is the real decision — the same one as choosing a sovereign block storage layer for anything else, plus one constraint. Do not assume a class multi-attaches because its name suggests it should. KubeVirt's docs note that RWX PVCs need no node affinity rule because the storage attaches simultaneously on multiple nodes — a statement about what RWX means, not a guarantee your class provisions it. Create one claim, run kubectl get pvc, read the ACCESS MODES column.

The Network Binding That Silently Disables Migration

The second gate is the pod network binding, and it is easy to walk into because bridge mode is what a virtualisation engineer instinctively reaches for. The same limitations list states that live migration is not allowed with a pod network binding of bridge interface type. A VM given a bridged primary interface to keep familiar L2 behaviour has forfeited live migration, and nothing says *why* until you read the condition.

Two more constraints belong on the checklist. Live migration requires ports 49152 and 49153 in the virt-launcher pod — declare them explicitly on a masquerade interface and migration stops working, a self-inflicted failure that looks exactly like a network fault. The primary interface must also carry the same name on source and target pods. Where a guest needs a real VLAN, put it on a secondary interface via Multus — the discipline that keeps bare-metal ingress predictable.

Licence-Sensitive Guests: Pinning, Hugepages and the SMT Trap

Databases and latency-sensitive guests want dedicated cores and large pages. KubeVirt delegates this to the Kubernetes CPU manager, which pins containers to host pCPUs only when the pod's QoS is Guaranteed: requests and limits equal, every container expressing CPU and memory requirements, and an integer CPU count. Only one half of that is obvious: the topology supplies the CPU side — sockets × cores × threads yields the integer count, which is why the dedicated-CPU examples set no explicit CPU request — while the memory side is yours to set. Hugepages carry their own precondition: they must be pre-allocated on the node at boot via kernel parameters and a restart, cannot be requested into existence at VM start, and require memory divisible by the page size.

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: ledger-db-01
  namespace: platform-vms
spec:
  runStrategy: Always
  dataVolumeTemplates:
    - metadata:
        name: ledger-db-01-root
      spec:
        storage:
          storageClassName: rook-ceph-block-rwx   # must provision RWX Block
          accessModes: [ReadWriteMany]            # the live-migration gate
          volumeMode: Block
          resources:
            requests:
              storage: 400Gi
        source:
          blank: {}   # scaffolding — a real guest arrives via CDI or Forklift
  template:
    metadata:
      labels:
        kubevirt.io/vm: ledger-db-01
      annotations:
        # Without this, the descheduler will not rebalance this VM at all.
        descheduler.alpha.kubernetes.io/evict: "true"
    spec:
      evictionStrategy: LiveMigrate
      domain:
        cpu:
          sockets: 1
          cores: 8
          threads: 1
          dedicatedCpuPlacement: true
          isolateEmulatorThread: true
        memory:
          hugepages:
            pageSize: 1Gi        # node must already have 1G pages reserved
        resources:
          requests:
            memory: 32Gi
          limits:
            memory: 32Gi         # equal to requests: Guaranteed QoS
        devices:
          disks:
            - name: root
              disk:
                bus: virtio
          interfaces:
            - name: default
              masquerade: {}     # bridge here would forfeit live migration
      networks:
        - name: default
          pod: {}
      volumes:
        - name: root
          dataVolume:
            name: ledger-db-01-root

That manifest contains a trap worth meeting on paper first. On a node with SMT enabled and the kubelet's CPUManager static policy option full-pcpus-only, a VM with an even CPU count plus dedicatedCpuPlacement and isolateEmulatorThread schedules successfully, then is rejected by the kubelet with SMT Alignment Error: requested 3 cpus not multiple cpus per core = 2 — the emulator thread makes the total odd. The documented fix is two changes on the KubeVirt CR, not the VM: enable the AlignCPUs gate and add the alpha.kubevirt.io/EmulatorThreadCompleteToEvenParity annotation, after which KubeVirt adds one or two dedicated CPUs to complete the count to even.

# featureGates is a list — a merge patch REPLACES it. Read the current value
# first and re-send the full set, or you will silently disable something.
# Only Alpha gates belong here — VMExport went GA in v1.9 and is always on.
kubectl -n kubevirt get kubevirt kubevirt \
  -o jsonpath='{.spec.configuration.developerConfiguration.featureGates}'

kubectl -n kubevirt patch kubevirt kubevirt --type=merge -p '{
  "metadata": {
    "annotations": {
      "alpha.kubevirt.io/EmulatorThreadCompleteToEvenParity": ""
    }
  },
  "spec": {
    "configuration": {
      "developerConfiguration": {
        "featureGates": ["AlignCPUs"]
      },
      "migrations": {
        "parallelMigrationsPerCluster": 5,
        "parallelOutboundMigrationsPerNode": 2,
        "bandwidthPerMigration": "1Gi",
        "completionTimeoutPerGiB": 800,
        "progressTimeout": 150
      }
    }
  }
}'

The defaults those numbers replace matter for planning: KubeVirt ships with five parallel migrations per cluster, two outbound per node, and 64MiB/s per migration. A whole-fleet drain at 64MiB/s is a different maintenance window from one at 1GiB/s — and the limit exists to stop migrations overwhelming the cluster, so raise it on a network you have measured.

Moving the Estate With Forklift — and Its ReadWriteOnce Default

Forklift is the open-source toolkit for the bulk move, migrating VMs from sources including VMware, oVirt, OpenStack, OVA and KubeVirt itself into KubeVirt. It works — and its defaults are where this article's central problem lives.

Forklift's planning documentation publishes default volume and access modes per provisioner. For openshift-storage.rbd.csi.ceph.com and kubernetes.io/rbd, the documented default is Block plus ReadWriteOnce. Set that beside KubeVirt's rule that live migration requires ReadWriteMany, and the collision is plain: a bulk migration onto Ceph RBD that does not override the StorageMap produces, by documented default, a fleet that cannot be live-migrated. Every VM boots. Every VM passes its smoke test. The estate finds out at the first drain, when the fix — recreating disks — is at its most expensive.

apiVersion: forklift.konveyor.io/v1beta1
kind: StorageMap
metadata:
  name: vsphere-to-rwx
  namespace: konveyor-forklift
spec:
  provider:
    source:
      name: vcenter-prod
      namespace: konveyor-forklift
    destination:
      name: host
      namespace: konveyor-forklift
  map:
    - source:
        id: datastore-1042        # vSphere datastore MOref
      destination:
        storageClass: rook-ceph-block-rwx
        accessMode: ReadWriteMany  # overrides the documented RWO default
        volumeMode: Block
---
apiVersion: forklift.konveyor.io/v1beta1
kind: Plan
metadata:
  name: ledger-tier
  namespace: konveyor-forklift
spec:
  warm: true                       # precopy now, cut over in the window
  targetNamespace: platform-vms
  provider:
    source:
      name: vcenter-prod
      namespace: konveyor-forklift
    destination:
      name: host
      namespace: konveyor-forklift
  map:
    network:
      name: vsphere-to-pod
      namespace: konveyor-forklift
    storage:
      name: vsphere-to-rwx
      namespace: konveyor-forklift
  vms:
    - id: vm-2871

Warm migration suits anything with a real cutover constraint, and it has documented edges. Disks copy incrementally using changed block tracking snapshots taken hourly by default; a VM supports up to 28 CBT snapshots, and warm migration may fail if the controller cannot create a new one. CBT must be enabled on each source VM and disk beforehand, and warm migration is not supported for OpenStack or OVA sources at all. Forklift is equally direct about vSphere throughput: a VDDK image is optional but highly recommended, and going without it can mean significantly lower speeds — while storing it in a public registry might violate VMware's licence terms. Build it; keep it in your own registry.

What KubeVirt Still Does Worse Than a Mature Hypervisor

An honest comparison serves a steering committee better than an enthusiastic one. Every gap below comes from KubeVirt's own documentation, not a competitor's table.

• No memory overcommit or ballooning. The docs say KubeVirt *does not yet* support classical memory overcommit management or ballooning — VirtualMachineInstances cannot give back allocated memory. Keep the qualifier; it is a roadmap gap, not a law. Size RAM as though it were permanent.
• KSM is the partial mitigation, and it is a trade. Kernel same-page merging can be started on the node; the more aggressively it is tuned, the more CPU it consumes.
• The descheduler ignores VMs by default. KubeVirt VMs are handled as pods with local storage, so it will not evict them unless you add the descheduler.alpha.kubernetes.io/evict annotation to the VMI template. No DRS-equivalent out of the box.
• No same-node volume migration. Volume migration relies on live migration, which only runs between separate nodes — moving a disk between classes on one host is impossible. It does work between two RWO PVCs on different hosts.
• Volume migration excludes real disk types. Forbidden for shareable disks (no multi-writer consistency guarantee), filesystem disks (virtiofs does not currently support live migration), and LUN disks.

One limitation belongs to this article rather than the software: everything above argues from documentation, not a benchmark. There are no measured migration times here, because none could be sourced honestly. If your decision needs *how fast*, that number must come from a proof-of-concept on your own hardware — the right place for it anyway.

Security and Audit Posture: What the First Public Audit Found

Security review is where a virtualisation change clears or stalls, so bring numbers. KubeVirt's first public security audit — run by Quarkslab through OSTIF — produced 15 findings with security impact: 0 Critical, 1 High (CVE-2025-64324), 7 Medium, 4 Low and 3 Informational. The project describes it as a critical step toward Graduation within the CNCF framework, and the first time it has been publicly audited.

Read that as it is. A first audit finding no Critical issues in a codebase that runs guest kernels is genuinely good; that it was the *first*, in 2025, for a project accepted to the CNCF in September 2019 says how long it went unexamined. KubeVirt remains at Incubating maturity, where it has been since 19 April 2022 — not Graduated. Put that sentence in the risk register. It pairs with the substrate: an immutable node OS under the hypervisor does more for the attack surface than any tuning above it, because a VM escape lands where there is no shell.

The Exit Ramp Out of KubeVirt Itself

Adopting KubeVirt to escape one virtualisation lock-in is only progress if leaving KubeVirt is cheaper than leaving what you left. The mechanism is documented: a declarative Export API plus virtctl vmexport, which downloads a volume compressed by default or raw on request. Raw is the point — that image boots under plain QEMU/KVM, oVirt, Proxmox or anything else reading a block device.

# 1. Snapshot the running VM so the export has a consistent point in time.
kubectl apply -f - <<'EOF'
apiVersion: snapshot.kubevirt.io/v1beta1
kind: VirtualMachineSnapshot
metadata:
  name: ledger-db-01-exit
  namespace: platform-vms
spec:
  source:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: ledger-db-01
EOF

kubectl -n platform-vms wait virtualmachinesnapshot/ledger-db-01-exit \
  --for=jsonpath='{.status.readyToUse}'=true --timeout=30m

# 2. Export it and pull the disk down as raw. --port-forward avoids needing
#    an ingress or route just to prove the exit works.
virtctl vmexport download ledger-db-01-exit \
  --namespace=platform-vms \
  --snapshot=ledger-db-01-exit \
  --volume=ledger-db-01-root \
  --format=raw \
  --output=ledger-db-01-root.img \
  --port-forward

# 3. Inspect it. This is metadata, not evidence the guest boots. Note that
#    "qemu-img check" is unavailable here: only qcow2, qed, parallels, vhdx,
#    vmdk and vdi support consistency checks, so raw exits 63.
qemu-img info ledger-db-01-root.img

# 4. The step that is actually proof: boot it outside KubeVirt.
qemu-system-x86_64 -machine q35,accel=kvm -m 4096 -smp 2 \
  -drive file=ledger-db-01-root.img,format=raw,if=virtio -nographic

Two version notes, because this detail rots. The export documentation still instructs adding VMExport to the KubeVirt CR's feature gates, but that gate reached GA in v1.9 — always enabled, no longer togglable — so the instruction is stale and the entry inert. Read your cluster's actual gate list, not a document: from v1.9 Beta gates are on by default and disabledFeatureGates is the only way out. The OCI path is the opposite case — --format=oci downloads an OCI image layout TAR and does need the OCIExport gate, still Alpha and off until enabled. The artefact then travels through an ordinary registry via skopeo, crane or oras, which your supply-chain controls already understand.

Click to zoom

The Long Game: Virtualisation as a Commodity You Own

Hardware virtualisation runs on kernel primitives — KVM, libvirt, QEMU — open source and older than most of the companies charging for them. What was ever proprietary was the management plane: the scheduler, the console, the licence server. KubeVirt's argument is that Kubernetes already is that plane, and you run it.

On a ten-year horizon the question is not which hypervisor is better this quarter. It is which one leaves you holding the three things that matter when terms change again: the scheduler, the storage, and a disk format anything can read. Provision the metal with tooling you control, make access modes a decision rather than a default, and rehearse the export. The next repricing is then a commercial event you can answer, not a bill you can only pay.