Leaving VMware Without Kubernetes: Proxmox VE in Production

Every conversation about leaving vSphere reaches the same fork, usually unnamed. One branch containerises the platform and runs the surviving VMs on Kubernetes. The other keeps the VM as the unit of operation and changes only who supplies the hypervisor. The first is a re-platforming programme wearing a migration's clothes; the second is a migration.

This catalogue already covers the first branch — KubeVirt: The VMware Exit for VMs You Cannot Containerise. This is the other branch, where Kubernetes never enters the design — and not an install walkthrough: results for setting up Proxmox are dominated by single-node installs that stop where production begins, the second node.

The branch you take before you migrate a single VM

The fork is decided by the fleet, not by taste. Put the VMs on Kubernetes when the estate is already Kubernetes-shaped — a platform team fluent in it, the VMs a shrinking remainder. The constraints then become Kubernetes constraints: KubeVirt documents that a VM on a PersistentVolumeClaim needs ReadWriteMany to live migrate, that live migration is blocked with a bridge-type pod network binding, and that ports 49152 and 49153 must be free in the virt-launcher pod.

Keep the VM as the unit when the estate is the other shape: stateful guests, vendor appliances supported only as an OVA, licence-pinned databases, Windows with a desktop on it. There Kubernetes asks a team to learn a new abstraction to keep running the old one; Proxmox VE asks for corosync quorum, a watchdog interval and a storage model. Cadence differs too: Kubernetes maintains its three most recent minor releases, roughly a year of patches each; Proxmox VE tracks Debian. Either way the nodes underneath still need bare-metal provisioning.

Click to zoom

What Broadcom actually changed, and what it forces

Start with primary sources. VMware's own vSphere Foundation FAQ, asked whether standalone SKUs exist for 9.0, answers: "No. Version 9.0 will only be available for VMware Cloud Foundation and VMware vSphere Foundation." Broadcom's knowledge-base article sets the unit — licensing counted on "the total number of physical CPU cores across all ESXi hosts you intend to license", with "a minimum of 16 physical cores for each CPU ... even if a CPU has fewer than 16".

Proxmox's model is deliberately boring by contrast: "A subscription is required for every occupied physical CPU socket. The number of CPU cores does not affect the price," with every node in a cluster on the same level. No feature is gated by tier — HA, live migration, clustering, backup and software-defined storage ship at every level; levels differ by support entitlement. At one socket the published annual totals are €1,100 PREMIUM, €550 STANDARD, €370 BASIC, €120 COMMUNITY.

The structurally interesting part is the licence, not the price: Proxmox VE code is licensed under the GNU Affero General Public License, version 3. A per-socket price can be raised; an AGPLv3 licence on the version you already run cannot be withdrawn. Only one of those is a dependency.

Cluster bring-up: quorum, corosync links, and the network you must not share

Proxmox is explicit about the floor: HA needs "at least three nodes for reliable quorum"; a two-node cluster borrows a third vote from a QDevice. The cluster stack "requires a reliable network with latencies under 5 milliseconds (LAN performance) between all nodes to operate stably", and above roughly 10 ms it "gets rather unlikely with more than three nodes". Corosync "does not use much bandwidth, it is sensitive to latency jitters", so Proxmox recommends a physically separated network: "Especially do not use a shared network for corosync and storage (except as a potential low-priority fallback in a redundant configuration)."

Hence a dedicated physical NIC for cluster traffic — Proxmox notes "a dedicated 1 Gbit NIC is enough in most situations" — plus a second link on a different physical network; corosync supports up to eight. What it will not accept as a substitute is a bond: "A single link backed by a bond can be problematic in certain failure scenarios." Redundancy inside one link is not two links.

The mechanics are short: UDP 5405–5412 between nodes, synchronised time, an SSH tunnel on TCP 22. All of it protects pmxcfs, the database-backed filesystem holding Proxmox VE configuration, replicated in real time by corosync — RAM-backed, capped at 128 MiB, and read-only on a node that loses quorum.

# Node 1 — corosync on its own NIC (link0), plus a separate link1.
pvecm create pve-prod --link0 10.10.0.11 --link1 10.20.0.11

# Node 2 — --link0 is not optional here. Node 3 uses .13 and .23.
pvecm add 10.10.0.11 --link0 10.10.0.12 --link1 10.20.0.12

pvecm status          # expect: 3 nodes, quorate — but no per-link state
corosync-cfgtool -n   # per-node link state; both enabled, LINK 0 connected

# token + consensus is the membership-reform budget. Measure it.
corosync-cmapctl | grep -Ew 'runtime.config.totem.token|runtime.config.totem.consensus'

The three clocks of a Proxmox HA failure

The part most worth getting right: Proxmox documents the same interval in three places at three levels of abstraction, and it is tempting to add them up. Do not: there are three clocks, and only three. Clock one runs constantly: every node running HA guests reports its presence to the cluster every ten seconds.

Clock two runs on the failed node and lasts about sixty seconds. The migration wiki puts it from outside: a node losing corosync connectivity to the quorate partition "will wait for about a minute to see if it can reconnect ... If not, the node will fence itself." The cluster-manager chapter states it numerically — HA nodes "fence themselves already after roughly one minute without a stable quorum" — and the HA manager chapter gives the mechanism: without quorum the node cannot reset its watchdog, which reboots it "after the watchdog has timed out (this happens after 60 seconds)". One countdown described three ways, not three in sequence. An active CRM with an idle LRM does not self-fence.

Clock three runs on the survivors and lasts about two minutes. The cluster "waits for some time (~2 minutes) in case the node comes back". Proxmox gives ha-manager's typical error-detection and failover time as about two minutes and draws the consequence itself — "so you can get no more than 99.999% availability", a ceiling imposed by the mechanism, not a figure any deployment achieves. The guest waits in the fence state, then recovers on an online, quorate node.

Click to zoom

The trap — an annotation on clock two, not a fourth clock. If corosync rides an LACP bond at default settings, "LACPDUs are only sent every 30 seconds, yielding a failover time of 90 seconds". Ninety is larger than sixty: the bond recovers into a node that already reset itself. The remedy is bond-lacp-rate fast on both node and switch — set on one side it requests an LACPDU per second from the other, and on both it "can reduce the failover time in the scenario above to 3 seconds and thus prevent fencing".

cat >/tmp/bond0.stanza <<'EOF'
auto bond0
iface bond0 inet manual
        bond-slaves eno3 eno4
        bond-miimon 100
        bond-mode 802.3ad
        bond-lacp-rate fast

auto vmbr1
iface vmbr1 inet static
        address 10.10.0.11/24
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
EOF

# Nothing below runs from here: /tmp is not what ifupdown2 reads. Merge the
# stanza into /etc/network/interfaces by hand first. Only then, on a node in
# HA maintenance mode, uncomment and run:
# ifreload -a
# grep -i 'lacp rate' /proc/net/bonding/bond0   # expect: fast

Membership reformation must also fit inside those sixty seconds. Corosync derives both timeouts from node count — token = 3000 + (number_of_nodes - 2) * token_coefficient, consensus = 1.2 * token — and with HA enabled Proxmox says "it is especially important that this time stays below 45 seconds to ensure that a new cluster membership is formed before the watchdog timeout of 60 seconds expires". Lowering the coefficient is graded "suggested" past 30 seconds, "recommended" past 40, "strongly recommended" past 45. That line is arithmetic, not a documented threshold: unset, the coefficient defaults to 650 ms, so the sum is 2.2 × (3000 + (n − 2) × 650) ms, and solving against 45000 gives n ≤ 28 — a ceiling, not a target, since Proxmox asks for room: "a safety margin below this 45 second limit is advisable". Since 9.2, new clusters get 125 ms in /etc/pve/corosync.conf; clusters upgraded from 8.x keep 650 ms — the estates worth measuring.

grep -q 'WATCHDOG_MODULE=' /etc/default/pve-ha-manager \
  || echo 'WATCHDOG_MODULE=' >> /etc/default/pve-ha-manager
sed -i 's/^#\?WATCHDOG_MODULE=.*/WATCHDOG_MODULE=iTCO_wdt/' /etc/default/pve-ha-manager
grep -n WATCHDOG_MODULE /etc/default/pve-ha-manager   # verify: not a no-op
systemctl restart watchdog-mux
systemctl is-active watchdog-mux

# Only once the intended watchdog is armed:
ha-manager add vm:101 --state started --max_restart 2 --max_relocate 2
ha-manager status

Shared storage: Ceph when you have the nodes, ZFS replication when you do not

Live migration and HA both need the guest's disk reachable from more than one node; Proxmox gives two non-interchangeable answers. Ceph is the hyper-converged one: "at least three (preferably) identical servers", at least twelve OSDs evenly distributed, one per physical disk, and at least 10 Gbps "to be used exclusively for Ceph traffic" — exclusively, because recovery traffic "will interfere with other services". Memory is the underestimated line: an OSD daemon "requires 4 GiB by default", 8 GiB per OSD recommended — so twelve OSDs is 96 GiB across the cluster, 32 GiB per node at the three-node minimum, before a guest.

Read that floor for what it is: Proxmox recommending its own product's configuration, not a neutral minimum. Ceph runs below it; Proxmox stops recommending it there. One setting turns a survivable failure permanent at any size: a replicated pool with min_size of 1 "allows I/O on an object when it has only 1 replica, which could lead to data loss, incomplete PGs or unfound objects". Ceph as a Kubernetes CSI layer is a different animal: Rook/Ceph vs Longhorn vs OpenEBS.

pveceph install --repository no-subscription   # on every node
pveceph init --network 10.10.20.0/24           # once, cluster-wide
pveceph mon create                             # per monitor node

pveceph osd create /dev/nvme0n1      # per disk, on every node
ceph osd tree                        # wait: OSDs up on all three nodes
pveceph pool create vm-rbd --size 3 --min_size 2 --pg_autoscale_mode on --add_storages 1

ceph osd pool get vm-rbd size        # expect: size: 3
ceph osd pool get vm-rbd min_size    # expect: min_size: 2 — never 1
ceph -s                              # expect: HEALTH_OK, PGs active+clean

Below that floor the honest answer is ZFS storage replication: redundancy for guests on local storage, snapshot-based, so after the initial full sync only new data moves. Proxmox allows HA alongside replication but warns "there may be some data loss between the last synced time and the time a node failed". The interval is therefore the ceiling on your recovery point objective, not the RPO itself — default fifteen minutes, minimum one, maximum a week. Writing "RPO: 15 minutes" into a policy records a ceiling and calls it a guarantee — the distinction Kubernetes disaster recovery meets from the other side.

Backup as the recovery tier: PBS retention, verification, and what prune does not delete

Replication is not backup: it copies corruption and deletion as faithfully as data. Proxmox Backup Server is the separate tier, outside the cluster's failure domain. Its mechanism explains its most surprising behaviour: PBS splits VM disk images into fixed-size chunks, typically 4 MiB, optionally encrypted client-side with AES-256 GCM, sent over TLS regardless. Chunks are shared between snapshots, which is why deleting a snapshot does not delete its data.

"When pruning a snapshot, only the snapshot metadata (manifest, indices, blobs, log and notes) is removed. The chunks containing the actual backup data ... have to be removed by a garbage collection run." Garbage collection "frees up space in a datastore by deleting all unused backup chunks" — weekly, as a documented starting point.

Retention arithmetic surprises: within each retained period only the latest backup is kept, and days without backups do not count, so a keep-daily of fourteen holds the last fourteen days with a backup, not fourteen calendar days. Verification is the third job and the first dropped: Proxmox recommends reverifying everything at least monthly even after a success, because drives degrade. A backup never restored and never reverified is a belief, not a recovery tier, the argument immutable backups takes to its conclusion.

# 1. Retention — keeps the latest backup within each retained period.
proxmox-backup-manager prune-job create vm-daily --store main \
  --schedule '03:30' \
  --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --keep-yearly 3

# 2. The job that actually frees space. Weekly is the documented start.
proxmox-backup-manager datastore update main --gc-schedule 'sat 02:00'

# 3. Verification, two tiers: cheap and frequent, then full.
proxmox-backup-manager verify-job create quick --store main \
  --schedule '05:00' --ignore-verified true --outdated-after 30
proxmox-backup-manager verify-job create full --store main \
  --schedule 'monthly' --ignore-verified false

Moving the fleet: ESXi import limits and what bites at scale

Proxmox ships an ESXi import path and its documentation is candid about the edges. Read the scope first: "Import was tested from ESXi version 6.5 up to version 8.0", on a wiki page that self-dates its supported-source list to November 2024. That is what it says today; it is not a statement about ESXi 9.

Three blockers change the plan, not the schedule; check them before sizing a wave. Disks backed by VMware vSAN cannot be imported; the documented workaround, moving them to other storage first, is a prerequisite project, not a step. Disks encrypted through a Storage Policy "cannot be imported"; the policy has to come off on the VMware side first, a change-control conversation. A datastore name containing + "might not work", and VMs carrying snapshots import "significantly slower".

VM=legacy-app-01

# 1. Pre-flight on the VMware side. Any hit stops this VM.
govc datastore.info | grep -i vsan            # vSAN-backed: blocked
govc collect -s "vm/$VM" config.keyId         # VM home encrypted: blocked
govc collect -json "vm/$VM" config.hardware.device \
  | grep -i keyId                             # encrypted disk: blocked
govc snapshot.tree -vm "$VM"                  # snapshots: much slower

# 2. All four clean? Only then take the guest down.
govc vm.power -s "$VM"                        # -off is the hard fallback
govc collect -s "vm/$VM" runtime.powerState   # expect: poweredOff

# 3. Register the source, then import by volume ID.
pvesm add esxi esxi-src --server 192.0.2.10 \
  --username migrate-svc --password 'CHANGE-ME' \
  --skip-cert-verification 1
pvesm list esxi-src                           # volume ID is column one
VOLID='CHANGE-ME'                             # paste from above
qm import 120 "$VOLID" --storage local-zfs

The failure mode at scale is concurrency. Proxmox's esxi-folder-fuse service "limits parallel connections to four and serializes retries of requests after getting rate-limited". Push past what ESXi serves and the migration stalls rather than slows: an overloaded API "will start blocking all requests, including all other running imports, which can result in hanging IO for guests that get live-imported". Live import trades downtime for risk — if it fails, all data written since it started is lost — and Proxmox advises against it on lossy networks. Run the fleet cold and serial.

The exit ramp from Proxmox, and the decade after the migration

A migration argument that omits its own exit is a sales argument. Proxmox's exit has three parts, worth designing on day one. The licence: AGPLv3 code cannot be withdrawn from the version you already run — the property Broadcom's customers discovered they did not have. The disk format: guest volumes are qcow2 or raw files, ZFS datasets, or RBD and LVM block volumes — raw bytes, never a proprietary container. And rehearsal: a restore from Proxmox Backup Server onto a non-Proxmox host is what makes this an exit ramp rather than an intention, the discipline that turns vendor lock-in into a number.

Then the cadence: roughly three years per major version, tied to Debian, with the FAQ publishing the table — Proxmox VE 8 on Debian 12 Bookworm, released 2023-06, Debian EOL 2026-07, Proxmox EOL 2026-08. That date is this month, for anyone still on 8. Direction of travel: 9.2 arrived 21 May 2026, its arm64 build 5 August 2026 — the project's first for a CPU architecture besides x86-64. Volume-chain snapshots remain a 9.0 technology preview, graduation still on the roadmap; a VM holding one on local storage cannot be migrated.

None of this makes Proxmox the answer. It makes it an answer with a different shape of risk: a smaller relearning cost, a support window measured in years, a licence that cannot be revoked, an operational model the team already has. If the fleet is Kubernetes-shaped, the KubeVirt branch is the better answer. If it is a few hundred stateful VMs that were never going to be containers, change the vendor and keep the model — and write down, before the first import, how you would leave the thing you are about to standardise on.