Kubernetes Block Storage You Control: Rook/Ceph, Longhorn, OpenEBS

Stateful workloads on Kubernetes stopped being controversial some time ago. Databases, message brokers, and object gateways all run in-cluster now, and the operators that manage them are mature. But almost every one of those operators assumes a volume already exists underneath it. Ask a database operator where its data lives and it will tell you: a PersistentVolumeClaim. Ask what makes that claim durable and it goes quiet, because that is somebody else's problem — and on a bare-metal cluster, somebody else is you.

That gap is where repatriation projects stall. A team moves compute off a hyperscaler, gets the control plane and the network right, then finds that the thing quietly holding the estate together was a managed block device — an EBS volume that survived instance termination, expanded on request, snapshotted on a schedule, and never made anyone think about failure domains. Replacing it is not a matter of picking a project off a landscape diagram. It is a decision about what kind of storage platform your team will operate.

This article makes that decision concretely: what the CSI contract obliges a block layer to provide, the three engines and how each replicates, the prerequisites that disqualify options before you benchmark, the config that separates a demo from a deployment, and the exit ramp. It is the block-storage counterpart to our guide on choosing a sovereign object tier — a different access pattern with different physics, not a decision to make by analogy from that one.

What the CSI Contract Actually Obliges You To Provide

The Container Storage Interface is a narrow contract: provision a volume, attach it to a node, mount it into a pod, then expand, snapshot, unmount, and delete it. Everything the engines compete on lives underneath that contract — which is why the contract, not the engine, is the durable seam in your architecture. Five obligations matter in practice, and each is somewhere a naive setup fails.

• Durability against the failure you expect. A block volume must survive the loss of the node it was attached to: synchronous replication across nodes, an external storage system, or an explicit decision that this workload replicates at the application layer instead.
• Access-mode semantics you can trust. Block volumes are ReadWriteOnce — one node at a time — and ReadWriteOncePod narrows that to a single pod, which is what you want for anything that would corrupt itself under concurrent writers. If you need many writers you need a filesystem, not a block device.
• Online expansion. Volumes are always sized wrong. allowVolumeExpansion plus a node-side filesystem resize is the difference between a kubectl patch and a migration.
• Snapshots with defined consistency. A CSI VolumeSnapshot copies a block device at a point in time. Unless the application was quiesced, it is crash-consistent — equivalent to pulling the power cable — not application-consistent.
• Topology awareness. The scheduler must know which nodes can reach a volume. WaitForFirstConsumer binding and correct topology keys are what stop a pod from being scheduled where its data is not.

The fourth obligation is quietly the most consequential. A crash-consistent snapshot of a database volume is usually recoverable, because journalling filesystems and write-ahead logs exist to make crash recovery survivable. Usually is not a recovery objective. If your RPO is a number someone signed, snapshots must be coordinated with the application — which is why a serious posture pairs CSI snapshots with the logical, application-aware backup path in our Kubernetes disaster-recovery guide.

Click to zoom

The Three Engines, Honestly Described

The filter for inclusion is the one we apply to every tier: a permissive licence, a real self-hosted deployment story rather than a hobbled community edition, governance that will not strand you, and an operational surface a team can carry at three in the morning.

Rook/Ceph RBD — the storage platform, not the storage feature

Ceph's RADOS Block Device is the most capable option here and the most demanding. A volume is striped into objects, the objects map to placement groups, and CRUSH deterministically distributes those groups across OSDs — one per disk — according to a failure domain you declare. Nothing in the path is special: no metadata server for block, no primary to fail over, and adding disks rebalances the cluster rather than partitioning it.

Rook makes this tractable. It has been a CNCF graduated project since October 2020 — accepted January 2018, incubating from September 2018 — the strongest governance signal in this comparison. Ceph continues its annual cadence, with Tentacle (v20) released in November 2025 after Squid (v19). Rook turns Ceph's surface into CRDs: a CephCluster describes the daemons, a CephBlockPool describes replication and failure domain, and a StorageClass exposes it to workloads.

The reason to choose it is leverage: the same disks serve RBD volumes, S3-compatible object storage through RGW, and shared filesystems through CephFS under one operational model. If you need more than one access pattern — and most platforms eventually do — Ceph is the only option here that does not mean a second system.

The reason to hesitate is that Ceph is a distributed system with its own vocabulary, and it is now yours. Production wants at least three storage nodes so three replicas land in three failure domains, three monitors for quorum, and a real understanding of min_size — because min_size: 2 versus min_size: 1 is the difference between refusing writes during a degraded window and accepting writes a later failure will lose.

Longhorn — replicated volumes a Kubernetes team can reason about

Longhorn takes the opposite position: instead of one distributed storage cluster, every volume gets its own tiny controller. A per-volume engine accepts writes and fans them synchronously to a configured number of replicas, each a sparse file on a separate node's disk. The blast radius of a bug is one volume, and the mental model is small enough to hold in your head.

It is a CNCF incubating project under Apache-2.0, with SUSE as primary sponsor — the commercial packaging is branded SUSE Storage while the upstream project remains Longhorn. The significant recent development is the V2 Data Engine, which reached general availability in v1.12.0 after arriving as a preview in v1.5.0. V1 exposes volumes through a user-space iSCSI target and the kernel initiator; V2 replaces that path with SPDK and NVMe-over-TCP in user space, which is where the latency difference comes from.

That performance is not free, and the prerequisites are a gate rather than a footnote: 1,024 pages of 2 MiB huge pages — 2 GiB — on every node running V2 volumes; the vfio_pci, uio_pci_generic, and nvme-tcp modules loaded persistently; Linux kernel 6.7 or later, because earlier kernels are subject to a memory-corruption issue; and one dedicated CPU core per node, because spdk_tgt polls and will consume a full core. On an immutable host OS those are machine-configuration changes, not in-cluster ones — exactly what Talos Linux makes declarative instead of a manual step someone forgets on node seven.

OpenEBS — Mayastor for latency, local engines for everything else

OpenEBS 4.x is two products with one name. Replicated PV Mayastor is a Rust NVMe-oF data plane: a per-volume nexus accepts I/O and replicates it synchronously over NVMe-over-TCP to replicas drawn from DiskPool resources. The local engines — LocalPV-LVM, LocalPV-ZFS, LocalPV-Hostpath — do something different and genuinely useful: node-local volumes with no replication, the correct answer for workloads that already replicate their own data.

Mayastor's prerequisites are as sharp as Longhorn V2's: 2 GiB of 2 MiB huge pages per storage node, the nvme-tcp and nvme-fabrics modules loaded, kernel 5.13 or later to mount NVMe-oF reliably, at least three worker nodes, two or more CPU cores per storage node, and node labels selecting which nodes run the I/O engine DaemonSet. NVMe devices are not merely recommended; the design assumes them.

The governance picture deserves a clear-eyed read, because it is where these three genuinely differ in risk. OpenEBS was accepted into the CNCF at sandbox level on 10 October 2024 — a re-entry, after the TOC moved it to archived in February 2024 and invited re-application once the concerns raised were addressed. Engineering is substantially funded by DataCore, which acquired the original vendor, MayaData, in November 2021. None of that makes the software bad; Mayastor is real and the local engines are excellent. It does mean one maturity tier below Longhorn and two below Rook, with a concentrated sponsor — weigh that when a volume you cannot lose will sit on it for a decade.

The Numbers That Decide It Before You Benchmark

Most of this decision is settled by arithmetic and hardware inventory, not throughput graphs. Three numbers do the work.

Usable capacity. Three-way replication means raw capacity divided by three. Twelve 3.84 TB NVMe drives across four nodes is roughly 46 TB raw and roughly 15 TB usable — before headroom for rebuilds, which you must reserve, because a cluster that cannot rebuild a failed OSD without filling up will fail closed at the worst moment. Erasure coding improves that ratio and suits the object tier; for block, replication stays the default.

Write amplification on the network. A synchronous three-replica write is one client write and two network writes, on the fabric your application traffic uses. That is why a dedicated storage network — or at minimum 25 GbE with the storage path isolated — belongs in the design rather than the second iteration.

What you are comparing against. The managed baseline is public: EBS gp3 lists at $0.08 per GB-month with 3,000 IOPS and 125 MB/s included, io2 at $0.125 per GB-month plus $0.065 per provisioned IOPS-month, and snapshots at $0.05 per GB-month incrementally. Run that for a 40 TB estate with snapshot retention and the storage line alone passes $4,000 a month, before replication traffic. That is the honest input to a repatriation model — the same discipline as our cloud-repatriation playbook, where the point is never that self-hosting is free, only that owning the asset changes the shape of the curve.

Click to zoom

Config That Is Actually Production-Grade

Each engine has a handful of settings that separate a demo from a deployment. For Ceph it is the pool's replication and failure domain — and refusing to let a size-1 pool exist by accident.

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: rbd-replicated
  namespace: rook-ceph
spec:
  failureDomain: host          # host, rack, or zone — must match reality
  replicated:
    size: 3
    requireSafeReplicaSize: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: block-replicated
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: rbd-replicated
  imageFormat: "2"
  imageFeatures: layering
  csi.storage.k8s.io/fstype: ext4
  # Encryption at rest with keys you hold, not keys the platform holds.
  encrypted: "true"
  encryptionKMSID: vault-transit
allowVolumeExpansion: true
reclaimPolicy: Retain          # Delete is a data-loss footgun on a shared class
volumeBindingMode: Immediate

reclaimPolicy: Retain earns its own sentence. With Delete, removing a PVC destroys the volume — correct for ephemeral CI workspaces, catastrophic for a database. Publish separate StorageClasses for the two intents and let the class name carry the meaning, because whoever deletes a namespace at 2 a.m. is not reading your pool spec.

For Longhorn: replica count, the anti-affinity that makes that count meaningful, and — on V2 — explicit data-engine selection.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-v2-nvme
provisioner: driver.longhorn.io
parameters:
  dataEngine: "v2"                     # requires hugepages + kernel >= 6.7
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
  fsType: ext4
  dataLocality: "best-effort"          # keep one replica on the writer's node
  replicaZoneSoftAntiAffinity: "true"
  # Per-volume encryption; the key lives in a Secret you control.
  encrypted: "true"
  csi.storage.k8s.io/provisioner-secret-name: longhorn-crypto
  csi.storage.k8s.io/provisioner-secret-namespace: longhorn-system
allowVolumeExpansion: true
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

volumeBindingMode: WaitForFirstConsumer is not a performance tweak but a correctness setting. Binding a volume before the pod is scheduled invites the scheduler to place the pod where the data is inconvenient — and on topology-constrained clusters, where it cannot run at all.

For Mayastor, the pool is the resource that matters: a DiskPool claims a whole block device on a labelled node, and volumes draw replicas from those pools.

apiVersion: openebs.io/v1beta3
kind: DiskPool
metadata:
  name: pool-node1-nvme0
  namespace: openebs
spec:
  node: worker-01
  disks:
    - /dev/disk/by-id/nvme-SAMSUNG_MZQL23T8HCLS_S64HNE0T123456
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mayastor-3
provisioner: io.openebs.csi-mayastor
parameters:
  protocol: nvmf
  repl: "3"
  fsType: ext4
  thin: "false"                # thick provisioning: fail at claim, not at 3 a.m.
allowVolumeExpansion: true
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

Thin provisioning is the other decision hiding in these manifests. Thin volumes let you over-commit and discover the shortfall when a write fails inside a running database; thick provisioning moves that failure to claim time, in a pull request, where it is a capacity conversation instead of an incident. Prefer the boring failure mode.

Whichever engine you run, instrument it before you trust it. All three export Prometheus metrics; the alerts that matter are degraded-replica count, rebuild activity, pool utilisation crossing the point where a rebuild can no longer complete, and per-volume p99 latency. Wire them into the self-hosted observability stack you already run — a storage tier whose degradation you learn about from application timeouts is unmonitored.

What an Auditor Will Ask About a Volume

A block volume holding regulated data is an in-scope system, and the questions are consistent across regimes. Four come up reliably.

• Encryption at rest, and who holds the key. All three engines support per-volume encryption via CSI secrets, and Ceph can source keys from an external KMS. The audit question is never whether the bytes are encrypted; it is whether a platform administrator can decrypt them without leaving a trace.
• Erasure. When a volume is deleted, is the data gone, or is it a freed extent a later thin allocation might hand to another tenant unzeroed? Confirm your engine's behaviour and pair reclaimPolicy with a documented wipe path if your regime demands one.
• Residency and attachment. Which machines, in which rack, in which jurisdiction, can attach this volume? On bare metal that is answerable exactly — a genuine advantage over a managed service where the answer is a region name.
• Restoration, rehearsed. Not whether snapshots exist, but when you last restored one into a fresh namespace and validated the application on top of it. An unrehearsed restore is a hypothesis.

The last is where most estates are weakest: a storage tier you have never recovered from has unknown properties. Rehearse the restore on a schedule with the recovery-objective arithmetic written down, and treat a rebuild after a deliberately failed disk as routine rather than an emergency. Storage failures are the ones where an untested assumption becomes permanent.

The Exit Ramp: Migrating Data You Cannot Re-Derive

Every other tier has a cheap exit because its state can be rebuilt — an index re-derived, a cache re-warmed, an image re-pulled. Block storage is the exception. The volume is the state, and moving it means copying every byte while something is writing to it. Design for that before you need it.

Three properties keep the choice reversible. First, workloads reference storage only through a StorageClass name — never a provisioner-specific parameter set buried in a Helm values file, and never a volumeName pinned to a particular PV. Swapping engines then means publishing a new class and re-issuing claims, not editing every chart you own.

Second, prefer application-level migration over block-level migration wherever the application supports it, because the application knows what consistency means. Adding a replica in the new StorageClass and promoting it is online, reversible, and verifiable, with a rollback that is simply not promoting. A dd between two block devices is none of those.

Third, where you must copy volumes, use a tool that understands CSI snapshots rather than a filesystem copy of a live mount. A snapshot-then-transfer path — Velero's data mover, or a controller built on VolumeSnapshot — gives you a defined consistency point and a restartable transfer. The dual-write-and-verify pattern that makes an object-tier migration safe does not translate cleanly to block, which is why this decision deserves more care up front.

The Long Game

Block storage is the most conservative decision in a platform, and it should be. Compute is fungible, networking reconfigurable, every stateless tier redeployable from a registry in minutes. The volumes are the part that cannot be re-derived from source control — and the part where a wrong decision compounds quietly, one unrehearsed restore at a time.

Which is why the governance read matters as much as the benchmark. Rook has been graduated for six years and rides a Ceph release train older than most of the cloud-native ecosystem. Longhorn is incubating, Apache-2.0, and has just shipped its SPDK engine to GA under a clear sponsor. OpenEBS re-entered the sandbox in October 2024 with concentrated backing and excellent local engines. All three are software you can read, fork, and run without asking permission; they differ in how many independent parties would keep them alive if the sponsor walked away. On a decade horizon that difference is the point of treating open source as a method rather than a licence checkbox.

So decide at the right altitude: how much storage platform do you want to operate, then pick the engine matching that answer rather than the one that wins a synthetic benchmark. Put every workload behind a StorageClass name. Replicate at exactly one layer and know which. Rehearse the restore. Then leave it alone — the best block tier is the one nobody has thought about since commissioning, holding data you could still get back if every vendor involved vanished tomorrow.