Replacing MinIO: Ceph vs SeaweedFS vs Garage

For most of the last decade, "self-hosted S3" had a one-word answer: MinIO. It was the default object store behind half the Kubernetes platforms ever built — the backup target for Velero, the WAL archive for Postgres, the chunk store for Loki and Thanos, the registry backend, the data lake. Then the default went dark. Through 2025 MinIO stripped the management console out of its community edition, stopped publishing Docker images, and in December 2025 moved the open-source server into maintenance mode. By February 2026 the project was marked *no longer maintained*, and on 25 April 2026 the `minio/minio` repository was archived read-only — no issues, no pull requests, no future. The thing under your backups stopped being a project and became an artifact.

This is not a post-mortem on MinIO. It is a decision framework for what comes next, because the wrong reaction to losing a default is to reach for the nearest drop-in replacement and reproduce the exact dependency you just got burned by. The object-storage tier is load-bearing infrastructure, and the teams that handle this well treat the replacement as an architecture decision, not a docker pull. The three open-source contenders worth serious evaluation in 2026 are Rook/Ceph's RADOS Gateway, SeaweedFS, and Garage, and they are not interchangeable. They encode three different bets about tenancy, object profile, jurisdiction, and how much operational weight you are willing to carry.

What follows is the honest comparison: what each system is, the axes that drive the choice, readable config, and how to leave whatever you pick. The goal is to own the storage your business depends on — and keep the exit ramp paved.

What Actually Happened to MinIO — and the Sovereignty Lesson In It

The lesson is in the sequence. MinIO was not acquired or shut down; it narrowed the open-source surface step by step: mid-2025 the full admin console (users, policies, lifecycle, monitoring) moved to commercial AIStor, leaving the open binary with little more than an object browser; Docker Hub images stopped; then maintenance mode, unmaintained, archived. A single vendor doing what single vendors eventually do with a project they fully control.

MinIO is licensed AGPLv3, and that license is irrevocable — the code that exists can be forked forever. So why is "just fork it" not the answer? Because a fork is a project, and a project needs maintainers to track CVEs and ship releases on a cadence regulated buyers can depend on. The community attempt to fork the console (OpenMaxIO) stalled almost immediately. Open *source* and open *project* are different guarantees, and only the second one protects you over a multi-year horizon. This is the same distinction that runs through all of our work: open source is a method, not a checkbox — what matters is governance, contributor diversity, and a license that keeps the exit ramp open, not the mere presence of a public repo.

Why the Object-Storage Tier Is Load-Bearing

It is tempting to treat object storage as a commodity. That framing is exactly why MinIO's exit caught so many teams flat-footed. The S3 endpoint is the substrate a sovereign platform quietly rests on. Your Velero disaster-recovery backups land in it. Your CloudNativePG WAL archive and point-in-time recovery stream to it continuously — lose the object store and your proven recovery point objective becomes a hope. Your self-hosted observability stack writes Loki log chunks, Tempo traces, and Thanos metric blocks to it. Container images, ML model artifacts, and the data lake all terminate there too.

The object store sits underneath all of them, so the choice deserves database-tier rigor: failure modes, operational burden, and jurisdiction controls first — speed fourth.

The Three Contenders, Honestly Described

Rook/Ceph RADOS Gateway (RGW). Ceph is a distributed storage system that provides block storage, file, and object from one cluster; Rook is the CNCF-graduated operator that runs it on Kubernetes. Object storage is exposed through the RADOS Gateway, a stateless S3/Swift front end you scale horizontally behind a load balancer. RGW has the most complete S3 API of the three — multipart, versioning, lifecycle, object lock, server-side encryption, bucket policies, STS-style tenancy — and Ceph is the only option here with erasure coding on the live write path plus battle-tested active-active multi-site replication. The price is operational weight: CRUSH maps, placement groups, OSD lifecycle, a multi-daemon topology, a real RAM floor per OSD, and a sane minimum of six storage nodes if you want 4+2 erasure coding at host failure domain (three if you stay replicated or use 2+1). Ceph is what you run when you need a multi-tenant object platform at scale and you have, or will build, Ceph expertise.

SeaweedFS. Apache-2.0 licensed and over a decade old, SeaweedFS began as a small-file distributed filesystem inspired by Facebook's Haystack paper, with an S3 gateway added on top. Its architecture is three tiers: a master tracking volume topology, volume servers that pack many small files into large volume files (giving O(1) disk seeks regardless of file count), and a filer that provides directory semantics and hosts the S3 API. Its superpower is billions of small objects — model artifacts, thumbnails, log fragments, CI caches — where it outclasses everything else, and its operations are far lighter than Ceph. The caveats are honest ones: S3 is a translation layer over the filer rather than a native design, and there are documented edge cases in multipart-plus-compression and high-concurrency filer consistency. It replicates on the write path; sealed, mostly-read volumes can later be converted to Reed-Solomon 10+4 erasure coding (1.4× overhead) via the erasure_coding worker / ec.encode (EC volumes then support reads and deletes only, not updates).

Garage. Written in Rust by the French non-profit Deuxfleurs and licensed AGPLv3, Garage is the youngest and most opinionated. It is a single statically linked binary with no external database — no etcd, no Postgres, no ZooKeeper — and it coordinates through CRDTs rather than a Raft/Paxos consensus on the data path, which is precisely what lets it stay responsive across high-latency links between distant nodes. It was designed from day one for heterogeneous, geo-distributed deployments: nodes in different cities, on different hardware, over the open internet. Its v2.0 release in mid-2025 added scoped admin tokens, key expiration, and website-redirect support. It deliberately forgoes erasure coding in favor of simple replication, and its S3 surface is narrower than RGW's. Garage is what you reach for when jurisdiction and low operational burden matter more than raw single-datacenter throughput.

A Decision Framework, Not a Default

Replacing one default with another is how you end up here again. Choose against the axes that distinguish these systems — six matter, in rough priority order.

• Jurisdiction and data residency. Do you need to guarantee that every object physically resides in specific countries? This is Garage's home turf — zone-aware replication places copies by geography as a first-class control. Ceph multi-site can do active-active across zones but at real operational cost. For most teams this axis, when it binds, binds hardest.
• Object profile: many small vs. fewer large. Billions of small objects favor SeaweedFS decisively. Large objects and aggregate throughput at scale favor Ceph. Garage handles small objects competently but is not optimized for peak local throughput.
• Tenancy. Rich multi-tenancy — per-user identities, bucket policies, quotas, isolation between teams — is Ceph RGW's strength. SeaweedFS and Garage offer access keys and bucket-level permissions suited to a trusted operator team, not a self-service multi-tenant platform.
• Operational burden. Garage is the lightest (one binary, no dependencies); SeaweedFS is moderate (three component types, well documented); Ceph is heaviest and demands genuine expertise. Be honest about the team you have, not the one you wish you had.
• Storage efficiency: erasure coding vs. replication. At hundreds of terabytes, Ceph's erasure coding (e.g. 4+2 ≈ 1.5× overhead) is dramatically cheaper than the 3× of replication. Below that, replication's simplicity usually wins. Ceph erasure-codes live data inline; SeaweedFS can EC-encode sealed warm volumes (RS 10+4, 1.4×) but not the write path; Garage is replication-only by design.
• S3 API completeness. If you depend on advanced features — object lock for WORM compliance, lifecycle transitions, STS — verify them explicitly. RGW is closest to AWS parity; SeaweedFS and Garage cover the core well but have edges. Test your actual workload's calls, not the marketing matrix.

Click to zoom

Architecture in Depth, with Config You Can Read

Each system's config reveals its philosophy more honestly than a feature table. Start with Ceph RGW under Rook: declare a CephObjectStore and the operator provisions the gateway and RADOS pools. Metadata stays replicated (never erasure-coded); the bulk data pool is erasure-coded for efficiency.

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: sovereign-store
  namespace: rook-ceph
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 3                 # metadata: 3x replication — never erasure-coded
  dataPool:
    failureDomain: host
    erasureCoded:
      # 4+2 with failureDomain: host requires >= 6 OSD hosts; use 2+1 on smaller clusters
      dataChunks: 4           # 4+2 erasure coding ≈ 1.5x overhead vs 3x replication
      codingChunks: 2
  preservePoolsOnDelete: true
  gateway:
    port: 80
    securePort: 443
    instances: 3              # RGW is stateless — scale horizontally behind a LB
    resources:
      requests: { cpu: "1", memory: "4Gi" }
  # A second zone here turns this into active-active multi-site replication.

SeaweedFS shows its filesystem heritage. There is no single binary that is "the object store"; you compose a master, volume servers, and a filer, and the S3 gateway is a thin process hung off the filer. The replication string (001, 010, 110) is SeaweedFS's compact encoding of how many copies to keep across servers, racks, and data centers.

# Master — tracks volume topology and the file → volume mapping.
weed master -mdir=/data/master -defaultReplication=010

# Volume servers — pack many small files into large volumes (O(1) seek).
weed volume -mserver=master:9333 -dir=/data/vol -max=200

# Filer holds directory semantics; the S3 gateway is layered on top of it.
weed filer -master=master:9333
weed s3 -filer=filer:8888 -config=/etc/seaweedfs/s3.json
# defaultReplication=010 → keep one extra copy on a different rack, same DC.

Garage is the inverse of Ceph's ceremony. Every node runs the same binary and config; there are no special roles. You set a replication factor, then assign each node to a zone — and the layout engine guarantees one replica per zone. Make each zone a jurisdiction and you have turned data residency into two lines of configuration.

# garage.toml (excerpt) — symmetric on every node, no external database.
replication_factor = 3
consistency_mode = "consistent"

# Then assign each node a zone = a jurisdiction. With replication_factor 3
# and three zones, every object gets exactly one replica in each:
#   garage layout assign <node-id> -z eu-central-de -c 2T
#   garage layout assign <node-id> -z eu-west-fr    -c 2T
#   garage layout assign <node-id> -z eu-north-se   -c 2T
#   garage layout apply --version 1
# Every object now physically resides in DE, FR, and SE. Nowhere else.

Click to zoom

Jurisdiction, GDPR, and the Sovereignty Boundary

Object storage is where data sovereignty stops being abstract. A bucket is a physical thing on physical disks in a physical country, and regulators increasingly care which one. The EU Data Act, applicable from 12 September 2025, forces portability and phases out cloud switching charges — capped at the provider's directly linked costs today, prohibited outright from 12 January 2027 — though egress billing for ordinary day-to-day use is unaffected; the CLOUD Act means a US-headquartered provider can be compelled to produce data regardless of which region it sits in. "Our data is in Frankfurt" and "we control who can compel access to our data" remain different statements — and only the second is sovereignty. This is the same gap that NIS2, DORA, and the EU AI Act have made legally material for regulated workloads.

Self-hosting the object tier closes that gap, and the system you pick determines how precisely. Garage lets you assert residency as configuration: a replica in Germany, one in France, one in Sweden, and provably nowhere else. Ceph multi-site can pin zones to locations but expects you to operate the clusters that back them. SeaweedFS supports cross-data-center replication but is not built for fine-grained geographic placement. For an auditor, the difference is concrete: residency stops being a provider's contractual promise and becomes a property of your own layout — the same audit-grade posture we hold for backups and the node OS underneath, where an immutable platform like Talos Linux makes the storage hosts auditable down to the boot.

Migrating Off MinIO Without a Big Bang — and the Exit Ramp for Next Time

The quiet good news: you can leave MinIO because it speaks S3, and so does everything you might move to. The S3 API is the portability contract that turns a vendor exit into a migration. The mechanical move is unglamorous — mirror, verify, cut over — and rclone or mc make it a background job, not a maintenance-window gamble.

# The S3 API is the portability contract. Mirror old → new, verify, then cut over.
rclone sync  minio:backups newstore:backups --checksum --transfers 32
rclone check minio:backups newstore:backups --download   # true byte-for-byte; plain --checksum misses multipart objects without Md5chksum

# Dual-write from the application during the bake period, then flip the
# endpoint in one config change once 'rclone check' is clean and stable.

Design the new deployment so the next exit is cheap: stable internal endpoint (not a vendor hostname), only the S3 features you need, credentials in your secrets layer. That is designing the exit ramp before you need it, applied to storage.

Choosing Well: A Recommendation Per Profile

For most teams the framework resolves cleanly. Jurisdiction across sites with a small team → Garage (accept replication-only as the price of control). Enormous small-object counts with lighter ops than Ceph → SeaweedFS (budget testing for S3 edges you touch). Multi-tenant object service, live-path erasure-coded efficiency at scale, or Ceph already for block/file → Rook/Ceph RGW (invest in the expertise; it does not run itself).

And if the honest answer is that no one on your team will own any of these yet, the disciplined move is to stay on a managed S3 for now and build the capability deliberately — the same operating-model shift that determines whether any cloud repatriation succeeds. Sovereignty is a goal pursued in proportion to readiness, not a mandate to self-host before you can operate it. Choosing to wait, with a plan to build, is itself a sovereign decision; drifting onto the next single-vendor default because it was the path of least resistance is not.

That is the payoff the S3 API standard delivers, and it shows up exactly when a vendor exits: when backups, the WAL archive, and the registry all speak S3 rather than a vendor's own API, migrating off a store that goes dark becomes a mirroring job to the new store, a content-level audit, and one changed endpoint — not an outage. The lesson to take from MinIO's archival, for anyone who had planned for it, was never 'pick a better store.' It was 'never let the store be something you can't walk away from.'

That is the sovereign object tier in one idea: not the fastest or most fashionable store, but the one whose tenancy, jurisdiction, and ops shape match the platform you run — and that you can leave when the landscape shifts. It will shift again. MinIO going dark was a reminder that defaults are someone else's business decision waiting to happen. Own the tier, keep to the portable standard, and the next reminder costs far less than this one did.