A Series-B logistics platform

The platform had started as a lean startup running on a single managed Kubernetes service with a handful of SaaS tools wired together. Over five years it grew to forty engineers, eight distinct product lines serving freight brokers and carriers, and a tooling profile that had become a material operating expense. Managed Kubernetes fees, proprietary CI/CD minute charges at scale, a SaaS code hosting platform with per-seat pricing that tracked headcount, a commercial container registry with image scanning add-ons, a SaaS identity provider, and a commercial monitoring platform — each individually justifiable when adopted, collectively a spend and vendor-dependency profile that no longer made sense at the team's size and maturity. The specific catalyst that moved the conversation from "we should do something about this" to "we need to act now" was a managed Kubernetes provider migration event. The provider announced a breaking change to their CNI implementation that required a cluster migration rather than an in-place upgrade. The team started planning the migration and discovered something concrete: they did not know how to perform a Kubernetes cluster migration. They had never needed to. The provider had always handled cluster lifecycle events, and the team's operational knowledge stopped at the kubectl surface. The migration took three times longer than the provider's documentation suggested, required escalations that the provider was slow to respond to, and produced a week of elevated anxiety across the engineering team. That event changed the frame from cost management to operational sovereignty: the team wanted to understand and own the platform they depended on.

The compounding tooling costs were a real number with a real CFO reviewing it, but they were not the primary driver. The primary driver was optionality: the team wanted to be able to answer "what do we do if vendor X doubles prices, deprecates this API, gets acquired, or has a multi-hour outage at a critical moment?" with a rehearsed plan, not a shrug and a support ticket. They had reached the stage where managed services that looked like cost efficiency from a startup stage were now genuine operating risks from a scale-up with enterprise customers and contracted SLAs.

The secondary driver was engineering team development as a deliberate strategy. The founders had made an explicit decision: they wanted to build a team with deep operational platform understanding, not a team that filed support tickets with SaaS vendors when something broke. That decision had real hiring and retention implications — engineers who wanted to build deep infrastructure competence were more interested in joining a team that operated its own platform than one that rented operational knowledge from vendors. The infrastructure architecture was also a talent architecture.

There was a practical catalyst on the cost side: a renewal cycle across several SaaS contracts was approaching within six months. The vendors had indicated price increases. The team needed to understand the fully-loaded cost of self-operation — including engineering time, hardware, and co-lo fees — before entering those renewal negotiations. If self-operation was cheaper on a fully-loaded basis, they should not renew. If it was comparable, the optionality and capability arguments still favoured self-operation. Only if it was materially more expensive than renewal was staying on managed services clearly correct.

The team had no prior experience with bare-metal Kubernetes, Talos Linux, distributed storage, or any of the OSS tools being considered as replacements. Any migration architecture had to support operational learning alongside migration — not as a precondition. The team could not spend six months in training before touching production; they needed to build operational confidence through direct experience on a non-production environment while production workloads migrated incrementally.

Physical infrastructure was constrained by the company's existing co-location arrangement: one rack in a single facility. Adding a second location for geographic HA was on the roadmap but not funded for the initial phase. The architecture had to function in a single-site configuration — with documented single points of failure clearly identified — and have a clear, documented path to multi-site that did not require re-platforming.

The CI/CD dependency was the most operationally sensitive. The existing pipeline had five years of configuration built on top of a SaaS CI/CD platform. It handled not just test runs but production deployments, with environment-specific secrets and approval gates that were integrated into the team's release process. Replacing it required either full pipeline migration (higher effort, cleaner result) or a transitional proxy period (lower initial effort, technical debt). The team chose full migration, which meant the CI/CD replacement had to be production-ready before the cut, not after.

Storage was the most technically demanding design decision. The team had used managed object storage and managed block storage without ever operating a storage layer. Distributed storage on bare metal — particularly Ceph — has a reputation for operational complexity. The architecture had to make Ceph manageable for a team encountering it for the first time, which meant Rook's operator model was necessary (not bare Ceph), the initial configuration had to be conservative (replication factor 3, no erasure coding until the team had Ceph operational experience), and the runbook had to cover failure modes in enough depth that the team could respond to a disk failure or OSD crash without escalating.

The engagement opened with a tooling audit: every SaaS dependency mapped by function, monthly cost, API surface the engineering team depended on (versus vendor-specific features they were actually using), and a candidate OSS replacement with a maturity assessment. The output was a ranked replacement list ordered by three factors: replacement maturity (how production-ready is the OSS equivalent, how large is the operational community), migration complexity (how much application code depends on vendor-specific APIs, how much pipeline configuration must be rebuilt), and cost impact (how much does replacing this tool move the cost number). Tools that ranked low on maturity or high on migration complexity were deprioritised; the goal was to ship a working platform, not to experiment with immature tooling.

The target architecture was designed as a layered build — not a migration plan requiring everything to move at once, but a platform that could be constructed alongside the existing environment and absorb workloads incrementally as each layer was validated. The sequence was: physical compute layer first (Talos nodes, Kubernetes, networking), then storage (Rook/Ceph), then platform services (Forgejo, Harbor, Keycloak, observability stack), then application workload migration, then CI/CD cutover last. Each layer was validated with non-production workloads before production traffic was admitted.

The principle applied consistently throughout tool selection: choose OSS tools with strong operational communities, comprehensive public runbooks, clear upgrade paths between major versions, and governance structures that meant the project would still exist in three years. The team would be operating these tools for years. Tools that were impressive in demonstrations but had thin operational documentation, small user communities, or governance structures dependent on a single company were excluded regardless of technical capability. This ruled out several technically capable options and kept the stack legible.

The migration model for each service was explicit about what would and would not migrate. Git history migrated fully — five years of commit history into Forgejo, verified by hash comparison after migration. CI/CD run history did not migrate — prior run logs were archived to Ceph object storage before the SaaS platform was cancelled, but the new pipeline started with a clean run history. This trade-off was documented and accepted; the team valued their git history significantly more than their CI run history. Being explicit about what migrated and what didn't avoided the failure mode of a migration that technically succeeded but left the team feeling like something important was lost without knowing what.

The full migration ran eleven months — incremental and deliberate, with production traffic running on both platforms simultaneously during the transition window for each service. The team operated the final platform independently from month six; the last four months of the engagement were observation, knowledge transfer, and runbook iteration, not active infrastructure construction. When the engagement formally closed, there was no hand-off meeting because there was nothing to hand off — the team had been operating the platform for four months already.

Representative outcomes across comparable engagements (not precise client figures): Tooling costs reduced by 50–70% compared to the prior SaaS footprint. The range reflects variation in the specific managed service mix: higher savings come from replacing commercial monitoring and CI/CD platforms (where per-seat and per-minute pricing at scale is expensive); lower savings come from engagements where the SaaS footprint was already partially rationalised before the migration began.

DR drills went from multi-week planning and execution exercises to sub-day rehearsals run by the team on a quarterly schedule. The prior environment's DR process required co-ordination with multiple SaaS vendors to understand recovery procedures; the new environment's DR process is entirely in the team's hands and documented in runbooks they wrote.

Zero SaaS hard-lock dependencies remain in the critical production path. Each replaced tool has a documented successor candidate in the runbook — if Forgejo were to become unmaintained, the migration path to another git host is documented; if Keycloak were to present an unacceptable upgrade complexity, the OIDC migration path is documented. The optionality is preserved structurally, not assumed.

Engineering team operational depth increased materially. This was the stated goal of the engagement from the founders' perspective, and the leading indicator — the team's comfort level with platform failure events — is qualitatively different from the cluster-migration incident that catalysed the engagement. The team now runs Ceph failure drills proactively. That change in posture is the outcome that persists when everything else is forgotten.

The existing application code — this was entirely a platform migration, not a rewrite. Zero lines of application code changed. The co-location rack (now running different compute on the same physical rails). The complete git history (migrated fully into Forgejo, hash-verified). The engineering culture of building rather than renting — the platform reinforces this on every operational interaction, because every failure mode is now the team's to diagnose and resolve, not a support ticket to file. The runbooks, architecture documentation, and GitOps repository belong to the team and are maintained by them. The on-call rotation for platform incidents is staffed entirely by the engineering team. No dependency on Stribog for day-two operations remains.