Hard Multi-Tenancy with vCluster: Retire 30 Clusters

The cluster count crept up the way it always does. A team wanted its own environment, so it got a cluster. A compliance scope needed separation, so it got a cluster. Staging, a data-science sandbox, a customer-specific deployment — each a reasonable request answered with a fresh Kubernetes cluster. Eighteen months later the platform team operates thirty clusters, and the monthly bill has a line item unrelated to workloads: the control-plane fee. On managed Kubernetes that fee is roughly $73 per cluster per month before a single pod is scheduled. Thirty clusters is $2,190 a month, every month, to run empty control planes — ignoring the real cost of thirty upgrade cycles, thirty add-on sets, and thirty audit surfaces.

So a consolidation wave arrived in 2026, and the pitch is seductive: collapse the fleet into one or two large clusters, give each team a namespace, and shed most of the managed control-plane fees. The problem is teams did not adopt separate clusters for fun. They adopted them for a boundary — a hard line so a mistake, compromised workload, or runaway process in one tenant cannot touch another. Naive consolidation dissolves that boundary and trades a budget problem for a security incident. The engineering question is not *whether* to consolidate, but how without losing the isolation the clusters quietly provided.

This article is a tenancy-isolation architecture, not a product pitch. It walks the spectrum from bare namespaces to hard tenancy, names what each layer isolates and still shares, and shows where vCluster earns its keep. It is distinct from policy enforcement — a Kyverno policy engine governs what a tenant may deploy, but never draws the isolation boundary. Here we care about the boundary.

Soft vs Hard: What 'Tenant' Actually Means

Multi-tenancy is not one problem; it is a spectrum of threat models, and the word 'tenant' hides which one you are solving. Soft multi-tenancy assumes cooperative tenants — teams in the same organization who might accidentally interfere (runaway jobs, mislabeled resources, a missing NetworkPolicy) rather than attack each other. Hard multi-tenancy assumes mutually-untrusted tenants — external customers, regulated workloads that must be provably separated, or internal teams where a breach in one must never permit lateral movement. Soft tenancy needs accident guardrails; hard tenancy needs a security boundary that holds under an adversary.

The reason this framing is load-bearing: almost every 'we do multi-tenancy' claim is soft tenancy wearing hard-tenancy language. Namespaces, ResourceQuotas, and RBAC prevent accidents well and stop almost nothing against a determined adversary with code execution in a pod. The shared kernel and shared API server are the real threat surface — neither is constrained by a namespace. Deciding which model you owe your tenants — contractually and under compliance — is the first architectural decision, and it dictates everything downstream.

Click to zoom

Read the spectrum left to right as surfaces you stop sharing. Namespaces stop sharing names and quota; Capsule stops sharing policy ownership; vCluster stops sharing the API server and control-plane state; node isolation stops sharing the scheduler's node pool; a kernel sandbox stops sharing the syscall boundary. Hard tenancy is not a product — it is the point where remaining shared surfaces are ones an adversary cannot cross. Paying for isolation past your threat model is waste.

Namespaces Are the Floor, Not the Answer

The baseline every cluster already has is the namespace, and it is genuinely useful. A namespace scopes names, is the unit RBAC binds to, and is where ResourceQuota and LimitRange attach. A competent soft-tenancy setup layers four things onto each tenant namespace: a ResourceQuota capping aggregate CPU/memory/storage and object counts; a LimitRange setting per-pod defaults so one workload cannot request the whole node; a default-deny NetworkPolicy so pods cannot reach other tenants; and RBAC RoleBindings scoping the tenant's service accounts to their own namespace. That configuration is fast and cheap, and for cooperative internal teams it is often sufficient.

The failure modes are precise. First, the shared API server: every tenant's requests hit one kube-apiserver, so expensive list/watch calls or a CRD conversion-webhook exploit attack infrastructure every other tenant depends on. Second, cluster-scoped resources: CRDs, ClusterRoles, PriorityClasses, and admission webhooks are not namespaced — two tenants cannot each install their own CRD version; whoever installs it wins. Third, and most important, the shared kernel: pods from different namespaces share one Linux kernel on the same nodes. A container escape — kernel CVE, misconfigured hostPath, privileged pod — crosses the namespace boundary as if it were not there, because at the kernel level it is not.

Namespace-as-a-Service: Capsule, and the HNC Cautionary Tale

The gap between 'a namespace' and 'a tenant' is that a real tenant usually owns *several* namespaces — dev, staging, prod — that must share the same policies. Capsule, a CNCF Sandbox project, models this with a Tenant custom resource: declare a tenant, its owners, and its guardrails once, and Capsule propagates RBAC, NetworkPolicies, resource quotas, and ingress/hostname restrictions across every namespace that tenant creates. Tenants can self-service namespaces within assigned limits without a platform ticket — namespaces become a product the platform offers.

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: team-a
spec:
  owners:
    - name: team-a-lead@example.com
      kind: User
  namespaceOptions:
    quota: 5                   # max namespaces this tenant may self-provision
  # Tenant-wide budget (scope: Tenant): aggregate hard limits across ALL of this tenant's namespaces, not per-namespace multiples
  resourceQuotas:
    scope: Tenant
    items:
      - hard:
          requests.cpu: "20"
          requests.memory: 64Gi
          limits.cpu: "40"
          limits.memory: 128Gi
  networkPolicies:
    items:
      - podSelector: {}
        policyTypes: ["Ingress", "Egress"]
        ingress:
          - from:
              - namespaceSelector:
                  matchLabels:
                    capsule.clastix.io/tenant: team-a
  # Confine the tenant's pods to a labelled node pool
  nodeSelector:
    tenant-pool: team-a

Capsule is lightweight, Kubernetes-native, and adds no extra control plane — that is exactly its ceiling. Tenants still share one API server and one etcd. There is no CRD isolation: a cluster-scoped resource stays cluster-scoped, and a tenant who needs their own CRD version, admission webhook, or cluster-admin cannot have it without affecting everyone. Capsule is superb *soft* tenancy — arguably the strongest form — but it does not cross into hard tenancy. It moves the boundary from one namespace to a governed group; it does not split the control plane.

There is a sharp optionality lesson here. For years the alternative was the Hierarchical Namespace Controller (HNC) from the Kubernetes Multi-Tenancy Working Group, which nested namespaces in a tree with policy inheritance. In April 2025 the HNC repository was archived by SIG Auth for lack of maintainers and adopters and moved to kubernetes-retired; upstream named no successor, leaving adopters to choose their own path (Capsule and vCluster being common landings). Teams that wired tenancy deep into HNC's nesting semantics inherited a migration they did not schedule. This is the long-game argument in miniature: the tenancy layer sits *underneath* every workload, so betting it on a component that can be archived is expensive. Prefer layers with the broadest maintenance base and cleanest exit — replacing this layer is a fleet-wide migration.

Where vCluster Earns Its Keep: the Virtual Control Plane

vCluster changes the shape of the problem by giving each tenant its own Kubernetes control plane — a real API server, controller-manager, and datastore — running as an ordinary workload inside a host namespace. From the tenant's kubectl it is indistinguishable from a dedicated cluster: own API endpoint, own CRDs, own ClusterRoles, own Kubernetes version. From the host's perspective, that virtual cluster is a StatefulSet and a handful of pods in one namespace. This is the move that pulls tenancy off the soft end of the spectrum: tenants no longer talk to the host API server at all.

Architecturally, a default vCluster runs as a control-plane StatefulSet pod holding the API server, controller-manager, syncer, and (with embedded SQLite) the datastore in one process. Enabling a deployed etcd backing store adds a separate etcd StatefulSet in the same host namespace. The default distribution is full upstream Kubernetes (k8s) with an embedded SQLite datastore. Both k0s and k3s were deprecated as distro options in v0.25; k0s was removed in v0.26, while k3s remained available until it too was removed, in v0.33. SQLite is replaceable with etcd: a chart-deployed etcd StatefulSet is available on the open-source tier, while embedded etcd is a Free-tier feature that requires connecting to vCluster Platform for license validation. High availability itself runs on the open-source tier too — both the deployed-etcd and embedded-etcd HA examples just set statefulSet.highAvailability.replicas: 3; external Postgres or MySQL backing stores are the Enterprise-licensed feature. The tenant API server handles control-plane objects locally. The syncer watches for pods that need to run and copies them into the host namespace so host kubelets schedule them; status flows back up. The tenant sees pods in their cluster; the host sees pods in a namespace.

controlPlane:
  distro:
    k8s:                      # full upstream Kubernetes (default distro)
      enabled: true
  backingStore:
    etcd:
      deploy:
        enabled: true        # dedicated etcd for this tenant's state
  # Default Service is ClusterIP (cluster-internal only). Other host namespaces can still reach that Service unless you add host NetworkPolicies.
  service:
    spec:
      type: ClusterIP
sync:
  toHost:
    pods:
      enabled: true
    ingresses:
      enabled: false         # block tenant-created host ingresses
  fromHost:
    nodes:
      enabled: true
      selector:              # pin tenant to labelled node pool
        labels:
          tenant-pool: team-a
policies:
  podSecurityStandard: restricted

What this isolates is substantial. Each tenant has an independent API server, so an expensive or malicious API call degrades only that tenant's control plane. CRD collisions disappear — every tenant installs CRDs and webhooks into their own API server at their own version, with zero blast radius on other tenants. A tenant can be cluster-admin inside their vCluster with no host privileges. For the shared-API-server and shared-CRD failure modes that make namespace tenancy leak, vCluster closes them by construction — which is why the consolidation math works: no per-tenant managed control-plane fee on the host.

Click to zoom

True Hard Tenancy: Node Isolation and Kernel Sandboxing

Closing the last gap means tenant pods must stop sharing nodes and, ideally, the kernel. Node isolation first: carve a dedicated pool per tenant (or trust-tier), taint those nodes, and give synced pods matching tolerations and a nodeSelector. A compromised pod is then confined to that tenant's nodes — lateral movement at the node level reaches nothing new. vCluster Private Nodes (v0.27+) goes further, letting external nodes join a tenant cluster with their own CNI and CSI for independent networking and storage rather than a shared host overlay.

The second half is the kernel. Dedicated nodes stop cross-tenant escape but not escape onto a node the tenant shares with *its own* workloads — unacceptable for the strictest regimes. gVisor (user-space kernel intercepting syscalls) and Kata Containers (per-pod microVMs), via RuntimeClass, sandbox the syscall boundary. A gVisor pod never speaks to the host kernel directly; a Kata pod runs in its own VM with its own kernel. With a per-tenant control plane and dedicated nodes, this is hard tenancy: control plane, nodes, and kernel all unshared.

# RuntimeClass (nodes need runsc/containerd-gvisor)
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
scheduling:
  nodeSelector:
    sandbox: gvisor
---
# Tenant workload: dedicated pool + sandbox
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tenant-workload
  namespace: team-a
spec:
  template:
    spec:
      runtimeClassName: gvisor
      nodeSelector:
        tenant-pool: team-a
      tolerations:
        - key: tenant
          operator: Equal
          value: team-a
          effect: NoSchedule
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: app
          image: registry.internal.example.com/team-a/app:1.4.2
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

Tenancy also meets the rest of platform hardening. A sandboxed node pool is only as trustworthy as the workloads admitted onto it — isolation and the policy engine that enforces what may run are complementary. Layer in identity-aware, zero-trust networking with Cilium and eBPF and the path between tenants is authenticated and default-denied, not merely namespaced. Hard tenancy is a stack; vCluster is the control-plane course, not the whole meal.

The Consolidation Economics: Retiring 30 Clusters Honestly

Return to the thirty clusters. At roughly $73 per managed control plane per month, thirty clusters spend $2,190 monthly on control planes alone, and consolidating tenants into virtual clusters on one or two host clusters can remove most of those managed control-plane fees (for example, 28 of 30 clusters retired is ~93% of the $73/cluster line item, before host CPU/memory for the in-cluster control planes) — not a free 80% industry benchmark. Honest accounting does not stop at that line. You move control-plane work onto host compute: each vCluster's API server, controller-manager, and datastore consume real CPU and memory, and a fleet of them is not free. The saving is genuine but smaller than the gross line — a trade of predictable managed fees for host capacity you operate yourself.

The larger saving is operational surface. Thirty clusters is thirty upgrade cycles, thirty CNI installs, thirty add-on sets, thirty audit boundaries, thirty places for config drift. Consolidating to a few host clusters running many vClusters collapses what the platform team upgrades and audits, while each *tenant* still gets a cluster-shaped experience they can version independently. This is the sovereignty argument, not just the cost one: you own a small, coherent substrate instead of renting thirty shallowly-understood ones — the same instinct as repatriating an over-fragmented cloud estate onto engineering you control.

The Exit Ramp: vCluster Is Standard Kubernetes

Any tenancy layer is a dependency underneath everything, so anti-lock-in is not optional: if this layer disappears or disappoints, what does leaving cost? vCluster answers well because a tenant virtual cluster is a conformant Kubernetes API. Manifests, Helm charts, and operators inside it are ordinary Kubernetes objects — nothing proprietary. Migrating onto a dedicated cluster is a kubectl get/apply against a new endpoint; source and destination speak the same API. That is a genuine exit ramp — the opposite of HNC, where bespoke nesting had to be unwound by hand when the project was archived.

There is a licensing seam worth naming for procurement. The vCluster core is open-source (Apache-2.0) and CNCF-adjacent — that core already covers running multiple StatefulSet replicas against a deployed etcd for high availability; the commercial platform layer (management UI, sleep/wake, fleet features, embedded etcd, external database backing stores) is a paid product from the vendor. Build on the open core and treat the platform layer as a convenience you can add or drop — keep tenant definitions, node pools, and policy in your own GitOps repository so the substrate, not the vendor's console, is source of truth. That way the engagement is with an open standard you can walk away from, not a console you would have to reproduce. Sovereignty here is concrete: tenancy lives in Git, in Kubernetes primitives, portable to any conformant cluster.

Decision Framework: When Each Tier Is the Right One

Isolation is not free; buying more than your threat model requires is waste. Match the tier to the tenant; revisit as trust boundaries change.

• Cooperative internal teams, shared budget, no compliance separation mandate: namespaces with ResourceQuota, LimitRange, default-deny NetworkPolicy, and scoped RBAC. Add Capsule when teams need multi-namespace self-service. Soft tenancy — correct for most internal platforms.
• Teams that need their own CRDs, operators, Kubernetes version, or cluster-admin — but still trust each other: vCluster on a shared node pool. Independent control planes remove CRD-collision and shared-API-server problems without dedicated-cluster cost.
• Mutually-untrusted tenants — external customers, regulated workloads, or any 'a breach here must not reach there' requirement: vCluster plus a dedicated tainted node pool plus a gVisor/Kata RuntimeClass. Hard tenancy: control plane, nodes, and kernel unshared. Nothing cheaper delivers the boundary.
• Separate regulatory jurisdictions, sovereignty mandates, or physical-separation clauses: dedicated clusters (or vCluster Private Nodes with independent CNI/CSI) may still be required. Consolidation has a floor; know yours before promising a saving you cannot safely deliver.
• Any tier: version tenancy definitions in Git and enforce them with an admission policy engine. Isolation boundary and policy layer are complementary; neither substitutes for the other.

Tenancy is one of the few decisions that is genuinely hard to reverse, because it sits beneath workloads rather than beside them. A cluster you can re-platform in a weekend; a tenancy model wired through thirty teams' assumptions you cannot. Pick the *lowest* tier that satisfies your threat model, express it in portable Kubernetes primitives in your own repository, and keep the exit ramp graded. Consolidation done this way is not cost-cutting that quietly weakens security — it is a sovereignty upgrade that also retires control-plane invoices.

If you are staring at a sprawl of per-team clusters and deciding which tenants can share a substrate, that assessment is the kind of engagement we do — mapping trust boundaries to the cheapest isolation tier that holds, before you commit a migration.