Hard Multi-Tenancy with vCluster: Retire 30 Clusters, Keep Isolation

The cluster count crept up the way it always does. A team wanted its own environment, so it got a cluster. A compliance scope needed separation, so it got a cluster. A staging environment, a data-science sandbox, a customer-specific deployment — each a reasonable request, each answered with a fresh Kubernetes cluster. Eighteen months later the platform team is operating thirty clusters, and the monthly bill has a line item that has nothing to do with running workloads: the control-plane fee. On managed Kubernetes that fee is roughly $73 per cluster per month before a single pod is scheduled. Thirty-two clusters is $2,336 a month, every month, to run empty control planes — and that number ignores the real cost, which is thirty upgrade cycles, thirty sets of add-ons, and thirty audit surfaces.

So a consolidation wave arrived in 2026, and the pitch is seductive: collapse the fleet back into one or two large clusters, give each team a namespace, reclaim 80% of the control-plane spend. The problem is that the teams did not adopt separate clusters for fun. They adopted them for a boundary — a hard line that says a mistake, a compromised workload, or a runaway process in one tenant cannot touch another. Naive consolidation dissolves exactly that boundary and trades a budget problem for a security incident. The interesting engineering question is not *whether* to consolidate. It is how to consolidate without losing the isolation the clusters were quietly providing.

This article is a tenancy-isolation architecture, not a product pitch. It walks the isolation spectrum from bare namespaces to fully hard tenancy, names precisely what each layer isolates and what it still shares, and shows where vCluster earns its keep and where it does not. It is deliberately distinct from policy enforcement — a Kyverno policy engine governs what a tenant may deploy, but it never draws the isolation boundary itself. Here we are concerned with the boundary.

Soft vs Hard: What 'Tenant' Actually Means

Multi-tenancy is not one problem; it is a spectrum of threat models, and the word 'tenant' hides which one you are solving. The distinction that matters is trust. Soft multi-tenancy assumes cooperative tenants — different teams inside the same organization who are not trying to attack each other but might accidentally interfere: a runaway job that starves the node, a mislabeled resource, a NetworkPolicy someone forgot to write. The isolation you need is guardrails against accident. Hard multi-tenancy assumes mutually-untrusted tenants — external customers, regulated workloads that must be provably separated, or internal teams where a breach in one must never permit lateral movement to another. The isolation you need is a security boundary that holds under an adversary.

The reason this framing is load-bearing: almost every 'we do multi-tenancy' claim is soft tenancy wearing hard-tenancy language. Namespaces, ResourceQuotas, and RBAC are excellent at preventing accidents and nearly useless against a determined adversary who has code execution in a pod. The Kubernetes threat surface at that point is the shared kernel and the shared API server — neither of which a namespace boundary constrains. Deciding which model you actually owe your tenants — contractually, and under your compliance regime — is the first architectural decision, and it dictates everything downstream.

Click to zoom

Read the spectrum left to right as a sequence of surfaces you stop sharing. Namespaces stop sharing names and quota. Capsule stops sharing policy ownership. vCluster stops sharing the API server and control-plane state. Node isolation stops sharing the scheduler's node pool. A kernel sandbox stops sharing the syscall boundary. Hard tenancy is not a product — it is the point on this line where the surfaces you still share are ones an adversary cannot cross. Where that point sits depends on your threat model, and paying for isolation past it is waste.

Namespaces Are the Floor, Not the Answer

The baseline every cluster already has is the namespace, and it is genuinely useful. A namespace scopes names, is the unit RBAC binds to, and is the object a ResourceQuota and LimitRange attach to. A competent soft-tenancy setup layers four things onto each tenant namespace: a ResourceQuota capping aggregate CPU/memory/storage and object counts, a LimitRange setting per-pod defaults so one workload cannot request the whole node, a default-deny NetworkPolicy so pods cannot reach other tenants' pods, and RBAC RoleBindings scoping the tenant's service accounts to their own namespace. That configuration is fast to stand up and operationally cheap, and for cooperative internal teams it is frequently the right and sufficient answer.

The failure modes are precise, and knowing them is how you decide when the floor is not enough. First, the shared API server: every tenant's requests hit one kube-apiserver, so a tenant who can craft expensive list/watch calls, or who exploits a CRD conversion webhook, is attacking infrastructure every other tenant depends on. Second, cluster-scoped resources: CRDs, ClusterRoles, PriorityClasses, and admission webhooks are not namespaced. Two tenants cannot each install their own version of a CRD; whoever installs it wins, and the other tenant inherits it. Third, and most important, the shared kernel: pods from different namespaces run on the same nodes and share one Linux kernel. A container escape — a kernel CVE, a misconfigured hostPath, a privileged pod — crosses the namespace boundary as if it were not there, because at the kernel level it is not.

Namespace-as-a-Service: Capsule, and the HNC Cautionary Tale

The gap between 'a namespace' and 'a tenant' is that a real tenant usually owns *several* namespaces — dev, staging, prod — that must share the same policies. Managing that by hand across a fleet is where namespace-as-a-service tooling comes in. Capsule, a CNCF project, models this with a Tenant custom resource: you declare a tenant, its owners, and its guardrails once, and Capsule propagates RBAC, NetworkPolicies, resource quotas, and ingress/hostname restrictions consistently across every namespace that tenant creates. Tenants can self-service new namespaces within their assigned limits without a platform-team ticket, which is the operational win — it turns namespaces into a product the platform offers.

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: team-a
spec:
  owners:
    - name: [email protected]
      kind: User
  # Cap how many namespaces this tenant may self-provision
  namespaceOptions:
    quota: 5
  # Propagated into EVERY namespace the tenant creates
  resourceQuotas:
    scope: Tenant
    items:
      - hard:
          requests.cpu: "20"
          requests.memory: 64Gi
          limits.cpu: "40"
          limits.memory: 128Gi
  networkPolicies:
    items:
      - podSelector: {}
        policyTypes: ["Ingress", "Egress"]
        ingress:
          - from:
              - namespaceSelector:
                  matchLabels:
                    capsule.clastix.io/tenant: team-a
  # Confine the tenant's pods to a labelled node pool
  nodeSelector:
    tenant-pool: team-a

Capsule is lightweight, Kubernetes-native, and adds no extra control plane — and that is exactly its ceiling. Tenants still share one API server and one etcd. There is no CRD isolation: a cluster-scoped resource is still cluster-scoped, and a tenant who needs their own version of a CRD, their own admission webhook, or cluster-admin inside their space cannot have it without affecting everyone. Capsule is superb *soft* tenancy — the strongest form of it, arguably — but it does not cross into hard tenancy. It moves the boundary from one namespace to a governed group of them; it does not split the control plane.

There is a sharp optionality lesson buried in this layer. For years the alternative pattern was the Hierarchical Namespace Controller (HNC) from the Kubernetes SIG-Multi-Tenancy group, which nested namespaces in a tree with policy inheritance. In April 2025 the HNC repository was archived and is no longer maintained, with the official guidance being to migrate to Capsule or vCluster. Teams that had wired their tenancy model deep into HNC's nesting semantics inherited a migration they did not schedule. This is the long-game argument in miniature: the tenancy layer sits *underneath* every workload you run, so betting it on a component that can be archived is an expensive place to be wrong. Prefer the layers with the broadest maintenance base and the cleanest exit, because the cost of replacing this particular layer is measured in fleet-wide migrations.

Where vCluster Earns Its Keep: the Virtual Control Plane

vCluster changes the shape of the problem by giving each tenant its own Kubernetes control plane — a real API server, controller-manager, and datastore — running as an ordinary workload inside a namespace of the host cluster. From the tenant's kubectl, it is indistinguishable from a dedicated cluster: they get their own API endpoint, install their own CRDs, define their own ClusterRoles, and run at their own Kubernetes version. From the host's perspective, that entire virtual cluster is just a StatefulSet and a handful of pods in one namespace. This is the single move that pulls tenancy off the soft end of the spectrum, because tenants no longer talk to the host's API server at all.

Architecturally, a vCluster is one pod containing a unified process: the API server, controller-manager, datastore, and a component called the syncer. The default distribution is k3s with an embedded SQLite datastore — swappable for full upstream Kubernetes (k8s) or k0s, and SQLite is replaceable with etcd, Postgres, or MySQL when you need HA or durability guarantees. The tenant's API server handles all their control-plane objects locally. The syncer is the clever part: it watches the tenant's virtual API server for pods that need to actually run, and it copies (syncs) those pods down into the host namespace so the host's real kubelets schedule and execute them. Status flows back up. The tenant sees their pods running in their cluster; the host sees pods running in a namespace.

controlPlane:
  distro:
    k8s:                      # full upstream Kubernetes, not k3s
      enabled: true
  backingStore:
    etcd:
      deploy:
        enabled: true        # dedicated etcd for this tenant's state
  # Tenant API server is not exposed to the shared network
  proxy:
    extraSANs: []
sync:
  toHost:
    pods:
      enabled: true          # syncer schedules tenant pods onto host nodes
    ingresses:
      enabled: false         # do not let tenants create host ingresses
  fromHost:
    nodes:
      enabled: true
      selector:              # tenant only sees its dedicated node pool
        labels:
          tenant-pool: team-a
policies:
  # Pods synced to the host inherit these hardening defaults
  podSecurityStandard: restricted

What this isolates is substantial and worth stating precisely. Each tenant has an independent API server, so an expensive or malicious API call degrades only that tenant's control plane, not the shared one. CRD collisions disappear — every tenant installs whatever CRDs and webhooks they like into their own API server, at their own version, with zero blast radius on other tenants. A tenant can even be cluster-admin inside their vCluster while having no privileges at all on the host. For the shared-API-server and shared-CRD failure modes that make namespace tenancy leak, vCluster closes them by construction. This is why it is materially stronger isolation than namespaces — and why the consolidation math works, since the host runs no per-tenant managed control-plane fee.

Click to zoom

True Hard Tenancy: Node Isolation and Kernel Sandboxing

Closing the last gap means the tenant's pods must stop sharing nodes and, ideally, stop sharing the kernel. Node isolation is the first half. You carve a dedicated node pool per tenant (or per trust-tier), taint those nodes so nothing schedules there by accident, and configure the tenant's synced pods with the matching tolerations and a nodeSelector. Now a compromised pod is confined to nodes that host only that tenant's workloads — lateral movement at the node level reaches nothing new. vCluster's Private Nodes feature (v0.27+) takes this further, letting external nodes join a tenant cluster directly with their own CNI and CSI stack, giving genuinely independent networking and storage rather than a shared host overlay.

The second half is the kernel. Dedicated nodes stop cross-tenant escape but do not stop a tenant escaping to the node it legitimately shares with *its own* workloads, and for the strictest regimes even that is unacceptable — you want the syscall boundary itself sandboxed. That is what gVisor (a user-space kernel that intercepts syscalls) and Kata Containers (lightweight per-pod microVMs) provide, wired in through a Kubernetes RuntimeClass. A pod that runs under gVisor never speaks to the host kernel directly; a Kata pod runs in its own VM with its own kernel. Combined with a per-tenant control plane and dedicated nodes, this is hard tenancy: control plane, nodes, and kernel are all unshared.

# 1. Register the sandbox runtime (nodes must have runsc/containerd-gvisor)
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
scheduling:
  nodeSelector:
    sandbox: gvisor            # only nodes advertising the sandbox runtime
---
# 2. Tenant workload: dedicated node pool + sandbox + hardened context
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tenant-workload
  namespace: team-a
spec:
  template:
    spec:
      runtimeClassName: gvisor  # syscalls hit gVisor, not the host kernel
      nodeSelector:
        tenant-pool: team-a     # dedicated, tainted node pool
      tolerations:
        - key: tenant
          operator: Equal
          value: team-a
          effect: NoSchedule
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: app
          image: registry.internal.example.com/team-a/app:1.4.2
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

This layering is also where tenancy meets the rest of your platform hardening. A dedicated, sandboxed node pool is only as trustworthy as the workloads admitted onto it, which is why the isolation boundary and the policy engine that enforces what may run inside it are complementary, not alternatives — the boundary contains the blast; the policy engine reduces the chance of a blast. Layer in identity-aware, zero-trust networking with Cilium and eBPF and the network path between tenants is authenticated and default-denied rather than merely namespaced. Hard tenancy is a stack, and vCluster is the control-plane course in it — not the whole meal.

The Consolidation Economics: Retiring 30 Clusters Honestly

Return to the thirty clusters. The headline saving is real: at roughly $73 per managed control plane per month, thirty-two clusters spend $2,336 monthly on control planes alone, and consolidating tenants into virtual clusters on one or two host clusters reclaims up to 80% of that — the widely-cited figure from 2026 consolidation case studies. But the honest accounting does not stop at the managed-control-plane line. You are moving that control-plane work onto host-cluster compute: each vCluster's API server, controller-manager, and datastore consume real CPU and memory on the host, and a fleet of them is not free. The saving is genuine but smaller than the gross line suggests, and it is a trade of predictable managed fees for host capacity you now operate yourself.

The larger, less-visible saving is operational surface. Thirty clusters is thirty upgrade cycles, thirty CNI installations, thirty sets of cluster add-ons, thirty audit boundaries, thirty places for configuration to drift. Consolidating to a small number of host clusters running many vClusters collapses the number of things the platform team upgrades and audits, while each *tenant* still gets a cluster-shaped experience they can version independently. This is the sovereignty argument, not just the cost one: you own a small, coherent, deeply-understood substrate instead of renting thirty shallowly-understood ones. It is the same instinct behind repatriating an over-fragmented cloud estate onto engineering you control — fewer, better-understood systems beat many rented ones.

The Exit Ramp: vCluster Is Standard Kubernetes

Any tenancy layer you adopt is a dependency underneath everything, so the anti-lock-in question is not optional: if this layer disappears or disappoints, what does leaving cost? vCluster answers well because a tenant's virtual cluster is a conformant Kubernetes API. The manifests, Helm charts, and operators a tenant runs inside a vCluster are ordinary Kubernetes objects — nothing is expressed in a proprietary dialect. Migrating a tenant off vCluster and onto a dedicated cluster is a kubectl get/apply of their resources against a new API endpoint, because the source and destination speak the same API. That is a genuine exit ramp, and it is the direct opposite of the HNC situation, where the tenancy model was expressed in a bespoke nesting semantics that had to be unwound by hand when the project was archived.

There is a licensing seam worth naming for procurement. The vCluster core is open-source (Apache-2.0) and CNCF-adjacent; the commercial platform layer (management UI, sleep/wake, fleet features) is a paid product from the vendor. The optionality-preserving posture is to build on the open core and treat the platform layer as a convenience you can add or drop, not a foundation you are married to — keep your tenant definitions, node pools, and policy in your own GitOps repository so the substrate, not the vendor's console, is your source of truth. That way the engagement is with an open standard you can walk away from, not a console you would have to reproduce. Sovereignty here is concrete: your tenancy model lives in Git, expressed in Kubernetes primitives, portable to any conformant cluster.

Decision Framework: When Each Tier Is the Right One

Isolation is not free, and buying more of it than your threat model requires is waste dressed as diligence. Match the tier to the tenant, and revisit the choice as trust boundaries change.

• Cooperative internal teams, shared budget, no compliance separation mandate: namespaces with ResourceQuota, LimitRange, default-deny NetworkPolicy, and scoped RBAC. Add Capsule when teams need self-service across multiple namespaces. This is soft tenancy, and for most internal platforms it is correct and sufficient.
• Teams that need their own CRDs, their own operators, their own Kubernetes version, or cluster-admin in their space — but still trust each other not to attack: vCluster on a shared node pool. Independent control planes remove the CRD-collision and shared-API-server problems without the cost of dedicated clusters.
• Mutually-untrusted tenants — external customers, regulated workloads, or any 'a breach here must not reach there' requirement: vCluster plus a dedicated tainted node pool plus a gVisor/Kata RuntimeClass. This is hard tenancy: control plane, nodes, and kernel all unshared. Nothing cheaper actually delivers the boundary.
• Tenants under separate regulatory jurisdictions, sovereignty mandates, or contractual physical-separation clauses: genuinely dedicated clusters (or vCluster Private Nodes with fully independent CNI/CSI) may still be required. Consolidation has a floor; know where yours is before you promise a saving you cannot safely deliver.
• Any tier: version the tenancy definitions in Git and enforce them with an admission policy engine. The isolation boundary and the policy layer are complementary; neither substitutes for the other.

The long-game framing is that tenancy is one of the few decisions that is genuinely hard to reverse, because it sits beneath the workloads rather than beside them. A cluster you can re-platform in a weekend; a tenancy model wired through thirty teams' assumptions you cannot. So the durable move is to pick the *lowest* tier that satisfies your real threat model, express it in portable Kubernetes primitives held in your own repository, and keep the exit ramp graded. Consolidation done this way is not a cost-cutting exercise that quietly weakens your security posture — it is a sovereignty upgrade that happens to also retire a stack of control-plane invoices.

If you are staring at a sprawl of per-team clusters and trying to decide which tenants can share a substrate and which genuinely cannot, that assessment is the kind of engagement we do — mapping your trust boundaries to the cheapest isolation tier that actually holds, before you commit a migration to it.