Terraform vs Kubernetes: Provisioning or Reconciling

Most pages answering this question stop at "Terraform provisions infrastructure, Kubernetes orchestrates containers, use both". True, and useless the first time a Deployment's replica count flaps between the number in your HCL and the number your autoscaler wants, or a load-balancer annotation vanishes on every apply.

Those failures share one cause. The same object has two owners, and the owners disagree about time: one acts when a pipeline runs, the other acts continuously. This article is about where to draw that line, what the documentation of each side says happens when you draw it badly, and what the operators that move cloud resources into a reconcile loop do to your exit plan.

Two different clocks: apply-and-exit versus a loop that never ends

Terraform runs to completion. Its record is the state file, and HashiCorp's state documentation is explicit about why it exists: Terraform "must store state about your workspace's managed infrastructure and configuration" and uses it "to map real world resources to your configuration". Before any operation it refreshes that record against reality, and the refresh is implicit: plan and apply "run an implicit in-memory refresh as part of their operations", while -refresh-only updates state without touching infrastructure and supersedes the deprecated terraform refresh subcommand.

Then the process exits. Between two runs, no Terraform process is watching — even scheduled drift detection is a periodic plan, not a loop. Drift is discovered when something plans, not when it happens.

Kubernetes inverts that. The controller documentation borrows its definition from control theory — "a control loop is a non-terminating loop that regulates the state of a system" — and says controllers "watch the state of your cluster, then make or request changes where needed", each trying "to move the current cluster state closer to the desired state". The same page accepts something no Terraform user would: "potentially, your cluster never reaches a stable state", and that is fine "as long as the controllers for your cluster are running and able to make useful changes".

"Provisioning versus orchestration" undersells the difference. One is an event: a reviewed diff, applied once, recorded. The other is a process: a desired state, defended indefinitely. Put an object under both and you have an event-driven owner and a continuous owner taking turns to overwrite each other — the continuous one always gets the last word until the next pipeline run.

Where the boundary usually belongs

A defensible default follows from the clocks. Things that change rarely, need a human-reviewed diff and must exist before the cluster does belong to Terraform: VPCs and subnets, IAM roles and policies, DNS zones, the managed Kubernetes cluster and node pools, managed databases and their backups. Things that change on every release, or are adjusted by other controllers at runtime, belong to a Kubernetes reconciler — usually GitOps through Argo CD or Flux: Deployments, Services, ConfigMaps, autoscalers, policies, CRD instances.

The grey zone is the platform layer in between: CRDs, the ingress controller, cert-manager, the GitOps controller itself. A workable test is the clock again. If a component is installed once per cluster and upgraded on a deliberate schedule, either owner can hold it, so pick one and record it. If anything else in the cluster will mutate it at runtime — a controller adding finalizers, a webhook injecting fields — give it to the reconciler, which already expects company.

The Terraform Kubernetes provider does not merely tolerate this split; its own documentation asks for it. When credentials for the provider are interpolated from other resources, "these resources SHOULD NOT be created in the same Terraform module where Kubernetes provider resources are also used", because this "will lead to intermittent and unpredictable errors which are hard to debug and diagnose". The recommendation that follows is unambiguous: the most reliable configuration is one where "the cluster itself and the Kubernetes provider resources can be managed with separate apply operations".

kubernetes_manifest, the resource for arbitrary objects, makes it a hard constraint. Its documentation says it "requires API access during planning time", so the cluster "cannot be created in the same apply operation", and HashiCorp recommends it "only for custom resources or resources not yet fully supported by the provider". A one-module cluster-plus-workloads layout fails at plan time on day one.

So the minimum shape is two root modules with two applies. Stage one builds the cluster; stage two reads it through data sources and configures the provider from those:

# Stage 2 root module — the cluster already exists; read it, do not create it.
data "aws_eks_cluster" "this" {
  name = "platform-eu-1"
}

data "aws_eks_cluster_auth" "this" {
  name = "platform-eu-1"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.this.token
}

Once that second stage exists, ask whether it should be Terraform at all. If what it manages ships on every release, a GitOps reconciler is the better owner, and stage two shrinks to the bootstrap: namespaces, the GitOps controller and its credentials. How that bootstrap scales across fleets is covered in the multi-cluster GitOps comparison; how secrets cross the same line is in the External Secrets Operator piece.

Click to zoom

What breaks when both own the same object

Each failure below is documented by the projects themselves, along with the fix they recommend.

Replica count: your HCL versus the autoscaler

The HorizontalPodAutoscaler documentation is direct. With an HPA enabled, "it is recommended that the value of spec.replicas of the Deployment and / or StatefulSet be removed from their manifest(s)". Otherwise every apply tells Kubernetes to scale "to the value of the spec.replicas key", which can result in "thrashing or flapping behavior". Terraform is just another applier here: each run resets the count, and the HPA scales it back. The same page warns of the cost of the fix — removing the field "may incur a one-time degradation of Pod counts as the default value of this key is 1" — and documents how to avoid it under both client-side and server-side apply. Follow it, not a bare merge.

Labels and annotations other systems wrote

Cloud controllers, service meshes and admission webhooks annotate objects for their own bookkeeping. The provider documentation states the result plainly: "Terraform will remove them since they are not presented in the code." The fix is provider-level ignore_annotations and ignore_labels, which take regular expressions. They do not reach pod templates: for those the provider points back to lifecycle.ignore_changes.

Server-Side Apply field managers

kubernetes_manifest "uses Server-side Apply to carry out apply operations", which puts Terraform inside Kubernetes' own ownership model. The Server-Side Apply reference records an owner for every field in managedFields; applying a different value to a field "owned by another manager will result in a conflict", deliberately, "to signal that the operation might undo another collaborator's changes". Forcing the write overrides the value and transfers ownership. The documented alternatives are to give up the claim by dropping the field from your configuration, or to match the server's value and become a shared manager.

The provider exposes this through a field_manager block with a name and a force_conflicts boolean, and through computed_fields, the fields "whose values can be altered by the API server during 'apply'", which defaults to metadata.annotations and metadata.labels. Name the manager after the owning pipeline so managedFields says which repository wrote what, and leave force_conflicts off: a conflict is the signal you want.

resource "kubernetes_manifest" "api_certificate" {
  manifest = {
    apiVersion = "cert-manager.io/v1"
    kind       = "Certificate"
    metadata = {
      name      = "api-tls"
      namespace = "payments"
    }
    spec = {
      secretName = "api-tls"
      dnsNames   = ["api.example.com"]
      issuerRef = {
        name = "letsencrypt-prod"
        kind = "ClusterIssuer"
      }
    }
  }

  field_manager {
    name            = "terraform-platform-stage2"
    force_conflicts = false
  }
}

The silent revert, in both directions

The fourth failure is the general case. A controller writes a field Terraform owns; the next plan proposes reverting it; whoever approves that plan without context undoes the controller. Or the reverse: Terraform writes a field a reconciler owns, and within one loop the reconciler puts it back, so the apply "succeeds" and changes nothing. Neither side logs an error. The only cure is a recorded owner per field, enforced by the tool's own escape hatches.

Concrete config: handing a field or an object across the line

The general Terraform mechanics — ignore_changes, removed blocks, -refresh-only — are covered in depth in the Terraform vs Ansible comparison. What follows is the Kubernetes-specific use of each.

Hand over a field. The lifecycle documentation frames ignore_changes as exactly this: it lets Terraform "share management responsibilities of a single object with a separate process". Ignored attributes are still used on create and ignored on update — so the value below is the starting replica count, and the HPA owns it afterwards. In kubernetes_deployment_v1, spec is a block list, so the path is spec[0].replicas, and replicas is a string so that an explicit zero can be told apart from unset:

resource "kubernetes_deployment_v1" "api" {
  metadata {
    name      = "api"
    namespace = "payments"
  }

  spec {
    replicas = "3"

    selector {
      match_labels = {
        app = "api"
      }
    }

    template {
      metadata {
        labels = {
          app = "api"
        }
      }

      spec {
        container {
          name  = "api"
          image = "registry.example.com/payments/api:1.4.2"
        }
      }
    }
  }

  lifecycle {
    # The HorizontalPodAutoscaler owns replica count after creation.
    ignore_changes = [spec[0].replicas]
  }
}

Hand over an object. When a workload moves to GitOps, Terraform must stop owning it without deleting it. A removed block with destroy = false does that; HashiCorp's documentation describes it as the way to "hand off management responsibilities to another tool or team after using Terraform for the initial provisioning". Merge the GitOps manifest first, confirm the reconciler reports it in sync, then delete the resource block and apply:

removed {
  from = kubernetes_deployment_v1.api

  lifecycle {
    destroy = false
  }
}

Prove the boundary holds. terraform plan -detailed-exitcode returns 0 for no changes, 1 for an error and 2 when changes are present. Run against the main branch — where configuration should already be applied — a 2 there most likely means something outside Terraform changed what Terraform owns, though a provider perpetual diff also returns 2, so read the plan before reverting anything. Pair it with the Kubernetes side of the ledger, the field managers on a contested object:

#!/usr/bin/env bash
# drift-gate.sh — run on main, where configuration is already applied.
# Usage: drift-gate.sh <terraform-root> [<namespace>/<deployment>]
set -euo pipefail

root="${1:?usage: drift-gate.sh <terraform-root> [<namespace>/<deployment>]}"
target="${2:-}"

if [[ -n "$target" ]]; then
  ns="${target%%/*}"
  name="${target##*/}"
  echo "Field managers on deployment/$name in $ns:"
  kubectl get deployment "$name" --namespace "$ns" --show-managed-fields \
    -o jsonpath='{range .metadata.managedFields[*]}{.manager}{"\t"}{.operation}{"\n"}{end}'
fi

rc=0
terraform -chdir="$root" plan -detailed-exitcode -input=false -no-color -out=drift.tfplan || rc=$?

case "$rc" in
  0) echo "No diff: live infrastructure matches configuration." ;;
  2) echo "Non-empty diff: likely drift outside Terraform; review the plan below." >&2
     terraform -chdir="$root" show -no-color drift.tfplan
     exit 2 ;;
  *) echo "terraform plan failed with exit code $rc" >&2
     exit 1 ;;
esac

The operators that blur it: Crossplane, tofu-controller, ACK

A family of projects asks the obvious question: if the control loop is the better model for workloads, why not for the database and the network too? Each puts cloud resources on the continuous clock, and each is a different answer.

Crossplane calls itself a control plane framework that configures software, "then monitors it throughout its lifecycle" and "automatically corrects the drift" — built on Kubernetes. The CNCF announced its graduation on 6 November 2025, citing the then-recent v2.0. The managed-resources documentation shows how far the loop reaches: forProvider is the "source of truth", and if "a user makes a change inside a Provider's web console, Crossplane reverts that change". Terraform would report a console edit at the next plan; Crossplane undoes it on its next poll.

Crossplane also builds the field-level hand-off into the schema. Settings in initProvider apply only at creation and "aren't enforced" afterwards — the documented example is an autoscaled node group whose desiredSize Crossplane does not reset. managementPolicies restricts what Crossplane may do; the default * is full control and an empty array pauses the resource. Both are beta features, enabled by default, and the provider determines support — check yours before relying on them. The crossplane.io/poll-interval annotation overrides the controller's default --poll-interval per resource. The NodeGroup below follows the shape of the documentation's own example:

apiVersion: eks.aws.m.upbound.io/v1beta1
kind: NodeGroup
metadata:
  namespace: platform
  name: workers-eu-1
  annotations:
    crossplane.io/poll-interval: "1h"
spec:
  # No LateInitialize, as Crossplane recommends alongside initProvider.
  managementPolicies: ["Observe", "Create", "Update", "Delete"]
  initProvider:
    scalingConfig:
      desiredSize: 3   # set once; the cluster autoscaler owns it afterwards
  forProvider:
    region: eu-west-1
    clusterName: platform-eu-1
    scalingConfig:
      maxSize: 12      # enforced: a console edit is reverted
      minSize: 3
    # Node role and subnet references omitted; see the provider's NodeGroup CRD.

tofu-controller, the Flux project formerly named tf-controller, keeps HCL and puts Terraform itself inside the loop. Per its documentation it runs plan and apply "inside Runner Pods", stores "the TFSTATE of the applied resources as a Secret inside the Kubernetes cluster", and performs drift detection by default (switched off with spec.disableDriftDetection). What happens next depends on one field. If spec.approvePlan is omitted, the default is plan-and-manually-apply: the controller plans and "a human must then review and manually apply the changes." Only with spec.approvePlan: auto is "the plan to fix that drift" generated and applied automatically — the setting that turns every console edit into a revert at the next interval. spec.approvePlan: disable is the drift-detection-only model for an existing state, letting you "decide to do things later when a drift occurs" — the gentler entry point.

ACK, AWS Controllers for Kubernetes, lets you define and manage AWS service resources directly from Kubernetes. The relevant property for this argument is scope: it is AWS-only. Crossplane's providers span clouds through one resource model, tofu-controller keeps portable HCL, and ACK binds your cloud definitions to one vendor's CRDs — a fine trade if that vendor is permanent, and a lock-in surface if it is not. The ECS vs Kubernetes analysis makes the same argument one layer up.

The audit and security angle

The two models produce different evidence, and an auditor will ask for whichever one you have not got. Terraform yields a reviewed plan, an approval on a pull request, an apply log and a state history: a strong answer to "who approved this change, and what did they see?" It is weak on "what happened between runs?", because nothing ran between runs.

A reconciler yields the opposite: managedFields naming the writer of each field, Kubernetes Events and controller logs, and a loop that actively removes out-of-band change. That is strong evidence of continuous enforcement, weak evidence of intent unless the Git history behind the reconciler carries the review. A mature boundary keeps both: pull-request review for the desired state wherever it lives, and the loop's records for what happened next.

The credential trade-off deserves a sentence in the threat model. Terraform's cloud credentials can live in a CI runner that exists only while a pipeline runs. Crossplane and tofu-controller need cloud credentials held inside the cluster, permanently, scoped to whatever they manage. The cluster becomes an administrative plane for the cloud account, so its RBAC, its etcd encryption and the Secret holding tofu-controller's TFSTATE all inherit the sensitivity of that account. Scope each provider credential to the resources it reconciles, never to the account.

Exit ramps and the long game

Every boundary in this article can be moved later, if it was drawn with moving in mind. HCL and state stay portable: the same configuration runs under OpenTofu, the open-source fork, covered in the OpenTofu migration guide. A removed block releases Terraform's claim only with lifecycle { destroy = false }, as in the snippet above; by default it destroys the object. Crossplane's managementPolicies can narrow a resource to observation before another owner takes it — where the provider supports it. tofu-controller's drift-only model lets you trial the loop without granting it write authority.

None of those ramps work if nobody knows who owns what. The durable artefact is not a tool choice; it is a short ownership register — per resource class, the owning tool, the repository, the escape hatches in use (ignore_changes, initProvider, ignore_annotations) and the reason. It outlives every tool on this page, and it is what lets a team move a platform between environments without discovering mid-migration that two systems believed they owned the database.

Moving an object across the line is then a short, reversible sequence:

1. Record the new owner in the register before touching either tool.
2. Make the new owner observe only — Crossplane Observe, tofu-controller drift-only, or a GitOps application with automated sync disabled — and confirm it reports no difference.
3. Release the old owner without deleting anything: a removed block with destroy = false on the Terraform side, or managementPolicies narrowed on the Crossplane side.
4. Grant the new owner write authority, then run the drift gate on both sides to prove only one of them still claims the object.

Terraform and Kubernetes will keep converging — more providers, more controllers that wrap cloud APIs. That makes the rule more important, not less: one owner per object, written down. The tools are replaceable. The boundary is the part you have to own.