Argo Rollouts: Evidence-Gated Progressive Delivery

A kubectl rollout restart is a leap of faith. The Deployment controller swaps pods according to its rolling-update strategy, the new ReplicaSet reaches full size, the old one drains, and you find out whether the release was good by watching your dashboards and your support queue. That model works because most releases are fine. The releases that are not fine are the expensive ones — and a plain Deployment gives you no structural way to limit their blast radius or to make the promotion decision on evidence rather than on hope.

Progressive delivery replaces that leap of faith with small, measured, reversible steps: expose a new version to a fraction of traffic, measure it against the metrics that define 'healthy,' and widen only when the evidence says so. If not, the system reverts automatically without paging anyone — typically in seconds under the default traffic-routing abort path (stable left at full capacity). Argo Rollouts makes this a first-class Kubernetes deployment primitive rather than a bespoke pipeline of shell scripts.

This article walks through production Argo Rollouts: canary versus blue-green, AnalysisTemplate/AnalysisRun gates, traffic shaping via mesh and Gateway API, and the trade-offs marketing material skips. Framing is audit-grade: promotion as an evidence-gated event, decisions recorded and exported for retention, every release reversible.

Why the Deployment Object Is Not Enough

The Kubernetes Deployment is a fine abstraction for what it does: it manages ReplicaSets and performs rolling updates with configurable surge and unavailability. What it does not do is reason about whether the new version is *good*. Its only health signal is the pod readiness probe. A pod that starts, passes its readiness check, and then serves 30% of requests with a 500 is, as far as the Deployment controller is concerned, perfectly healthy. The rollout proceeds, the old ReplicaSet is scaled to zero, and the controller reports success while your error rate climbs.

Worse, the Deployment's rollback is manual and coarse. kubectl rollout undo exists, but by the time a human correlates the alert, finds the bad release, and issues the command, the bad version has served full traffic for minutes. Mean-time-to-recovery is bounded by human reaction time — a floor that blocks elite DORA failed-deployment recovery measured in minutes, not hours.

Argo Rollouts replaces the Deployment with a Rollout resource — a drop-in superset that keeps the familiar spec.template pod definition but adds a spec.strategy that is either canary or blueGreen. Crucially, it adds the ability to pause between steps and to run an Analysis against live metrics, making promotion conditional on evidence. The readiness probe still gates whether a pod receives traffic; the analysis gates whether the *release* is allowed to proceed. Those are two different questions, and conflating them is exactly the gap that turns a routine deploy into an incident.

Canary: Shifting Traffic in Measured, Reversible Steps

The canary strategy introduces the new version alongside the stable version and shifts a controlled percentage of traffic to it in steps. Between steps, the rollout can pause — either for a fixed duration, indefinitely until a human promotes, or for the duration of an automated analysis. Each step is a checkpoint. The new version never receives full traffic until it has survived every checkpoint you defined.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-api
  namespace: payments
spec:
  replicas: 10
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: checkout-api
          image: registry.internal.corp/checkout-api:2.14.0
  strategy:
    canary:
      # Stable and canary Services front the two ReplicaSets.
      canaryService: checkout-api-canary
      stableService: checkout-api-stable
      trafficRouting:
        plugins:
          # Mesh-agnostic traffic shaping via the Gateway API plugin.
          argoproj-labs/gatewayAPI:
            httpRoute: checkout-api-route
            namespace: payments
      steps:
        - setWeight: 5
        - pause: { duration: 2m }          # bake time at 5%
        - analysis:                         # evidence gate at 5%
            templates:
              - templateName: success-rate-latency
            args:
              - name: service
                value: checkout-api-canary
        - setWeight: 25
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate-latency
            args:
              - name: service
                value: checkout-api-canary
        - setWeight: 50
        - pause: {}                         # indefinite — manual promotion gate
        - setWeight: 100

Read the steps top to bottom: 5% for a two-minute bake, then analysis; if it passes, 25% for five minutes, another gate; 50%, then an indefinite pause requiring kubectl argo rollouts promote before 100%. That unbounded pause: {} is the deliberate manual gate — change-advisory, business sign-off, or on-call review inside an otherwise automated pipeline. Automated metric gates catch technical regressions; the human owns the business decision — what makes this defensible in a regulated environment.

Click to zoom

There are two distinct mechanisms hiding behind the word 'canary,' and conflating them causes confusion. Without a trafficRouting block, Argo Rollouts approximates the traffic weight by controlling the *replica counts* of the canary and stable ReplicaSets behind a single Service — 5% weight means roughly 1 canary pod for every 19 stable pods, with traffic split by chance via kube-proxy. That is crude and granular only to the nearest pod. With a trafficRouting provider — a service mesh, an ingress controller, or the Gateway API plugin shown above — the controller sets an *actual* weight on the router, decoupling traffic percentage from pod count. For anything where 5% must mean 5%, you want real traffic routing, covered next.

Traffic Shaping: Service Mesh, Ingress, and the Gateway API

Argo Rollouts does not implement traffic splitting itself. It delegates to whatever data plane you already run, manipulating that data plane's native resources to shift weight. This is a deliberate and sovereign design choice: the controller is an orchestrator of traffic policy, not a proxy in the request path. Your latency and availability never depend on the rollout controller being healthy, because it is never on the data path. That separation matters — a delivery tool that sits inline with production traffic is a delivery tool that can take production down.

Historically the integration was per-provider: a native Istio integration that edits VirtualService weights, an Nginx integration that manipulates a canary Ingress, an AWS ALB integration, and an SMI (Service Mesh Interface) TrafficSplit integration. SMI is effectively end-of-life, and the project's center of gravity in 2026 has moved decisively to the Kubernetes Gateway API plugin, which speaks one standard interface and works across any conformant implementation — Istio, Cilium, Contour, Traefik, kgateway, and others. That path still rides the traffic-router plugin system, which the project documents as alpha-quality; treat it as production-ready only with a vendored binary and a tested upgrade path. This is the optionality argument in concrete form: you write the rollout once against a standard API, and you can change the underlying mesh or gateway without rewriting your delivery configuration.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-rollouts-config
  namespace: argo-rollouts
data:
  trafficRouterPlugins: |-
    - name: argoproj-labs/gatewayAPI
      location: "file:///plugins/rollouts-plugin-trafficrouter-gatewayapi"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: checkout-api-route
  namespace: payments
spec:
  parentRefs:
    - name: payments-gateway
  rules:
    - backendRefs:
        # Argo Rollouts rewrites these weights as the canary advances.
        - name: checkout-api-stable
          port: 8080
          weight: 100
        - name: checkout-api-canary
          port: 8080
          weight: 0

The mesh integration also unlocks the richer analysis: when traffic flows through a mesh, the mesh emits per-version request metrics — success rate, latency percentiles, request volume — keyed by the destination subset. Your AnalysisTemplate can query those mesh metrics directly, which is more reliable than instrumenting the application, because the measurement happens at the infrastructure layer and is consistent across every service. This pairs naturally with the eBPF-based identity and observability that a Cilium-class data plane provides: the same layer that enforces zero-trust policy also produces the golden signals your promotion gate consumes.

AnalysisTemplates: Turning Metrics Into an Automated Verdict

The AnalysisTemplate is where progressive delivery stops being a slow-motion deploy and becomes an evidence-gated one. A template defines one or more metrics, each with a provider query, a successCondition (or failureCondition), and a cadence. When a canary reaches an analysis step, the controller instantiates an AnalysisRun — a CRD that records every measurement, its timestamp, and the pass/fail/inconclusive verdict. The AnalysisRun is the in-cluster artifact that makes promotion auditable — a queryable record of the measurements that justified advancing or aborting. It is not long-term storage: successfulRunHistoryLimit and unsuccessfulRunHistoryLimit default to 5 each, and optional ttlStrategy can delete completed runs sooner, so export AnalysisRuns (or Events) before they are reaped.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-latency
  namespace: payments
spec:
  args:
    - name: service
  metrics:
    - name: success-rate
      interval: 30s
      count: 5                     # five measurements over ~2.5 minutes
      # Abort after two failing measurements (transient blips tolerated).
      failureLimit: 2
      successCondition: "result[0] >= 0.99"
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(
              istio_requests_total{
                destination_service_name="{{args.service}}",
                response_code!~"5.."
              }[1m]
            )) /
            sum(rate(
              istio_requests_total{
                destination_service_name="{{args.service}}"
              }[1m]
            ))
    - name: p95-latency-ms
      interval: 30s
      count: 5
      failureLimit: 2
      # p95 latency under 400ms for the canary subset.
      successCondition: "result[0] <= 400"
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            histogram_quantile(0.95,
              sum(rate(
                istio_request_duration_milliseconds_bucket{
                  destination_service_name="{{args.service}}"
                }[1m]
              )) by (le)
            )

Two design details carry most of the value. First, count and interval make the gate a *sustained-evidence* test: requiring the success condition to hold across five measurements over two-and-a-half minutes prevents a single fortunate sample from greenlighting a flapping release. Second, failureLimit tolerates transient noise — a failureLimit: 2 means the run aborts only after two failing measurements, so one momentary blip from a node hiccup does not roll back an otherwise-good canary. Tuning these two parameters is the difference between a gate that is trusted and one that everyone learns to override.

Argo Rollouts ships metric providers for Prometheus, Datadog, New Relic, CloudWatch, Graphite, Wavefront, an InfluxDB/Kubernetes-job/web provider, and — through the plugin system — arbitrary external sources. The web and job providers are the escape hatch: if your verdict depends on something no metrics backend exposes — a synthetic transaction, a business KPI from a data warehouse, the result of a policy evaluation — you wrap it in an HTTP endpoint or a Kubernetes Job and the analysis consumes the result. The gate is only as good as the signal you point it at; choosing the metric that actually distinguishes a bad release from a good one is the genuinely hard part, and it is service-specific work no tool can do for you.

Automatic Rollback: What Actually Happens on Abort

When an AnalysisRun reaches its failureLimit, the rollout is *aborted*. Abort is a specific, well-defined state, not a vague 'something went wrong.' On abort, Argo Rollouts immediately shifts traffic routing back to the stable ReplicaSet — setting the canary weight to zero — and, with traffic routing configured, scales the canary down after abortScaleDownDelaySeconds (ignored for basic canary). By default with traffic routing the stable version is left at 100% throughout, so it is already warm and at full capacity — no cold-start, no image pull, no new ReplicaSet to spin up; recovery is a weight change at the router and completes in seconds. That instant path is not true if dynamicStableScale: true (see the dynamicStableScale trade-off below): stable has been scaled down and must scale back up before it can absorb full traffic.

That is why progressive delivery collapses failed-deployment recovery: by default, recovery is a traffic reroute to a version that never stopped running at full capacity, not a redeploy. A plain Deployment rollback is a fresh rolling update of the old image — schedule pods, pull images, clear readiness — while the bad version still serves. The default Rollout abort path (stable left at 100%) is bounded by data-plane re-weight latency, typically single-digit seconds; with dynamicStableScale: true, abort still shifts weight back but waits on stable scale-up.

spec:
  progressDeadlineSeconds: 600
  progressDeadlineAbort: true
  strategy:
    canary:
      # Forensic window: applies only with traffic routing (ignored for basic canary).
      # Default is 30s; 600 keeps aborted canary pods ~10 minutes for capture.
      abortScaleDownDelaySeconds: 600
      stableService: checkout-api-stable
      canaryService: checkout-api-canary
      trafficRouting:
        plugins:
          argoproj-labs/gatewayAPI:
            httpRoute: checkout-api-route
            namespace: payments
      steps:
        - setWeight: 10
        - analysis:
            templates:
              - templateName: success-rate-latency
            args:
              - name: service
                value: checkout-api-canary

One operational nuance worth internalising: an aborted rollout does not automatically retry. It sits in the aborted state until you either retry it (kubectl argo rollouts retry rollout checkout-api) or push a new revision. That is the correct default — an automated system that keeps re-attempting a release the metrics just rejected would be an outage amplifier, not a safety mechanism. The human decision after an abort is 'fix forward or investigate,' and the AnalysisRun record tells you exactly which metric tripped, when, and by how much, so that decision starts from evidence rather than archaeology.

Blue-Green: Preview Verification and Instant Cutover

Canary is the right tool when you can tolerate a small fraction of users hitting the new version during evaluation. Some workloads cannot: a schema migration that is not backward-compatible, a release where even 5% exposure to a bad version is unacceptable, or a service where you need to run a full verification suite against the new version *before* any real user touches it. For those, blue-green is the correct strategy.

In the blue-green model, Argo Rollouts stands up the new version (green) at full replica count alongside the running version (blue), but routes zero production traffic to it. Two Services front the workload: the activeService, which carries production traffic and points at blue, and the previewService, which points at green and is reachable only by your verification tooling and internal testers. You run your pre-promotion analysis and smoke tests against the preview Service. When — and only when — they pass, the controller flips the active Service's selector from blue to green in a single atomic operation. Every user moves to the new version at once; there is no intermediate state where both versions serve production traffic.

Click to zoom

spec:
  strategy:
    blueGreen:
      activeService: checkout-api-active     # production traffic
      previewService: checkout-api-preview   # verification traffic only
      autoPromotionEnabled: false            # require explicit promotion
      # Gate the cutover on evidence from the preview environment.
      prePromotionAnalysis:
        templates:
          - templateName: smoke-and-success-rate
        args:
          - name: service
            value: checkout-api-preview
      # Watch the new version after it takes production traffic; a failing
      # post-promotion run auto-rolls-back to blue.
      postPromotionAnalysis:
        templates:
          - templateName: success-rate-latency
        args:
          - name: service
            value: checkout-api-active
      # Keep blue running for 5 minutes after cutover for instant rollback.
      scaleDownDelaySeconds: 300

The honest trade-off of blue-green is cost: you run two full copies of the workload simultaneously. For a service at 10 replicas, the cutover window doubles your footprint to 20. For a memory-heavy service — an inference server, a large in-memory cache — that doubling can be the difference between a routine deploy and a capacity event. Canary with basic replica weighting runs only a few extra canary pods at a time; with traffic routing (the Gateway API path above), the stable ReplicaSet stays at 100% by default, so by the final weight step you also run ~2× replicas — the same doubling as blue-green — unless you set canary.dynamicStableScale: true (which trades away instant abort, because stable must scale back up). The choice between them is partly a risk decision and partly a budget decision: blue-green buys you a clean pre-promotion verification environment and an instant atomic cutover, and you pay for it in transient compute.

The Promotion State Machine: Steps, Gates, Pause, and Abort

Underneath both strategies is a state machine — what you need to reason about a stuck, paused, or unexpected rollout. Transitions are driven by step completion, analysis verdicts, and manual actions. The controller reconciles live state toward the steps' desired state like the rest of Kubernetes, so a rollout is debuggable with the same describe and status instincts you already have.

Click to zoom

The states that matter operationally: a rollout is Progressing while it works through its steps; Paused when it hits a pause step (either a timed pause or an indefinite manual gate); Healthy once all steps complete and the new version is fully promoted and stable; and Degraded when an analysis fails (abort + traffic back to stable) or when progressDeadlineSeconds elapses without progress. A failed AnalysisRun aborts automatically; a missed progress deadline only surfaces ProgressDeadlineExceeded and does not abort unless progressDeadlineAbort: true is set (default is false; the abort-controls snippet above sets progressDeadlineAbort: true so a missed deadline aborts). Reading kubectl argo rollouts get rollout <name> shows you the current step, the live traffic weight, the AnalysisRun status, and which state the rollout is in — the single command that answers 'what is this release doing right now.'

Step types compose freely, which is what makes the model expressive enough for real release policies without bespoke code. The table below is the working vocabulary.

# Advance past the current pause step to the next step.
kubectl argo rollouts promote checkout-api -n payments

# Break glass: skip ALL remaining steps and analyses, go straight to 100%.
# Use only in a declared incident; it bypasses every evidence gate.
kubectl argo rollouts promote checkout-api --full -n payments

# Immediately abort: shift all traffic back to stable.
kubectl argo rollouts abort checkout-api -n payments

# Restart an aborted rollout from the first step.
kubectl argo rollouts retry rollout checkout-api -n payments

# Watch the live state machine: step, weight, analysis, health.
kubectl argo rollouts get rollout checkout-api -n payments --watch

The promote --full break-glass path deserves a governance note. It exists for the real case where you must ship immediately — a security hotfix, a revenue-stopping bug — and cannot wait for the full analysis cadence. But it bypasses every evidence gate, so it should be a permissioned, audited action: scope it via RBAC so only senior on-call can invoke it, and ensure the invocation lands in your audit log. An evidence-gated pipeline with an unaudited universal override is not actually evidence-gated. The break-glass path being *visible and recorded* is what keeps it honest.

GitOps, Audit Evidence, and the Promotion Record

Argo Rollouts resources are ordinary Kubernetes manifests, which means they belong in Git and flow through the same GitOps delivery pipeline as the rest of your platform. The Rollout, its AnalysisTemplates, the HTTPRoute, and the Services are declarative artifacts under version control. A release is a commit that bumps the image tag; the promotion policy — the steps, the gates, the thresholds — is itself reviewed, versioned, and auditable. This is the open-source-as-method discipline applied to delivery: the rules that govern how software reaches production are themselves code, in the open, changeable only through review.

There is a well-known interaction to handle: when Argo CD manages a Rollout via GitOps and Argo Rollouts is mutating the live object's traffic weights and ReplicaSet counts mid-rollout, Argo CD can perceive that live mutation as drift and try to revert it. The resolution is the Argo Rollouts health check extension for Argo CD — which teaches Argo CD to understand Rollout health and not fight the controller — plus careful use of ignoreDifferences on the fields the rollout controller owns. Get this wrong and your GitOps engine and your rollout controller will oscillate; get it right and a Rollout is just another resource Argo CD keeps honest.

The audit value compounds only if you export it. Every AnalysisRun is an in-cluster CRD recording metric values, thresholds, timestamps, and the verdict that justified a promotion or abort. The controller keeps only the last successfulRunHistoryLimit / unsuccessfulRunHistoryLimit runs per Rollout (both default to 5); optional ttlStrategy can delete completed runs sooner. Ship the exported AnalysisRuns — with the Rollout's status transitions and Kubernetes Events — to a compliance-grade audit store, and you answer by construction what every auditor and post-incident review asks: *on what evidence was this version promoted?* Not 'someone clicked approve,' but the exact metrics, thresholds, and moment the system decided the release was safe. That is audit-grade delivery — the promotion decision is a retained export, not a short history ring left in the cluster.

Shipping Safely at Scale Is the Long Game

The instinct that progressive delivery slows teams down is backwards. Teams ship slowly when shipping is dangerous — unbounded full-traffic regression risk leads to batching, low-traffic windows, and heavyweight review. Make each release small, measured, and automatically reversible, and that fear evaporates: you ship more often *because* each ship is safer. Elite DORA performance follows — high deployment frequency and low change-failure rate from the same safety mechanism.

Argo Rollouts governs deployment time — which build is running. The complementary surface is runtime: self-hosted feature flags behind OpenFeature change what an already-promoted build does, per request, with no rollout at all. Different plane, different failure mode, and worth owning for the same reasons.

The decision framework is straightforward. Use canary for the common case — stateless services, backward-compatible changes, anywhere a small evaluated traffic fraction is acceptable. Use blue-green when even minimal exposure is unacceptable, when you need a clean pre-promotion verification environment, or when atomic cutover matters more than doubled footprint. In both cases the analysis gate makes promotion evidence-gated and automatic rollback bounds the cost of being wrong. Preserve optionality at the traffic-routing layer — write against the Gateway API and you are not married to one mesh.

None of this is free. You need a metrics stack good enough to define 'healthy' precisely, a data plane that can shift traffic by weight, the discipline to choose gate metrics that actually distinguish good releases from bad, and the operational maturity to treat the break-glass override as the audited exception it is. That is real engineering investment — and it is exactly the kind of investment that pays back over the decade a serious platform operates, not the quarter. If you are designing a delivery platform and want the promotion policy, the analysis gates, and the audit trail built to survive a compliance review, that is the kind of engagement we do.