Progressive Delivery with Argo Rollouts: Evidence-Gated Canary and Blue-Green on Kubernetes

A kubectl rollout restart is a leap of faith. The Deployment controller swaps pods according to its rolling-update strategy, the new ReplicaSet reaches full size, the old one drains, and you find out whether the release was good by watching your dashboards and your support queue. That model works because most releases are fine. The releases that are not fine are the expensive ones — and a plain Deployment gives you no structural way to limit their blast radius or to make the promotion decision on evidence rather than on hope.

Progressive delivery is the discipline of replacing that leap of faith with a sequence of small, measured, reversible steps. You expose a new version to a fraction of traffic, you measure it against the metrics that define 'healthy' for your service, and you only widen the exposure when the evidence says the version is good. If the evidence says otherwise, the system reverts — automatically, in seconds, without paging anyone. Argo Rollouts is the Kubernetes controller that makes this a first-class deployment primitive rather than a bespoke pipeline of shell scripts and prayer.

This article is a principal-engineer-level walkthrough of how Argo Rollouts actually works in production: the canary and blue-green strategies and when each is correct, the AnalysisTemplate/AnalysisRun machinery that turns Prometheus queries into automated promotion gates, the traffic-shaping integration with service meshes and the Gateway API, and the honest trade-offs — including one named cost that the marketing material tends to skip. The framing throughout is audit-grade: promotion as an evidence-gated event, every decision recorded, every release reversible. That is the posture that lets you ship faster at scale precisely because each individual ship is safer.

Why the Deployment Object Is Not Enough

The Kubernetes Deployment is a fine abstraction for what it does: it manages ReplicaSets and performs rolling updates with configurable surge and unavailability. What it does not do is reason about whether the new version is *good*. Its only health signal is the pod readiness probe. A pod that starts, passes its readiness check, and then serves 30% of requests with a 500 is, as far as the Deployment controller is concerned, perfectly healthy. The rollout proceeds, the old ReplicaSet is scaled to zero, and the controller reports success while your error rate climbs.

Worse, the Deployment's rollback story is manual and coarse. kubectl rollout undo exists, but someone has to decide to run it, and by the time a human has correlated the alert, found the bad release, and issued the command, the bad version has been serving full production traffic for minutes. The mean-time-to-recovery is bounded below by human reaction time. For a team trying to hit elite DORA numbers — where failed-deployment recovery time is measured in minutes, not hours — that floor is the problem.

Argo Rollouts replaces the Deployment with a Rollout resource — a drop-in superset that keeps the familiar spec.template pod definition but adds a spec.strategy that is either canary or blueGreen. Crucially, it adds the ability to pause between steps and to run an Analysis against live metrics, making promotion conditional on evidence. The readiness probe still gates whether a pod receives traffic; the analysis gates whether the *release* is allowed to proceed. Those are two different questions, and conflating them is exactly the gap that turns a routine deploy into an incident.

Canary: Shifting Traffic in Measured, Reversible Steps

The canary strategy introduces the new version alongside the stable version and shifts a controlled percentage of traffic to it in steps. Between steps, the rollout can pause — either for a fixed duration, indefinitely until a human promotes, or for the duration of an automated analysis. Each step is a checkpoint. The new version never receives full traffic until it has survived every checkpoint you defined.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-api
  namespace: payments
spec:
  replicas: 10
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: checkout-api
          image: registry.internal.corp/checkout-api:2.14.0
  strategy:
    canary:
      # Stable and canary Services front the two ReplicaSets.
      canaryService: checkout-api-canary
      stableService: checkout-api-stable
      trafficRouting:
        plugins:
          # Mesh-agnostic traffic shaping via the Gateway API plugin.
          argoproj-labs/gatewayAPI:
            httpRoute: checkout-api-route
            namespace: payments
      steps:
        - setWeight: 5
        - pause: { duration: 2m }          # bake time at 5%
        - analysis:                         # evidence gate at 5%
            templates:
              - templateName: success-rate-latency
            args:
              - name: service
                value: checkout-api-canary
        - setWeight: 25
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate-latency
            args:
              - name: service
                value: checkout-api-canary
        - setWeight: 50
        - pause: {}                         # indefinite — manual promotion gate
        - setWeight: 100

Read the steps top to bottom: 5% of traffic for a two-minute bake, then an analysis gate; if it passes, 25% for five minutes, another gate; 50%, then an indefinite pause that requires a human kubectl argo rollouts promote before going to 100%. That last unbounded pause: {} is the deliberate manual gate — the point where a change-advisory approval, a business sign-off, or an on-call review can sit in the middle of an otherwise automated pipeline. The combination of automated metric gates and a human gate is what makes this defensible in a regulated environment: the machine catches the technical regressions, the human owns the business decision.

Click to zoom

There are two distinct mechanisms hiding behind the word 'canary,' and conflating them causes confusion. Without a trafficRouting block, Argo Rollouts approximates the traffic weight by controlling the *replica counts* of the canary and stable ReplicaSets behind a single Service — 5% weight means roughly 1 canary pod for every 19 stable pods, with traffic split by chance via kube-proxy. That is crude and granular only to the nearest pod. With a trafficRouting provider — a service mesh, an ingress controller, or the Gateway API plugin shown above — the controller sets an *actual* weight on the router, decoupling traffic percentage from pod count. For anything where 5% must mean 5%, you want real traffic routing, covered next.

Traffic Shaping: Service Mesh, Ingress, and the Gateway API

Argo Rollouts does not implement traffic splitting itself. It delegates to whatever data plane you already run, manipulating that data plane's native resources to shift weight. This is a deliberate and sovereign design choice: the controller is an orchestrator of traffic policy, not a proxy in the request path. Your latency and availability never depend on the rollout controller being healthy, because it is never on the data path. That separation matters — a delivery tool that sits inline with production traffic is a delivery tool that can take production down.

Historically the integration was per-provider: a native Istio integration that edits VirtualService weights, an Nginx integration that manipulates a canary Ingress, an AWS ALB integration, and an SMI (Service Mesh Interface) TrafficSplit integration. SMI is effectively end-of-life, and the project's center of gravity in 2026 has moved decisively to the Kubernetes Gateway API plugin, which speaks one standard interface and works across any conformant implementation — Istio, Cilium, Contour, Traefik, kgateway, and others. This is the optionality argument in concrete form: you write the rollout once against a standard API, and you can change the underlying mesh or gateway without rewriting your delivery configuration.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-rollouts-config
  namespace: argo-rollouts
data:
  trafficRouterPlugins: |-
    - name: argoproj-labs/gatewayAPI
      location: "file:///plugins/rollouts-plugin-trafficrouter-gatewayapi"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: checkout-api-route
  namespace: payments
spec:
  parentRefs:
    - name: payments-gateway
  rules:
    - backendRefs:
        # Argo Rollouts rewrites these weights as the canary advances.
        - name: checkout-api-stable
          port: 8080
          weight: 100
        - name: checkout-api-canary
          port: 8080
          weight: 0

The mesh integration also unlocks the richer analysis: when traffic flows through a mesh, the mesh emits per-version request metrics — success rate, latency percentiles, request volume — keyed by the destination subset. Your AnalysisTemplate can query those mesh metrics directly, which is more reliable than instrumenting the application, because the measurement happens at the infrastructure layer and is consistent across every service. This pairs naturally with the eBPF-based identity and observability that a Cilium-class data plane provides: the same layer that enforces zero-trust policy also produces the golden signals your promotion gate consumes.

AnalysisTemplates: Turning Metrics Into an Automated Verdict

The AnalysisTemplate is where progressive delivery stops being a slow-motion deploy and becomes an evidence-gated one. A template defines one or more metrics, each with a provider query, a successCondition (or failureCondition), and a cadence. When a canary reaches an analysis step, the controller instantiates an AnalysisRun — a CRD that records every measurement, its timestamp, and the pass/fail/inconclusive verdict. The AnalysisRun is the artifact that makes the promotion auditable: it is a durable, queryable record of exactly what evidence justified advancing or aborting the release.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-latency
  namespace: payments
spec:
  args:
    - name: service
  metrics:
    - name: success-rate
      interval: 30s
      count: 5                     # five measurements over ~2.5 minutes
      # Abort after two failing measurements (transient blips tolerated).
      failureLimit: 2
      successCondition: "result[0] >= 0.99"
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(
              istio_requests_total{
                destination_service_name="{{args.service}}",
                response_code!~"5.."
              }[1m]
            )) /
            sum(rate(
              istio_requests_total{
                destination_service_name="{{args.service}}"
              }[1m]
            ))
    - name: p95-latency-ms
      interval: 30s
      count: 5
      failureLimit: 2
      # p95 latency under 400ms for the canary subset.
      successCondition: "result[0] <= 400"
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            histogram_quantile(0.95,
              sum(rate(
                istio_request_duration_milliseconds_bucket{
                  destination_service_name="{{args.service}}"
                }[1m]
              )) by (le)
            )

Two design details carry most of the value. First, count and interval make the gate a *sustained-evidence* test: requiring the success condition to hold across five measurements over two-and-a-half minutes prevents a single fortunate sample from greenlighting a flapping release. Second, failureLimit tolerates transient noise — a failureLimit: 2 means the run aborts only after two failing measurements, so one momentary blip from a node hiccup does not roll back an otherwise-good canary. Tuning these two parameters is the difference between a gate that is trusted and one that everyone learns to override.

Argo Rollouts ships metric providers for Prometheus, Datadog, New Relic, CloudWatch, Graphite, Wavefront, an InfluxDB/Kubernetes-job/web provider, and — through the plugin system — arbitrary external sources. The web and job providers are the escape hatch: if your verdict depends on something no metrics backend exposes — a synthetic transaction, a business KPI from a data warehouse, the result of a policy evaluation — you wrap it in an HTTP endpoint or a Kubernetes Job and the analysis consumes the result. The gate is only as good as the signal you point it at; choosing the metric that actually distinguishes a bad release from a good one is the genuinely hard part, and it is service-specific work no tool can do for you.

Automatic Rollback: What Actually Happens on Abort

When an AnalysisRun reaches its failureLimit, the rollout is *aborted*. Abort is a specific, well-defined state, not a vague 'something went wrong.' On abort, Argo Rollouts immediately shifts traffic routing back to the stable ReplicaSet — setting the canary weight to zero — and scales the canary down according to the abortScaleDownDelaySeconds. The stable version, which has been running the whole time, is already warm and at full capacity. There is no cold-start, no image pull, no new ReplicaSet to spin up. Recovery is a weight change at the router, which is why it completes in seconds.

This is the structural reason progressive delivery improves your failed-deployment recovery time so dramatically: the 'recovery' is not a redeploy of the previous version, it is a traffic reroute to a version that never stopped running. Compare that to a plain Deployment rollback, which is a fresh rolling update of the old image — pods to schedule, images to pull, readiness gates to clear — all while the bad version is still serving. The Rollout's abort path is bounded by how fast your data plane can re-weight, typically single-digit seconds.

spec:
  # Mark the rollout Degraded if a step makes no progress in 10 minutes.
  progressDeadlineSeconds: 600
  # On a degraded rollout, automatically roll back to the last stable.
  progressDeadlineAbort: true
  strategy:
    canary:
      # Keep aborted-canary pods for 30s so logs/heap dumps can be captured
      # before the ReplicaSet is scaled to zero.
      abortScaleDownDelaySeconds: 30
      stableService: checkout-api-stable
      canaryService: checkout-api-canary
      steps:
        - setWeight: 10
        - analysis:
            templates:
              - templateName: success-rate-latency
            args:
              - name: service
                value: checkout-api-canary

One operational nuance worth internalising: an aborted rollout does not automatically retry. It sits in the aborted state until you either retry it (kubectl argo rollouts retry rollout checkout-api) or push a new revision. That is the correct default — an automated system that keeps re-attempting a release the metrics just rejected would be an outage amplifier, not a safety mechanism. The human decision after an abort is 'fix forward or investigate,' and the AnalysisRun record tells you exactly which metric tripped, when, and by how much, so that decision starts from evidence rather than archaeology.

Blue-Green: Preview Verification and Instant Cutover

Canary is the right tool when you can tolerate a small fraction of users hitting the new version during evaluation. Some workloads cannot: a schema migration that is not backward-compatible, a release where even 5% exposure to a bad version is unacceptable, or a service where you need to run a full verification suite against the new version *before* any real user touches it. For those, blue-green is the correct strategy.

In the blue-green model, Argo Rollouts stands up the new version (green) at full replica count alongside the running version (blue), but routes zero production traffic to it. Two Services front the workload: the activeService, which carries production traffic and points at blue, and the previewService, which points at green and is reachable only by your verification tooling and internal testers. You run your pre-promotion analysis and smoke tests against the preview Service. When — and only when — they pass, the controller flips the active Service's selector from blue to green in a single atomic operation. Every user moves to the new version at once; there is no intermediate state where both versions serve production traffic.

Click to zoom

spec:
  strategy:
    blueGreen:
      activeService: checkout-api-active     # production traffic
      previewService: checkout-api-preview   # verification traffic only
      autoPromotionEnabled: false            # require explicit promotion
      # Gate the cutover on evidence from the preview environment.
      prePromotionAnalysis:
        templates:
          - templateName: smoke-and-success-rate
        args:
          - name: service
            value: checkout-api-preview
      # Watch the new version after it takes production traffic; a failing
      # post-promotion run auto-rolls-back to blue.
      postPromotionAnalysis:
        templates:
          - templateName: success-rate-latency
        args:
          - name: service
            value: checkout-api-active
      # Keep blue running for 5 minutes after cutover for instant rollback.
      scaleDownDelaySeconds: 300

The honest trade-off of blue-green is cost: you run two full copies of the workload simultaneously. For a service at 10 replicas, the cutover window doubles your footprint to 20. For a memory-heavy service — an inference server, a large in-memory cache — that doubling can be the difference between a routine deploy and a capacity event. Canary, by contrast, runs only a few extra canary pods at a time. The choice between them is partly a risk decision and partly a budget decision: blue-green buys you a clean pre-promotion verification environment and an instant atomic cutover, and you pay for it in transient compute.

The Promotion State Machine: Steps, Gates, Pause, and Abort

Underneath both strategies is a state machine, and understanding it is what lets you reason about a rollout that is stuck, paused, or behaving unexpectedly. A Rollout is always in one of a small set of states, and the transitions between them are driven by step completion, analysis verdicts, and manual actions. The controller reconciles the live state toward the desired state defined by the steps, the same way the rest of Kubernetes works — which means a rollout is debuggable with the same describe and status-reading instincts you already have.

Click to zoom

The states that matter operationally: a rollout is Progressing while it works through its steps; Paused when it hits a pause step (either a timed pause or an indefinite manual gate); Healthy once all steps complete and the new version is fully promoted and stable; and Degraded when an analysis fails or the progressDeadlineSeconds elapses without progress. The Degraded transition is the one that triggers automatic rollback. Reading kubectl argo rollouts get rollout <name> shows you the current step, the live traffic weight, the AnalysisRun status, and which state the rollout is in — the single command that answers 'what is this release doing right now.'

Step types compose freely, which is what makes the model expressive enough for real release policies without bespoke code. The table below is the working vocabulary.

# Advance past the current pause step to the next step.
kubectl argo rollouts promote checkout-api -n payments

# Break glass: skip ALL remaining steps and analyses, go straight to 100%.
# Use only in a declared incident; it bypasses every evidence gate.
kubectl argo rollouts promote checkout-api --full -n payments

# Immediately abort: shift all traffic back to stable.
kubectl argo rollouts abort checkout-api -n payments

# Restart an aborted rollout from the first step.
kubectl argo rollouts retry rollout checkout-api -n payments

# Watch the live state machine: step, weight, analysis, health.
kubectl argo rollouts get rollout checkout-api -n payments --watch

The promote --full break-glass path deserves a governance note. It exists for the real case where you must ship immediately — a security hotfix, a revenue-stopping bug — and cannot wait for the full analysis cadence. But it bypasses every evidence gate, so it should be a permissioned, audited action: scope it via RBAC so only senior on-call can invoke it, and ensure the invocation lands in your audit log. An evidence-gated pipeline with an unaudited universal override is not actually evidence-gated. The break-glass path being *visible and recorded* is what keeps it honest.

GitOps, Audit Evidence, and the Promotion Record

Argo Rollouts resources are ordinary Kubernetes manifests, which means they belong in Git and flow through the same GitOps delivery pipeline as the rest of your platform. The Rollout, its AnalysisTemplates, the HTTPRoute, and the Services are declarative artifacts under version control. A release is a commit that bumps the image tag; the promotion policy — the steps, the gates, the thresholds — is itself reviewed, versioned, and auditable. This is the open-source-as-method discipline applied to delivery: the rules that govern how software reaches production are themselves code, in the open, changeable only through review.

There is a well-known interaction to handle: when Argo CD manages a Rollout via GitOps and Argo Rollouts is mutating the live object's traffic weights and ReplicaSet counts mid-rollout, Argo CD can perceive that live mutation as drift and try to revert it. The resolution is the Argo Rollouts health check extension for Argo CD — which teaches Argo CD to understand Rollout health and not fight the controller — plus careful use of ignoreDifferences on the fields the rollout controller owns. Get this wrong and your GitOps engine and your rollout controller will oscillate; get it right and a Rollout is just another resource Argo CD keeps honest.

The audit value is the part that compounds. Every AnalysisRun is a durable CRD recording the metric values, the thresholds, the timestamps, and the verdict that justified a promotion or an abort. Ship those — along with the Rollout's own status transitions and the Kubernetes Events it emits — to a compliance-grade audit store, and you have answered, by construction, the question every auditor and every post-incident review asks: *on what evidence was this version promoted to production?* Not 'someone clicked approve,' but a queryable record of the exact metrics, the exact thresholds, and the exact moment the system decided the release was safe. That is what audit-grade rigor means for delivery — the promotion decision is not a memory, it is a record.

Shipping Safely at Scale Is the Long Game

The instinct that progressive delivery slows teams down is backwards. Teams ship slowly when shipping is dangerous — when every release carries the unbounded risk of a full-traffic regression, the rational response is to batch changes, schedule them for low-traffic windows, and require heavyweight manual review. Make each release small, measured, and automatically reversible, and that fear evaporates. You ship more often *because* each ship is safer, not in spite of it. This is the through-line of elite DORA performance: high deployment frequency and low change-failure rate are not in tension; the same safety mechanism produces both.

The decision framework is straightforward. Use canary for the common case — stateless services, backward-compatible changes, anywhere a small fraction of evaluated traffic is acceptable and you want the gentlest possible risk gradient. Use blue-green when even minimal exposure to a bad version is unacceptable, when you need a clean pre-promotion verification environment, or when an atomic cutover matters more than the doubled footprint costs. In both cases, the analysis gate is what converts a deployment into an evidence-gated event, and the automatic rollback is what bounds the cost of being wrong. The traffic-routing layer is where you preserve optionality — write against the Gateway API and you are not married to any one mesh.

None of this is free. You need a metrics stack good enough to define 'healthy' precisely, a data plane that can shift traffic by weight, the discipline to choose gate metrics that actually distinguish good releases from bad, and the operational maturity to treat the break-glass override as the audited exception it is. That is real engineering investment — and it is exactly the kind of investment that pays back over the decade a serious platform operates, not the quarter. If you are designing a delivery platform and want the promotion policy, the analysis gates, and the audit trail built to survive a compliance review, that is the kind of engagement we do.