Progressive Canary Rollouts with Argo Rollouts

You want a production-ready canary that walks traffic up in weighted steps and aborts itself on a metric regression — this is the complete Argo Rollouts and Istio configuration to do it.

When to use this pattern

  • You run on Kubernetes with Istio (or another supported traffic provider) and a Prometheus-compatible metrics backend.
  • You want automated, metric-gated promotion rather than manual traffic shifting.
  • You need the rollout to abort and revert on its own when the new version regresses.

Prerequisites

Complete working example

# rollout.yaml — Rollout + services + VirtualService + AnalysisTemplate
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: app }
spec:
  replicas: 10
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      containers:
        - name: app
          image: ghcr.io/acme/app:PLACEHOLDER     # bumped to trigger a rollout
          ports: [{ containerPort: 8080 }]
          readinessProbe: { httpGet: { path: /ready, port: 8080 } }
  strategy:
    canary:
      canaryService: app-canary
      stableService: app-stable
      trafficRouting:
        istio:
          virtualService:
            name: app-vs
            routes: [primary]
      steps:
        - setWeight: 5
        - pause: { duration: 5m }
        - analysis:
            templates: [{ templateName: canary-health }]
            args: [{ name: canary-svc, value: app-canary }]
        - setWeight: 25
        - pause: { duration: 5m }
        - analysis:
            templates: [{ templateName: canary-health }]
            args: [{ name: canary-svc, value: app-canary }]
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100
---
apiVersion: v1
kind: Service
metadata: { name: app-stable }
spec: { selector: { app: web }, ports: [{ port: 80, targetPort: 8080 }] }
---
apiVersion: v1
kind: Service
metadata: { name: app-canary }
spec: { selector: { app: web }, ports: [{ port: 80, targetPort: 8080 }] }
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata: { name: app-vs }
spec:
  hosts: [app.example.com]
  gateways: [app-gateway]
  http:
    - name: primary                 # Argo rewrites these weights per step
      route:
        - destination: { host: app-stable }
          weight: 100
        - destination: { host: app-canary }
          weight: 0
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: canary-health }
spec:
  args: [{ name: canary-svc }]
  metrics:
    - name: error-rate
      interval: 1m
      count: 5
      failureLimit: 1               # one breach aborts the rollout
      successCondition: result < 0.02
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(istio_requests_total{destination_service_name="{{args.canary-svc}}",response_code=~"5.."}[1m]))
            / sum(rate(istio_requests_total{destination_service_name="{{args.canary-svc}}"}[1m]))
Three Components, Three Responsibilities The rollout controller owns the step schedule and the promote-or-abort decision. The traffic router owns how requests are split. The metrics provider owns whether the current step looks healthy. Each is replaceable, and confusing their responsibilities is the usual source of a stuck rollout. rollout controller owns the schedule and the decision traffic router applies the weight it is told metrics provider answers healthy or not A rollout stuck at step one is almost always a metrics provider returning no data — not a controller or router fault.

Step-by-step walkthrough

The Rollout object replaces a standard Deployment. Argo manages two ReplicaSets — stable and canary — and rewrites the Istio VirtualService weights as it advances through steps.

The traffic ladder (setWeight / pause / analysis) is the canary itself: 5% for 5 minutes with analysis, then 25%, then 50% for a longer window, then 100%. Each analysis block runs the canary-health template; a failure aborts the whole rollout.

Stable and canary services give Istio two destinations to weight between. The VirtualService starts at 100/0 and Argo adjusts it — you never edit those weights by hand.

The AnalysisTemplate queries Istio’s istio_requests_total for the canary’s 5xx ratio, sampled five times at one-minute intervals. failureLimit: 1 means a single breaching sample aborts. The full query design, including latency and baseline comparison, is covered in automating canary analysis with Prometheus metrics.

Rollout States and What Each Requires From You Progressing means the schedule is advancing and nothing is needed. Paused means a manual step is waiting for a human decision. Degraded means analysis failed and the rollout has aborted. A rollout that stays paused indefinitely is usually waiting for an approval nobody knows exists. Progressing advancing on schedule — do nothing Paused waiting for a human — often unnoticed Degraded analysis failed, traffic already reverted Alert on Paused as well as Degraded: a rollout waiting quietly for approval blocks every later release behind it.

Verification

# Trigger a rollout by bumping the image
kubectl argo rollouts set image app app=ghcr.io/acme/app:$GIT_SHA

# Watch the weighted steps and analysis status live
kubectl argo rollouts get rollout app --watch

Expected output during a healthy rollout shows the weight climbing (5 → 25 → 50 → 100) with each AnalysisRun marked ✔ Successful, ending in Status: ✔ Healthy. A regression shows ✖ Failed analysis and Status: ✖ Degraded with canary weight back at 0.

To drive it manually:

kubectl argo rollouts promote app     # advance past a pause
kubectl argo rollouts abort app       # halt and revert to stable
Pushing During a Rollout Restarts It A new revision pushed while a rollout is at step three does not continue from step three. The controller starts a fresh rollout from step one, so the analysis windows already completed are discarded. Frequent pushes can therefore prevent a rollout from ever reaching full traffic. step 1 step 2 step 3 new revision pushed step 1 again step 2 The three completed analysis windows on the left are discarded, not credited. On a busy repository this is why rollouts appear to never finish — the schedule keeps being reset.

Common pitfalls

  • Analysis query returns no data. If metrics are not labelled by service/version, istio_requests_total{destination_service_name="app-canary"} is empty and the run fails with “no data points.” Test the query in the Prometheus UI first.
  • First weight too small for the traffic. At 5% of a low-traffic service, five one-minute samples may cover too few requests to detect a regression. Raise the first setWeight or lengthen interval/count so each analysis sees a meaningful request count.
  • No session affinity. Without consistent hashing at the Istio DestinationRule, users flap between versions. Add affinity as shown in the canary release guide.

Understanding what the controller owns

Argo Rollouts is easiest to operate once the division of responsibility is clear, because most confusing behaviour comes from attributing a problem to the wrong component.

The controller owns the step schedule and the promote-or-abort decision. It does not move traffic itself and it does not evaluate metrics itself; it decides when those things should happen and reacts to their results. A rollout stuck between steps is almost always the controller waiting for something else.

The traffic router — an ingress controller, a service mesh, or the platform’s own weighting — applies whatever weight the controller sets. If traffic is not actually split in the proportions the rollout reports, the fault is here, and it is usually a mismatch between the router the rollout is configured for and the one actually installed.

The metrics provider answers a single question: does this analysis run pass. It returns success, failure, or inconclusive, and the controller acts on that. A rollout that never advances past its first analysis step is nearly always a provider returning no data — a wrong query, a missing label, or a service account without permission to query.

The failure modes worth recognising on sight

Three symptoms account for most operational time with this tool. A rollout stuck at step one with no visible error is an analysis run returning no data; check the provider’s query against the metrics backend directly before touching anything else. A rollout that restarts from the beginning repeatedly is receiving new revisions mid-rollout, which discards completed analysis windows — on a busy repository this can prevent a rollout ever finishing, and the fix is release batching rather than tuning. A rollout paused indefinitely is waiting on a manual step nobody knows exists, which blocks every subsequent release behind it.

That last one deserves an alert of its own. Degraded rollouts are noticed because something is broken; paused rollouts are silent, and a queue of undeployed changes builds up behind them for days before anyone investigates.

Analysis templates as shared assets

The most useful structural habit is treating analysis templates as reusable, reviewed assets rather than as per-service configuration. A single template defining the error-rate and latency comparisons — parameterised by service name — means the query correctness argument is made once and inherited everywhere, and improving it improves every rollout at the same time.

The alternative, where each service copies and adapts a template, produces a fleet of subtly different queries whose differences nobody remembers. When one of them turns out to be wrong, finding the others with the same flaw becomes an archaeology exercise across dozens of manifests.

← Back to Canary Releases & Progressive Rollouts