Canary Releases & Progressive Rollouts

The problem a canary solves is exposure: with an all-at-once deploy, a latent bug reaches 100% of users before any metric can warn you. A canary release routes a small, growing slice of production traffic to the new version and analyses real metrics at each step, so a regression is caught while it affects 5% of users, not all of them. This page covers the traffic-laddering mechanics, the automated analysis that gates each step, and the abort conditions that make the rollout reverse itself without a human in the loop.

Prerequisites

How Progressive Traffic Shifting Works Under the Hood

A canary controller manages two ReplicaSets — stable and canary — behind a traffic splitter. Instead of a single cutover, the controller walks a ladder of weights: 5%, then 25%, then 50%, then 100%. Between steps it pauses, and during each pause it runs an analysis that queries your metrics backend and compares the canary’s behaviour to the stable baseline.

Three mechanisms make this safe:

  1. Weighted routing. The splitter directs the configured percentage of requests to the canary ReplicaSet. With a mesh this is precise to the request; with replica-count ratios it is approximate (3 canary pods out of 20 ≈ 15%).
  2. Automated analysis. At each pause, an AnalysisRun executes metric queries. If the canary’s error rate or latency exceeds the failure condition, the run fails and the rollout aborts — traffic shifts back to stable automatically. The full query mechanics are covered in automating canary analysis with Prometheus metrics.
  3. Session affinity. Routing by a stable hash of user or session ID keeps a given user on one version for the analysis window, so their experience is consistent and per-user metrics are attributable.

The result is a rollout that is a guarded step function: it can only move forward when the evidence says the new version is at least as healthy as the old one.

Canary Traffic Ladder with Analysis Gates A step chart where canary traffic weight climbs from 5 to 25 to 50 to 100 percent. Between each step sits an analysis gate; passing advances to the next weight, failing aborts back to zero percent (the stable version). 100% 50% 25% 5% rollout time → analysis analysis analysis abort → 0% (stable)

Step-by-Step Implementation

Step 1 — Declare the rollout and traffic ladder

Replace the standard Deployment with an Argo Rollouts Rollout that describes the canary steps.

# k8s/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: app }
spec:
  replicas: 10
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      containers:
        - name: app
          image: ghcr.io/acme/app:PLACEHOLDER
          readinessProbe: { httpGet: { path: /ready, port: 8080 } }
  strategy:
    canary:
      canaryService: app-canary     # mesh routes weighted traffic here
      stableService: app-stable
      trafficRouting:
        istio:
          virtualService: { name: app-vs, routes: [primary] }
      steps:
        - setWeight: 5
        - pause: { duration: 5m }
        - analysis: { templates: [{ templateName: error-rate-and-latency }] }
        - setWeight: 25
        - pause: { duration: 5m }
        - analysis: { templates: [{ templateName: error-rate-and-latency }] }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100

Verification: kubectl argo rollouts get rollout app shows the current step, weight, and analysis status in a live view.

Step 2 — Bind the analysis template

The AnalysisTemplate is what turns pauses into gates. This is the abridged shape; the full Prometheus queries live in the dedicated guide.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: error-rate-and-latency }
spec:
  metrics:
    - name: error-rate
      interval: 1m
      count: 5                     # five measurements across the window
      failureLimit: 1             # abort after one breaching measurement
      successCondition: result < 0.02   # canary 5xx rate under 2%
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{app="web",version="canary",code=~"5.."}[1m]))
            / sum(rate(http_requests_total{app="web",version="canary"}[1m]))

Verification: After the first pause, kubectl argo rollouts get rollout app shows the AnalysisRun as Successful; a Failed run auto-aborts.

Step 3 — Enforce session affinity

Configure the mesh or ingress to hash on a stable key so users do not flap between versions.

# Istio DestinationRule — consistent hashing keeps a user on one version
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata: { name: app-affinity }
spec:
  host: app
  trafficPolicy:
    loadBalancer:
      consistentHash:
        httpCookie: { name: canary-affinity, ttl: 3600s }

Verification: Send repeated requests with the same cookie and confirm the x-version response header stays constant across the analysis window.

Step 4 — Watch the promotion or abort

# Follow the rollout to completion; non-zero exit means it aborted
kubectl argo rollouts get rollout app --watch

Verification: A healthy rollout ends in Phase: Healthy at 100% weight. An aborted rollout returns to the stable ReplicaSet at 0% canary weight, and kubectl argo rollouts status app reports Degraded.

Configuration Reference

Option Type Default Effect
setWeight integer (0–100) — Percentage of traffic sent to the canary at that step
pause.duration duration indefinite Analysis window length; omit duration to require manual promotion
analysis.count integer 1 Number of metric measurements per run; more measurements resist noise
failureLimit integer 0 Breaching measurements tolerated before the run fails and aborts
successCondition expression — The boolean the metric result must satisfy to pass
consistentHash object none Enables session affinity so users stay on one version

Integration with Upstream and Downstream Topics

Canary sits alongside the other deployment strategies:

  • Sibling — blue-green. Blue-green deployments flip 100% at once; canary trades that speed for a smaller blast radius. Frontends often use blue-green while backend services canary.
  • Downstream — analysis. The gate quality depends entirely on the Prometheus analysis template — bad queries mean a false sense of safety.
  • Downstream — rollback. An aborted analysis is an automated rollback; wire the abort event into your incident runbook so on-call is notified even though recovery is automatic.
  • Complementary — feature flags. Feature-flag-driven release management runs a second, per-feature ramp inside the canary binary.

Performance and Cost Impact

Metric Value Notes
Extra capacity during rollout ~10–15% Only the canary ReplicaSet is additional, not a full second environment
Time to full rollout 20–40 min Sum of pause windows; longer windows mean more signal, slower ship
Blast radius before first gate ≤ 5% Bounded by the first setWeight
Detection latency 1–5 min Set by interval × count; shorter risks noise, longer risks exposure
Rollback latency on abort 1–2 min Traffic shift back to stable ReplicaSet
Minimum viable traffic ~1000 req/window Below this, error-rate math is too noisy to gate on

The tension is window length versus exposure time. Longer windows detect subtle regressions but keep the rollout — and its risk — open longer. Size windows to a request count, not a wall-clock time, so low-traffic hours do not weaken the gate.

Troubleshooting

Error: analysis run failed: no data points

Cause: The Prometheus query returns empty because metrics are not labelled by version, so the canary series does not exist.

Fix: Ensure the app emits a version label (or the mesh injects one) and that both canary and stable values appear in the metrics. Test the query directly in the Prometheus UI before binding it to the rollout.

Symptom: Users report inconsistent behaviour during rollout

Cause: No session affinity, so round-robin routing bounces a user between the canary and stable versions on successive requests.

Fix: Add consistent hashing (Step 3). Confirm with repeated requests carrying the same affinity cookie that the served version stays constant.

Symptom: Canary passes analysis but production degrades after promotion

Cause: The analysis window was too short or the first weight too small to surface a load-dependent regression that only appears at higher traffic.

Fix: Add a longer pause at 50% before the final jump to 100%, and include a saturation metric (CPU, memory, queue depth) in the analysis so load-sensitive regressions are caught before full promotion.

Symptom: Rollout stalls indefinitely at a pause step

Cause: A pause step with no duration requires manual promotion, so the rollout waits for a human.

Fix: Either add a duration to make the step time-boxed, or promote explicitly with kubectl argo rollouts promote app. Reserve indefinite pauses for the final production gate if you want a human sign-off there.

Frequently Asked Questions

How much traffic should the first canary step receive?

Start at the smallest slice that still produces statistically meaningful metrics within your analysis window. For high-traffic services 5% is standard. For a low-traffic endpoint, 5% may be only a handful of requests per minute — too few to detect a 1–2% error-rate regression — so use a larger first step or lengthen the window to accumulate a minimum request count. Always size the analysis by request volume, not clock time.

Should canary metrics be compared to absolute thresholds or the baseline?

Compare primarily to the live stable baseline, so a normal error rate does not falsely fail the gate. But add absolute-floor guardrails as a backstop — for example, abort if the canary 5xx rate exceeds 2% regardless of baseline. The relative comparison catches regressions the new version introduces; the absolute floor catches shared outages that degrade both versions equally and would otherwise slip through a purely relative check.

What causes flapping between versions during a canary?

Round-robin routing without session affinity: each request is independently routed, so a user hits the canary, then stable, then canary again, seeing inconsistent behaviour and generating unattributable metrics. Route by a consistent hash of session or user ID (Step 3) so a given user stays on one version for the whole analysis window.

Can I run canary releases without a service mesh?

Yes. An ingress controller with weighted backends (NGINX canary annotations), CDN percentage routing at the edge, or simply the ratio of canary-to-stable replica counts all approximate traffic weighting. A service mesh gives request-precise control and the cleanest per-version metrics, but you can start with ingress weights and adopt a mesh later as precision needs grow.

← Back to Deployment Strategies & Progressive Delivery