Automating Canary Analysis with Prometheus Metrics

The gate that makes a canary release safe is only as good as its metric queries — this page shows how to write Prometheus-backed analysis that catches real regressions without false alarms.

When to use this pattern

  • You run canary rollouts (for example with Argo Rollouts) and want promotion decisions driven by data.
  • You have Prometheus scraping request metrics labelled by version.
  • You need to distinguish a real regression from normal baseline noise.

Prerequisites

Complete working example

# analysis-template.yaml — error-rate (baseline + floor) and p95 latency
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: canary-health }
spec:
  args:
    - name: app
      value: web
  metrics:
    # 1. Primary gate: canary error rate relative to the stable baseline.
    - name: error-rate-ratio
      interval: 1m
      count: 5                      # five samples across the pause window
      failureLimit: 1              # one breach aborts
      successCondition: result < 2.0     # canary no worse than 2x baseline
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            (
              sum(rate(http_requests_total{app="{{args.app}}",version="canary",code=~"5.."}[2m]))
              / clamp_min(sum(rate(http_requests_total{app="{{args.app}}",version="canary"}[2m])), 0.001)
            )
            /
            clamp_min(
              sum(rate(http_requests_total{app="{{args.app}}",version="stable",code=~"5.."}[2m]))
              / clamp_min(sum(rate(http_requests_total{app="{{args.app}}",version="stable"}[2m])), 0.001)
            , 0.0001)

    # 2. Backstop: absolute error-rate floor, independent of baseline.
    - name: error-rate-floor
      interval: 1m
      count: 5
      failureLimit: 1
      successCondition: result < 0.05    # abort if canary 5xx exceeds 5% outright
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="{{args.app}}",version="canary",code=~"5.."}[2m]))
            / clamp_min(sum(rate(http_requests_total{app="{{args.app}}",version="canary"}[2m])), 0.001)

    # 3. Performance gate: p95 latency in seconds.
    - name: p95-latency
      interval: 1m
      count: 5
      failureLimit: 2
      successCondition: result < 0.4      # p95 under 400ms
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            histogram_quantile(0.95,
              sum(rate(http_request_duration_seconds_bucket{app="{{args.app}}",version="canary"}[2m])) by (le)
            )
Which Signal Detects Which Kind of Regression Error rate detects broken code paths within a minute. Latency detects performance regressions and blocked calls within about three minutes. Saturation detects memory and connection leaks over ten minutes or more. Traffic detects routing faults, where the canary receives less than its configured share. SIGNAL → WHAT IT CATCHES → HOW FAST error rate broken code paths — visible within a minute latency p95 slow queries and blocking calls — about three minutes saturation memory and connection leaks — ten minutes or more A five-minute canary window can only see the first two rows, which is worth knowing before trusting it with a leak.

Step-by-step walkthrough

Metric 1 — error-rate ratio (primary gate). The query computes the canary’s 5xx ratio and divides it by the stable version’s 5xx ratio. successCondition: result < 2.0 passes as long as the canary is no more than twice as error-prone as the live baseline. Comparing to the baseline — not to zero — means a normal 0.5% error rate does not fail the gate. clamp_min guards against divide-by-zero when a series is momentarily empty.

Metric 2 — absolute floor (backstop). A pure ratio has a blind spot: if a shared dependency outage pushes both versions to 10% errors, the ratio stays near 1.0 and the canary passes while production burns. The floor query aborts whenever the canary’s absolute 5xx rate exceeds 5%, regardless of baseline.

Metric 3 — p95 latency. Error rate misses slow-but-successful regressions. histogram_quantile(0.95, …) reads p95 from the latency histogram; failureLimit: 2 tolerates one noisy spike but aborts on a sustained one.

Sample sizing. interval: 1m × count: 5 gives a five-minute analysis with five decision points. The [2m] range in each query smooths per-scrape noise. On a low-traffic service, widen interval and the range so each sample still covers hundreds of requests — sizing by request volume, not clock time, is what keeps the gate trustworthy.

Without a Version Label, No Canary Query Is Possible Metrics labelled only by service produce one time series covering both versions, so a canary regression is diluted by stable traffic and may be invisible at five per cent weight. Adding a version label produces separate series and makes the comparison possible at all. labelled by service only one series — both versions averaged together a 5% canary moves the average by almost nothing labelled by service + version two series — directly comparable the canary's own error rate is visible in isolation Add the version label before building any analysis: without it the queries look right and measure the wrong thing.

Verification

Test every query in the Prometheus UI before binding it, then dry-run the analysis:

# Paste each query into http://prometheus:9090/graph and confirm it returns a value
# Then run the template standalone against a live canary:
kubectl argo rollouts get analysisrun --rollout app

# Force a regression check: deploy a version that returns 500s to the canary and
# confirm the AnalysisRun transitions to Failed within one interval.

Expected: a healthy canary shows all three metrics ✔; an injected 500-heavy build flips error-rate-floor to ✖ Failed and aborts the rollout.

Skip the First Ninety Seconds of Every Window Immediately after a canary starts receiving traffic, its latency is inflated by cold caches, JIT warm-up and connection pool establishment. Including that period in the analysis makes almost every canary look like a regression. Excluding the first ninety seconds removes the effect. warm-up 0–90s exclude representative window — this is what the analysis reads CANARY ANALYSIS WINDOW Cold caches, JIT warm-up and connection pool setup all inflate the first minute or so of any new version's metrics. Without the exclusion, healthy releases abort routinely and the team stops trusting the gate.

Common pitfalls

  • Ratio without a floor. As above, a baseline-relative gate alone misses shared outages. Always pair it with an absolute floor.
  • Querying an instant vector where a range is needed. rate() requires a range selector ([2m]); using an instant selector returns no data and the run fails spuriously.
  • failureLimit: 0 on a noisy metric. Zero tolerance turns a single scrape blip into an abort. Allow one or two breaches on latency, and rely on count for confidence rather than hair-trigger limits.

Getting the query right before the automation

Automated analysis is only as good as the query underneath it, and three properties separate a query that measures the release from one that measures the weather.

Compare concurrently, not historically. The canary’s error rate means nothing on its own; what matters is its ratio to the stable version’s over the identical window. A concurrent comparison cancels out traffic patterns, upstream incidents, and time-of-day effects, because both versions experience them equally. A historical baseline confounds all of them with the release, which is why historically-baselined canaries abort on Monday mornings.

Exclude the warm-up. A version that has just started receiving traffic has cold caches, an un-warmed JIT, and empty connection pools. Its first sixty to ninety seconds are unrepresentative by construction. Including them makes almost every canary look like a latency regression, and the usual response — widening the threshold until the false positives stop — also disables detection of real ones.

Require a minimum sample. A ratio computed over forty requests is noise. The analysis should refuse to conclude anything until the canary has served enough requests for the comparison to be meaningful, and should treat “not enough data yet” as a distinct state from “healthy”. Conflating the two means a canary receiving no traffic at all — because the routing is misconfigured — reports as passing.

Choosing thresholds you can defend

Thresholds set by intuition tend to be either so tight that they fire constantly or so loose that they never fire. The defensible approach is to compute them from history: take the distribution of the metric’s version-to-version ratio across the last few dozen healthy releases, and set the threshold outside its normal range. That produces a number with a stated false-positive rate rather than a round figure someone liked.

It also produces an uncomfortable but useful finding on many systems: the natural variation between two identical versions is wider than people expect, and a threshold tight enough to catch a one per cent regression would have fired on a quarter of past healthy releases. Knowing that up front is better than discovering it after the third unnecessary rollback.

What to do when analysis is inconclusive

The third state — enough time has passed but the data is too sparse to decide — needs an explicit policy. Automatically promoting is unsafe; automatically aborting punishes low-traffic services for being low-traffic. The pragmatic answer is to hold the current weight, extend the window once, and escalate to a human if it is still inconclusive. That keeps the mechanism honest without either rubber-stamping or blocking releases on services that will never generate canary-scale traffic.

← Back to Canary Releases & Progressive Rollouts