Automating Canary Analysis with Prometheus Metrics

The gate that makes a canary release safe is only as good as its metric queries — this page shows how to write Prometheus-backed analysis that catches real regressions without false alarms.

When to use this pattern

  • You run canary rollouts (for example with Argo Rollouts) and want promotion decisions driven by data.
  • You have Prometheus scraping request metrics labelled by version.
  • You need to distinguish a real regression from normal baseline noise.

Prerequisites

Complete working example

# analysis-template.yaml — error-rate (baseline + floor) and p95 latency
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: canary-health }
spec:
  args:
    - name: app
      value: web
  metrics:
    # 1. Primary gate: canary error rate relative to the stable baseline.
    - name: error-rate-ratio
      interval: 1m
      count: 5                      # five samples across the pause window
      failureLimit: 1              # one breach aborts
      successCondition: result < 2.0     # canary no worse than 2x baseline
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            (
              sum(rate(http_requests_total{app="{{args.app}}",version="canary",code=~"5.."}[2m]))
              / clamp_min(sum(rate(http_requests_total{app="{{args.app}}",version="canary"}[2m])), 0.001)
            )
            /
            clamp_min(
              sum(rate(http_requests_total{app="{{args.app}}",version="stable",code=~"5.."}[2m]))
              / clamp_min(sum(rate(http_requests_total{app="{{args.app}}",version="stable"}[2m])), 0.001)
            , 0.0001)

    # 2. Backstop: absolute error-rate floor, independent of baseline.
    - name: error-rate-floor
      interval: 1m
      count: 5
      failureLimit: 1
      successCondition: result < 0.05    # abort if canary 5xx exceeds 5% outright
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="{{args.app}}",version="canary",code=~"5.."}[2m]))
            / clamp_min(sum(rate(http_requests_total{app="{{args.app}}",version="canary"}[2m])), 0.001)

    # 3. Performance gate: p95 latency in seconds.
    - name: p95-latency
      interval: 1m
      count: 5
      failureLimit: 2
      successCondition: result < 0.4      # p95 under 400ms
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            histogram_quantile(0.95,
              sum(rate(http_request_duration_seconds_bucket{app="{{args.app}}",version="canary"}[2m])) by (le)
            )

Step-by-step walkthrough

Metric 1 — error-rate ratio (primary gate). The query computes the canary’s 5xx ratio and divides it by the stable version’s 5xx ratio. successCondition: result < 2.0 passes as long as the canary is no more than twice as error-prone as the live baseline. Comparing to the baseline — not to zero — means a normal 0.5% error rate does not fail the gate. clamp_min guards against divide-by-zero when a series is momentarily empty.

Metric 2 — absolute floor (backstop). A pure ratio has a blind spot: if a shared dependency outage pushes both versions to 10% errors, the ratio stays near 1.0 and the canary passes while production burns. The floor query aborts whenever the canary’s absolute 5xx rate exceeds 5%, regardless of baseline.

Metric 3 — p95 latency. Error rate misses slow-but-successful regressions. histogram_quantile(0.95, …) reads p95 from the latency histogram; failureLimit: 2 tolerates one noisy spike but aborts on a sustained one.

Sample sizing. interval: 1m × count: 5 gives a five-minute analysis with five decision points. The [2m] range in each query smooths per-scrape noise. On a low-traffic service, widen interval and the range so each sample still covers hundreds of requests — sizing by request volume, not clock time, is what keeps the gate trustworthy.

Verification

Test every query in the Prometheus UI before binding it, then dry-run the analysis:

# Paste each query into http://prometheus:9090/graph and confirm it returns a value
# Then run the template standalone against a live canary:
kubectl argo rollouts get analysisrun --rollout app

# Force a regression check: deploy a version that returns 500s to the canary and
# confirm the AnalysisRun transitions to Failed within one interval.

Expected: a healthy canary shows all three metrics ✔; an injected 500-heavy build flips error-rate-floor to ✖ Failed and aborts the rollout.

Common pitfalls

  • Ratio without a floor. As above, a baseline-relative gate alone misses shared outages. Always pair it with an absolute floor.
  • Querying an instant vector where a range is needed. rate() requires a range selector ([2m]); using an instant selector returns no data and the run fails spuriously.
  • failureLimit: 0 on a noisy metric. Zero tolerance turns a single scrape blip into an abort. Allow one or two breaches on latency, and rely on count for confidence rather than hair-trigger limits.

← Back to Canary Releases & Progressive Rollouts