Wiring Rollback to Error-Rate and Latency Thresholds

You want rollback to fire on the degradations a health check passes — a rising error rate or creeping latency — without rolling back perfectly good deploys on random noise. This page designs those metric thresholds, the statistical half of automated rollback triggers and runbook integration.

When to use this pattern

  • Your deploys pass health checks but you have still shipped regressions that only showed up as elevated errors or latency.
  • You have a Prometheus-compatible metrics backend with per-version request metrics.
  • You want thresholds that distinguish a real regression from normal baseline noise.

Prerequisites

Complete working example

#!/usr/bin/env bash
# scripts/metric-rollback-gate.sh
# Exit 0 = healthy (keep the deploy). Exit 1 = breach (roll back).
set -uo pipefail
PROM="http://prometheus:9090/api/v1/query"

q() { curl -sf "$PROM" --data-urlencode "query=$1" | jq -r '.data.result[0].value[1] // "0"'; }

# --- Config ---
REL_MULT=2.0        # allow new error rate up to 2x the pre-deploy baseline
ABS_CEIL=0.05       # absolute backstop: never tolerate >5% errors regardless of baseline
P95_CEIL=0.4        # p95 latency ceiling in seconds
SAMPLES=5           # consecutive samples required to confirm a breach
INTERVAL=20         # seconds between samples

# Baseline error rate captured just before this deploy (passed in as an env var).
BASE="${BASELINE_ERROR_RATE:-0.005}"

breaches=0
for i in $(seq 1 "$SAMPLES"); do
  ERR=$(q 'sum(rate(http_requests_total{app="web",code=~"5.."}[2m]))/sum(rate(http_requests_total{app="web"}[2m]))')
  P95=$(q 'histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{app="web"}[2m])) by (le))')

  # A sample breaches if: relative error too high, OR absolute error too high, OR p95 too high.
  bad=$(awk -v e="$ERR" -v b="$BASE" -v m="$REL_MULT" -v c="$ABS_CEIL" -v p="$P95" -v pc="$P95_CEIL" \
    'BEGIN{ print ((b>0 && e > b*m) || e > c || p > pc) ? 1 : 0 }')

  if [ "$bad" = "1" ]; then
    breaches=$((breaches+1))
    echo "Sample $i BREACH — err=$ERR (base=$BASE) p95=${P95}s"
  else
    breaches=0                       # reset: breaches must be CONSECUTIVE
    echo "Sample $i ok — err=$ERR p95=${P95}s"
  fi
  [ "$breaches" -ge "$SAMPLES" ] && { echo "Sustained breach — trigger rollback"; exit 1; }
  sleep "$INTERVAL"
done
echo "No sustained breach — deploy healthy."
exit 0

Capture the baseline immediately before deploying, and pass it in:

      - name: Capture pre-deploy baseline error rate
        id: base
        run: |
          B=$(curl -sf "http://prometheus:9090/api/v1/query" \
            --data-urlencode 'query=sum(rate(http_requests_total{app="web",code=~"5.."}[10m]))/sum(rate(http_requests_total{app="web"}[10m]))' \
            | jq -r '.data.result[0].value[1] // "0.005"')
          echo "rate=$B" >> "$GITHUB_OUTPUT"

      - name: Metric rollback gate
        id: metric
        continue-on-error: true
        env:
          BASELINE_ERROR_RATE: ${{ steps.base.outputs.rate }}
        run: bash scripts/metric-rollback-gate.sh
      # ...then the same 'if: steps.metric.outcome == failure' rollback + page steps
      # from the parent guide's workflow.
Absolute Thresholds Fire on Traffic, Not on Regressions During a marketing spike, absolute latency rises above a fixed 400 millisecond threshold and triggers a rollback even though the new version performs identically to the old. A threshold expressed relative to the stable version's concurrent baseline stays flat and does not fire. absolute threshold: p95 > 400ms canary p95 — rises with the spike, crosses the line, fires wrongly canary ÷ stable, measured concurrently — flat, because nothing regressed Express thresholds as a ratio to the stable version over the same window, and traffic variation cancels out.

Step-by-step walkthrough

Baseline-relative error rate. The gate compares the post-deploy 5xx ratio to the pre-deploy baseline (e > b*m), so a service whose normal error rate is 0.5% is not rolled back for being at 0.5%. Comparing to zero would make every deploy look like a regression.

Absolute ceiling backstop. A pure ratio is blind to shared outages: if a dependency failure pushes errors up everywhere, the ratio against a now-also-elevated baseline stays low. The ABS_CEIL of 5% catches that — any absolute error rate above the ceiling breaches regardless of baseline.

p95 latency. Error rate misses regressions that are slow but successful. histogram_quantile(0.95, …) reads p95 from the latency histogram; exceeding P95_CEIL breaches even at a normal error rate.

Consecutive-sample requirement. breaches counts consecutive bad samples and resets on any good one. A single noisy spike does not roll back the deploy; only a sustained breach across all SAMPLES does. This is what separates a real regression from transient noise — the same principle behind canary analysis windows.

The Four Outcomes of a Rollback Threshold A threshold that fires on a genuine regression is a prevented incident. Firing on a healthy release costs a wasted rollback and erodes trust. Staying silent on a healthy release is the normal case. Staying silent on a genuine regression is the outcome the whole mechanism exists to avoid. release is genuinely bad release is fine fires prevented incident — the point of the system wasted rollback, and trust erodes silent the outcome the mechanism exists to avoid the normal case — most deploys land here Tune toward the top-right rather than the bottom-left: a wasted rollback is recoverable, a missed regression is an outage.

Verification

# Test each query in the Prometheus UI first, then dry-run the gate against a healthy deploy:
BASELINE_ERROR_RATE=0.005 bash scripts/metric-rollback-gate.sh; echo "exit=$?"   # → exit=0

# Force a regression (route synthetic 5xx traffic) and confirm sustained breach → exit 1:
# The log shows five consecutive BREACH lines, then "Sustained breach — trigger rollback".

Expected: a healthy deploy exits 0; a real regression produces consecutive breaches and exits 1, driving the rollback step.

How Much Traffic a Threshold Needs Before It Means Anything Detecting a five per cent error-rate increase needs roughly 1,000 canary requests. A one per cent increase needs about 25,000. A tenth of a per cent needs upward of two million. A threshold evaluated on fewer requests than its regression size requires will fire on noise. CANARY REQUESTS NEEDED TO DISTINGUISH THE REGRESSION FROM NOISE 5% error increase ~1,000 requests — a five-minute window is plenty 1% error increase ~25,000 — raise the weight or lengthen the window 0.1% error increase ~2,000,000 — no realistic canary window reaches this A threshold set finer than your traffic supports does not detect small regressions; it manufactures false ones.

Common pitfalls

  • Absolute-only thresholds. A fixed “> 1% errors” rolls back services whose healthy baseline is already near 1%, and misses services whose baseline is 0.01%. Anchor to the baseline and keep the absolute value as a backstop only.
  • Window too short. Five samples at 2-second intervals on a low-traffic service cover too few requests to be meaningful. Size INTERVAL and the [2m] range so each sample reflects hundreds of requests.
  • Non-consecutive breach counting. Counting total (not consecutive) breaches rolls back on scattered noise. Reset the counter on any healthy sample so only sustained regressions trigger.

Setting thresholds you can defend

A threshold picked because it sounded reasonable is a threshold nobody can defend when it fires during a release everyone believes is fine — and that conversation, repeated twice, is how automated rollback gets switched off.

The defensible method is to derive it from history. Collect the version-to-version metric ratio across the last few dozen healthy releases: for each, the new version’s error rate divided by the stable version’s over the same window. That distribution describes the natural variation between two versions that are, by assumption, equally good. Set the threshold outside it — commonly at the 99th percentile — and you have a number with a stated false-positive rate rather than an opinion.

The exercise frequently produces an uncomfortable finding: natural variation is wider than people assume, and a threshold tight enough to catch a small regression would have fired on a noticeable fraction of past healthy releases. That is worth knowing before the automation is trusted, not after the third unnecessary rollback.

Ratios, not absolutes

Absolute thresholds — p95 above 400 milliseconds, error rate above one per cent — fire on traffic patterns rather than on releases. A marketing spike raises latency for both versions equally, crosses the line, and triggers a rollback of a release that changed nothing relevant.

Expressing the threshold as a ratio between the canary and the stable version over the same window removes the entire class of false positive, because both versions experience the spike identically and the ratio stays flat. It costs nothing but a different query, and it is the single highest-value change to most threshold configurations.

Sizing the window to the regression

The last piece is honesty about what the available traffic can detect. Distinguishing a five per cent error-rate increase from noise takes on the order of a thousand canary requests; distinguishing a tenth of a per cent takes millions. A threshold set finer than the traffic supports does not detect small regressions — it manufactures false ones from sampling noise.

Where the traffic genuinely cannot support the sensitivity you want, the honest options are to raise the canary weight, lengthen the window, or accept that regressions below a certain size will be caught by the full rollout rather than by the gate. Pretending otherwise produces a gate that is simultaneously noisy and blind.

← Back to Automated Rollback Triggers and Runbook Integration