Wiring Rollback to Error-Rate and Latency Thresholds

You want rollback to fire on the degradations a health check passes — a rising error rate or creeping latency — without rolling back perfectly good deploys on random noise. This page designs those metric thresholds, the statistical half of automated rollback triggers and runbook integration.

When to use this pattern

  • Your deploys pass health checks but you have still shipped regressions that only showed up as elevated errors or latency.
  • You have a Prometheus-compatible metrics backend with per-version request metrics.
  • You want thresholds that distinguish a real regression from normal baseline noise.

Prerequisites

Complete working example

#!/usr/bin/env bash
# scripts/metric-rollback-gate.sh
# Exit 0 = healthy (keep the deploy). Exit 1 = breach (roll back).
set -uo pipefail
PROM="http://prometheus:9090/api/v1/query"

q() { curl -sf "$PROM" --data-urlencode "query=$1" | jq -r '.data.result[0].value[1] // "0"'; }

# --- Config ---
REL_MULT=2.0        # allow new error rate up to 2x the pre-deploy baseline
ABS_CEIL=0.05       # absolute backstop: never tolerate >5% errors regardless of baseline
P95_CEIL=0.4        # p95 latency ceiling in seconds
SAMPLES=5           # consecutive samples required to confirm a breach
INTERVAL=20         # seconds between samples

# Baseline error rate captured just before this deploy (passed in as an env var).
BASE="${BASELINE_ERROR_RATE:-0.005}"

breaches=0
for i in $(seq 1 "$SAMPLES"); do
  ERR=$(q 'sum(rate(http_requests_total{app="web",code=~"5.."}[2m]))/sum(rate(http_requests_total{app="web"}[2m]))')
  P95=$(q 'histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{app="web"}[2m])) by (le))')

  # A sample breaches if: relative error too high, OR absolute error too high, OR p95 too high.
  bad=$(awk -v e="$ERR" -v b="$BASE" -v m="$REL_MULT" -v c="$ABS_CEIL" -v p="$P95" -v pc="$P95_CEIL" \
    'BEGIN{ print ((b>0 && e > b*m) || e > c || p > pc) ? 1 : 0 }')

  if [ "$bad" = "1" ]; then
    breaches=$((breaches+1))
    echo "Sample $i BREACH — err=$ERR (base=$BASE) p95=${P95}s"
  else
    breaches=0                       # reset: breaches must be CONSECUTIVE
    echo "Sample $i ok — err=$ERR p95=${P95}s"
  fi
  [ "$breaches" -ge "$SAMPLES" ] && { echo "Sustained breach — trigger rollback"; exit 1; }
  sleep "$INTERVAL"
done
echo "No sustained breach — deploy healthy."
exit 0

Capture the baseline immediately before deploying, and pass it in:

      - name: Capture pre-deploy baseline error rate
        id: base
        run: |
          B=$(curl -sf "http://prometheus:9090/api/v1/query" \
            --data-urlencode 'query=sum(rate(http_requests_total{app="web",code=~"5.."}[10m]))/sum(rate(http_requests_total{app="web"}[10m]))' \
            | jq -r '.data.result[0].value[1] // "0.005"')
          echo "rate=$B" >> "$GITHUB_OUTPUT"

      - name: Metric rollback gate
        id: metric
        continue-on-error: true
        env:
          BASELINE_ERROR_RATE: ${{ steps.base.outputs.rate }}
        run: bash scripts/metric-rollback-gate.sh
      # ...then the same 'if: steps.metric.outcome == failure' rollback + page steps
      # from the parent guide's workflow.

Step-by-step walkthrough

Baseline-relative error rate. The gate compares the post-deploy 5xx ratio to the pre-deploy baseline (e > b*m), so a service whose normal error rate is 0.5% is not rolled back for being at 0.5%. Comparing to zero would make every deploy look like a regression.

Absolute ceiling backstop. A pure ratio is blind to shared outages: if a dependency failure pushes errors up everywhere, the ratio against a now-also-elevated baseline stays low. The ABS_CEIL of 5% catches that — any absolute error rate above the ceiling breaches regardless of baseline.

p95 latency. Error rate misses regressions that are slow but successful. histogram_quantile(0.95, …) reads p95 from the latency histogram; exceeding P95_CEIL breaches even at a normal error rate.

Consecutive-sample requirement. breaches counts consecutive bad samples and resets on any good one. A single noisy spike does not roll back the deploy; only a sustained breach across all SAMPLES does. This is what separates a real regression from transient noise — the same principle behind canary analysis windows.

Verification

# Test each query in the Prometheus UI first, then dry-run the gate against a healthy deploy:
BASELINE_ERROR_RATE=0.005 bash scripts/metric-rollback-gate.sh; echo "exit=$?"   # → exit=0

# Force a regression (route synthetic 5xx traffic) and confirm sustained breach → exit 1:
# The log shows five consecutive BREACH lines, then "Sustained breach — trigger rollback".

Expected: a healthy deploy exits 0; a real regression produces consecutive breaches and exits 1, driving the rollback step.

Common pitfalls

  • Absolute-only thresholds. A fixed “> 1% errors” rolls back services whose healthy baseline is already near 1%, and misses services whose baseline is 0.01%. Anchor to the baseline and keep the absolute value as a backstop only.
  • Window too short. Five samples at 2-second intervals on a low-traffic service cover too few requests to be meaningful. Size INTERVAL and the [2m] range so each sample reflects hundreds of requests.
  • Non-consecutive breach counting. Counting total (not consecutive) breaches rolls back on scattered noise. Reset the counter on any healthy sample so only sustained regressions trigger.

← Back to Automated Rollback Triggers and Runbook Integration