Triggering Automatic Rollback on Failed Health Checks

You want a deploy that reverts itself the moment the new version fails a post-deploy health check, with no human watching the graph — this is the complete health-triggered rollback workflow, the binary-signal half of automated rollback triggers and runbook integration.

When to use this pattern

  • You deploy a service with a meaningful health endpoint and want hard failures caught instantly.
  • You keep the previous version available (image history, warm standby) so revert is fast.
  • You want on-call notified automatically even though recovery is hands-off.

Prerequisites

Complete working example

// health.ts — a DEEP health check: healthy only if dependencies are reachable
import { db, redis } from "./clients";

export async function health(): Promise<Response> {
  const checks: Record<string, boolean> = {};
  try { await db.query("SELECT 1"); checks.database = true; } catch { checks.database = false; }
  try { await redis.ping();        checks.cache = true;    } catch { checks.cache = false; }
  const ok = Object.values(checks).every(Boolean);
  return new Response(JSON.stringify({ status: ok ? "ok" : "degraded", checks }), {
    status: ok ? 200 : 503,
    headers: { "content-type": "application/json" },
  });
}
# .github/workflows/deploy-health-rollback.yml
name: Deploy with Health Rollback
on:
  push: { branches: [main] }

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Configure kubectl
        run: echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > "$HOME/.kube/config" && mkdir -p "$HOME/.kube"

      - name: Record last known-good image
        id: lkg
        run: |
          LKG=$(kubectl get deploy app -o jsonpath='{.spec.template.spec.containers[0].image}')
          echo "image=$LKG" >> "$GITHUB_OUTPUT"

      - name: Deploy new version
        run: |
          kubectl set image deploy/app app=ghcr.io/${{ github.repository }}:${{ github.sha }}
          kubectl rollout status deploy/app --timeout=180s

      - name: Health gate — poll with bounded retries
        id: health
        continue-on-error: true
        run: |
          URL="https://app.example.com/health"
          for i in $(seq 1 18); do            # 18 × 10s = up to 3 minutes of warm-up
            CODE=$(curl -s -o /tmp/h.json -w '%{http_code}' "$URL" || echo 000)
            if [ "$CODE" = "200" ] && jq -e '.status == "ok"' /tmp/h.json >/dev/null; then
              echo "Healthy after $((i*10))s"; exit 0
            fi
            echo "Attempt $i: HTTP $CODE $(jq -c '.checks // {}' /tmp/h.json 2>/dev/null)"
            sleep 10
          done
          echo "Health never reached ok within timeout"; exit 1

      - name: Roll back to last known-good
        if: steps.health.outcome == 'failure'
        run: |
          kubectl set image deploy/app app=${{ steps.lkg.outputs.image }}
          kubectl rollout status deploy/app --timeout=120s
          echo "Rolled back to ${{ steps.lkg.outputs.image }}"

      - name: Page on-call
        if: steps.health.outcome == 'failure'
        run: |
          curl -sf -X POST https://events.pagerduty.com/v2/enqueue \
            -H 'Content-Type: application/json' \
            -d '{"routing_key":"${{ secrets.PD_ROUTING_KEY }}","event_action":"trigger","payload":{"summary":"Health-check rollback: ${{ github.sha }} reverted to ${{ steps.lkg.outputs.image }}","severity":"error","source":"ci-health-rollback"}}'

      - name: Fail the run if rolled back
        if: steps.health.outcome == 'failure'
        run: exit 1
The Gate Runs Before Users Do The new version is deployed to capacity that receives no user traffic. The health probe exercises it there. Only a passing probe opens the router to real requests, so a failed deploy is never user-visible; a probe placed after the traffic switch can only detect damage already done. deploy to idle no user traffic yet health probe 3 consecutive passes pass open the router fail roll back — no user ever saw it Probing after the switch turns a prevented incident into a short one. The ordering is the whole value of the gate.

Step-by-step walkthrough

Deep health endpoint. health.ts returns 200 only when the database query and cache ping both succeed. This is the crux: a shallow endpoint that returns 200 on process start would let a deploy with an unreachable database pass the gate. The checks object also gives the log a precise reason for failure.

Recording last known-good. Before changing the image, the workflow reads the currently-serving image reference and stores it as a step output. Rollback targets this specific image, not a relative “previous,” avoiding the loop where you revert to another bad version.

Bounded-retry health gate. The poll runs up to 18 times at 10-second intervals — three minutes — to tolerate normal warm-up (JIT, cache priming, connection pools) before declaring failure. continue-on-error: true lets the workflow reach the rollback step instead of aborting on the failed gate.

Revert and notify. On failure, the image is set back to last known-good and the rollout is awaited, then PagerDuty is triggered with the from/to versions. A final exit 1 marks the run red so the failure is visible in the deploy history even though recovery was automatic.

What Each Health Probe Actually Proves A liveness probe proves only that the process is running and will pass while every request returns 500. A readiness probe additionally proves the database and cache respond. A synthetic transaction probe exercises a real user path end to end and is the only one that would have caught the last three incidents. liveness the process is running passes while every request returns 500 readiness dependencies answer database, cache, queue all reachable synthetic transaction a real user path completes the only probe that would have caught the last three Rollback triggered by the leftmost probe fires late or never; triggered by the rightmost it fires on the failure users experience.

Verification

# Simulate a broken deploy: ship an image whose /health returns 503, then watch the run.
# Expected timeline in the Actions log:
#   Deploy new version            ✅
#   Health gate                   ❌ (18 attempts, checks show database:false)
#   Roll back to last known-good  ✅  → image reverts
#   Page on-call                  ✅  → PagerDuty incident created
#   Fail the run if rolled back   ❌  → run marked red

# Confirm production is back on the good image:
kubectl get deploy app -o jsonpath='{.spec.template.spec.containers[0].image}'

Expected: the run reverts to the recorded image, pages on-call, and finishes red.

Probe Interval × Failure Threshold = Detection Delay With a thirty-second probe interval and a threshold of three consecutive failures, the worst-case detection delay is ninety seconds plus one interval, or two minutes. Halving the interval to fifteen seconds halves the delay without changing how many consecutive failures are required. 30s INTERVAL × 3 FAILURES fail 1 fail 2 fail 3 — rollback fires worst case ≈ 2 minutes of user-visible failure Lowering the threshold to 2 halves the delay and roughly doubles false positives; halving the interval halves it for free. Tune the interval first — it is the term with no downside.

Common pitfalls

  • Shallow health endpoint. Returning 200 on liveness alone makes the gate meaningless — the deploy “passes” while the database is down. Always assert dependencies, and add metric thresholds for degradations a health check misses.
  • No warm-up tolerance. A single immediate check rolls back healthy deploys that just needed 20 seconds to warm up. Poll with bounded retries.
  • Relative rollback target. kubectl rollout undo reverts to the previous ReplicaSet, which may itself be bad. Pin to a recorded last-known-good image instead, per the parent guide.

Designing the probe the rollback depends on

The rollback trigger is only as good as the probe it reads, and probes are usually written for the orchestrator’s restart logic rather than for a deploy gate. The two want different things.

An orchestrator’s liveness probe answers “should I restart this process”. It is deliberately shallow, because a deep check that fails during a transient dependency blip would cause restart storms. A deploy gate wants the opposite: a check deep enough that passing it means the release genuinely works, evaluated once rather than continuously.

That difference argues for a separate endpoint. A /health/deep that asserts database connectivity, migration revision, cache reachability and one representative read path costs a few hundred milliseconds, which is unacceptable every five seconds and entirely fine as a gate run a handful of times during a deploy.

Choosing the assertion

The most useful deep check is a synthetic transaction — a real user path exercised end to end, such as fetching a known record through the same code path a request would take. It catches the class of failure that a connectivity check misses: the dependency answers, the query is malformed, and every user request returns a 500 while every probe returns 200.

Keep the synthetic path read-only and idempotent, or the probe becomes a source of load and of data. A read of a known fixture record is enough; anything that writes creates its own cleanup problem and eventually its own incident.

Failure handling that does not overreact

Two parameters decide how the gate behaves under transient noise: the interval between probes and the number of consecutive failures required. Their product is the worst-case detection delay, and both have costs. A short interval costs a little load. A low failure threshold costs false positives during ordinary blips.

The asymmetry favours tuning the interval first: halving it halves detection time at essentially no cost, whereas halving the threshold roughly doubles the false-positive rate. Teams routinely do the opposite, lowering the threshold to one because detection felt slow, and then spend weeks investigating rollbacks that were triggered by a single dropped connection.

← Back to Automated Rollback Triggers and Runbook Integration