Triggering Automatic Rollback on Failed Health Checks

You want a deploy that reverts itself the moment the new version fails a post-deploy health check, with no human watching the graph — this is the complete health-triggered rollback workflow, the binary-signal half of automated rollback triggers and runbook integration.

When to use this pattern

  • You deploy a service with a meaningful health endpoint and want hard failures caught instantly.
  • You keep the previous version available (image history, warm standby) so revert is fast.
  • You want on-call notified automatically even though recovery is hands-off.

Prerequisites

Complete working example

// health.ts — a DEEP health check: healthy only if dependencies are reachable
import { db, redis } from "./clients";

export async function health(): Promise<Response> {
  const checks: Record<string, boolean> = {};
  try { await db.query("SELECT 1"); checks.database = true; } catch { checks.database = false; }
  try { await redis.ping();        checks.cache = true;    } catch { checks.cache = false; }
  const ok = Object.values(checks).every(Boolean);
  return new Response(JSON.stringify({ status: ok ? "ok" : "degraded", checks }), {
    status: ok ? 200 : 503,
    headers: { "content-type": "application/json" },
  });
}
# .github/workflows/deploy-health-rollback.yml
name: Deploy with Health Rollback
on:
  push: { branches: [main] }

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Configure kubectl
        run: echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > "$HOME/.kube/config" && mkdir -p "$HOME/.kube"

      - name: Record last known-good image
        id: lkg
        run: |
          LKG=$(kubectl get deploy app -o jsonpath='{.spec.template.spec.containers[0].image}')
          echo "image=$LKG" >> "$GITHUB_OUTPUT"

      - name: Deploy new version
        run: |
          kubectl set image deploy/app app=ghcr.io/${{ github.repository }}:${{ github.sha }}
          kubectl rollout status deploy/app --timeout=180s

      - name: Health gate — poll with bounded retries
        id: health
        continue-on-error: true
        run: |
          URL="https://app.example.com/health"
          for i in $(seq 1 18); do            # 18 × 10s = up to 3 minutes of warm-up
            CODE=$(curl -s -o /tmp/h.json -w '%{http_code}' "$URL" || echo 000)
            if [ "$CODE" = "200" ] && jq -e '.status == "ok"' /tmp/h.json >/dev/null; then
              echo "Healthy after $((i*10))s"; exit 0
            fi
            echo "Attempt $i: HTTP $CODE $(jq -c '.checks // {}' /tmp/h.json 2>/dev/null)"
            sleep 10
          done
          echo "Health never reached ok within timeout"; exit 1

      - name: Roll back to last known-good
        if: steps.health.outcome == 'failure'
        run: |
          kubectl set image deploy/app app=${{ steps.lkg.outputs.image }}
          kubectl rollout status deploy/app --timeout=120s
          echo "Rolled back to ${{ steps.lkg.outputs.image }}"

      - name: Page on-call
        if: steps.health.outcome == 'failure'
        run: |
          curl -sf -X POST https://events.pagerduty.com/v2/enqueue \
            -H 'Content-Type: application/json' \
            -d '{"routing_key":"${{ secrets.PD_ROUTING_KEY }}","event_action":"trigger","payload":{"summary":"Health-check rollback: ${{ github.sha }} reverted to ${{ steps.lkg.outputs.image }}","severity":"error","source":"ci-health-rollback"}}'

      - name: Fail the run if rolled back
        if: steps.health.outcome == 'failure'
        run: exit 1

Step-by-step walkthrough

Deep health endpoint. health.ts returns 200 only when the database query and cache ping both succeed. This is the crux: a shallow endpoint that returns 200 on process start would let a deploy with an unreachable database pass the gate. The checks object also gives the log a precise reason for failure.

Recording last known-good. Before changing the image, the workflow reads the currently-serving image reference and stores it as a step output. Rollback targets this specific image, not a relative “previous,” avoiding the loop where you revert to another bad version.

Bounded-retry health gate. The poll runs up to 18 times at 10-second intervals — three minutes — to tolerate normal warm-up (JIT, cache priming, connection pools) before declaring failure. continue-on-error: true lets the workflow reach the rollback step instead of aborting on the failed gate.

Revert and notify. On failure, the image is set back to last known-good and the rollout is awaited, then PagerDuty is triggered with the from/to versions. A final exit 1 marks the run red so the failure is visible in the deploy history even though recovery was automatic.

Verification

# Simulate a broken deploy: ship an image whose /health returns 503, then watch the run.
# Expected timeline in the Actions log:
#   Deploy new version            ✅
#   Health gate                   ❌ (18 attempts, checks show database:false)
#   Roll back to last known-good  ✅  → image reverts
#   Page on-call                  ✅  → PagerDuty incident created
#   Fail the run if rolled back   ❌  → run marked red

# Confirm production is back on the good image:
kubectl get deploy app -o jsonpath='{.spec.template.spec.containers[0].image}'

Expected: the run reverts to the recorded image, pages on-call, and finishes red.

Common pitfalls

  • Shallow health endpoint. Returning 200 on liveness alone makes the gate meaningless — the deploy “passes” while the database is down. Always assert dependencies, and add metric thresholds for degradations a health check misses.
  • No warm-up tolerance. A single immediate check rolls back healthy deploys that just needed 20 seconds to warm up. Poll with bounded retries.
  • Relative rollback target. kubectl rollout undo reverts to the previous ReplicaSet, which may itself be bad. Pin to a recorded last-known-good image instead, per the parent guide.

← Back to Automated Rollback Triggers and Runbook Integration