Writing a Rollback Runbook CI Can Execute

The runbook says “revert to the previous release and notify the channel”. At 3am, under pressure, that sentence expands into six commands nobody can remember, run against a system nobody wants to guess about. A rollback runbook that CI executes removes the recall problem entirely — and, just as importantly, can be rehearsed on a Tuesday afternoon.

When to use this pattern

  • You have an automated rollback trigger but the action it fires is still a human following prose.
  • Rollback correctness currently depends on which engineer is on call.
  • You want to rehearse the rollback path regularly without touching production.

Prerequisites

What makes a runbook executable

The gap between prose and script is not syntax — it is that prose can be vague and a script cannot. Three ambiguities have to be resolved before anything can be automated, and resolving them is most of the value even if you never write the script.

From Ambiguous Prose to Executable Steps Three rows. The prose instruction to revert to the previous release becomes a query against recorded deploy state for the last release that passed smoke tests. The instruction to switch traffic becomes an idempotent operation safe to repeat. The instruction to confirm it worked becomes an explicit assertion with a timeout and a failure path. PROSE RUNBOOK SAYS SCRIPT MUST DECIDE "revert to the previous release" previous by time? by tag? the one before the bad one? last release marked healthy in deploy state queried, not remembered "switch traffic back" what if someone already did half of it? idempotent set-to-target, never toggle safe to run five times "confirm the site is back" confirm how, for how long, and then what? assert version + error rate for 90s non-zero exit escalates to a human Writing the right-hand column is the work. The script is what falls out of it.

Complete working example

#!/usr/bin/env bash
# scripts/rollback.sh — executable rollback runbook.
#   DRY_RUN=1 ./scripts/rollback.sh production     # rehearse, change nothing
#   ./scripts/rollback.sh production               # execute
set -euo pipefail
ENV="${1:?usage: rollback.sh <environment>}"
DRY="${DRY_RUN:-0}"
run() { if [ "$DRY" = "1" ]; then echo "DRY-RUN: $*"; else "$@"; fi; }

STATE="s3://deploy-state/${ENV}/history.json"     # append-only deploy log

# 1. Resolve the target: the newest release recorded healthy that is NOT the current one.
aws s3 cp "$STATE" - > /tmp/history.json
CURRENT=$(jq -r '.[-1].release' /tmp/history.json)
TARGET=$(jq -r --arg cur "$CURRENT" \
  '[.[] | select(.healthy == true and .release != $cur)] | last | .release // empty' /tmp/history.json)
[ -n "$TARGET" ] || { echo "FATAL: no healthy prior release recorded — escalate to a human"; exit 2; }
echo "current=${CURRENT}  target=${TARGET}"

# 2. Refuse to proceed if the bad release contracted the schema — code rollback cannot fix that.
if jq -e --arg cur "$CURRENT" '.[] | select(.release == $cur) | .destructive_migration == true' \
     /tmp/history.json >/dev/null; then
  echo "FATAL: ${CURRENT} included a destructive migration. Code rollback is unsafe."
  echo "       Follow the forward-fix path; see the expand-contract guide."
  exit 3
fi

# 3. Confirm the target artifact still exists BEFORE changing anything.
run aws s3api head-object --bucket shop-releases --key "${ENV}/${TARGET}.tar.gz"

# 4. Repoint traffic. `set` semantics, not `toggle` — running this twice is a no-op.
run aws s3 cp "s3://shop-releases/${ENV}/${TARGET}.tar.gz" /tmp/rb.tar.gz
run bash -c "tar -xzf /tmp/rb.tar.gz -C /tmp/rb && aws s3 sync /tmp/rb/dist s3://shop-assets"
run aws s3 cp "/tmp/rb/dist/index.html" "s3://shop-assets/index.html" --cache-control 'no-cache'
run aws cloudfront create-invalidation --distribution-id E123EXAMPLE --paths '/index.html'

# 5. Verify: the served version must be the target, and stay healthy for 90 seconds.
if [ "$DRY" != "1" ]; then
  deadline=$(( SECONDS + 90 )); ok=0
  while (( SECONDS < deadline )); do
    served=$(curl -sf https://shop.example.com/version.json | jq -r .release || echo none)
    [ "$served" = "$TARGET" ] && ok=1 || ok=0
    sleep 10
  done
  [ "$ok" = "1" ] || { echo "FATAL: served version is '${served}', expected '${TARGET}'"; exit 4; }
fi

# 6. Record it. The rollback is itself a deploy and must appear in the history.
run bash -c "jq --arg r '${TARGET}' --arg c '${CURRENT}' \
  '. + [{release:\$r, healthy:true, rolled_back_from:\$c}]' /tmp/history.json \
  | aws s3 cp - '${STATE}'"
echo "rollback complete: ${CURRENT} → ${TARGET}"
# .github/workflows/rollback.yml — triggerable by a human or by an alert webhook.
name: Rollback
on:
  workflow_dispatch:
    inputs:
      environment: { description: Environment, required: true, default: production }
      dry_run:     { description: Rehearse only, type: boolean, default: false }
  repository_dispatch:
    types: [rollback-requested]        # fired by the alerting integration

permissions: { contents: read, id-token: write }

jobs:
  rollback:
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment || 'production' }}
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/rollback
          aws-region: eu-west-1
      - run: ./scripts/rollback.sh "${{ inputs.environment || 'production' }}"
        env:
          DRY_RUN: ${{ inputs.dry_run && '1' || '0' }}

Step-by-step walkthrough

Resolve the target from recorded state. “The previous release” is ambiguous the moment two deploys happen in an hour, or the moment the previous one was itself bad. Querying an append-only history for the last release marked healthy removes the ambiguity and makes the choice auditable afterwards.

Refuse rather than guess. Exit code 3 — destructive migration — is the most important line in the script. A rollback into a contracted schema produces a worse outage than the one being fixed, and the only correct automated behaviour is to stop and say so. This is the operational reason the expand-contract discipline matters.

Check the artifact exists before touching anything. A rollback that fails halfway because the target aged out of storage is strictly worse than one that refuses at the start. This is also why retention policy and rollback window have to agree, as discussed in setting artifact retention policies.

Use set semantics, never toggle. “Switch to the other colour” is unrunnable twice; “make the live pointer equal $TARGET” is safe any number of times. Incident automation gets re-run, by people and by retries, and idempotence is what makes that harmless.

Verify for a duration, not once. A single successful request can be served by one healthy edge while others are still serving the bad release. Sampling for 90 seconds catches partial propagation, which is the most common way a rollback is declared complete while a fraction of users still see the fault.

Record the rollback as a deploy. If it is not in the history, the next rollback will compute the wrong target — the system would still believe the bad release is current.

Verification

Rehearse it, on a schedule, without an incident:

# 1. Dry run against production — prints every action, changes nothing.
DRY_RUN=1 ./scripts/rollback.sh production

Expected:

current=2026-07-31-8d31f0a  target=2026-07-30-c47b912
DRY-RUN: aws s3api head-object --bucket shop-releases --key production/2026-07-30-c47b912.tar.gz
DRY-RUN: aws s3 cp s3://shop-releases/production/2026-07-30-c47b912.tar.gz /tmp/rb.tar.gz
…
rollback complete: 2026-07-31-8d31f0a → 2026-07-30-c47b912
# 2. Execute for real against staging, then assert the served version changed.
./scripts/rollback.sh staging
curl -sf https://staging.shop.example.com/version.json | jq -r .release

# 3. Prove idempotence — a second run must be a no-op, not a re-toggle.
./scripts/rollback.sh staging && echo "second run OK"

The third check is the one worth automating weekly. A rollback path that has not been executed in three months is a hypothesis, not a capability.

Each non-zero exit means something specific, and the on-call response differs per code — which is why they are distinct rather than all being exit 1.

Exit Codes and the Response Each One Requires Four outcomes. Exit zero means rollback complete and verified, requiring only a post-incident note. Exit two means no healthy prior release was recorded, requiring a human to choose a target. Exit three means the bad release contained a destructive migration, requiring a forward fix. Exit four means traffic did not move, requiring investigation of routing or cache. EXIT MEANING WHAT THE RESPONDER DOES 0 rolled back and verified for 90s write the incident note; nothing urgent 2 no healthy prior release in history pick a target by hand — automation cannot decide 3 destructive migration in the bad release forward-fix — do not roll back the code 4 served version never became the target check routing and edge cache, then re-run

Keeping the runbook honest over time

Executable runbooks rot in a specific way: the script keeps working while the assumptions underneath it quietly stop being true. The artifact retention window shrinks below the rollback window. A new environment is added and never wired into the deploy history. The verification endpoint changes shape and the version assertion starts comparing null to null.

The defence is a scheduled rehearsal that exercises the whole path against a non-production environment and fails loudly. Run it weekly, treat a failure as a real defect with an owner, and include the dry run against production so that target resolution — the part most likely to break silently — is checked against real state every week. A rehearsal that only runs against staging will not catch a production history file that stopped being written to.

The Weekly Rehearsal That Keeps Rollback Real A scheduled weekly job runs two things: a full rollback execution against staging and a dry run against production. Between them they catch three decay conditions — expired artifacts, unregistered environments, and a changed verification contract — each of which would otherwise surface during an incident. weekly schedule Tuesday 14:00 execute against staging the full path, for real dry run against production resolves the real target target artifact aged out of retention verification endpoint changed shape new environment never registered in history Each of these is trivial to fix on a Tuesday and expensive to discover during an outage.

Common pitfalls

Rebuilding instead of re-pointing. Rebuilding from a previous commit takes minutes and may not reproduce the artifact that was actually running. Roll back to a stored artifact.

A script that assumes a human is watching. Prompts, interactive confirmations, and read calls hang forever in CI. Every decision must be a flag or a query.

Not recording the rollback. The deploy history is the input to the next rollback’s target resolution. Skipping the write means the second rollback of an incident picks exactly the release you just backed out of.

← Back to Automated Rollback Triggers & Runbook Integration