Blocking Deploys on Config Drift with a Parity Gate

You want a pipeline step that refuses to deploy when a candidate’s configuration has drifted from production — a missing key or an undeclared extra one — before it causes a “worked in preview, broke in prod” incident. This is the complete, copy-paste config-drift gate that implements the key-diff check from environment parity validation gates.

When to use this pattern

  • You promote config through stages and have been bitten by a key present in one environment but not another.
  • You want an explicit, auditable allow-list for the keys that are supposed to differ.
  • You need the check to block merges, not just warn.

Prerequisites

Complete working example

#!/usr/bin/env bash
# scripts/config-drift-gate.sh — fails the pipeline on config-key drift vs production
set -euo pipefail

# Keys that are ALLOWED to differ between environments (explicit, auditable).
# Extend this list deliberately; every entry is a documented exception.
ALLOWED_OVERRIDES='^(APP_ENV|DATABASE_URL|REDIS_URL|API_BASE_URL|SENTRY_DSN|PREVIEW_URL)$'

# 1. Production baseline key set (read-only).
prod_keys() {
  kubectl get configmap app-config -n production -o json \
    | jq -r '.data | keys[]'
}

# 2. Candidate key set from the rendered preview/staging config JSON.
candidate_keys() {
  jq -r '.data | keys[]' candidate-config.json
}

# 3. Strip allow-listed keys, sort, and diff the remainder.
strip() { grep -Ev "$ALLOWED_OVERRIDES" | sort; }

PROD=$(prod_keys | strip)
CAND=$(candidate_keys | strip)

if diff <(echo "$PROD") <(echo "$CAND") >/dev/null; then
  echo "✅ Config parity OK — non-override key sets match production."
  exit 0
fi

echo "❌ CONFIG DRIFT — candidate diverges from production (< prod, > candidate):"
diff <(echo "$PROD") <(echo "$CAND") || true
echo
echo "Resolve by declaring the key in both environments, or add it to ALLOWED_OVERRIDES if it is a legitimate stage-specific key."
exit 1
# .github/workflows/config-drift-gate.yml
name: Config Drift Gate
on:
  pull_request:
    types: [opened, synchronize]
jobs:
  drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Render candidate config
        run: ./scripts/render-config.sh > candidate-config.json
      - name: Configure kubectl (read-only prod access)
        run: echo "${{ secrets.KUBE_CONFIG_RO }}" | base64 -d > "$HOME/.kube/config" && mkdir -p "$HOME/.kube"
      - name: Run config-drift gate
        run: bash scripts/config-drift-gate.sh    # non-zero exit blocks the merge
Compare Key Sets, Never Values The production and candidate key sets are compared after removing allow-listed overrides. A key present in production but absent from the candidate is a missing requirement. A key present only in the candidate is undeclared. Values are excluded because they are supposed to differ between environments. AFTER STRIPPING ALLOW-LISTED KEYS production keys FEATURE_X · MAILER_MODE RATE_LIMIT · CDN_HOST candidate keys FEATURE_X · MAILER_MODE CDN_HOST · DEBUG_PANEL the diff missing: RATE_LIMIT undeclared: DEBUG_PANEL Both findings block. One is a missing requirement, the other is configuration that exists nowhere in version control.

Step-by-step walkthrough

The allow-list. ALLOWED_OVERRIDES is an explicit regex of keys that are meant to differ between environments — the database URL, the app env name, the preview URL. Every entry is a documented, auditable exception. Keeping it explicit means the gate stays strict about everything else.

Reading both key sets. prod_keys pulls the production ConfigMap’s keys with read-only credentials; candidate_keys reads the rendered candidate. Both are reduced to sorted key lists — values are never compared, because values are supposed to differ.

The diff. After stripping allow-listed keys and sorting, diff of the two sets is empty only when they match exactly. A < line means a key exists in production but not the candidate (a missing required key); a > line means the candidate has an undeclared extra key. Either fails the job.

Blocking the merge. Because the script exits non-zero on drift and the workflow runs on pull requests, registering the drift job as a required status check in branch protection prevents a divergent config from merging at all — the gate is preventive, not advisory.

Where Config Drift Comes From Fifty-four per cent of drift originates in a manual edit made directly to production during an incident. Twenty-nine per cent comes from keys added for a preview and never promoted. Seventeen per cent comes from configuration that exists only on an unmerged branch. ORIGIN OF DRIFT INCIDENTS manual production edit during an incident — 54% preview-only key — 29% branch — 17% The majority is created deliberately, under pressure, by someone fixing something — and never written back. Which is why the gate has to run on every deploy rather than only when configuration files change.

Verification

# Passing case: identical non-override keys → exit 0
bash scripts/config-drift-gate.sh; echo "exit=$?"   # → exit=0

# Failing case: inject an undeclared key into the candidate and re-run
jq '.data.UNDECLARED_KEY = "x"' candidate-config.json > tmp && mv tmp candidate-config.json
bash scripts/config-drift-gate.sh; echo "exit=$?"   # → prints "> UNDECLARED_KEY", exit=1

Expected: the clean candidate passes; the injected key produces a > diff line and a non-zero exit that blocks the deploy.

Required Check or Advisory Job As a required status check the gate blocks the merge, so drift cannot reach the main branch. As an advisory job it reports and is routinely ignored, because a job that never blocks anything stops being read within weeks. required status check drift cannot merge — the gate is preventive costs a few seconds per pull request advisory job reports, blocks nothing, read for two weeks costs the same and prevents nothing If the gate is too noisy to be required, fix the noise — do not demote it, because a demoted gate is a deleted gate.

Common pitfalls

  • Comparing values instead of keys. Value comparison fails on every legitimate difference (different DB URLs) and trains people to bypass the gate. Diff keys only, as the script does.
  • An implicit or ever-growing allow-list. If every failure is “fixed” by adding the key to ALLOWED_OVERRIDES, the gate erodes to nothing. Treat each addition as a reviewed exception with a reason, not a reflex.
  • Running after deploy. A drift gate that runs post-deploy can only roll back a running environment. Run it pre-provision and as a required check so drift never starts, per the parent guide.

Where drift comes from, and why the gate must run every time

It is worth understanding the source of drift, because it explains why a gate that only runs when configuration files change catches almost nothing.

The dominant source is a manual edit made directly to production. Someone is fixing an incident at two in the morning, adds a variable through a console to unblock the system, and the fix works. The change is never written back to version control because the incident is over and the pressure is gone. Nothing in the repository changed, so a gate triggered by file changes never runs, and the divergence persists until it causes the next incident.

The second source is a variable added for a preview or a staging experiment and never promoted. It exists in one environment, works there, and is discovered missing only when the feature reaches production — which is precisely the case the gate exists to catch, and again involves no committed change to any configuration file.

The third and smallest source is configuration living on an unmerged branch, where a feature’s variable is defined in a branch that has not landed while the feature’s code has.

All three share a property: they are invisible to anything that watches the repository. The gate has to read the environments themselves, on every deploy, or it reads a version of reality that is up to months out of date.

Reading production safely

Comparing against production requires production access from the pipeline, which is a legitimate concern. The mitigation is that the gate needs only key names, never values — so the credential it uses can be scoped to list keys and nothing else. On Kubernetes that is a role granting get on a single ConfigMap; on a secret manager it is a list permission on one path. Neither can read a secret’s contents.

Setting that scope explicitly is worth the ten minutes. A gate running with broad production read access is a new and unnecessary target, and the narrow permission is both sufficient and easy to justify in a review.

Making the failure message actionable

The gate’s value collapses if its output is a bare non-zero exit. The message should name each key, say which side it is missing from, and state the two legitimate resolutions: declare the key in both environments, or add it to the allow-list with a reason. Without that, the reflex response to a red gate is to add whatever the failing key is to the allow-list, which is how the gate erodes.

A useful refinement is to print the allow-list’s current size alongside any failure. A team watching that number grow is far more likely to question an addition than one that only ever sees a single key name in isolation.

← Back to Environment Parity Validation Gates