Automating Preview Environment Teardown to Control Costs

You Googled this because your preview environment bill keeps climbing from environments that never got cleaned up — PRs closed without teardown, branches deleted directly, abandoned drafts left running. This page reclaims that spend with two independent mechanisms: immediate teardown on PR close and a scheduled idle-TTL sweep that does not depend on webhooks, both building on the ephemeral preview routing setup.

When to use this pattern

  • You run per-PR preview environments and have seen orphaned namespaces accumulate.
  • You want cost attribution (chargeback) per team and a hard cap on how long any preview survives.
  • You need teardown that survives missed webhooks and direct branch deletions.

Prerequisites

Complete working example

# .github/workflows/preview-teardown.yml — two triggers, one safety net
name: Preview Teardown
on:
  pull_request:
    types: [closed]          # immediate teardown
  schedule:
    - cron: "0 */2 * * *"    # idle sweep every 2 hours (webhook-independent)

jobs:
  # 1. Immediate teardown when a PR closes (merged or not).
  close:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - name: Configure kubectl
        run: echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > "$HOME/.kube/config" && mkdir -p "$HOME/.kube"
      - name: Delete the PR namespace
        run: |
          kubectl delete namespace "preview-${{ github.event.pull_request.number }}" \
            --ignore-not-found --wait=false
          echo "Torn down preview-${{ github.event.pull_request.number }}"

  # 2. Scheduled safety net: destroy environments idle past the TTL.
  sweep:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - name: Configure kubectl
        run: echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > "$HOME/.kube/config" && mkdir -p "$HOME/.kube"
      - name: Destroy idle preview namespaces
        run: |
          IDLE_SECONDS=$((4 * 3600))     # 4h idle TTL
          HARD_TTL_SECONDS=$((24 * 3600)) # 24h absolute cap
          NOW=$(date +%s)
          kubectl get ns -l preview=true -o json | jq -r --argjson now "$NOW" \
            --argjson idle "$IDLE_SECONDS" --argjson hard "$HARD_TTL_SECONDS" '
            .items[]
            | .metadata as $m
            | ($m.annotations["preview/last-traffic"] // $m.annotations["preview/created"] | tonumber) as $last
            | ($m.annotations["preview/created"] | tonumber) as $created
            | select( ($now - $last) > $idle or ($now - $created) > $hard )
            | $m.name' \
          | while read -r ns; do
              echo "Sweeping idle/expired namespace: $ns"
              kubectl delete namespace "$ns" --ignore-not-found --wait=false
            done

At provisioning time, every namespace must be tagged so the sweep and chargeback can find it:

# Run in the provisioning job, once per PR
kubectl label namespace "preview-${PR}" preview=true team="${TEAM}" pr="${PR}"
NOW=$(date +%s)
kubectl annotate namespace "preview-${PR}" \
  "preview/created=${NOW}" "preview/last-traffic=${NOW}" --overwrite
Three Teardown Triggers, Three Different Leaks Closing a pull request catches the normal case. An idle timeout catches environments whose pull request stays open for days without being used. A scheduled orphan sweep catches environments whose teardown failed or whose pull request was deleted, which no event will ever fire for. pull request closed the normal case — most environments idle timeout open but unused for 48 hours — most cost scheduled orphan sweep failed teardowns — no event will fire Without the third trigger, a handful of environments survive forever and quietly become the largest line on the bill.

Step-by-step walkthrough

Immediate teardown (close job). The pull_request: closed event fires on both merge and manual close. Deleting the namespace cascades to the deployment, service, ingress, and secret — everything the preview created. --wait=false returns immediately so the job does not block on finalizers.

Tagging at creation. The preview=true label is what the sweep selects on; team powers chargeback; preview/created and preview/last-traffic timestamps drive the TTL math. Without these, the sweep cannot distinguish a preview from any other namespace.

Idle sweep (sweep job). Running on a two-hour cron independent of webhooks, it lists labelled namespaces and deletes any idle beyond the 4-hour TTL or past the 24-hour hard cap. The jq expression falls back to created when no last-traffic annotation exists, so environments are never immortal. This is the safety net that catches the environments PR-close teardown misses.

Chargeback. Because every namespace carries a team label, a separate reporting query can sum compute per team — feeding the same cost discipline as tracking CI/CD compute costs for platform teams.

Teardown Is More Than Stopping the Container Stopping compute is the obvious step. The database, persistent volumes, DNS records, TLS certificates and object-storage prefixes all survive it and continue to cost money or consume quota. Certificates in particular count against rate limits, which is how a preview system stops being able to issue new ones. WHAT SURVIVES A NAIVE TEARDOWN compute ✓ removed database volumes DNS record TLS certificate Everything after the first box keeps costing money, and certificates keep consuming an issuance rate limit. Tag every resource with the pull request number at creation, then delete by tag rather than by name. Deleting by name misses anything a later change renamed — deleting by tag does not.

Verification

# After closing a PR, the namespace is gone within a minute
kubectl get ns preview-123    # → Error: namespaces "preview-123" not found

# Simulate an orphan: create a namespace with an old last-traffic and run the sweep logic
kubectl create ns preview-999
kubectl label ns preview-999 preview=true
kubectl annotate ns preview-999 "preview/created=1" "preview/last-traffic=1"
# The next scheduled sweep (or a manual run) deletes preview-999 as expired.

Expected: closed PRs are reclaimed immediately, and any namespace whose timestamps exceed the TTL is swept on the next schedule.

An Idle Timer Must Count Automated Traffic Too A nightly end-to-end suite starts against a preview whose idle timer counts only browser traffic from humans. Twenty minutes into the run the timer expires and the environment is destroyed, and the failure is reported as flaky tests rather than as a teardown race. automated suite running — no human traffic idle timer fires, environment destroyed remaining specs fail with connection refused Reported as flaky tests, investigated as a test problem, and never traced back to the teardown policy. Have the test job hold an explicit lease, or count all requests rather than only browser ones.

Common pitfalls

  • Relying on PR-close alone. Webhooks miss; branches get force-deleted. Without the scheduled sweep, orphans accumulate silently until the bill spikes. Always run both.
  • No hard TTL. An environment that keeps receiving a trickle of automated traffic (uptime pings, bots) can dodge the idle check forever. The HARD_TTL_SECONDS cap guarantees an upper bound regardless of activity.
  • Untagged namespaces. If provisioning forgets the preview=true label, the sweep cannot see the environment and it lives forever. Make tagging part of the provisioning step, not an afterthought.

Three triggers, because one is never enough

Teardown fails in ways that no single trigger covers, which is why a production-grade setup uses three.

The close event handles the normal case and covers the large majority of environments. It is also the trigger most likely to be the only one implemented, and on its own it leaves two gaps.

The idle timeout handles the long-lived pull request. An environment whose review has stalled costs money continuously while nobody looks at it, and since most of a preview’s billed lifetime is idle, this is where nearly all the recoverable spend sits. Suspending after a day of no traffic, with resume on the next request, typically removes most of the cost without any reviewer noticing.

The scheduled sweep handles everything the events missed: a teardown job that failed on a transient API error, a pull request deleted rather than closed, a resource created by a job that was cancelled before it recorded anything. No event will ever fire for these, so only a periodic reconciliation against the set of open pull requests finds them. Without it, a handful of environments survive indefinitely and eventually become the largest single line on the bill.

Teardown is more than stopping the container

Stopping compute is the obvious part and rarely the expensive part. Databases, persistent volumes, DNS records, TLS certificates and object-storage prefixes all survive a naive teardown and continue to cost money or consume quota. Certificates in particular count against issuance rate limits, so accumulated orphans eventually prevent new previews from getting working TLS at all — a failure that presents as an unrelated infrastructure problem.

Tag every resource with the pull request number at creation and delete by tag. Deleting by name works until the first naming-convention change, after which the sweep silently stops finding older resources and they persist forever.

Not racing the work still in progress

An idle timer that counts only browser traffic will happily destroy an environment while an automated suite is running against it, and the resulting failures are reported as flaky tests rather than as a teardown race — which means the actual cause goes uninvestigated for a long time.

Either have long-running jobs hold an explicit lease that suppresses teardown, or count all requests rather than only interactive ones. The lease is cleaner because it also covers work that generates little traffic, such as a job waiting on an external callback.

← Back to Ephemeral Preview URL Management and Routing