Deployment Strategies & Progressive Delivery

Progressive delivery replaces the all-at-once production push with a controlled, observable, reversible rollout — new code reaches users in slices that can be halted or reversed the instant a metric moves the wrong way. This guide is for platform engineers and release owners who need to ship frontend and full-stack applications frequently without trading away production stability.

Architecture Overview

Every progressive delivery strategy is a variation on the same control loop: deploy the new version without exposing it, verify it against health signals, shift traffic incrementally, and promote or roll back based on what the metrics say. The diagram below shows that loop end to end.

The three infrastructure strategies — blue-green, canary, and rolling — differ only in how the traffic slice is shifted. Feature-flag-driven release management sits on top of all of them, moving the exposure decision from infrastructure into application code.

Core Concepts

Pattern 1 — Atomic Blue-Green Cutover (Foundational)

When to use it: Any service where you can briefly run two versions and need an instant, reversible cutover. This is the baseline production-grade strategy and the simplest to reason about.

Blue-green keeps two environments — one live (blue), one idle (green). You deploy to the idle one, validate it in isolation, then flip the router. Because the previous version stays warm, rollback is a second flip, not a redeploy.

# .github/workflows/blue-green.yml — cutover job (abridged)
name: Blue-Green Deploy
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Determine idle color
        id: color
        run: |
          # Read which color currently serves live traffic from the router config
          LIVE=$(kubectl get service app-router -o jsonpath='{.spec.selector.color}')
          IDLE=$([ "$LIVE" = "blue" ] && echo green || echo blue)
          echo "idle=$IDLE" >> "$GITHUB_OUTPUT"

      - name: Deploy to idle environment
        run: kubectl set image deploy/app-${{ steps.color.outputs.idle }} app=ghcr.io/${{ github.repository }}:${{ github.sha }}

      - name: Smoke test idle before cutover
        run: |
          # Hit the idle environment directly, bypassing the live router
          kubectl rollout status deploy/app-${{ steps.color.outputs.idle }} --timeout=120s
          curl -sf "http://app-${{ steps.color.outputs.idle }}.internal/health" | jq -e '.status == "ok"'

      - name: Flip the router (atomic cutover)
        run: |
          # A single patch redirects 100% of traffic — the old color stays warm for rollback
          kubectl patch service app-router -p \
            '{"spec":{"selector":{"color":"${{ steps.color.outputs.idle }}"}}}'

Common mis-configurations:

• Deleting or scaling down the old environment immediately after cutover — you lose the instant rollback path. Keep it warm for at least one analysis window.
• Smoke-testing through the live router instead of hitting the idle environment directly — the test then validates the old version, not the new one.
• Forgetting database backward compatibility — the two versions share one database, so schema changes must be additive (expand-then-contract), or the flip breaks the version still serving traffic.

For static frontends, the same cutover is achieved with a symlink swap rather than a router patch — see atomic symlink deploys for static frontends.

Pattern 2 — Metric-Gated Canary (Intermediate)

When to use it: High-traffic services where even a brief exposure of a bad version to 100% of users is unacceptable, and you have enough traffic for statistically meaningful analysis within minutes.

A canary routes a small slice — typically 5% — to the new version, holds for an analysis window, and widens only if the new version’s metrics match the stable baseline. The rollout is a step function guarded at every step.

# Argo Rollouts canary spec — the traffic ladder with automated analysis
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: app
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5          # 5% of traffic to the new version
        - pause: { duration: 5m }   # analysis window
        - analysis:             # automated metric check — aborts on failure
            templates:
              - templateName: error-rate-and-latency
        - setWeight: 25
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100        # full promotion

The analysis step is the heart of the pattern: it queries your metrics backend and fails the rollout automatically if the new version regresses. See automating canary analysis with Prometheus metrics for the full AnalysisTemplate.

Common mis-configurations:

• Analysis windows too short to accumulate signal — a 30-second window on a low-traffic endpoint sees too few requests to detect a 1% error-rate regression. Size the window to guarantee a minimum request count, not a fixed clock time.
• Comparing the canary against absolute thresholds instead of the live baseline — a 0.5% error rate might be normal for your app; compare canary-vs-stable, not canary-vs-zero.
• Routing canary traffic by round-robin without session affinity — a user bouncing between versions on every request sees inconsistent behaviour.

Pattern 3 — Progressive Delivery with Release Control (Advanced / Production-Grade)

When to use it: Large teams shipping many changes per day, where deployment frequency must be decoupled from release risk and individual features need independent kill switches.

At scale, the infrastructure rollout and the feature release are separated entirely. Code ships to production continuously (often dark), and a feature flag governs exposure. This lets you deploy a canary of the binary while simultaneously running a percentage rollout of a feature inside it — two independent control loops.

// Release control lives in application code, not the pipeline.
// The deploy already happened; this decides who sees the feature.
import { flags } from "./flags";

export async function checkoutHandler(req: Request, user: User) {
  // Deploy and release are independent: this code is live for everyone,
  // but the new flow is gated to a growing cohort with an instant kill switch.
  if (await flags.isEnabled("new-checkout-flow", { userId: user.id })) {
    return newCheckout(req, user);   // ramped 1% → 50% → 100% via the flag service
  }
  return legacyCheckout(req, user);  // stable path — the automatic fallback
}

Scale and failure modes at this level:

• Flag debt. Every temporary flag is a branch in production that must be removed after rollout. Track flag age and fail CI when a “temporary” flag exceeds its TTL — otherwise the codebase accretes dead toggles that eventually cause an incident.
• Long-lived schema skew. With deploy and release decoupled, old and new code paths coexist for days. Database migrations must stay backward-compatible across the entire flag lifetime, not just the deploy window.
• Flag-service availability. The flag evaluation service becomes a production dependency. Cache flag state locally with a stale-but-available fallback so a flag-service outage defaults to the stable path rather than erroring.

Environment & Toolchain Matrix

Which tools you already run for pipeline architecture largely dictate this choice: teams on a service mesh get canary weighting for free, while teams on managed static hosting lean on atomic deploys and edge routing.

Cost & Performance Trade-offs

The dominant cost lever is the duration of the doubled-capacity window in blue-green. Retire the old environment as soon as the analysis window closes — but not before — and use self-hosted runners to keep the deploy pipeline itself off billable minutes.

Failure Modes & Remediation

1. Cutover flips before the new environment is ready

Root cause: The router is patched before pods pass readiness, so traffic hits a version that is still starting. Symptom: a spike of connection-refused or 502 errors at the exact moment of cutover.

Fix: Gate the flip on kubectl rollout status completing and a direct health check succeeding against the idle environment, as in Pattern 1. Never flip on deploy-submitted; flip on ready-and-verified.

2. Canary looks healthy but the baseline is unhealthy

Root cause: The analysis compares the canary against a stable version that is itself degraded, so a regression passes the gate because “both are equally bad.”

Fix: Add absolute floor guardrails alongside the relative comparison — e.g. abort if canary error rate exceeds 2% regardless of baseline. Relative comparison catches regressions; absolute floors catch shared outages.

3. Rollback fails because the old version was already scaled to zero

Root cause: A cost-optimization step scaled down the previous version immediately after cutover, so there is nothing warm to roll back to.

Fix: Keep the previous version at minimum viable replicas until the analysis window closes. Encode this as a policy in the pipeline, not a manual step. Full runbook in automated rollback triggers and runbook integration.

4. Schema migration breaks the version still serving traffic

Root cause: A destructive migration (dropped column, renamed table) ran during deploy while the old version — still live during blue-green validation — expected the old schema.

Fix: Use expand-then-contract migrations: add the new column, deploy code that writes both, backfill, switch reads, and only drop the old column in a later release once no running version references it.

5. Feature flag left on for a cohort after rollback

Root cause: Infrastructure rolled back but the feature flag stayed enabled, so the reverted binary still exposed a half-finished feature via a flag it no longer fully supported.

Fix: Treat the flag kill switch as part of the rollback runbook. Automated rollbacks that touch flagged features must flip the flag off as a first action, before or alongside the traffic shift.

Frequently Asked Questions

Related

• Blue-Green Deployments for Full-Stack Apps — two-environment cutover mechanics, database compatibility, and warm-standby rollback.
• Canary Releases & Progressive Rollouts — laddered traffic weighting with automated metric analysis and abort conditions.
• Feature-Flag-Driven Release Management — decoupling deploy from release, cohort ramps, kill switches, and flag lifecycle hygiene.
• Automated Rollback Triggers & Runbook Integration — threshold configuration and incident-runbook wiring for hands-off recovery.
• Preview Environments & Environment Parity — validate deployment behaviour per-PR before it ever reaches a production rollout.

← Back to site index