Deployment Strategies & Progressive Delivery

Progressive delivery replaces the all-at-once production push with a controlled, observable, reversible rollout — new code reaches users in slices that can be halted or reversed the instant a metric moves the wrong way. This guide is for platform engineers and release owners who need to ship frontend and full-stack applications frequently without trading away production stability.

Architecture Overview

Every progressive delivery strategy is a variation on the same control loop: deploy the new version without exposing it, verify it against health signals, shift traffic incrementally, and promote or roll back based on what the metrics say. The diagram below shows that loop end to end.

The three infrastructure strategies — blue-green, canary, and rolling — differ only in how the traffic slice is shifted. Feature-flag-driven release management sits on top of all of them, moving the exposure decision from infrastructure into application code.

Core Concepts

Pattern 1 — Atomic Blue-Green Cutover (Foundational)

When to use it: Any service where you can briefly run two versions and need an instant, reversible cutover. This is the baseline production-grade strategy and the simplest to reason about.

Blue-green keeps two environments — one live (blue), one idle (green). You deploy to the idle one, validate it in isolation, then flip the router. Because the previous version stays warm, rollback is a second flip, not a redeploy.

# .github/workflows/blue-green.yml — cutover job (abridged)
name: Blue-Green Deploy
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Determine idle color
        id: color
        run: |
          # Read which color currently serves live traffic from the router config
          LIVE=$(kubectl get service app-router -o jsonpath='{.spec.selector.color}')
          IDLE=$([ "$LIVE" = "blue" ] && echo green || echo blue)
          echo "idle=$IDLE" >> "$GITHUB_OUTPUT"

      - name: Deploy to idle environment
        run: kubectl set image deploy/app-${{ steps.color.outputs.idle }} app=ghcr.io/${{ github.repository }}:${{ github.sha }}

      - name: Smoke test idle before cutover
        run: |
          # Hit the idle environment directly, bypassing the live router
          kubectl rollout status deploy/app-${{ steps.color.outputs.idle }} --timeout=120s
          curl -sf "http://app-${{ steps.color.outputs.idle }}.internal/health" | jq -e '.status == "ok"'

      - name: Flip the router (atomic cutover)
        run: |
          # A single patch redirects 100% of traffic — the old color stays warm for rollback
          kubectl patch service app-router -p \
            '{"spec":{"selector":{"color":"${{ steps.color.outputs.idle }}"}}}'

Common mis-configurations:

• Deleting or scaling down the old environment immediately after cutover — you lose the instant rollback path. Keep it warm for at least one analysis window.
• Smoke-testing through the live router instead of hitting the idle environment directly — the test then validates the old version, not the new one.
• Forgetting database backward compatibility — the two versions share one database, so schema changes must be additive (expand-then-contract), or the flip breaks the version still serving traffic.

For static frontends, the same cutover is achieved with a symlink swap rather than a router patch — see atomic symlink deploys for static frontends.

Pattern 2 — Metric-Gated Canary (Intermediate)

When to use it: High-traffic services where even a brief exposure of a bad version to 100% of users is unacceptable, and you have enough traffic for statistically meaningful analysis within minutes.

A canary routes a small slice — typically 5% — to the new version, holds for an analysis window, and widens only if the new version’s metrics match the stable baseline. The rollout is a step function guarded at every step.

# Argo Rollouts canary spec — the traffic ladder with automated analysis
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: app
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5          # 5% of traffic to the new version
        - pause: { duration: 5m }   # analysis window
        - analysis:             # automated metric check — aborts on failure
            templates:
              - templateName: error-rate-and-latency
        - setWeight: 25
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100        # full promotion

The analysis step is the heart of the pattern: it queries your metrics backend and fails the rollout automatically if the new version regresses. See automating canary analysis with Prometheus metrics for the full AnalysisTemplate.

Common mis-configurations:

• Analysis windows too short to accumulate signal — a 30-second window on a low-traffic endpoint sees too few requests to detect a 1% error-rate regression. Size the window to guarantee a minimum request count, not a fixed clock time.
• Comparing the canary against absolute thresholds instead of the live baseline — a 0.5% error rate might be normal for your app; compare canary-vs-stable, not canary-vs-zero.
• Routing canary traffic by round-robin without session affinity — a user bouncing between versions on every request sees inconsistent behaviour.

Pattern 3 — Progressive Delivery with Release Control (Advanced / Production-Grade)

When to use it: Large teams shipping many changes per day, where deployment frequency must be decoupled from release risk and individual features need independent kill switches.

At scale, the infrastructure rollout and the feature release are separated entirely. Code ships to production continuously (often dark), and a feature flag governs exposure. This lets you deploy a canary of the binary while simultaneously running a percentage rollout of a feature inside it — two independent control loops.

// Release control lives in application code, not the pipeline.
// The deploy already happened; this decides who sees the feature.
import { flags } from "./flags";

export async function checkoutHandler(req: Request, user: User) {
  // Deploy and release are independent: this code is live for everyone,
  // but the new flow is gated to a growing cohort with an instant kill switch.
  if (await flags.isEnabled("new-checkout-flow", { userId: user.id })) {
    return newCheckout(req, user);   // ramped 1% → 50% → 100% via the flag service
  }
  return legacyCheckout(req, user);  // stable path — the automatic fallback
}

Scale and failure modes at this level:

• Flag debt. Every temporary flag is a branch in production that must be removed after rollout. Track flag age and fail CI when a “temporary” flag exceeds its TTL — otherwise the codebase accretes dead toggles that eventually cause an incident.
• Long-lived schema skew. With deploy and release decoupled, old and new code paths coexist for days. Database migrations must stay backward-compatible across the entire flag lifetime, not just the deploy window.
• Flag-service availability. The flag evaluation service becomes a production dependency. Cache flag state locally with a stale-but-available fallback so a flag-service outage defaults to the stable path rather than erroring.

Environment & Toolchain Matrix

Which tools you already run for pipeline architecture largely dictate this choice: teams on a service mesh get canary weighting for free, while teams on managed static hosting lean on atomic deploys and edge routing.

Cost & Performance Trade-offs

The dominant cost lever is the duration of the doubled-capacity window in blue-green. Retire the old environment as soon as the analysis window closes — but not before — and use self-hosted runners to keep the deploy pipeline itself off billable minutes.

Failure Modes & Remediation

1. Cutover flips before the new environment is ready

Root cause: The router is patched before pods pass readiness, so traffic hits a version that is still starting. Symptom: a spike of connection-refused or 502 errors at the exact moment of cutover.

Fix: Gate the flip on kubectl rollout status completing and a direct health check succeeding against the idle environment, as in Pattern 1. Never flip on deploy-submitted; flip on ready-and-verified.

2. Canary looks healthy but the baseline is unhealthy

Root cause: The analysis compares the canary against a stable version that is itself degraded, so a regression passes the gate because “both are equally bad.”

Fix: Add absolute floor guardrails alongside the relative comparison — e.g. abort if canary error rate exceeds 2% regardless of baseline. Relative comparison catches regressions; absolute floors catch shared outages.

3. Rollback fails because the old version was already scaled to zero

Root cause: A cost-optimization step scaled down the previous version immediately after cutover, so there is nothing warm to roll back to.

Fix: Keep the previous version at minimum viable replicas until the analysis window closes. Encode this as a policy in the pipeline, not a manual step. Full runbook in automated rollback triggers and runbook integration.

4. Schema migration breaks the version still serving traffic

Root cause: A destructive migration (dropped column, renamed table) ran during deploy while the old version — still live during blue-green validation — expected the old schema.

Fix: Use expand-then-contract migrations: add the new column, deploy code that writes both, backfill, switch reads, and only drop the old column in a later release once no running version references it.

5. Feature flag left on for a cohort after rollback

Root cause: Infrastructure rolled back but the feature flag stayed enabled, so the reverted binary still exposed a half-finished feature via a flag it no longer fully supported.

Fix: Treat the flag kill switch as part of the rollback runbook. Automated rollbacks that touch flagged features must flip the flag off as a first action, before or alongside the traffic shift.

Frequently Asked Questions

Sequencing Adoption

Progressive delivery is usually adopted in the wrong order, which is why teams end up with a canary they cannot trust or a rollback that fails when used. The dependencies between the pieces are real, and following them saves rework.

First, make deploys reversible. Before any traffic-shifting exists, a deploy must be something you can put back. That means immutable, addressable artifacts, a recorded history of what was deployed where, and a rollback that re-points at an existing artifact rather than rebuilding from source. Nothing later works without it: a canary that detects a problem and cannot revert is a slower way to have an outage.

Second, make schema changes non-blocking. Two application versions must be able to run against one database, because every strategy above rolling depends on it. This is the step most often skipped, and skipping it produces the worst failure available — a rollback that succeeds mechanically and leaves the application broken.

Third, get per-version observability. Metrics labelled by release, so the canary’s error rate is distinguishable from the stable version’s over the same window. Without this, canary analysis compares the new version against yesterday and fires on Monday-morning traffic.

Fourth, add traffic shifting. Weighted routing is the easy part and the part teams reach for first. It is genuinely straightforward once the three prerequisites exist, and near-useless before.

Fifth, automate the decision. A canary evaluated by a human who checks a dashboard when they remember is a manual process with extra steps. Automating promote-or-abort is what converts the mechanism into a control.

What to expect at each stage

The first two steps produce no visible improvement in release speed, which makes them politically harder than the ones that follow. What they produce is a change in the cost of being wrong: an incident that previously required a restore now requires a five-minute revert. That is worth measuring and stating, because otherwise the work reads as infrastructure for its own sake.

The last three compound. Once per-version metrics and weighted routing both exist, adding automated analysis is a small increment, and the release cadence can rise substantially because each release carries less risk. Teams commonly find that the constraint moves from deployment risk to review throughput at this point, which is a better problem to have and a signal that the sequence is complete.

Whichever combination you adopt, the prerequisite is the same: two application versions must tolerate one set of backing services simultaneously. That property is what every strategy above rolling depends on, and it is bought once — by sequencing schema changes across releases — rather than per strategy.

None of this is adopted in an afternoon, and the order matters more than the pace. Reversible deploys first, backward-compatible schema changes second, per-version observability third, traffic shifting fourth, automated decisions last.

Related

• Blue-Green Deployments for Full-Stack Apps — two-environment cutover mechanics, database compatibility, and warm-standby rollback.
• Canary Releases & Progressive Rollouts — laddered traffic weighting with automated metric analysis and abort conditions.
• Feature-Flag-Driven Release Management — decoupling deploy from release, cohort ramps, kill switches, and flag lifecycle hygiene.
• Automated Rollback Triggers & Runbook Integration — threshold configuration and incident-runbook wiring for hands-off recovery.
• Preview Environments & Environment Parity — validate deployment behaviour per-PR before it ever reaches a production rollout.

← Back to site index