Feature-Flag-Driven Release Management

The operational pain here is that deployment and release are usually the same event: the moment you deploy, every user is exposed, so a risky feature and a routine bug fix carry the same all-or-nothing risk. Feature flags break that coupling — code ships to production dark, and a runtime toggle decides who sees it, when, and for how long. This page covers how to run releases as independent, reversible flag operations: cohort ramps, instant kill switches, safe evaluation under flag-service failure, and the lifecycle discipline that keeps flags from rotting into production liabilities.

Prerequisites

How Flag Evaluation Works Under the Hood

A feature flag is a runtime decision function: given an evaluation context (user ID, plan tier, region), the flag service returns a variant. The application branches on that variant. Because the branch lives in code that is already deployed to everyone, changing the release is just changing the flag rule — no pipeline runs, no traffic shift.

Four properties make this production-safe:

  1. Ship dark. New code merges and deploys behind a flag defaulted to off in production. The deploy is invisible; the deployment already succeeded before anyone decides to release.
  2. Ramp by cohort. The flag rule grows the exposed set — 1% of users, then internal staff, then a region, then everyone — while metrics for each cohort are compared to the control group.
  3. Kill switch. Flipping the flag off is a sub-second, infrastructure-free rollback. It is faster than any blue-green flip or canary abort because nothing redeploys.
  4. Resilient evaluation. The SDK caches the last-known ruleset locally. If the flag service is unreachable, evaluation falls back to the cached value or a hard-coded default — always the stable path — so a flag outage degrades to safe, not broken.
Deploy-Release Decoupling with Feature Flags A single deployed binary holds both a new path and a stable path. A request carries an evaluation context to a flag evaluator, which consults a local cache backed by the flag service and returns a variant; enabled users take the new path, everyone else takes the stable path. Request Flag Evaluator context → variant Flag Service local cache fallback New Path (enabled) ramped cohort Stable Path (default) everyone else · fallback one deployed binary — release ≠ deploy

Step-by-Step Implementation

Step 1 — Wrap the new path in a flag

Use OpenFeature so the code is provider-agnostic. The default must be the stable path.

// checkout.ts
import { OpenFeature } from "@openfeature/server-sdk";

const client = OpenFeature.getClient();

export async function handleCheckout(req: Request, user: User) {
  // Default `false` → stable path. Shipping this line does NOT release the feature.
  const useNewFlow = await client.getBooleanValue("new-checkout-flow", false, {
    targetingKey: user.id,
    plan: user.plan,
    region: user.region,
  });

  return useNewFlow ? newCheckout(req, user) : legacyCheckout(req, user);
}

Verification: Deploy with the flag off and confirm 100% of traffic still takes legacyCheckout — the deploy released nothing.

Step 2 — Ramp by cohort

Grow exposure in the flag service, not in code. A typical ladder: internal users → 1% → 10% → 50% → 100%.

// flag rule (managed in the flag service UI or as config)
{
  "key": "new-checkout-flow",
  "defaultVariant": "off",
  "rules": [
    { "if": { "attribute": "email", "endsWith": "@acme.com" }, "serve": "on" },
    { "rollout": { "on": 10, "off": 90 }, "bucketBy": "targetingKey" }
  ]
}

Verification: Query metrics split by flag variant; the on cohort’s conversion and error rate should track the off control within tolerance before you widen the rollout.

Step 3 — Guarantee a safe fallback

Configure the SDK so a flag-service outage returns cached or default values, never an error.

OpenFeature.setProvider(
  new FlagProvider({
    cacheTtlSeconds: 60,        // serve cached rules for up to 60s
    staleWhileRevalidate: true, // keep serving stale rules if refresh fails
    onError: () => "off",       // last resort: stable path
  })
);

Verification: Block the flag service at the network level in staging and confirm requests still succeed on the stable path with no 5xx.

Step 4 — Enforce flag lifecycle in CI

Record each flag’s TTL and fail the build when one overstays.

#!/usr/bin/env bash
# scripts/flag-ttl-check.sh — run in CI
set -euo pipefail
NOW=$(date +%s)
fail=0
# flags.json: [{ "key": "...", "created": <epoch>, "ttlDays": 30 }]
jq -c '.[]' flags.json | while read -r flag; do
  key=$(jq -r '.key' <<<"$flag")
  created=$(jq -r '.created' <<<"$flag")
  ttl=$(jq -r '.ttlDays' <<<"$flag")
  age=$(( (NOW - created) / 86400 ))
  if (( age > ttl )); then
    echo "FLAG DEBT: '$key' is ${age}d old (TTL ${ttl}d) — remove it or extend TTL with justification"
    fail=1
  fi
done
exit $fail

Verification: Add a flag with a past-due created timestamp and confirm CI fails with the FLAG DEBT message.

Three Places a Flag Can Be Evaluated Evaluating in the client SDK from a cached ruleset adds no latency but can serve a stale value for up to the poll interval. Evaluating at the edge adds a few milliseconds and is consistent within the edge's cache. Evaluating server-side per request is always current and adds a network hop unless the ruleset is local. client SDK, cached rules 0 ms — but up to one poll interval out of date at the edge a few ms, consistent within each edge location server, per request always current — the only place a kill switch is trustworthy A kill switch evaluated from a cached client ruleset is not a kill switch — it is a request to stop, eventually.

Configuration Reference

Option Type Default Effect
Default variant enum off The value served with no matching rule; must be the stable path in production
bucketBy string targetingKey Attribute the percentage rollout hashes on; keep stable so a user’s bucket does not change
cacheTtlSeconds integer 30–60 How long cached rules are served before refresh; higher survives longer outages
staleWhileRevalidate boolean true Serve stale rules when refresh fails instead of erroring
ttlDays (flag registry) integer 30 Age after which CI flags the toggle as debt
Evaluation location enum server Server-side for backend logic; client-side for UI — never gate security on client flags

Integration with Upstream and Downstream Topics

Feature flags layer on top of the infrastructure deployment strategies:

  • Complementary — canary. A flag ramp can run inside a canary rollout: the binary is canaried per-request while a feature is flag-ramped per-user, two independent control loops.
  • Complementary — blue-green. After a blue-green cutover, flags let you release features from the freshly live version gradually rather than all at once.
  • Downstream — rollback. A flag kill switch must be part of the rollback runbook; an infra rollback that leaves a flag on can re-expose a broken feature.
  • Upstream — config parity. Flag defaults and rulesets are configuration; keep them consistent across stages using the same discipline as synchronizing environment variables across stages.

Performance and Cost Impact

Metric Value Notes
Flag evaluation latency < 1 ms Local cache hit; no network call on the hot path
Kill-switch propagation 1–60 s Bounded by cacheTtlSeconds; lower for faster kills, higher for outage resilience
Rollback speed (flag off) sub-second at origin Faster than any infrastructure rollback
Flag-service outage impact 0 (safe fallback) Requests continue on the stable path
Cost of flag debt compounding Each stale flag multiplies untested branch combinations and test matrix size

The real cost of flags is not runtime — it is debt. A codebase with dozens of long-lived flags has an exponential number of untested path combinations. The lifecycle check in Step 4 is the single most important control.

What Happens When the Flag Service Is Unreachable Failing open serves the new behaviour when the service cannot be reached, which turns a flag-service outage into an uncontrolled release. Failing closed serves the previous behaviour, which is almost always the safer default and the one a kill switch requires. fail open — serve the new path a flag-service outage becomes a full release and the kill switch stops working at the same moment fail closed — serve the old path an outage degrades to known-good behaviour the safe default for almost every flag Set the default in code, not in the service: a default fetched from the service is not a default at all.

Troubleshooting

Symptom: Feature appears for users outside the target cohort

Cause: The percentage rollout buckets on an unstable attribute (session ID that rotates, or request timestamp), so a user’s bucket changes between requests.

Fix: Bucket on a stable targetingKey such as a persistent user ID. Confirm the same user consistently resolves to the same variant across sessions.

Symptom: Requests error when the flag service is down

Cause: The SDK is configured to throw on evaluation failure instead of returning a default.

Fix: Set an onError default of the stable variant and enable staleWhileRevalidate (Step 3). Flag evaluation must be non-throwing on the request path.

Symptom: A rolled-back deploy still shows the broken feature

Cause: Infrastructure rolled back but the flag stayed enabled, so the reverted binary still exposed the flagged path.

Fix: Add the flag kill switch to the automated rollback action so flipping the flag off happens first, before or alongside the traffic shift. Coordinate this in the rollback runbook.

Symptom: CI green but production has 40 stale flags

Cause: No lifecycle enforcement — flags are created but never removed, and nothing fails the build.

Fix: Adopt the flag-ttl-check.sh gate (Step 4) as a required status check. Treat flag removal as the definition-of-done for a rollout, not optional cleanup.

Frequently Asked Questions

What is the difference between a feature flag and a canary release?

A canary shifts infrastructure traffic between two binaries at the request level; a feature flag toggles a code path inside a single binary at the feature level per user. Canary answers “is this build healthy?”; flags answer “who should see this feature?”. They compose cleanly — you can run a per-user flag ramp inside a per-request canaried binary, giving you two independent, reversible control loops.

How do I prevent feature flag debt?

Give every temporary flag a TTL and a named owner in a registry, and fail CI when a flag exceeds its TTL without justification. Treat flag removal — deleting the toggle and the now-dead branch — as the final, required step of a rollout, not optional cleanup. Long-lived flags multiply untested path combinations; a dozen boolean flags is 4096 theoretical states, most never exercised, and that is where latent incidents hide.

What happens if the flag service is unavailable?

Nothing user-visible, if configured correctly. The SDK serves the last-known ruleset from a local cache (staleWhileRevalidate) and, if it has no cached value, returns a hard-coded default that is always the stable path. Evaluation must never throw on the request path and must never default a half-finished feature to on. Test this explicitly by blocking the flag service in staging.

Do feature flags replace the need for schema migrations to be backward compatible?

No — they raise the stakes. Because a flag keeps old and new code paths alive at the same time for the flag’s entire lifetime, which may be days or weeks, the database schema must remain compatible with both paths that whole time. Use expand-then-contract migrations and do not drop or rename anything the stable path still reads until the flag is fully rolled out and removed.

Governing Flags So They Stay an Asset

Flags start as a way to reduce release risk and become, without governance, the largest source of unexplained behaviour in a codebase. Three rules keep them useful.

Every flag declares an owner and an expiry at creation. Not in a document — in the same place the flag is defined, enforced by a check that fails a pull request adding a release flag without both. An expiry is a statement that this flag is temporary, and the absence of one is a statement that it is not, which is a decision worth making deliberately.

Kill switches are a separate, permanent category. A flag that exists to disable a subsystem in an emergency is not debt and should never be proposed for removal. Tagging it as such removes an entire class of false positive from the cleanup automation and stops people learning to ignore its suggestions.

Removal is automated up to the pull request. Detecting that a flag has served one value to everyone for longer than the rollback window is mechanical; deciding that the losing branch is genuinely dead is not. Generating the removal pull request automatically and requiring a human to merge it puts the effort exactly where judgement is needed.

The cost that justifies the effort

The argument for governance is easier to make with the second-order cost named. Each live flag doubles the notional paths through the code it guards, and flags compose — three overlapping flags in one flow describe eight behaviours, of which the test suite probably exercises two. That is where the real expense sits: not in the conditional, but in the fact that nobody can any longer state what the code does without enumerating flag states.

It also degrades incident response directly. The first question during an incident is what changed, and a flag toggled by someone in another team three weeks ago is a change with no commit, no review, and no entry in the deploy history that the rollback automation consults. Recording flag changes into the same timeline as deploys is a small piece of work that repays itself the first time it is needed.

One last operational detail: set each flag’s default in code rather than fetching it from the service. When the flag service is unreachable, a default that lives in the service is not a default at all, and the failure mode — serving the new path because nothing said otherwise — turns a provider outage into an uncontrolled release.

← Back to Deployment Strategies & Progressive Delivery