Weighted Canary Routing with NGINX Ingress

The canary pods are running and healthy, and now something has to send them a slice of real traffic. With the NGINX ingress controller that is a second Ingress object and three annotations — no service mesh, no new component, and no change to the application.

When to use this pattern

  • You already run the NGINX ingress controller and do not want to introduce a mesh for this.
  • You need percentage-based traffic splitting plus a deterministic route for your own testers.
  • The canary decision loop — analyse, then raise or abort — is described in canary releases and progressive rollouts and you need the routing half.

Prerequisites

How the controller decides

Two Ingress objects claim the same host. The one annotated as a canary is not a separate route — the controller merges it into the primary’s configuration as a conditional upstream, evaluated per request in a fixed order.

Canary Match Precedence, Per Request An incoming request is checked against the canary-by-header annotation first, then the canary-by-cookie annotation, then the weight. A match at any stage routes to the canary service; no match falls through to the stable service. request 1 · header match x-canary: always 2 · cookie match sticky, set on first route 3 · weight 10 of every 100 canary service — new version stable service — everything unmatched Precedence is fixed: a header match beats a cookie, and a cookie beats the weight. That is what makes tester routing deterministic.

Complete working example

# k8s/ingress-stable.yaml — the production route, unchanged during a canary.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop-stable
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 10m
spec:
  ingressClassName: nginx
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: shop-stable, port: { number: 80 } }
# k8s/ingress-canary.yaml — same host, merged in as a conditional upstream.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"          # required; without it this REPLACES the route
    nginx.ingress.kubernetes.io/canary-weight: "10"     # 10% of unmatched traffic

    # Deterministic route for testers, evaluated before the weight.
    nginx.ingress.kubernetes.io/canary-by-header: "x-canary"
    nginx.ingress.kubernetes.io/canary-by-header-value: "always"

    # Pin a routed session so a user does not alternate between versions mid-visit.
    nginx.ingress.kubernetes.io/canary-by-cookie: "shop_canary"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "shop_canary_route"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "1800"
spec:
  ingressClassName: nginx
  rules:
    - host: shop.example.com                            # MUST match the stable host exactly
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: shop-canary, port: { number: 80 } }
#!/usr/bin/env bash
# scripts/canary-step.sh — raise the weight, or abort. Idempotent by design.
set -euo pipefail
NS=production
weight() { kubectl -n "$NS" annotate ingress shop-canary --overwrite \
  "nginx.ingress.kubernetes.io/canary-weight=$1"; }

case "${1:?usage: canary-step.sh <weight|abort|promote>}" in
  abort)
    weight 0                                    # stop sending traffic immediately…
    kubectl -n "$NS" delete ingress shop-canary # …then remove the route entirely
    ;;
  promote)
    # Point the STABLE service at the new version, then drop the canary ingress.
    # Order matters: deleting the canary first would send its share back to the old version.
    kubectl -n "$NS" patch service shop-stable \
      -p '{"spec":{"selector":{"app":"shop","version":"canary"}}}'
    kubectl -n "$NS" delete ingress shop-canary
    ;;
  *) weight "$1" ;;
esac

# Verify what the controller actually loaded, not what we asked for.
kubectl -n "$NS" get ingress shop-canary \
  -o jsonpath='{.metadata.annotations.nginx\.ingress\.kubernetes\.io/canary-weight}{"\n"}' 2>/dev/null || true

Step-by-step walkthrough

canary: "true" is not optional and not cosmetic. Without it, a second ingress for the same host is a conflicting route, and the controller resolves the conflict by oldest-creation-timestamp — which usually means your canary silently takes all traffic or none. The annotation is what turns it into a merged conditional.

The host must match character for character. A canary ingress on www.shop.example.com against a stable on shop.example.com is not a canary at all; it is a separate site that receives no traffic and reports healthy.

Header routing exists so you can test deterministically. Percentage routing is useless for verification — you cannot tell whether the response you got came from the canary. curl -H 'x-canary: always' always reaches the new version, which makes smoke testing a canary possible before opening it to real users.

Stickiness matters more for frontends than for APIs. A single page load fetches a document and then a dozen hashed assets. Without stickiness each of those is routed independently, so a user can get the new document and the old JavaScript bundle — which fails to load, because the asset names differ. Cookie affinity keeps a session on one version and removes that entire failure mode.

One Page Load, With and Without Session Affinity Without affinity, a single page load routes each request independently: the document comes from the canary while three hashed asset requests go to stable, which does not have those filenames, producing a failed load. With affinity, the cookie set on the first request keeps every subsequent request on the canary. NO AFFINITY — every request routed independently GET / → canary app.4f9c.js → stable vendor.a91f.js → stable main.77c3.css → canary 404 on two assets stable has different hashes The user sees a blank page. Nothing is unhealthy; the two versions simply disagree about filenames. COOKIE AFFINITY — first route pins the session GET / → canary sets shop_canary_route app.4f9c.js → canary vendor.a91f.js → canary page loads correctly Affinity matters far more for a bundled frontend than for an API, because a page load is a dozen correlated requests.

Promotion order is the one thing worth rehearsing. Repointing the stable service first means the canary’s share and the stable share are both serving the new version at the moment the canary ingress is deleted. Deleting the canary ingress first returns its 10% to the old version for as long as the promotion takes — a brief, confusing regression in the middle of a successful rollout.

Verification

# 1. The controller merged the canary rather than creating a competing route.
kubectl -n production exec deploy/ingress-nginx-controller -- \
  cat /etc/nginx/nginx.conf | grep -A3 'shop.example.com' | grep -i canary

# 2. Header routing is deterministic — every request hits the canary.
for _ in $(seq 5); do
  curl -s -H 'x-canary: always' https://shop.example.com/version.json | jq -r .release
done                                    # → the canary release, 5 times
# 3. Weight is approximately honoured over a sample.
for _ in $(seq 200); do curl -s https://shop.example.com/version.json | jq -r .release; done \
  | sort | uniq -c | sort -rn

Expected at weight 10:

    181 2026-07-30-c47b912   ← stable
     19 2026-07-31-8d31f0a   ← canary

A sample of 200 will not land exactly on 20; anything between roughly 12 and 30 is consistent with a 10% split. A result of 0 or 200 means the merge did not happen — check step 1 before adjusting anything else.

What the routing layer cannot tell you

Weighted routing is a traffic mechanism, not a safety mechanism, and the distinction matters because the annotations make it feel like more than it is.

The ingress will happily send 10% of users to a version that is returning 500s. Nothing in the routing layer reads metrics, and nothing aborts on its own — the abort is a human or an automation calling the script above. That means a canary with no analysis attached is simply a partial outage with extra steps, affecting a tenth of users for as long as nobody looks. The analysis loop, the metric thresholds and the automatic abort are the part that makes the rollout safe, and they live outside the ingress entirely: in automating canary analysis with Prometheus metrics, or in a controller as described in progressive canary rollouts with Argo Rollouts.

The second limit is statistical. At 10% of a service handling 40 requests per second, a canary window of five minutes gives roughly 1,200 canary requests — enough to detect a gross error-rate regression, nowhere near enough to detect a 0.2% increase in checkout failures. Either lengthen the window, raise the weight, or accept that small regressions will be found by the stable rollout rather than by the canary. Pretending otherwise is how a canary comes to be trusted for guarantees it never provided.

What a Canary Window Can Actually Detect Four configurations of weight and window duration with their resulting canary request volume and the smallest error-rate regression each can distinguish from noise: 1,200 requests detects about a five per cent regression, 6,000 detects two per cent, 24,000 detects one per cent, and 120,000 detects a fifth of a per cent. CONFIGURATION CANARY REQUESTS SMALLEST DETECTABLE 10% weight, 5-minute window 1,200 ~5% error-rate rise 10% weight, 25-minute window 6,000 ~2% 40% weight, 25-minute window 24,000 ~1% 40% weight, 2-hour window 120,000 ~0.2%

Common pitfalls

Omitting the canary annotation. The second ingress becomes a conflicting route rather than a weighted one, and the resolution is not the one you expect.

Leaving a canary ingress at weight 0 after an abort. It stops routing but stays in the controller’s configuration and in everyone’s mental model. Delete it; a zero-weight canary is the thing someone re-raises by accident next week.

Assuming assets follow the document. Without stickiness they do not, and a mixed page is the most confusing failure a canary can produce. The content-hashed naming described in versioning frontend build artifacts with content hashes prevents the wrong-bytes case, but the missing-asset case still needs affinity.

← Back to Canary Releases & Progressive Rollouts