Running E2E Tests Against Preview Environments

A preview environment that nothing tests is a screenshot: it proves the build compiled, not that the change works. Wiring an end-to-end suite to each per-pull-request deployment turns that screenshot into a gate — but only if the suite waits for the environment to be genuinely ready, addresses it by its actual URL, isolates its data from every other concurrent run, and leaves enough evidence behind that a failure can be diagnosed without re-running anything.

This guide covers those four problems in order, against the ephemeral deployments produced by automated preview deployments on pull requests and the per-branch addresses from ephemeral preview URL management and routing. The failure modes are all timing and isolation; almost none of them are the tests themselves.

Prerequisites

Why “Deployed” Is Not “Ready”

The single largest source of flaky preview tests is starting the suite when the platform says the deployment succeeded. That signal means the artifact was accepted and routing was updated. It says nothing about whether the origin has booted, whether migrations finished, whether the first render is still cold, or whether DNS for a freshly created wildcard hostname has propagated to the runner’s resolver.

The Gap Between "Deploy Succeeded" and "Environment Ready" A horizontal timeline from 0 to 90 seconds. The deploy job reports success at 8 seconds. DNS resolves at 22 seconds, the container finishes booting at 35 seconds, migrations complete at 48 seconds and the first render warms the cache at 61 seconds. A shaded false-start window spans 8 to 61 seconds, after which the readiness probe passes. false-start window — tests here fail for reasons unrelated to the change deploy job reports success 8s wildcard DNS resolves 22s container accepts traffic 35s migrations complete 48s readiness probe passes 61s start the E2E suite here A 200 from the edge during this window is a cached shell, not a working app.

The fix is a readiness endpoint that asserts something meaningful. A handler that returns 200 OK unconditionally is worse than no probe, because it converts an obvious failure into an intermittent one. A useful probe checks that the database connection pool answers a trivial query, that the migration table is at the expected revision, and that any mandatory downstream service responds — and returns 503 otherwise, so the poller keeps waiting instead of proceeding.

The second half of the readiness problem is the poller. It must distinguish “not ready yet” (keep waiting) from “will never be ready” (fail fast). A poll loop that treats connection refused, 404, and 503 identically will happily spend its entire ten-minute budget waiting for a deployment that failed, and then report a timeout that tells nobody anything. Fail immediately on a 404 at the origin, because that means routing is wrong rather than slow.

Step-by-Step Implementation

1. Resolve the preview URL from the deploy job

Never reconstruct the URL from the branch name in the test job. Branch names get sanitised — slashes become dashes, uppercase is folded, long names are truncated with a hash — and the reconstruction eventually disagrees with what the deploy actually produced. Pass the real value through a job output.

jobs:
  deploy-preview:
    runs-on: ubuntu-latest
    outputs:
      url: ${{ steps.deploy.outputs.preview-url }}     # single source of truth
    steps:
      - uses: actions/checkout@v4
      - id: deploy
        run: |
          url=$(./scripts/deploy-preview.sh)           # prints e.g. https://pr-482.preview.example.com
          echo "preview-url=$url" >> "$GITHUB_OUTPUT"

  e2e:
    needs: deploy-preview
    runs-on: ubuntu-latest
    env:
      BASE_URL: ${{ needs.deploy-preview.outputs.url }}
      RUN_ID: pr-${{ github.event.number }}-${{ github.run_attempt }}

Including run_attempt in the namespace matters: a re-run of a failed job must not inherit records the previous attempt created and left behind.

2. Gate on real readiness

#!/usr/bin/env bash
# scripts/wait-for-ready.sh — poll until the app is genuinely serving, or fail with a reason.
set -euo pipefail
URL="${1:?usage: wait-for-ready.sh <base-url>}"
DEADLINE=$(( SECONDS + 300 ))     # 5 minutes; longer hides real provisioning regressions

while (( SECONDS < DEADLINE )); do
  code=$(curl -s -o /tmp/ready.json -w '%{http_code}' --max-time 10 "$URL/api/ready" || echo 000)
  case "$code" in
    200) jq -e '.database == "ok" and .migrations == "current"' /tmp/ready.json >/dev/null \
           && { echo "ready after ${SECONDS}s"; exit 0; } ;;
    404) echo "FATAL: /api/ready returned 404 — routing is wrong, not slow"; exit 1 ;;
    000|502|503) : ;;             # still starting: keep polling
    *)   echo "unexpected status $code"; ;;
  esac
  sleep 5
done

echo "FATAL: not ready after 300s — last payload:"; cat /tmp/ready.json; exit 1

Printing the last payload on timeout is what turns a “preview never came up” ticket into a one-line diagnosis. The deadline deserves the same care. Five minutes is deliberately tight: a preview that regularly needs eight will keep needing more as the application grows, and a generous timeout converts a provisioning regression into a slow, invisible tax on every pull request. Treat repeated timeouts as a signal to profile the provisioning path — usually a migration that has grown, or an image pull that is no longer cached on the runner — rather than as a reason to raise the number.

One more detail separates a readiness probe that works from one that merely looks like it does: the probe must be served by the same origin the tests will hit, not by an edge or a load balancer that answers on the application’s behalf. A CDN that serves a cached 200 for /api/ready while the origin is down will pass the gate every time.

3. Point the suite at the resolved URL

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  // Falls back to local dev so the same config works on a laptop.
  use: {
    baseURL: process.env.BASE_URL ?? 'http://localhost:3000',
    trace: 'retain-on-failure',        // full timeline, only for the specs that failed
    video: 'retain-on-failure',
    screenshot: 'only-on-failure',
  },
  // A preview environment is remote and shared: retry once for transient network faults,
  // never more, or genuine flakiness stops being visible.
  retries: process.env.CI ? 1 : 0,
  timeout: 45_000,
  expect: { timeout: 10_000 },         // remote round trips are slower than local ones
  reporter: [['github'], ['html', { open: 'never' }]],
});

4. Isolate every record the suite creates

Two pull requests running the same checkout spec against the same preview database will collide on a unique email, a cart identifier, or a slug. Namespacing is cheap and survives shared infrastructure.

// tests/fixtures.ts — every created entity carries the run namespace.
import { test as base } from '@playwright/test';

const RUN = process.env.RUN_ID ?? 'local';

export const test = base.extend<{ ns: (name: string) => string }>({
  ns: async ({}, use) => {
    // "checkout-user" → "[email protected]"
    await use((name) => `${name}+${RUN}@e2e.example.com`);
  },
});

Pair it with a teardown that deletes by namespace prefix rather than by a hard-coded list, so a spec that adds a new entity type does not silently start leaking rows. Prefix teardown also degrades safely: if the job is cancelled before teardown runs, the orphaned rows are still identifiable by run identifier, and a scheduled sweep can remove anything whose pull request has closed. A teardown that enumerates entity types by hand fails in the opposite direction — it appears to work, while every newly added entity accumulates forever until the preview database is large enough to slow the seeding step that all the other tests depend on.

Run-Scoped Data Namespacing in a Shared Preview Database Two test runs, pr-482 and pr-509, each write user, cart and order records prefixed with their own run identifier into one shared database. A teardown step for each run deletes only rows matching its own prefix, leaving the other run untouched. run pr-482-1 checkout spec run pr-509-1 checkout spec shared preview database user checkout-user+pr-482-1@… cart cart-pr-482-1-a91f user checkout-user+pr-509-1@… cart cart-pr-509-1-77c3 teardown 482 DELETE … LIKE '%pr-482-1%' teardown 509 DELETE … LIKE '%pr-509-1%' Disjoint by construction — neither run can see, mutate, or delete the other's rows.

Configuration Reference

Option Type Default Effect
baseURL string none Root every relative navigation resolves against; set from the deploy job output
trace enum off retain-on-failure keeps a full timeline for failed specs only
retries integer 0 One retry absorbs transient network faults; more hides genuine flakiness
expect.timeout ms 5000 Per-assertion wait; remote previews need roughly double the local value
workers integer CPU count Concurrent specs — also concurrent writers against the shared preview data
--max-time (curl) seconds none Caps a single readiness probe so one hung request cannot consume the budget
RUN_ID string none Namespace prefix that keeps concurrent runs’ records disjoint

Integration with Adjacent Topics

Parity gates run before the suite, not instead of it. Environment parity validation gates catch structural divergence — a missing config key, a schema behind production — before the environment starts. E2E tests catch behavioural failures afterwards. Running only one of the two leaves a predictable gap: parity alone never executes the app, and tests alone report a config bug as a mysterious 500.

Secrets reach the suite the same way they reach the app. The test job needs sandbox credentials for third-party services, and those should arrive through the same short-lived mechanism described in secrets injection for preview environments rather than as long-lived repository secrets copied into a second place.

Teardown must not race the suite. The cost controls in automating preview environment teardown will happily destroy an environment while tests are still running against it if the idle timer counts only human traffic. Make the test job hold a lease, or set the idle threshold above the suite’s worst-case duration.

Performance and Cost Impact

The dominant cost is not the tests — it is waiting. On a representative pull request, a full-suite gate spends 61 seconds waiting for readiness, 14 minutes running every spec, and 40 seconds uploading artifacts, for roughly 16 minutes of blocking feedback. Splitting the suite into a critical-path subset for pull requests and a full run on merge cuts the blocking path to about 5 minutes while keeping total coverage identical.

Deciding which specs belong in the blocking subset is a routing question, not a judgement call, and it is worth writing the rule down so it survives the next person who adds a spec.

Routing a Spec to the Right Trigger A decision tree. First question: does the spec cover a revenue or authentication path? If yes and it runs under ninety seconds it joins the pull-request gate; if yes but slower it joins the merge run. If no, a spec depending on a third-party sandbox goes nightly, and everything else goes to the merge run. Revenue or auth path? yes no Under 90s? median wall-clock Third-party sandbox? yes no yes no Pull-request gate — blocking target: under 5 minutes total Merge run — non-blocking fails the main branch, alerts the author Nightly full suite sandbox quotas reset between runs Write the rule into the spec tags, not a wiki page — an untagged spec should default to the merge run.

Two numbers are worth tracking per repository. The first is readiness wait as a share of the job: if it exceeds 20%, the provisioning path is the problem, not the tests, and no amount of test tuning will help. The second is the retry rate: a critical-path suite retrying more than about 2% of specs is telling you the isolation or the readiness gate is incomplete, and that number climbs quietly until a genuine regression is dismissed as “the usual flake”. Combining this with the shard mechanics in parallelizing test suites in CI brings the critical-path run under three minutes for most suites, which is the threshold below which developers stop context-switching while they wait.

Troubleshooting

net::ERR_NAME_NOT_RESOLVED on the first spec, passing on retry. Wildcard DNS for the freshly created hostname had not propagated to the runner’s resolver when the suite started. Add a resolution check to the readiness script — getent hosts "$host" in a loop before the first HTTP probe — so the wait happens once, in the poller, rather than as a random first-spec failure.

Tests pass locally against the preview URL but fail in CI. Almost always a missing sandbox credential in the CI job’s environment, causing a third-party widget to fail to load and a selector to time out. Compare the two environments’ variable keys, not values, with the drift check from blocking deploys on config drift with a parity gate.

expect(locator).toBeVisible() timed out only on the first run after a deploy. The origin was cold and the first render exceeded the assertion timeout. Warm it in the readiness script with a single request to the heaviest route before declaring ready, rather than raising every timeout in the suite.

Duplicate-key errors that appear only when two pull requests are open. Data isolation is incomplete: some entity is still created with a fixed identifier. Grep the specs for literal emails, slugs, and identifiers, and route all of them through the namespace helper.

Frequently Asked Questions

Why do E2E tests fail immediately after the deploy job reports success?

Because “deployed” means the platform accepted the artifact and updated routing, not that the application is serving. During the gap the edge can return 200 for a cached shell while the origin is still booting and migrations are still applying. Poll a readiness endpoint that asserts dependency health, and only then start the suite.

Should the whole E2E suite run on every pull request?

No. Run a critical-path subset — authentication, the primary conversion flow, and anything the diff touches — as the required check, and the full suite on merge to the main branch plus a nightly schedule. A twenty-minute required check gets bypassed within a month, and a bypassed check protects nothing.

How should test data be isolated between concurrent preview runs?

Namespace every created record with the run identifier, including the run attempt, and tear down by prefix match. Namespacing works on shared databases and costs nothing at provision time; per-run database creation is cleaner in theory but adds a minute or more to every pull request.

Is it safe to run tests against a preview that shares production services?

Only for genuinely read-only dependencies. Anything the suite can write to — payments, transactional email, analytics, outbound webhooks — must point at a sandbox, or a test run will send real messages to real people and skew production dashboards. Enforce it with a parity check on the outbound endpoint configuration, not a team convention.

← Back to Preview Environments & Environment Parity