Capturing Playwright Traces for Flaky Preview Failures

The test failed against the preview, passed on re-run, and there is nothing to look at. Preview-only failures are the hardest kind to diagnose because the environment is gone within the hour — so the evidence has to be captured during the failing run, not reconstructed afterwards.

When to use this pattern

  • Tests fail against preview environments and pass locally or on re-run.
  • Your current debugging loop starts with “re-run it with more logging”.
  • Preview environments are torn down before anyone can inspect them by hand.

Prerequisites

What a trace contains, and why it beats logs

A trace is not a log file. It is a recorded timeline of the browser session — every action, a DOM snapshot before and after each one, the network exchange, and the console — replayable in a viewer that lets you step through the failure.

What a Trace Records, Aligned on One Timeline Four parallel tracks over a nine-second test. The action track shows navigation, a click, a fill and a failing assertion. The snapshot track shows DOM captures at each action. The network track shows a request that returned 502 shortly before the assertion. The console track shows an unhandled rejection at the same moment. ACTIONS SNAPSHOTS NETWORK CONSOLE goto /checkout click Pay fill card expect(confirmation) ✗ timeout DOM captured before and after every action — inspectable, not a screenshot GET /api/cart 200 POST /api/pay → 502 Unhandled rejection: gateway 0s 4s 9s The 502 four seconds before the assertion is the answer — and it is invisible in a test log that only records the assertion.

Complete working example

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  // One retry absorbs genuine network transients against a shared preview.
  retries: process.env.CI ? 1 : 0,

  use: {
    baseURL: process.env.BASE_URL,
    // on-first-retry keeps the trace for a spec that failed then passed — the
    // flaky case, which retain-on-failure discards because the run ended green.
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
    // Record request and response bodies: the 502 above is only diagnosable with them.
    contextOptions: { recordHar: { mode: 'minimal', content: 'embed' } },
  },

  // Give every artifact a name that identifies which run produced it.
  outputDir: `test-results/${process.env.RUN_LABEL ?? 'local'}`,
  reporter: process.env.CI ? [['blob'], ['github']] : [['list']],
});
// tests/fixtures.ts — attach the preview's identity to every trace.
import { test as base } from '@playwright/test';

export const test = base.extend({
  page: async ({ page }, use, testInfo) => {
    // Annotations appear in the trace metadata and in the HTML report.
    testInfo.annotations.push(
      { type: 'preview-url', description: process.env.BASE_URL ?? 'unset' },
      { type: 'preview-commit', description: process.env.PREVIEW_SHA ?? 'unset' },
    );
    // Console errors are the single most useful correlate of a preview-only failure.
    const consoleErrors: string[] = [];
    page.on('console', (m) => m.type() === 'error' && consoleErrors.push(m.text()));
    page.on('pageerror', (e) => consoleErrors.push(`pageerror: ${e.message}`));

    await use(page);

    if (testInfo.status !== testInfo.expectedStatus && consoleErrors.length) {
      await testInfo.attach('console-errors', {
        body: consoleErrors.join('\n'), contentType: 'text/plain',
      });
    }
  },
});
# .github/workflows/e2e.yml — upload labelled, and only what is useful.
      - name: Run tests
        run: npx playwright test --shard=${{ matrix.shard }}/4
        env:
          BASE_URL:     ${{ needs.deploy-preview.outputs.url }}
          PREVIEW_SHA:  ${{ github.event.pull_request.head.sha }}
          RUN_LABEL:    pr${{ github.event.number }}-s${{ matrix.shard }}-a${{ github.run_attempt }}

      - uses: actions/upload-artifact@v4
        if: failure()          # only when something actually failed
        with:
          # The name is the whole findability story: pull request, shard, attempt.
          name: traces-pr${{ github.event.number }}-s${{ matrix.shard }}-a${{ github.run_attempt }}
          path: test-results/
          retention-days: 14   # long enough to outlive a review cycle
          compression-level: 9

Step-by-step walkthrough

on-first-retry, not retain-on-failure. This is the setting that matters for flakiness specifically. retain-on-failure keeps traces for specs that ended red; a spec that failed and then passed ends green and is classified flaky, so its evidence can be dropped. on-first-retry records the retry attempt precisely for that case — which is the case you are trying to diagnose.

Which Trace Mode Keeps Evidence for a Flaky Spec Three settings compared. With trace off, neither outcome produces evidence. With retain-on-failure, a hard failure keeps its trace but a spec that passed on retry does not. With on-first-retry, both outcomes preserve a trace of the failing attempt. spec fails outright fails, then passes on retry trace: 'off' no evidence no evidence trace: 'retain-on-failure' trace kept run ended green — trace discarded trace: 'on-first-retry' trace kept trace kept — the flaky case The top-right cell is where preview flakiness lives, and two of the three settings leave it empty.

Record the network with bodies. A 502 with an empty body tells you the request failed; a 502 with the gateway’s response tells you which upstream. On a preview environment this is often the entire diagnosis, because the failure is infrastructural rather than in the application.

Annotate with the preview identity. Traces are read days later, by which time the environment is gone and the pull request has moved on several commits. Recording the URL and commit inside the artifact means the trace is self-describing rather than requiring a reconstruction of which deploy it came from.

Capture console and page errors explicitly. Playwright’s trace records console output, but attaching the errors as a plain-text file means they are visible in the report summary without opening the viewer — often enough to identify the problem from the pull request page alone.

Label the upload with pull request, shard and attempt. Four shards times two attempts is eight artifacts named test-results; finding the right one becomes a chore, and chores get skipped. The name is the difference between a trace being used and a trace merely existing.

Verification

# 1. A failing run produced an artifact, and it contains a trace.
gh run download -n "traces-pr482-s2-a1" -D /tmp/tr
find /tmp/tr -name 'trace.zip' | head

# 2. Open it. This is the whole point — it replays locally, offline.
npx playwright show-trace /tmp/tr/**/trace.zip

# 3. The annotations survived, so the trace identifies its own environment.
unzip -p /tmp/tr/**/trace.zip test.trace 2>/dev/null | head -c 2000 | grep -o 'preview-[a-z]*'

Expected from the third command:

preview-url
preview-commit

If no artifact exists for a failed run, check that the upload step uses if: failure() rather than the default — the default is success(), which skips the upload in exactly the situation it was added for. That single mistake accounts for most “we have traces configured but there is never one to look at” reports.

Reading a preview failure in the right order

Traces make a failure diagnosable, but only if you look at the tracks in an order that separates the two possibilities: the application is broken, or the environment is.

Start with the network track, not the failing assertion. A non-2xx response, a request that never completed, or a request to an unexpected host tells you immediately that the problem is environmental — a missing configuration value, an unready dependency, or a sandbox credential that was never injected. Only when the network track is clean is the assertion itself worth studying, and then the DOM snapshot immediately before the failure usually shows the answer: an element rendered but hidden, a loading state that never resolved, a different copy than the selector expects.

The habit matters because the default instinct — read the assertion, then the code — leads to a long investigation of application logic for what turns out to be a preview that never received NEXT_PUBLIC_API_URL. Checking the environment first costs thirty seconds and resolves a large fraction of preview-only failures outright, and the ones it does not resolve are then genuinely about the code.

Triage Order, and Where Failures Actually Resolve Four triage steps in order. The network track resolves about forty per cent of preview failures, console errors a further twenty per cent, the DOM snapshot before the failure a further twenty-five per cent, and the remaining fifteen per cent require reading the assertion and application code. 1 · network track non-2xx? wrong host? resolves ~40% 2 · console errors missing config, CSP, CORS resolves ~20% 3 · DOM snapshot rendered but hidden? stuck? resolves ~25% 4 · the assertion genuinely the code ~15% Starting at step 4 — the instinctive order — means investigating application logic for a problem that is usually environmental. Steps 1 and 2 take under a minute combined and settle three preview failures in five.

Common pitfalls

trace: 'on'. Recording every spec produces gigabytes per run and slows the suite noticeably. Failure-scoped capture gives the same diagnostic value at a fraction of the cost.

Uploading with the default if condition. actions/upload-artifact runs only on success unless told otherwise, so the artifacts you most want are exactly the ones not uploaded. Use if: failure() or if: always().

One-day retention on failure artifacts. A trace from Friday afternoon is gone by Monday. Debug output generally deserves short retention, as covered in setting artifact retention policies — but failure traces are the documented exception, and two weeks is a reasonable value.

← Back to Running E2E Tests Against Preview Environments