Running Accessibility Checks Against Preview Deployments

Accessibility regressions are cheap to fix in the pull request that introduced them and expensive to fix six months later, when the offending pattern has been copied into fourteen components. A scan against the preview deployment puts the feedback at the point of change — and a baseline keeps it from blocking every merge on pre-existing debt.

When to use this pattern

  • You have preview deployments and no automated accessibility signal.
  • The codebase has existing violations, so a strict gate would block everything.
  • Contrast and labelling issues keep reaching production and being reported by users.

Prerequisites

Why the preview, and not a component test

Most accessibility violations are properties of a composed page, not of a component in isolation. Scanning the deployment is what makes them visible.

What Only a Full-Page Scan Can See Four violation types. A missing accessible name is detectable in a component scan. Colour contrast, duplicate element ids and heading order all depend on the surrounding page and are only detectable when the composed deployment is scanned. component scan preview page scan missing accessible name on a button detected detected contrast against the real background background unknown detected duplicate id across two components only one present detected heading order across the layout no surrounding headings detected Three of the four most common violation classes are invisible until the components are composed into a real page.

Complete working example

// tests/a11y.spec.ts — scan the deployed preview, diff against a baseline.
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
import { readFileSync, existsSync } from 'node:fs';

const ROUTES = ['/', '/products', '/products/example-item', '/cart', '/checkout', '/account'];

// A baseline is a set of violation fingerprints that already existed on main.
type Finding = { rule: string; selector: string; route: string; impact: string };
const key = (f: Finding) => `${f.route}::${f.rule}::${f.selector}`;
const baseline = new Set<string>(
  existsSync('a11y-baseline.json')
    ? JSON.parse(readFileSync('a11y-baseline.json', 'utf8')).map(key)
    : [],
);

for (const route of ROUTES) {
  test(`accessibility: ${route}`, async ({ page }, testInfo) => {
    await page.goto(route, { waitUntil: 'networkidle' });

    const results = await new AxeBuilder({ page })
      .withTags(['wcag2a', 'wcag2aa', 'wcag21aa'])   // the level you actually commit to
      // Disable nothing globally; exclude only third-party embeds you cannot change.
      .exclude('#intercom-container')
      .analyze();

    const found: Finding[] = results.violations.flatMap((v) =>
      v.nodes.map((n) => ({ rule: v.id, selector: n.target.join(' '), route, impact: v.impact ?? 'minor' })),
    );

    // Only NEW findings fail. Pre-existing ones are tracked, not blocking.
    const regressions = found.filter((f) => !baseline.has(key(f)));

    await testInfo.attach('a11y-findings', {
      body: JSON.stringify(found, null, 2), contentType: 'application/json',
    });

    expect(regressions, regressions.map((r) =>
      `${r.impact.toUpperCase()} ${r.rule} at ${r.selector}`).join('\n')).toEqual([]);
  });
}
# .github/workflows/a11y.yml
name: Accessibility
on: [pull_request]

permissions: { contents: read, pull-requests: write }

jobs:
  a11y:
    needs: deploy-preview
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci --ignore-scripts
      - run: npx playwright install --with-deps chromium

      - run: npx playwright test tests/a11y.spec.ts
        env:
          BASE_URL: ${{ needs.deploy-preview.outputs.url }}

      - if: always()
        uses: actions/upload-artifact@v4
        with: { name: a11y-findings, path: test-results/, retention-days: 14 }
# scripts/refresh-a11y-baseline.sh — run on main, never on a pull request.
set -euo pipefail
BASE_URL="https://www.example.com" npx playwright test tests/a11y.spec.ts || true
jq -s 'add | unique_by(.route + .rule + .selector)' \
  test-results/**/a11y-findings.json > a11y-baseline.json
echo "baseline now holds $(jq length a11y-baseline.json) known findings"

Step-by-step walkthrough

Scan the deployment, waiting for the network to settle. A scan that runs before client-side rendering completes reports violations for a page that never existed and misses the ones in the rendered result. waitUntil: 'networkidle' is the blunt but reliable version; a wait for a known post-render element is better where one exists.

Choose the tag set deliberately. wcag2a, wcag2aa and wcag21aa correspond to a commitment you can state. Including best-practice adds advisory rules that are often right but are not a standard, and mixing them means a failure no longer tells you whether you broke a commitment.

The baseline is what makes this adoptable. A codebase with 60 existing violations cannot turn on a strict gate — the first pull request would fail for reasons unrelated to itself, and the gate would be disabled within a week. Fingerprinting existing findings and failing only on new ones lets the check ship today and the debt shrink deliberately.

Baselined Findings Inform; New Findings Block A scan returns sixty-three findings. Sixty match the baseline and are reported without blocking. Three are new and fail the check, each named with its rule, selector and impact so the author can act immediately. scan result 63 findings 60 match the baseline reported, not blocking tracked as backlog, ordered by impact removed from the file as they are fixed 3 are new this pull request introduced them CRITICAL button-name at .modal > button check fails — named, located, fixable now The split is what makes the gate both immediately adoptable and genuinely preventive.

Fingerprint on route, rule and selector. Coarser keys (rule only) let a genuine new violation hide behind an old one elsewhere. Finer keys (including text content) break on every copy change, filling the baseline with noise.

Refresh the baseline only from the main branch. Regenerating it during a pull request would silently absorb that pull request’s own regressions — the baseline would grow to include exactly the violations the check exists to catch.

Verification

# 1. The check fails on a deliberate regression.
git apply <<'PATCH'
--- a/src/components/Button.tsx
+++ b/src/components/Button.tsx
-      <button aria-label="Close">×</button>
+      <button>×</button>
PATCH
BASE_URL="$PREVIEW_URL" npx playwright test tests/a11y.spec.ts; echo "exit=$?"

Expected — a named rule, a selector, and a non-zero exit:

CRITICAL button-name at .modal > button
exit=1
# 2. Pre-existing findings do not fail the run.
git checkout -- src/components/Button.tsx
BASE_URL="$PREVIEW_URL" npx playwright test tests/a11y.spec.ts; echo "exit=$?"   # → exit=0

# 3. The baseline is shrinking over time, not growing.
git log --format='%ad %H' --date=short -- a11y-baseline.json | head -5 | while read -r d h; do
  printf '%s %s\n' "$d" "$(git show "$h:a11y-baseline.json" | jq length)"
done

The third command is the one to keep an eye on. A baseline that grows means findings are being absorbed rather than fixed — usually because someone regenerated it from a branch rather than from main.

Making the debt shrink instead of settle

A baseline solves adoption and creates a new risk: a permanent list nobody ever revisits. Two practices keep it moving.

The first is ordering the baseline by impact and treating the critical and serious entries as ordinary backlog items with owners, rather than as an undifferentiated blob. Axe’s impact rating is a reasonable proxy for user harm, and a list of nine critical findings is a plausible sprint; a list of sixty mixed findings is not, and reads as unfixable.

The second is a ratchet: when a pull request happens to fix a baselined finding, remove it from the baseline in the same change. That way the file only ever shrinks through normal work, and nobody has to schedule an accessibility project to make progress. Pairing this with the review-time signal — the findings comment on the pull request — means the remaining human effort goes to what automation genuinely cannot see: keyboard order, focus management, and whether the labelling makes sense to someone who cannot see the layout.

Baseline Trajectory Under Two Policies Starting from sixty baselined findings, an absorb-on-regeneration policy grows the baseline to about ninety over six months. A ratchet policy, where fixes remove entries and regressions are blocked, shrinks it to about twenty-two over the same period. 0 30 60 90 findings absorbed on every regeneration ratchet: fixes remove, regressions blocked M1 M4 M6 Neither line requires a dedicated project — the difference is entirely in whether the baseline is allowed to absorb new findings.

Common pitfalls

Failing on the whole violation list rather than on regressions. The gate gets disabled the first week. Baseline first, ratchet after.

Scanning only the homepage. It is usually the most polished page in the product. Scan the routes where the interesting composition happens — forms, checkout, account settings.

Treating a passing scan as accessibility. Automated rules cover roughly a third of WCAG criteria. The scan buys back review time for keyboard and screen-reader testing; it does not replace it.

Re-run the baseline generation after any design-system upgrade. A change to shared component styles can resolve dozens of baselined findings at once, and leaving them in the file makes the remaining debt look larger than it is while hiding the improvement the upgrade actually delivered.

← Back to Running E2E Tests Against Preview Environments