Parallelizing Test Suites in CI

A frontend test suite grows until it becomes the pipeline’s critical path, and by the time anyone measures it, a pull request costs eighteen minutes of which fifteen are tests. Sharding fixes the wall-clock problem — but only if the split is balanced, the reports are merged, and the shard count stops at the point where per-job overhead eats the gains.

This guide covers how shard assignment actually works, how to balance splits on recorded durations rather than file counts, how to merge coverage so quality gates still see the whole picture, and how to pick a shard count from your own numbers rather than a round one. It pairs with incremental builds and affected detection, which reduces how much you test; sharding reduces how long testing what remains takes.

Prerequisites

• A test runner with native shard support: Playwright 1.40+, Jest 29+, or Vitest 1.0+
• CI with a job matrix (GitHub Actions strategy.matrix, GitLab parallel)
• Dependency caching already in place, or per-shard install cost will dominate
• Write access to a durable store for timing data — repository file, cache entry, or artifact
• A coverage tool that can merge reports (nyc, istanbul-merge, or vitest --merge-reports)

How Shard Assignment Works

Every sharding implementation is the same three-line algorithm: enumerate the test units in a deterministic order, partition that ordered list into N groups, and have runner i execute group i. Everything that goes wrong with sharding comes from one of those three steps.

Enumeration order must be stable. Runners glob the filesystem independently, and glob order is not guaranteed identical across machines or across operating systems. If shard 2 enumerates a different list than shard 3, some files run twice and others never run at all — and a test that never runs reports as a pass. Sort the discovered list explicitly before partitioning; every mature runner does this internally, which is the main reason to prefer native --shard over a hand-rolled split pipeline.

Partitioning is the only interesting decision. The default is contiguous or round-robin assignment by index, which balances file counts and therefore balances nothing that matters. Two files differ in runtime by two orders of magnitude in any suite that mixes unit and browser tests. Duration-balanced partitioning — greedy longest-first bin packing over recorded timings — takes ten lines and typically halves the worst shard.

Execution must be independent. Sharded tests run in fresh processes on separate machines, so anything shared implicitly (a seeded database row, a fixture file written by an earlier test, an ordering assumption) breaks. This is the part teams underestimate: the first sharded run surfaces every latent inter-test dependency at once.

There is one more property worth naming, because it decides whether sharding is even the right tool. Sharding only helps when the suite is CPU- or browser-bound on the runner. If the tests spend most of their time waiting on a shared staging API or a single database, splitting them across four machines produces four clients contending for the same bottleneck — the wall-clock barely moves and the failure rate climbs as the dependency saturates. Check where the time actually goes before adding runners: if a serial run shows 80% of each spec’s duration in network wait, the fix is a faster or isolated backend, not more parallelism. Sharding multiplies whatever the tests already do, including their contention.

Step-by-Step Implementation

1. Measure before splitting

You cannot balance what you have not measured, and the measurement is also how you decide the shard count.

# Playwright: emit a JSON report with per-spec durations from one serial run.
npx playwright test --reporter=json > timings.json

# Rank the slowest specs — this is the list that determines your floor.
jq -r '.suites[].specs[] | "\(.tests[0].results[0].duration/1000 | floor)s\t\(.file)"' \
  timings.json | sort -rn | head -10

If the single slowest file takes four minutes, no shard count will bring the suite below four minutes. That file must be split before parallelism helps — which is a test-design fix, not a CI fix.

2. Declare the matrix

# .github/workflows/test.yml
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false          # a failure in shard 1 must not hide failures in shard 3
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: pnpm }
      - run: pnpm install --frozen-lockfile
      - run: npx playwright install --with-deps chromium   # ~40s: the per-shard overhead floor
      - name: Run shard ${{ matrix.shard }}
        run: npx playwright test --shard=${{ matrix.shard }}/4 --reporter=blob
      - uses: actions/upload-artifact@v4
        with:
          name: blob-report-${{ matrix.shard }}
          path: blob-report
          retention-days: 1

fail-fast: false is not optional. With the default, the first shard to fail cancels the others, so a developer fixes one failure, pushes, and discovers a second — turning a single red run into three round trips.

3. Balance the split on recorded durations

Store the timings from step 1 in the repository and partition greedily: sort files longest-first, and repeatedly assign the next file to whichever shard currently has the least accumulated time.

// scripts/plan-shards.mjs — greedy longest-processing-time bin packing.
import { readFileSync } from 'node:fs';

const timings = JSON.parse(readFileSync('test-timings.json', 'utf8')); // { "specs/checkout.spec.ts": 84.2, ... }
const shardCount = Number(process.argv[2] ?? 4);
const shards = Array.from({ length: shardCount }, () => ({ files: [], total: 0 }));

// Longest first: placing big items early is what keeps the tail balanced.
for (const [file, seconds] of Object.entries(timings).sort((a, b) => b[1] - a[1])) {
  const target = shards.reduce((min, s) => (s.total < min.total ? s : min));
  target.files.push(file);
  target.total += seconds;
}

// Unknown files (new tests with no timing yet) go to the lightest shard.
console.log(JSON.stringify(shards.map((s) => s.files)));

Verify the plan is actually balanced before trusting it:

node scripts/plan-shards.mjs 4 | jq -r '.[] | length'   # file counts will differ — that is correct

A spread wider than about 15% between the heaviest and lightest shard means the timing data is stale or one file is too large to pack around. Greedy longest-first packing is not optimal in the theoretical sense, but it is provably within a third of optimal and needs no tuning — and in practice the residual imbalance is dominated by run-to-run variance on shared runners anyway, so a smarter packer buys nothing measurable.

4. Merge the reports

Per-shard reports are individually useless to a coverage gate. Merge them in a dependent job.

  merge:
    needs: test
    if: always()             # merge and report even when a shard failed
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with: { pattern: blob-report-*, path: all-blobs, merge-multiple: true }
      - run: npx playwright merge-reports --reporter=html,github ./all-blobs
      - uses: actions/upload-artifact@v4
        with: { name: html-report, path: playwright-report }

The if: always() condition is what makes the merge job a reporting step rather than a second gate. Without it, a single failing shard skips the merge, the developer gets four separate partial reports and no coverage number, and the most common response is to re-run the whole matrix to find out what actually broke. The fan-out/fan-in shape below is the topology worth aiming for: the matrix decides pass or fail, and a single downstream job turns four partial results into one artifact a human can read.

For unit-test coverage the equivalent is nyc merge followed by a single threshold check:

npx nyc merge ./coverage-shards ./coverage/merged.json
npx nyc report --temp-dir ./coverage --check-coverage --lines 85   # gate on the merged total

Configuration Reference

Note that workers and shard count compound: four shards each running four workers is sixteen concurrent browsers, which will exhaust a shared database or a rate-limited staging API long before it exhausts CPU.

Integration with Adjacent Topics

Sharding sits between two neighbours in this section and changes the economics of both.

Affected detection runs first. Incremental builds and affected detection in monorepos decides which suites need to run at all; sharding then splits whatever survived that filter. Applying them in the other order — sharding the full suite and letting each shard skip — pays the per-shard setup cost for shards that then do nothing.

Caching sets the overhead floor. Every shard repeats checkout, install, and browser download. Without the dependency caching described in best practices for caching npm vs Yarn vs pnpm in CI, an eight-shard matrix pays eight cold installs and can be slower end to end than four warm ones.

Concurrency limits cap the ceiling. A 12-shard matrix on a queue with 10 free runners does not run 12-wide; it runs 10-wide and then 2-wide, which is worse than an 8-shard matrix that fits. The queue behaviour is covered in optimizing pipeline concurrency and queue limits.

Performance and Cost Impact

Wall-clock falls with shard count; total runner minutes rise, because every shard repays the same fixed setup. The crossing point is the only number that matters, and it is specific to your setup cost.

Read the flat tail carefully: past six shards this suite buys 30 seconds of wall-clock for 18 extra runner-minutes per run. At 200 runs a week that is 60 runner-hours a month for half a minute a developer barely notices. The right stopping point is where the marginal wall-clock gain drops below roughly a minute — usually when the slowest shard approaches the fixed setup cost, here 70 seconds.

Troubleshooting

Error: No tests found on a subset of shards. The shard count in the command does not match the matrix length — --shard=${{ matrix.shard }}/4 with a five-entry matrix produces a shard 5 of 4. Derive both from one source: build the matrix from fromJSON over a computed list, or hard-code the denominator next to the matrix definition and review them together.

Shard timings drift apart over weeks. The timing file is stale; new specs were added and all landed in whichever shard was lightest at generation time. Regenerate timings on a schedule — a nightly full serial run that commits the updated file is the simplest durable approach — rather than during pull-request runs, where the data would be contaminated by shard-level parallelism.

Coverage drops sharply after enabling sharding. The gate is reading one shard’s report instead of the merged one, usually because the artifact download pattern matched a single file. Confirm the merged report exists and covers the expected file count before believing a coverage regression that appeared on the same commit that enabled sharding.

Tests pass serially and fail when sharded. Real inter-test coupling, exposed. The usual causes are a shared database record created by an earlier test in the same file order, a fixture written to a fixed path, or a global mock left installed. Reproduce locally with --shard=2/4 to get the exact subset, then fix the isolation — retrying the shard hides a bug that will eventually appear in production.

Frequently Asked Questions

Related

• Incremental Builds & Affected Detection in Monorepos — decides which suites need to run before sharding splits them.
• Optimizing Pipeline Concurrency & Queue Limits — the runner capacity that caps how wide a matrix can actually go.
• Implementing Remote Build Caching with Turborepo — removes the rebuild each shard would otherwise repeat.
• Running E2E Tests Against Preview Environments — where a sharded browser suite most often points.

← Back to Build Optimization & Caching Strategies