Merging Coverage Reports from Parallel Test Shards

You sharded the test suite, and coverage fell from 87% to 23% on the same commit. Nothing regressed — the gate is reading one shard’s report. Coverage is only meaningful across the whole suite, so the shards’ raw data has to be reassembled before any threshold is applied.

When to use this pattern

  • A sharded suite reports coverage, and the numbers no longer match the serial run.
  • Your coverage gate started failing on the same commit that introduced sharding.
  • You want one report a reviewer can open, not four partial ones per run.

Prerequisites

Why per-shard numbers cannot be combined by averaging

Coverage is a set union over executed lines, not an average of percentages. Two shards each reporting 30% do not make 30%, or 60% — they make whatever the union of their executed lines is, which depends entirely on overlap.

Per-Shard Coverage Against the Merged Union Four shards each execute a different subset of the codebase, reporting between nineteen and thirty-one per cent individually. Their union covers eighty-seven per cent. An average of the four percentages would give twenty-five per cent, which describes nothing real. CODEBASE — each bar is the same set of files shard 1 28% shard 2 31% shard 3 25% shard 4 19% merged union 87% Averaging the four would give 25.75% — a number with no meaning. The union is the only correct combination.

Complete working example

# .github/workflows/test.yml
name: Test
on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci --ignore-scripts

      - name: Test shard ${{ matrix.shard }}
        run: |
          npx jest --shard=${{ matrix.shard }}/4 \
            --coverage --coverageReporters=json \
            --coverageDirectory="coverage-${{ matrix.shard }}"
          # Rename so four shards' files can share one directory after download.
          mv "coverage-${{ matrix.shard }}/coverage-final.json" \
             "coverage-${{ matrix.shard }}/shard-${{ matrix.shard }}.json"

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: coverage-${{ matrix.shard }}
          path: coverage-${{ matrix.shard }}/shard-${{ matrix.shard }}.json
          retention-days: 1

  coverage:
    needs: test
    if: always()          # report even when a shard failed; partial data still informs
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci --ignore-scripts

      - uses: actions/download-artifact@v4
        with: { pattern: coverage-*, path: shards, merge-multiple: true }

      - name: Merge and gate
        run: |
          ls shards/                                   # expect shard-1..4.json
          npx nyc merge shards .nyc_output/merged.json
          npx nyc report --reporter=text-summary --reporter=lcov \
            --temp-dir .nyc_output \
            --check-coverage --lines 85 --branches 75 --functions 80

      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: coverage-html, path: coverage/lcov-report, retention-days: 7 }
// jest.config.cjs — count files that no test imported, or they vanish from the denominator.
module.exports = {
  collectCoverage: true,
  // Without collectCoverageFrom, only files a test actually required are instrumented,
  // so an entirely untested module is simply absent — and the percentage looks better.
  collectCoverageFrom: ['src/**/*.{ts,tsx}', '!src/**/*.d.ts', '!src/**/*.stories.tsx'],
  coverageReporters: ['json'],        // raw only in shards; rendering happens after the merge
};

Step-by-step walkthrough

Emit json, not lcov or text, inside the shards. coverage-final.json is istanbul’s raw counter data and is the only format that merges losslessly. lcov is a rendered summary; merging rendered summaries loses the per-statement counts the union needs.

Rename the files before uploading. Every shard produces coverage-final.json, and merge-multiple: true downloads them into one directory where three of the four would overwrite the first. Making the name unique per shard is a one-line change that prevents a silent, hard-to-diagnose data loss.

Which Coverage Format Merges, and Which Is a Dead End Four shards each emit raw coverage JSON, which merges into a single dataset. From that dataset, lcov, an HTML report and a text summary are rendered. Arrows from lcov and HTML back toward the merge step are shown crossed out, because rendered formats cannot be merged. shard-1.json shard-2.json shard-3.json shard-4.json nyc merge union of counters merged.json lcov (for services) html (for humans) text-summary (for the gate) Rendering is one-way: nothing on the right can be fed back into the merge. Shards must emit raw JSON. This is also why a shard that emits only text-summary leaves you with no recoverable data at all.

collectCoverageFrom is what makes the number honest. By default, coverage instruments only files a test imported, so a module with no tests at all does not appear in the denominator. Enumerating the source glob explicitly means an untested file shows as 0% rather than disappearing — which is usually a several-point difference on a real codebase.

Gate after merging, never inside a shard. Placing --check-coverage in the shard job guarantees failure, because each shard legitimately covers roughly a quarter of the code. Thresholds belong exclusively in the merge job.

if: always() on the coverage job. When a shard fails, the merged report is built from whatever arrived. That is a partial picture and should not gate — but it is exactly what a developer wants to look at while fixing the failure.

Verification

# 1. All four raw files arrived and are non-trivial.
ls -la shards/ && jq -s 'map(keys | length) | {files_per_shard: .}' shards/*.json

# 2. The merge really unions rather than replacing.
npx nyc merge shards .nyc_output/merged.json
jq 'keys | length' .nyc_output/merged.json          # ≥ the largest single shard's count

# 3. The merged total matches a serial run on the same commit.
npx jest --coverage --coverageReporters=text-summary

Expected — merged and serial agree within rounding:

Statements   : 87.14% ( 4021/4615 )
Branches     : 76.02% ( 1102/1450 )
Functions    : 81.33% (  610/750  )
Lines        : 87.41% ( 3944/4512 )

If the merged file has roughly the same key count as one shard, the rename step is missing and shards overwrote each other. That is the single most common cause of a mysterious post-sharding coverage drop.

Reporting coverage that people act on

A merged number that is correct but invisible changes nothing. Two additions turn the report into something reviewers use.

The first is a per-pull-request diff view: coverage on changed lines rather than the repository total. A 0.2% movement in a whole-repo figure is noise, but “eleven of the twenty-three lines you added are untested” is specific and actionable. Most coverage services compute this from the merged lcov plus the pull request’s diff, and it is the single change that moves coverage from a number people argue about to feedback people respond to.

The second is treating the threshold as a ratchet rather than a target. Fixing the gate at the current value and raising it only when the merged figure has comfortably exceeded it for a fortnight prevents both the slow erosion that comes from a permanently-too-low bar and the sudden blockage that comes from setting an aspirational one. Combined with the shard mechanics in parallelizing test suites in CI, it keeps a fast suite honest without turning coverage into a negotiation.

Repository Total vs Changed-Lines Coverage The repository total moves from 87.14 per cent to 87.34 per cent, a change too small to act on. The changed-lines view for the same pull request shows twelve of twenty-three added lines covered and eleven untested, pointing at specific code. repository total 87.14% → 87.34% +0.20 percentage points indistinguishable from noise changed lines in this pull request 12 of 23 added lines covered src/checkout/refund.ts lines 41–51 untested Both come from the same merged report. Only the right-hand one tells a reviewer what to do next.

Common pitfalls

Uploading a rendered report instead of raw JSON. An HTML or lcov report cannot be merged back into counter data. Always keep coverage-final.json per shard.

Merging across different source revisions. If one shard ran on a re-triggered older commit, its line numbers no longer match and the merge silently produces nonsense. Ensure all shards share one github.sha.

Requiring the coverage job as a blocking check while it runs if: always(). It will report on partial data when a shard failed. Gate on the aggregate job that also asserts the shards passed, as described in splitting lint, test and build into parallel jobs.

Store the merged report as an artifact even when the gate passes. The single most useful thing a coverage report does is answer “was this file covered before my change”, and that question is asked days later, by which time an un-uploaded report no longer exists anywhere.

← Back to Parallelizing Test Suites in CI