Cumulative coverage and test gain

This page extends code coverage. It assumes you can already bind, run and ingest a window for Go or LLVM, and read documents with reflow coverage documents|gaps|lines|diff. To decide which gaps deserve a test and when to stop, see when to stop adding tests.

Before and after a new test

Coverage is process-wide: nothing attributes lines to a single flow. To learn what one new test adds without running the existing suite again, run only the new test in its own window, then compare two cumulative documents built from the same frozen binary: one from the baseline windows’ raw files, and one from those files plus the new window’s.

1. Record a baseline: run the existing tests in one or more run windows on one frozen instrumented binary. Keep every window’s cover directory unchanged after ingest, with its expected process count and apply keys. Build the baseline cumulative document from them and note its ID.
2. Find candidates: gaps --document <baseline> lists missed ranges. Read them with lines --with-source and pick reachable behaviour that no existing test asserts: an error path, an empty state, a boundary, a state transition.
3. Write the test. Record its purpose (the behaviour and risk) and its oracle (where the expected values come from) before you look at the delta.
4. Run only the new test once, in its own window: a fresh, empty cover directory for each producer, bound to the same frozen binaries (do not rebuild or re-sign them), with the same exclusions. Keep that directory too.
5. Build a new cumulative document from the baseline windows’ raw files plus the new window’s, then compare: diff --document <new cumulative> --base <baseline cumulative> --json. When the pair is likeForLike, the delta comes from the new test’s processes.

A diff between documents of different scenarios (for example one test’s window against another’s) is a scenario comparison. It can even be likeForLike, but it is never a test gain.

diff classifies the pair:

• likeForLike: same format, producer, toolchain, binary, mode, instrumented set, exclusions, tree and coverable code. gain reports the hit delta per metric for the whole comparison, plus newly hit and newly missed ranges per file on the current page (pageComplete is false while more pages follow). A net line delta of 0 can still hide lines that moved: check newlyHit and newlyMissed.
• descriptive: something other than tests changed (reasons such as tree_changed, toolchain_changed, binary_changed or exclude_changed). Totals are shown, gain is null, and the change must not be reported as test gain. To get a like-for-like pair, use the same instrumented binaries from the same commit on both sides; a rebuild can change the binary (binary_changed) even from the same source.
• Documents from different formats or producers are refused (format_mismatch, producer_kind_mismatch).

Cumulative documents from saved windows

There is no command that merges stored documents. Each ingest makes one document, and documents are never combined afterwards. Ingest does merge every raw file in its cover directory, so you can assemble one cumulative document per language from windows you already ran, with bind and ingest only and without running anything again. It is not a separate feature: record how each cumulative document was assembled.

Requirements

• One frozen binary. Every included window ran the exact same executable: the same file and SHA-256, never a rebuild or re-sign. Bind that file; ingest refuses binary_changed.
• The source it was built from. Run bind and ingest from a clean checkout of the commit the binary was built from. Go’s bind checks the binary’s VCS stamp against HEAD. LLVM binaries carry no stamp, so the source binding is whatever checkout you bind: choose the right commit.
• The same scope as the documents you compare. Use the same --collection, the same --exclude globs and, for LLVM, the same --path-strip.
• One fresh directory per language. Bind a new, empty directory outside the worktree, then copy the raw files in (bind refuses a directory that is not empty). Go and LLVM files never share a directory or a document.

Process count and batch keys

• --processes is the sum of each included window’s expected processes for that language, taken from what you launched (launch records, step states), never from the files you copied. Never lower it to match the files.
• Pass every included window’s apply keys with --batch: at most 64 keys, each with at most 1,000 runs (batch_too_large otherwise). Each key must have runs in the collection you pass (unknown_batch otherwise). A flow that ran in several windows counts once in distinctFlows.
• If a window is missing a file it should have, leave that whole window out (its files, its share of the count and its keys) and say so, or stop. Never keep its keys while dropping its missing process.

Go

cd <clean checkout at the binary's commit>
reflow coverage bind --binary <frozen Go binary> --cover-dir <new-dir> \
  --collection <id> [--exclude <glob>]... --json
cp -p <window-1-dir>/covmeta.* <new-dir>/
cp -p <window-dir>/covcounters.* <new-dir>/   # every included window
reflow coverage ingest --cover-dir <new-dir> --processes <N> \
  --batch <key-1> --batch <key-2> --collection <id> --json

• Copy exactly one covmeta.* file. For one binary it is the same file in every window; compare them with cmp first, and stop if they differ.
• Copy the covcounters.* files with their names unchanged. Do not copy reflow-start.* markers or tmp.* files.
• Counter file names carry the process ID. If two windows reused the same process ID, ingest refuses duplicate_pid. Do not rename the files to get around it: split those windows into separate cumulative documents, or report the limitation.
• The document is exit-only: copied files cannot be verified at process start.

LLVM (Swift and other languages)

cd <clean checkout at the commit the binary was built from>
reflow coverage bind --format llvm-cov --binary <frozen Mach-O or ELF> \
  --cover-dir <new-dir> --path-strip <same prefix> --collection <id> --json
cp -p <window-dir>/<name>.profraw <new-dir>/<window>-<name>.profraw   # every profile
reflow coverage ingest --format llvm-cov --cover-dir <new-dir> --processes <N> \
  --batch <key-1> --batch <key-2> --collection <id> --json

• Process IDs repeat across windows, so prefix each copied file name with its window. Names may use letters, digits and ._%+- and must end in .profraw. Renaming changes only the name, never the contents.
• Ingest refuses any other file, a count other than N (profile_count_mismatch), and byte-identical profiles (profile_duplicate), even when they came from different processes. Copy each profile once; if two genuine profiles are identical, split the scope or report it.
• Reflow does not check that a profile came from the bound binary; llvm-cov drops what does not match. Copy only profiles written by launches of that exact executable.

Reading and labelling the result

• Read it like any document: gaps, files, lines, and diff --document <new cumulative> --base <previous cumulative>. With one frozen binary, one checkout and the same excludes, the pair is likeForLike; anything else is descriptive and never a test gain.
• Go and LLVM stay separate documents and separate diffs. Never blend them.
• Keep a record of each cumulative document: the source directories, the apply keys and the expected-process sums it was built from, and any window left out and why.