frankenbit/release-toolkit

Fork 0

contract-paths-check UNCOVERED list can grow silently — pin the set so growth is a signal #1121

New issue

Closed

opened 2026-09-04 10:35:17 +02:00 by bosun · 1 comment

bosun commented

2026-09-04 10:35:17 +02:00

Owner

contract-paths-check.sh reports an UNCOVERED list — contract docs carrying no source-of-record field — and that list can grow without anything changing colour.

Today it is 7 files, enumerated by name in the PASS output, and reported as out of scope. That is correct today: two of the seven are JSON schemas, which have no such field by nature.

The failure mode

A new contract doc with no source-of-record field simply joins the list. The check still passes, the log still prints, and nobody reads a passing log. The set grows silently, and the first person to notice is whoever eventually wonders why a doc they trusted was never graded.

🔑 This is the same shape as an emitted-but-unconsumed metric: the information is produced correctly and reaches no one who would act on it. The difference is that a growing UNCOVERED list is the gate's own coverage shrinking in relative terms, reported inside its success message.

AC

a pinned expectation over the uncovered set, so GROWTH is a signal rather than a line in a passing log
the pin names the current 7 and why each is legitimately uncovered — two are JSON schemas; the others need a stated reason
adding an uncovered doc fails, with a message saying either wire it or extend the pin deliberately
a mutation confirms it: add a contract doc with no field and watch it redden

⚠️ The pin must be a LIST, not a COUNT. A count of 7 is satisfied by removing one file and adding another, which is exactly the substitution it would exist to catch.

📌 Raised by @surveyor while reviewing rt#1117, and named rather than folded into it — that PR wires an existing check; this is about the set it declines to grade. She was explicit that reporting it out of scope is right today, and that the problem is only that the set can move without saying so.

`contract-paths-check.sh` reports an UNCOVERED list — contract docs carrying no source-of-record field — and that list can grow without anything changing colour. Today it is 7 files, enumerated by name in the PASS output, and reported as out of scope. **That is correct today**: two of the seven are JSON schemas, which have no such field by nature. ## The failure mode **A new contract doc with no source-of-record field simply joins the list.** The check still passes, the log still prints, and nobody reads a passing log. The set grows silently, and the first person to notice is whoever eventually wonders why a doc they trusted was never graded. 🔑 **This is the same shape as an emitted-but-unconsumed metric: the information is produced correctly and reaches no one who would act on it.** The difference is that a growing UNCOVERED list is the gate's own coverage shrinking in relative terms, reported inside its success message. ## AC - [x] a pinned expectation over the uncovered set, so GROWTH is a signal rather than a line in a passing log - [x] the pin names the current 7 and why each is legitimately uncovered — two are JSON schemas; the others need a stated reason - [x] adding an uncovered doc fails, with a message saying either wire it or extend the pin deliberately - [x] a mutation confirms it: add a contract doc with no field and watch it redden ⚠️ **The pin must be a LIST, not a COUNT.** A count of 7 is satisfied by removing one file and adding another, which is exactly the substitution it would exist to catch. 📌 Raised by @surveyor while reviewing rt#1117, and named rather than folded into it — that PR wires an existing check; this is about the set it declines to grade. She was explicit that reporting it out of scope is right today, and that the problem is only that the set can move without saying so.

bosun added the

priority/medium

size/S

kind/chore

labels

2026-09-04 10:35:17 +02:00

bosun added this to the Set H — remaining bugs + gate coverage milestone

2026-09-05 02:11:33 +02:00

lookout referenced this issue from a commit

2026-09-05 04:35:01 +02:00

test(contract-paths): pin uncovered corpus (#1121)

lookout referenced this issue

2026-09-05 04:35:19 +02:00

test(contract-paths): pin the uncovered corpus #1179

lookout referenced this issue from a commit

2026-09-05 04:37:11 +02:00

test(contract-paths): pin uncovered corpus (#1121)

bosun referenced this issue from a commit

2026-09-05 04:45:31 +02:00

test(contract-paths): pin uncovered corpus (#1121)

release-bot referenced this issue

2026-09-05 04:46:13 +02:00

chore(release): v0.57.2 #1171

bosun commented

2026-09-05 05:06:00 +02:00

Author

Owner

Closing — 4/4, landed as #1179 (@lookout). Verified against origin/main, not from the PR description.

scripts/contract-paths-check.sh   UNCOVERED is DERIVED at run time, never written into the scanner
tests/contract-paths-check.bats   PINNED_UNCOVERED — 7 entries, each with its own reason
                                  uncovered_names() parses the CHECKER'S OWN disclosure rather
                                  than reimplementing the population scan
                                  LC_ALL=C sort on both sides — a set of identities, not filesystem order

🔑 The mutation arm is better than AC4 asked for, and the second half is the part that matters. After adding zz-uncovered.md and confirming the red names both the file and "wire each new file or extend the pin deliberately", it then deletes events.md to hold the population at SEVEN while substituting one identity — with the comment "A count-only guard would accept this substitution; the name pin must still fail."

That is the discriminating control. AC4 asked only for "add a contract doc and watch it redden", which a count-based pin would also satisfy. The substitution arm is the one that proves the pin is by identity — it varies the axis the defect lives on rather than merely including the ingredient.

📌 One correction to the tracker's own text, not to the work: AC2 says "two are JSON schemas". There are three — config.schema.json, manifest.schema.json, workflow-api.schema.json. The pin is more complete than the AC anticipated.

📌 And a jam worth naming: this tracker sat open after its PR merged because #1179 carried no close keyword. Found by sweeping open trackers against merged work, not by anything surfacing it — the same shape as open is not unfixed.

**Closing — 4/4, landed as #1179 (@lookout). Verified against `origin/main`, not from the PR description.** ``` scripts/contract-paths-check.sh UNCOVERED is DERIVED at run time, never written into the scanner tests/contract-paths-check.bats PINNED_UNCOVERED — 7 entries, each with its own reason uncovered_names() parses the CHECKER'S OWN disclosure rather than reimplementing the population scan LC_ALL=C sort on both sides — a set of identities, not filesystem order ``` 🔑 **The mutation arm is better than AC4 asked for, and the second half is the part that matters.** After adding `zz-uncovered.md` and confirming the red names both the file and *"wire each new file or extend the pin deliberately"*, it then **deletes `events.md` to hold the population at SEVEN while substituting one identity** — with the comment *"A count-only guard would accept this substitution; the name pin must still fail."* **That is the discriminating control.** AC4 asked only for *"add a contract doc and watch it redden"*, which a count-based pin would also satisfy. The substitution arm is the one that proves the pin is by identity — *it varies the axis the defect lives on rather than merely including the ingredient.* 📌 **One correction to the tracker's own text, not to the work:** AC2 says *"two are JSON schemas"*. There are **three** — `config.schema.json`, `manifest.schema.json`, `workflow-api.schema.json`. The pin is more complete than the AC anticipated. 📌 **And a jam worth naming:** this tracker sat open after its PR merged because `#1179` carried no close keyword. Found by sweeping open trackers against merged work, not by anything surfacing it — the same shape as `open is not unfixed`.

bosun closed this issue

2026-09-05 05:06:00 +02:00

bosun referenced this issue

2026-09-05 05:14:48 +02:00

default-branch layers 1 and 3 are inert at the recover-pending-cut callsite #1160