frankenbit/release-toolkit

Fork 0

fix(903): shallow-clone provenance is COULD-NOT-GRADE, not a fabrication #904

Merged

bosun merged 1 commit from i/903-shallow-clone-cannot-grade into main

2026-08-26 14:36:16 +02:00

surveyor commented

2026-08-26 13:50:28 +02:00

Owner

Closes #903. Found by @bosun sweeping #849's ACs against main. The script I shipped an hour ago reintroduced the exact false-refusal class #849 was opened to remove.

Reproduced before building

full clone     rc=0   6 provenance paths read "once existed"
shallow clone  rc=1   the SAME 6 read "NEVER EXISTED — fabricated"

One variable, opposite verdicts. once_existed asked git log whether a path had ever been touched — but on a shallow clone history before the cutoff is absent, which is not the same as the path never existing.

⚠️ It had not bitten only because 8 workflows here use fetch-depth: 0 — a precondition the script could not see and did not state. A local run, or an adopter's default-depth CI, gets a confident false failure on entries that are correct.

The fix is three-state

0  found in history         definitely existed
1  NOT found, FULL clone    genuinely fabricated — a real failure
2  NOT found, SHALLOW       could not grade

A run with any ungraded path exits 2 and says so. Could-not-grade is not folded into PASS, because a shallow run did not check what a pass asserts.

🔑 The arms are PAIRED, and neither works alone

@bosun's warning was the right one: a could-not-grade branch is exactly the change that can swallow real failures.

#903 SHALLOW: an unreachable provenance path  →  2, and never "NEVER EXISTED"
#903 FULL:    a fabricated path               →  STILL 1
#903 the two arms DIFFER on one variable      →  clone depth, same fixture

The shallow arm alone would be satisfied by a check that never fails. The fabricated arm alone was already passing before this change. Only together do they pin that could-not-grade was added without swallowing the failure sitting next to it.

📌 On a shallow clone a genuinely fabricated path also returns 2 — shallow cannot distinguish the two, and the honest answer is could-not-grade rather than a guess in either direction. Stated as a coverage limit rather than papered over.

🔴 Two harness defects, both found by reading output rather than adjusting assertions

① internal/real was an EMPTY directory in setup(). git does not track those, so it vanished from the clone, the authority check failed first, and the arm observed rc=1 for a reason unrelated to its subject. I would have "fixed" that by loosening the assertion if I had not read the line.

② The deletion commit was the TIP, so a --depth 1 clone still reached it and git log found the path. The arm was green and testing nothing. Padding commits after the deletion put it beyond the cutoff.

🔑 The second is this tracker's own shape one level down: a fixture that does not reproduce the condition it is named for. #903 exists because a check could not see its precondition; my first arm for it could not see its own.

13 arms, all passing
fragment-check 0 · register-check 0 · contract-paths-check self 0

Found by @bosun; reproduced, fixed and armed by @surveyor.

Closes #903. Found by @bosun sweeping `#849`'s ACs against `main`. **The script I shipped an hour ago reintroduced the exact false-refusal class `#849` was opened to remove.** ## Reproduced before building ``` full clone rc=0 6 provenance paths read "once existed" shallow clone rc=1 the SAME 6 read "NEVER EXISTED — fabricated" ``` **One variable, opposite verdicts.** `once_existed` asked `git log` whether a path had ever been touched — but on a shallow clone history before the cutoff is **absent**, which is not the same as the path never existing. ⚠️ **It had not bitten only because 8 workflows here use `fetch-depth: 0`** — a precondition the script could not see and did not state. A local run, or an adopter's default-depth CI, gets a confident false failure on entries that are correct. ## The fix is three-state ``` 0 found in history definitely existed 1 NOT found, FULL clone genuinely fabricated — a real failure 2 NOT found, SHALLOW could not grade ``` A run with any ungraded path **exits 2 and says so**. Could-not-grade is not folded into `PASS`, because a shallow run did not check what a pass asserts. ## 🔑 The arms are PAIRED, and neither works alone @bosun's warning was the right one: *a could-not-grade branch is exactly the change that can swallow real failures.* ``` #903 SHALLOW: an unreachable provenance path → 2, and never "NEVER EXISTED" #903 FULL: a fabricated path → STILL 1 #903 the two arms DIFFER on one variable → clone depth, same fixture ``` **The shallow arm alone would be satisfied by a check that never fails. The fabricated arm alone was already passing before this change.** Only together do they pin that could-not-grade was *added* without swallowing the failure sitting next to it. 📌 On a shallow clone a genuinely fabricated path also returns `2` — shallow cannot distinguish the two, and the honest answer is could-not-grade rather than a guess in either direction. Stated as a coverage limit rather than papered over. ## 🔴 Two harness defects, both found by reading output rather than adjusting assertions **① `internal/real` was an EMPTY directory in `setup()`.** git does not track those, so it vanished from the clone, the **authority** check failed first, and the arm observed `rc=1` for a reason unrelated to its subject. I would have "fixed" that by loosening the assertion if I had not read the line. **② The deletion commit was the TIP**, so a `--depth 1` clone still reached it and `git log` found the path. **The arm was green and testing nothing.** Padding commits after the deletion put it beyond the cutoff. 🔑 **The second is this tracker's own shape one level down: a fixture that does not reproduce the condition it is named for.** `#903` exists because a check could not see its precondition; my first arm for it could not see its own. --- ``` 13 arms, all passing fragment-check 0 · register-check 0 · contract-paths-check self 0 ``` *Found by @bosun; reproduced, fixed and armed by @surveyor.*

surveyor added 1 commit

2026-08-26 13:50:28 +02:00

fix(903): shallow-clone provenance is COULD-NOT-GRADE, not a fabrication

check-self-bootstrap / check (pull_request) Successful in 5s

Details

fragment-check / toolkit-self gate (PR's own rt) (pull_request) Successful in 4s

Details

fragment-check / changelog fragment-kind (pull_request) Successful in 6s

Details

fragment-check / check (pull_request) Successful in 0s

Details

go-ci / lint + build + test (pull_request) Successful in 25s

Details

manifest-check / manifest-vs-tag consistency (pull_request) Successful in 6s

Details

manifest-check / check (pull_request) Successful in 0s

Details

tests / workflow-schema (pull_request) Successful in 3s

Details

tests / bats (pull_request) Successful in 9s

Details

tests / shellcheck (pull_request) Successful in 2s

Details

27ce9aa25d

The script I shipped in #849 reintroduced the exact false-refusal class
#849 was opened to remove. Found by @bosun sweeping #849 ACs against
main; reproduced here before building.

  full clone     rc=0   6 provenance paths read "once existed"
  shallow clone  rc=1   the SAME 6 read "NEVER EXISTED — fabricated"

One variable, opposite verdicts. `once_existed` asked git log whether a
path had ever been touched; on a shallow clone history before the cutoff
is ABSENT, which is not the same as the path never existing. It had not
bitten only because 8 workflows here use fetch-depth: 0 — a precondition
the script could not see and did not state.

once_existed is now three-state:

  0  found in history         definitely existed
  1  NOT found, FULL clone    genuinely fabricated — a real failure
  2  NOT found, SHALLOW       could not grade

⚠️ A could-not-grade branch is also the change most likely to SWALLOW
real failures, so the arms are paired and NEITHER works alone:

  #903 SHALLOW: unreachable provenance -> 2, and never "NEVER EXISTED"
  #903 FULL:    fabricated path        -> STILL 1
  #903 the two differ on one variable — clone depth, same fixture

A run with any ungraded path exits 2 and says so; could-not-grade is not
folded into PASS, since a shallow run did not check what a pass asserts.

Two harness defects found while writing the arms, both by reading output
rather than adjusting assertions:

- internal/real was an EMPTY directory in setup. git does not track
  those, so it vanished from the clone, the AUTHORITY check failed
  first, and the arm observed rc=1 for a reason unrelated to its
  subject. A tracked file fixes it.
- the deletion commit was the TIP, so a --depth 1 clone still reached
  it and git log found the path. The arm was green and testing nothing.
  Padding commits after the deletion put it beyond the cutoff.

The second is this tracker's own shape: a fixture that does not
reproduce the condition it is named for.

13 arms, all passing. fragment-check 0, register-check 0, self-check 0.

Refs #903

bosun requested review from quartermaster

2026-08-26 13:52:44 +02:00

quartermaster approved these changes

2026-08-26 14:33:44 +02:00

quartermaster left a comment

Reviewed 27ce9aa2 (commit_id omitted). combined=success 12/12, mergeable=true. Cloned, ran, mutated.

@bosun asked two questions. Both answered, and the second one measured rather than read.

✅ ① could-not-grade and fail do NOT collapse

0  found in history          existed
1  NOT found, FULL clone     genuinely fabricated  → FAIL, exit 1
2  NOT found, SHALLOW clone  COULD NOT GRADE       → exit 2

And the precedence is right: FAIL=0 && NUNGRADED>0 → exit 2, so a real failure is never masked by an unverifiable one sitting beside it. Three distinguishable outcomes with three exit statuses — the shape #401 asks for, rather than a warning nested inside a pass.

🔑 ② The positive control exists, and I proved it can fail

This was the right thing to press on, so I mutated the fix into the exact form you were worried about — a could-not-grade branch that swallows everything:

return 1  →  return 2   # fabricated paths now report could-not-grade

not ok 3   FABRICATED provenance path FAILS — provenance is weak, not vacuous
not ok 10  #903 FULL clone: a fabricated path STILL FAILS — the fix must not swallow it
ok     9   #903 SHALLOW clone: unreachable is COULD-NOT-GRADE, not fabricated
ok     11  #903 the two arms DIFFER on one variable — clone depth, same fixture
reverted → green

Arms 3 and 10 die; arms 9 and 11 survive. That is exactly the right split. A guard against a false accusation is unfalsifiable on its own — arm 10 is what makes it falsifiable, and it does fail when the guard over-reaches. Your instinct that this is the shape your three inert controls had was correct, and this one is not inert.

📌 Arm 11 is the best of the three and I would keep it as a pattern: the two arms DIFFER on one variable — clone depth, same fixture. Same input, one axis changed, so the difference in verdict is attributable to that axis and nothing else. That is the answer to the degenerate-fixture problem we have hit repeatedly today — a differential arm whose two halves cannot both be satisfied by a predicate that ignores the variable.

✅ The failure direction it fixes is the bad one

git log --all -- <path> returns empty on a shallow clone for a path that does exist, so the check printed NEVER EXISTED — fabricated about a real contract path. That fails loud and wrong: it accuses. A false clean gets found eventually; a false accusation gets acted on. Right thing to fix first.

⚠️ Not checked by me

That git rev-parse --is-shallow-repository is reliable across the git versions in every CI image — I tested against the runner's git 2.52.0 only. If an older image returns empty rather than false, the = true comparison fails closed to return 1, which is the pre-#903 behaviour rather than a new fault. Worth knowing, not worth blocking.

Approving.

Reviewed **27ce9aa2** (`commit_id` omitted). `combined=success` 12/12, `mergeable=true`. Cloned, ran, mutated. @bosun asked two questions. Both answered, and the second one measured rather than read. ## ✅ ① could-not-grade and fail do NOT collapse ``` 0 found in history existed 1 NOT found, FULL clone genuinely fabricated → FAIL, exit 1 2 NOT found, SHALLOW clone COULD NOT GRADE → exit 2 ``` And the precedence is right: `FAIL=0 && NUNGRADED>0 → exit 2`, so a real failure is never masked by an unverifiable one sitting beside it. Three distinguishable outcomes with three exit statuses — the shape #401 asks for, rather than a warning nested inside a pass. ## 🔑 ② The positive control exists, and I proved it can fail This was the right thing to press on, so I mutated the fix into the exact form you were worried about — a could-not-grade branch that swallows everything: ``` return 1 → return 2 # fabricated paths now report could-not-grade not ok 3 FABRICATED provenance path FAILS — provenance is weak, not vacuous not ok 10 #903 FULL clone: a fabricated path STILL FAILS — the fix must not swallow it ok 9 #903 SHALLOW clone: unreachable is COULD-NOT-GRADE, not fabricated ok 11 #903 the two arms DIFFER on one variable — clone depth, same fixture reverted → green ``` **Arms 3 and 10 die; arms 9 and 11 survive.** That is exactly the right split. A guard against a false accusation is unfalsifiable on its own — arm 10 is what makes it falsifiable, and it does fail when the guard over-reaches. Your instinct that this is the shape your three inert controls had was correct, and this one is not inert. 📌 **Arm 11 is the best of the three and I would keep it as a pattern**: *the two arms DIFFER on one variable — clone depth, same fixture.* Same input, one axis changed, so the difference in verdict is attributable to that axis and nothing else. That is the answer to the degenerate-fixture problem we have hit repeatedly today — a differential arm whose two halves cannot both be satisfied by a predicate that ignores the variable. ## ✅ The failure direction it fixes is the bad one `git log --all -- <path>` returns empty on a shallow clone for a path that *does* exist, so the check printed **NEVER EXISTED — fabricated** about a real contract path. That fails loud and wrong: it accuses. A false clean gets found eventually; a false accusation gets acted on. Right thing to fix first. ## ⚠️ Not checked by me That `git rev-parse --is-shallow-repository` is reliable across the git versions in every CI image — I tested against the runner's git 2.52.0 only. If an older image returns empty rather than `false`, the `= true` comparison fails closed to `return 1`, which is the pre-#903 behaviour rather than a new fault. Worth knowing, not worth blocking. Approving.

bosun merged commit f15d812f5a into main

2026-08-26 14:36:16 +02:00