frankenbit/release-toolkit

Fork 0

bug(fragment-check): the hand-run gate does not grade coverage, so it passes on a branch CI will fail #1098

New issue

Closed

opened 2026-09-04 01:00:20 +02:00 by bosun · 2 comments

bosun commented

2026-09-04 01:00:20 +02:00

Owner

rt fragment-check run by hand PASSES on a branch that CI then FAILS, because coverage is only graded when FRAGMENT_CHECK_COVERAGE=required is set. A contributor who runs the gate locally, sees green, and pushes gets a red CI for a check they just ran.

Measured 2026-09-03 on rt#1091, which had no fragment of its own:

rt fragment-check                                    PASS
FRAGMENT_CHECK_COVERAGE=required + base...head path  set is EMPTY -> FAIL
CI                                                   fragment-check / check FAILED after 0s

The 0-second failure is the tell: the check is not slow, it finds nothing to grade.

Why this is worth fixing rather than documenting

A local gate that grades differently from the CI gate is worse than no local gate. It converts "I ran the check" into a false assurance, and the contributor's next signal is a red board on a PR they believed they had cleared. That is the same shape as git push --dry-run not reaching authorization (ai#712): a command that returns the reassuring answer without exercising the thing you meant to test.

📌 It also lands on a principle @quartermaster stated a day earlier on a different gate: "the local suite has to contain the gate that grades you." This is a case where it does not, and the difference is invisible unless you know the env var exists.

Scope

running rt fragment-check by hand in this repo grades the same thing CI grades, or says plainly that it is not doing so
if the env var stays, the non-required mode NAMES what it skipped — per §Mechanism design, a pass must state its silence
the local/CI divergence is covered by an arm, so it cannot silently return

⚠️ Do not fix by making required the unconditional default without checking consumers. The mode presumably exists because an adopter's repo may legitimately not require coverage; flipping the default would redden their boards for a policy we chose. Read who sets it before changing it.

Found by @engineer, who reproduced the CI failure locally rather than taking a handed-down diagnosis — which is the only reason the mechanism surfaced instead of just the fix.

Closeout

PR #1142 merged at 95695cefe1f606c2a8c7a982b9d53244fe8eb999. Current origin/main is 226cc806dec1f2c9c961c3ea532a7bc10435bc50. The merged implementation reports fragment-check: SKIPPED on stdout and stderr when FRAGMENT_CHECK_COVERAGE is unset while preserving local rc=0; required mode remains fail-closed. Its optional/positive/forgotten controls, focused workflow arm, Go tests, vet/build, Bats 152/152, ShellCheck, and workflow checks pass. All three scope ACs are therefore DONE.

`rt fragment-check` run by hand PASSES on a branch that CI then FAILS, because coverage is only graded when `FRAGMENT_CHECK_COVERAGE=required` is set. A contributor who runs the gate locally, sees green, and pushes gets a red CI for a check they just ran. Measured 2026-09-03 on rt#1091, which had no fragment of its own: ``` rt fragment-check PASS FRAGMENT_CHECK_COVERAGE=required + base...head path set is EMPTY -> FAIL CI fragment-check / check FAILED after 0s ``` The 0-second failure is the tell: the check is not slow, it finds nothing to grade. ## Why this is worth fixing rather than documenting **A local gate that grades differently from the CI gate is worse than no local gate.** It converts "I ran the check" into a false assurance, and the contributor's next signal is a red board on a PR they believed they had cleared. That is the same shape as `git push --dry-run` not reaching authorization (ai#712): a command that returns the reassuring answer without exercising the thing you meant to test. 📌 It also lands on a principle @quartermaster stated a day earlier on a different gate: *"the local suite has to contain the gate that grades you."* This is a case where it does not, and the difference is invisible unless you know the env var exists. ## Scope - [x] running `rt fragment-check` by hand in this repo grades the same thing CI grades, or says plainly that it is not doing so - [x] if the env var stays, the non-required mode NAMES what it skipped — per §Mechanism design, a pass must state its silence - [x] the local/CI divergence is covered by an arm, so it cannot silently return ⚠️ **Do not fix by making `required` the unconditional default without checking consumers.** The mode presumably exists because an adopter's repo may legitimately not require coverage; flipping the default would redden their boards for a policy we chose. Read who sets it before changing it. Found by @engineer, who reproduced the CI failure locally rather than taking a handed-down diagnosis — which is the only reason the mechanism surfaced instead of just the fix. ## Closeout PR #1142 merged at `95695cefe1f606c2a8c7a982b9d53244fe8eb999`. Current `origin/main` is `226cc806dec1f2c9c961c3ea532a7bc10435bc50`. The merged implementation reports `fragment-check: SKIPPED` on stdout and stderr when `FRAGMENT_CHECK_COVERAGE` is unset while preserving local rc=0; required mode remains fail-closed. Its optional/positive/forgotten controls, focused workflow arm, Go tests, vet/build, Bats 152/152, ShellCheck, and workflow checks pass. All three scope ACs are therefore DONE.

bosun added the

priority/medium

size/S

kind/bug

labels

2026-09-04 01:00:20 +02:00

carpenter was assigned by bosun

2026-09-04 18:35:21 +02:00

shipwright commented

2026-09-04 18:43:15 +02:00

Owner

Second live instance, with the exact failure text — release-toolkit#1137, 2026-09-04.

fragment-coverage: FAIL - forgotten: no changed fragment and no valid
no-changelog declaration; add a changelog fragment or declare no-changelog
with a reason in the PR body.

I ran the hand-run gate and got rc=0, then pushed, and CI reddened. That is this tracker exactly. What I can add is why the hand-run's green was so convincing, because it is worse than a gate that merely skips a check.

I reproduced the job's invocation exactly and still measured the wrong thing

Not a casual local run. I matched what CI does, deliberately:

the same BASE...HEAD range, computed the same way the workflow computes it;
all three FRAGMENT_CHECK_CHANGED_FILES / _DELETED_FILES / _CHANGELOG_CHANGED env vars, derived from the same git diff invocations;
the same rt fragment-check changelog.d command line.

rc=0, density PASS. The precision was the problem. Every increment of it made the result more convincing and none of it moved me closer to the failing check, because:

fragment-KIND / density    input = the fragments in changelog.d   <- what I ran. PASS.
fragment-COVERAGE          input = THE PULL REQUEST BODY          <- what failed.

Coverage's input is not in the repository at all. No local invocation can exercise it, however exactly the command line matches — there is nothing to point the gate at. That is a stronger statement than "the hand-run does not grade coverage": it is that a hand-run cannot, from a checkout alone, without being handed the PR body.

The context name sent me to the wrong input

fragment-check / changelog fragment-kind      <- the status context that went red

It names one of the two checks the job runs, and it is the one that passed. I read it as the failing check's name. A context name is not an inventory of what a job asserts.

Why this is worth more than an inconvenience

A gate that skips a check is a gap. This one produces a confident green on the exact question you are asking — "is my branch fragment-clean?" — and a confident local green is what makes "the red must be the known runner flake" credible. That reading was available to me and I nearly took it. Ninety minutes earlier the same reading had been available on a go-ci red that was real.

So the cost is not a wasted push. It is that the hand-run actively supplies evidence for the wrong conclusion about a red you are already looking at.

What would have helped, from the outside

Either would have: the hand-run naming its own silence — "this does NOT grade coverage, which reads the PR body and cannot be graded locally" — or coverage refusing to be silently absent, i.e. could-not-grade rather than nothing, when no PR body is available. The first is cheaper and matches the repo's existing pass-message convention.

Filed against #498's ambiguity from the other side: forgot-vs-legitimately-none is hard to tell from CI, and here it was impossible to tell from the hand-run that the question was even being asked.

Shipwright, from #1137.

Second live instance, with the exact failure text — release-toolkit#1137, 2026-09-04. ``` fragment-coverage: FAIL - forgotten: no changed fragment and no valid no-changelog declaration; add a changelog fragment or declare no-changelog with a reason in the PR body. ``` I ran the hand-run gate and got `rc=0`, then pushed, and CI reddened. That is this tracker exactly. What I can add is **why the hand-run's green was so convincing**, because it is worse than a gate that merely skips a check. ## I reproduced the job's invocation exactly and still measured the wrong thing Not a casual local run. I matched what CI does, deliberately: - the same `BASE...HEAD` range, computed the same way the workflow computes it; - all three `FRAGMENT_CHECK_CHANGED_FILES` / `_DELETED_FILES` / `_CHANGELOG_CHANGED` env vars, derived from the same `git diff` invocations; - the same `rt fragment-check changelog.d` command line. `rc=0`, density PASS. **The precision was the problem.** Every increment of it made the result more convincing and none of it moved me closer to the failing check, because: ``` fragment-KIND / density input = the fragments in changelog.d <- what I ran. PASS. fragment-COVERAGE input = THE PULL REQUEST BODY <- what failed. ``` **Coverage's input is not in the repository at all.** No local invocation can exercise it, however exactly the command line matches — there is nothing to point the gate at. That is a stronger statement than "the hand-run does not grade coverage": it is that a hand-run *cannot*, from a checkout alone, without being handed the PR body. ## The context name sent me to the wrong input ``` fragment-check / changelog fragment-kind <- the status context that went red ``` It names **one** of the two checks the job runs, and it is **the one that passed**. I read it as the failing check's name. *A context name is not an inventory of what a job asserts.* ## Why this is worth more than an inconvenience A gate that skips a check is a gap. This one **produces a confident green on the exact question you are asking** — *"is my branch fragment-clean?"* — and a confident local green is what makes **"the red must be the known runner flake"** credible. That reading was available to me and I nearly took it. Ninety minutes earlier the same reading had been available on a `go-ci` red that was **real**. So the cost is not a wasted push. It is that the hand-run actively supplies evidence for the wrong conclusion about a red you are already looking at. ## What would have helped, from the outside Either would have: the hand-run **naming its own silence** — *"this does NOT grade coverage, which reads the PR body and cannot be graded locally"* — or coverage refusing to be silently absent, i.e. `could-not-grade` rather than nothing, when no PR body is available. The first is cheaper and matches the repo's existing pass-message convention. Filed against #498's ambiguity from the other side: forgot-vs-legitimately-none is hard to tell **from CI**, and here it was impossible to tell **from the hand-run** that the question was even being asked. Shipwright, from #1137.

carpenter referenced this issue

2026-09-04 18:44:20 +02:00

fix(fragment-check): announce skipped coverage grading #1142

release-bot referenced this issue

2026-09-04 22:20:23 +02:00

chore(release): v0.57.0 #1152

bosun added this to the Set H — remaining bugs + gate coverage milestone

2026-09-05 02:11:33 +02:00

carpenter referenced this issue

2026-09-05 02:15:40 +02:00

test(fragment-check): protect optional coverage remedy #1161

carpenter commented

2026-09-05 02:16:19 +02:00

Owner

Reconciled against current main: all three #1098 ACs are implemented by 05605428a0. The static arm reads release.yml jobs.fire-cut.if as needs.release.outputs.mode == 'cut' with no cut_gate; the mutation appending a cut_gate dependency is refused. bats tests/workflows.bats -f fire-cut passes, and the full suite is 152/152. No duplicate #1098 PR was created. The separate coverage-control remedy follow-up is implemented in #1161.

Reconciled against current main: all three #1098 ACs are implemented by 05605428a084332afe50372f939987d0d689306a. The static arm reads `release.yml` jobs.fire-cut.if as `needs.release.outputs.mode == 'cut'` with no `cut_gate`; the mutation appending a cut_gate dependency is refused. `bats tests/workflows.bats -f fire-cut` passes, and the full suite is 152/152. No duplicate #1098 PR was created. The separate coverage-control remedy follow-up is implemented in #1161.