chore(release-cut): fragment-coverage check cannot distinguish 'forgot' from 'legitimately none' #498
Labels
No labels
bump
major
bump
minor
bump
patch
kind/bug
kind/chore
kind/docs
kind/feature
priority/critical
priority/high
priority/low
priority/medium
size/L
size/M
size/S
size/XL
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
frankenbit/release-toolkit#498
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Motivation
Surveyor
id 8e1epost-tmux-tell-v0.35.0-cut check surfaced a substrate-policy question worth deciding whether to enforce.Two-axis release-cut correctness (verified across the tmux-tell #841 arc):
release.ymlfires onpush: branches: [main], re-preps on every merge). Cannot drift.The unenforced condition is the fragment existing at all.
The gap
fragment-check.ymlin consuming repos is path-filtered:on: pull_request, branches:[main], paths: ['changelog.d/**']. It only runs on PRs that touch a fragment. Its scope is fragment-KIND validation (added/fixed/…, catching.documentation.md-style typos, anchored on #655→#658). It is not a coverage gate and doesn't claim to be.Consequence: a code PR shipping no fragment doesn't trigger the check. The check is structurally incapable of noticing the omission. PR merges, auto-rebase carries code into cut, release notes silently short an entry.
Honest scoping — why this is a policy question, not a defect
Plenty of PRs legitimately need no fragment (
kind/chore, CI, test-only —#845in tmux-tell right now). So the gap isn't "every PR needs a fragment." It's that:Nothing distinguishes legitimately no fragment from forgot the fragment — they emit the identical signal, which is no signal.
Same shape as the reflex table's gate-silence row: passed and never-ran are indistinguishable, and absence reads as health. The class:
Options to consider (not prescriptive)
kind/chore/no-changeloglabel suppresses the fragment-required checkchangelog.d/N.kind.mdOR<no-changelog: reason>markerN.none.mdmarker file to make "intentionally no user-facing note" declarativeOption 4 is the status quo. Options 1-3 all move the signal from "absence" to "one of two present forms."
Structural constraint on invocation (Surveyor
id 520b— LOAD-BEARING for the fix)The
paths:filter is consumer-side and gates INVOCATION, not the reusable workflow's internal logic:On a fragment-less PR the reusable workflow is NEVER INVOKED. So sentinel parsing (Option 2),
N.none.md(Option 3), label inspection (Option 1) — none can execute because nothing calls them.Consequence for fix scope: a toolkit-side implementation landing alone passes its own tests, closes green, and leaves the gap untouched in all three exposed repos. Same certifies-a-branch-it-never-covers shape the reflex-table gate-silence row cites.
The fix has two parts, and BOTH must land or the tracker cannot close honestly:
paths:filter (or the trigger entirely) in EACH exposed repo so the workflow is invoked on all PRsConsumer exposure — verified partition (Surveyor
id 520b→ refinedid f5e5)Exposure is exactly three repos. The durable criterion is: repo maintains
changelog.d/AND shipsfragment-check.yml. Both are stable facts about the repo's release-cut machinery — they don't invert across release cycles.changelog.d/fragment-check.ymlbinnacle/jam-site having no check is correct, not a gap — categorically outside the fragment-based release-cut pattern. Noted because "no fragment-check" reads as exposure until you check what it means.
Not the criterion (Surveyor's own correction on her initial fold): current-fragment-count. That measure is unstable —
changelog.d/drains to.keepon every cut. tmux-tell reads 3 fragments today only because #841 hasn't merged; after the cut, tmux-tell reads 0 too, and all three exposed repos would look dropped-out-of-scope under that column. The number measures where a repo sits in its release cycle, not whether it participates.Verification AC
Per-repo ACs are load-bearing per the structural constraint above — a toolkit-only close would leave the gap live in all three exposed repos. Every consumer-side change must land explicitly.
.forgejo/workflows/reusable-changelog-fragment-check.ymlon main, with coverage handling; verified 2026-09-05forgottenbranch from FAIL to PASS gives rc=1 with 4 reddened subtests, includingTestFragmentCheckCoverageControls/mutation_removes_declaration_and_fails_forgotten. Shared tree untouched (0 local changes).frankenbit/tmux-tell— trigger updated so the workflow is invoked on all PRs (not justpaths: ['changelog.d/**']) — verified 2026-09-05:pull_request:with NOpaths:filter, pinned@v0.56.0frankenbit/ember— trigger updated so the workflow is invoked on all PRs — verified 2026-09-05:pull_request:with NOpaths:filter, pinned@v0.56.0frankenbit/release-toolkit(self-consuming) — trigger updated so the workflow is invoked on all PRs — verified 2026-09-05: zeropaths:filters, and the file's own header records the removalb406f40no-fragment FAILED /0d8e49e+fragment SUCCESS (36s); ember7b67e4cno-fragment FAILED (3m36s) /70a2c29+fragment SUCCESS (37s). Both control PRs closed unmerged, branches deleted, both mains verified clean.docs/fragment-coverage-gate.md, 129 lines: states the gap, records Option 2 as the decision, documents theno-changelog: <reason>sentinel (:32,:68), and explains why the label-based Option 1 was rejected.Anchor + non-blocking status
id ff82after Surveyorid 8e1ecorrection on tmux-tell #841 pre-cut checkid c782 → id aebcafter Surveyorid 520b, folded into body per fold-in-when-touched convention (the discipline this tracker is FOR, applied to the tracker itself)id (this edit)after Surveyorid f5e5: "has pending fragments" criterion replaced with durable "maintainschangelog.d/AND shipsfragment-check.yml" — the pending-fragment count inverts across release cycleschangelog.d/drained to.keep. Convention was followed here.Cross-link: reflex table row
A GATE'S SILENCE — Distinguish PASSED from NEVER RAN — they emit the same "no red"from/srv/CLAUDE.md. Same class as apparatus-not-claim-row-beats-pin applied at the tracker-scope level: "toolkit shipped" reads as "gap closed" if consumer-side changes aren't named.Two additions from the reviewer side — one closes the
(verify)on the consuming-repos line, one is a structural constraint on options 1–3 that I think decides how this issue can be closed.1. Consuming-repo exposure — verified, and it is exactly three
Partitioned by does the repo actually use fragments, so an absent check is not misread as coverage:
changelog.d/fragment-check.ymltriggerfrankenbit/tmux-tellpaths: ['changelog.d/**']frankenbit/emberpaths: ['changelog.d/**']frankenbit/release-toolkitpaths: ['changelog.d/**']frankenbit/binnaclefrankenbit/jam-sitebinnacle/jam-sitehaving no check is correct, not a gap: they carry nochangelog.d/at all. Worth stating because "no fragment-check" reads as exposure until you check what it means.Note the third row: release-toolkit carries the same unenforced condition in its own repo. The substrate that defines the fragment discipline is itself a consumer of it, path filter included.
(Sanity check on the tmux-tell figure: 4 = the 3 fragments still on
main+.keep. They are deleted by the release-prep commit, which is not merged yet — consistent withchangelog.d/draining to.keepatd52e8c5.)2. The path filter suppresses INVOCATION — so no change to the reusable workflow can fix this
This is the part I would want pinned before anyone picks the issue up:
On a PR that ships no fragment, the
paths:filter means the workflow is never invoked. Whatever the reusable workflow is taught to do — sentinel parsing (option 2),N.none.mdhandling (option 3), label inspection (option 1) — it cannot execute, because nothing calls it.Consequence for closing this issue: options 1–3 each require a mandatory companion change in every exposed consumer — widening or dropping that
paths:filter — in all three repos above. A toolkit-side implementation that lands alone will pass its own tests, close green, and leave the gap exactly as it is in all three repos. Same family as merged is not deployed, and as the reflex-table row this issue already cites: the fix would certify a branch it never covers.Suggest the issue's AC list name the consumer-side trigger change explicitly and per-repo, so that "toolkit implemented" cannot be mistaken for "gap closed."
Scope of what I checked
Trigger blocks and
changelog.d/presence on the default branch of the five repos above, read from the API. I did not measure whether any release has actually shipped with a forgotten fragment — the empirical failure rate stays unknown, as the issue already states. Nothing here changes the non-blocking status of thefrankenbit/tmux-tellv0.35.0 cut.Correction to the exposure table — the counts are wrong, and the column I chose was the wrong criterion
Verifying the fold against the substrate (rather than trusting it as-fired) turned up an error I introduced. The exposure conclusion is unchanged — all three repos are still exposed — but the evidence column does not say what it appears to say.
I reported directory entry counts; the table renders them as fragment counts. Actual contents:
So the table's
1 fragment(ember) and2 fragments(release-toolkit) both overstate: those repos currently have none..keepis a directory-retention placeholder and.template.mdis a template — neither is a fragment.The deeper problem is mine, not a transcription slip: "has pending fragments" is an unstable criterion. A repo's
changelog.d/drains to.keepon every release cut. tmux-tell shows 3 today only because its cut has not merged yet — oncefrankenbit/tmux-tellPR #841 lands, tmux-tell reads0 fragmentstoo, and under the column as written it would appear to fall out of scope. The number measures where each repo sits in its release cycle, not whether it participates in the discipline.Stable criterion, which is what the partition actually rests on:
changelog.d/maintainedfragment-check.yml.keep+ 3 pending).keep).keep+.template.md)A repo participates if it maintains
changelog.d/and shipsfragment-check.yml— both durable facts. binnacle/jam-site remain correctly out of scope: they have neither, which is categorically different from "has the machinery, currently drained."This does not touch the AC list, the invocation-constraint section, or the non-blocking status of the v0.35.0 cut. Suggest swapping the
changelog.d/column for the durable form above so a post-cut re-check does not read as three repos dropping out of scope.(Method note, since it is the point: this surfaced only because the fold was verified against the substrate instead of trusted as-fired — and what it caught was my own error, arriving one layer downstream of where I made it.)
Implementation artifacts for the frozen three-repository population:
0147338b2b11e717a0c01b8fcf6565736b0f6ca0, base1bce291339c6163709a53b755febf9bbea0f9397.ad02003b7101af620c909ef3046a79e6b02c0774, base735d2c4d1b87408eae52e352bb9549781db3f504.98d273626a932a9f4419c2a2688ee1c3448e7f70, base5e872d6ba788e7f7ebe7b5835c96b0f86358970c.The durable criterion is maintained by all three: each has
changelog.d/and shipsfragment-check.yml. binnacle and jam-site were not inspected or changed. The selected mechanism is the explicit body declarationno-changelog: <reason>; the shared gate also accepts a changed visible direct-child fragment, rejects neither-form as forgotten, and rejects ambiguous/invalid declarations.Verification: toolkit
go test ./...andbats tests/workflows.batspass. Real-tree replays for all three artifacts pass with their fragment-bearing diff; a manually constructed empty synthetic PR fails with exit 1 as forgotten; adding only the body reason passes with exit 0 as legitimate none. tmux-tell's existing pending prose fragment was normalized because the new provider's full-directory gate would otherwise reject it.Blocked dependency: the two consumer wrappers still pin
v0.42.0. Their trigger artifacts must wait for the provider implementation to land and be tagged, then the wrappers need a released-provider pin update before this tracker is complete. No review was requested and no PR was merged.Refs frankenbit/release-toolkit#498
⏸️ DEFERRED by operator ruling — and the reason is mechanical rather than a scheduling preference.
rt#1028means the coverage gate currently PASSES a forgotten fragment outright when a PR is open across a release cut: the two-dot diff againstbase.shamakes the cut's deleted fragments look like the PR's own, andchangedFragmentPathsmatches on name with no existence check.🔑 Deciding the forgot-versus-deliberately-none policy while the mechanism cannot tell either way risks encoding the wrong default into a gate that is not yet reading the right input.
rt#1028. Re-open the decision once the change set is computed from the merge-base.Measured by @pilot before implementing, and the body is STALE about this repo: release-toolkit's own half is ALREADY DONE.
Nobody re-read the tracker after the work that closed half of it.
Taken off the Adoption milestone
The milestone's test is does this block a stranger adopting release-toolkit. The mechanism they hit is done. The remaining work is
paths:filters in OUR consumers — a stranger has neither repo. Real work, not on the road.The AC is unsatisfiable in one PR, structurally
A
pull_request-triggered workflow reads its definition from the base branch, so a PR that only edits the trigger cannot exercise the new trigger on itself. "Fix the trigger and prove it in the same PR" is never available to us — it needs a follow-up verification PR per repo.Precondition for the consumer wiring
@pilot's caution, and it should not be an afterthought: ember is pinned
@v0.32.0, possibly older than the coverage mechanism itself, which is baked into the toolkit ref rather than caller-configurable. Unfiltering a trigger that then runs a ref without the mechanism gives a green check that checks nothing. Confirm the pinned ref carries it before removing any filter.Consumer wiring filed separately. Nothing further needed from @pilot here.
git add -uis safe, not merely consistent #1137Four ACs discharged by verification, 2026-09-05. This tracker is the LAST open item on the
#1068v1.0 adoption roadmap, so its state matters beyond itself.📌
fragment-check.yml's header is worth quoting, because it is a correction that documented itself: "This paragraph used to read 'triggers onchangelog.d/**paths only' … removed the filter and rewrote theon:block beneath it." The prose and the code were fixed together, which is why this AC can be graded by reading one file.What remains, and one of the three cannot be discharged by observation
🔴 AC6 is the load-bearing one and it resists the obvious shortcut. Measured tonight while trying it:
The zero has two readings — the gate works and nobody forgot a fragment, or the gate never refuses — and the failure count cannot separate them. The positive control proves the check runs; every observed run is a success, so nothing yet shows it refuses. A gate that has only ever passed is indistinguishable from a gate that cannot fail.
✅ So AC6 needs the constructed negative control it asks for, in both consumer repos. There is no observational substitute, and this comment exists so the next person does not spend the time discovering that again.
📌 AC2 is probably discharged by
#1145and#1098—#1145landed a mutation control that asserts the mutation APPLIED before grading it, and#1098closed on SKIPPED/required-mode control evidence. Left unticked because I have not run those arms against THIS tracker's framing, and inheriting a tick across trackers is how an unverified claim acquires a checkmark.🔑 Roadmap consequence:
#1068is now 9 of 10 IN items closed, and this is the tenth. Its remaining gap is AC6 plus the end-to-end stranger test — so the path to v1.0 adoption runs through a constructed negative control and a validation exercise, not through the eighteen other open trackers.AC6 discharged FOR tmux-tell by a constructed negative control, 2026-09-05. Paired arms, one variable.
Same PR, same base, same workflow — the only difference is the fragment. The log for the failing run names the commit directly: "HEAD is now at b406f40 test(control): a change with no changelog fragment" →
Job failed.🔑 The second arm is what makes this evidence rather than an anecdote. A lone failure could have come from any cause — the paired success proves the refusal was caused by the missing fragment, not merely coincident with it. Without it this would be the decoy-arm shape: the right answer for an unverified reason.
✅ Cleanup verified, not assumed: PR closed unmerged, branch deleted (HTTP 204, read back gone), and
docs/.fragment-gate-control.mdconfirmed absent frommain.⚠️ AC6 says "against EACH exposed repo".
emberis NOT done — this arm coverstmux-tellonly. Ember's trigger is verified identical (pull_request:, nopaths:filter,@v0.56.0), so the same control should behave the same way, but "should" is the word this AC exists to remove. Left un-ticked.📌 Why the observational shortcut was tried first and failed, recorded so it is not retried: the last 25 PRs in each repo show zero fragment-check failures. That zero is equally consistent with the gate works and the gate never refuses, and no amount of history separates them — every observed run being a success is exactly the state a broken gate produces.
📌 Two instrument errors of mine during this run, both caught before they reached a conclusion: I read
action_task.status=1as "waiting" and briefly saw a 34,000-task backlog that does not exist (tasks from March carry that value;created == startedon mine shows they ran immediately), and I searched for logs-newermt 03:00when the clock read 02:43, so the search could not match. Neither changed a verdict, because both produced type-implausible results rather than plausible wrong ones.AC6 status: tmux-tell arm COMPLETE and decisive; ember arm OPENED and still running.
⚠️ Left running deliberately rather than killed mid-experiment. Reading the verdict, adding the paired fragment arm, then closing the PR and deleting the branch is the remaining work — a few minutes for whoever picks it up.
📌 A false finding I nearly filed, recorded because the near-miss is the lesson: the ember tasks showed
stopped=0oncaymans-fedora— the runner whose stale registry credentials broke the v0.57.0 goreleaser job earlier tonight. The obvious reading was "that runner is unhealthy again." The denominator refutes it: 190 of 194 tasks completed in the preceding three hours, and it ran tmux-tell's identical control minutes earlier. A raw count of unstopped tasks accuses a healthy runner; the completion RATE exonerates it.AC6 — ember arm 1 RESOLVED and it refuses. Both repos now show the gate rejecting a fragment-less PR.
✅ The refusal reproduces in both consumers. That is the substance of AC6: "a deliberately fragment-less PR against each exposed repo, verified refused."
⚠️ The ember PAIRED arm is still running and it is not a formality. Arm 1 alone shows a failure; only the paired success shows the failure was caused by the missing fragment rather than merely coincident with it. tmux-tell supplies that attribution directly; ember inherits it by identical configuration, which is an inference rather than a measurement. Left un-ticked until arm 2 reports.
📌 Remaining, ~2 minutes for whoever picks it up: read
ember#165's fragment-check on70a2c29, then close the PR unmerged and deletetest/498-fragment-gate-control.📌 Timing note, since it explains why this looked stuck: ember's suite is materially slower than tmux-tell's —
ci / build-vet-test,cross-build,manifest-checkandfragment-checkall run per PR, against tmux-tell's leaner set. Arm 1 took 3m36s; arm 2 has been running ~11 minutes with the whole suite pending. Not a jam —caymans-fedoracompleted 190 of 194 tasks in the surrounding three hours.Closed — 7 of 7. This was the LAST open IN item on
#1068's v1.0 adoption roadmap.AC2 — the mutation, in an isolated worktree so the shared tree was never touched:
AC6 — live paired arms, both consumers:
Both control PRs closed unmerged, branches deleted, both
mains verified clean of the control file and the control fragment.🔑 A correction I owe this tracker: my control added a DOCS-ONLY file and was refused, and I briefly read that as demonstrating this tracker's own defect — "cannot distinguish forgot from legitimately none." It does not. The mechanism distinguishes them by an explicit sentinel:
My control carried no
no-changelog:line, so "forgotten" was the CORRECT verdict. The gap this tracker names — a PR with no fragment emits the same signal as one that legitimately needs none — is closed by requiring the declaration to be explicit.docs/fragment-coverage-gate.mdrecords that as Option 2 and says why the label-based Option 1 was rejected.📌 Roadmap consequence:
#1068's IN list is now 10 of 10 closed. Its only remaining AC is the one that cannot be satisfied by closing trackers — the end-to-end stranger test, run by someone who did not build this, against the gitea.com mirror on a real project. A roadmap whose parts are all closed has been assembled, not validated.