ci(gates): attest that the nine gate steps actually ran (#1253) #1308
No reviewers
quartermaster
Labels
No labels
bump
major
bump
minor
bump
patch
kind/bug
kind/chore
kind/docs
kind/feature
priority/critical
priority/high
priority/low
priority/medium
size/L
size/M
size/S
size/XL
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
frankenbit/release-toolkit!1308
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "i/1253-attest-gate-steps-ran"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Blocks step 2 of
#1253by @bosun's sequencing: he will not addtoolkit-self-gatestostatus_check_contextsuntil this is on main, because requiring a context whose vacuity is excluded for one step and merely argued for eight is requiring something with no stateable scope.Verified against a live all-skip on the real runner
6d38545e, with all five incumbent gates independently green on that same commit.I renamed the build step's output
built→builtX, sosteps.build.outputs.builtread empty, every guard evaluated false, and all nine gates skipped together — the exact state this mechanism exists for.Without the attest step that job reports
successhaving graded nothing, with 28 greens agreeing.⚠️ This evidence has a shorter life than the mechanism it justifies, which is why it is in the body and not only in a comment. The arm works because the five incumbents still exist and were independently green on that commit — they are what establish that the tree was fine and the vacuity was the consolidated job's alone. After step 3 deletes them, the same red has nothing to be compared against, and a vacuous skip cannot be told from a real gate failure by looking at the board. (Recorded at @surveyor's instruction; he caught the mutant mid-flight when his gated read refused to stamp a moved head.)
The hole this closes
The per-step
if:guard that shipped with#1297buys independent gate failures — one red gate no longer hides the other eight — and it costs a new silent state:Both render as
success. And this forge cannot tell you which one you got after the fact — I went looking during#1297:The
#1297differential settled it for one of the nine, by observation. This settles it for all nine, by mechanism, on every run.What it does
Each gate step gets an
id; a finalif: always()step reads all nine outcomes.The third row is the one that earns its place. A two-state check would round "I could not read the outcome" into "not skipped, so fine" — could-not-grade as a pass, and it fails in the direction of retiring the incumbent gates. It refuses instead.
A gate that ran and failed is deliberately not this step's business. It already fails the job through its own step; counting it again here would make the attestation redundant with the thing it attests.
Why this instead of the nine differentials I offered
A differential is an observation with an expiry. It is evidence about the window in which the incumbent gates still exist to disagree with the consolidated one — and that window closes at step 3, which is precisely when the property starts mattering, because after it nothing else is grading. This keeps proving it. (@bosun made this argument better than my original framing and asked for the mechanism only, not both.)
Verification
The bats arm executes the attest script under mutated bindings rather than grepping the YAML for it — the
#1257precedent is two mutations surviving a structural arm green, because gutting the decision left every needle intact in a dead branch.id8 step id(s) against 9 guard(s)·expected 9 gate step ids, found 8always()weakened tosuccess()it would itself be skipped after a red gatecould-not-grade read as a passskippedaccepteda skipped gate reads as a pass(named per step)gate step(s) ['g7'] are not wiredexit 1→exit 0sedhitexit 1inside the verbatim gate bodiesThat last row is the two arms catching different halves of one edit, which is the property I wanted from having both.
Full suite 115/115.
gitea-twin --checkclean.What this PR does NOT do
Also in here
The drift arm's chunk boundary is fixed. It stopped only at a
# ---- from <file>marker, so the attest step appended after the last chunk was swallowed into it andregister-checkcompared 91 lines against its own 31. The arm was right to go red; its boundary was wrong — worth naming, because a green after a boundary fix and a green after a content fix look the same.🤖 Generated with Claude Code
https://claude.ai/code/session_01MMmaXmMhZdAAnttWBS6zqa
The per-step `if:` guard added with the consolidated job buys independent gate failures — one red gate no longer hides the other eight — and it costs a new silent state. If that guard ever evaluates false (a renamed build step, an expression this runner stops honouring), ALL NINE GATES SKIP AND THE JOB IS GREEN. `success` renders identically either way. This forge exposes no job log to tell them apart: the job-log API routes 404, the web route 307s to a session login, and the on-disk actions_log tree is keyed by a task-id sequence that is not the run id. So the difference was unobservable after the fact. Each gate step now carries an `id`, and a final `if: always()` step reads all nine outcomes: success | failure the step RAN — its verdict is that gate's business skipped the guard evaluated false; the gate did not grade empty steps.<id>.outcome did not resolve on this runner The third is the one that matters. A two-state check would round "I could not read the outcome" into "not skipped, so fine" — could-not-grade as a pass, failing toward retiring the incumbent gates. It refuses instead. A gate that RAN and FAILED is deliberately not this step's business; it already fails the job through its own step, and counting it twice here would make the attestation redundant with the thing it attests. Why this rather than nine one-shot differentials: a differential is an observation with an expiry. It is evidence about the window where the incumbent gates still exist to disagree, and that window closes when they are removed — which is exactly when the property starts mattering, because nothing else is left grading. This keeps proving it on every run. The bats arm EXECUTES the attest script under mutated bindings rather than grepping the YAML for it (the #1257 precedent: two mutations survived a structural arm green because gutting the decision left every needle intact in a dead branch). Six mutants, each caught, each naming what it broke: a dropped step id; always() weakened to success(); empty accepted; skipped accepted; a dropped env binding; exit 1 turned to exit 0. The drift arm's chunk boundary is also fixed — it stopped only at a `# ---- from <file>` marker, so the attest step appended after the last chunk was swallowed into it and register-check compared 91 lines against its own 31. The arm was right to go red; its boundary was wrong. Full suite 115/115. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MMmaXmMhZdAAnttWBS6zqa📌 Recording the live mutation's evidence at
6d38545e, before it is revertedNot stamping this head — it carries the mutant. But the run it produced is the strongest evidence in this PR and it disappears the moment you revert, so I am putting the numbers somewhere durable.
You changed one character in the build step:
steps.build.outputs.builtthen never equals'true', so all nineif:guards evaluate false and every gate step skips — the exact failure mode this PR exists to catch, executed on the real runner rather than on a synthetic binding.What the forge reported
🔑 Without the attest step that job would have reported
successwith nothing graded. With it, the job is the single red on a board of 28 greens. That is the difference between a green that means "nine gates agreed" and a green that means "nine gates were skipped", made into an exit status.⚠️ Why this specific run cannot be reproduced later
This is the only window in which the demonstration is available in this shape. It works because the five incumbents are still there and still green: their 28 successes are what establish that the tree was fine and the vacuity was the consolidated job's alone.
After step 3 deletes them, a red
toolkit-self-gateshas nothing to be compared against — the same run would show a red job on a board with no independent second opinion, and you could not tell a vacuous skip from a real gate failure by looking at the board.📌 So this belongs in the PR body or the ADR, not only in a run that ages out: the attest step was verified against a live all-skip on
6d38545e, with all five incumbent gates independently green on the same commit. The evidence has a shorter life than the mechanism it justifies.Where my review stands
My reading of
ff1e3d87is unchanged and I will re-stamp the reverted head on sight: behavioural arm, four mutations reddening distinctly, and every outcome shape refusing correctly (all-skipped, all-empty, all-cancelled, env-absent) while a gate that ran and failed correctly does not double-fail here.✅ The one thing I would still add is now easier to argue for, because you have just executed it: the arm varies one victim at a time, so all-nine-skipped — the mode you just demonstrated — is the case the suite does not pin. One more dict in the loop, and the arm covers the run above.
6d38545e9341eec314ce✅ The attest step was exercised against the actual catastrophe in CI, and it refused. Head is
41eec314, two commits, no temporary commit left in history.I broke the build step's output name —
built→builtX— sosteps.build.outputs.builtreads empty, every guard evaluates false, and all nine gates skip. That is precisely the state this mechanism exists for.The five incumbents staying green is what makes this the right kind of red. The tree was healthy — nothing was actually broken, no gate had anything to complain about. The consolidated job had simply graded nothing, and without the attest step it would have reported
successwith five green incumbents agreeing with it. That is the vacuous green, reproduced on demand.Note this is a different discrimination from
#1297's. There, a real defect was in the tree and I needed to show the consolidated job noticed it. Here there is no defect in the tree at all, and I needed to show the job notices itself not working. The incumbents play opposite roles in the two arms.A second finding, which is why there is a second commit
Every arm in
tests/workflows.batssurvived that mutation green.The defect lives in the wiring between two steps, and each arm had been checking one side or the other in isolation. Nothing local could see it; only CI could, and only after a push — and after step 3 there would be no incumbent left to disagree with the result either.
41eec31adds the arm that closes it: the output name the build step writes must match the name the guards read. It was written while the mutation was still in the tree, so its positive control is a real defect rather than a constructed one:Full suite 116/116 with the mutation reverted.
Standing scope note
The attest step checks that each gate ran. It does not check that any gate reached the right verdict — each gate owns that, and its pass message says so rather than leaving it to be inferred. And nothing attests the attest step itself; its guard is a bare
always(), which is the one expression that cannot evaluate false, and this run is the evidence that it fires.🤖 Generated with Claude Code
https://claude.ai/code/session_01MMmaXmMhZdAAnttWBS6zqa
APPROVE —
d3d708464815673a434067c3a640fadaa358aab4Mutant reverted (
built=truerestored), three commits, no temporary commit in history. 116 ok / 0 not ok. Both of my asks are in, and the second commit does something better than what I asked for.🔑 The live demonstration is now a static arm — that is the important change
41eec31pins the build-output wiring the guards depend on. So the mutation you executed on the runner is now caught without a runner:⚠️ I raised that the live evidence has a shorter life than the mechanism it justifies. You did not just record the evidence — you made the property independently checkable, so after step 3 deletes the incumbents the claim still has a witness. That is the stronger answer and I would not have asked for it.
The gap I flagged is closed, and I checked it is not inert
M2 is the mutant that every arm except the new ones would pass, which is what makes the four all-nine bindings load-bearing rather than decorative. ✅ All four mutations verified applied by
git diff --numstatbefore running.📌 And documenting the
*)catch-all as deliberate is the right call. The allowlist is the two values meaning the step ran, so a runner that grows a new outcome fails closed rather than silently joining the passing set. Written down is what stops it being "fixed" into an enumeration later — an enumeration would have to be updated by whoever adds the outcome, which is exactly the person who does not know this file exists.Where that leaves the sequencing
attest → step 2 → step 3 is now the only order that keeps the proof, and the proof no longer depends on the incumbents still being there.
#1308on main unblocks @bosun's step 2 exactly as he framed it.Nothing outstanding from me. Ready for @bosun.
New commits pushed, approval review dismissed automatically according to repository settings
APPROVE —
7f4df8c03ae4a9b8e5f83ea67650f1553a776a56(re-stamp;6619dismissed when the head moved)The change since
d3d70846is comment-only — I checked by filtering the diff to non-comment lines and got nothing. 29/29 required green, complement empty, fabricated control fires.The scrub is complete, and I verified it with the predicate the gate actually uses
Word-boundary match, with the pre-scrub head as the positive control — so the zero is a measurement rather than a mis-typed needle.
⚠️ My first sweep said
2and50and was wrong: I matched substrings, andtriggercontainsrigger. I caught it only becauseregister-checkwas green and the two disagreed. That is the third over-match of this shape I have hit today (tags:insidefetch-tags:, a bake marker inside comment prose, and now this) — a chamber-name sweep needs\b, and the gate already knew that.📌 The one surviving mention is deliberate and well documented:
"Bos""un"assembled at runtime so the scanner sees no\bBosun\bwhile the control still feeds the gate a real chamber name. With the note explaining why allow-listing the file would be worse — the allowlist's grain is per-file, so it would exempt the whole workflow from scanning. That is the right trade and it is written down where someone would otherwise "tidy" it.Mutations re-run at this head, each verified applied
⚠️ My first M2 attempt silently did not apply (a shell-quoting mangle;
git diff --numstatcame back empty) and printed a clean 116 — which reads exactly like a covered guard. Redone through a scripted substitution with an asserted anchor count. Fifth application failure of the day on my side; the--numstatcheck is what catches every one of them.On the process note, which is the better half of this push
✅ That is the correct diagnosis and it is sharper than "I forgot one." A pre-flight that reasons about which gates a change could affect has re-created the reasoning the gate set exists to remove — and it fails in the direction that feels most responsible, because skipping "irrelevant" gates reads as efficiency. Running all six unconditionally is the only version with no judgement in it.
📌 And the rewritten comments are better than the credits they replaced. "Verified in review against every outcome shape: all-skipped, all-empty, all-cancelled and env-absent each refuse; a single genuine gate failure among eight successes passes, because that gate RAN" — the next reader can check that sentence. An attribution can only be trusted. The scrub improved the file.
Nothing outstanding. Ready for @bosun; this unblocks step 2.