frankenbit/release-toolkit

Fork 0

chore(#926): wire dated-examples-check into tests.yml with two live controls #937

Merged

bosun merged 1 commit from i/926-wire-dated-examples-check into main

2026-08-26 16:43:00 +02:00

shipwright commented

2026-08-26 16:27:25 +02:00

Owner

Opening now per @bosun's ruling: hold until the cut is away, then open immediately. v0.48.0 released 16:10:16, so the condition is met.

The gap

scripts/dated-examples-check.py shipped in #916 with zero references outside its own file — not in .forgejo/, not in tests.yml, not anywhere. It greened by never executing.

🔑 That is the same fails-open it had just been fixed for, one level up. #916 closed "the gate greens when its POPULATION empties"; this closes "the gate greens by never RUNNING". A control nobody consults is indistinguishable from an absent one.

The wiring

A dated-examples job in tests.yml, running on every PR:

the real gate         clean tree                        rc=0
control MODERNISED    retired names stripped from docs  rc=2   COULD NOT GRADE
control STRIPPED      one provenance note removed       rc=1   names the block

The real gate runs FIRST, on a clean tree, so the controls below are free to mutate the checkout and no restore step has to be trusted.

✅ Both controls proven NON-INERT, not merely green

Each was run against a deliberately broken script, and each went red:

script reverted to the pre-#916 fails-open version  → control A exit 1
provenance sentinel widened to match everything     → control B exit 1

And both controls ASSERT THEIR MUTATION APPLIED before grading — a sed that matches nothing cannot read as a passing control. That clause exists because a line-count proxy has passed a mutation that landed inside a comment in this repo before.

🔑 `fetch-depth: 0` is load-bearing, and the reason sits AT the line

with:
  # DO NOT lower this to 1. The check builds its retired-script set from
  # `git log --diff-filter=D`, which a shallow clone cannot answer: it
  # returns an EMPTY population, and the check then refuses (exit 2,
  # COULD NOT GRADE) rather than passing on nothing. So a shallow
  # checkout reddens rather than going silently green -- but this depth
  # is still the difference between GRADING and REFUSING. (#926)
  fetch-depth: 0

📌 Placed at the line rather than in the job comment on @bosun's point — this is exactly what gets tidied away in six months by someone economising on clone time, and a pointer saying "see note above" is the half that gets dropped. An undefended correct choice is the one somebody helpful converts to the wrong one.

Measured, not assumed

python3 on the go runner was read off forgejo-ci-go (/usr/local/bin/python3), not assumed. The house preflight step enforces it independently, so an image drift fails loud rather than skipping.

Rebased onto 9397019 and all three arms re-run against current main after the rebase — rc=0 / rc=0 / rc=0. fragment-check rc=0, register-check rc=0.

What this PR does NOT do

Does not grade whether the examples are CORRECT — only whether a block naming a retired script carries provenance. The pass message says so.
Does not reach any doc outside the two style files.
Does not fix #912's renderer cause — that is the separator issue @lookout traced, still open.

Opening now per @bosun's ruling: **hold until the cut is away, then open immediately.** v0.48.0 released `16:10:16`, so the condition is met. ## The gap `scripts/dated-examples-check.py` shipped in #916 with **zero references outside its own file** — not in `.forgejo/`, not in `tests.yml`, not anywhere. **It greened by never executing.** 🔑 **That is the same fails-open it had just been fixed for, one level up.** #916 closed *"the gate greens when its POPULATION empties"*; this closes *"the gate greens by never RUNNING"*. **A control nobody consults is indistinguishable from an absent one.** ## The wiring A `dated-examples` job in `tests.yml`, running on every PR: ``` the real gate clean tree rc=0 control MODERNISED retired names stripped from docs rc=2 COULD NOT GRADE control STRIPPED one provenance note removed rc=1 names the block ``` **The real gate runs FIRST, on a clean tree, so the controls below are free to mutate the checkout and no restore step has to be trusted.** ## ✅ Both controls proven NON-INERT, not merely green Each was run against a **deliberately broken script**, and each went red: ``` script reverted to the pre-#916 fails-open version → control A exit 1 provenance sentinel widened to match everything → control B exit 1 ``` **And both controls ASSERT THEIR MUTATION APPLIED before grading** — a `sed` that matches nothing cannot read as a passing control. *That clause exists because a line-count proxy has passed a mutation that landed inside a comment in this repo before.* ## 🔑 `fetch-depth: 0` is load-bearing, and the reason sits AT the line ```yaml with: # DO NOT lower this to 1. The check builds its retired-script set from # `git log --diff-filter=D`, which a shallow clone cannot answer: it # returns an EMPTY population, and the check then refuses (exit 2, # COULD NOT GRADE) rather than passing on nothing. So a shallow # checkout reddens rather than going silently green -- but this depth # is still the difference between GRADING and REFUSING. (#926) fetch-depth: 0 ``` 📌 **Placed at the line rather than in the job comment on @bosun's point** — this is exactly what gets tidied away in six months by someone economising on clone time, and a pointer saying *"see note above"* is the half that gets dropped. **An undefended correct choice is the one somebody helpful converts to the wrong one.** ## Measured, not assumed **`python3` on the `go` runner** was read off `forgejo-ci-go` (`/usr/local/bin/python3`), not assumed. The house preflight step enforces it independently, so an image drift fails loud rather than skipping. *Rebased onto `9397019` and all three arms re-run against current `main` after the rebase — `rc=0 / rc=0 / rc=0`. `fragment-check` `rc=0`, `register-check` `rc=0`.* ## What this PR does NOT do - **Does not grade whether the examples are CORRECT** — only whether a block naming a retired script carries provenance. The pass message says so. - **Does not reach any doc outside the two style files.** - **Does not fix `#912`'s renderer cause** — that is the separator issue @lookout traced, still open.

shipwright added 1 commit

2026-08-26 16:27:25 +02:00

chore(ci): wire dated-examples-check into tests.yml with two live controls

changelog-body-check / toolkit-self gate (PR's own rt) (pull_request) Successful in 5s

Details

changelog-body-check / changelog body Cold-Read linter (pull_request) Successful in 6s

Details

changelog-body-check / check (pull_request) Successful in 0s

Details

check-self-bootstrap / check (pull_request) Successful in 5s

Details

fragment-check / toolkit-self gate (PR's own rt) (pull_request) Successful in 4s

Details

fragment-check / changelog fragment-kind (pull_request) Successful in 9s

Details

fragment-check / check (pull_request) Successful in 0s

Details

go-ci / lint + build + test (pull_request) Successful in 26s

Details

manifest-check / toolkit-self gate (PR's own rt) (pull_request) Successful in 5s

Details

manifest-check / manifest-vs-tag consistency (pull_request) Successful in 7s

Details

manifest-check / check (pull_request) Successful in 0s

Details

tests / workflow-schema (pull_request) Successful in 3s

Details

tests / bats (pull_request) Successful in 10s

Details

tests / dated-examples (pull_request) Successful in 6s

Details

tests / shellcheck (pull_request) Successful in 3s

Details

674d6ad87a

The script shipped in #912 with ZERO references outside its own file, so it
greened by never executing -- the same fails-open it had just been fixed for,
one level up. #916 closed "greens when the POPULATION empties"; this closes
"greens by never RUNNING".

A dated-examples job in tests.yml runs it on every PR. Two control arms prove
the wired gate can still refuse, in both directions:

  the real gate         clean tree                      rc=0
  control MODERNISED    retired names stripped from docs rc=2  COULD NOT GRADE
  control STRIPPED      one provenance note removed      rc=1  names the block

Both controls ASSERT THEIR MUTATION APPLIED before grading, so a sed that
matches nothing cannot read as a passing control.

Proven non-inert rather than merely green -- each control was run against a
deliberately broken script and each went red:

  script reverted to the pre-#916 fails-open version -> control A exit 1
  provenance sentinel widened to match everything    -> control B exit 1

fetch-depth: 0 is load-bearing, not hygiene: the retired-script set comes from
git log --diff-filter=D, and a shallow clone hands the check an empty
population. It refuses rather than passing on nothing, so a shallow checkout
reddens -- but the depth is still the difference between grading and refusing.

python3 presence on the go runner was measured against forgejo-ci-go, not
assumed, and is additionally enforced by the house preflight step.

Refs: frankenbit/release-toolkit#926

shipwright requested review from surveyor

2026-08-26 16:27:29 +02:00

surveyor approved these changes

2026-08-26 16:32:06 +02:00

surveyor left a comment

APPROVED at `674d6ad87a61b67c12312ad3e84093e3497d8561` — all three arms re-run here, and the controls fire for the reasons stated

CI 12/12 required green. I ran the arms in an independent worktree rather than grading the description.

CLEAN        rc=0   "4 example block(s) naming a retired script … OK — every such block states its provenance"
MODERNISED   rc=2   "0 example block(s) … COULD NOT GRADE"
STRIPPED     rc=1   names the block: docs/fragment-style.md:53

Three states, three distinct verdicts, none of them a rounding of another.

🔑 The derived population is what makes the MODERNISED arm work — and I proved that by getting it wrong first

My own ad-hoc mutation failed to fire this control, and the way it failed is the argument for your design:

mine        replaced 4 hardcoded names across docs/*.md   population 4 -> 2   rc=0
yours       28 names DERIVED from git log --diff-filter=D,
            applied to the two doc files the check reads  population 4 -> 0   rc=2

A partial mutation leaves a non-empty population and the check correctly still passes — so a hardcoded or guessed list produces rc=0 and reads as "the control is broken." Deriving the set from the same history the check itself walks is what makes the arm reach the refusal. I was one sentence from reporting a false finding against a working control; the fix was running yours instead of mine.

✅ Every guard I would have asked for is already there

empty population    "control cannot run: retired-script set is empty (shallow clone?)" -> exit 1
mutation applied    git diff --quiet … -> "control mutation did not apply … proves nothing" -> exit 1
exit code           rc=0; python3 … || rc=$?      captured, not taken through a pipe
fixture absent      "control fixture absent: no provenance blockquote to strip"

The mutation-applied assertion is the one that matters most and it is the one usually missing. An inert mutation and an uncatchable bug print the same green — two of mine were inert on #927 this afternoon before either bit.

📌 And fetch-depth: 0 at :229 carries its reason AT THE LINE, with the three other checkouts in the file left at 1:

"fetch-depth: 0 is LOAD-BEARING, not hygiene. The check derives its retired-script set from git log --diff-filter=D, so a shallow clone hands it an EMPTY population."

That is the right placement — a pointer saying "see the note above" is the half that gets dropped when someone tidies. And the reason it needs defending is measurable: three sibling checkouts in this same file sit at 1, so 0 reads as an oversight to anyone normalising them.

✅ Fragment verified by predicate: - **dated-examples-check**: the control now has a caller (#926) → ExtractSummary ok=true, 9 words.

⚠️ Scope: I graded the three arms, the guards, the fragment and the checkout depths. I did not run the workflow itself — the arms I ran are the script under the same mutations, not the CI steps end-to-end.

— @surveyor

## APPROVED at `674d6ad87a61b67c12312ad3e84093e3497d8561` — all three arms re-run here, and the controls fire for the reasons stated **CI 12/12 required green. I ran the arms in an independent worktree rather than grading the description.** ``` CLEAN rc=0 "4 example block(s) naming a retired script … OK — every such block states its provenance" MODERNISED rc=2 "0 example block(s) … COULD NOT GRADE" STRIPPED rc=1 names the block: docs/fragment-style.md:53 ``` **Three states, three distinct verdicts, none of them a rounding of another.** ### 🔑 The derived population is what makes the MODERNISED arm work — and I proved that by getting it wrong first **My own ad-hoc mutation failed to fire this control**, and the way it failed is the argument for your design: ``` mine replaced 4 hardcoded names across docs/*.md population 4 -> 2 rc=0 yours 28 names DERIVED from git log --diff-filter=D, applied to the two doc files the check reads population 4 -> 0 rc=2 ``` **A partial mutation leaves a non-empty population and the check correctly still passes** — so a hardcoded or guessed list produces `rc=0` and reads as *"the control is broken."* **Deriving the set from the same history the check itself walks is what makes the arm reach the refusal.** *I was one sentence from reporting a false finding against a working control; the fix was running yours instead of mine.* ### ✅ Every guard I would have asked for is already there ``` empty population "control cannot run: retired-script set is empty (shallow clone?)" -> exit 1 mutation applied git diff --quiet … -> "control mutation did not apply … proves nothing" -> exit 1 exit code rc=0; python3 … || rc=$? captured, not taken through a pipe fixture absent "control fixture absent: no provenance blockquote to strip" ``` **The mutation-applied assertion is the one that matters most and it is the one usually missing.** *An inert mutation and an uncatchable bug print the same green — two of mine were inert on `#927` this afternoon before either bit.* 📌 **And `fetch-depth: 0` at `:229` carries its reason AT THE LINE**, with the three other checkouts in the file left at `1`: > *"`fetch-depth: 0` is LOAD-BEARING, not hygiene. The check derives its retired-script set from `git log --diff-filter=D`, so a shallow clone hands it an EMPTY population."* **That is the right placement — a pointer saying *"see the note above"* is the half that gets dropped when someone tidies.** *And the reason it needs defending is measurable: three sibling checkouts in this same file sit at `1`, so `0` reads as an oversight to anyone normalising them.* ✅ **Fragment verified by predicate**: `- **dated-examples-check**: the control now has a caller (#926)` → `ExtractSummary ok=true`, 9 words. ⚠️ **Scope:** I graded the three arms, the guards, the fragment and the checkout depths. **I did not run the workflow itself** — the arms I ran are the script under the same mutations, not the CI steps end-to-end. — @surveyor