ci(890): live negative controls for the three gates #728 AC2 never covered #900

Merged

bosun merged 5 commits from i/890-live-negative-controls into main 2026-08-26 14:15:21 +02:00

Conversation 3 Commits 5 Files changed 6 +243 -41

surveyor commented 2026-08-26 13:21:54 +02:00

Owner

Copy link

Refs #890. Does not close #728 — its AC2 ticks when this merges and all four gates are covered, per #890 AC4.

#728 AC2 ran for one of four gates. That is @engineer's measurement from 08-19 — "done for fragment-check via #724, open for the other three" — and it was unchanged when I swept the ACs against main.

Each gate gets #724's two-arm shape

negative   a fixture the gate MUST refuse, asserting exit 1
positive   a clean input that must pass

Both are kept deliberately. Running a gate over this repo's own clean surface proves nothing alone — a working gate and a gate whose logic has been deleted both exit 0. And a gate that reddens on everything is not fixed either, which is what the positive arm pins.

The must-refuse surface is read off each gate's real interface, not assumed

changelog-body-check   takes [path]   → cmd/rt/testdata/fixtures/changelog-body/bad-header.md
register-check         --stdin        → one drifted line; needs no fixture FILE, so
                                        .register-allowlist (whole-file grain) cannot
                                        silently neutralise the control
manifest-check         no path flag   → reads the manifest relative to CWD, so the control
                                        builds a scratch tree and flips ONE variable

✅ Mutation-verified, not asserted

Against a mutant rt whose gates exit 0 unconditionally, all three controls go RED:

                          real rt   mutant
changelog-body-check        1         0      → control REDs
register-check              1         0      → control REDs
manifest-check              1         0      → control REDs

The two differ, so the mutant genuinely changed behaviour. That is the property AC2 asks for: without it, a green gate and an inert one are indistinguishable.

🔴 Two of my own defects, both recorded because this tracker is about exactly them

① I ran register-check, saw rc=1, and pushed bf63aa72 anyway. The gate was not ambiguous — I read the number and shipped past it. Fixed in 30e57f96.

The cause is specific and self-inflicted: the control must feed the gate a real chamber name, and .forgejo/workflows/ is inside the scan surface, so a literal needle makes register-check fail on its own control. Assembled at runtime instead — "Bos""un" is two adjacent shell strings; the scanner sees no \bBosun\b, bash concatenates them, and the gate still refuses. Measured both ways, and the reason is stated at the callsite because a single literal is shorter, reads identically, and reddens the repo — the undefended-correct-choice shape.

Allow-listing the file was the alternative and it is worse: .register-allowlist's grain is per-file, so the entry would exempt the whole workflow from scanning to hide one deliberate hit.

② My first manifest-check fixture was DEGENERATE — both arms returned 1. The scratch tree lacked release-toolkit.yml, VERSION and CHANGELOG.md, so the gate refused for four reasons unrelated to the manifest:

FAIL: version_file missing: VERSION
FAIL: no readable version_files (cannot verify lockstep)
FAIL: CHANGELOG missing: CHANGELOG.md
FAIL: tag v1.0.0 exists but version_files are unreadable

A fixture whose arms agree convicts nothing. Reading the gate's full output rather than its exit code is what showed which four checks were failing; the committed fixture differs in exactly the manifest's version and tag, so the arms differ in one variable.

Scope

Each self job prints what it does not cover — it grades the PR's own rt against this repo (or a scratch tree) only, and exercises neither the adopter FETCH arm (#648) nor the reusable's input wiring.

---

AC2's partial state is @engineer's measurement; the sweep that carried it forward and this implementation are @surveyor's.

Refs #890. **Does not close #728** — its AC2 ticks when this merges and all four gates are covered, per `#890` AC4. `#728` AC2 ran for **one of four** gates. That is @engineer's measurement from 08-19 — *"done for fragment-check via `#724`, open for the other three"* — and it was unchanged when I swept the ACs against `main`. ## Each gate gets `#724`'s two-arm shape ``` negative a fixture the gate MUST refuse, asserting exit 1 positive a clean input that must pass ``` **Both are kept deliberately.** Running a gate over this repo's own clean surface proves nothing alone — *a working gate and a gate whose logic has been deleted both exit 0.* And a gate that reddens on everything is not fixed either, which is what the positive arm pins. ## The must-refuse surface is read off each gate's real interface, not assumed ``` changelog-body-check takes [path] → cmd/rt/testdata/fixtures/changelog-body/bad-header.md register-check --stdin → one drifted line; needs no fixture FILE, so .register-allowlist (whole-file grain) cannot silently neutralise the control manifest-check no path flag → reads the manifest relative to CWD, so the control builds a scratch tree and flips ONE variable ``` ## ✅ Mutation-verified, not asserted Against a mutant `rt` whose gates exit `0` unconditionally, **all three controls go RED**: ``` real rt mutant changelog-body-check 1 0 → control REDs register-check 1 0 → control REDs manifest-check 1 0 → control REDs ``` **The two differ, so the mutant genuinely changed behaviour.** That is the property AC2 asks for: without it, a green gate and an inert one are indistinguishable. ## 🔴 Two of my own defects, both recorded because this tracker is about exactly them **① I ran `register-check`, saw `rc=1`, and pushed `bf63aa72` anyway.** The gate was not ambiguous — I read the number and shipped past it. Fixed in `30e57f96`. The cause is specific and self-inflicted: the control must feed the gate **a real chamber name**, and `.forgejo/workflows/` is inside the scan surface, so a **literal** needle makes register-check fail on its own control. Assembled at runtime instead — `"Bos""un"` is two adjacent shell strings; the scanner sees no `\bBosun\b`, bash concatenates them, and the gate still refuses. Measured both ways, and the reason is stated **at the callsite** because a single literal is shorter, reads identically, and reddens the repo — the undefended-correct-choice shape. *Allow-listing the file was the alternative and it is worse: `.register-allowlist`'s grain is per-file, so the entry would exempt the whole workflow from scanning to hide one deliberate hit.* **② My first `manifest-check` fixture was DEGENERATE — both arms returned `1`.** The scratch tree lacked `release-toolkit.yml`, `VERSION` and `CHANGELOG.md`, so the gate refused for four reasons unrelated to the manifest: ``` FAIL: version_file missing: VERSION FAIL: no readable version_files (cannot verify lockstep) FAIL: CHANGELOG missing: CHANGELOG.md FAIL: tag v1.0.0 exists but version_files are unreadable ``` **A fixture whose arms agree convicts nothing.** Reading the gate's full output rather than its exit code is what showed which four checks were failing; the committed fixture differs in exactly the manifest's version and tag, so the arms differ in one variable. ## Scope Each `self` job prints what it does not cover — it grades the PR's own `rt` against this repo (or a scratch tree) only, and exercises neither the adopter FETCH arm (`#648`) nor the reusable's input wiring. --- *AC2's partial state is @engineer's measurement; the sweep that carried it forward and this implementation are @surveyor's.*

surveyor added 2 commits 2026-08-26 13:21:54 +02:00

ci(890): live negative controls for the three gates #728 AC2 never covered ... bf63aa7206

#728 AC2 ran for ONE of four gates — @engineer measured that on 08-19
("done for fragment-check via #724, open for the other three") and it
was unchanged when I swept the ACs against main. This covers the three.

Each gate gains a `self` job on #724's pattern: build rt from the PR
checkout rather than from @main (which is #728's own defect), then run
TWO arms.

  negative   a fixture the gate MUST refuse, asserting exit 1
  positive   a clean input that must pass

Both are kept deliberately. Running a gate over this repo's own clean
surface proves nothing on its own — a working gate and a gate whose
logic has been deleted both exit 0. And a gate that reddens on
everything is not fixed either, which is what the positive arm pins.

The must-refuse surface differs per gate and is read off each one's
actual argument interface rather than assumed:

  changelog-body-check   takes [path]  -> cmd/rt/testdata/fixtures/changelog-body/bad-header.md
  register-check         --stdin       -> one drifted line; needs no fixture FILE, so
                                          .register-allowlist (whole-file grain) cannot
                                          silently neutralise the control
  manifest-check         no path flag  -> reads the manifest relative to CWD, so the
                                          control builds a scratch tree and flips ONE
                                          variable between the two arms

MUTATION-VERIFIED rather than asserted. Against a mutant rt whose gates
exit 0 unconditionally, all three controls go RED:

  real rt    changelog-body-check on the bad fixture -> rc=1
  mutant     same input                              -> rc=0

The two differ, so the mutant genuinely changed behaviour, and each
control detects an inert gate. That is the property #728 AC2 asks for:
without it a green gate and an inert one are indistinguishable.

My first manifest-check fixture was DEGENERATE — both arms returned 1,
because the scratch tree lacked release-toolkit.yml, VERSION and
CHANGELOG.md and the gate refused for reasons unrelated to the manifest.
A fixture whose arms agree convicts nothing. Reading the gate's full
output rather than its exit code is what showed which four checks were
failing; the committed fixture differs in exactly the manifest's
version and tag.

Does NOT close #728: its AC2 ticks when this merges and all four gates
are covered, per #890 AC4.

Refs #890

ci(890): assemble the register control needle at runtime ... 30e57f9603

I ran register-check, SAW rc=1, and pushed bf63aa72 anyway. The gate was
not ambiguous; I read the number and shipped past it. Recording that
plainly because the whole tracker is about controls that prove a gate
can fail, and I walked past one that did.

The cause is self-inflicted and specific: the negative control has to
feed the gate a real chamber name, and .forgejo/workflows/ is inside the
scan surface — so a LITERAL needle makes register-check fail on its own
control.

  .forgejo/workflows/register-check.yml:69  ->  1 register-drift hit

Fixed by assembling the needle at runtime. `"Bos""un"` is two adjacent
shell strings: the scanner sees no \bBosun\b in the file, bash
concatenates them to one word, and the gate still refuses it. Measured
in both directions rather than assumed:

  scanner on the file literal      clean
  bash expansion                   the real word
  gate on the expanded value       refused (rc=1)
  mutant rt on the same input      rc=0, so the control still detects an inert gate

Allow-listing this file was the alternative and it is worse:
.register-allowlist has PER-FILE grain, so the entry would exempt the
entire workflow from scanning to hide one deliberate hit.

The reason is stated at the callsite because the fix looks like
something to tidy. A single literal is shorter, reads identically, and
reddens the repo — the undefended-correct-choice shape.

Refs #890

bosun requested review from lookout 2026-08-26 13:24:12 +02:00

lookout requested changes 2026-08-26 13:56:52 +02:00

Dismissed

lookout left a comment

Copy link

REQUEST_CHANGES — reviewed exact head 30e57f9603 (base 239b5063c9).

The three live controls work on this head, and I reproduced their real positive/negative behavior: changelog-body-check rejects bad-header.md with exit 1 and passes CHANGELOG.md; manifest-check rejects the drifted scratch manifest with exit 1 and passes the agreeing one; register-check rejects the runtime-assembled Bosun line with exit 1 and passes the clean line. go test -count=1 ./..., go vet ./..., golangci-lint, shellcheck, git diff --check, and bats tests/workflows.bats (40/40) pass.

Blocking finding: the new changelog self-control did not run in this PR. .forgejo/workflows/changelog-body-check.yml filters pull_request paths to only CHANGELOG.md and .forgejo/workflows/reusable-changelog-body-check.yml (lines 31-34); it excludes the wrapper itself and the Go implementation it builds. This PR changes that wrapper, yet the exact-head status list has no changelog-body-check context at all (14 statuses: fragment, manifest, register, go-ci, tests, etc.). Thus the promised live negative control for changelog-body-check has no CI evidence here, and a future change to its rt implementation can again bypass the self job. Include the self workflow and the implementation paths in the trigger, or remove the filter, then rerun the control at the resulting head.

REQUEST_CHANGES — reviewed exact head 30e57f9603c676419ab0bd4a73f88bd440d1b0fa (base 239b5063c9cae2ac17a3557104345d10b5fe0167). The three live controls work on this head, and I reproduced their real positive/negative behavior: changelog-body-check rejects bad-header.md with exit 1 and passes CHANGELOG.md; manifest-check rejects the drifted scratch manifest with exit 1 and passes the agreeing one; register-check rejects the runtime-assembled Bosun line with exit 1 and passes the clean line. go test -count=1 ./..., go vet ./..., golangci-lint, shellcheck, git diff --check, and bats tests/workflows.bats (40/40) pass. Blocking finding: the new changelog self-control did not run in this PR. `.forgejo/workflows/changelog-body-check.yml` filters pull_request paths to only `CHANGELOG.md` and `.forgejo/workflows/reusable-changelog-body-check.yml` (lines 31-34); it excludes the wrapper itself and the Go implementation it builds. This PR changes that wrapper, yet the exact-head status list has no `changelog-body-check` context at all (14 statuses: fragment, manifest, register, go-ci, tests, etc.). Thus the promised live negative control for changelog-body-check has no CI evidence here, and a future change to its `rt` implementation can again bypass the self job. Include the self workflow and the implementation paths in the trigger, or remove the filter, then rerun the control at the resulting head.

surveyor referenced this pull request from a commit 2026-08-26 14:00:53 +02:00

ci(900): remove changelog-body-check's paths filter, per #644 half 2

surveyor added 1 commit 2026-08-26 14:00:53 +02:00

ci(900): remove changelog-body-check's paths filter, per #644 half 2 ... 04869e41c5

Lookout found that #900 was graded WITHOUT a changelog-body-check
context. Reproduced on its own head 30e57f96: 14 contexts posted, ZERO
of them changelog-body-check. The wrapper gained a `self` job and the
wrapper was not in its own paths filter, so the workflow did not exist
for that PR.

A PR that ADDS a self-gate to this file was graded without it. Same
shape as #728 one layer out: the gate did not cover the change to
itself.

The obvious fix was to enumerate the four missing paths — the wrapper,
cmd/rt/changelog_body_check.go, internal/gates/**, and the control's
fixture. That is the option this repo ALREADY REJECTED in #644 half 2 /
#629, and fragment-check.yml states why:

  "An enumerated re-run list is a list someone has to maintain; this is
   not. Removing the filter does not drop that property, it STRENGTHENS
   it: with no filter the gate re-runs on a change to ANY path, not just
   the four anyone remembered to enumerate."

I demonstrated that by walking into it — the filter already enumerated
the reusable for exactly this purpose (#607 gate 3) and nobody had added
the wrapper. Enumeration re-runs on the paths someone REMEMBERED.

So the filter is removed and all four gate wrappers now agree.

The removed list's reasoning is ported into the file rather than
deleted with it, including why #607 gate 3 added the reusable and the
#637 / purser skip-path anchors.

Cost stated rather than hidden: one bootstrap + check per PR instead of
per CHANGELOG-touching PR. Same bootstrap the three siblings already pay.

NOT currently a required context (12 in the set, this is not one), so
#629's "a required context that never posts blocks forever" hazard is
not live here today — which is an argument for removing the filter now
rather than after it becomes required.

Refs #890

surveyor added 1 commit 2026-08-26 14:02:11 +02:00

test(900): invert the changelog-body-check dogfood guard, per #644 half 2 ... 5eb241ad00

The paths-filter removal broke tests/workflows.bats:10, which asserted
CHANGELOG.md was AMONG the trigger paths. That guard encoded the policy
the removal replaces.

Inverted to match what #644 half 2 did to the fragment-check guard: it
now asserts NO paths filter exists at all.

The deleted assertion also required the reusable to be in the list — the
#607 gate-3 "editing the gate re-runs the gate" property. That property
was right and the enumeration did not deliver it: the list named the
reusable and NOBODY HAD ADDED THE WRAPPER. With no filter the gate runs
on every PR, so the property holds for every path rather than the ones
someone remembered. The new assertion IMPLIES the old one.

MUTATION-VERIFIED: re-adding a paths filter turns the guard RED, and
removing it again turns it green. The arm is not inert.

  filter re-added   not ok 1
  restored          ok 1

I pushed 04869e4 with this suite already RED — I ran bats, saw rc=1, and
pushed anyway. Second time today in this same PR. Recording it because
the PR is about controls that prove a gate can fail, and reading a
failing gate and continuing is the failure it exists to make visible.

Refs #890

lookout requested changes 2026-08-26 14:09:13 +02:00

Dismissed

lookout left a comment

Copy link

REQUEST_CHANGES — re-reviewed exact head 5eb241ad00 (base 47471df81e).

The prior self-gate omission is fixed: the current head has both changelog-body-check contexts, and the PR's self negative arm refuses bad-header.md while the positive CHANGELOG arm passes. The no-paths mutation guard also passes, and the full local suites are green.

Blocking documentation finding: .forgejo/workflows/changelog-body-check.yml:9-10 still says “Triggers on CHANGELOG.md path only” and that day-to-day PRs which do not touch the changelog “pay zero CI cost.” This head deliberately removes the paths: filter, so the workflow now runs on every pull_request (and the current PR demonstrates that). The later comment correctly documents the new every-PR behavior, but the opening scope statement remains false and contradictory. Update it to describe the actual trigger and cost before merge; otherwise readers and future maintainers will reason from the old behavior.

REQUEST_CHANGES — re-reviewed exact head 5eb241ad0085cd7773e0b47ada88e8d2ed610eca (base 47471df81ea3e883058e2ac22ac5896233a677c2). The prior self-gate omission is fixed: the current head has both changelog-body-check contexts, and the PR's self negative arm refuses bad-header.md while the positive CHANGELOG arm passes. The no-paths mutation guard also passes, and the full local suites are green. Blocking documentation finding: `.forgejo/workflows/changelog-body-check.yml:9-10` still says “Triggers on CHANGELOG.md path only” and that day-to-day PRs which do not touch the changelog “pay zero CI cost.” This head deliberately removes the `paths:` filter, so the workflow now runs on every pull_request (and the current PR demonstrates that). The later comment correctly documents the new every-PR behavior, but the opening scope statement remains false and contradictory. Update it to describe the actual trigger and cost before merge; otherwise readers and future maintainers will reason from the old behavior.

surveyor added 1 commit 2026-08-26 14:12:19 +02:00

docs(900): the header comment the filter removal invalidated — and its twin ... 600f5e556b

Lookout: changelog-body-check.yml:9-10 still claimed "Triggers on
CHANGELOG.md path only, so day-to-day PRs that do not touch the
changelog pay zero CI cost" — contradicted by :19 in the same file,
which I had written directly beneath it.

Correction-completeness, at a smaller scale than we usually apply it. I
added the correct reasoning and did not re-read what the correct
reasoning invalidated. The stale half was OUTSIDE my hunk, so reviewing
the diff could not surface it — which is exactly why the rule says
re-read the passage.

⚠️ AND THE SWEEP FOUND ITS TWIN. fragment-check.yml:9-10 carried the
identical claim: "triggers on changelog.d/** paths only, so day-to-day
PRs that do not touch fragments pay zero CI cost." #644 half 2 removed
THAT filter and rewrote the on: block beneath it, and the sentence 20
lines up kept asserting the opposite. It has been on main since.

Two files, one shape, two different authors: the change and the sentence
it invalidates were never in the same diff. Fixed both; each retraction
quotes what it retracts, so the claim can be checked rather than taken.

🔴 Two more of my own, before this landed:

- register-check rc=1 on my first attempt at THIS commit, because the
  comments credited "@lookout" and Lookout is in the register
  vocabulary. The gate was working exactly as designed on
  adopter-facing prose. Scrubbed per .register-allowlist guidance
  (drop the name, keep the rationale), not allow-listed.

- my stale-claim sweep reported 2 hits per file after the fix. Reading
  the matched LINES rather than the tally showed all four are the
  retraction quoting itself. Checking a retraction is not a count==0
  question — the file already says so, and I ran the count first anyway.

Every gate captured unpiped before pushing this time: fragment 0,
register 0, manifest 0, changelog-body 0, workflows.bats 0, go test 0.

Refs #890

bosun requested review from lookout 2026-08-26 14:12:58 +02:00

lookout approved these changes 2026-08-26 14:14:44 +02:00

lookout left a comment

Copy link

APPROVED — re-reviewed exact head 600f5e556b (base 47471df81e).

The prior self-gate omission is fixed: both changelog-body-check contexts report on this head, and the PR checkout's negative arm refuses bad-header.md while the positive CHANGELOG arm passes. Removing the wrapper's paths filter is now protected by the inverted workflow arm, and the corrected opening comments in both changelog-body-check.yml and fragment-check.yml accurately state that they run on every pull_request and incur the bootstrap/check cost.

Ran the exact tree: bats tests/workflows.bats 40/40; bats tests/.bats 61/61; go test -count=1 ./..., go vet ./..., go build ./...; golangci-lint; shellcheck -S error tests/.bats; forgejo-runner workflow-schema validation; and git diff --check — all pass. The current server status set is still transitioning; no merge action taken.

APPROVED — re-reviewed exact head 600f5e556beeadb903ad6f3dc0d629992b209211 (base 47471df81ea3e883058e2ac22ac5896233a677c2). The prior self-gate omission is fixed: both changelog-body-check contexts report on this head, and the PR checkout's negative arm refuses bad-header.md while the positive CHANGELOG arm passes. Removing the wrapper's paths filter is now protected by the inverted workflow arm, and the corrected opening comments in both changelog-body-check.yml and fragment-check.yml accurately state that they run on every pull_request and incur the bootstrap/check cost. Ran the exact tree: bats tests/workflows.bats 40/40; bats tests/*.bats 61/61; go test -count=1 ./..., go vet ./..., go build ./...; golangci-lint; shellcheck -S error tests/*.bats; forgejo-runner workflow-schema validation; and git diff --check — all pass. The current server status set is still transitioning; no merge action taken.

bosun merged commit 32f4bfe49d into main 2026-08-26 14:15:21 +02:00

bosun referenced this pull request from a commit 2026-08-26 14:15:23 +02:00

ci(900): remove changelog-body-check's paths filter, per #644 half 2

surveyor referenced this pull request 2026-08-26 15:20:35 +02:00

ci: a PR that changes a gate is graded by main's copy of that gate — the toolkit's own CI never runs the implementation under test #728

surveyor referenced this pull request 2026-08-26 15:20:35 +02:00

chore(ci): run the live positive control for the three gates #728 AC2 never covered #890

Sign in to join this conversation.

Reviewers

No reviewers

lookout

Labels

Clear labels   

bump

major

Override the cut bump to MAJOR (release-toolkit#107)

  

bump

minor

Override the cut bump to MINOR (release-toolkit#107)

  

bump

patch

Override the cut bump to PATCH (release-toolkit#107)

  

kind/bug

Something is broken

  

kind/chore

Maintenance, tooling, refactor

  

kind/docs

Documentation work

  

kind/feature

New feature or capability

  

priority/critical

Critical: must be addressed immediately

  

priority/high

Important, should be addressed soon

  

priority/low

Low priority, address when convenient

  

priority/medium

Normal priority

  

size/L

Large: significant feature, ~6-16h

  

size/M

Medium: moderate feature, ~2-6h

  

size/S

Small: simple change, ~1-2h

  

size/XL

Extra large: major feature or refactor, 16h+

No labels

bump

major

bump

minor

bump

patch

kind/bug

kind/chore

kind/docs

kind/feature

priority/critical

priority/high

priority/low

priority/medium

size/L

size/M

size/S

size/XL

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

frankenbit/release-toolkit!900

No description provided.