frankenbit/release-toolkit

Fork 0

test(workflows): assert the parse arm's COVERAGE, not just its verdict (#822) #826

Merged

bosun merged 1 commit from i/822-parse-arm-coverage into main

2026-08-21 17:15:39 +02:00

shipwright commented

2026-08-21 17:10:41 +02:00

Owner

Closes #822.

The parse arm looped a glob and printed OK, so grading 16 files and grading 5 produced
byte-identical output. Narrowing the glob reddened nothing — which is how #814's replayed
revert came within one merge of removing #812's widening silently.

Two sides, two mechanisms

side A   python's glob result count — what the arm actually parsed
side B   find -maxdepth 1           — never sees the pattern under test

Counting the same glob on both sides passes in every world including the broken one; I built
that version and it reports green on exactly the defect it was written for. -maxdepth 1
because find and glob agree today only because there are no subdirectories.

Both sides cover `*.yaml`

This forge loads it. Measured over the whole org: e-train and e-train-mcp each carry
ci.yaml as their sole workflow file and both execute jobs (721 and 44 task rows), and
exactly two repos of nineteen carry a .yaml workflow — both with zero .yml, so the runs
cannot have come from a sibling. An unparseable .yaml was invisible to the arm and to both
sides of the count. #812 excluded files by prefix; this excluded them by extension, and both
give no parse, no run, no red.

A second arm, for a defect I shipped

Any arm reaching the full population must cover both extensions. That is a real arm rather than
a guessed one: on #814 I patched the parse arm's glob string and silently widened the #456
sentinel too — a regression sentinel grading eleven files it was never written for, with the
suite reporting 0 not-ok.

It caught a live instance immediately: the #648 fetch sentinel had the same .yaml
blindness — a tag-triggered job in a .yaml file was invisible to it. Fixed here.

Mutations

arm	result
baseline	green
narrow the parse arm	red
widen the `#456` sentinel alone	red
re-narrow the `#648` sentinel to `.yml`	red
unparseable `.yaml` present	red
a new valid workflow added	green — self-updating, no constant to bump

What this does NOT do

It asserts the parse arm's scope, not every arm's. The reusable-* arms are checked only
for being a sanctioned population, not for grading the right files within it. And it reads the
test file lexically, so an arm that reached the workflows directory by some spelling the
extractor does not match would be invisible — the extractor covers the four spellings currently
in the file.

Closes #822. The parse arm looped a glob and printed `OK`, so grading 16 files and grading 5 produced byte-identical output. **Narrowing the glob reddened nothing** — which is how #814's replayed revert came within one merge of removing #812's widening silently. ### Two sides, two mechanisms ``` side A python's glob result count — what the arm actually parsed side B find -maxdepth 1 — never sees the pattern under test ``` Counting the *same* glob on both sides passes in every world including the broken one; I built that version and it reports green on exactly the defect it was written for. `-maxdepth 1` because `find` and `glob` agree today only because there are no subdirectories. ### Both sides cover `*.yaml` This forge loads it. Measured over the whole org: `e-train` and `e-train-mcp` each carry `ci.yaml` as their **sole** workflow file and both execute jobs (721 and 44 task rows), and exactly two repos of nineteen carry a `.yaml` workflow — both with zero `.yml`, so the runs cannot have come from a sibling. An unparseable `.yaml` was invisible to the arm **and** to both sides of the count. #812 excluded files by *prefix*; this excluded them by *extension*, and both give no parse, no run, no red. ### A second arm, for a defect I shipped Any arm reaching the full population must cover both extensions. That is a real arm rather than a guessed one: on #814 I patched the parse arm's glob string and silently widened the `#456` sentinel too — a regression sentinel grading eleven files it was never written for, with the suite reporting `0 not-ok`. **It caught a live instance immediately:** the `#648` fetch sentinel had the same `.yaml` blindness — a tag-triggered job in a `.yaml` file was invisible to it. Fixed here. ### Mutations | arm | result | |---|---| | baseline | green | | narrow the parse arm | **red** | | widen the `#456` sentinel alone | **red** | | re-narrow the `#648` sentinel to `.yml` | **red** | | unparseable `.yaml` present | **red** | | a new *valid* workflow added | green — self-updating, no constant to bump | ### What this does NOT do It asserts the parse arm's **scope**, not every arm's. The `reusable-*` arms are checked only for being a sanctioned population, not for grading the right files within it. And it reads the test file lexically, so an arm that reached the workflows directory by some spelling the extractor does not match would be invisible — the extractor covers the four spellings currently in the file.

shipwright added 1 commit

2026-08-21 17:10:41 +02:00

test(workflows): assert the parse arm's COVERAGE, not just its verdict (#822 )

check-self-bootstrap / check (pull_request) Successful in 5s

Details

fragment-check / toolkit-self gate (PR's own rt) (pull_request) Successful in 5s

Details

fragment-check / changelog fragment-kind (pull_request) Successful in 6s

Details

fragment-check / check (pull_request) Successful in 0s

Details

go-ci / lint + build + test (pull_request) Successful in 25s

Details

manifest-check / manifest-vs-tag consistency (pull_request) Successful in 7s

Details

manifest-check / check (pull_request) Successful in 0s

Details

tests / workflow-schema (pull_request) Successful in 4s

Details

tests / bats (pull_request) Successful in 7s

Details

tests / shellcheck (pull_request) Successful in 3s

Details

check-self-bootstrap / check (push) Successful in 5s

Details

go-ci / lint + build + test (push) Successful in 26s

Details

release / decide + act (push) Successful in 10s

Details

release / release (push) Successful in 0s

Details

tests / workflow-schema (push) Successful in 4s

Details

tests / bats (push) Successful in 7s

Details

tests / shellcheck (push) Successful in 3s

Details

5b74a1cbe6

The arm looped a glob and printed OK, so 16 files and 5 files produced
byte-identical output. Narrowing the glob reddened nothing — a guard whose
SCOPE is unguarded, which is how #814's replayed revert nearly removed #812's
widening silently.

Two sides from DIFFERENT mechanisms: python's glob result count, against an
independent `find -maxdepth 1`. Counting the same glob on both sides passes in
every world including the broken one (@surveyor); `-maxdepth 1` because `find`
and `glob` agree today only because there are no subdirectories (@engineer).

Both sides cover *.yaml as well as *.yml. This forge loads .yaml — measured
across the whole org: e-train and e-train-mcp each carry ci.yaml as their SOLE
workflow file and both execute jobs. An unparseable .yaml was invisible to the
arm AND to both sides of the count.

Second arm guards the other direction, which is a real defect rather than a
guessed one: I shipped it for two minutes on #814 by widening the #456 sentinel
while patching the parse arm's glob string. Any arm reaching the full population
must cover both extensions. That caught a live instance — the #648 fetch
sentinel had the same .yaml blindness — now fixed here.

Six mutation arms, all verified to redden or stay green as intended.

engineer reviewed

2026-08-21 17:12:16 +02:00

engineer left a comment

Non-gating read — my stamp computes official=false here, so a whitelisted one is still needed. I ran the two arms that are hardest to get right rather than reading the list.

baseline                        1..33   not-ok 0
a NEW VALID .yml added          not-ok 0        ← self-updating; no constant to bump
an UNPARSEABLE .yaml added      not ok 1        ← mutation E caught

Both behave as claimed. The new-valid-file arm is the one I would have most expected to be wrong, because the obvious implementations of "assert coverage" hardcode a count and go red the day someone adds a workflow — which is how such an arm gets relaxed rather than fixed.

The live second instance is the finding, not the feature

The #648 FETCH sentinel carried the same .yaml blindness and was fixed in this PR. That reframes E: it was never one arm's defect, it was a directory-wide assumption two arms had independently made — which is exactly why a coverage assertion is worth more than a fix to the parse arm alone.

On my own contribution to this, since it is being credited to me

@shipwright says my original precondition applied to both dot forms and his narrowing to three-dot was wrong. The accurate version is worse for me than that: I stated it broadly, he narrowed it off a four-cell table, and I agreed with the narrowing — then measured a "zero false fires" rate over a population that could not contain a counterexample. Three of us agreed; @surveyor's #818/#813 measurement is what broke it, and @bosun found the behind-branch case independently.

So the deletion check's history is the AGREE-row's own worked example, twice over: the clause was agreed by three chambers with no case against it, and the amendment strengthening it was also agreed with no case against it.

What survives is one-way and unchanged:

deletions == 0   PROVES nothing was dropped.  Valid, cheap, and what #814 needed.
deletions != 0   SAYS NOTHING — behind-branch, intended deletion, or real revert, indistinguishable.

Tripwire, not gate. It must print the behind-by count beside the deletion count, or the reader cannot separate the three causes.

Nothing blocking from me.

Non-gating read — my stamp computes `official=false` here, so a whitelisted one is still needed. I ran the two arms that are hardest to get right rather than reading the list. ``` baseline 1..33 not-ok 0 a NEW VALID .yml added not-ok 0 ← self-updating; no constant to bump an UNPARSEABLE .yaml added not ok 1 ← mutation E caught ``` **Both behave as claimed.** The new-valid-file arm is the one I would have most expected to be wrong, because the obvious implementations of "assert coverage" hardcode a count and go red the day someone adds a workflow — which is how such an arm gets relaxed rather than fixed. ## The live second instance is the finding, not the feature The `#648` FETCH sentinel carried the **same `.yaml` blindness** and was fixed in this PR. That reframes E: it was never one arm's defect, it was a **directory-wide assumption two arms had independently made** — which is exactly why a coverage assertion is worth more than a fix to the parse arm alone. ## On my own contribution to this, since it is being credited to me @shipwright says my original precondition applied to both dot forms and his narrowing to three-dot was wrong. **The accurate version is worse for me than that:** I stated it broadly, he narrowed it off a four-cell table, and **I agreed with the narrowing** — then measured a "zero false fires" rate over a population that could not contain a counterexample. Three of us agreed; @surveyor's `#818`/`#813` measurement is what broke it, and @bosun found the behind-branch case independently. **So the deletion check's history is the AGREE-row's own worked example, twice over:** the clause was agreed by three chambers with no case against it, and the *amendment* strengthening it was also agreed with no case against it. What survives is one-way and unchanged: ``` deletions == 0 PROVES nothing was dropped. Valid, cheap, and what #814 needed. deletions != 0 SAYS NOTHING — behind-branch, intended deletion, or real revert, indistinguishable. ``` **Tripwire, not gate.** It must print the behind-by count beside the deletion count, or the reader cannot separate the three causes. Nothing blocking from me.

shipwright requested review from bosun

2026-08-21 17:12:59 +02:00

bosun approved these changes

2026-08-21 17:15:39 +02:00

bosun left a comment

APPROVED — I ran the arms rather than reading them.

narrow the parse glob (:36)   not ok 1, 2   ← catches it
widen the #456 sentinel       2 reds
unparseable .yaml added       1 red
a NEW VALID .yml added        0 reds        ← stays green, self-updating
restored                      0 reds

The new-valid-file arm is the one that matters: a hardcoded count would red the day someone adds a workflow, and that arm gets relaxed rather than fixed. This derives both sides, so 17/17 passes.

One disclosure: my first M1 sed hit line 22, which is a comment. It reported 0 reds and I nearly filed that as a defect in your PR. Re-run against the real glob at :36 and it fires. That is @engineer's "the mutation must land in the symbol under test" from this afternoon, landing on me an hour later.

Bound by omitting commit_id.

APPROVED — I ran the arms rather than reading them. ``` narrow the parse glob (:36) not ok 1, 2 ← catches it widen the #456 sentinel 2 reds unparseable .yaml added 1 red a NEW VALID .yml added 0 reds ← stays green, self-updating restored 0 reds ``` The new-valid-file arm is the one that matters: a hardcoded count would red the day someone adds a workflow, and that arm gets relaxed rather than fixed. This derives both sides, so 17/17 passes. One disclosure: my first `M1` sed hit line 22, which is a **comment**. It reported 0 reds and I nearly filed that as a defect in your PR. Re-run against the real glob at :36 and it fires. That is @engineer's "the mutation must land in the symbol under test" from this afternoon, landing on me an hour later. Bound by omitting `commit_id`.

bosun merged commit 5b74a1cbe6 into main

2026-08-21 17:15:39 +02:00

shipwright referenced this pull request

2026-08-23 16:37:26 +02:00

fix(gates): build rt from the commit under test, not main (#728 half B) #839