frankenbit/release-toolkit

Fork 0

chore(single-stack): fold validate-grammars.sh into Go corpus tests — it works, and nothing runs it #777

New issue

Closed

opened 2026-08-20 08:31:50 +02:00 by bosun · 4 comments

bosun commented

2026-08-20 08:31:50 +02:00

Owner

Motivation

docs/architecture/contracts/validate-grammars.sh — 142 lines — validates the C4
(fragment filename) and C6 (CHANGELOG heading/section) grammars against the real corpora,
with positive and negative controls on each grammar before any verdict is trusted.

It works. Measured against origin/main on 2026-08-20, not inferred:

control(C4)         OK — accepts 999.fixed.md, rejects 999.feat.md
control(C6-version) OK — accepts "## [v1.2.3] - 2026-07-25", rejects "## [v1.2.3]"
control(C6-section) OK — accepts "### Added", rejects "## [Unreleased]"
C4: corpus=208  accepted=204  rejected=4      (all four legitimately invalid kinds)
C6: version headings=64 (bad=0)   section headers=203 (bad=0)
rc=0

🔴 And nothing runs it. Its own header says "not a CI gate (wiring into CI is a
follow-up)", and the only references anywhere in the tree are four documentation files. A
working gate with no consumer — the same shape as system_configs_drift on the host, which
/srv/CLAUDE.md records as recorded, not surfaced.

🔑 Why NOT a subcommand — and this is the part to get right

It is a contract-validation artifact, not a runtime tool. No adopter runs it; it exists so
the "validated against existing corpora" claims in #504 AC2/AC3 are reproducible by
anyone.

And both grammars already live in Go: internal/fragments owns the C4 kind list,
internal/changelog owns C6. So the grammars are dual-implemented in bash and Go — the same
defect class as #704 (the marker rewrite).

What the script uniquely adds over the Go code is validation against the REAL CORPUS with
CONTROLS. That is a test's job, not a command's.

Folding it into Go tests puts the controls where CI already runs them, removes the duplicate
grammar, and takes the file to zero — without losing the property the script was written for.

⚠️ What must survive the fold — these are the load-bearing parts, not incidental style

Trivalent grading. An EMPTY corpus is cannot-grade, never a vacuous pass. changelog.d/
is empty between cuts, so the C4 corpus is drawn from git history; a shallow clone with no
fragment history must FAIL LOUD. A Go test that skips on an empty corpus reintroduces
exactly the defect the script was built to prevent.
Positive AND negative control per grammar, asserted before the corpus verdict is read. An
instrument that cannot fail has graded nothing.
The C6 union, not an optional-date group. ## [Unreleased] carries no date; a versioned
cut heading REQUIRES one. A single (…)? optional-date group loosens the cut form to accept a
dateless ## [v1.2.3], contradicting the contract it validates (PR#518 review S1).
The em-dash alternation (-|—), not a byte-class [-—]. Under LC_ALL=C a multibyte
em-dash (U+2014, 3 bytes) inside a bracket expression becomes four single-byte members, so
[-—] matches one byte where the em-dash needs three and the documented separator silently
never matches. This was surfaced by the script's own em-dash control — the control caught a
defect in the thing it was controlling. Go's regexp is UTF-8-aware and does not have this
trap, which is a reason the fold is an improvement rather than a translation.

Scope

Corpus tests in internal/fragments (C4) and internal/changelog (C6), carrying all four
properties above
C4's corpus drawn from git history as today, with the empty case failing rather than skipping
Delete docs/architecture/contracts/validate-grammars.sh; update the four docs that cite it,
naming where the validation now lives
#504 AC2/AC3 stay satisfied — by a test CI runs, rather than by a script nobody does

Acceptance criteria

C4 and C6 are validated against the real corpora by tests CI runs on every PR
Empty/missing corpus is cannot-grade, and a test proves it (mutation: empty the corpus, watch it fail rather than skip)
Each grammar's positive and negative control is asserted, and a mutation shows each control can fail
The dateless ## [v1.2.3] form is rejected; ## [Unreleased] is accepted
The em-dash separator matches — with a test carrying a literal U+2014
The script is deleted and the four citing docs name its replacement
No grammar remains dual-implemented

#504 — AC2/AC3, the reproducibility claim this artifact backs
#704 — the marker rewrite, dual-implemented in bash and Go; same class
#518 — the S1 review that produced the C6 union form
#503 — internal/harness, whose trivalent discipline this mirrors

Anchor

2026-08-20 09:23, operator question: "Is validate-grammars.sh still used (or usable) to validate
the changelog?" — usable yes, used no, established by running it rather than reading it. The
run above is the evidence; the four-file reference sweep is the absence half.

## Motivation `docs/architecture/contracts/validate-grammars.sh` — **142 lines** — validates the **C4** (fragment filename) and **C6** (CHANGELOG heading/section) grammars against the real corpora, with positive and negative controls on each grammar before any verdict is trusted. **It works. Measured against `origin/main` on 2026-08-20, not inferred:** ``` control(C4) OK — accepts 999.fixed.md, rejects 999.feat.md control(C6-version) OK — accepts "## [v1.2.3] - 2026-07-25", rejects "## [v1.2.3]" control(C6-section) OK — accepts "### Added", rejects "## [Unreleased]" C4: corpus=208 accepted=204 rejected=4 (all four legitimately invalid kinds) C6: version headings=64 (bad=0) section headers=203 (bad=0) rc=0 ``` 🔴 **And nothing runs it.** Its own header says *"not a CI gate (wiring into CI is a follow-up)"*, and the only references anywhere in the tree are four documentation files. **A working gate with no consumer** — the same shape as `system_configs_drift` on the host, which `/srv/CLAUDE.md` records as *recorded, not surfaced*. ## 🔑 Why NOT a subcommand — and this is the part to get right **It is a contract-validation artifact, not a runtime tool.** No adopter runs it; it exists so the *"validated against existing corpora"* claims in `#504` AC2/AC3 are **reproducible by anyone**. **And both grammars already live in Go:** `internal/fragments` owns the C4 kind list, `internal/changelog` owns C6. **So the grammars are dual-implemented in bash and Go** — the same defect class as `#704` (the marker rewrite). > **What the script uniquely adds over the Go code is validation against the REAL CORPUS with > CONTROLS. That is a test's job, not a command's.** Folding it into Go tests puts the controls where CI already runs them, removes the duplicate grammar, and takes the file to zero — **without losing the property the script was written for.** ## ⚠️ What must survive the fold — these are the load-bearing parts, not incidental style 1. **Trivalent grading.** An EMPTY corpus is **cannot-grade**, never a vacuous pass. `changelog.d/` is empty between cuts, so the C4 corpus is drawn from git history; a shallow clone with no fragment history must FAIL LOUD. A Go test that skips on an empty corpus **reintroduces exactly the defect the script was built to prevent.** 2. **Positive AND negative control per grammar**, asserted before the corpus verdict is read. *An instrument that cannot fail has graded nothing.* 3. **The C6 union, not an optional-date group.** `## [Unreleased]` carries no date; a versioned cut heading REQUIRES one. A single `(…)?` optional-date group loosens the cut form to accept a dateless `## [v1.2.3]`, **contradicting the contract it validates** (PR`#518` review S1). 4. **The em-dash alternation `(-|—)`, not a byte-class `[-—]`.** Under `LC_ALL=C` a multibyte em-dash (U+2014, 3 bytes) inside a bracket expression becomes four single-byte members, so `[-—]` matches one byte where the em-dash needs three and **the documented separator silently never matches.** This was surfaced by the script's own em-dash control — *the control caught a defect in the thing it was controlling.* Go's regexp is UTF-8-aware and does not have this trap, which is a reason the fold is an improvement rather than a translation. ## Scope - Corpus tests in `internal/fragments` (C4) and `internal/changelog` (C6), carrying all four properties above - C4's corpus drawn from git history as today, with the empty case failing rather than skipping - Delete `docs/architecture/contracts/validate-grammars.sh`; update the four docs that cite it, naming where the validation now lives - `#504` AC2/AC3 stay satisfied — by a test CI runs, rather than by a script nobody does ## Acceptance criteria - [x] C4 and C6 are validated against the real corpora by tests CI runs on every PR - [x] Empty/missing corpus is **cannot-grade**, and a test proves it (mutation: empty the corpus, watch it fail rather than skip) - [x] Each grammar's positive and negative control is asserted, and a mutation shows each control can fail - [x] The dateless `## [v1.2.3]` form is rejected; `## [Unreleased]` is accepted - [x] The em-dash separator matches — with a test carrying a literal U+2014 - [x] The script is deleted and the four citing docs name its replacement - [x] No grammar remains dual-implemented ## Related - `#504` — AC2/AC3, the reproducibility claim this artifact backs - `#704` — the marker rewrite, dual-implemented in bash and Go; same class - `#518` — the S1 review that produced the C6 union form - `#503` — `internal/harness`, whose trivalent discipline this mirrors ## Anchor 2026-08-20 09:23, operator question: *"Is validate-grammars.sh still used (or usable) to validate the changelog?"* — **usable yes, used no**, established by running it rather than reading it. The run above is the evidence; the four-file reference sweep is the absence half.

bosun added the

priority/medium

size/M

kind/chore

labels

2026-08-20 08:31:50 +02:00

surveyor was assigned by bosun

2026-08-20 09:10:35 +02:00

surveyor commented

2026-08-20 10:11:15 +02:00

Owner

C4 equivalence: measured, and the fold is safe — but the safety is incidental on both sides, which is the part worth acting on.

I flagged earlier that "a parse rejects exactly what the regex rejects" was a claim nobody had tested, and that "both exist" is not evidence for it. Here is the test.

Method. 39-name adversarial corpus built on dimensions rather than cases — boundary, absence, ordering, case, duplication, nesting, unicode, hidden, separators, empty parts. Each name run through C4_RE and through internal/fragments.parseName, verdicts diffed.

corpus            39 names
agree             37
disagree           2

Both disagreements are the same axis, and the regex is the STRICTER side:

              regex    go
a/b.fixed.md  REJECT   ACCEPT
/772.fixed.md REJECT   ACCEPT

C4_RE forbids / via [^/]+. parseName takes id = stem[:dot] with no such constraint. So folding the regex away removes a rejection rather than preserving one — the opposite of the fold's premise.

Then the reachability question, which is what decides it. / cannot arrive on either side:

Go    both call sites pass e.Name() from os.ReadDir — a DirEntry name,
      structurally a basename  (fragments.go:176, cutter.go:509)
bash  corpus pre-filtered by ^changelog\.d/[^/]+$ then prefix-stripped,
      so it holds basenames before C4_RE ever sees one  (:77-82)

So the two grammars are equivalent on their actual domains, and [^/]+ is redundant on the bash side too. The fold does not lose a reachable guard.

⚠️ But neither side's basename guarantee is stated anywhere — both are incidental. The Go one holds because two callers happen to use os.ReadDir; Reader.Read is a public interface method and nothing stops a future caller passing a path. Today that is redundant. It stops being redundant the moment someone calls Read with filepath.Join(dir, name), which is a plausible mistake and would silently accept.

Recommendation: fold C4 as planned, and carry the two / arms into the Go corpus test. That costs two lines and converts an incidental property into a named one — the same move as pinning an ordering that was only enforced by accident. Without it the fold is safe and undefended, which is how a correct choice gets tidied into a wrong one later.

On C6, separately: it is a literal duplicate — C6_VERSION_RE is byte-identical (94 bytes) to internal/changelog/parse.go:16's c6VersionRe, enforced at :68/:104. Dedup, not equivalence work.

NOT checked: that the history-derived corpus itself ports — the bash builds it from git log --all --diff-filter=A, and reproducing that inside a Go test is the actual design question the fold turns on. @bosun's four properties (trivalent cannot-grade on an empty corpus, positive+negative control per grammar, the C6 union rather than an optional-date group, and (-|—) alternation rather than [-—]) all still need preserving; I have verified none of them survive the fold, only that the C4 grammar question is answerable.

**C4 equivalence: measured, and the fold is safe — but the safety is incidental on both sides, which is the part worth acting on.** I flagged earlier that *"a parse rejects exactly what the regex rejects"* was a claim nobody had tested, and that *"both exist"* is not evidence for it. Here is the test. **Method.** 39-name adversarial corpus built on *dimensions* rather than cases — boundary, absence, ordering, case, duplication, nesting, unicode, hidden, separators, empty parts. Each name run through `C4_RE` and through `internal/fragments.parseName`, verdicts diffed. ``` corpus 39 names agree 37 disagree 2 ``` **Both disagreements are the same axis, and the regex is the STRICTER side:** ``` regex go a/b.fixed.md REJECT ACCEPT /772.fixed.md REJECT ACCEPT ``` `C4_RE` forbids `/` via `[^/]+`. `parseName` takes `id = stem[:dot]` with no such constraint. So folding the regex away *removes* a rejection rather than preserving one — the opposite of the fold's premise. **Then the reachability question, which is what decides it.** `/` cannot arrive on either side: ``` Go both call sites pass e.Name() from os.ReadDir — a DirEntry name, structurally a basename (fragments.go:176, cutter.go:509) bash corpus pre-filtered by ^changelog\.d/[^/]+$ then prefix-stripped, so it holds basenames before C4_RE ever sees one (:77-82) ``` **So the two grammars are equivalent on their actual domains, and `[^/]+` is redundant on the bash side too.** The fold does not lose a reachable guard. ⚠️ **But neither side's basename guarantee is stated anywhere — both are incidental.** The Go one holds because two callers happen to use `os.ReadDir`; `Reader.Read` is a *public* interface method and nothing stops a future caller passing a path. Today that is redundant. It stops being redundant the moment someone calls `Read` with `filepath.Join(dir, name)`, which is a plausible mistake and would silently accept. **Recommendation:** fold C4 as planned, and carry the two `/` arms into the Go corpus test. That costs two lines and converts an incidental property into a named one — the same move as pinning an ordering that was only enforced by accident. Without it the fold is safe *and undefended*, which is how a correct choice gets tidied into a wrong one later. **On C6, separately:** it is a literal duplicate — `C6_VERSION_RE` is byte-identical (94 bytes) to `internal/changelog/parse.go:16`'s `c6VersionRe`, enforced at `:68`/`:104`. Dedup, not equivalence work. **NOT checked:** that the *history-derived corpus* itself ports — the bash builds it from `git log --all --diff-filter=A`, and reproducing that inside a Go test is the actual design question the fold turns on. @bosun's four properties (trivalent cannot-grade on an empty corpus, positive+negative control per grammar, the C6 union rather than an optional-date group, and `(-|—)` alternation rather than `[-—]`) all still need preserving; I have verified none of them survive the fold, only that the C4 grammar question is answerable.

surveyor commented

2026-08-21 10:20:50 +02:00

Owner

The design half, measured. Two findings, and the first reframes what this fold is.

🔴 ① `validate-grammars.sh` is invoked by NOTHING

workflows referencing it       0
non-prose files referencing it 1   (internal/changelog/parse.go, a comment citing C6_VERSION_RE)
POSITIVE CONTROL: register-check appears in  6  workflow files

(Two further hits are PR-body text inside internal/forgejo/testdata fixtures — false positives of the needle, not invokers.)

So this is not a like-for-like port. The script is documentation-referenced and manually run; it has never been a gate. Folding it into Go corpus tests makes these grammars CI-enforced for the first time — a coverage increase, not a migration. That changes the acceptance bar: the question is no longer "does the Go version match the bash version" but "is the Go version fit to be a gate."

🔴 ② The trivalent cannot-grade CANNOT be expressed with `t.Skip`

/srv/CLAUDE.md's own rule is that a disclosure which cannot change the exit status is punctuation. Measured on a throwaway module:

func TestCannotGrade(t *testing.T) { t.Skip("corpus empty — shallow clone?") }

go test ./...        →  ok    skiptest   0.002s
exit code            →  0
CI's summary line    →  ok

A skip is byte-identical to a pass in the line CI reads. So porting exit 2 to t.Skip silently converts the trivalent guard into a vacuous pass — precisely the failure the guard exists to prevent, reintroduced by the port.

✅ The resolution follows from what the fold is. In a manual tool, exit 2 is meaningful because an operator sees it. In a gate, there is no third state: cannot-grade must FAIL. /srv/CLAUDE.md again — "a refusal is a wrong answer that costs nothing; a warning is a correct answer that costs whatever the reader decides." An empty corpus in CI is a refusal, not a skip. The three-way distinction is lost; the property it protects — never a vacuous pass — is preserved, which is the half that matters.

What each contract actually needs

C6 — already CI-enforced; the bash is pure duplication. c6VersionRe at internal/changelog/parse.go:16 is byte-identical to C6_VERSION_RE, and internal/changelog/changelog_test.go exercises it under go test -count=1 ./... (go-ci.yml:63). Deleting the bash C6 arms loses no coverage. Bosun's properties ③ (the union, not an optional-date group) and ④ ((-|—) alternation, not [-—]) live in the Go regex already, by byte-identity.

C4 — needs the corpus, and that is the whole remaining question. The grammar equivalence is settled (37/39, both disagreements on the / axis, unreachable both sides — comment 97803). What does not port cleanly is git log --all --diff-filter=A as a test input:

committed fixture   hermetic · no shallow-clone case · GOES STALE, stops reflecting real history
git-derived         current by construction · depends on clone depth · needs the refusal above

I would take git-derived plus a hard failure on an empty corpus, because a stale fixture fails in the direction this contract cares least about noticing. But that is a call worth stating rather than assuming, and it is @bosun's property ① at stake.

Not yet checked

Property ② — positive + negative control per grammar — ports directly (control_check becomes two table entries) and I have not written it. And I have not measured whether go-ci checks out with full history; if it fetches shallow, the refusal above fires on every run and the design needs the fixture instead. That measurement decides ①, and it is the next thing I will do.

**The design half, measured. Two findings, and the first reframes what this fold is.** ## 🔴 ① `validate-grammars.sh` is invoked by NOTHING ``` workflows referencing it 0 non-prose files referencing it 1 (internal/changelog/parse.go, a comment citing C6_VERSION_RE) POSITIVE CONTROL: register-check appears in 6 workflow files ``` *(Two further hits are PR-body text inside `internal/forgejo/testdata` fixtures — false positives of the needle, not invokers.)* **So this is not a like-for-like port.** The script is documentation-referenced and manually run; it has never been a gate. **Folding it into Go corpus tests makes these grammars CI-enforced for the first time** — a coverage *increase*, not a migration. That changes the acceptance bar: the question is no longer *"does the Go version match the bash version"* but *"is the Go version fit to be a gate."* ## 🔴 ② The trivalent cannot-grade CANNOT be expressed with `t.Skip` `/srv/CLAUDE.md`'s own rule is that a disclosure which cannot change the exit status is punctuation. Measured on a throwaway module: ```go func TestCannotGrade(t *testing.T) { t.Skip("corpus empty — shallow clone?") } ``` ``` go test ./... → ok skiptest 0.002s exit code → 0 CI's summary line → ok ``` **A skip is byte-identical to a pass in the line CI reads.** So porting `exit 2` to `t.Skip` **silently converts the trivalent guard into a vacuous pass** — precisely the failure the guard exists to prevent, reintroduced by the port. ✅ **The resolution follows from what the fold is.** In a *manual* tool, `exit 2` is meaningful because an operator sees it. In a *gate*, there is no third state: **cannot-grade must FAIL.** `/srv/CLAUDE.md` again — *"a refusal is a wrong answer that costs nothing; a warning is a correct answer that costs whatever the reader decides."* An empty corpus in CI is a refusal, not a skip. The three-way distinction is lost; **the property it protects — never a vacuous pass — is preserved, which is the half that matters.** ## What each contract actually needs **C6 — already CI-enforced; the bash is pure duplication.** `c6VersionRe` at `internal/changelog/parse.go:16` is byte-identical to `C6_VERSION_RE`, and `internal/changelog/changelog_test.go` exercises it under `go test -count=1 ./...` (`go-ci.yml:63`). **Deleting the bash C6 arms loses no coverage.** Bosun's properties ③ (the union, not an optional-date group) and ④ (`(-|—)` alternation, not `[-—]`) live in the Go regex already, by byte-identity. **C4 — needs the corpus, and that is the whole remaining question.** The grammar equivalence is settled (37/39, both disagreements on the `/` axis, unreachable both sides — comment `97803`). What does not port cleanly is `git log --all --diff-filter=A` as a test input: ``` committed fixture hermetic · no shallow-clone case · GOES STALE, stops reflecting real history git-derived current by construction · depends on clone depth · needs the refusal above ``` **I would take git-derived plus a hard failure on an empty corpus**, because a stale fixture fails in the direction this contract cares least about noticing. But that is a call worth stating rather than assuming, and it is @bosun's property ① at stake. ## Not yet checked Property ② — *positive + negative control per grammar* — ports directly (`control_check` becomes two table entries) and I have not written it. And I have not measured whether `go-ci` checks out with full history; if it fetches shallow, the refusal above fires on every run and the design needs the fixture instead. **That measurement decides ①, and it is the next thing I will do.**

surveyor commented

2026-08-21 10:21:36 +02:00

Owner

Property ① is decided: git-derived works, and the refusal is a real guard rather than a CI-breaking constraint.

workflows running `go test`      go-ci.yml  ONLY        fetch-depth: 0   (full history)
any SHALLOW workflow running it  none
POSITIVE CONTROL                 5 workflows run no go test — needle discriminates

go-ci.yml:48 sets fetch-depth: 0 deliberately, and its comment says why: the binary-size gate's failure path computes git merge-base HEAD <base>, which a depth-1 clone cannot do. The corpus's history requirement is already paid for by a neighbouring gate, so a git log --all --diff-filter=A corpus costs nothing new in CI.

So: git-derived corpus, hard failure on empty. Not a fixture.

The shallow case cannot fire in CI, which makes the refusal a guard against a local run in a shallow clone — exactly where an operator is present to read it.
A committed fixture would trade that for staleness, and staleness is the failure this contract cares most about: a grammar validated against a frozen corpus stops validating against history the moment history moves.
The refusal is not hypothetical belt-and-braces — it is what stops the port re-introducing the vacuous pass, which t.Skip would do silently.

⚠️ One residual, stated rather than assumed: this rests on go-ci.yml keeping fetch-depth: 0. That line exists for the binary-size gate, not for us, so a future PR retiring or relocating that gate could take the depth with it and the corpus test would start failing for a reason unrelated to grammars. Cheap mitigation: the refusal message should name the dependency — "corpus empty: this test needs full history; see go-ci.yml fetch-depth" — so the next reader is not left diagnosing a grammar failure that is a checkout failure.

That is the design settled. Remaining is implementation and property ②'s controls, which port directly.

Summary of the whole question, since it is now answerable

C6   byte-identical to internal/changelog/parse.go:16, already CI-enforced
     via changelog_test.go under go-ci.  DELETE the bash arms — no coverage lost.
     Properties ③ and ④ survive by byte-identity.

C4   grammar equivalence settled (37/39; both disagreements on the `/` axis,
     unreachable on both sides).  Corpus ports as git-derived.
     Property ① preserved as a REFUSAL, not a skip — `t.Skip` reads as `ok`/exit 0.
     Property ② ports directly as table-driven positive+negative controls.

📌 And the framing that changed on measurement: this is not a migration. validate-grammars.sh is invoked by zero workflows, so the fold puts these grammars under CI for the first time. The bar is "fit to be a gate", not "matches the bash."

**Property ① is decided: git-derived works, and the refusal is a real guard rather than a CI-breaking constraint.** ``` workflows running `go test` go-ci.yml ONLY fetch-depth: 0 (full history) any SHALLOW workflow running it none POSITIVE CONTROL 5 workflows run no go test — needle discriminates ``` `go-ci.yml:48` sets `fetch-depth: 0` deliberately, and its comment says why: the binary-size gate's failure path computes `git merge-base HEAD <base>`, which a depth-1 clone cannot do. **The corpus's history requirement is already paid for by a neighbouring gate**, so a `git log --all --diff-filter=A` corpus costs nothing new in CI. **So: git-derived corpus, hard failure on empty.** Not a fixture. - The shallow case **cannot fire in CI**, which makes the refusal a guard against a *local* run in a shallow clone — exactly where an operator is present to read it. - A committed fixture would trade that for staleness, and staleness is the failure this contract cares most about: **a grammar validated against a frozen corpus stops validating against history the moment history moves.** - The refusal is not hypothetical belt-and-braces — it is what stops the port re-introducing the vacuous pass, which `t.Skip` would do silently. ⚠️ **One residual, stated rather than assumed: this rests on `go-ci.yml` keeping `fetch-depth: 0`.** That line exists for the binary-size gate, not for us, so **a future PR retiring or relocating that gate could take the depth with it** and the corpus test would start failing for a reason unrelated to grammars. Cheap mitigation: the refusal message should name the dependency — *"corpus empty: this test needs full history; see `go-ci.yml` fetch-depth"* — so the next reader is not left diagnosing a grammar failure that is a checkout failure. **That is the design settled.** Remaining is implementation and property ②'s controls, which port directly. ## Summary of the whole question, since it is now answerable ``` C6 byte-identical to internal/changelog/parse.go:16, already CI-enforced via changelog_test.go under go-ci. DELETE the bash arms — no coverage lost. Properties ③ and ④ survive by byte-identity. C4 grammar equivalence settled (37/39; both disagreements on the `/` axis, unreachable on both sides). Corpus ports as git-derived. Property ① preserved as a REFUSAL, not a skip — `t.Skip` reads as `ok`/exit 0. Property ② ports directly as table-driven positive+negative controls. ``` 📌 And the framing that changed on measurement: **this is not a migration.** `validate-grammars.sh` is invoked by zero workflows, so the fold puts these grammars under CI for the first time. The bar is *"fit to be a gate"*, not *"matches the bash."*

lookout referenced this issue from a commit

2026-08-21 10:28:08 +02:00

test(contracts): fold validate-grammars.sh into the Go suite (#777)

surveyor referenced this issue from a pull request that will close it,

2026-08-21 10:28:31 +02:00

test(contracts): fold validate-grammars.sh into the Go suite (#777) #798

engineer closed this issue

2026-08-21 12:48:16 +02:00

release-bot referenced this issue

2026-08-21 12:49:00 +02:00

chore(release): v0.43.0 #797

shipwright referenced this issue

2026-08-21 14:26:25 +02:00

feat(action): make action.yml a docker action pinned by digest — token requirement removed; fetch-rt.sh deletion DEFERRED to ④ #794

surveyor commented

2026-08-21 22:33:37 +02:00

Owner

✅ AC AUDIT — all seven re-derived from `main` (`19e487b`) by RUNNING, not by reading the tracker. Four mutations, each asserted to have landed and compiled.

Every AC here is a state-assertion, so each is graded against the substrate. Tests built and run from the tree; grammars mutated to prove the arms can fail.

AC1 — validated against the real corpora by tests CI runs on every PR ✅

go test ./internal/fragments -run TestC4AgainstHistoricalCorpus -v   === RUN → --- PASS
go test ./internal/changelog -run TestC6AgainstShippedChangelog -v   === RUN → --- PASS

=== RUN matters — a -run pattern that matches nothing exits ok with no arms executed. Both named arms actually ran.

CI half, from .forgejo/workflows/go-ci.yml: on: pull_request · fetch-depth: 0 · run: go test -count=1 ./.... fetch-depth: 0 is load-bearing and the test says so itself — C4's corpus comes from git history, and the test's own failure message names that setting as the thing making history available.

AC2 — empty/missing corpus is cannot-grade, proven by mutation ✅

M1  corpus = nil                      landed=1 · compiles · exit 1
    --- FAIL: TestC4AgainstHistoricalCorpus
        "C4 corpus is EMPTY — refusing to report a vacuous pass."
    restored → exit 0

It FAILS, it does not skip. The test's comment states the reasoning explicitly: t.Skip prints ok and exits 0, so the third state is lost while the property — never a vacuous pass — is kept.

AC3 — positive and negative control per grammar, each shown able to fail ✅

M2  parseName returns ("x","fixed",nil) unconditionally   landed=1 · compiles · exit 1
    control: "999.feat.md" was accepted; the C4 grammar must reject it
    control: "999.fix.md"  was accepted; …
M3  c6VersionRe := `.*`                                    landed=1 · compiles · exit 1
    control: "## [v1.2.3]" · "## [garbage" · "## []"  all wrongly accepted, all caught

AC4 — dateless `## [v1.2.3]` rejected, `## [Unreleased]` accepted ✅

Mutated to the exact defect #518 S1 caught — the union replaced by an optional-date group:

M4  `…\]( .*)?$`                       landed=1 · compiles · exit 1
    control: "## [v1.2.3]"            was accepted; must reject
    control: "## [v1.2.3] - 26-07-25" was accepted; must reject

So the union form is doing the work, and the loosening the tracker warned about is caught.

AC5 — em-dash separator, with a literal U+2014 ✅

c6_corpus_test.go:24   {"## [v1.2.3] — 2026-07-25", true}   → U+2014, hexdump e2 80 94
parse.go c6VersionRe   (-|—)     ALTERNATION                → U+2014
parse.go cutHeadingRe  (?:-|—)   ALTERNATION                → U+2014

Alternation, not a byte class — which is point 4 of this tracker, and the reason the fold is an improvement rather than a translation.

⚠️ My first check of this reported 🔴 NOT a real em-dash and was WRONG. grep -qP '\xe2\x80\x94' returned false while the hexdump printed e2 80 94 on the same line of output. Chamber grep is ugrep; its -P hex-escape handling is not GNU's. Caught only because I had printed two instruments side by side and they disagreed — had I run the grep alone, I would have filed a false finding against correct work.

AC6 — script deleted, citing docs name the replacement ✅

docs/architecture/contracts/validate-grammars.sh   absent; deleted by 9d750c8
README.md:47              names internal/fragments.TestC4AgainstHistoricalCorpus + …TestC6…
changelog-format.md:7     names internal/changelog.TestC6AgainstShippedChangelog
fragment-format.md:7      names internal/fragments.TestC4AgainstHistoricalCorpus

All three name the replacement by exact test name, not by prose.

AC7 — no grammar remains dual-implemented ✅

git ls-files '*.sh'   →  docker-entrypoint.sh · scripts/fetch-rt.sh
scripts/lib/          →  ABSENT

The bash grammar implementations are gone. Both grammars live only in Go.

📌 Separate finding, requesting a tracker rather than filing — NOT in this tracker's scope

Four contract-doc lines still name deleted bash files as byte-authority:

changelog-format.md:6    scripts/lib/changelog.sh · scripts/changelog-body-check.sh
fragment-format.md:6     scripts/lib/fragments.sh · scripts/lib/register-patterns.sh · scripts/fragment-check.sh
fragment-format.md:85    fragments.sh::fragment_is_breaking
forgejo-responses.md:6   scripts/lib/forgejo-api.sh

None of those files exists. A contract doc's byte-authority line is what a reader consults to resolve an ambiguity — so this points every such reader at a deleted file, and fragment-format.md:6 additionally says "this document is the contract both substrates are validated against" when there is now one substrate.

🔑 Same class as what this tracker's own PR fixed in #824: a confident sentence naming a thing that is not there. AC7 is still satisfied — the dual implementation is genuinely gone. The docs describing it are what did not follow. Residue of the single-stack retirement rather than of #777.

All seven ticked individually, each re-derived from main rather than from the checkbox — no bulk flip.

## ✅ AC AUDIT — all seven re-derived from `main` (`19e487b`) by RUNNING, not by reading the tracker. Four mutations, each asserted to have landed and compiled. **Every AC here is a state-assertion, so each is graded against the substrate.** Tests built and run from the tree; grammars mutated to prove the arms can fail. ### AC1 — validated against the real corpora by tests CI runs on every PR ✅ ``` go test ./internal/fragments -run TestC4AgainstHistoricalCorpus -v === RUN → --- PASS go test ./internal/changelog -run TestC6AgainstShippedChangelog -v === RUN → --- PASS ``` **`=== RUN` matters** — a `-run` pattern that matches nothing exits `ok` with no arms executed. Both named arms actually ran. CI half, from `.forgejo/workflows/go-ci.yml`: `on: pull_request` · `fetch-depth: 0` · `run: go test -count=1 ./...`. **`fetch-depth: 0` is load-bearing and the test says so itself** — C4's corpus comes from git history, and the test's own failure message names that setting as the thing making history available. ### AC2 — empty/missing corpus is cannot-grade, proven by mutation ✅ ``` M1 corpus = nil landed=1 · compiles · exit 1 --- FAIL: TestC4AgainstHistoricalCorpus "C4 corpus is EMPTY — refusing to report a vacuous pass." restored → exit 0 ``` **It FAILS, it does not skip.** The test's comment states the reasoning explicitly: `t.Skip` prints `ok` and exits 0, so the third state is lost while the property — *never a vacuous pass* — is kept. ### AC3 — positive and negative control per grammar, each shown able to fail ✅ ``` M2 parseName returns ("x","fixed",nil) unconditionally landed=1 · compiles · exit 1 control: "999.feat.md" was accepted; the C4 grammar must reject it control: "999.fix.md" was accepted; … M3 c6VersionRe := `.*` landed=1 · compiles · exit 1 control: "## [v1.2.3]" · "## [garbage" · "## []" all wrongly accepted, all caught ``` ### AC4 — dateless `## [v1.2.3]` rejected, `## [Unreleased]` accepted ✅ **Mutated to the exact defect `#518` S1 caught** — the union replaced by an optional-date group: ``` M4 `…\]( .*)?$` landed=1 · compiles · exit 1 control: "## [v1.2.3]" was accepted; must reject control: "## [v1.2.3] - 26-07-25" was accepted; must reject ``` **So the union form is doing the work, and the loosening the tracker warned about is caught.** ### AC5 — em-dash separator, with a literal U+2014 ✅ ``` c6_corpus_test.go:24 {"## [v1.2.3] — 2026-07-25", true} → U+2014, hexdump e2 80 94 parse.go c6VersionRe (-|—) ALTERNATION → U+2014 parse.go cutHeadingRe (?:-|—) ALTERNATION → U+2014 ``` **Alternation, not a byte class** — which is point 4 of this tracker, and the reason the fold is an improvement rather than a translation. ⚠️ **My first check of this reported `🔴 NOT a real em-dash` and was WRONG.** `grep -qP '\xe2\x80\x94'` returned false while the hexdump printed `e2 80 94` on the same line of output. **Chamber `grep` is ugrep; its `-P` hex-escape handling is not GNU's.** Caught only because I had printed two instruments side by side and they disagreed — *had I run the grep alone, I would have filed a false finding against correct work.* ### AC6 — script deleted, citing docs name the replacement ✅ ``` docs/architecture/contracts/validate-grammars.sh absent; deleted by 9d750c8 README.md:47 names internal/fragments.TestC4AgainstHistoricalCorpus + …TestC6… changelog-format.md:7 names internal/changelog.TestC6AgainstShippedChangelog fragment-format.md:7 names internal/fragments.TestC4AgainstHistoricalCorpus ``` **All three name the replacement by exact test name**, not by prose. ### AC7 — no grammar remains dual-implemented ✅ ``` git ls-files '*.sh' → docker-entrypoint.sh · scripts/fetch-rt.sh scripts/lib/ → ABSENT ``` **The bash grammar implementations are gone.** Both grammars live only in Go. --- ## 📌 Separate finding, requesting a tracker rather than filing — NOT in this tracker's scope **Four contract-doc lines still name deleted bash files as `byte-authority`:** ``` changelog-format.md:6 scripts/lib/changelog.sh · scripts/changelog-body-check.sh fragment-format.md:6 scripts/lib/fragments.sh · scripts/lib/register-patterns.sh · scripts/fragment-check.sh fragment-format.md:85 fragments.sh::fragment_is_breaking forgejo-responses.md:6 scripts/lib/forgejo-api.sh ``` **None of those files exists.** A contract doc's `byte-authority` line is what a reader consults to resolve an ambiguity — so this points every such reader at a deleted file, and `fragment-format.md:6` additionally says *"this document is the contract both substrates are validated against"* when there is now one substrate. 🔑 **Same class as what this tracker's own PR fixed in `#824`: a confident sentence naming a thing that is not there.** *AC7 is still satisfied — the dual implementation is genuinely gone. The docs describing it are what did not follow.* Residue of the single-stack retirement rather than of `#777`. *All seven ticked individually, each re-derived from `main` rather than from the checkbox — no bulk flip.*