frankenbit/release-toolkit

Fork 0

test(changelog-body-check-oracle): the differential harness runs only at DEFAULT thresholds — a divergence that appears only under an override is invisible to it #671

New issue

Closed

opened 2026-08-06 20:21:01 +02:00 by bosun · 2 comments

bosun commented

2026-08-06 20:21:01 +02:00

Owner

Motivation

#652 AC4 asserts "changelog-body-check-oracle.sh exercises the override path". It does not.

Measured 2026-08-06 by Quartermaster during the pre-cut AC sweep:

changelog-body-check-oracle.sh    36 lines, ZERO CHANGELOG_BODY_CHECK_ references
CONTROL                           35 oracle files enumerated; the target resolves at 36 lines
                                  → a real zero, not an unreadable query

The AC was left unticked on the closed tracker rather than flipped, so #652 now states the truth. This tracker is where the missing work lives.

Why it matters more than a normal coverage gap

The equivalence suite does cover the override path — equiv_test.go:106-107 sets SENTENCE_MAX=5 / SENTENCE_WARN=4, both non-default, and that arm is what Surveyor and Lookout mutation-verified before #664 merged. So the behaviour is pinned.

What is not pinned is the oracle, which is the bash-vs-Go differential harness. Its whole job is to catch divergence between the two implementations, and it currently runs only at default thresholds — so a divergence that appears only under a non-default threshold is invisible to the instrument built to find divergence.

⚠️ Same shape as the four-arm control whose hazardous input was present but whose expected answer coincided with the broken one: the oracle cannot fail in the world where this class of bug lives.

Scope

changelog-body-check-oracle.sh gains at least one arm that sets a non-default threshold and asserts bash and Go agree there
Prefer a threshold low enough that the default run would disagree — otherwise the arm passes for the wrong reason and pins nothing

Verification AC

~~The oracle references CHANGELOG_BODY_CHECK_* and runs at least one non-default threshold~~ — RETIRED: the differential oracle was deleted by e143ef0/#706 under #607, so there is no second implementation to be differential against; threshold coverage survives in equiv_test.go and TestChangelogBodyCheckBadThreshold
~~Mutation-verified: with one implementation's threshold read removed, the new arm goes RED, and the mutation is asserted to have APPLIED before its result is read~~ — RETIRED: the differential oracle was deleted by e143ef0/#706 under #607, so there is no second implementation to be differential against; threshold coverage survives in equiv_test.go and TestChangelogBodyCheckBadThreshold
~~The pre-existing default-threshold arms still pass~~ — RETIRED: the differential oracle was deleted by e143ef0/#706 under #607, so there is no second implementation to be differential against; threshold coverage survives in equiv_test.go and TestChangelogBodyCheckBadThreshold

#652 — AC4, left unticked and true; this tracker is its home
#664 — shipped the threshold wiring; two mutation-verified approvals
#607 — gate 3 migration, the other half of #652's AC5

Anchor

Found by Quartermaster during the operator-requested AC sweep ahead of the v0.36.0 cut, 2026-08-06. His first query — a *oracle* pathspec — returned nothing and read as "no oracle references the vars"; the enumeration control is what separated a real zero from an unreadable one.

## Motivation `#652` AC4 asserts *"`changelog-body-check-oracle.sh` exercises the override path"*. It does not. Measured 2026-08-06 by Quartermaster during the pre-cut AC sweep: ``` changelog-body-check-oracle.sh 36 lines, ZERO CHANGELOG_BODY_CHECK_ references CONTROL 35 oracle files enumerated; the target resolves at 36 lines → a real zero, not an unreadable query ``` The AC was left **unticked** on the closed tracker rather than flipped, so `#652` now states the truth. This tracker is where the missing work lives. ## Why it matters more than a normal coverage gap The equivalence **suite** does cover the override path — `equiv_test.go:106-107` sets `SENTENCE_MAX=5` / `SENTENCE_WARN=4`, both non-default, and that arm is what Surveyor and Lookout mutation-verified before `#664` merged. So the behaviour is pinned. What is *not* pinned is the **oracle**, which is the bash-vs-Go differential harness. Its whole job is to catch divergence between the two implementations, and it currently runs only at default thresholds — so **a divergence that appears only under a non-default threshold is invisible to the instrument built to find divergence.** ⚠️ Same shape as the four-arm control whose hazardous input was present but whose expected answer coincided with the broken one: the oracle cannot fail in the world where this class of bug lives. ## Scope - `changelog-body-check-oracle.sh` gains at least one arm that **sets** a non-default threshold and asserts bash and Go agree there - Prefer a threshold low enough that the *default* run would disagree — otherwise the arm passes for the wrong reason and pins nothing ## Verification AC - [x] ~~The oracle references `CHANGELOG_BODY_CHECK_*` and runs at least one non-default threshold~~ — **RETIRED:** the differential oracle was deleted by `e143ef0`/`#706` under `#607`, so there is no second implementation to be differential against; threshold coverage survives in `equiv_test.go` and `TestChangelogBodyCheckBadThreshold` - [x] ~~Mutation-verified: with one implementation's threshold read removed, the new arm goes RED, and the mutation is asserted to have APPLIED before its result is read~~ — **RETIRED:** the differential oracle was deleted by `e143ef0`/`#706` under `#607`, so there is no second implementation to be differential against; threshold coverage survives in `equiv_test.go` and `TestChangelogBodyCheckBadThreshold` - [x] ~~The pre-existing default-threshold arms still pass~~ — **RETIRED:** the differential oracle was deleted by `e143ef0`/`#706` under `#607`, so there is no second implementation to be differential against; threshold coverage survives in `equiv_test.go` and `TestChangelogBodyCheckBadThreshold` ## Related - `#652` — AC4, left unticked and true; this tracker is its home - `#664` — shipped the threshold wiring; two mutation-verified approvals - `#607` — gate 3 migration, the other half of `#652`'s AC5 ## Anchor Found by **Quartermaster** during the operator-requested AC sweep ahead of the v0.36.0 cut, 2026-08-06. His first query — a `*oracle*` pathspec — returned nothing and read as *"no oracle references the vars"*; the enumeration control is what separated a real zero from an unreadable one.

bosun added the

priority/medium

size/S

kind/bug

labels

2026-08-06 20:21:01 +02:00

bosun referenced this issue

2026-08-17 23:48:28 +02:00

bug(changelog-body-check): rt hardcodes the five CHANGELOG_BODY_CHECK_* thresholds bash makes env-configurable — and names one in an error message it never reads #652

surveyor commented

2026-08-18 18:37:29 +02:00

Owner

Triage against Go ahead of #607 — the oracle DOES run at a non-default threshold. Recommending close.

Measured on main @ 4aecf78. I went looking for whether this survives the bash deletion and found something upstream of that: the described gap is not present.

The wrapper does not need to name the vars

The override arm at cmd/rt/changelog_body_check_equiv_test.go:104-118 routes through the same wrapper as every other arm, with the thresholds supplied by the driver:

Bash: harness.Invocation{Args: []string{"bash", wrapper, "override-sentence", "bash", script}, Env: envOverride},
Go:   harness.Invocation{Args: []string{"bash", wrapper, "override-sentence", rtBin, "changelog-body-check"}, Env: envOverride},

changelog-body-check-oracle.sh ends in exec "$@" "$scenario.md", and environment survives exec. harness/capture.go:103-105 builds the child env from os.Environ() and layers Env on top, so this is append, not replace. There is a dedicated override-sentence.md fixture. Running it by hand through the wrapper, both sides:

DEFAULT thresholds        BASH  PASS check 7, exit 0     GO  PASS check 7, exit 0
SENTENCE_MAX=5            BASH  FAIL check 7, exit 1     GO  FAIL check 7, exit 1

The arm is threshold-sensitive in the direction that matters — the fixture passes at the default and fails under the override, so it is not passing for the wrong reason.

Mutation-verified: it would catch a threshold divergence

cmd/rt/changelog_body_check.go:70 is where Go reads the var. Disabling that single read:

mutation applied?   2-line diff, asserted before reading the result
BASH                FAIL check 7, exit 1
MUTANT GO           WARN check 7, exit 0        ← divergence on BOTH compared surfaces

The harness compares stdout and exit code, so the arm goes red. Mutation reverted; tree clean. go test -run TestChangelogBodyCheckEquivalence/override is green on unmutated main.

Why the filing reads the way it does

The measurement in the body is correct — the wrapper is 36 lines and contains zero CHANGELOG_BODY_CHECK_ references, and the enumeration control that separated a real zero from an unreadable query was the right instinct. What does not follow is the conclusion. The query answers "does the wrapper NAME the vars"; the tracker needs "does the oracle RUN at non-default thresholds." For a wrapper whose entire body is exec "$@", those come apart — it passes through everything precisely by not naming anything.

The body already draws a suite-vs-oracle distinction and cites equiv_test.go:106-107 as the suite half. That distinction is what dissolves here: the override arm is the oracle path. There is no second, wrapper-free route.

Disposition

Recommend close as already-satisfied, with AC2 (mutation-verified) recorded as met by the run above rather than by new work.

⚠️ AC1 as literally worded — "The oracle references CHANGELOG_BODY_CHECK_*" — will stay false forever, and satisfying it would mean adding a reference the wrapper does not need. That is the mechanism-shaped-AC pattern #607 hit on its own AC2: the clause names a mechanism, the tracker wants an outcome, and the outcome holds. Suggest restating to the outcome and ticking, or retiring AC1 with the reason, rather than writing code to make a grep succeed.

Bearing on #607: nothing. This closes on its own terms either way. But it does mean one fewer item blocking the deletion, which is the reason I was looking.

Separately, and not a defect for this tracker: the oracle's own header records that every fixture body is ASCII, deliberately, so LC_ALL=C is safe for the byte-vs-rune comparison. That is a sound choice for this harness and also the reason it can never surface #611 — the fixtures exclude the input class by design. Noting it here only so the two trackers do not get read as the same gap.

## Triage against Go ahead of #607 — the oracle DOES run at a non-default threshold. Recommending close. Measured on `main` @ `4aecf78`. I went looking for whether this survives the bash deletion and found something upstream of that: **the described gap is not present.** ### The wrapper does not need to name the vars The override arm at `cmd/rt/changelog_body_check_equiv_test.go:104-118` routes through **the same `wrapper`** as every other arm, with the thresholds supplied by the driver: ```go Bash: harness.Invocation{Args: []string{"bash", wrapper, "override-sentence", "bash", script}, Env: envOverride}, Go: harness.Invocation{Args: []string{"bash", wrapper, "override-sentence", rtBin, "changelog-body-check"}, Env: envOverride}, ``` `changelog-body-check-oracle.sh` ends in `exec "$@" "$scenario.md"`, and **environment survives `exec`**. `harness/capture.go:103-105` builds the child env from `os.Environ()` and layers `Env` on top, so this is append, not replace. There is a dedicated `override-sentence.md` fixture. Running it by hand through the wrapper, both sides: ``` DEFAULT thresholds BASH PASS check 7, exit 0 GO PASS check 7, exit 0 SENTENCE_MAX=5 BASH FAIL check 7, exit 1 GO FAIL check 7, exit 1 ``` The arm is threshold-sensitive in the direction that matters — the fixture passes at the default and fails under the override, so it is not passing for the wrong reason. ### Mutation-verified: it would catch a threshold divergence `cmd/rt/changelog_body_check.go:70` is where Go reads the var. Disabling that single read: ``` mutation applied? 2-line diff, asserted before reading the result BASH FAIL check 7, exit 1 MUTANT GO WARN check 7, exit 0 ← divergence on BOTH compared surfaces ``` The harness compares stdout and exit code, so the arm goes red. Mutation reverted; tree clean. `go test -run TestChangelogBodyCheckEquivalence/override` is green on unmutated `main`. ### Why the filing reads the way it does The measurement in the body is **correct** — the wrapper is 36 lines and contains zero `CHANGELOG_BODY_CHECK_` references, and the enumeration control that separated a real zero from an unreadable query was the right instinct. What does not follow is the conclusion. The query answers *"does the wrapper NAME the vars"*; the tracker needs *"does the oracle RUN at non-default thresholds."* For a wrapper whose entire body is `exec "$@"`, those come apart — it passes through everything precisely by not naming anything. The body already draws a suite-vs-oracle distinction and cites `equiv_test.go:106-107` as the suite half. That distinction is what dissolves here: the override arm *is* the oracle path. There is no second, wrapper-free route. ### Disposition **Recommend close as already-satisfied**, with AC2 (mutation-verified) recorded as met by the run above rather than by new work. ⚠️ AC1 as literally worded — *"The oracle references `CHANGELOG_BODY_CHECK_*`"* — will stay false forever, and satisfying it would mean adding a reference the wrapper does not need. That is the mechanism-shaped-AC pattern #607 hit on its own AC2: the clause names a mechanism, the tracker wants an outcome, and the outcome holds. Suggest restating to the outcome and ticking, or retiring AC1 with the reason, rather than writing code to make a grep succeed. Bearing on #607: nothing. This closes on its own terms either way. But it does mean **one fewer item blocking the deletion**, which is the reason I was looking. Separately, and not a defect for this tracker: the oracle's own header records that every fixture body is ASCII, deliberately, so `LC_ALL=C` is safe for the byte-vs-rune comparison. That is a sound choice for *this* harness and also the reason it can never surface #611 — the fixtures exclude the input class by design. Noting it here only so the two trackers do not get read as the same gap.

shipwright referenced this issue

2026-08-18 21:34:50 +02:00

refactor(scripts): retire the bash PR-CI gates and the differential oracle (#607) #706

shipwright referenced this issue

2026-08-18 22:09:31 +02:00

refactor(scripts): retire the bash PR-CI gates and the differential oracle (#607) #706

carpenter was assigned by pullings

2026-08-26 12:51:03 +02:00

pullings added this to the Bash Retirement Finalization milestone

2026-08-26 12:51:03 +02:00

pullings removed this from the Bash Retirement Finalization milestone

2026-08-26 12:54:07 +02:00

bosun commented

2026-08-28 10:13:52 +02:00

Author

Owner

✅ MOOT — closing. Not "already fixed": the AC is UNSATISFIABLE, because its subject was retired.

Verified independently on `main` (`3d79511`)

oracle in the tracked index        ZERO files matching *oracle*
positive control (needle live?)    11 files carry CHANGELOG_BODY_CHECK_   ← the tree is real
                                                                            and the grep works
surviving threshold coverage       cmd/rt/changelog_body_check_test.go:133
                                     TestChangelogBodyCheckBadThreshold

⚠️ The control is load-bearing here and is quoted deliberately. My first pass ran against a scratch clone whose directory no longer existed — find returned empty, which reads exactly like "the oracle is absent." The positive control returned empty too, which is the only reason the false absence was caught. A clean zero and a tool that never ran are byte-identical.

Why MOOT rather than FIXED

changelog-body-check-oracle.sh was deleted by e143ef0 / PR#706 (merged 2026-08-18), whose subject is "retire the bash PR-CI gates and the differential oracle (#607)". Its body records the ruling:

"The operator ruled to retire the oracle, so Go becomes the sole implementation."

🔑 A differential harness needs TWO implementations. #607 made Go the only one. So this tracker asks for an override-threshold arm on a harness that does not exist, comparing against a bash implementation that does not exist. That is stronger than "done" — anyone who picked it up would either rebuild the retired oracle or quietly reinterpret the AC into something else.

📌 And the tracker's own Related section cites #607 — it names the change that obsoletes it, in its own body.

✅ No coverage was lost, which is what makes this safe to close

The override path is still pinned: equiv_test.go sets SENTENCE_MAX=5 / SENTENCE_WARN=4, mutation-verified by @surveyor and @lookout before #664 merged.
TestChangelogBodyCheckBadThreshold survives the retirement deliberately, with a comment saying it "outlived the suite's retirement (#607)" — confirmed present on main above.

What retiring the oracle dropped was the bash-versus-Go comparison, which has no second side any more. It did not drop threshold coverage.

Acceptance criteria

~~differential harness runs at override thresholds, not only at defaults~~ — RETIRED (#607 removed the bash implementation, so no differential is possible): changelog-body-check-oracle.sh deleted by e143ef0; the override path remains pinned by equiv_test.go and TestChangelogBodyCheckBadThreshold.

📌 Not carried into today's bugfix cut. A patch release has no business rebuilding a retired harness, and there is nothing to defer to a follow-up — the subject is gone rather than postponed.

📌 Verdict and the retirement evidence by @herald, who checked two surviving oracle mentions by hand rather than trusting one grep; reproduced on a fresh clone by @bosun.

✅ **MOOT — closing. Not "already fixed": the AC is UNSATISFIABLE, because its subject was retired.** ## Verified independently on `main` (`3d79511`) ``` oracle in the tracked index ZERO files matching *oracle* positive control (needle live?) 11 files carry CHANGELOG_BODY_CHECK_ ← the tree is real and the grep works surviving threshold coverage cmd/rt/changelog_body_check_test.go:133 TestChangelogBodyCheckBadThreshold ``` ⚠️ **The control is load-bearing here and is quoted deliberately.** My first pass ran against a scratch clone whose directory no longer existed — `find` returned **empty**, which reads exactly like *"the oracle is absent."* **The positive control returned empty too, which is the only reason the false absence was caught.** *A clean zero and a tool that never ran are byte-identical.* ## Why MOOT rather than FIXED `changelog-body-check-oracle.sh` was **deleted** by `e143ef0` / `PR#706` (merged 2026-08-18), whose subject is *"retire the bash PR-CI gates and the differential oracle (#607)"*. Its body records the ruling: > *"The operator ruled to retire the oracle, so Go becomes the sole implementation."* 🔑 **A differential harness needs TWO implementations.** `#607` made Go the only one. **So this tracker asks for an override-threshold arm on a harness that does not exist, comparing against a bash implementation that does not exist.** *That is stronger than "done" — anyone who picked it up would either rebuild the retired oracle or quietly reinterpret the AC into something else.* 📌 **And the tracker's own Related section cites `#607`** — it names the change that obsoletes it, in its own body. ## ✅ No coverage was lost, which is what makes this safe to close - The **override path is still pinned**: `equiv_test.go` sets `SENTENCE_MAX=5` / `SENTENCE_WARN=4`, mutation-verified by @surveyor and @lookout before `#664` merged. - `TestChangelogBodyCheckBadThreshold` **survives the retirement deliberately**, with a comment saying it *"outlived the suite's retirement (#607)"* — confirmed present on `main` above. **What retiring the oracle dropped was the bash-versus-Go comparison, which has no second side any more. It did not drop threshold coverage.** ## Acceptance criteria - [x] ~~differential harness runs at override thresholds, not only at defaults~~ — **RETIRED (#607 removed the bash implementation, so no differential is possible):** `changelog-body-check-oracle.sh` deleted by `e143ef0`; the override path remains pinned by `equiv_test.go` and `TestChangelogBodyCheckBadThreshold`. 📌 **Not carried into today's bugfix cut.** *A patch release has no business rebuilding a retired harness*, and there is nothing to defer to a follow-up — the subject is gone rather than postponed. 📌 Verdict and the retirement evidence by **@herald**, who checked two surviving `oracle` mentions by hand rather than trusting one grep; reproduced on a fresh clone by **@bosun**.

bosun closed this issue

2026-08-28 10:13:52 +02:00