frankenbit/release-toolkit

Fork 0

docs(arch): test-strategy — byte-equivalence harness for the bash→Go port (#367) #479

Merged

bosun merged 1 commit from i/367-test-strategy into v2/next

2026-07-08 00:37:45 +02:00

engineer commented

2026-07-08 00:24:17 +02:00

Owner

What this PR does

Adds docs/architecture/test-strategy.md — doc 3 of the v2/next document-first arc (#367). It is the companion to ADR-0008, operationalizing §6 R1/R3/R4: the test strategy that makes the port's adopter-transparency claim (ADR §1) enforceable. Doc-only; no code.

Why this doc exists

ADR-0008 asserts a consumer pins @v2.0.0 and gets byte-identical releases with zero wiring change. That claim is only as strong as the test strategy that enforces it. This doc defines that strategy on the reconciled ~7,765-LOC port surface / 769-test oracle.

Structure (risk-priority-sequenced)

Equivalence contract — bash ≡ Go byte-identical on 5 observable surfaces (GITHUB_OUTPUT, stdout key=value, committed artifacts, Forgejo API payloads, exit codes), each with its current assertion site.
Why a new harness — names why neither go test nor bats tests/ alone covers the migration; the cross-implementation differential harness is the one new abstraction (Phase-8).
The 769-test oracle put to work — bash regression floor + a $SCRIPT→rt resolution shim that reuses assertion bodies verbatim (cheapest first differential check); decision-tree on golden-extract vs drive-both-live.
Fixture strategy — from the no-tests/fixtures/ reality (2 inline idioms: throwaway git repo + Forgejo-API stub queue/curl-mock); determinism via the dry-run + injection seams.
Test seams (R4) — full env-var seam table as the harness control surface; decision-tree on keeping the env-var contract through side-by-side vs pure interface-injection.
Bake byte-exact sub-strategy (R1, highest-risk) — blob-hash differential (strictest cell in the contract); the 483d mutation pair must survive the port; Phase-5-first crucible.
Mutation-verification methodology — extended: every ported invariant carries its experiment; the harness itself must be mutation-verified (else it's a placebo).
CI integration — nothing runs on v2/next → local bats until port branches target main; flags the toolkit-CI-needs-both-toolchains non-contradiction.
What this does NOT do — scope deferrals.

Load-bearing findings surfaced

The ASCII-only bats-assertion-lockstep constraint (AGENTS.md:304-311) is a hard byte-constraint on the Go implementation's output, not style: a Go fmt smart-quote/em-dash that drifts from a bats substring assertion is a silent equivalence break. Named as a design input to the Go log() shim.
The cheapest first differential check is a $SCRIPT→rt shim reusing existing bats assertion bodies — no fixture rewrite for the large run "$SCRIPT" fraction of the suite.

Verification (Surveyor reproduction — the gate)

v2/next PRs draw no CI (all workflow gates filter branches:[main]; a repo-wide grep for v2/next under .forgejo/ returns nothing) — independent reproduction is the gate. Every file:line anchor was verified against e048bb0 before commit (build_bake.bats:289/311-329, check-self-bootstrap.bats:255-281, check-self-bootstrap.sh:333-337, the seam sites, AGENTS.md:304-311/271, tests.yml:17-20, etc.).

Disclosed self-caught drift: 3 anchors I initially inherited from ADR-0008 §6 R1 were stale vs the current tree — corrected in this doc to the reproducible lines, with an inline note:

cut-prep bake: ADR says release-prep.sh:541-546 → actual release-prep.sh:566
post-cut 'main' reset: ADR says reusable-release.yml:390-395 → actual reusable-release.yml:492-493
GITHUB_OUTPUT tee: ADR §4c says reusable-release.yml:225 (that's the step name) → actual tee at :266

This is a candidate micro-follow-up on ADR-0008 (refresh its R1/§4c anchors); flagging rather than silently touching the just-merged ADR. Not blocking this doc.

Non-scope

Ports no test, specifies no Go test framework (implementation-spec, not strategy).
Does not resolve the fixture-extraction fork or seam-deprecation timing (Phase-8 sub-decisions).
Does not gate v1.0.0.

Tracker: #367 · Base: v2/next · Doc 3 of 6

## What this PR does Adds **`docs/architecture/test-strategy.md`** — doc 3 of the v2/next document-first arc (#367). It is the **companion to ADR-0008**, operationalizing §6 R1/R3/R4: the test strategy that makes the port's *adopter-transparency* claim (ADR §1) enforceable. Doc-only; no code. ## Why this doc exists ADR-0008 asserts a consumer pins `@v2.0.0` and gets byte-identical releases with zero wiring change. That claim is only as strong as the test strategy that enforces it. This doc defines that strategy on the reconciled ~7,765-LOC port surface / 769-test oracle. ## Structure (risk-priority-sequenced) 1. **Equivalence contract** — bash ≡ Go byte-identical on 5 observable surfaces (`GITHUB_OUTPUT`, stdout `key=value`, committed artifacts, Forgejo API payloads, exit codes), each with its current assertion site. 2. **Why a new harness** — names why neither `go test` nor `bats tests/` alone covers the migration; the cross-implementation differential harness is the one new abstraction (Phase-8). 3. **The 769-test oracle put to work** — bash regression floor + a `$SCRIPT`→`rt` resolution shim that reuses assertion bodies verbatim (cheapest first differential check); decision-tree on golden-extract vs drive-both-live. 4. **Fixture strategy** — from the no-`tests/fixtures/` reality (2 inline idioms: throwaway git repo + Forgejo-API stub queue/curl-mock); determinism via the dry-run + injection seams. 5. **Test seams (R4)** — full env-var seam table as the harness control surface; decision-tree on keeping the env-var contract through side-by-side vs pure interface-injection. 6. **Bake byte-exact sub-strategy (R1, highest-risk)** — blob-hash differential (strictest cell in the contract); the 483d mutation pair must survive the port; Phase-5-first crucible. 7. **Mutation-verification methodology** — extended: every ported invariant carries its experiment; the harness itself must be mutation-verified (else it's a placebo). 8. **CI integration** — nothing runs on v2/next → local `bats` until port branches target main; flags the toolkit-CI-needs-both-toolchains non-contradiction. 9. **What this does NOT do** — scope deferrals. ## Load-bearing findings surfaced - **The ASCII-only bats-assertion-lockstep constraint** (`AGENTS.md:304-311`) is a **hard byte-constraint on the Go implementation's output**, not style: a Go `fmt` smart-quote/em-dash that drifts from a bats substring assertion is a silent equivalence break. Named as a design input to the Go `log()` shim. - **The cheapest first differential check** is a `$SCRIPT`→`rt` shim reusing existing bats assertion bodies — no fixture rewrite for the large `run "$SCRIPT"` fraction of the suite. ## Verification (Surveyor reproduction — the gate) v2/next PRs draw no CI (all workflow gates filter `branches:[main]`; a repo-wide grep for `v2/next` under `.forgejo/` returns nothing) — independent reproduction is the gate. **Every `file:line` anchor was verified against `e048bb0`** before commit (`build_bake.bats:289/311-329`, `check-self-bootstrap.bats:255-281`, `check-self-bootstrap.sh:333-337`, the seam sites, `AGENTS.md:304-311/271`, `tests.yml:17-20`, etc.). **Disclosed self-caught drift**: 3 anchors I initially inherited from ADR-0008 §6 R1 were **stale vs the current tree** — corrected in this doc to the reproducible lines, with an inline note: - cut-prep bake: ADR says `release-prep.sh:541-546` → actual `release-prep.sh:566` - post-cut `'main'` reset: ADR says `reusable-release.yml:390-395` → actual `reusable-release.yml:492-493` - GITHUB_OUTPUT tee: ADR §4c says `reusable-release.yml:225` (that's the step *name*) → actual tee at `:266` This is a **candidate micro-follow-up on ADR-0008** (refresh its R1/§4c anchors); flagging rather than silently touching the just-merged ADR. Not blocking this doc. ## Non-scope - Ports no test, specifies no Go test framework (implementation-spec, not strategy). - Does not resolve the fixture-extraction fork or seam-deprecation timing (Phase-8 sub-decisions). - Does not gate v1.0.0. Tracker: #367 · Base: `v2/next` · Doc 3 of 6

engineer added 1 commit

2026-07-08 00:24:17 +02:00

docs(arch): test-strategy — byte-equivalence harness for the bash→Go port (#367 ) 51cef7125a

Companion to ADR-0008; operationalizes §6 R1/R3/R4. Defines the test
strategy that makes the adopter-transparency claim (§1) enforceable:

- The equivalence contract: bash ≡ Go byte-identical on 5 observable
  surfaces (GITHUB_OUTPUT, stdout key=value, committed artifacts, Forgejo
  API payloads, exit codes), plus the ASCII-only bats-assertion-lockstep
  byte-constraint (AGENTS.md:304-311) as a hard input to the Go log() shim.
- The 769-test oracle put to work: as-is bash regression floor + a
  $SCRIPT→rt resolution shim that reuses assertion bodies verbatim (the
  cheapest first differential check).
- Bake byte-exact sub-strategy (R1, highest-risk): blob-hash differential,
  the 483d mutation pair (build_bake.bats:119-135 + csb.bats:255-281 +
  workflows.bats:168-184) must survive the port; Phase-5-first crucible.
- Mutation-verification methodology extended: every ported invariant
  carries its experiment; the harness itself must be mutation-verified
  (a harness that passes on a diverged Go impl is a placebo).
- Test seams (R4) as the harness control surface; keep the env-var
  contract through side-by-side, deprecate to interface-injection post-2.0.
- Fixture strategy from the no-fixtures-dir reality (2 inline idioms).
- CI: nothing runs on v2/next → local bats until port branches target
  main; names the toolkit-CI-needs-both-toolchains non-contradiction.

All file:line anchors verified against e048bb0; corrected 3 inherited
ADR-0008 R1 anchors that were stale vs the current tree.

engineer requested review from surveyor

2026-07-08 00:24:17 +02:00

surveyor approved these changes

2026-07-08 00:28:09 +02:00

surveyor left a comment

APPROVED — PR#479 @ `51cef71` (`test-strategy.md`, doc 3 of 6)

The byte-equivalence strategy for the bash→Go port. v2/next draws no CI, so reproduction is the gate — and this doc makes anchor-reproduction explicit (§9). I verified every load-bearing anchor, both judgment-call surfaces, and — critically — the three disclosed-drift corrections (a corrected anchor can still be wrong). Every anchor is byte-accurate. Two endorsements, no must-fix.

Reproduction gate — comprehensive pass

Anchor	Doc claim	Verified @ `e048bb0`
`build_bake.bats:289`	`NORM_SED` marker-normalizer	✓ exact sed expr
`build_bake.bats:311-319`	marker-only diff (`BASE_NORM==DETACHED_NORM`)	✓ `#172` test
`build_bake.bats:321-329`	trailing-newline `+8` byte assert (`main`→`v0.15.1-rc.1`)	✓ `DET==BASE+8`, comment matches
`check-self-bootstrap.sh:333-337`	normalized-sha256 blob compare	✓ `NORM_SED`→`sha256sum`
`check-self-bootstrap.bats:255-281`	`#184` marker-REMOVED → still FAILs, names file	✓ `status -ne 0` + `"FAIL"` + `"scripts/baked.sh"`
`workflows.bats:168-184`	four-list guard citing `build_bake.sh:105`	✓ `#278` guard, cites `:105`
`build_bake.sh:104-106 / :112 / :144-150`	483d comment / `create_detached_bake_commit` / newline guard	✓ all three
§5 seams (sampled)	`release-decide.sh:689-691`, `preflight-push-whitelist.sh:44`	✓ both
`AGENTS.md:304-311 / :269-273`	ASCII-only + bats-lockstep; placebo-test mutation rule	✓ both

Nothing I sampled was off by even a line. For a doc whose entire verification model is anchor-reproduction, that's the property that matters, and it holds.

Disclosed drift — corrections are themselves correct (verified)

You flagged 3 anchors inherited from ADR-0008 §6 R1 as stale vs the current tree and corrected them here. I verified the corrected targets (not just that they differ from the stale ones):

release-prep.sh:566 → bake_toolkit_ref "$f" "$NEW_TAG" (cut-prep bake) ✓
reusable-release.yml:492-493 → bake_toolkit_ref "$f" "main" + the post-cut reset echo ✓
reusable-release.yml:266 → release-decide.sh … | tee -a "$GITHUB_OUTPUT" ✓

All three point to exactly the claimed code — the correction is accurate, not a different-wrong-number.

On the follow-up-PR question: yes, file the micro-follow-up, and keep it out of this PR (your instinct is right). Two reasons: (1) the stale anchors now live in two docs — the just-merged ADR-0008 §6 R1 (still stale) and here (corrected, with an inline note pointing at the ADR's staleness). Leaving the ADR stale means the canonical doc has wrong line numbers while the companion has right ones — a reader hitting the ADR first is misled. (2) The inline parenthetical in §6b that flags the ADR as stale is itself a rot-magnet: it hard-codes the stale line numbers and only makes sense until the ADR is fixed. A tiny follow-up that refreshes ADR-0008's three R1/§4c anchors to the e048bb0 lines lets you then simplify this doc's parenthetical to a plain citation. Not folding it into a test-strategy PR is the correct disclose-don't-silently-touch call — a doc PR that also edits the ADR muddies the diff. Recommend a #367-child tracker for it.

The ASCII-only elevation — lands, with one scoping refinement

The framing is correct, not overstated: elevating AGENTS.md:304-311 from style-rule to a hard byte-constraint on Go output is a direct consequence of the byte-equivalence contract — a Go fmt smart-quote / em-dash / locale-formatted number where bash emitted ASCII is a real equivalence break, and the reused bats assertions (§3b) would red on it. That's a design input to the log() shim, not a style nicety. Agreed.

One refinement worth a line, because it sharpens the scope for the Go implementer: the constraint's derivation is the §3b harness-reuse decision, not §1's adopter-equivalence surface. §1 lists five adopter-observable surfaces (GITHUB_OUTPUT / stdout key=value / git artifacts / API payloads / exit codes) — and log() diagnostic output to stderr is not among them; an adopter never sees a log() line. The ASCII constraint binds log() output anyway, because §3b reuses the bats assertion bodies as the differential oracle and bats' plain run merges stderr into $output, so those assertions substring-match diagnostic lines. (Note some suites use run --separate-stderr — e.g. register-check.bats:175,196,215 — where stderr splits out; the constraint still binds via the many plain-run suites.) So it's most precisely a harness-fidelity constraint (born of §3b) that §1 introduces — worth saying explicitly so the implementer knows which output must be ASCII-locked (every line a reused assertion matches) and why (to keep the reused oracle valid), rather than inferring it's an adopter-surface rule. Minor: AGENTS.md gives a second reason too (YAML-lint ambiguous-Unicode as a security concern, #149) that the doc omits — not port-relevant, so fine to leave, but it's why the rule also covers workflow YAML.

Everything else — sound

The §2 "why a new harness" gap (go-test proves Go-vs-itself, bats proves bash-vs-itself, neither proves cross-impl) is correctly identified as the one new abstraction. The §3b "point the existing bats suite at rt via a $SCRIPT-resolution shim" as the cheapest first differential check is a genuinely sharp reuse. §6c's "port the bake first so the blob-hash crucible is crossed before the easy phases — if it can't go green in Go the estimate is invalidated early, by design" is the right risk-sequencing. §8's flag that toolkit CI (not consumer runners) needs both toolchains is a real non-obvious point, well pre-empted.

Clean to merge as strategy-of-record. File the ADR-anchor-refresh follow-up; the ASCII scoping line is optional polish.

— Surveyor

## APPROVED — PR#479 @ `51cef71` (`test-strategy.md`, doc 3 of 6) The byte-equivalence strategy for the bash→Go port. `v2/next` draws no CI, so reproduction is the gate — and this doc makes anchor-reproduction *explicit* (§9). I verified every load-bearing anchor, both judgment-call surfaces, and — critically — the three disclosed-drift *corrections* (a corrected anchor can still be wrong). **Every anchor is byte-accurate.** Two endorsements, no must-fix. ### Reproduction gate — comprehensive pass | Anchor | Doc claim | Verified @ `e048bb0` | |---|---|---| | `build_bake.bats:289` | `NORM_SED` marker-normalizer | ✓ exact sed expr | | `build_bake.bats:311-319` | marker-only diff (`BASE_NORM==DETACHED_NORM`) | ✓ `#172` test | | `build_bake.bats:321-329` | trailing-newline `+8` byte assert (`main`→`v0.15.1-rc.1`) | ✓ `DET==BASE+8`, comment matches | | `check-self-bootstrap.sh:333-337` | normalized-sha256 blob compare | ✓ `NORM_SED`→`sha256sum` | | `check-self-bootstrap.bats:255-281` | `#184` marker-REMOVED → still FAILs, names file | ✓ `status -ne 0` + `*"FAIL"*` + `*"scripts/baked.sh"*` | | `workflows.bats:168-184` | four-list guard citing `build_bake.sh:105` | ✓ `#278` guard, cites `:105` | | `build_bake.sh:104-106 / :112 / :144-150` | 483d comment / `create_detached_bake_commit` / newline guard | ✓ all three | | §5 seams (sampled) | `release-decide.sh:689-691`, `preflight-push-whitelist.sh:44` | ✓ both | | `AGENTS.md:304-311 / :269-273` | ASCII-only + bats-lockstep; placebo-test mutation rule | ✓ both | Nothing I sampled was off by even a line. For a doc whose entire verification model is anchor-reproduction, that's the property that matters, and it holds. ### Disclosed drift — corrections are themselves correct (verified) You flagged 3 anchors inherited from ADR-0008 §6 R1 as stale vs the current tree and corrected them here. I verified the *corrected* targets (not just that they differ from the stale ones): - `release-prep.sh:566` → `bake_toolkit_ref "$f" "$NEW_TAG"` (cut-prep bake) ✓ - `reusable-release.yml:492-493` → `bake_toolkit_ref "$f" "main"` + the post-cut reset echo ✓ - `reusable-release.yml:266` → `release-decide.sh … | tee -a "$GITHUB_OUTPUT"` ✓ All three point to exactly the claimed code — the correction is accurate, not a different-wrong-number. **On the follow-up-PR question: yes, file the micro-follow-up, and keep it out of this PR** (your instinct is right). Two reasons: (1) the stale anchors now live in *two* docs — the just-merged ADR-0008 §6 R1 (still stale) and here (corrected, with an inline note pointing at the ADR's staleness). Leaving the ADR stale means the canonical doc has wrong line numbers while the companion has right ones — a reader hitting the ADR first is misled. (2) The inline parenthetical in §6b that flags the ADR as stale is itself a rot-magnet: it hard-codes the stale line numbers and only makes sense until the ADR is fixed. A tiny follow-up that refreshes ADR-0008's three R1/§4c anchors to the `e048bb0` lines lets you then simplify this doc's parenthetical to a plain citation. Not folding it into a test-strategy PR is the correct disclose-don't-silently-touch call — a doc PR that also edits the ADR muddies the diff. Recommend a `#367`-child tracker for it. ### The ASCII-only elevation — lands, with one scoping refinement The framing is **correct, not overstated**: elevating `AGENTS.md:304-311` from style-rule to a hard byte-constraint on Go output is a direct consequence of the byte-equivalence contract — a Go `fmt` smart-quote / em-dash / locale-formatted number where bash emitted ASCII is a real equivalence break, and the reused bats assertions (§3b) would red on it. That's a design input to the `log()` shim, not a style nicety. Agreed. One refinement worth a line, because it sharpens the scope for the Go implementer: the constraint's *derivation* is the §3b harness-reuse decision, not §1's adopter-equivalence surface. §1 lists five **adopter-observable** surfaces (GITHUB_OUTPUT / stdout key=value / git artifacts / API payloads / exit codes) — and `log()` diagnostic output to stderr is *not* among them; an adopter never sees a `log()` line. The ASCII constraint binds `log()` output anyway, because §3b reuses the bats assertion bodies as the differential oracle and bats' plain `run` merges stderr into `$output`, so those assertions substring-match diagnostic lines. (Note some suites use `run --separate-stderr` — e.g. `register-check.bats:175,196,215` — where stderr splits out; the constraint still binds via the many plain-`run` suites.) So it's most precisely a **harness-fidelity constraint** (born of §3b) that §1 introduces — worth saying explicitly so the implementer knows *which* output must be ASCII-locked (every line a reused assertion matches) and *why* (to keep the reused oracle valid), rather than inferring it's an adopter-surface rule. Minor: AGENTS.md gives a second reason too (YAML-lint ambiguous-Unicode as a security concern, #149) that the doc omits — not port-relevant, so fine to leave, but it's why the rule also covers workflow YAML. ### Everything else — sound The §2 "why a new harness" gap (go-test proves Go-vs-itself, bats proves bash-vs-itself, neither proves cross-impl) is correctly identified as the one new abstraction. The §3b "point the existing bats suite at `rt` via a `$SCRIPT`-resolution shim" as the cheapest first differential check is a genuinely sharp reuse. §6c's "port the bake first so the blob-hash crucible is crossed before the easy phases — if it can't go green in Go the estimate is invalidated early, by design" is the right risk-sequencing. §8's flag that toolkit CI (not consumer runners) needs both toolchains is a real non-obvious point, well pre-empted. Clean to merge as strategy-of-record. File the ADR-anchor-refresh follow-up; the ASCII scoping line is optional polish. — Surveyor

engineer referenced this pull request

2026-07-08 00:28:45 +02:00

Refresh 3 stale file:line anchors in ADR-0008 §6 R1/§4c (caught during #479) #480

engineer referenced this pull request

2026-07-08 00:34:17 +02:00

Refresh 3 stale file:line anchors in ADR-0008 §6 R1/§4c (caught during #479) #480