feat(scaffold): equivalence-harness framework (bash-oracle runner + observable-surface capture + diff) #503

Closed
opened 2026-07-24 22:42:29 +02:00 by bosun · 1 comment
Owner

Build the byte-equivalence harness that gates every subsequent phase. Load-bearing — subsequent phases' gates use this instrument.

Scope

  • Harness package internal/harness (or a top-level harness/ module — decide at implementation): given (a) a bash script + args + fixture input and (b) the equivalent Go binary invocation, capture the five observable surfaces per test-strategy.md §1 (GITHUB_OUTPUT, stdout key=value, committed git artifacts, Forgejo API payloads, exit codes), diff, and report byte-mismatches with structured error output.
  • Positive-control mechanism: harness rejects tests whose positive-arm cannot fire (per §5 absence-needs-positive-control discipline).
  • Fixture directory conventions: testdata/fixtures/<phase>/<case>/ with input + expected-output structure.
  • Integration with go test via a TestHarness_* naming convention or dedicated harness test-runner.

AC

  • Harness compiles + runs against a trivial hand-authored case and reports byte-identical — verified in PR#512 16a21c5
  • Harness reddens correctly on a planted mismatch — verified: instrument mutation-verified (3 narrow mutations per PR#512 body); positive-control (empty-both → cannot-grade, not vacuous-green) implemented
  • Harness handles LC_ALL=C properly — verified in PR#512: LC_ALL=C forced non-overridably (per §5 locale-safe subprocess pattern)
  • Harness documentation in docs/architecture/test-strategy.md reflects the implementation shape — verified: test-strategy.md §2a landed in PR#512 with implementation shape

Refs ADR-0009 §3.1 (approach — harness as load-bearing gate mechanism), §3.3 phase 0a, §5 (absence-needs-positive-control, locale-safe subprocess); test-strategy.md §1, §3b, §7.

Build the byte-equivalence harness that gates every subsequent phase. **Load-bearing** — subsequent phases' gates use this instrument. ## Scope - Harness package `internal/harness` (or a top-level `harness/` module — decide at implementation): given (a) a bash script + args + fixture input and (b) the equivalent Go binary invocation, capture the five observable surfaces per test-strategy.md §1 (GITHUB_OUTPUT, stdout key=value, committed git artifacts, Forgejo API payloads, exit codes), diff, and report byte-mismatches with structured error output. - Positive-control mechanism: harness rejects tests whose positive-arm cannot fire (per §5 absence-needs-positive-control discipline). - Fixture directory conventions: `testdata/fixtures/<phase>/<case>/` with input + expected-output structure. - Integration with `go test` via a `TestHarness_*` naming convention or dedicated harness test-runner. ## AC - [x] Harness compiles + runs against a trivial hand-authored case and reports byte-identical — verified in PR#512 16a21c5 - [x] Harness reddens correctly on a planted mismatch — verified: instrument mutation-verified (3 narrow mutations per PR#512 body); positive-control (empty-both → cannot-grade, not vacuous-green) implemented - [x] Harness handles LC_ALL=C properly — verified in PR#512: LC_ALL=C forced non-overridably (per §5 locale-safe subprocess pattern) - [x] Harness documentation in `docs/architecture/test-strategy.md` reflects the implementation shape — verified: test-strategy.md §2a landed in PR#512 with implementation shape Refs ADR-0009 §3.1 (approach — harness as load-bearing gate mechanism), §3.3 phase 0a, §5 (absence-needs-positive-control, locale-safe subprocess); test-strategy.md §1, §3b, §7.
bosun closed this issue 2026-07-25 00:07:37 +02:00
Author
Owner

Closed by hand — PR#512 merged as 16a21c595d39c04b4d201d84eae5bd3fb5ed0202 on v2/next via server-side rebase (pre-merge head was 314fb2e; rebased onto v2/next @ 3af1442; Surveyor stamp 4536 carried per Forgejo's server-side-rebase stamp-preservation semantics). Per ADR-0009 §3.3 phase-0a discipline: no close-keyword in PR body (avoids Forgejo positional-close-parser trap; matches #510 pattern). All 4 ACs verified + ticked in PR#512 body; equivalence-harness framework now live at internal/harness, testdata/fixtures conventions established, LC_ALL=C locale-safe subprocess pattern enforced non-overridably. Phase 0a status: 2/3 complete (#501 + #503); #502 pending token-fix cycle (PR#514).

**Closed by hand** — PR#512 merged as `16a21c595d39c04b4d201d84eae5bd3fb5ed0202` on v2/next via server-side rebase (pre-merge head was 314fb2e; rebased onto v2/next @ 3af1442; Surveyor stamp 4536 carried per Forgejo's server-side-rebase stamp-preservation semantics). Per ADR-0009 §3.3 phase-0a discipline: no close-keyword in PR body (avoids Forgejo positional-close-parser trap; matches #510 pattern). All 4 ACs verified + ticked in PR#512 body; equivalence-harness framework now live at `internal/harness`, testdata/fixtures conventions established, LC_ALL=C locale-safe subprocess pattern enforced non-overridably. Phase 0a status: 2/3 complete (#501 + #503); #502 pending token-fix cycle (PR#514).
Sign in to join this conversation.
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
frankenbit/release-toolkit#503
No description provided.