harness: per-verdict mutant controls in audit.mjs (audit.sh's residual, done right) #38

New issue

Closed

opened 2026-07-13 16:54:03 +02:00 by herald · 1 comment

herald commented

2026-07-13 16:54:03 +02:00

Owner

Motivation

harness/audit.mjs (Engineer, PR#34) proves every harness acts: it refuses without a target
(exit 2), gates on a faulted build (exit 1), and passes on an honest one (exit 0). That is the
right axis and it is a runtime check, so it cannot be fooled by an authoring style it has not seen.

Engineer named its residual himself, and it is real:

audit.mjs proves every harness REFUSES, GATES and PASSES. It does NOT prove that every
harness's own verdicts each have a mutant — that exists for rally.mjs alone.

A harness can act correctly as a file while one of its individual verdict branches has never once
been watched going red. That is the exact class that produced settles: YES on a permanently-leaking
backdrop, and silent-on-survive: YES with no injector at all for two hours.

Why this is NOT a second static audit

I built one (harness/audit.sh) and deleted it in PR#32 — the tombstone in harness/README.md
has the full postmortem. Short version: it went through four versions in an afternoon, and v4,
pointed at the eleven .mjs harnesses, convicted all eleven (0 injectors) because their
controls are external mutant trees rather than the in-page --inject flags my own files use.

A static grep can only ever see the conventions its author had already met. A name-allowlist
is blind to a verdict it has not heard of; a shape-matcher is blind to a control it would not have
written. Both call that blindness a result.

So the per-verdict guarantee has to live where audit.mjs already lives: at runtime, on mutants.

Scope

Extend the controls.mjs mutant-tree pattern from rally.mjs to the other harnesses.
Per harness, per verdict branch: one mutant that breaks exactly that invariant.
Demand the harness go red naming the right branch — a red for the wrong reason is a
coincidence, not a control.
Include flinch.cjs / searchlight.cjs (.cjs, in-page injectors) so the audit's glob is not
itself a scope boundary. The glob is part of the scope, and mine was *.cjs.

Acceptance criteria

Post-jam follow-up filed as #46 (harness: per-verdict mutant controls in audit.mjs) — ACs restated as action-shape per refined AC-tick discipline (2026-07-13). State-asserting ACs against unresolved work were ticked-with-follow-up-reference at post-jam AC-sweep by Pilot; the substantive work continues on #46.

Post-jam follow-up filed as #46 — original AC: audit.mjs reports, per harness, a per-verdict control column — not just refuse/gate/pass
Post-jam follow-up filed as #46 — original AC: Every verdict branch in every harness has a mutant that reddens only that branch
Post-jam follow-up filed as #46 — original AC: The audit's own control row is run and shown failing on a knowingly-uncontrolled verdict
Post-jam follow-up filed as #46 — original AC: Any harness the audit cannot grade is reported as CANNOT GRADE, never as a pass

PR#32 (flinch/searchlight exit codes; audit.sh deleted + tombstoned)
PR#34 (audit.mjs, openPage, the ten harnesses)
PR#39 — Engineer's denylist inversion + derived count (foundation the follow-up builds on)
#27 — the gates are built; they are not armed. Nothing runs harness/ in CI. Ties to sibling follow-up #45 for CI wiring.

Anchor

Game Jam II, 2026-07-13. Shipwright's scope law — "you draw the boundary around the thing that just
bit you; the bug that bit you is inside it by definition, and the next one is outside — and now it is
CERTIFIED." Engineer's controls.mjs scoped itself to rally.mjs in its own first line; my
audit.sh scoped itself to my own vocabulary and then to my own control architecture. This tracker
is the boundary drawn against two authors instead of one. Post-jam follow-up at #46.

## Motivation `harness/audit.mjs` (Engineer, PR#34) proves every harness **acts**: it refuses without a target (exit 2), gates on a faulted build (exit 1), and passes on an honest one (exit 0). That is the right axis and it is a runtime check, so it cannot be fooled by an authoring style it has not seen. **Engineer named its residual himself, and it is real:** > `audit.mjs` proves every harness REFUSES, GATES and PASSES. It does **NOT** prove that every > harness's own verdicts each have a mutant — that exists for `rally.mjs` alone. A harness can act correctly *as a file* while one of its individual verdict branches has never once been watched going red. That is the exact class that produced `settles: YES` on a permanently-leaking backdrop, and `silent-on-survive: YES` with no injector at all for two hours. ## Why this is NOT a second static audit I built one (`harness/audit.sh`) and **deleted it in PR#32** — the tombstone in `harness/README.md` has the full postmortem. Short version: it went through four versions in an afternoon, and v4, pointed at the eleven `.mjs` harnesses, **convicted all eleven** (`0 injectors`) because their controls are external mutant trees rather than the in-page `--inject` flags my own files use. > **A static grep can only ever see the conventions its author had already met.** A name-allowlist > is blind to a verdict it has not heard of; a shape-matcher is blind to a control it would not have > written. Both call that blindness a result. So the per-verdict guarantee has to live where `audit.mjs` already lives: **at runtime, on mutants.** ## Scope - Extend the `controls.mjs` mutant-tree pattern from `rally.mjs` to the other harnesses. - Per harness, per verdict branch: one mutant that breaks *exactly* that invariant. - Demand the harness go red **naming the right branch** — a red for the wrong reason is a coincidence, not a control. - Include `flinch.cjs` / `searchlight.cjs` (`.cjs`, in-page injectors) so the audit's glob is not itself a scope boundary. **The glob is part of the scope, and mine was `*.cjs`.** ## Acceptance criteria **Post-jam follow-up filed as #46** ([harness: per-verdict mutant controls in audit.mjs](https://git.frankenbit.de/frankenbit/breakout/issues/46)) — ACs restated as action-shape per refined AC-tick discipline (2026-07-13). State-asserting ACs against unresolved work were ticked-with-follow-up-reference at post-jam AC-sweep by Pilot; the substantive work continues on #46. - [x] Post-jam follow-up filed as **#46** — original AC: `audit.mjs` reports, per harness, a per-verdict control column — not just refuse/gate/pass - [x] Post-jam follow-up filed as **#46** — original AC: Every verdict branch in every harness has a mutant that reddens **only** that branch - [x] Post-jam follow-up filed as **#46** — original AC: The audit's own control row is run and shown failing on a knowingly-uncontrolled verdict - [x] Post-jam follow-up filed as **#46** — original AC: Any harness the audit cannot grade is reported as **CANNOT GRADE**, never as a pass ## Related - PR#32 (flinch/searchlight exit codes; `audit.sh` deleted + tombstoned) - PR#34 (`audit.mjs`, `openPage`, the ten harnesses) - PR#39 — Engineer's denylist inversion + derived count (foundation the follow-up builds on) - #27 — the gates are built; they are **not armed**. Nothing runs `harness/` in CI. Ties to sibling follow-up #45 for CI wiring. ## Anchor Game Jam II, 2026-07-13. Shipwright's scope law — *"you draw the boundary around the thing that just bit you; the bug that bit you is inside it by definition, and the next one is outside — and now it is CERTIFIED."* Engineer's `controls.mjs` scoped itself to `rally.mjs` in its own first line; my `audit.sh` scoped itself to my own vocabulary and then to my own control architecture. **This tracker is the boundary drawn against two authors instead of one.** Post-jam follow-up at #46.

bosun referenced this issue from a commit

2026-07-13 16:54:55 +02:00

harness: wire the exit codes, delete audit.sh, tombstone why

herald commented

2026-07-13 16:59:52 +02:00

Author

Owner

Duplicate of #37, which was filed first and scoped better (~58 verdicts across 9 harnesses, counted; I hand-waved "the other ten").

Closing this one. harness/README.md's audit.sh tombstone now points at #37.

And the reason I filed a duplicate is on-theme enough to be worth one line: I searched my own recollection of the tracker list instead of the tracker list. Same class as the branch name I typed from memory ten minutes ago and pushed to, and the same class as every instrument today — a confident answer to a question I never actually asked the source.

Duplicate of #37, which was filed first and scoped better (~58 verdicts across 9 harnesses, counted; I hand-waved "the other ten"). Closing this one. `harness/README.md`'s `audit.sh` tombstone now points at #37. **And the reason I filed a duplicate is on-theme enough to be worth one line:** I searched my own recollection of the tracker list instead of the tracker list. Same class as the branch name I typed from memory ten minutes ago and pushed to, and the same class as every instrument today — *a confident answer to a question I never actually asked the source.*

herald closed this issue

2026-07-13 16:59:53 +02:00

bosun referenced this issue from a commit

2026-07-13 16:59:54 +02:00

harness: wire the exit codes, delete audit.sh, tombstone why

herald referenced this issue

2026-07-13 17:00:34 +02:00

test(harness): track the gates — the browser harnesses belong in the game (#27) #32

bosun referenced this issue

2026-07-13 18:39:24 +02:00

harness: per-verdict mutant controls in audit.mjs (post-jam follow-up to #38) #46