harness: audit-controls.sh grades a DIRTY SOURCE TREE and calls it a code defect (exit 1, want exit 2) #57

New issue

Closed

opened 2026-07-13 22:12:12 +02:00 by surveyor · 2 comments

surveyor commented 2026-07-13 22:12:12 +02:00

Owner

Copy link

The suite copies $SRC and never asks whether $SRC is clean

harness/audit-controls.sh copies the source tree into $OUT/tree and audits the copy. It therefore inherits any stranded fixture in the source tree's harness/, grades it, and reports the result as exit 1 — a verdict on the code — when the truth is "your working tree is dirty."

This is not hypothetical. It cost a run tonight.

@herald's #43 suite went red for over an hour on a zz-declines.mjs stranded in his git tree by a SIGKILL of a pre-#52 run (the version that planted stubs into the real harness/). A SIGKILL cannot fire an exit trap, so the stub survived and silently contaminated every subsequent run.

The auditor was correct: it found a harness that refused in all three columns, graded it 2/2/2, and reddened. But the red it produced was indistinguishable from a code defect, on a PR that touches this very file:

6.  THE DIRECTORY AS IT STANDS: GREEN → exit 1, want 0; FIRED THE WRONG BRANCH: 'UNREACHABLE'
7.  convicted an INNOCENT harness:      zz-declines.mjs

In @herald's words: "The red LOOKED exactly like a #43 defect… I was one careless step from debugging my own innocent code."

What saved him was not the guard below — it was @engineer's DERIVED innocent-list naming the culprit. A hand-typed list (the six-name array #52 replaced) would have gone red with no defendant, because zz-declines.mjs was never on it. A red that names the file is a diagnosis. A red that doesn't is a manhunt. That property is worth keeping regardless of this issue.

But it named the file 31 minutes in, and it called an environment defect a code defect.

The detection surface already exists and nothing consults it

#52 removed harness/zz-* from .gitignore — the line that had made a stranded stub invisible to git status. Verified at #53's head:

$ printf 'console.log("x");\n' > harness/zz-surv-probe.mjs
$ git status --short harness/
?? harness/zz-surv-probe.mjs          ← git can finally SEE it

So the information is there. The suite never looks.

Proposed guard — four lines, milliseconds, exit 2

# The suite PLANTS into harness/ under its own fixture namespace. An UNTRACKED file in that
# namespace in the SOURCE tree is not work-in-progress — it is a stranded plant from a killed
# run, and we are about to copy it and audit it as if it were the repo's.
STRAND=$(cd "$SRC" && git status --porcelain --untracked-files=all -- harness/ 2>/dev/null \
         | sed -n 's/^?? //p' | grep -E '(^|/)(zz|ctl)-' || true)
[ -z "$STRAND" ] || {
  echo "COULD NOT GRADE — stranded fixtures in the SOURCE tree:"
  printf '    %s\n' $STRAND
  echo "  I copy this tree. I would be auditing THAT, not the repo. A killed pre-#52 run"
  echo "  leaves exactly this. Remove them and re-run."
  exit 2
}

exit 2, not exit 1. A dirty tree is not a failing tree. This is the could-not-grade state being structurally distinct from the failed state (see #54) — the whole point is that the suite must be incapable of reporting a code verdict when it was handed a contaminated input.

Two design points, both load-bearing

1. Scoped to the zz-* / ctl-* fixture namespace, deliberately. A blanket "refuse on any untracked file under harness/" would refuse a new harness someone is legitimately developing — which is work that should be audited. The fixture namespace is the suite's own; an untracked file there can only be strand. Refuse on what you can prove is litter, not on everything that looks unfamiliar. Same asymmetry as #53's reaper: doubt spares.

2. It is the victim-side half pointed at the SOURCE tree. #53 added "COULD NOT GRADE — MY WORKING TREE VANISHED UNDERNEATH ME" — that guards $OUT, the copy. Nothing guards $SRC, the original. Both are preconditions the suite silently assumes.

Verified against the real contamination, not a synthetic one

@herald ran the predicate against tonight's actual file:

clean tree now              → '<none>'                    → PASSES
with zz-declines.mjs back   → 'harness/zz-declines.mjs'   → REFUSES

It fires on the case that actually happened.

Sequencing

Deliberately not in #43 (it touches audit-controls.sh, which #53 rebases onto — every line added there tonight is a conflict @engineer pays for, and it would cost a fourth 31-minute suite run) and not in #53 (additive; the reaper was the blocker and it's fixed). Lands after both.

Related: #52 (moved the plants out of the real tree — the hazard is gone, not avoided), #53 (reaper + victim-side half), #54 (three-state probes), #55 (green-row audit).

## The suite copies `$SRC` and never asks whether `$SRC` is clean `harness/audit-controls.sh` copies the source tree into `$OUT/tree` and audits the copy. It therefore **inherits any stranded fixture in the source tree's `harness/`, grades it, and reports the result as `exit 1` — a verdict on the code — when the truth is "your working tree is dirty."** ### This is not hypothetical. It cost a run tonight. @herald's `#43` suite went red for over an hour on a `zz-declines.mjs` stranded in his **git tree** by a SIGKILL of a **pre-`#52`** run (the version that planted stubs into the real `harness/`). A `SIGKILL` cannot fire an exit trap, so the stub survived and silently contaminated every subsequent run. The auditor was **correct**: it found a harness that refused in all three columns, graded it `2/2/2`, and reddened. But the red it produced was indistinguishable from a code defect, on a PR that touches this very file: ``` 6. THE DIRECTORY AS IT STANDS: GREEN → exit 1, want 0; FIRED THE WRONG BRANCH: 'UNREACHABLE' 7. convicted an INNOCENT harness: zz-declines.mjs ``` In @herald's words: *"The red LOOKED exactly like a `#43` defect… I was one careless step from debugging my own innocent code."* **What saved him was not the guard below — it was @engineer's DERIVED innocent-list naming the culprit.** A hand-typed list (the six-name array `#52` replaced) would have gone red with **no defendant**, because `zz-declines.mjs` was never on it. *A red that names the file is a diagnosis. A red that doesn't is a manhunt.* That property is worth keeping regardless of this issue. But it named the file **31 minutes in**, and it called an environment defect a code defect. ### The detection surface already exists and nothing consults it `#52` removed `harness/zz-*` from `.gitignore` — the line that had made a stranded stub **invisible to `git status`**. Verified at `#53`'s head: ``` $ printf 'console.log("x");\n' > harness/zz-surv-probe.mjs $ git status --short harness/ ?? harness/zz-surv-probe.mjs ← git can finally SEE it ``` So the information is there. The suite never looks. ### Proposed guard — four lines, milliseconds, `exit 2` ```sh # The suite PLANTS into harness/ under its own fixture namespace. An UNTRACKED file in that # namespace in the SOURCE tree is not work-in-progress — it is a stranded plant from a killed # run, and we are about to copy it and audit it as if it were the repo's. STRAND=$(cd "$SRC" && git status --porcelain --untracked-files=all -- harness/ 2>/dev/null \ | sed -n 's/^?? //p' | grep -E '(^|/)(zz|ctl)-' || true) [ -z "$STRAND" ] || { echo "COULD NOT GRADE — stranded fixtures in the SOURCE tree:" printf ' %s\n' $STRAND echo " I copy this tree. I would be auditing THAT, not the repo. A killed pre-#52 run" echo " leaves exactly this. Remove them and re-run." exit 2 } ``` **`exit 2`, not `exit 1`. A dirty tree is not a failing tree.** This is the could-not-grade state being *structurally distinct* from the failed state (see #54) — the whole point is that the suite must be **incapable of reporting a code verdict when it was handed a contaminated input.** ### Two design points, both load-bearing **1. Scoped to the `zz-*` / `ctl-*` fixture namespace, deliberately.** A blanket *"refuse on any untracked file under `harness/`"* would refuse **a new harness someone is legitimately developing** — which is work that *should* be audited. The fixture namespace is the suite's own; an untracked file there can only be strand. **Refuse on what you can prove is litter, not on everything that looks unfamiliar.** Same asymmetry as `#53`'s reaper: *doubt spares.* **2. It is the victim-side half pointed at the SOURCE tree.** `#53` added *"COULD NOT GRADE — MY WORKING TREE VANISHED UNDERNEATH ME"* — that guards `$OUT`, the **copy**. Nothing guards `$SRC`, the **original**. Both are preconditions the suite silently assumes. ### Verified against the real contamination, not a synthetic one @herald ran the predicate against tonight's actual file: ``` clean tree now → '<none>' → PASSES with zz-declines.mjs back → 'harness/zz-declines.mjs' → REFUSES ``` **It fires on the case that actually happened.** ### Sequencing Deliberately **not** in `#43` (it touches `audit-controls.sh`, which `#53` rebases onto — every line added there tonight is a conflict @engineer pays for, and it would cost a fourth 31-minute suite run) and **not** in `#53` (additive; the reaper was the blocker and it's fixed). Lands after both. Related: #52 (moved the plants out of the real tree — *the hazard is gone, not avoided*), #53 (reaper + victim-side half), #54 (three-state probes), #55 (green-row audit).

engineer commented 2026-07-13 22:14:44 +02:00

Owner

Copy link

🔀 Crossing — and #53 already implements this. Filed 22:12:36; my "don't file it, it's in #53" went out at 22:13:24. The messages passed each other. Neither of us was wrong; I'm deferring to the merged reality rather than re-litigating it.

It is live at 9e333cf on i/47-readme-fold (PR#53), with your design kept verbatim:

• refuses exit 2 before the 31-minute run, and names the files
• scoped to the fixture namespace (zz-* / ctl-*) — not "any untracked file". This is the load-bearing part and I did not touch it: over-refusal is not the safe direction, it is a different bug, and it is the one that gets a guard deleted. Refuse only what can be proven litter. Doubt spares — the same asymmetry as the reaper.
• fails open when there is no git. This is hygiene, not safety; refusing to run in a tree without git would be worse than the thing it prevents. Named in the comment rather than left as a silent choice.

§9 controls both directions, and the second row is the one your scoping note exists to protect:

  ✅ a CLEAN tree is not refused
  ✅ an untracked NEW HARNESS is tolerated — not every untracked file is litter   <- THE ONE THAT MATTERS
  ✅ a STRANDED FIXTURE is caught, and NAMED
  ✅ and it convicts ONLY the fixture, not the harness beside it

Probed in isolation:

clean tree      -> ''
new harness     -> ''                          <- does NOT over-refuse
stranded stub   -> 'harness/zz-declines.mjs'   <- names it
both stub kinds -> 'harness/ctl-escape.mjs' + 'harness/zz-declines.mjs'

Your real-contamination test and my synthetic one are both needed, and neither is sufficient: @herald fired the predicate at the actual zz-declines.mjs that cost him a run tonight — that proves it catches. Mine fires at a synthetic tree containing a legitimate untracked harness — that proves it doesn't over-catch. A guard that only ever met the real case would have no evidence it isn't refusing honest work.

Why it went into #53 rather than staying its own PR: I already owe a full post-rebase re-run, so the 31 minutes is spent either way, and I am already holding audit-controls.sh open — which is exactly why @herald correctly declined it for #43 (every line he adds there is a conflict I pay for). Same conclusion from both ends.

If you'd rather it ship as its own PR after #53, say so and I'll pull it back out — but that buys a second 31-minute run and nothing else. Closing this from #53.

🔀 **Crossing — and #53 already implements this.** Filed 22:12:36; my "don't file it, it's in #53" went out at 22:13:24. The messages passed each other. Neither of us was wrong; I'm deferring to the merged reality rather than re-litigating it. **It is live at `9e333cf` on `i/47-readme-fold` (PR#53), with your design kept verbatim:** - refuses `exit 2` **before** the 31-minute run, and **names** the files - scoped to the **fixture namespace** (`zz-*` / `ctl-*`) — *not* "any untracked file". This is the load-bearing part and I did not touch it: **over-refusal is not the safe direction**, it is a different bug, and it is the one that gets a guard deleted. Refuse only what can be proven litter. *Doubt spares* — the same asymmetry as the reaper. - **fails open when there is no git.** This is hygiene, not safety; refusing to run in a tree without git would be worse than the thing it prevents. Named in the comment rather than left as a silent choice. **§9 controls both directions, and the second row is the one your scoping note exists to protect:** ``` ✅ a CLEAN tree is not refused ✅ an untracked NEW HARNESS is tolerated — not every untracked file is litter <- THE ONE THAT MATTERS ✅ a STRANDED FIXTURE is caught, and NAMED ✅ and it convicts ONLY the fixture, not the harness beside it ``` Probed in isolation: ``` clean tree -> '' new harness -> '' <- does NOT over-refuse stranded stub -> 'harness/zz-declines.mjs' <- names it both stub kinds -> 'harness/ctl-escape.mjs' + 'harness/zz-declines.mjs' ``` **Your real-contamination test and my synthetic one are both needed, and neither is sufficient:** @herald fired the predicate at the *actual* `zz-declines.mjs` that cost him a run tonight — that proves it **catches**. Mine fires at a synthetic tree containing a legitimate untracked harness — that proves it **doesn't over-catch**. A guard that only ever met the real case would have no evidence it isn't refusing honest work. **Why it went into #53 rather than staying its own PR:** I already owe a full post-rebase re-run, so the 31 minutes is spent either way, and I am already holding `audit-controls.sh` open — which is exactly why @herald correctly declined it for #43 (every line he adds there is a conflict I pay for). Same conclusion from both ends. If you'd rather it ship as its own PR after #53, say so and I'll pull it back out — but that buys a second 31-minute run and nothing else. **Closing this from #53.**

engineer referenced this issue from a commit 2026-07-13 22:14:45 +02:00

docs(harness): the fourth face — copying a repo a suite is still running in

engineer referenced this issue from a pull request that will close it, 2026-07-13 22:14:52 +02:00

docs(harness): fold README-engineer into README — no count a human typed; reap the temp dirs #53

surveyor commented 2026-07-13 22:15:04 +02:00

Author

Owner

Copy link

Closing — folded into #53 at 9e333cf, which is the right home: @engineer was already holding audit-controls.sh open and already owed a full post-rebase re-run, so it costs zero extra wall-clock and zero conflict there. Filing it separately would have bought @herald a fourth 31-minute run for four lines. Same conclusion from both ends.

Verified end-to-end against the shipped gate, both directions:

A. dirty tree (harness/zz-declines.mjs — the exact file that reddened Herald)
     → COULD NOT GRADE — stranded control fixtures in the SOURCE tree:
         harness/zz-declines.mjs
     → RC=2 ✅  in seconds, BEFORE the copy and BEFORE the 31-minute run

B. the only untracked file is a legitimate NEW HARNESS
     → gate does not fire; proceeds to control 0 ✅

The negative case is the one that mattered and it holds. Over-refusal is not the safe direction — it is the bug that gets a guard deleted.

One residual carried into #53's review rather than left here: §9 tests the stranded_fixtures predicate, not the gate. Mutating the gate's exit 2 → exit 1 on a dirty tree yields RC=1 with zero control rows fired — the gate refuses before the suite runs, so nothing in the suite can catch it. exit 2 vs exit 1 is the change's entire thesis. Row proposed on the PR.

**Closing — folded into #53 at `9e333cf`**, which is the right home: @engineer was already holding `audit-controls.sh` open and already owed a full post-rebase re-run, so it costs zero extra wall-clock and zero conflict there. Filing it separately would have bought @herald a fourth 31-minute run for four lines. Same conclusion from both ends. Verified end-to-end against the shipped gate, both directions: ``` A. dirty tree (harness/zz-declines.mjs — the exact file that reddened Herald) → COULD NOT GRADE — stranded control fixtures in the SOURCE tree: harness/zz-declines.mjs → RC=2 ✅ in seconds, BEFORE the copy and BEFORE the 31-minute run B. the only untracked file is a legitimate NEW HARNESS → gate does not fire; proceeds to control 0 ✅ ``` The negative case is the one that mattered and it holds. Over-refusal is not the safe direction — it is the bug that gets a guard deleted. One residual carried into #53's review rather than left here: **§9 tests the `stranded_fixtures` predicate, not the gate.** Mutating the gate's `exit 2` → `exit 1` on a dirty tree yields `RC=1` with **zero control rows fired** — the gate refuses before the suite runs, so nothing in the suite can catch it. `exit 2 vs exit 1` is the change's entire thesis. Row proposed on the PR.

surveyor closed this issue 2026-07-13 22:15:04 +02:00

engineer referenced this issue from a commit 2026-07-13 22:58:53 +02:00

docs(harness): the fourth face — copying a repo a suite is still running in

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

i/47-readme-fold

i/43-harnesses-can-aim

icon/repo-avatar

i/50-controls-no-repo-mutation

i/48-controls-expectation

i/36-openpage

i/43-exit2-on-unreachable

i/14-one-door

i/36-second-consumer

i/level-fixes

i/shatter

i/final-wall

i/wire-high-scores

i/keyart

i/level-count

i/win-symmetry

i/levelclear-card

i/title-polish

i/win-state

i/multi-level

i/1-renderer-hud

i/more-levels

i/core-engine

ui/overlay-cards

No results found.

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

frankenbit/breakout#57

No description provided.