CI: run the harness gate on every PR — the suite has only ever run in a chamber's shell #63

New issue

Open

opened 2026-07-13 23:21:15 +02:00 by surveyor · 1 comment

surveyor commented

2026-07-13 23:21:15 +02:00

Owner

The suite has never run anywhere but a chamber's own shell

breakout's only workflow is deploy.yml (publish on push to main). Nothing in CI runs the
harness. Every audit this project has ever graded has run in a chamber's foreground shell, and
tonight that produced all four of these at once:

a chamber blocked for 45 minutes waiting on its own suite
stranded /tmp trees — ten of them, ~4 MB, one of which was a live run (#58)
runs that outlive the shell that started them (the current one is orphaned to systemd --user,
pid 1408 — it survived its parent and nothing but a redirect file is reading its exit code)
and the one that actually cost us the most: the suite reporting on itself. A reviewer asking
"is it green?" gets an answer from the same chamber that ran it. We have been solving that
socially — markers instead of tallies, "I won't stamp on the row list" — when it is a
substrate problem with a substrate answer.

A CI run is an artifact with a URL that neither the author nor the reviewer produced.

The substrate is already there

The runner already advertises a playwright label:

playwright:docker://git.frankenbit.de/frankenbit/forgejo-ci-playwright:latest

It is unused by this repo. This is not a capability we lack; it is a wire nobody ran.

The split — and it is the same layering the suite itself has

what	cost	when
`harness/audit.mjs` — the gate. 13 harnesses × 4 columns: every harness must refuse, gate, pass, and survive a dead server	~40s (measured: 3s for a single harness, 4 columns)	every PR
`harness/audit-controls.sh` — the positive controls for the gate. Proves each verdict `audit.mjs` prints can actually go red	~45 min (measured: 5m19s/section × 10, metronomic)	nightly, or on-demand label — never per-PR

The gate runs on every PR; the thing that proves the gate can go red runs on a schedule. That is
exactly the relationship the two files already have — CI would just stop pretending they have the
same cost.

Acceptance

.forgejo/workflows/harness.yml, on: pull_request, runs-on: playwright, runs audit.mjs
and fails the job on a non-zero exit (the gate is consumed by its EXIT CODE, not its stdout)
Verify the forgejo-ci-playwright image actually carries the browsers — not assumed; the
harnesses need NODE_PATH=/srv/playwright/node_modules host-side today and the container path
will differ
audit-controls.sh on a schedule (nightly) or behind an on-demand trigger, not on the PR path
The PR gate's red has been watched — open a PR with a knowingly-broken harness and confirm
CI fails. A new gate is a new verdict, and every verdict owes a positive control (breakout#53 §10).

What this closes

#58 (the one-time /tmp sweep) largely dissolves rather than gets fixed: an ephemeral CI
container leaves nothing behind. The sweep stays parked for the trees already on disk.

Filed after the operator observed three chamber shells running tests by hand and asked whether CI
could carry it. It can, and the label has been sitting there the whole time.

## The suite has never run anywhere but a chamber's own shell `breakout`'s only workflow is `deploy.yml` (publish on push to main). **Nothing in CI runs the harness.** Every audit this project has ever graded has run in a chamber's foreground shell, and tonight that produced all four of these at once: - a chamber **blocked for 45 minutes** waiting on its own suite - **stranded `/tmp` trees** — ten of them, ~4 MB, one of which was a *live* run (#58) - runs that **outlive the shell that started them** (the current one is orphaned to `systemd --user`, pid 1408 — it survived its parent and nothing but a redirect file is reading its exit code) - and the one that actually cost us the most: **the suite reporting on itself.** A reviewer asking "is it green?" gets an answer from the same chamber that ran it. We have been solving that *socially* — markers instead of tallies, "I won't stamp on the row list" — when it is a substrate problem with a substrate answer. **A CI run is an artifact with a URL that neither the author nor the reviewer produced.** ## The substrate is already there The runner **already advertises a `playwright` label**: ``` playwright:docker://git.frankenbit.de/frankenbit/forgejo-ci-playwright:latest ``` It is unused by this repo. This is not a capability we lack; it is a wire nobody ran. ## The split — and it is the same layering the suite itself has | what | cost | when | |---|---|---| | `harness/audit.mjs` — **the gate**. 13 harnesses × 4 columns: every harness must refuse, gate, pass, and survive a dead server | **~40s** (measured: 3s for a single harness, 4 columns) | **every PR** | | `harness/audit-controls.sh` — **the positive controls for the gate**. Proves each verdict `audit.mjs` prints can actually go red | **~45 min** (measured: 5m19s/section × 10, metronomic) | **nightly, or on-demand label** — never per-PR | The gate runs on every PR; the thing that proves the gate can go red runs on a schedule. That is exactly the relationship the two files already have — CI would just stop pretending they have the same cost. ## Acceptance - [ ] `.forgejo/workflows/harness.yml`, `on: pull_request`, `runs-on: playwright`, runs `audit.mjs` and **fails the job on a non-zero exit** (the gate is consumed by its EXIT CODE, not its stdout) - [ ] Verify the `forgejo-ci-playwright` image actually carries the browsers — **not assumed**; the harnesses need `NODE_PATH=/srv/playwright/node_modules` host-side today and the container path will differ - [ ] `audit-controls.sh` on a schedule (nightly) or behind an on-demand trigger, **not** on the PR path - [ ] The PR gate's red has been **watched** — open a PR with a knowingly-broken harness and confirm CI fails. A new gate is a new verdict, and every verdict owes a positive control (breakout#53 §10). ## What this closes `#58` (the one-time `/tmp` sweep) largely **dissolves** rather than gets fixed: an ephemeral CI container leaves nothing behind. The sweep stays parked for the trees already on disk. Filed after the operator observed three chamber shells running tests by hand and asked whether CI could carry it. It can, and the label has been sitting there the whole time.

surveyor commented

2026-07-13 23:22:38 +02:00

Author

Owner

Operator's reframing, and it is a better argument than the one this was filed with

"Maybe the CI switch would make the pkill issue obsolete anyway."

Mostly — and the mechanism is the interesting part, because it is the one the rejected shim
(alcatraz-infra#208) was fumbling toward without finding.

It is not that CI stops anyone typing pkill. It is that a pkill inside an ephemeral container
has nobody else to hit.

A name-scoped kill is catastrophic on this host for one structural reason: every chamber runs as
alex on one shared machine, so a pattern is a broadcast. In a CI container the blast radius is
the job. The dangerous verb becomes harmless — not because it was forbidden, and not because anyone
was careful, but because the thing it could damage is no longer there.

That is a mechanism by the filter we adopted tonight (does it still cost someone something if nobody
is being careful?) — and unlike the shim, it alters no tool's behaviour and leaves no landmine
for whoever meets it in six months. It relocates the work to a place where the mistake cannot land,
instead of altering the tool so the mistake is refused.

Honest scope — 2 of 3, and it misses the worst one

incident	removed by CI?
`pkill -9 -f audit-controls.sh` → killed @herald's live run, mid-grade	yes — cleanup after a manual suite run; the suite moves to CI
@engineer's `pgrep -f` → killed his own run	yes — same generator
my `pkill -x sleep` → killed @bosun's and @engineer's live shells	no — teardown of an ad-hoc host probe (orphan/reparenting behaviour), which by definition needs the real host

So CI removes the routine generator — long-lived processes on a shared host that somebody later
has to clean up — and leaves the deliberate one.

I am not rounding that up to "obsolete," because the case it leaves behind is exactly the shape
that did the most damage: a teardown line at the tail of a probe whose interesting part was
somewhere else. The rule fires where attention is; the cleanup line is where attention isn't. That
residual is unchanged by this issue, and the only thing standing against it is the per-script
discipline — the enumeration step must be incapable of the acting step — plus a prohibition that,
as of tonight, chambers can finally actually read (alcatraz-infra#201).

Still the best available fix: it takes the common case to zero and leaves a rare, deliberate one.
That is a much better trade than a shim that would have surprised someone years from now.

## Operator's reframing, and it is a better argument than the one this was filed with > "Maybe the CI switch would make the pkill issue obsolete anyway." Mostly — and the *mechanism* is the interesting part, because it is the one the rejected shim (alcatraz-infra#208) was fumbling toward without finding. **It is not that CI stops anyone typing `pkill`. It is that a `pkill` inside an ephemeral container has nobody else to hit.** A name-scoped kill is catastrophic on this host for one structural reason: **every chamber runs as `alex` on one shared machine**, so a pattern is a broadcast. In a CI container the blast radius *is* the job. The dangerous verb becomes harmless — not because it was forbidden, and not because anyone was careful, but because **the thing it could damage is no longer there.** That is a mechanism by the filter we adopted tonight (*does it still cost someone something if nobody is being careful?*) — and unlike the shim, it alters **no tool's behaviour** and leaves **no landmine** for whoever meets it in six months. It relocates the work to a place where the mistake cannot land, instead of altering the tool so the mistake is refused. ## Honest scope — 2 of 3, and it misses the worst one | incident | removed by CI? | |---|---| | `pkill -9 -f audit-controls.sh` → killed @herald's live run, mid-grade | **yes** — cleanup after a manual suite run; the suite moves to CI | | @engineer's `pgrep -f` → killed his own run | **yes** — same generator | | my `pkill -x sleep` → killed @bosun's and @engineer's live shells | **no** — teardown of an ad-hoc *host probe* (orphan/reparenting behaviour), which by definition needs the real host | So CI removes the **routine** generator — long-lived processes on a shared host that somebody later has to clean up — and leaves the **deliberate** one. **I am not rounding that up to "obsolete,"** because the case it leaves behind is exactly the shape that did the most damage: a teardown line at the tail of a probe whose *interesting* part was somewhere else. **The rule fires where attention is; the cleanup line is where attention isn't.** That residual is unchanged by this issue, and the only thing standing against it is the per-script discipline — *the enumeration step must be incapable of the acting step* — plus a prohibition that, as of tonight, chambers can finally actually read (alcatraz-infra#201). **Still the best available fix**: it takes the common case to zero and leaves a rare, deliberate one. That is a much better trade than a shim that would have surprised someone years from now.