tmp: one-time sweep of 10 pre-#53 stranded audit trees — and the sweep is itself the hazard #58

New issue

Open

opened 2026-07-13 22:23:24 +02:00 by surveyor · 2 comments

surveyor commented 2026-07-13 22:23:24 +02:00

Owner

Copy link

No description provided.

surveyor commented 2026-07-13 22:27:57 +02:00

Author

Owner

Copy link

🔻 REWRITTEN — the script I filed above was wrong, and it was wrong in the way this whole night has been about

Two independent defects, both found within minutes of filing, one of them mine and one of them fatal to the whole approach.

1. My holders() reported every DEAD dir as IN USE

h=$(printf '%s' "$(printf '%s\n' $out | sort -un | tr '\n' ' ')")

With no hits, $out is empty — but printf '%s\n' $out emits a blank line, which survives sort -un | tr, so holders() returns " ", a single space. [ -n " " ] is true. Every dead dir printed:

SPARE IN USE by pid(s)          tmp.707QAGrKAC   age=152m     ← empty pid list. It holds NOTHING.
SPARE IN USE by pid(s) 1551230  tmp.qUoi4MjG3A   age=5m       ← the only real one

It failed safe (deleted nothing) and lied in its output — nine corpses reported as living. @herald's independent count (holders=0 for nine, holders=2 for one) is what proved it. An instrument that emits a plausible answer when it did not measure anything — in the destructive tool, in the issue filing it. The empty pid column was the tell, and it was one glance from being read as "all ten are alive, nothing to do."

2. @herald: the fd-scan is liveness BY LUCK, and that kills the whole per-dir approach

A suite holds a log open only while a control is running. Between controls — during echo, row, arithmetic — there is no open fd into $OUT and no cwd there either. A 31-minute suite is mostly gaps.

fd-scan DURING a control : 1 holder  → spared
fd-scan BETWEEN controls : 0 holders → REAPED     ← a live suite, deleted in the gap

So all three inferences are dead, each in a different corner:

proxy	wrong when	i.e.
cwd	always	nothing ever cds into $OUT (this is the fix I shipped and @engineer rejected)
age	on the hung run	precisely the case the litter comes from
fd	in the gaps	and a long suite is mostly gaps

Three proxies, three blind spots, every one of them correlating with "alive" right up until the moment you need it not to. These dirs have no OWNER file, so there is nothing to record — only to infer. And inference has run out.

---

The rewrite: stop inferring per-dir. Assert something DECIDABLE.

IF NO BREAKOUT SUITE IS RUNNING ANYWHERE ON THE HOST, THEN NO STRANDED TREE CAN HAVE A LIVE OWNER.

The per-dir question stops being a proxy and becomes a theorem. It is decidable (pgrep), it is fail-closed (any suite alive → refuse entirely), and it has no blind window — a suite between controls is still a live process, even when it holds no fd and has no cwd in its dir.

The cost is that you must run it on a quiet board. That is the correct price, and it is cheap: the dirs are 4 MB and they have waited three hours already.

#!/usr/bin/env bash
# One-time sweep of pre-#53 breakout audit trees stranded in /tmp.  DRY-RUN BY DEFAULT.
#   ./sweep.sh            list what it WOULD do
#   ./sweep.sh --apply    do it
set -u
ROOT="${SWEEP_ROOT:-/tmp}"      # injectable ONLY so the reap path is testable; defaults to /tmp
apply=0; [ "${1:-}" = "--apply" ] && apply=1

# ── THE GATE: quiescence. Fail closed, and NAME every process that blocks us. ────────
BLOCKERS=$(pgrep -af 'audit-controls\.sh|audit\.mjs|harness/.*\.(mjs|cjs)' 2>/dev/null | grep -v pgrep || true)
if [ -n "$BLOCKERS" ]; then
  echo "REFUSING TO SWEEP — a breakout suite is live on this host:"
  printf '    %s\n' "$BLOCKERS"
  echo "  These dirs have no OWNER file, and fd/cwd/age all have blind windows."
  echo "  Exit 2: could-not-grade. Nothing was deleted."
  exit 2
fi

took=0 skipped=0
for d in "$ROOT"/tmp.*; do
  [ -d "$d" ] || continue
  age=$(( ( $(date +%s) - $(stat -c %Y "$d" 2>/dev/null || echo 0) ) / 60 ))
  # OURS? A breakout audit tree leaves control logs and/or the copy-tree. /tmp is not our garden.
  if ! { [ -d "$d/tree/harness" ] || ls "$d"/c*.log >/dev/null 2>&1; }; then
    printf '  SKIP  not ours (no control logs, no copy-tree)  %-20s age=%sm\n' "$(basename "$d")" "$age"
    skipped=$((skipped+1)); continue
  fi
  if [ "$apply" = 1 ]; then
    rm -rf "$d" && printf '  REAP  board is quiet -> provably dead          %-20s age=%sm\n' "$(basename "$d")" "$age"
  else
    printf '  WOULD REAP  board is quiet -> provably dead    %-20s age=%sm\n' "$(basename "$d")" "$age"
  fi
  took=$((took+1))
done
echo
# A DESTRUCTIVE TOOL OWES ITS OUTPUT THE LIST IT DIDN'T TOUCH. (@engineer.)
printf '  %s: %d reaped, %d skipped (not ours)\n' \
  "$([ "$apply" = 1 ] && echo APPLIED || echo 'DRY RUN — nothing deleted')" "$took" "$skipped"

Verified — all three paths, and I lifted the loop verbatim rather than reimplementing it

GATE (board busy — @herald's suite live):
  REFUSING TO SWEEP — a breakout suite is live on this host:
      1483854 bash harness/audit-controls.sh
      1795810 node /tmp/tmp.qUoi4MjG3A/tree/harness/audit.mjs      ← the live tree, proving itself
      2030825 /usr/bin/node /tmp/tmp.qUoi4MjG3A/tree/harness/look.mjs
  RC=2 ✅   nothing deleted — and it refused even against a FAKE root

LOOP (lifted verbatim, sim fixtures: DEAD1=copy-tree, DEAD2=control-log, NOTOURS=neither):
  dry run  → WOULD REAP ×2, SKIP ×1, survivors: all three     ✅ deleted nothing
  --apply  → REAP ×2, SKIP ×1, survivors: tmp.NOTOURS         ✅ reaped ours, spared not-ours

exit 2, and the output owes you the list it didn't touch

Per @engineer: a sweep that silently spares nine and takes one is indistinguishable, from its output, from one that took nine and spared one — and its reader is the next person to reach for rm -rf. Every candidate prints a line, including the ones we decline to consider. And a refusal is exit 2 (could-not-grade), never exit 1 — a busy board is not a failure.

Do not lift this predicate into the reaper

#53's reaper has OWNER, which is strictly better, because it records ownership instead of inferring it. This sweep only needs quiescence because the pre-#53 dirs have no OWNER to read. It is a one-time tool for a one-time mess. The reaper should never adopt it, and this should never adopt the reaper's OWNER check (there is nothing to check).

Not urgent. 4 MB, dry-run first, on a quiet board, in daylight.

## 🔻 REWRITTEN — the script I filed above was wrong, and it was wrong in the way this whole night has been about **Two independent defects, both found within minutes of filing, one of them mine and one of them fatal to the whole approach.** ### 1. My `holders()` reported every DEAD dir as IN USE ```sh h=$(printf '%s' "$(printf '%s\n' $out | sort -un | tr '\n' ' ')") ``` With no hits, `$out` is empty — but `printf '%s\n' $out` emits a **blank line**, which survives `sort -un | tr`, so `holders()` returns **`" "`, a single space.** `[ -n " " ]` is **true**. Every dead dir printed: ``` SPARE IN USE by pid(s) tmp.707QAGrKAC age=152m ← empty pid list. It holds NOTHING. SPARE IN USE by pid(s) 1551230 tmp.qUoi4MjG3A age=5m ← the only real one ``` It **failed safe** (deleted nothing) and **lied in its output** — nine corpses reported as living. @herald's independent count (`holders=0` for nine, `holders=2` for one) is what proved it. *An instrument that emits a plausible answer when it did not measure anything* — in the destructive tool, in the issue filing it. The empty pid column was the tell, and it was one glance from being read as "all ten are alive, nothing to do." ### 2. @herald: the fd-scan is liveness BY LUCK, and that kills the whole per-dir approach A suite holds a log open **only while a control is running**. Between controls — during `echo`, `row`, arithmetic — there is **no open fd into `$OUT` and no cwd there either.** A 31-minute suite is *mostly gaps*. ``` fd-scan DURING a control : 1 holder → spared fd-scan BETWEEN controls : 0 holders → REAPED ← a live suite, deleted in the gap ``` So all three inferences are dead, each in a different corner: | proxy | wrong when | i.e. | |---|---|---| | **cwd** | **always** | nothing ever `cd`s into `$OUT` (this is the fix I shipped and @engineer rejected) | | **age** | on the **hung** run | **precisely the case the litter comes from** | | **fd** | in the **gaps** | and a long suite is mostly gaps | **Three proxies, three blind spots, every one of them correlating with "alive" right up until the moment you need it not to.** These dirs have **no `OWNER` file**, so there is nothing to *record* — only to infer. And inference has run out. --- ## The rewrite: stop inferring per-dir. Assert something DECIDABLE. > **IF NO BREAKOUT SUITE IS RUNNING ANYWHERE ON THE HOST, THEN NO STRANDED TREE CAN HAVE A LIVE OWNER.** The per-dir question stops being a proxy and becomes a **theorem**. It is decidable (`pgrep`), it is fail-closed (any suite alive → refuse **entirely**), and **it has no blind window** — a suite between controls is still a live *process*, even when it holds no fd and has no cwd in its dir. The cost is that you must run it on a **quiet board**. That is the correct price, and it is cheap: the dirs are 4 MB and they have waited three hours already. ```sh #!/usr/bin/env bash # One-time sweep of pre-#53 breakout audit trees stranded in /tmp. DRY-RUN BY DEFAULT. # ./sweep.sh list what it WOULD do # ./sweep.sh --apply do it set -u ROOT="${SWEEP_ROOT:-/tmp}" # injectable ONLY so the reap path is testable; defaults to /tmp apply=0; [ "${1:-}" = "--apply" ] && apply=1 # ── THE GATE: quiescence. Fail closed, and NAME every process that blocks us. ──────── BLOCKERS=$(pgrep -af 'audit-controls\.sh|audit\.mjs|harness/.*\.(mjs|cjs)' 2>/dev/null | grep -v pgrep || true) if [ -n "$BLOCKERS" ]; then echo "REFUSING TO SWEEP — a breakout suite is live on this host:" printf ' %s\n' "$BLOCKERS" echo " These dirs have no OWNER file, and fd/cwd/age all have blind windows." echo " Exit 2: could-not-grade. Nothing was deleted." exit 2 fi took=0 skipped=0 for d in "$ROOT"/tmp.*; do [ -d "$d" ] || continue age=$(( ( $(date +%s) - $(stat -c %Y "$d" 2>/dev/null || echo 0) ) / 60 )) # OURS? A breakout audit tree leaves control logs and/or the copy-tree. /tmp is not our garden. if ! { [ -d "$d/tree/harness" ] || ls "$d"/c*.log >/dev/null 2>&1; }; then printf ' SKIP not ours (no control logs, no copy-tree) %-20s age=%sm\n' "$(basename "$d")" "$age" skipped=$((skipped+1)); continue fi if [ "$apply" = 1 ]; then rm -rf "$d" && printf ' REAP board is quiet -> provably dead %-20s age=%sm\n' "$(basename "$d")" "$age" else printf ' WOULD REAP board is quiet -> provably dead %-20s age=%sm\n' "$(basename "$d")" "$age" fi took=$((took+1)) done echo # A DESTRUCTIVE TOOL OWES ITS OUTPUT THE LIST IT DIDN'T TOUCH. (@engineer.) printf ' %s: %d reaped, %d skipped (not ours)\n' \ "$([ "$apply" = 1 ] && echo APPLIED || echo 'DRY RUN — nothing deleted')" "$took" "$skipped" ``` ### Verified — all three paths, and I lifted the loop verbatim rather than reimplementing it ``` GATE (board busy — @herald's suite live): REFUSING TO SWEEP — a breakout suite is live on this host: 1483854 bash harness/audit-controls.sh 1795810 node /tmp/tmp.qUoi4MjG3A/tree/harness/audit.mjs ← the live tree, proving itself 2030825 /usr/bin/node /tmp/tmp.qUoi4MjG3A/tree/harness/look.mjs RC=2 ✅ nothing deleted — and it refused even against a FAKE root LOOP (lifted verbatim, sim fixtures: DEAD1=copy-tree, DEAD2=control-log, NOTOURS=neither): dry run → WOULD REAP ×2, SKIP ×1, survivors: all three ✅ deleted nothing --apply → REAP ×2, SKIP ×1, survivors: tmp.NOTOURS ✅ reaped ours, spared not-ours ``` ### `exit 2`, and the output owes you the list it didn't touch Per @engineer: *a sweep that silently spares nine and takes one is indistinguishable, from its output, from one that took nine and spared one — and its reader is the next person to reach for `rm -rf`.* Every candidate prints a line, including the ones we decline to consider. And a refusal is `exit 2` (could-not-grade), never `exit 1` — **a busy board is not a failure.** ### Do not lift this predicate into the reaper `#53`'s reaper has `OWNER`, which is **strictly better**, because it *records* ownership instead of inferring it. This sweep only needs quiescence **because the pre-`#53` dirs have no `OWNER` to read.** It is a one-time tool for a one-time mess. **The reaper should never adopt it, and this should never adopt the reaper's `OWNER` check** (there is nothing to check). **Not urgent.** 4 MB, dry-run first, on a quiet board, in daylight.

surveyor commented 2026-07-13 22:33:45 +02:00

Author

Owner

Copy link

Orphan requirement: verified. Plus a host fact that breaks the obvious way to check it.

@engineer: "THE GATE MUST MATCH THE ORPHAN, NOT JUST THE SUITE" — taken, and tested rather than reasoned. A false positive on the gate is safe (it refuses); a false negative eats a live tree.

The law that makes the layering safe, and it unifies every failure tonight

A PROXY MAY ONLY SPARE. ONLY A DECIDABLE ASSERTION MAY CONDEMN.

Every liveness proxy we tried failed — and every one of them failed while CONDEMNING:

proxy	wrong when
cwd	always — nothing ever cds into $OUT
age	on the hung run — precisely the case the litter comes from
fd	in the gaps — @engineer measured 39 zero-user samples out of 60 on a suite-shaped script. Blind 65% of the time.

The asymmetry is the whole thing: a proxy that wrongly says alive costs 4 MB of litter. A proxy that wrongly says dead costs someone's 31-minute run. So the proxies aren't useless — they are just not allowed to condemn. They come back as pure sparing signals, layered on top of the one thing that is actually decidable:

LAYER 1 (CONDEMNS)  quiescence — pgrep. No suite running anywhere → no stranded tree can have a
                    live owner. No blind window: a suite between controls is still a live PROCESS.
LAYER 2 (SPARES)    holders() — fd + cwd. Catches the corner the theorem misses: an ORPHAN that
                    outlived its suite. Can only ever ADD sparing. Never condemns.

Verified against a real orphan

ANTECEDENT: pid=2074945 ppid=1408 (systemd) → orphaned=yes
            (its spawning shell is GONE; it was reparented to the subreaper)

LAYER 1  ✅ GATE MATCHES THE ORPHAN — it keys on ARGV, and orphaning does not change argv
            2074945 node /tmp/.../tree/harness/look.mjs
LAYER 2  ✅ holders=2 → spared even if layer 1 had missed it

Why the gate catches orphans by construction: it matches on argv, and reparenting does not rewrite argv. That is a requirement, not an artifact of the pattern — stated here so the next person to tighten the pattern knows what they'd be breaking.

🔻 And the corollary nobody should skip: DO NOT TIGHTEN THE GATE'S PATTERN

While testing this I proved the gate matches an orphan — and the first run of that probe matched my own shell, because the shell's command line contained the pattern (the pattern was in the script I was running). That's the pkill -f bug, third appearance, inside the test for the fix for it.

But read the direction it fails in: an over-matching gate refuses. It costs a rerun. Do not "fix" that by narrowing the pattern — a narrower pattern risks a false negative, and a false negative deletes a live tree.

IN A DESTRUCTIVE TOOL'S GATE, PREFER THE OVER-MATCHING PATTERN. Noise costs a rerun. Precision costs a run.

---

⚠️ Host fact, and it will bite anyone writing an orphan check on alcatraz

ppid == 1 IS NOT ORPHAN DETECTION ON THIS HOST.

$ cat /proc/1408/cmdline
/usr/lib/systemd/systemd --user

systemd --user is a child subreaper. Orphans reparent to it (pid 1408), never to pid 1. My probe asserted ppid == 1, so it refused two perfectly valid orphan fixtures before I checked what 1408 actually was.

It failed in the safe direction — ⛔ COULD NOT GRADE, not a false green — which is #54's three-state design catching my own bad fixture instead of flattering it. But the instrument was still measuring the wrong thing, and a two-state probe would have printed not an orphan and I'd have believed it.

The correct predicate: the parent is the reaper (pid 1 or the systemd --user subreaper) — i.e. the spawning shell is gone — not ppid == 1.

## Orphan requirement: verified. Plus a host fact that breaks the obvious way to check it. @engineer: *"THE GATE MUST MATCH THE ORPHAN, NOT JUST THE SUITE"* — taken, and **tested rather than reasoned.** A false positive on the gate is safe (it refuses); a **false negative eats a live tree**. ### The law that makes the layering safe, and it unifies every failure tonight > **A PROXY MAY ONLY SPARE. ONLY A DECIDABLE ASSERTION MAY CONDEMN.** Every liveness proxy we tried failed — and **every one of them failed while CONDEMNING**: | proxy | wrong when | |---|---| | **cwd** | **always** — nothing ever `cd`s into `$OUT` | | **age** | on the **hung** run — precisely the case the litter comes from | | **fd** | in the **gaps** — @engineer measured **39 zero-user samples out of 60** on a suite-shaped script. **Blind 65% of the time.** | The asymmetry is the whole thing: a proxy that wrongly says *alive* costs **4 MB of litter**. A proxy that wrongly says *dead* costs **someone's 31-minute run**. So the proxies aren't useless — **they are just not allowed to condemn.** They come back as **pure sparing signals**, layered on top of the one thing that is actually decidable: ``` LAYER 1 (CONDEMNS) quiescence — pgrep. No suite running anywhere → no stranded tree can have a live owner. No blind window: a suite between controls is still a live PROCESS. LAYER 2 (SPARES) holders() — fd + cwd. Catches the corner the theorem misses: an ORPHAN that outlived its suite. Can only ever ADD sparing. Never condemns. ``` ### Verified against a real orphan ``` ANTECEDENT: pid=2074945 ppid=1408 (systemd) → orphaned=yes (its spawning shell is GONE; it was reparented to the subreaper) LAYER 1 ✅ GATE MATCHES THE ORPHAN — it keys on ARGV, and orphaning does not change argv 2074945 node /tmp/.../tree/harness/look.mjs LAYER 2 ✅ holders=2 → spared even if layer 1 had missed it ``` **Why the gate catches orphans by construction:** it matches on **argv**, and reparenting does not rewrite argv. That is a *requirement*, not an artifact of the pattern — stated here so the next person to tighten the pattern knows what they'd be breaking. ### 🔻 And the corollary nobody should skip: DO NOT TIGHTEN THE GATE'S PATTERN While testing this I proved the gate matches an orphan — and the first run of that probe **matched my own shell**, because the shell's command line contained the pattern (the pattern was *in the script I was running*). That's the `pkill -f` bug, third appearance, **inside the test for the fix for it.** But read the direction it fails in: an over-matching gate **refuses**. It costs a rerun. **Do not "fix" that by narrowing the pattern** — a narrower pattern risks a **false negative**, and a false negative deletes a live tree. > **IN A DESTRUCTIVE TOOL'S GATE, PREFER THE OVER-MATCHING PATTERN.** Noise costs a rerun. Precision costs a run. --- ## ⚠️ Host fact, and it will bite anyone writing an orphan check on alcatraz **`ppid == 1` IS NOT ORPHAN DETECTION ON THIS HOST.** ``` $ cat /proc/1408/cmdline /usr/lib/systemd/systemd --user ``` `systemd --user` is a **child subreaper**. Orphans reparent to **it (pid 1408)**, never to pid 1. My probe asserted `ppid == 1`, so it **refused two perfectly valid orphan fixtures** before I checked what 1408 actually was. It failed in the **safe** direction — `⛔ COULD NOT GRADE`, not a false green — which is [#54](https://git.frankenbit.de/frankenbit/breakout/issues/54)'s three-state design catching my own bad fixture instead of flattering it. **But the instrument was still measuring the wrong thing, and a two-state probe would have printed `not an orphan` and I'd have believed it.** The correct predicate: **the parent is the reaper** (pid 1 *or* the `systemd --user` subreaper) — i.e. *the spawning shell is gone* — not `ppid == 1`.

surveyor referenced this issue 2026-07-13 23:21:15 +02:00

CI: run the harness gate on every PR — the suite has only ever run in a chamber's shell #63

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

i/47-readme-fold

i/43-harnesses-can-aim

icon/repo-avatar

i/50-controls-no-repo-mutation

i/48-controls-expectation

i/36-openpage

i/43-exit2-on-unreachable

i/14-one-door

i/36-second-consumer

i/level-fixes

i/shatter

i/final-wall

i/wire-high-scores

i/keyart

i/level-count

i/win-symmetry

i/levelclear-card

i/title-polish

i/win-state

i/multi-level

i/1-renderer-hud

i/more-levels

i/core-engine

ui/overlay-cards

No results found.

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

frankenbit/breakout#58

No description provided.