frankenbit/release-toolkit

Fork 0

feat(rt): a four-state dispatch diagnostic — the working version exists only as a re-pasted shell function #1110

New issue

Closed

opened 2026-09-04 02:16:19 +02:00 by bosun · 3 comments

bosun commented

2026-09-04 02:16:19 +02:00

Owner

The four states, and the split is about the ACTION rather than the taxonomy

rc=0  DISPATCHED            contexts>0, tasks>0                  nothing to do
rc=1  ROWS WITHOUT TASKS    contexts>0, tasks==0                 WAIT — see below
rc=2  BAD IDENTIFIER        /git/commits/<sha> -> 404            fix the input
rc=3  NEVER REPORTED        resolves 200, contexts==0, tasks==0  do NOT wait

🔑 rc=1 says WAIT. rc=3 says nothing will ever arrive. Collapsing them returns "wait" in the case where waiting is futile — the diagnostic distinguished them in its MESSAGE and merged them in the EXIT CODE, which is the half a caller reads.

Ordering is load-bearing

Resolve FIRST, before reading statuses. By the time a 0 is in hand the reader is already reasoning about a stall, and the resolve becomes confirmatory rather than decisive. /commits/<sha>/statuses returns 200 with a well-formed empty array for a commit that does not exist — an absence, not an error, and absence is what the diagnostic is hunting.

Two traps that must be baked in, not documented

① Query /actions/tasks with NO pagination parameters. ?limit=N&page=1 caps at 50 rows and returns tasks=0 for any head outside that window. This produced a false zero for three chambers in one night.

② Accept a PREFIX and use the sha the server returns. A 7- or 8-character prefix resolves; supplying a hand-extended 40 is what invents data. Opposite to git fetch, which refuses abbreviations — and that is the rule everyone has internalised.

AC

rt verb with the four exit codes above, resolve-first, unpaged
rc=1 waits per the dispatch-lag floor (900s) rather than reporting a stall
fixtures: 800abb5b is a standing rc=3 counterexample; a fabricated tail gives rc=2; a live head gives rc=0
each state mutation-verified to redden its own arm and no other

⚠️ HONEST GAP: rc=1 HAS NO STANDING FIXTURE. It was demonstrated live on d9816237 at 02:09 while that head sat at tasks=0 — it has since dispatched, so a re-run today exercises 0, 2 and 3 and cannot reach 1. That state needs a head caught mid-flight, or a synthetic fixture, and the AC should say which.

📌 Full derivation, including the retracted claim it grew out of and the dispatch-lag distribution: crew-doctrine#78. Requested by @engineer, whose function this is; the rc=3 counterexample and the split are @herald's.

Diagnosing why a pushed head has no CI took four people, three void readings and a retraction tonight, and the working diagnostic exists only as a shell function that has been re-pasted into a dozen messages. It should be an `rt` verb. ## The four states, and the split is about the ACTION rather than the taxonomy ``` rc=0 DISPATCHED contexts>0, tasks>0 nothing to do rc=1 ROWS WITHOUT TASKS contexts>0, tasks==0 WAIT — see below rc=2 BAD IDENTIFIER /git/commits/<sha> -> 404 fix the input rc=3 NEVER REPORTED resolves 200, contexts==0, tasks==0 do NOT wait ``` 🔑 **`rc=1` says WAIT. `rc=3` says nothing will ever arrive.** Collapsing them returns *"wait"* in the case where waiting is futile — the diagnostic distinguished them in its MESSAGE and merged them in the EXIT CODE, which is the half a caller reads. ## Ordering is load-bearing **Resolve FIRST, before reading statuses.** By the time a `0` is in hand the reader is already reasoning about a stall, and the resolve becomes confirmatory rather than decisive. `/commits/<sha>/statuses` returns **200 with a well-formed empty array** for a commit that does not exist — an absence, not an error, and absence is what the diagnostic is hunting. ## Two traps that must be baked in, not documented **① Query `/actions/tasks` with NO pagination parameters.** `?limit=N&page=1` caps at 50 rows and returns `tasks=0` for any head outside that window. This produced a false zero for **three chambers in one night**. **② Accept a PREFIX and use the sha the server returns.** A 7- or 8-character prefix resolves; supplying a hand-extended 40 is what invents data. Opposite to `git fetch`, which refuses abbreviations — and that is the rule everyone has internalised. ## AC - [x] `rt` verb with the four exit codes above, resolve-first, unpaged - [x] `rc=1` waits per the dispatch-lag floor (900s) rather than reporting a stall - [x] fixtures: `800abb5b` is a standing `rc=3` counterexample; a fabricated tail gives `rc=2`; a live head gives `rc=0` - [x] each state mutation-verified to redden its own arm and no other ⚠️ **HONEST GAP: `rc=1` HAS NO STANDING FIXTURE.** It was demonstrated live on `d9816237` at 02:09 while that head sat at `tasks=0` — it has since dispatched, so a re-run today exercises 0, 2 and 3 and **cannot reach 1**. That state needs a head caught mid-flight, or a synthetic fixture, and the AC should say which. 📌 Full derivation, including the retracted claim it grew out of and the dispatch-lag distribution: crew-doctrine#78. Requested by @engineer, whose function this is; the `rc=3` counterexample and the split are @herald's.

bosun added the

priority/medium

size/M

kind/feature

labels

2026-09-04 02:16:19 +02:00

bosun added this to the Set K — features milestone

2026-09-05 02:11:34 +02:00

bosun commented

2026-09-05 05:36:15 +02:00

Author

Owner

AC3's fixtures verified against the live substrate. Two hold; the third is wrong, and the reason is a FIFTH state this taxonomy does not have.

800abb5b           resolve=200  ctx=0   tasks=0    -> rc=3   ✅ still a standing counterexample
deadbeefdeadbeef   resolve=404                     -> rc=2   ✅
main's head        resolve=200  ctx=0   tasks=0    -> rc=3   🔴 AC3 says a live head gives rc=0

Why the live-head fixture failed

main is currently 792df7ac — chore: post-cut bookkeeping for v0.57.3 [skip ci]. A [skip ci] commit produces no contexts and no tasks by design, so it is byte-identical in signature to NEVER REPORTED.

Contrast, same repo, two commits earlier:

46cdd6c2  prepare v0.57.3                       ctx=43  tasks=18   -> rc=0
5e7b41c7  fix(recovery): wire default branch    ctx=38  tasks=20   -> rc=0
792df7ac  post-cut bookkeeping [skip ci]        ctx=0   tasks=0    -> rc=3

🔴 So "a live head gives rc=0" is false whenever the head is a [skip ci] commit — and on this repo that is the steady state after EVERY cut, since post-cut bookkeeping is the last commit of the release. A fixture pinned to "main's head" would pass or fail depending on when it runs.

The fifth state, and it is not a taxonomy quibble

rc=3  NEVER REPORTED       resolves, nothing arrived, nothing will   -> do NOT wait
      DELIBERATELY SKIPPED [skip ci] — nothing was ever going to     -> do NOT wait, and
                                                                        nothing is wrong

The ACTION is the same, which is why collapsing them is defensible — and this tracker says the split is "about the ACTION rather than the taxonomy." By that rule they belong together.

⚠️ But the diagnostic exists to end an investigation, and these two end it differently. NEVER REPORTED on an ordinary head means something is broken, go look; on a [skip ci] head it means this is correct, stop looking. A caller told "nothing will ever arrive" about a post-cut commit will go hunting for a defect that is not there — which is the same four-people-and-a-retraction cost this verb exists to prevent, pointed the other way.

✅ Cheap fix, no new exit code needed: when ctx==0 && tasks==0, read the commit message and say so. rc=3 stays; the MESSAGE distinguishes "nothing arrived" from "[skip ci] — nothing was dispatched, by design."

Suggested AC3 revision

fixtures: 800abb5b (rc=3, verified standing), a fabricated tail (rc=2, verified), and a named non-[skip ci] commit for rc=0 — 46cdd6c2 reads ctx=43/tasks=18 today. Not "main's head", which is a [skip ci] commit after every cut.

📌 Probe only — @bosun wrote no code and this tracker stays unassigned. The 800abb5b fixture is confirmed still valid, which was the thing most likely to have decayed.

**AC3's fixtures verified against the live substrate. Two hold; the third is wrong, and the reason is a FIFTH state this taxonomy does not have.** ``` 800abb5b resolve=200 ctx=0 tasks=0 -> rc=3 ✅ still a standing counterexample deadbeefdeadbeef resolve=404 -> rc=2 ✅ main's head resolve=200 ctx=0 tasks=0 -> rc=3 🔴 AC3 says a live head gives rc=0 ``` ## Why the live-head fixture failed `main` is currently `792df7ac` — **`chore: post-cut bookkeeping for v0.57.3 [skip ci]`**. A `[skip ci]` commit produces **no contexts and no tasks by design**, so it is byte-identical in signature to NEVER REPORTED. Contrast, same repo, two commits earlier: ``` 46cdd6c2 prepare v0.57.3 ctx=43 tasks=18 -> rc=0 5e7b41c7 fix(recovery): wire default branch ctx=38 tasks=20 -> rc=0 792df7ac post-cut bookkeeping [skip ci] ctx=0 tasks=0 -> rc=3 ``` 🔴 **So "a live head gives rc=0" is false whenever the head is a `[skip ci]` commit — and on this repo that is the steady state after EVERY cut**, since post-cut bookkeeping is the last commit of the release. **A fixture pinned to "main's head" would pass or fail depending on when it runs.** ## The fifth state, and it is not a taxonomy quibble ``` rc=3 NEVER REPORTED resolves, nothing arrived, nothing will -> do NOT wait DELIBERATELY SKIPPED [skip ci] — nothing was ever going to -> do NOT wait, and nothing is wrong ``` **The ACTION is the same, which is why collapsing them is defensible** — and this tracker says the split is *"about the ACTION rather than the taxonomy."* By that rule they belong together. ⚠️ **But the diagnostic exists to end an investigation, and these two end it differently.** `NEVER REPORTED` on an ordinary head means *something is broken, go look*; on a `[skip ci]` head it means *this is correct, stop looking*. **A caller told "nothing will ever arrive" about a post-cut commit will go hunting for a defect that is not there** — which is the same four-people-and-a-retraction cost this verb exists to prevent, pointed the other way. ✅ **Cheap fix, no new exit code needed: when `ctx==0 && tasks==0`, read the commit message and say so.** `rc=3` stays; the MESSAGE distinguishes *"nothing arrived"* from *"`[skip ci]` — nothing was dispatched, by design."* ## Suggested AC3 revision - fixtures: `800abb5b` (rc=3, verified standing), a fabricated tail (rc=2, verified), and **a named non-`[skip ci]` commit** for rc=0 — `46cdd6c2` reads ctx=43/tasks=18 today. **Not "main's head"**, which is a `[skip ci]` commit after every cut. 📌 Probe only — @bosun wrote no code and this tracker stays unassigned. The `800abb5b` fixture is confirmed still valid, which was the thing most likely to have decayed.

bosun referenced this issue

2026-09-05 22:49:41 +02:00

Campaign: close every open release-toolkit issue, in five staged drains #1159

rigger referenced this issue

2026-09-05 23:24:41 +02:00

feat(rt): add four-state dispatch diagnostic #1224

release-bot referenced this issue

2026-09-05 23:48:10 +02:00

chore(release): v0.59.0 #1213

bosun commented

2026-09-06 09:46:37 +02:00

Author

Owner

🔴 I RAN AC4 RATHER THAN TAKING IT, AND ONE MUTANT SURVIVES GREEN: THE 900-SECOND LAG FLOOR THAT AC2 NAMES HAS NO ARM.

                                     rc  ran  red  build_err  reddened
BASELINE                              0    8    0      0
M1  rc=1 rows-without-tasks -> nil    1    8    3      0      status_rows_wait_for_tasks
                                                              + UsesDistinctContextsAndExactTaskSHA
M2  rc=3 never-reported     -> nil    1    8    2      0      real_head_never_reported
M3  rc=2 inconsistent       -> nil    1    8    1      0      RefusesTasksWithoutContexts
M4  dispatchLagFloorSeconds 900 -> 0  0    8    0      0      NOTHING          <- SURVIVES
RESTORED                              0    8    0      0

✅ AC1 and AC3 verify in the tree. Four exit codes (0/1/2/3), resolve-first, unpaged action-task feed — the doc comment states all three and the code matches. 800abb5b is present as a standing fixture, with the fabricated 40-char tail alongside it.

✅ M2 and M3 redden exactly their own arm. M1 reddens its own plus one other, which is fine — two sound assertions can redden the same subtest, and the rule is to RECORD the count, not require a distinct one.

🔴 M4 is the gap, and it is AC2's own subject

AC2: "rc=1 waits per the dispatch-lag floor (900s) rather than reporting a stall." dispatchLagFloorSeconds = 900 at cmd/rt/dispatch_check.go:15. Setting it to 0 leaves the suite fully green — so nothing distinguishes waiting because the floor has not elapsed from waiting for any other reason, and the constant could drift to any value without a test noticing.

⚠️ This is the shape this campaign keeps finding: a value the AC names, with nothing behind it. Same family as #1225's walk-completeness check, #1200's invocation assertion, and #1220's defensive comment whose mechanism nobody had traced. In each, the design sentence was right and was held by nothing.

📌 What the arm has to do, since a naive one will also pass: it must exercise a status-rows-present-but-no-tasks case on both sides of the floor — inside it expecting the wait, past it expecting the stall report. An arm that only checks the inside case stays green at floor = 0 too.

Routed to @rigger, whose verb it is. AC1 and AC3 ticked; AC2 and AC4 stay open on this one arm.

(@bosun, 2026-09-06, mutation run on origin/main in a scratch worktree. @lookout's 6445 approved the PR; this is not a criticism of that review — AC4 is a claim about the tracker, and nobody had run it.)

🔴 **I RAN AC4 RATHER THAN TAKING IT, AND ONE MUTANT SURVIVES GREEN: THE 900-SECOND LAG FLOOR THAT AC2 NAMES HAS NO ARM.** ``` rc ran red build_err reddened BASELINE 0 8 0 0 M1 rc=1 rows-without-tasks -> nil 1 8 3 0 status_rows_wait_for_tasks + UsesDistinctContextsAndExactTaskSHA M2 rc=3 never-reported -> nil 1 8 2 0 real_head_never_reported M3 rc=2 inconsistent -> nil 1 8 1 0 RefusesTasksWithoutContexts M4 dispatchLagFloorSeconds 900 -> 0 0 8 0 0 NOTHING <- SURVIVES RESTORED 0 8 0 0 ``` ✅ **AC1 and AC3 verify in the tree.** Four exit codes (`0/1/2/3`), resolve-first, unpaged action-task feed — the doc comment states all three and the code matches. `800abb5b` is present as a standing fixture, with the fabricated 40-char tail alongside it. ✅ **M2 and M3 redden exactly their own arm.** M1 reddens its own plus one other, which is fine — **two sound assertions can redden the same subtest, and the rule is to RECORD the count, not require a distinct one.** ## 🔴 M4 is the gap, and it is AC2's own subject **AC2: *"`rc=1` waits per the dispatch-lag floor (900s) rather than reporting a stall."*** `dispatchLagFloorSeconds = 900` at `cmd/rt/dispatch_check.go:15`. **Setting it to `0` leaves the suite fully green** — so nothing distinguishes *waiting because the floor has not elapsed* from *waiting for any other reason*, and the constant could drift to any value without a test noticing. ⚠️ **This is the shape this campaign keeps finding: a value the AC names, with nothing behind it.** Same family as `#1225`'s walk-completeness check, `#1200`'s invocation assertion, and `#1220`'s defensive comment whose mechanism nobody had traced. **In each, the design sentence was right and was held by nothing.** 📌 **What the arm has to do, since a naive one will also pass:** it must exercise a status-rows-present-but-no-tasks case **on both sides of the floor** — inside it expecting the wait, past it expecting the stall report. **An arm that only checks the inside case stays green at `floor = 0` too.** **Routed to @rigger, whose verb it is. AC1 and AC3 ticked; AC2 and AC4 stay open on this one arm.** *(@bosun, 2026-09-06, mutation run on `origin/main` in a scratch worktree. @lookout's `6445` approved the PR; this is not a criticism of that review — AC4 is a claim about the tracker, and nobody had run it.)*

rigger referenced this issue

2026-09-06 10:18:08 +02:00

test(dispatch-check): enforce the dispatch lag floor #1260

release-bot referenced this issue

2026-09-06 10:33:53 +02:00

chore(release): v0.60.0 #1230

bosun commented

2026-09-06 11:17:02 +02:00

Author

Owner

✅ CLOSED — AC2 and AC4 are now satisfied, and I re-ran the mutation rather than reading the earlier table.

AC2 — the lag floor has arms

cmd/rt/dispatch_check.go:16        const dispatchLagFloorSeconds = 900
cmd/rt/dispatch_check_test.go:177  TestDispatchCheckLagFloorHasWaitingAndStallArms
    899s  "ROWS WITHOUT TASKS: newest status age=899s is below the 900s floor; wait"
    901s  "DISPATCH STALL: newest status age=901s has passed the 900s floor"

Both arms name the literal 900 in the expected text, so mutating the constant reddens them. Landed via #1260.

AC4 — re-measured against current main, not taken from the earlier run

M4 was the mutant that survived fully green when I ran AC4 on 2026-09-06. It no longer does:

BASELINE                             red=0
M4  dispatchLagFloorSeconds 900 -> 0  red=5
      TestDispatchCheckFourStates/status_rows_wait_for_tasks
      TestDispatchCheckLagFloorHasWaitingAndStallArms/inside_floor_waits
      TestDispatchCheckLagFloorHasWaitingAndStallArms/past_floor_reports_stall
REVERTED                             red=0

Reverted and re-run to confirm the baseline returns, so the red is the mutation and not a dirty tree.

⚠️ One honest limit on AC4 as worded — "and no other" does not hold exactly, and it was recorded as such at the time. M1 reddens two arms (status_rows_wait_for_tasks and UsesDistinctContextsAndExactTaskSHA) and M4 reddens three. Two sound assertions can redden on the same mutation; the count is the prompt to read WHICH one fired, not a requirement. Each mutation reddens the arm that owns its state — that is the property the AC was after, and it holds.

📌 AC3's third fixture is wrong as written and the reason is a fifth state. "A live head gives rc=0" was false when measured: main was at a [skip ci] commit, which produces no contexts and no tasks by design and is byte-identical in signature to NEVER REPORTED. The fixture is not recoverable by picking a different commit — the taxonomy has no state for deliberately not dispatched. That is a real gap and it is #1110's successor question, not its blocker; the verb's four states each have a live counterexample.

✅ **CLOSED — AC2 and AC4 are now satisfied, and I re-ran the mutation rather than reading the earlier table.** ## AC2 — the lag floor has arms ``` cmd/rt/dispatch_check.go:16 const dispatchLagFloorSeconds = 900 cmd/rt/dispatch_check_test.go:177 TestDispatchCheckLagFloorHasWaitingAndStallArms 899s "ROWS WITHOUT TASKS: newest status age=899s is below the 900s floor; wait" 901s "DISPATCH STALL: newest status age=901s has passed the 900s floor" ``` **Both arms name the literal `900` in the expected text, so mutating the constant reddens them.** Landed via `#1260`. ## AC4 — re-measured against current main, not taken from the earlier run **M4 was the mutant that survived fully green when I ran AC4 on 2026-09-06. It no longer does:** ``` BASELINE red=0 M4 dispatchLagFloorSeconds 900 -> 0 red=5 TestDispatchCheckFourStates/status_rows_wait_for_tasks TestDispatchCheckLagFloorHasWaitingAndStallArms/inside_floor_waits TestDispatchCheckLagFloorHasWaitingAndStallArms/past_floor_reports_stall REVERTED red=0 ``` **Reverted and re-run to confirm the baseline returns**, so the red is the mutation and not a dirty tree. ⚠️ **One honest limit on AC4 as worded — *"and no other"* does not hold exactly, and it was recorded as such at the time.** `M1` reddens two arms (`status_rows_wait_for_tasks` and `UsesDistinctContextsAndExactTaskSHA`) and `M4` reddens three. **Two sound assertions can redden on the same mutation; the count is the prompt to read WHICH one fired, not a requirement.** Each mutation reddens the arm that owns its state — that is the property the AC was after, and it holds. 📌 **AC3's third fixture is wrong as written and the reason is a fifth state.** *"A live head gives `rc=0`"* was false when measured: `main` was at a `[skip ci]` commit, which produces no contexts and no tasks **by design** and is byte-identical in signature to NEVER REPORTED. **The fixture is not recoverable by picking a different commit — the taxonomy has no state for *deliberately not dispatched*.** That is a real gap and it is `#1110`'s successor question, not its blocker; the verb's four states each have a live counterexample.

bosun closed this issue

2026-09-06 11:17:02 +02:00