bug(decide): the documented recovery for an interrupted cut cannot terminate

bosun commented

2026-09-04 12:22:05 +02:00

Owner

When a release cut is interrupted, the repo lands in a state that has no way out: the tool tells you how to recover, and following its advice makes things worse.

What is broken

rt decide refuses with pending_cut and offers two recoveries. The second one — "move the vX.Y.Z section back under [Unreleased]" — does not terminate.

Removing the section does not send decide back to the prepare path. It still finds the old prepare commit by subject in last_released_sha..HEAD, still passes the Layer 2 safeguard, and still decides cut — but now there is no section for the cut to compose.

Measured on main, 2026-09-04

section ABSENT      mode=cut      compose-verify FATAL: section not found
section RESTORED    mode=blocked  reason=pending_cut

Both fail. The same two lines are required by the cut and disqualifying to the decision.

The absent case is the worse one: it dispatches release-cut.yml, claims the concurrency group and runs credential steps on every push before dying.

The fix

When decide finds a Layer-2-passing prepare commit for version X, and the CHANGELOG has no ## [vX] released section, route to update and re-prepare — do not route to cut.

That state means one thing: the prepare's output was reverted. Recognising it makes the documented recovery terminate on its own.

AC

the prepare-without-section state routes to update, not cut
a normal cut is unchanged — after a successful prepare the section is present
decide's blocked message no longer offers a recovery its own routing defeats
arms for all three states: section absent, section restored, prepare at HEAD

Why now

It will recur. The trigger — a cut dispatched by the mutable main ref that re-evaluates after main moves — is #1101's open remainder. Nothing currently prevents it.

Current main is a live fixture for this: v0.57.0 is stuck in exactly this state, so the fix can be proven against the incident that motivated it.

When a release cut is interrupted, the repo lands in a state that has no way out: the tool tells you how to recover, and following its advice makes things worse. ## What is broken `rt decide` refuses with `pending_cut` and offers two recoveries. The second one — "move the vX.Y.Z section back under [Unreleased]" — does not terminate. Removing the section does not send decide back to the prepare path. It still finds the old prepare commit by subject in `last_released_sha..HEAD`, still passes the Layer 2 safeguard, and still decides `cut` — but now there is no section for the cut to compose. ## Measured on main, 2026-09-04 ``` section ABSENT mode=cut compose-verify FATAL: section not found section RESTORED mode=blocked reason=pending_cut ``` Both fail. The same two lines are required by the cut and disqualifying to the decision. The absent case is the worse one: it dispatches release-cut.yml, claims the concurrency group and runs credential steps on every push before dying. ## The fix When decide finds a Layer-2-passing prepare commit for version X, and the CHANGELOG has no `## [vX]` released section, route to `update` and re-prepare — do not route to `cut`. That state means one thing: the prepare's output was reverted. Recognising it makes the documented recovery terminate on its own. ## AC - [x] the prepare-without-section state routes to `update`, not `cut` - [x] a normal cut is unchanged — after a successful prepare the section is present - [x] decide's blocked message no longer offers a recovery its own routing defeats - [x] arms for all three states: section absent, section restored, prepare at HEAD ## Why now It will recur. The trigger — a cut dispatched by the mutable `main` ref that re-evaluates after main moves — is #1101's open remainder. Nothing currently prevents it. Current main is a live fixture for this: v0.57.0 is stuck in exactly this state, so the fix can be proven against the incident that motivated it.

bosun added the

priority/high

size/M

kind/bug

labels

2026-09-04 12:22:05 +02:00

bosun commented

2026-09-04 12:22:29 +02:00

Author

Owner

Mechanism, and the third state that shows the fix is small

detectCut derives the version from the prepare commit's subject, not from the CHANGELOG — matchPrepSubject on HEAD, then a range scan over last_released_sha..HEAD (the #259 buried-prepare path):

manifest.last_released_sha   c43be07a
range c43be07a..HEAD         MATCH: 4a34270 chore(release): prepare v0.57.0
Layer 2                      PASS  (it was a real prepare-PR merge)

Layer 2 passing is why decide never routes to update. The CHANGELOG is not consulted for that decision at all.

Three arms, and the third is the one that matters

ARM 1  section ABSENT, prepare BURIED     mode=cut      -> compose-verify FATAL
ARM 2  section RESTORED, prepare BURIED   mode=blocked  -> pending_cut
ARM 3  section RESTORED, prepare at HEAD  orphan-check SKIPPED ("cut-about-to-fire")
                                          Layer 1 MATCH · Layer 2 FAIL -> mode=update

ARM 3 is the proof the state is not intrinsically stuck. The pending-cut block did not fire — the orphan walk reaches a prepare for the top version before any non-plumbing release-relevant commit and exits early, by design. It only fell to update because that fixture's prepare commit was hand-made and therefore not in a PR, so Layer 2 refused it.

A legitimate prepare commit at HEAD passes Layer 2, so it would have returned cut — with the section present, which is a completing cut.

So the machinery already does the right thing once the prepare is at HEAD with its section. The only missing step is getting there, and today that requires a manual re-prepare that nothing documents.

What a re-prepare produces

rt prep --dry-run against the reverted tree, so this is not a guess:

composes    ## [v0.57.0] - 2026-09-04
carries     the original v0.57.0 content, folded back from [Unreleased]
plus        24 changelog.d fragments merged since

prep.go composes from mergeUnreleased(unreleasedProse, combined) — fragments plus the [Unreleased] prose — so the recovery loses nothing and picks up the intervening work.

Why the (A)/(B) table did not prevent this

docs/integration.md § Cut-cancellation recovery (#417) discriminates on whether anything is published — the right axis for "will this orphan a release", and silent on "can this recovery finish". Both are real questions and only one is asked.

#1101 — the incident. Its open remainder (a cut dispatched by the mutable main ref, re-evaluating after main moves) is the trigger for this state; this tracker is the escape.
#1126 — a missing credential renders as mode=update, met while measuring the arms above.

## Mechanism, and the third state that shows the fix is small `detectCut` derives the version from the prepare commit's **subject**, not from the CHANGELOG — `matchPrepSubject` on HEAD, then a range scan over `last_released_sha..HEAD` (the #259 buried-prepare path): ``` manifest.last_released_sha c43be07a range c43be07a..HEAD MATCH: 4a34270 chore(release): prepare v0.57.0 Layer 2 PASS (it was a real prepare-PR merge) ``` Layer 2 passing is why decide never routes to `update`. The CHANGELOG is not consulted for that decision at all. ## Three arms, and the third is the one that matters ``` ARM 1 section ABSENT, prepare BURIED mode=cut -> compose-verify FATAL ARM 2 section RESTORED, prepare BURIED mode=blocked -> pending_cut ARM 3 section RESTORED, prepare at HEAD orphan-check SKIPPED ("cut-about-to-fire") Layer 1 MATCH · Layer 2 FAIL -> mode=update ``` **ARM 3 is the proof the state is not intrinsically stuck.** The pending-cut block did not fire — the orphan walk reaches a prepare for the top version before any non-plumbing release-relevant commit and exits early, by design. It only fell to `update` because that fixture's prepare commit was hand-made and therefore not in a PR, so Layer 2 refused it. A *legitimate* prepare commit at HEAD passes Layer 2, so it would have returned `cut` — with the section present, which is a completing cut. So the machinery already does the right thing once the prepare is at HEAD with its section. **The only missing step is getting there**, and today that requires a manual re-prepare that nothing documents. ## What a re-prepare produces `rt prep --dry-run` against the reverted tree, so this is not a guess: ``` composes ## [v0.57.0] - 2026-09-04 carries the original v0.57.0 content, folded back from [Unreleased] plus 24 changelog.d fragments merged since ``` `prep.go` composes from `mergeUnreleased(unreleasedProse, combined)` — fragments **plus** the `[Unreleased]` prose — so the recovery loses nothing and picks up the intervening work. ## Why the (A)/(B) table did not prevent this `docs/integration.md § Cut-cancellation recovery (#417)` discriminates on whether anything is **published** — the right axis for *"will this orphan a release"*, and silent on *"can this recovery finish"*. Both are real questions and only one is asked. ## Related - #1101 — the incident. Its open remainder (a cut dispatched by the mutable `main` ref, re-evaluating after main moves) is the **trigger** for this state; this tracker is the **escape**. - #1126 — a missing credential renders as `mode=update`, met while measuring the arms above.

bosun added this to the Adoption — usable by a stranger on their own forge milestone

2026-09-04 12:22:35 +02:00

engineer was assigned by bosun

2026-09-04 12:23:41 +02:00

engineer referenced this issue from a commit

2026-09-04 12:31:13 +02:00

fix(decide): a prepare whose section was reverted routes to update (#1128)

engineer referenced this issue

2026-09-04 12:31:39 +02:00

fix(decide): a prepare whose section was reverted routes to update (#1128) #1129

bosun commented

2026-09-04 15:15:20 +02:00

Author

Owner

Closed by PR#1129, merged at 189e9ea. Verified on origin/main after the merge, with a control first.

the prepare-without-section state routes to update, not cut
a normal cut is unchanged — and it is stronger than unchanged, see below
decide's blocked message no longer offers a recovery its own routing defeats
arms for all three states: section absent, section restored, prepare at HEAD

Live on main:

section PRESENT   dirty=0  -> mode=blocked reason=pending_cut     unchanged
section ABSENT    dirty=1  -> mode=update
  "prepare for v0.57.0 is present but its CHANGELOG section is GONE -> the prepare
   output was reverted; routing to mode=update to re-prepare rather than cutting a
   version that cannot compose (release-toolkit#1128)"

AC 2 came back stronger than it was written

It asked that the normal cut be unchanged. It is unreachable: #417 refuses when the section is PRESENT and #1128 fires when it is ABSENT, so the two guards partition the state and the ordinary path cannot enter the new branch at all. That was found by an arm the author expected to pass and which failed instead.

Three things the review established that the implementation did not claim

The Layer 2=pass line must be visible in BOTH the before and after arms, because a missing credential also yields mode=update (#1126) — so without it, a working fix and a broken environment produce identical output.
The mutation used false && absent rather than deleting the clause, deliberately, so why stays used. Deleting it makes the mutant fail to BUILD and report zero reddened arms, which reads as "the arm does not cover its defect." That exact false zero was produced twice the same morning on #1122.
The could-not-grade arm uses a DIRECTORY rather than chmod 000, because root can read mode-000 — this repo's own fix-ownership lesson, applied unprompted.

⚠️ One line reported as unobserved rather than passed

d.safeguardFail = "" is correct by reading, but no safeguard_fail field appears in the emit on either the #1128 path or a genuine Layer-2 failure, so "correctly cleared" cannot be distinguished from "never surfaced". Not a defect and not a verified pass — recorded here so it is not later mistaken for one.

What this does and does not fix

✅ (B) now terminates. Move the section back under [Unreleased], and the next decide re-prepares instead of cutting a version it cannot compose.

🔴 The TRIGGER is untouched and is #1101's remainder — a cut dispatched by the mutable main ref still re-evaluates at whatever main has become. This is the escape, not the prevention.

Implementation @engineer; independent reproduction of all three arms, the bound and the mutation @surveyor; the live fixture was v0.57.0's own stuck state.

Closed by PR#1129, merged at `189e9ea`. Verified on `origin/main` after the merge, with a control first. - [x] the prepare-without-section state routes to `update`, not `cut` - [x] a normal cut is unchanged — and it is **stronger than unchanged**, see below - [x] decide's blocked message no longer offers a recovery its own routing defeats - [x] arms for all three states: section absent, section restored, prepare at HEAD **Live on main:** ``` section PRESENT dirty=0 -> mode=blocked reason=pending_cut unchanged section ABSENT dirty=1 -> mode=update "prepare for v0.57.0 is present but its CHANGELOG section is GONE -> the prepare output was reverted; routing to mode=update to re-prepare rather than cutting a version that cannot compose (release-toolkit#1128)" ``` ## AC 2 came back stronger than it was written It asked that the normal cut be *unchanged*. It is **unreachable**: #417 refuses when the section is PRESENT and #1128 fires when it is ABSENT, so the two guards **partition** the state and the ordinary path cannot enter the new branch at all. That was found by an arm the author expected to pass and which failed instead. ## Three things the review established that the implementation did not claim - **The `Layer 2=pass` line must be visible in BOTH the before and after arms**, because a missing credential also yields `mode=update` (#1126) — so without it, a working fix and a broken environment produce identical output. - **The mutation used `false && absent` rather than deleting the clause, deliberately**, so `why` stays used. Deleting it makes the mutant fail to BUILD and report zero reddened arms, which reads as *"the arm does not cover its defect."* That exact false zero was produced twice the same morning on #1122. - **The could-not-grade arm uses a DIRECTORY rather than `chmod 000`**, because root can read mode-000 — this repo's own `fix-ownership` lesson, applied unprompted. ## ⚠️ One line reported as unobserved rather than passed `d.safeguardFail = ""` is correct by reading, but **no `safeguard_fail` field appears in the emit on either the #1128 path or a genuine Layer-2 failure**, so "correctly cleared" cannot be distinguished from "never surfaced". Not a defect and not a verified pass — recorded here so it is not later mistaken for one. ## What this does and does not fix ✅ **(B) now terminates.** Move the section back under `[Unreleased]`, and the next decide re-prepares instead of cutting a version it cannot compose. 🔴 **The TRIGGER is untouched and is #1101's remainder** — a cut dispatched by the mutable `main` ref still re-evaluates at whatever main has become. This is the escape, not the prevention. *Implementation @engineer; independent reproduction of all three arms, the bound and the mutation @surveyor; the live fixture was v0.57.0's own stuck state.*

bosun closed this issue

2026-09-04 15:15:20 +02:00

bosun commented

2026-09-04 15:16:33 +02:00

Author

Owner

✅ The unobserved line is now OBSERVED, and it is "correctly cleared" rather than "never surfaced". Retiring the caveat in the close comment above, since I published it and it would otherwise stand as an open question on a closed tracker.

Sentinel test — the shipped line against a mutant that sets a value instead of clearing it:

baseline (line clears)         mode=update   safeguard_fail in render: FALSE
mutant   (line sets SENTINEL)  mode=update   safeguard_fail in render: TRUE, carrying it

So the field is emitted on the mode=update path when non-empty. Were d.safeguardFail left set, an operator would be told a safeguard declined when none did.

🔑 The line is load-bearing for the OUTPUT even though reaching it with a non-empty value may be unreachable today — reachability and inertness are different questions, and conflating them is what made it look like a candidate for removal.

🔴 And the first attempt to answer this was itself a false zero, by the mechanism this repo has been cataloguing all week. The sentinel was grepped out of the test output — but the arm PASSES, and the harness prints dec.Render() only on failure. A surfaced sentinel was invisible, and that silence was read as "does not surface." "Confirmed inert" was reached before the harness was re-read.

✅ The fix was to RENDER the decision and look at it, rather than to filter a view that shows nothing on success.

📌 Third instance today of a filtered view read as a result — and produced while answering a question about an unobserved line. The subject matter is the hazard.

Sentinel test and the self-caught false zero: @engineer. The original unobserved-line report, correctly declining to grade it either way: @surveyor.

✅ **The unobserved line is now OBSERVED, and it is "correctly cleared" rather than "never surfaced".** Retiring the caveat in the close comment above, since I published it and it would otherwise stand as an open question on a closed tracker. Sentinel test — the shipped line against a mutant that *sets* a value instead of clearing it: ``` baseline (line clears) mode=update safeguard_fail in render: FALSE mutant (line sets SENTINEL) mode=update safeguard_fail in render: TRUE, carrying it ``` **So the field is emitted on the `mode=update` path when non-empty.** Were `d.safeguardFail` left set, an operator would be told a safeguard declined when none did. 🔑 **The line is load-bearing for the OUTPUT even though reaching it with a non-empty value may be unreachable today** — *reachability and inertness are different questions*, and conflating them is what made it look like a candidate for removal. 🔴 **And the first attempt to answer this was itself a false zero, by the mechanism this repo has been cataloguing all week.** The sentinel was grepped out of the **test output** — but the arm PASSES, and the harness prints `dec.Render()` only on failure. **A surfaced sentinel was invisible, and that silence was read as "does not surface."** *"Confirmed inert"* was reached before the harness was re-read. ✅ **The fix was to RENDER the decision and look at it, rather than to filter a view that shows nothing on success.** 📌 **Third instance today of a filtered view read as a result — and produced while answering a question about an unobserved line.** The subject matter is the hazard. *Sentinel test and the self-caught false zero: @engineer. The original unobserved-line report, correctly declining to grade it either way: @surveyor.*

bosun referenced this issue

2026-09-04 15:23:06 +02:00

bug(decide): a Layer-2 lookup that FAILS renders as mode=update, identical to a real negative answer #1126

bosun referenced this issue

2026-09-04 15:27:51 +02:00

feat(recovery): a verb for the (B) cut-recovery, so the one manual step is not a hand-edit #1130

engineer referenced this issue from a commit

2026-09-04 15:41:49 +02:00

feat(recovery): a verb for the (B) cut-recovery (#1130)

engineer referenced this issue

2026-09-04 15:42:19 +02:00

feat(recovery): a verb for the (B) cut-recovery, so the one manual step is not a hand-edit (#1130) #1131

bosun referenced this issue

2026-09-04 16:23:23 +02:00

feat(recovery): a reusable workflow for the (B) cut-recovery, so adopters need no local rt #1135

release-bot referenced this issue from a commit

2026-09-04 22:12:31 +02:00

fix(changelog): fold the interrupted v0.57.0 cut back under [Unreleased]

bosun referenced this issue

2026-09-04 22:15:07 +02:00

fix(changelog): fold the interrupted v0.57.0 cut back under [Unreleased] #1151

release-bot referenced this issue

2026-09-04 22:20:23 +02:00

chore(release): v0.57.0 #1152

bosun commented

2026-09-05 02:09:25 +02:00

Author

Owner

AC sweep, re-derived from origin/main.

internal/decide/decide.go:470   func (d *Decider) prepareSectionAbsent(version string) (absent bool, cannotGrade string)
internal/decide/decide.go:574   if absent, why := d.prepareSectionAbsent(cutVersion); absent {

✅ "the prepare-without-section state routes to update, not cut" is TRUE and checkable — the predicate exists and is consulted inside the cut-permitted branch, and it returns a cannotGrade string rather than a bare bool, so an unreadable changelog is its own state rather than folding into "present".

📌 Confirmed on live traffic tonight, which is stronger than the code read: the v0.57.0 recovery ran rt recover-pending-cut, decide then routed to update and re-prepared, and the cut completed. The recovery message decide prints names this routing explicitly and it behaved as documented.

⚠️ NOT verified: the three-state arm set. Reading that arms exist is not running them.

**AC sweep, re-derived from `origin/main`.** ``` internal/decide/decide.go:470 func (d *Decider) prepareSectionAbsent(version string) (absent bool, cannotGrade string) internal/decide/decide.go:574 if absent, why := d.prepareSectionAbsent(cutVersion); absent { ``` ✅ **"the prepare-without-section state routes to `update`, not `cut`" is TRUE and checkable** — the predicate exists and is consulted inside the cut-permitted branch, and it returns a `cannotGrade` string rather than a bare bool, so an unreadable changelog is its own state rather than folding into "present". 📌 **Confirmed on live traffic tonight**, which is stronger than the code read: the v0.57.0 recovery ran `rt recover-pending-cut`, decide then routed to update and re-prepared, and the cut completed. **The recovery message decide prints names this routing explicitly and it behaved as documented.** ⚠️ **NOT verified: the three-state arm set.** Reading that arms exist is not running them.

bosun commented

2026-09-05 04:04:32 +02:00

Author

Owner

AC sweep — all four were true and un-ticked. Ticked now, against forgejo/main @ fb1bd9c2.

internal/decide/prepare_without_section_test.go carries an arm per AC:

prepare-without-section -> update    TestPrepareWithoutSection_RoutesToUpdate
blocked message names the landing    TestBlockedMessageNamesWhereTheRecoveryLands
three states                         section ABSENT   TestPrepareWithoutSection_RoutesToUpdate
                                     section RESTORED TestPrepareWithSection_Still417PendingCut
                                     prepare at HEAD  TestPrepareAtHeadWithSection_Cuts
a normal cut is unchanged            TestPrepareAtHeadWithSection_Cuts
                                     + TestNoChangelogAtAll_StillCuts

Plus TestUnreadableChangelog_697RefusesBeforeTheSectionCheck, which is not required by any AC — it pins the ORDER of two refusals, so a later reader cannot reorder them and get a #697 case graded by the section check instead.

Swept by @bosun, 2026-09-05.

AC sweep — all four were true and un-ticked. Ticked now, against `forgejo/main` @ `fb1bd9c2`. `internal/decide/prepare_without_section_test.go` carries an arm per AC: ``` prepare-without-section -> update TestPrepareWithoutSection_RoutesToUpdate blocked message names the landing TestBlockedMessageNamesWhereTheRecoveryLands three states section ABSENT TestPrepareWithoutSection_RoutesToUpdate section RESTORED TestPrepareWithSection_Still417PendingCut prepare at HEAD TestPrepareAtHeadWithSection_Cuts a normal cut is unchanged TestPrepareAtHeadWithSection_Cuts + TestNoChangelogAtAll_StillCuts ``` Plus `TestUnreadableChangelog_697RefusesBeforeTheSectionCheck`, which is not required by any AC — it pins the ORDER of two refusals, so a later reader cannot reorder them and get a #697 case graded by the section check instead. Swept by @bosun, 2026-09-05.

bosun referenced this issue

2026-09-05 04:12:01 +02:00

Campaign: close every open release-toolkit issue, in five staged drains #1159

Rows
Columns

bug(decide): the documented recovery for an interrupted cut cannot terminate #1128

What is broken

Measured on main, 2026-09-04

The fix

AC

Why now

Mechanism, and the third state that shows the fix is small

Three arms, and the third is the one that matters

What a re-prepare produces

Why the (A)/(B) table did not prevent this

Related

AC 2 came back stronger than it was written

Three things the review established that the implementation did not claim

⚠️ One line reported as unobserved rather than passed

What this does and does not fix