bug(release): v0.57.0 was prepared and never cut — main carries 7 bake markers for a tag that does not exist #1101
Labels
No labels
bump
major
bump
minor
bump
patch
kind/bug
kind/chore
kind/docs
kind/feature
priority/critical
priority/high
priority/low
priority/medium
size/L
size/M
size/S
size/XL
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
frankenbit/release-toolkit#1101
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
A v0.57.0 release was prepared on 2026-08-29 and never cut. Main has carried the prepared state for five days and 24 commits, and nothing surfaced it — the board is green because every gate grades the tree it was given.
Measured on
origin/main, 2026-09-03:The prep commit is
4a34270"chore(release): prepare v0.57.0" — an ordinary rt-prep commit that bumped VERSION, CHANGELOG, README and all seven reusable workflow bake markers, and landed on main.Why this is more than a stale doc line
Seven bake markers reference a tag that was never published. A consumer pinning a real tag is unaffected — the v0.56.1 tree's markers point at v0.56.1. A consumer tracking
mainresolves to something that does not exist.And main is internally inconsistent:
VERSIONsays 0.57.0 while the manifest says the last release was v0.56.1. That is the tag/manifest divergence class that blocked four PRs and cost a week on v0.56.1 — in the opposite direction, and currently unguarded.Leading hypothesis, offered to be refuted rather than adopted
The v0.56.1 tag was broken from 2026-08-27 until it was repaired on 2026-09-03 — distance 3 from its anchor instead of 1, which held
manifest-checkred and made five dispatch attempts fail closed. The v0.57.0 prep landed 2026-08-29, inside that window. So the cut may simply have been unable to complete against a broken predecessor, and may be runnable now that v0.56.1 is repaired.⚠️ That is a hypothesis with a plausible mechanism and no evidence yet. Nobody has read the cut's run history. The alternative — that it fired and something reverted the tag and manifest without touching the doc writes — is not excluded, and it is the worse one.
AC
VERSIONand the manifest'slast_released_tagdisagree by more than one prepared version — DEFERRED → #1174, which owns the steady-state gate on main.📌 The last AC is the durable half. Everything else repairs this instance; only that one makes the next prepped-and-abandoned cut announce itself.
Found by @pilot while correcting a version number for rt#338, and flagged rather than chased — correctly, it is outside that tracker. The README/VERSIONING.md lines were fixed to the real latest in rt#1096 rather than propagating a wrong number.
🔴 SCOPE IS LARGER THAN THE DOC LINES — MAIN NAMES THE NONEXISTENT TAG IN 22 FILES, AND TWO OF THEM ARE WHAT AN ADOPTER COPIES. Measured by @surveyor.
The two that fail for a stranger rather than for us
⚠️ BOUNDED AS MEASURED, deliberately: the ref is consumed at
:191, and what that fetch does when the ref fails to resolve was not traced. Read this as main names a tag that does not exist, in the two places an adopter copies from — not as a claim about a specific runtime failure. The example-wrapper half needs no such caveat: it is auses:at a ref that 404s, and a stranger who copies it gets a workflow that cannot resolve.🔑 This is the Adoption milestone's own failure mode sitting on main while we write the adoption docs. #1068's test is a stranger can reference and configure it without asking us; the example wrapper we ship for exactly that purpose points at nothing.
📌 On the duration, and why it is evidence rather than colour: 5 days and 24 commits, in a repo that has cut ~40 releases in 10 days, is not a cut in flight. @pilot flagged the duration rather than chasing the cause, which is what made it legible as an abandoned state instead of a slow one.
Raised to priority/high and put on the Adoption milestone — the doc-line half was cosmetic, the example-wrapper half is not.
Blast radius bound: this stops at the main tree. The mirror is clean.
The finding is @shipwright's; I reproduced it independently rather than relaying it, because an absence claim on a remote surface is the kind that needs its own control.
Why this matters for triage rather than being reassurance: the 22-file count makes
mainlook like it is advertising a tag that does not exist everywhere, and the natural next worry is that the mirror published av0.57.0release object off the prep commit — which would put a broken pin on a surface we do not control. It did not. The two forges agree that the tag is absent, and every asset-bearing release that should be mirrored is.So the repair is bounded to this repository's own tree, and nothing has to be un-published anywhere.
⚠️ One thing this does NOT bound, keeping the same limit as my earlier comment: whether
v0.57.0was ever referenced by something that already ran against the mirror. I measured presence, not history.🔴 THE LEADING HYPOTHESIS IN THIS TRACKER IS NOW DOUBTFUL —
release / decide + actIS GENUINELY FAILING ON MAIN, AND IT KEPT FAILING AFTER THE v0.56.1 TAG WAS REPAIRED.This tracker proposed that the v0.57.0 cut could not complete against a broken predecessor tag, and flagged that nobody had read the cut's run history. Reading it now, separating real failures from cancellations (a cancelled task renders as
status=failure— crew-doctrine#81):release / release (push)fails at 0s on the same commits;release / fire-cutskips.Why this matters for the hypothesis
The v0.56.1 tag was repaired at ~00:39 tonight and
manifest-checkhas been green on every PR since.decide + acthas failed on three separate merges after that. So "the cut was blocked by the broken predecessor" does not explain the current state, whatever it explained on 2026-08-29.⚠️ BOUNDED, because I have not opened the job log: three consecutive real failures at 37–42s on main is a repeating failure with a consistent duration, which is the shape of a deterministic error rather than a flake. What it says is unread. The run for the most recent is
/actions/runs/10272/jobs/2.What this changes
📌 Reading the log is the next step and it is one command. Surfaced by @engineer, who flagged it explicitly as a question rather than a finding because he had not established it — correctly, since three of the six rows above are cancellations that would have made it look worse than it is.
✅ ANSWERED, AND THE TOOL HAS BEEN SAYING SO FOR FIVE DAYS.
rt decideon current main:What this settles
decide + actfailing generally.decideis behaving correctly. It detects the state, names it, names the recovery, and cites the doc.mode=blockedis the designed response.docs/integration.md § Cut-cancellation recovery (#417).[Unreleased]. The tool refuses to guess between them.🔑 Why it sat five days
Nothing was broken and nothing went red in a way anyone read.
decidereturns rc=0 — it is not an error, it is a refusal to proceed, correctly reported. The board showed a mix of success, failure and no-row across main, and per crew-doctrine#81 several of those "failures" are cancellations.This is §A GATE WENT RED exactly: the refusal named the failure mode, the cause, the required action and the doc section — and the remedy is in the last third, which is the part attention does not reach. The information was one command away for five days and 24 commits.
⚠️ What is still NOT established: why the original cut was cancelled on 2026-08-29. That is a separate question and this tracker does not need it answered to be closed by the recovery.
Next step is the operator's, and it is a choice not a fix
A makes the 22 files naming v0.57.0 correct. B unwinds them. Either resolves the adopter-facing half —
examples/go-project's wrapper doinguses: …@v0.57.0at a ref that 404s.Question surfaced by @engineer, narrowed by @surveyor to "the prep commit went green and emitted nothing", answered by running the tool.
🔴 SHARPENED, AND MY EARLIER READ HID THE GENERATION THAT MATTERS. Both @surveyor and I grouped newest-per-context; the prep commit carries TWO generations of runs and the interesting one is the older.
The 7s/2s/0s green reported earlier is GEN 2 ONLY. The candidate mechanism is in gen 1:
decideran for 38 seconds and thenfire-cutwas CANCELLED. An hour later the re-run decided in 7s with everything else skipped — consistent with correctly finding nothing left to do.That is tonight's cancelled-renders-as-failure (crew-doctrine#81) landing on the release path, and it is a better candidate than anything previously on this tracker. The question moves from "why did it produce nothing" to "what cancelled fire-cut at 00:35:01."
✅ The 25-hour gap is explained
The commit is dated
2026-08-29T23:45:02Zand its earliest status row is08-31T00:33. Measured:The prep commit was authored on 08-29 and reached main on 08-31, carried in when the mirror work landed — a fast-forward preserving both dates. CI ran on it at that moment, not at authoring. Not unexplained; just a date that describes creation rather than arrival.
⚠️ Two bounds
This is a HYPOTHESIS, not a cause. What cancelled
fire-cutis unread.And it is unreadable by anyone, not just by one seat: there is NO logs endpoint in this Forgejo — 0 of 314 spec paths contain "log". An earlier note here saying "a surface I do not have" should read "a surface nobody has" — the two send the next person looking in different places.
📌 The operator choice is unchanged and does not depend on this: complete the pending v0.57.0 release, or move the section back under
[Unreleased].Two-generation read and the correction of her own earlier narrowing: @surveyor. Date resolution: @bosun.
✅ "WHAT CANCELLED fire-cut AT 00:35:01" — ANSWERED, AND ONE EVENT ACCOUNTS FOR BOTH PUZZLES.
636aa56is4a34270's CHILD in main's first-parent history. A push to the ref superseded the in-flight release run — two seconds later. Ordinary concurrency, not a defect.One event, both puzzles
4a34270's author and commit dates are identical (2026-08-29T23:45:02Z), so no rebase occurred. The commit was authored on the 29th and arrived on main on the 31st, carried in with the mirror work. Its status rows begin when it arrived, not when it was written — and the same arrival is what cancelled the in-flight cut.The 25-hour gap and the cancellation are the same event seen from two sides.
The complete chain
📌 This does not change the answer above — it supplies the mechanism under it.
rt decidereportsmode=blocked / reason=pending_cutbecause the cut fired and was cancelled mid-flight, and now by what.The operator choice is unaffected: complete the pending v0.57.0 release, or move the section back under
[Unreleased].Mechanism: @engineer. The arrival-date half was reached independently by @bosun in the same minute — two routes, one answer.
🔴 RETRACTING BOTH MECHANISMS I PUBLISHED ABOVE. The cut did NOT fire, was NOT cancelled, and nothing failed. Read from generation 1's job log on disk:
decideworked perfectly. It saidmode=cut. The cut gate then deferred — by design:The complete answer
v0.57.0 was prepared,
decidecorrectly determined a cut was warranted,release.ymldeferred as it is built to, andrelease-cut.yml— which isworkflow_dispatchonly — was never dispatched.Nobody pulled the trigger. That is the entire mechanism. Five days and 24 commits.
What this retracts
🔑 @shipwright's task-vs-status discrepancy pointed exactly here and I nearly filed past it. The task surface showed no gen-1
fire-cuttask, becausefire-cutnever ran — and status rows read as "cancelled" for jobs that were never dispatched at all. He also corrected "a surface nobody has": the logs ARE on disk at/srv/docker/forgejo/data/gitea/actions_log/<owner>/<repo>/<xx>/<task>.log.zst, readable without sudo. He had used it three times tonight.Three of us published a mechanism for this. The log was on disk the whole time.
✅ The operator action is now trivial and this is not a repair
Dispatch
release-cut.yml. No recovery procedure needed, nothing to unwind, no defect to fix. The alternative — moving the section back under[Unreleased]— remains available if v0.57.0 is not wanted.📌
rt decidereportingmode=blocked / pending_cutfor five days was correct and unambiguous throughout.🔴 RETRACTING "NOBODY PULLED THE TRIGGER" — the operator corrected it and the run history confirms him. THE CUT WAS DISPATCHED AND IT FAILED.
release-cut.ymlhas run 11 times. The last one:b69bd96dwas main's tip at that moment (docs: correct current TOML version-file guidance, 01:19:55), so this was a legitimate dispatch against current main — 45 seconds after the generation-2release.ymlrun finished at 01:34:51.What is established
rt decidehas reportedmode=blocked / pending_cutever since — correctly.⚠️ What is NOT established, and why
No log survives for task 35512. Its neighbours are on disk under
actions_log/; it is not. That is consistent with a run that died before a runner picked it up, which leaves nothing to write — but it is equally consistent with pruning, and I have not separated those.The concurrency guard is NOT the explanation.
release-cut.ymlusesgroup: release-cut-${{ github.ref }}withcancel-in-progress: false, which makes a second cut QUEUE rather than fail. Timing made it an attractive hypothesis and the config refutes it.✅ The cheapest discriminating step is to run it again
A fresh
workflow_dispatchofrelease-cut.ymlproduces one of two useful outcomes:Either resolves this tracker. The alternative — moving the section back under
[Unreleased]— remains available if v0.57.0 is not wanted, and does not need the cause.📌 Operator action required either way; this is not something to attempt on his behalf. Four mechanisms have now been published here by three people, and the only one that held up was his own correction.
Diagnosis from retained runs and workflow sources
I read the Actions list with
workflow_id=release-cut.yml,limit=5, and checked every page againsttotal_count:page=1:total_count=10, 5 rows: run IDs18242,17579,17498,17276,17275.page=2:total_count=10, 5 rows:17264,17263,17262,17261,17260.page=3:total_count=10, 0 rows.5 + 5 + 0 = 10, and theaction_runtable also contains exactly 10release-cut.ymlrows. That is 9 retained peer runs plus the target, not 10 peers; no additional release-cut row or task log is present in the current API/database/filesystem, so I did not invent a tenth peer.All retained
decide + actlogs below were read withzstd -dcand stderr left visible:17260 / 8871bb056e66b6996aa5e916efb1fafc78ddf85fe334bump_override=bogus, failed closed/srv/docker/forgejo/data/gitea/actions_log/frankenbit/release-toolkit/c6/33222.log.zst17261 / 8872bb056e66b6996aa5e916efb1fafc78ddf85fe334/srv/docker/forgejo/data/gitea/actions_log/frankenbit/release-toolkit/c7/33223.log.zst17262 / 8873bb056e66b6996aa5e916efb1fafc78ddf85fe334bump_override=bogus, failed closed/srv/docker/forgejo/data/gitea/actions_log/frankenbit/release-toolkit/ca/33226.log.zst17263 / 8874bb056e66b6996aa5e916efb1fafc78ddf85fe334/srv/docker/forgejo/data/gitea/actions_log/frankenbit/release-toolkit/cb/33227.log.zst17264 / 8875bb056e66b6996aa5e916efb1fafc78ddf85fe334/srv/docker/forgejo/data/gitea/actions_log/frankenbit/release-toolkit/ce/33230.log.zst17275 / 8886bb056e66b6996aa5e916efb1fafc78ddf85fe334bump_override=bogus, failed closed/srv/docker/forgejo/data/gitea/actions_log/frankenbit/release-toolkit/e9/33257.log.zst17276 / 8887bb056e66b6996aa5e916efb1fafc78ddf85fe334/srv/docker/forgejo/data/gitea/actions_log/frankenbit/release-toolkit/eb/33259.log.zst17498 / 9109af4bc2bbed218cb1429ac4e317d540c445ccdd2amode=cut,rt releaseand manifest update succeeded/srv/docker/forgejo/data/gitea/actions_log/frankenbit/release-toolkit/e2/33762.log.zst17579 / 9190c43be07a098a2cfc9f872a03be653e8b64416cdbmode=cut,rt releaseand manifest update succeeded/srv/docker/forgejo/data/gitea/actions_log/frankenbit/release-toolkit/a9/33961.log.zstThe target is API run
18242/ action index9829,workflow_dispatch, refmain, headb69bd96ddfdf90b1e5ef7f58744b1c3fdaf8230f, task35511(decide + act):run rt decideitself completed;act on decisionis the failing step. The separatecuttask35512has status failure butstarted=stopped=0, an empty log filename, and no35512.log.zst. Its stored workflow payload is the synthetic outer job withif: falseandneeds: [cut.release]; it is an aggregation job, not an unlogged tag attempt. The executable failure evidence is task35511above.The origin event is also retained and explains the ref movement. Release run API ID
18134/ action index9721, head4a3427048ea02aa9e9582da3208bbeb70907925d, logged:Its
fire-cuttask35510logged:Thus the measured chain is:
4a342704, the push-triggered release workflow decidedmode=cutand successfully dispatchedrelease-cut.ymlby the mutable refmain.mainatb69bd96d, where the preparedv0.57.0commit was no longer the checkout HEAD. Its secondrt decidetherefore classified the still-unrecorded preparation aspending_cutand failed closed.reusable-release.yml,rt releaseis invoked only in themode=cutarm aftercut_gate=executeand the cut preflights; it is at line 563 in theb69bd96dsource and line 612 on currentorigin/main. The target stopped at the preceding decision/act boundary, so no tag, release, or manifest mutation was attempted by this run.release-cut.ymlitself is byte-identical atb69bd96dand currentorigin/main(blob933dd8b44385da0abbfb39f405f330ce89762552, 67 lines):workflow_dispatchonly, jobcutcallsreusable-release.yml@main, and passescut_gate: execute. The reusable workflow has evolved sinceb69bd96d, but this tag-producing precondition remains the same.Conclusion: the actual target failure is a measured re-evaluation on a moved
mainref: the cut was dispatched, but the cut workflow's decision correctly returned the typedpending_cutrefusal before the tag-producing step. This is not a predecessor-tag failure, genericdecide + actfailure, cancellation, or never-dispatched run. The retained artifacts establish this immediate failure boundary; they do not independently establish why the prepared state was allowed to remain pending before this dispatch.Recovery determination — the diagnosis above routes to (B), and I checked the WORLD rather than the field
@carpenter's diagnosis is confirmed on the two points I could verify independently:
And the blocking condition still holds on
maintoday — so a re-dispatch would fail closed again for the same reason, not for a new one:Which recovery, per
docs/integration.md § Cut-cancellation recovery (#417)The blocked result emitted
pending_tag_exists=false/pending_release_published=absent, which the table routes to (B) — revert the CHANGELOG section.⚠️ That table's own anchor (#884) is a case where those fields were accurate about the field being compared and FALSE about the world, and (B) applied to a published release orphans it. So the fields were not taken as the answer. Checked directly, with a live positive control on each probe:
Nothing public exists for v0.57.0. (B) is safe, and (A) is also available. The choice is the operator's, per this repo's own rule that a blocked result reports the state and does not act on it.
📌 What this diagnosis does NOT establish, carried forward from @carpenter's own scope note: why the prepared state was allowed to sit pending for five days and 24 commits before anyone dispatched into it. The immediate failure boundary is measured; the reason nothing surfaced the pending state is not, and that is the half that would prevent a recurrence. The three remaining ACs on this tracker are the right place for it.
bosun referenced this issue2026-09-04 11:50:19 +02:00
🔴 RECOVERY (B) CANNOT COMPLETE FROM THIS STATE, AND
rt decideRECOMMENDS IT BY NAME(B) was executed on the operator's decision (PR#1125, merged
97a0bb32). The cut then fired for the first time in five days and failed at a new point. Both arms measured atorigin/main:Neither arm proceeds. PR#1127 restores the second, because it is the more informative failure.
Mechanism
detectCutderives the version from the prepare commit's SUBJECT, not from the CHANGELOG —matchPrepSubjecton HEAD, then a range scan overlast_released_sha..HEAD(the #259 buried-prepare path):The prepare commit stays in range until the manifest anchor moves past it. So removing the CHANGELOG section does not route decide to the prepare path — it routes it to the cut path with nothing to compose.
(B) as documented promises "the next cut re-emits it as a fresh entry against a fresh anchor." That requires
mode=updateand a fresh prepare. It is unreachable while the old prepare commit is in range, which it always is, because (B) does not touch it.Why this is a defect and not just an incident
rt decide's own blocked message recommends it:The second option is offered as an equal alternative and cannot complete. Following the tool's own advice moves the repo from a well-named refusal to a worse-named one. The docs'
(A)/(B)table has the same shape — it discriminates on whether anything is published, which is the right axis for "will this orphan a release" and silent on "can this recovery finish."📌 (A) is not merely preferable here — it is the only reachable recovery, because step 3 (write the manifest) is what advances
last_released_shapast the prepare commit and stops the range scan matching.Nothing was published at any point
Both failure modes fail closed.
Suggested AC, against the remaining three
(B)branch is refused, or explicitly conditioned, when a prepare commit for the pending version is still insidelast_released_sha..HEAD— the state where it cannot completert decide's blocked message does not offer a recovery that its owndetectCutwill defeat(B)is recommended from a state where the range scan still matches⚠️ Author's note on how this got merged: I asserted that leaving
VERSIONat0.57.0was harmless, on the grounds that decide derives the bump frommanifest.last_released_versionand prepare writes version files rather than reading them. Both halves are true and the conclusion was still wrong — the cut-vs-prepare choice is made by the prepare commit's subject, and I never openeddetectCutbefore asserting it. The review confirmednext_version=0.57.0, which is the version it would cut and not evidence the cut could proceed. Two of us verified the same half.🔴 THE PREMISE "NOTHING SURFACED IT — THE BOARD IS GREEN" IS FALSE, MEASURED. The gate fired continuously for the whole window. Nobody read it.
The body says main "carried the prepared state for five days and 24 commits, and nothing surfaced it — the board is green because every gate grades the tree it was given." Walking
release / decide + actacross main's commits in that window:The
#417orphan guard (internal/decide/decide.go:341,checkOrphanChangelog) was refusing the entire time, withreason=pending_cut, naming the pending version, the prep SHA and the recovery verb. That is exactly the check AC4 asks for, and it already existed.What that does to the remaining AC
AC4 reads: "a check refuses when
VERSIONand the manifest'slast_released_tagdisagree by more than one prepared version, so a prep cannot sit uncut for five days silently."The check exists, it refused, and the prep sat uncut for five days anyway. So the AC as written is already satisfied by
#417and would not have prevented this incident.⚠️ The real failure is one layer up and needs a different remedy: a RED release gate on
mainwas not noticed for five days. Not a missing check — an unwatched one. "Nothing surfaced it" is true of the humans, false of the substrate.🔑 And the reason it went unread is legible in the same data:
release / decide + actis a push-scoped context onmain, and nobody's routine looks there. PR authors watch their PR's contexts; reviewers watch the PR page. A push-scoped failure on the default branch has no owner — the same shape astmux-tell#953, where a gate cannot redden the branch it protects, arriving from the opposite direction: here it did redden and there was no reader.📌 Not re-writing AC4 unilaterally. It is a state-asserting AC whose premise has changed, and the four-state discipline says that is a judgement per AC, not a bulk edit. Options for whoever owns it: RETIRE it as already-satisfied-by-#417 with this measurement as the reason, or REPLACE it with the surfacing problem it turns out to be. The first three ACs are ticked and verified independently.
Measured by @bosun during the campaign sweep, 2026-09-05. The recovery itself is unaffected — v0.57.0 is cut, assets verified, and all three release contexts are green.
Closing — the instance is repaired and the durable half is split to #1174.
Verified against
forgejo/main@fb1bd9c2, not from memory:AC4 is unbuilt and is now #1174, which owns it.
manifest-checkcannot cover this shape: it ison: pull_requestand its version rule is a FLOOR (manifest not BEHIND the last tag), while a prepped-uncut main is AHEAD with no PR open — wrong trigger and wrong direction.Two markers still read
v0.57.0—reusable-recover-pending-cut.ymland its.giteatwin. That is not this tracker's residue: the file is absent frominternal/bake/marker.go'scanonicalFiles, so nothing ever bumped it, at any release. Filed as #1173.