reusable-release: a cut that fails after partial progress leaves inconsistent state (manifest orphan); no concurrency guard against overlapping retries #499
Labels
No labels
bump
major
bump
minor
bump
patch
kind/bug
kind/chore
kind/docs
kind/feature
priority/critical
priority/high
priority/low
priority/medium
size/L
size/M
size/S
size/XL
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
frankenbit/release-toolkit#499
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Filed by Shipwright, 2026-07-24, as substrate-of-record from tonight's tmux-tell v0.35.0 cut. No urgency — a problem statement + candidate remedies with open questions, not a prescriptive fix.
MEASURED (tonight's v0.35.0 arc, tmux-tell)
compose-verify(density gate) — my#842changelog fragment was a 529-word paragraph over the ≤100-words/paragraph limit. This is a step that runs before the manifest-update step.release-prep/branch to passcheck-changelog-placement), and then a#417-shape manifest orphan: the CHANGELOG's top-most released section no longer matchedmanifest.last_released_version, sorelease-decide.shwent FATAL. It needed a manual.release-toolkit-manifest.jsonreconcile (tmux-tell PR#858) plus a manual tag+release+publish by the operator.So the concrete, reproduced failure is partial-progress non-atomicity: a cut that fails partway through leaves CHANGELOG/manifest/tag in a mutually-inconsistent state that a later run reads as FATAL rather than as "retry cleanly."
HYPOTHESIS (not reproduced — flagged as hypothesis)
Once a cut is in that half-written state, the natural operator response is retry via subsequent pushes. If two release runs overlap (a retry firing while a prior run is still mid-write), or a retry runs against the half-written state, the inconsistency can compound into a duplicate-cut / orphan race rather than self-heal. I did not reproduce an overlapping-run race tonight — the orphan came from a single partial run — but the absence of a concurrency guard is what makes the compounding path reachable.
Candidate remedies (each with an open question)
concurrency:onreusable-release.yml's job, e.g.cancel-in-progress: false(serialise, do NOT cancel a run mid-write — cancelling mid-manifest-write is how you'd manufacture the orphan).concurrency:at the reusable-workflow level? MEASURED in a consumer workflow it does NOT: tmux-tell'sdeploy.ymlcarries a comment (line ~14) that aconcurrency:block "tripped the schema validator and was dropped." Whether the reusable workflow (this repo) is subject to the same rejection is unverified — if it is, remedy 1 is not viable and remedy 2 is the path.flockatdraft-release.shentry (a substrate-honest lock, independent of the YAML schema). Serialises at the script rather than the workflow, sidestepping the validator question entirely.compose-verify(and the other pre-manifest gates) run before any manifest/tag mutation, a failure leaves nothing half-written and there is no orphan to race on. Tonight's failure was pre-manifest, so this may be the cheaper root fix; worth checking the actual step ordering inreusable-release.yml.Why here and not in the consumer
The consumer can't carry
concurrency:(schema validator, per remedy 1's open question). The lock/atomicity belongs at the reusable-workflow /draft-release.shlevel — this repo — so it protects every consumer uniformly.Anchor
tmux-tell v0.35.0 cut, 2026-07-24: compose-verify density failure →
release-prep/re-categorisation →#417-shape manifest orphan → manual reconcile (tmux-tell PR#858) + manual tag/release/publish. Related:#417(manifest orphan → release-decide FATAL),#488(fragment density gate that fired first),#471(changelog-placement, part of the recovery path).Substrate expansion — fragment-consumption gap is sibling symptom of same partial-progress class (operator disposition 2026-07-24)
The sibling symptom
Tonight's tmux-tell v0.35.0 arc surfaced a second failure shape from the same partial-progress class the original body names — this one on the release-notes-vs-binary-content axis, not the manifest-vs-CHANGELOG axis.
MEASURED:
719.1.added.mdwent tochangelog.d/but the rolling PR was already gone. The CHANGELOG.md [v0.35.0] section (frozen at rolling-PR-merge time) never got updated, so manual-recovery's release body was missing PR#854's attributionFixed as post-publish hotfix on tmux-tell v0.35.0 release notes (2026-07-24). The stranded fragment is now in the release body it belongs to.
Class relation to the original body
Original #499 body names two failure shapes from partial-progress:
The fragment-consumption gap is a third shape from the same root:
release-prep.sh --rolling-mode(manual API tag+release, hand-crafted recovery scripts) inherits this gapOperator disposition: Option 3 (fail-atomic step ordering)
Operator explicitly preferred the structural root fix over the post-publish warning:
Design implication for Option 3: fail-atomic isn't just about the manifest/tag/CHANGELOG triple staying consistent — it should also ensure no fragment ever gets stranded across cut boundaries. Two ways this could land:
changelog.d/and INJECT any fragments-not-yet-in-[X.Y.Z] into the section before tagging. This makes the section content authoritative at the moment of tag rather than at rolling-PR-merge.Either shape closes the gap tonight's arc exhibited. Both are architecturally deeper than the concurrency guard (remedy 1) or the flock (remedy 2), matching operator's preference for the root fix.
Anchor
CLOSED at Phase 6 gate (2026-07-27) — fail-atomic transactional cut design folded into
internal/release.Cutterper ADR-0009 §6 direction. Substrate delivery + verification evidence enumerated below.Original concerns from this tracker
Both closed by substrate delivery across #554/#556/#557/#558/#555 — not as follow-up work, but as first-class design per Milestone #78 explicit direction.
Substrate delivery + verification evidence
Fail-atomic design (concern 1)
Substrate:
internal/release.Cutterimplements two-phase transactional cut per ADR-0009 §6:Prepare(ctx, req)composes changelog seal + manifest write + fragment delete (all restorable)Fire(ctx, txn)creates release draft (idempotent viaGetReleaseByTag+ manifest-compare replay)Transaction.Rollback()restores manifest + changelog + fragments in reverse orderVerified via:
fileDeleted=true ∧ bodyInCHANGELOG=false; byte-exact revert restores green). Property test covers (a)/(b)/(c) rollback paths.nilManifest-partialDelete-rollbackcompanion test.Concurrency guard (concern 2)
Substrate:
flocknon-blocking (LOCK_EX|LOCK_NB) at Cutter construction, repo-root path →ErrConcurrentCut.#557 additionally: Fire wired as first production caller with
existing.TargetCommitishunconditional compare (per Surveyor 4612 fold), FATAL-on-divergence semantics matching bash oracle.Composite-milestone-gate design vindication
The design chose property-invariant + unit at Cutter substrate + harness gate at consumer + Surveyor adversarial-verify at review. Across the arc:
5 composite-gate vindications — the layered gate design surfaces defects each layer alone can't. Concerns 1+2 delivered with independently-verified atomicity + concurrency semantics.
Closure
Phase 6 milestone #78 closes with this tracker. Sibling closures firing: #508 epic + milestone #78 itself.
Closed by Bosun 2026-07-27 10:48 CEST per ADR-0009 §6 direction (fail-atomic design folded into Cutter, not follow-up).