#1444 - bug(release): v0.62.4's post-cut manifest bookkeeping never landed — main's release gate is blocked - frankenbit/release-toolkit

bosun commented

2026-09-07 20:11:27 +02:00

Owner

Every push to main has failed the release gate since 19:02 because v0.62.4's post-cut bookkeeping never landed, and the repo now reads as a prepared-but-uncut release that has in fact shipped.

Measured — 2026-09-07 20:0x, against `origin/main` `3c8687c2`

VERSION                    0.62.4
tag v0.62.4                exists, ancestor of HEAD
release v0.62.4            published 18:54:29, assets=3
manifest last_released_*   0.62.3 / v0.62.3 / d4917b72     <- STALE

rt decide                  mode=blocked  reason=pending_cut
                           pending_tag_exists=true  pending_release_published=true
rt prepared-uncut-check    rc=1  "MANIFEST DRIFT, not an uncut release"

Consequence: release / decide + act (push) is red on 45312fa6, 4aade268, 0c7f4d4e, 3c8687c2 — four consecutive main pushes. No cut can be decided while this stands.

Likely cause, stated as a hypothesis

rt post-cut direct-pushes the manifest to main under path α. Three release-cut runs failed at 18:53:35, 18:54:47 and 18:57:49 — inside a window where four unrelated merges landed on main (18:51:47, 18:53:53, 18:57:28). A lost push race fits the timing. This forge exposes no job log through the API, so the cause is not established — only the residue is.

AC

The manifest records v0.62.4 and rt decide leaves blocked
prepared-uncut-check returns rc=0 on main
Whether post-cut's push can lose a race, and what it does when it does, is ANSWERED -- PushAuthed is a bare git push with no retry and no rebase (internal/prep/git.go:387), the job commits on top of its own checkout e45075a1, and merge-base --is-ancestor e45075a1 596ce1e4 is TRUE, so the push could not fast-forward. Established from code and graph; no log was readable. Original text: is answered — or this tracker says it could not be, naming what was tried

Anchor

@bosun, diagnosing the CI red the operator reported. Second instance: #1356 (v0.61.1, same residue, 2026-09-06).

Every push to main has failed the release gate since 19:02 because v0.62.4's post-cut bookkeeping never landed, and the repo now reads as a prepared-but-uncut release that has in fact shipped. ## Measured — 2026-09-07 20:0x, against `origin/main` `3c8687c2` ``` VERSION 0.62.4 tag v0.62.4 exists, ancestor of HEAD release v0.62.4 published 18:54:29, assets=3 manifest last_released_* 0.62.3 / v0.62.3 / d4917b72 <- STALE rt decide mode=blocked reason=pending_cut pending_tag_exists=true pending_release_published=true rt prepared-uncut-check rc=1 "MANIFEST DRIFT, not an uncut release" ``` Consequence: `release / decide + act (push)` is red on `45312fa6`, `4aade268`, `0c7f4d4e`, `3c8687c2` — four consecutive main pushes. No cut can be decided while this stands. ## Likely cause, stated as a hypothesis `rt post-cut` direct-pushes the manifest to main under path α. Three `release-cut` runs failed at 18:53:35, 18:54:47 and 18:57:49 — inside a window where four unrelated merges landed on main (18:51:47, 18:53:53, 18:57:28). A lost push race fits the timing. **This forge exposes no job log through the API, so the cause is not established** — only the residue is. ## AC - [x] The manifest records v0.62.4 and `rt decide` leaves `blocked` - [x] `prepared-uncut-check` returns rc=0 on main - [x] Whether post-cut's push can lose a race, and what it does when it does, is ANSWERED -- `PushAuthed` is a bare `git push` with no retry and no rebase (`internal/prep/git.go:387`), the job commits on top of its own checkout `e45075a1`, and `merge-base --is-ancestor e45075a1 596ce1e4` is TRUE, so the push could not fast-forward. Established from code and graph; no log was readable. Original text: is answered — or this tracker says it could not be, naming what was tried ## Anchor @bosun, diagnosing the CI red the operator reported. Second instance: `#1356` (v0.61.1, same residue, 2026-09-06).

bosun added the

priority/high

size/S

kind/bug

labels

2026-09-07 20:11:27 +02:00

bosun referenced this issue from a commit

2026-09-07 20:14:11 +02:00

fix(release-state): record v0.62.4 in the manifest

~~bosun referenced this issue 2026-09-07 20:14:29 +02:00~~

fix(release-state): record v0.62.4 in the manifest #1446

bosun referenced this issue from a commit

2026-09-07 20:24:20 +02:00

chore: re-trigger CI — the PR opened at 20:14:29 created ZERO workflow runs

bosun referenced this issue

2026-09-07 20:26:09 +02:00

fix(release-state): record v0.62.4 in the manifest #1446

bosun referenced this issue from a commit

2026-09-07 20:30:12 +02:00

fix(changelog): split the 43-word sentence fragment-check refused

bosun commented

2026-09-07 20:36:49 +02:00

Author

Owner

AC3 answered — from the CODE and the commit graph, because this forge exposes no job log

The hypothesis on the body said "a lost push race fits the timing" and called the cause unestablished. It is now established well enough to act on, without a log.

① `post-cut`'s push has no retry and no rebase

// internal/prep/git.go:387
func PushAuthed(ctx context.Context, dir, token string, pushArgs ...string) error {
    return withTokenConfig(ctx, dir, token, func(extraEnv []string) error {
        _, err := runGit(ctx, dir, extraEnv, append([]string{"push"}, pushArgs...)...)
        return err
    })
}

A bare git push. post_cut.go:176 calls it with origin <baseBranch>; a non-fast-forward is returned as an error and runPostCut aborts before verifyPostCutManifest.

It does fetch — twice — and neither fetch rebases. postCutPrecheck (:344) fetches to READ the remote manifest; verifyPostCutManifest (:595) fetches to CHECK the push landed. There is no fetch-then-rebase between the commit and the push.

② The commit graph makes the rejection unavoidable, not merely likely

The job commits the manifest on top of its own checkout, which is the triggering commit e45075a1.

18:48:38Z  e45075a1  chore(release): prepare v0.62.4     <- the cut's checkout
18:53:35   release-cut run 25956 starts
18:53:51   596ce1e4 lands on main                        <- e45075a1 IS its ancestor
18:54:29   release v0.62.4 published, assets=3           <- tag + publish PRECEDE the push
18:54:47   release-cut run 25963 starts
18:57:25   9c760691 lands on main
18:57:49   release-cut run 25973 starts

git merge-base --is-ancestor e45075a1 596ce1e4 → true. Main moved past the checkout before any of the three runs could reach the push, so a bare push of a commit parented on e45075a1 could not fast-forward. No log is needed to know that.

③ The residue is exactly what that failure leaves

Tag and release are created by rt release, which runs BEFORE the bookkeeping. So the observable state — tag present, release published with assets, manifest untouched — is the signature of an abort at the push and at no other step.

And it predicts the second instance: v0.61.1 has the same signature (tag on the prepare commit, no pin action image commit, no manifest write). Two for two.

What is NOT established

Nobody read a ! [rejected] line. If the push failed for another reason the remedy is unchanged, but the diagnosis would be. Recorded as an inference from code plus graph, not as a log reading.

The gap this opens, which is bigger than the fix

There is no verb that completes an interrupted cut. rt recover-pending-cut moves the section back under [Unreleased] and is destructive once the release has published — prepared-uncut-check says so in its own refusal. So the only path is a hand-written manifest commit, twice now.

## AC3 answered — from the CODE and the commit graph, because this forge exposes no job log The hypothesis on the body said "a lost push race fits the timing" and called the cause unestablished. It is now established well enough to act on, without a log. ### ① `post-cut`'s push has no retry and no rebase ```go // internal/prep/git.go:387 func PushAuthed(ctx context.Context, dir, token string, pushArgs ...string) error { return withTokenConfig(ctx, dir, token, func(extraEnv []string) error { _, err := runGit(ctx, dir, extraEnv, append([]string{"push"}, pushArgs...)...) return err }) } ``` A bare `git push`. `post_cut.go:176` calls it with `origin <baseBranch>`; a non-fast-forward is returned as an error and `runPostCut` aborts before `verifyPostCutManifest`. **It does fetch — twice — and neither fetch rebases.** `postCutPrecheck` (`:344`) fetches to READ the remote manifest; `verifyPostCutManifest` (`:595`) fetches to CHECK the push landed. There is no fetch-then-rebase between the commit and the push. ### ② The commit graph makes the rejection unavoidable, not merely likely The job commits the manifest **on top of its own checkout**, which is the triggering commit `e45075a1`. ``` 18:48:38Z e45075a1 chore(release): prepare v0.62.4 <- the cut's checkout 18:53:35 release-cut run 25956 starts 18:53:51 596ce1e4 lands on main <- e45075a1 IS its ancestor 18:54:29 release v0.62.4 published, assets=3 <- tag + publish PRECEDE the push 18:54:47 release-cut run 25963 starts 18:57:25 9c760691 lands on main 18:57:49 release-cut run 25973 starts ``` **`git merge-base --is-ancestor e45075a1 596ce1e4` → true.** Main moved past the checkout before any of the three runs could reach the push, so a bare push of a commit parented on `e45075a1` **could not fast-forward.** No log is needed to know that. ### ③ The residue is exactly what that failure leaves Tag and release are created by `rt release`, which runs BEFORE the bookkeeping. So the observable state — tag present, release published with assets, manifest untouched — is the signature of an abort at the push and at no other step. **And it predicts the second instance:** `v0.61.1` has the same signature (tag on the prepare commit, no `pin action image` commit, no manifest write). Two for two. ### What is NOT established Nobody read a `! [rejected]` line. If the push failed for another reason the remedy is unchanged, but the diagnosis would be. Recorded as an inference from code plus graph, not as a log reading. ## The gap this opens, which is bigger than the fix **There is no verb that completes an interrupted cut.** `rt recover-pending-cut` moves the section back under `[Unreleased]` and is destructive once the release has published — `prepared-uncut-check` says so in its own refusal. So the only path is a hand-written manifest commit, twice now.

bosun referenced this issue

2026-09-07 20:37:30 +02:00

bug(post-cut): the manifest push cannot survive a merge landing during the cut, and nothing completes an interrupted cut #1447

bosun referenced this issue from a commit

2026-09-07 20:42:21 +02:00

fix(release-state): record v0.62.4 in the manifest

bosun referenced this issue from a commit

2026-09-07 20:42:21 +02:00

chore: re-trigger CI — the PR opened at 20:14:29 created ZERO workflow runs

bosun referenced this issue from a commit

2026-09-07 20:42:21 +02:00

fix(changelog): split the 43-word sentence fragment-check refused

bosun commented

2026-09-07 20:42:47 +02:00

Author

Owner

Closing. #1446 merged at 70911562. All three ACs re-derived on merged main, not on the branch.

$ git log --oneline -1
70911562 fix(changelog): split the 43-word sentence fragment-check refused

rt decide                mode=update   last_released_version=0.62.4    (was: blocked / pending_cut)
rt prepared-uncut-check  rc=0 — PASS                                    (was: rc=1, MANIFEST DRIFT)
rt manifest-check        rc=0 — all checks passed
manifest                 {v: 0.62.4, tag: v0.62.4, sha: e45075a1}

AC3 was answered before the merge, from the code and the commit graph — PushAuthed is a bare git push with no retry and no rebase, the job commits on top of its own checkout e45075a1, and merge-base --is-ancestor e45075a1 596ce1e4 is TRUE, so the push could not fast-forward. Nobody read a ! [rejected] line and the tracker says so.

The repair is #1447, with @carpenter — bounded retry with rebase, plus the arm that lands a commit on the base branch between the checkout and the push and watches the unfixed code fail. Without that arm a retry loop cannot be shown to work.

📌 What this incident cost, for whoever reads #1447: four consecutive main pushes red, no cut possible for 78 minutes, and a hand-written manifest commit — the second in two days.

Closing. `#1446` merged at `70911562`. **All three ACs re-derived on merged `main`, not on the branch.** ``` $ git log --oneline -1 70911562 fix(changelog): split the 43-word sentence fragment-check refused rt decide mode=update last_released_version=0.62.4 (was: blocked / pending_cut) rt prepared-uncut-check rc=0 — PASS (was: rc=1, MANIFEST DRIFT) rt manifest-check rc=0 — all checks passed manifest {v: 0.62.4, tag: v0.62.4, sha: e45075a1} ``` AC3 was answered before the merge, from the code and the commit graph — `PushAuthed` is a bare `git push` with no retry and no rebase, the job commits on top of its own checkout `e45075a1`, and `merge-base --is-ancestor e45075a1 596ce1e4` is TRUE, so the push could not fast-forward. **Nobody read a `! [rejected]` line and the tracker says so.** **The repair is `#1447`**, with @carpenter — bounded retry with rebase, plus the arm that lands a commit on the base branch between the checkout and the push and watches the unfixed code fail. Without that arm a retry loop cannot be shown to work. 📌 **What this incident cost, for whoever reads `#1447`:** four consecutive main pushes red, no cut possible for 78 minutes, and a hand-written manifest commit — the second in two days.

bosun closed this issue

2026-09-07 20:42:47 +02:00