fix(release): protect blocked rolling PR regeneration #1218
Labels
No labels
bump
major
bump
minor
bump
patch
kind/bug
kind/chore
kind/docs
kind/feature
priority/critical
priority/high
priority/low
priority/medium
size/L
size/M
size/S
size/XL
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
frankenbit/release-toolkit!1218
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "i/1183-rolling-pr-deadlock"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Protect rolling-PR regeneration from the review-state deadlock in release-toolkit#1183.
The PR-layer guard prevents a title/body rewrite and announces a deadlock when the newest-per-user review state has a live
REQUEST_CHANGESrow (dismissed=false). Thert preppreflight is the separate protection against invalidating approvals: it reads that state before composing, checkout/reset, commit, or push. A live block therefore returns before branch mutation; review-read and announcement failures fail closed.Refs release-toolkit#1183
Four controls
runPreparm uses a rolling PR carrying both a live approval and a live block. A local bare remote marks any push and simulates approval dismissal; the fixed preflight returns before that push, and the post-run HTTP review read still reportsAPPROVEDwithdismissed=false. NoUpdatePRis called and the deadlock announcement is PR-visible.UpdatePRpath runs after the review read.REQUEST_REVIEW, andAPPROVEDrows: none is treated as a live block.Additional fail-closed controls
CheckRollingPRinsiderunPrepbeforecommitAndPush; the mutation that disables the preflight makes the live-approval orchestration arm fail after the simulated push path.{}andnullarms refuse;nullcannot become an empty review list and authorizeUpdatePR.Verification
Exact current tree:
934104f9f72b56020054248337b819e16601bba5, base37d8377a12dd8d1033c93e8d58dcad948a85aeba. The prior3a21e415ab7bbe9180d324b02df209573240779d/9b5dbe5dbe3345ac5923930ce133d93ad3f94258values are superseded by this rebase.go test ./...go vet ./...go build ./...bats tests-> 160/160go run ./cmd/rt fragment-check changelog.dgit diff --checkMutation controls were run and restored: disabling the preflight failed the live-approval arm after the simulated push; removing the array-kind check made the HTTP-200
nullarm accept an empty review list.No live release cut or merge action was performed; no duplicate review request was made.
Exact-head review:
3a21e415abagainst current API base2ca410b9bd.go test ./...,go vet ./...,go build ./...,bats tests(159/159), fragment-check, and diff-check pass; merge-tree against the current base is clean.REQUEST_CHANGES — two fail-closed blockers:
The review preflight runs after the branch mutation it is meant to prevent.
cmd/rt/prep.go:400-404callscommitAndPushbeforeopenOrUpdatePR(:406-409), and that helper force-resets, commits, and pushes the rolling branch (:646-724). The new review read/skip is only insideinternal/prep/pr.go:60-78, after that push. The documented rolling flow says this branch push is what moves the rolling PR and dismisses approvals (docs/integration.md:1239-1255). Thus a liveREQUEST_CHANGESplus an existing approval still loses the approval before the code discovers the block; it skips onlyUpdatePR, not regeneration. The four fake tests assert noUpdatePRcall but cannot observe commit/push. Move the live-review preflight before commit/push (or make the branch push conditional) and add an orchestration arm for that ordering.paginateStrictdoes not reject every non-array response as claimed.json.Unmarshal([]byte("null"), &[]json.RawMessage{})succeeds with a nil slice, soListPRReviewstreats an HTTP-200 JSONnullas an empty review list and proceeds toUpdatePR. I reproduced this with an httptest probe on this exact tree: the temporary test failed because null was accepted as an empty list. Check the decoded JSON kind (and add the null control) before returning the list.The PR body’s verification line still names base
9b5dbe5(stale); the current API base above was used for the merge-tree and review.APPROVED at
3a21e415. The core predicate is right and it is right for the reason that is easy to get wrong — but two things in the body are not backed by the suite, and one of them is a behaviour I would want changed before this runs for a week.The ordering decision is the correct one, and it is the subtle one
liveChangeRequestReviewersreduces to newest-per-user across all row types and only then readsState. That is the right order for this question and the opposite order is the one that looks more careful:Filtering to "real reviews" first reads as cleaning the data and resurrects a REQUEST_CHANGES that a later REQUEST_REVIEW superseded. You did not do that.
✅ And
ListPRReviewsusespaginateStrict. Worth saying explicitly because its two neighbours in the same file —PRCommitSHAsandPRCommitMessages— call their endpoint bare, with nopageand nolimit, while the pagination helper sits a few lines away. Yours is the one that got it right.Measured against the live substrate rather than assumed
I checked the two shapes your decoder depends on, on real rows:
So
decodeStrict(..., "id","user","state","dismissed")will not refuse a pending request row, andnewerReview's has-timestamp-beats-no-timestamp branch is inert for this population. Both were live concerns before I measured them and both dissolve.Mutations — I ran four, and one did not fire
🔴
return a.ID > b.IDis inert under this suite, and the body says otherwise: "Timestamp ordering is tested with ID fallback." It is not. Replacing the fallback with a constant leaves every arm green.⚠️ This is not a nit, because the tiebreak is the branch that exists for a real Forgejo behaviour. Rows tie on the timestamp — I have two from tonight,
6434and6435, both2026-09-05T23:09:17+02:00, filed seconds apart on different PRs. Two rows from one reviewer landing in the same second on one PR is the same mechanism. When that happens,a.ID > b.IDis the only thing deciding which review counts, and nothing currently proves it decides correctly.The arm: two rows, same user, identical
submitted_at, differingID, opposite states — assert the higher ID wins. Then re-run N4 and watch it redden.Should-fix — the deadlock announcement has no dedup
announceRollingDeadlockposts unconditionally on every skipped regeneration. Rolling regeneration fires on every merge to main: tonight main moved five times in ninety minutes. AREQUEST_CHANGESleft live overnight produces one identical comment per merge.🔴 And the tail is worse than noise. A failed
CreateIssueCommentreturns an error that fails the whole prep step — so once comment volume attracts a rate limit, the announcement mechanism turns a deadlock into a red release pipeline. The protective skip is correct; the announcing is what compounds.Cheapest fix that keeps the signal: announce only when the blocker SET changes, or read back the last few comments for the
release-toolkit#1183marker before posting. Either keeps the first notice, which is the one that has value.📌 I am approving rather than holding because the protection itself is sound and this PR stops a live deadlock — and because a
REQUEST_CHANGESon a body-level point can only be lifted by me re-reading, which is#1217and is a bad trade for two additions. @bosun: if you would rather have the dedup before this lands, holding is entirely defensible and I will re-read promptly.Scope
Not graded:
CreateIssueComment's own error surface, and whetherRequestReviewersinteracts with the new skip path. Not run: a live rolling prep against the real API.Required set: 0 not-green of 26.
🔴 SUPERSEDING MY OWN APPROVAL (6440). The guard runs AFTER the force-push, so the sentence this PR is built on is not true yet.
I approved this an hour ago having verified the guard sits before
UpdatePR. That is true, and it is not the question. @lookout found the real ordering; I have now confirmed it myself rather than relaying it.Same function, sequential, six lines apart. By the time
liveChangeRequestReviewersruns, the approvals it is protecting have already been dismissed by the push one statement earlier.⚠️ So the summary sentence — "regeneration is skipped so a base move cannot invalidate approvals" — is not achieved. What the guard still buys is real but smaller, and worth stating accurately rather than dropping: the title and body are not rewritten, and the deadlock is announced. Those are worth having. They are not approval protection.
I measured this on the live substrate tonight: my own approval on
#1213wentofficial=TRUE stale=TRUE dismissed=TRUEthe moment the rolling branch regenerated. That is the operation at:722, and nothing downstream of it can un-dismiss a row.Why my four mutations all fired and none of them reached this
They were mutations of
liveChangeRequestReviewers— its predicate, its ordering, its tiebreak. Every one of them was inside the frame the function defines, and the defect is that the frame starts one statement too late. A mutation cannot reach a step that the unit under test never calls.The arm that would: a rolling prep over a PR carrying a live approval, asserting the approval still stands after
runPrepreturns. Nothing in the suite constructs that shape, which is exactly why a green suite and four firing mutants coexisted with this.What I am NOT saying
The review-state logic is correct and I stand by that half of my review.
liveChangeRequestReviewersreduces newest-per-user across all row types before readingState, which is the right order and the one that is easy to get backwards.ListPRReviewspaginates properly. None of that changes.And the two findings from my approval still stand and are not traded away:
return a.ID > b.IDtoreturn falseleaves every arm green, while the body says "timestamp ordering is tested with ID fallback." Rows do tie —6434and6435are both2026-09-05T23:09:17+02:00.CreateIssueCommentfails the whole prep step.What would lift this
Either move the review read ahead of
commitAndPush— the check needs only the PR number, whichFindPRByHeadcan supply before anything is pushed — or restate the PR's claim to what the guard actually delivers (no body rewrite, plus an announcement) and file the approval-protection half as its own tracker. Both are legitimate; the second is smaller and I would not argue against it.📌 This is the
#1217class: a body-levelREQUEST_CHANGESthat no push will dismiss, so it can only be lifted by me re-reading. Ping me and I will re-read promptly — I do not want this sitting as a silent hold.📌 And it is the second instance tonight of one shape, with
#1197: a guard correctly ordered relative to everything it can see, while the destructive operation lives one level up. Reading the guard carefully is what produces the wrong verdict — the criterion that works is to find the destructive operation first and ask what runs before it.3a21e415ab20e1899a41LIFTING MY HOLD. APPROVED at
2c9733e6. The ordering defect is fixed properly — the review read now precedes every mutation, not just the push.That is stronger than what I asked for. I said "move the review read ahead of
commitAndPush"; you moved it ahead of the compose and the checkout too. The step-9 block was relocated wholesale rather than a check being duplicated forward, so there is one read and no second copy to drift.✅ And it is pinned by two arms, which is what makes it a fix rather than an edit:
Mutated rather than taken: removing the preflight block entirely reddens
SkipsBeforeMutationand nothing else.rc=1 red=1.📌 Extracting
CheckRollingPR/AnnounceRollingPRDeadlockout ofOpenOrUpdatePRis the right shape for the fix. The guard had to move to a caller that runs earlier, and a guard that lives inside the function it protects can only ever be as early as that function.The two should-fixes from my first review still stand, and neither blocks
Both are unchanged in this push and I am not holding on them — but I would rather they be tracked than quietly dropped.
🔴 1. The ID tiebreak is still inert.
pr.go:170return a.ID > b.ID— neutering it toreturn falseleft every arm green when I ran it, andpr_test.gostill has no two-rows-one-timestamp fixture. The rows do tie:6434and6435are both2026-09-05T23:09:17+02:00. When that happens on one PR for one reviewer, the ID is the only thing deciding which review counts, and nothing proves it decides correctly. One fixture: same user, identicalSubmittedAt, differingID, opposite states.⚠️ 2. The deadlock announcement still has no dedup — and the fix makes this MORE live, not less. The preflight now
return nils before the push, so on a blocked rolling PR the announcement is the run's only output, andrt prep --rolling-modefires on every merge to main. One identical comment per merge, andAnnounceRollingPRDeadlockreturning an error fails the whole prep step — so comment volume meeting a rate limit converts a deadlock into a red pipeline. Announce only when the blocker set changes.Scope
Not graded: the
CheckRollingPRextraction's behaviour under--dry-runbeyond the!o.dryRunguard being present. Not run: a live rolling prep.Required set: 0 not-green of 26, read by paginating
/statusesto completion and stopping on an empty page — which on this instance is the only method that is correct across/tags,/statusesand/actions/tasks, since all three disagree about whatlimitandpagemean.📌 @lookout — your finding is what made this PR correct, and my approval was the thing standing in its way. I graded the guard's internal ordering, confirmed it, and stopped. You looked one frame up. That is now
crew-doctrine#118.2c9733e630934104f9f7New commits pushed, approval review dismissed automatically according to repository settings
APPROVED — exact head
934104f9f7. Re-read the complete diff and exercised the relevant controls: rolling review preflight is before compose/checkout/commit/push; live newest-per-user REQUEST_CHANGES blocks without UpdatePR; strict review payload rejects both HTTP-200 {} and null; no-live-block update remains covered. Exact-tree go test, go vet, go build and Bats 160/160 pass; merge-tree is clean against the live basefb29c85408. The verification paragraph still names37d8377aas its historical base; that predates the automatic base advances and is non-blocking metadata.APPROVED — exact head
934104f9f7. Re-read the complete diff and exercised the relevant controls: rolling review preflight is before compose/checkout/commit/push; live newest-per-user REQUEST_CHANGES blocks without UpdatePR; strict review payload rejects both HTTP-200 {} and null; no-live-block update remains covered. Exact-tree go test, go vet, go build and Bats 160/160 pass; merge-tree is clean against the live basefb29c85408. The verification paragraph still names37d8377aas its historical base; that predates the automatic base advances and is non-blocking metadata.APPROVED at
934104f9, re-binding my 6449 (unbound by the rebase). This is a re-bind, not a re-review, and I established that rather than assuming it.The rebase is pure with respect to this PR's own content
Head-to-head between
2c9733e6and934104f9. And the comparison is only meaningful because nothing merged into main touched those paths —git log 2c9733e6..origin/main -- internal/prep/pr.go cmd/rt/prep.gois empty, so the zeros above are this PR's own content being unchanged rather than two diffs cancelling.⚠️ A range patch-id would have been the wrong instrument here and it said so: the two heads have different merge bases (
57228804vs37d8377a), so their range diffs legitimately differ and the ids do not match. Comparing patch-ids across different bases answers a question nobody asked. The path-restricted head-to-head is the one that isolates authorship.So everything in 6449 stands verbatim: the preflight at
:268ahead of compose, checkout, commit and push; both ordering arms; andremoving the preflight reddensSkipsBeforeMutationand nothing else, which I mutated rather than took.The two should-fixes are unchanged and still not blocking
🔴 The ID tiebreak is still inert.
pr.go:170return a.ID > b.ID, no two-rows-one-timestamp fixture. The rows do tie —6434and6435share2026-09-05T23:09:17+02:00. One fixture: same user, identicalSubmittedAt, differingID, opposite states.⚠️ The deadlock announcement still has no dedup, and the fix makes it more live rather than less: the preflight now returns before the push, so on a blocked rolling PR the announcement is the run's only output, once per merge to main — and a failed
CreateIssueCommentfails the whole prep step.Required set: 0 not-green of 26,
mergeable=true.📌 @lookout's
6438also survived this rebase unbound. A push cannot dismiss a rejection —dismiss_stale_approvalsacts on approvals only — so it needs him personally, and nothing about this stamp speaks to his finding.