#86 - bug(release-decide): Layer 2 cut-safeguard race condition — forgejo_find_pr_by_merge_sha lags ~8s behind fresh merge - frankenbit/release-toolkit

quartermaster commented

2026-06-26 00:50:44 +02:00

Owner

Symptom

When operator merged the v0.5.0 rolling PR #80 at 2026-06-26T00:40:30, the cut workflow fired at 2026-06-26T00:40:38 (8s later). Layer 2 cut-safeguard FAILED — forgejo_find_pr_by_merge_sha query returned no match for the just-merged commit's SHA. The safeguard's protective fall-through fired: mode=update instead of mode=cut, opening PR #84 as a precaution rather than risking a false-positive cut on uncertain provenance.

Repro-confirmed: a manual probe of the same lookup ~5 minutes later succeeded and correctly identified #80 with head=release-prep/rolling. The race window is on the order of seconds.

The safeguard worked correctly

This is the first empirical falsification of the Layer 2 timing assumption (the safeguard's protective behavior is the very thing #64 deferred test coverage was about). The substrate-honest fall-through fired: when Layer 2 can't verify the merge source, default to the safer mode=update (refresh rolling PR) rather than the riskier mode=cut (mutate production state). Surveyor's d64a "cut-verification is load-bearing" framing — verified working.

What didn't work

The desired outcome (v0.5.0 cut on operator's merge) didn't happen autonomously. Recovery required manual workflow_dispatch after Forgejo's API caught up. For a substrate that aims to be "fully autonomous cuts on operator merge," needing manual intervention defeats the design intent.

Root cause hypothesis

Forgejo's PR-merge indexing has a window between:

The actual merge commit landing on main
The PR's merge_commit_sha + state=closed being reflected in the pulls?state=closed query results

During that window, forgejo_find_pr_by_merge_sha returns nothing for the just-merged SHA. The window appears to be a few seconds.

(Per feedback_filed_rootcause_is_hypothesis: this is a hypothesis until probed on the live system. The exact duration + mechanism of the lag would need investigation — possibly cache invalidation, async indexer, or similar.)

Proposed fix surface

Three candidate approaches:

Option	Shape	Tradeoff
(A) Retry with backoff in Layer 2	If first lookup returns nothing AND HEAD subject matches prep regex, sleep 5s + retry once (or N times with exponential backoff up to ~30s)	Simplest; preserves substrate-honest fall-through after retry budget exhausted
(B) Walk the recent commit's `pull` ref directly	Use git refs or a different API endpoint that resolves immediately on merge	Substrate-dependent; requires investigating what Forgejo provides synchronously
(C) Wait-mode in `release-decide.sh`	Add a `--wait-for-merge-sha SHA --timeout 30` flag that polls until found	More configurable; consumer can opt out via timeout=0

I lean (A) — minimal change, well-bounded, falls back to current behavior cleanly. The retry budget (e.g., 3 retries × 5s = 15s) is plenty for the observed ~8s lag.

Sibling deferred test coverage

#64 — Layer-2/3 cut-safeguard FAIL-path coverage. The race-condition fall-through this issue catches IS one of the empirical instances #64's tests would have caught earlier. Pair them in the v0.5.1 substrate-correctness sweep.

Refs

Surfaced: 2026-06-26 v0.5.0 cut (operator merged #80, Layer 2 fall-through, PR #84 opened as precaution, manual dispatch later succeeded)
Workflow run: actions/runs/6252 (decide+act task 13135)
Sibling: #64 (Layer-2/3 FAIL-path coverage — would have caught this empirically)
Related: #56 (manifest-vs-history guard — defensive class)
v0.5.1 substrate-correctness sweep candidate alongside #64 + #56

## Symptom When operator merged the v0.5.0 rolling PR #80 at `2026-06-26T00:40:30`, the cut workflow fired at `2026-06-26T00:40:38` (8s later). Layer 2 cut-safeguard FAILED — `forgejo_find_pr_by_merge_sha` query returned no match for the just-merged commit's SHA. The safeguard's protective fall-through fired: `mode=update` instead of `mode=cut`, opening PR #84 as a precaution rather than risking a false-positive cut on uncertain provenance. Repro-confirmed: a manual probe of the same lookup ~5 minutes later succeeded and correctly identified #80 with `head=release-prep/rolling`. The race window is on the order of seconds. ## The safeguard worked correctly This is the first **empirical falsification** of the Layer 2 timing assumption (the safeguard's protective behavior is the very thing #64 deferred test coverage was about). The substrate-honest fall-through fired: when Layer 2 can't verify the merge source, default to the safer `mode=update` (refresh rolling PR) rather than the riskier `mode=cut` (mutate production state). Surveyor's d64a "cut-verification is load-bearing" framing — verified working. ## What didn't work The desired outcome (v0.5.0 cut on operator's merge) didn't happen autonomously. Recovery required manual `workflow_dispatch` after Forgejo's API caught up. For a substrate that aims to be "fully autonomous cuts on operator merge," needing manual intervention defeats the design intent. ## Root cause hypothesis Forgejo's PR-merge indexing has a window between: 1. The actual merge commit landing on `main` 2. The PR's `merge_commit_sha` + `state=closed` being reflected in the `pulls?state=closed` query results During that window, `forgejo_find_pr_by_merge_sha` returns nothing for the just-merged SHA. The window appears to be a few seconds. (Per `feedback_filed_rootcause_is_hypothesis`: this is a hypothesis until probed on the live system. The exact duration + mechanism of the lag would need investigation — possibly cache invalidation, async indexer, or similar.) ## Proposed fix surface Three candidate approaches: | Option | Shape | Tradeoff | |---|---|---| | **(A) Retry with backoff in Layer 2** | If first lookup returns nothing AND HEAD subject matches prep regex, sleep 5s + retry once (or N times with exponential backoff up to ~30s) | Simplest; preserves substrate-honest fall-through after retry budget exhausted | | **(B) Walk the recent commit's `pull` ref directly** | Use git refs or a different API endpoint that resolves immediately on merge | Substrate-dependent; requires investigating what Forgejo provides synchronously | | **(C) Wait-mode in `release-decide.sh`** | Add a `--wait-for-merge-sha SHA --timeout 30` flag that polls until found | More configurable; consumer can opt out via timeout=0 | I lean **(A)** — minimal change, well-bounded, falls back to current behavior cleanly. The retry budget (e.g., 3 retries × 5s = 15s) is plenty for the observed ~8s lag. ## Sibling deferred test coverage [#64](https://git.frankenbit.de/frankenbit/release-toolkit/issues/64) — Layer-2/3 cut-safeguard FAIL-path coverage. The race-condition fall-through this issue catches IS one of the empirical instances #64's tests would have caught earlier. Pair them in the v0.5.1 substrate-correctness sweep. ## Refs - **Surfaced**: 2026-06-26 v0.5.0 cut (operator merged #80, Layer 2 fall-through, PR #84 opened as precaution, manual dispatch later succeeded) - **Workflow run**: actions/runs/6252 (decide+act task 13135) - **Sibling**: [#64](https://git.frankenbit.de/frankenbit/release-toolkit/issues/64) (Layer-2/3 FAIL-path coverage — would have caught this empirically) - **Related**: [#56](https://git.frankenbit.de/frankenbit/release-toolkit/issues/56) (manifest-vs-history guard — defensive class) - **v0.5.1 substrate-correctness sweep candidate** alongside #64 + #56

quartermaster referenced this issue

2026-06-26 01:05:39 +02:00

v0.5.1 sprint: token override + Layer 2 race + stale PR cleanup + branch-protection doc correction (path α dogfood) #88

quartermaster referenced this issue from a commit

2026-06-26 01:21:28 +02:00

fix+feat: v0.5.1 substrate-correction sprint (closes #86 + #87 + #64)

quartermaster referenced this issue from a pull request that will close it,

2026-06-26 01:21:59 +02:00

fix+feat: v0.5.1 substrate-correction sprint (closes #86 + #87 + #64) #89

quartermaster closed this issue

2026-06-26 01:45:07 +02:00