chore(post-cut-bookkeeping): investigate why manifest catch-up doesn't fire automatically #602

New issue

Closed

opened 2026-07-30 13:04:43 +02:00 by bosun · 1 comment

bosun commented

2026-07-30 13:04:43 +02:00

Owner

Motivation

Post-cut bookkeeping ([skip ci] commit that updates .release-toolkit-manifest.json to reflect the just-shipped version) has failed to fire on the last three empirical release cuts, each requiring a hand-update PR to reconcile the manifest with shipped reality:

Repo	Version	Hand-update PR	Cause noted
tmux-tell	v0.36.0	PR#871	Option A cut (workflow chain fast-failed; API-tag-bypass)
tmux-tell	v0.36.1	PR#876	Option A cut (same workflow chain issue)
release-toolkit	v0.34.0	PR#601	Post-cut bookkeeping never fired (v0.34.0 was normal cut path, not Option A — this is the load-bearing observation)

The tmux-tell instances at least had a clear cause (Option A bypass skipped the toolkit's own release-workflow path). The release-toolkit v0.34.0 case is different: PR#552 merged cleanly via the release-toolkit's own release pipeline (release-decide + release + install + validate all fired successfully — see workflow runs 2456-2465 on 2026-07-29 21:52-22:47 CEST). Yet the manifest never got its [skip ci] bookkeeping update.

This suggests the post-cut bookkeeping mechanism either:

Isn't wired for release-toolkit's own release path (only for adopters?)
Is wired but has a subtle failure mode (permissions? token scope? condition mismatch?)
Was never designed to update the manifest as part of post-cut (in which case the manifest catch-up is the expected steady-state and worth codifying as "run this after every cut")

Scope

Investigation, not implementation. Determine:

Design intent: what SHOULD happen to the manifest post-cut? Is it:
- (a) Auto-updated by the release workflow (rolling-PR-style follow-up commit)?
- (b) Auto-updated by a downstream [skip ci] job triggered by release: published?
- (c) Expected steady-state: manifest updates via next rolling-release-PR cycle (i.e., it catches up when the next release-prep runs)?
- (d) Something else?
Empirical state: check the workflow tree + scripts + reusable workflows for the mechanism. Which of (a)-(d) is actually wired?
Gap analysis: if design intent ≠ empirical state, what's the specific failure mode?
Fix path decision:
- If (a)/(b) intended but broken: fix the mechanism + verify on next cut
- If (c) intended: verify this class of drift is self-healing on next rolling PR (test: check whether v0.35.0's rolling PR will regenerate manifest correctly given v0.34.0 shipped state)
- If gap is by-design: document the expected hand-update pattern as steady-state + codify as chamber discipline

Verification AC

Design intent named (which of (a)-(d)) — (a): the cut workflow owns post-cut bookkeeping. Not a downstream release-published job, not next-cycle self-heal.
Empirical state named — main 1bce291 wires rt release immediately followed by rt post-cut, with a separate manifest post-condition. (a) is wired.
Gap analysis complete — root cause source- and run-proven on the v0.34.0 cut (task 19682, eff2d9d1): rt release's Cutter wrote the cut manifest into the LOCAL checkout, the old manifest-precheck read that local last_released_sha as evidence of a prior REMOTE landing, and logged idempotent skip. A false local skip — not a token gap and not a wiring omission.
Fix path decided — already landed on main as a74a4c9c, which makes the post-cut boundary remote-authoritative: only matching manifest state on remote main may skip; absent, unreadable or mismatching remote proceeds. Its mutation controls cover the local-current-run-write case. No implementation made in this investigation.

PR#601 — v0.34.0 manifest catch-up (this investigation's motivation)
tmux-tell PR#871, PR#876 — same-shape hand-updates for Option A cuts (excluded from this investigation's scope; those had a clear cause)
ADR-0009 (Go port architecture) — release-toolkit's own release-cut path is what shipped in Phase 7
Phase 8 milestone #81 (Prerelease cutting write-side) — this investigation may surface something worth folding into the pre-1.0.0 arc

Anchor

Filed 2026-07-30 by Bosun per operator direction (post-PR#601 merge). Three empirical instances suggest something is genuinely non-firing at the substrate layer; investigation-shape rather than fix-shape until the root cause is named. Priority/low because manifest catch-up is small + not user-facing; but worth resolving before v1.0.0 arc so the release process is substrate-honest for the stability commitment.

## Motivation Post-cut bookkeeping (`[skip ci]` commit that updates `.release-toolkit-manifest.json` to reflect the just-shipped version) has failed to fire on the last three empirical release cuts, each requiring a hand-update PR to reconcile the manifest with shipped reality: | Repo | Version | Hand-update PR | Cause noted | |------|---------|---------------|-------------| | tmux-tell | v0.36.0 | [PR#871](https://git.frankenbit.de/frankenbit/tmux-tell/pulls/871) | Option A cut (workflow chain fast-failed; API-tag-bypass) | | tmux-tell | v0.36.1 | [PR#876](https://git.frankenbit.de/frankenbit/tmux-tell/pulls/876) | Option A cut (same workflow chain issue) | | release-toolkit | v0.34.0 | [PR#601](https://git.frankenbit.de/frankenbit/release-toolkit/pulls/601) | Post-cut bookkeeping never fired (v0.34.0 was normal cut path, not Option A — this is the load-bearing observation) | The tmux-tell instances at least had a clear cause (Option A bypass skipped the toolkit's own release-workflow path). The **release-toolkit v0.34.0 case is different**: PR#552 merged cleanly via the release-toolkit's own release pipeline (release-decide + release + install + validate all fired successfully — see workflow runs 2456-2465 on 2026-07-29 21:52-22:47 CEST). Yet the manifest never got its `[skip ci]` bookkeeping update. This suggests the post-cut bookkeeping mechanism either: - Isn't wired for release-toolkit's own release path (only for adopters?) - Is wired but has a subtle failure mode (permissions? token scope? condition mismatch?) - Was never designed to update the manifest as part of post-cut (in which case the manifest catch-up is the expected steady-state and worth codifying as "run this after every cut") ## Scope Investigation, not implementation. Determine: 1. **Design intent**: what SHOULD happen to the manifest post-cut? Is it: - (a) Auto-updated by the release workflow (rolling-PR-style follow-up commit)? - (b) Auto-updated by a downstream `[skip ci]` job triggered by `release: published`? - (c) Expected steady-state: manifest updates via next rolling-release-PR cycle (i.e., it catches up when the next release-prep runs)? - (d) Something else? 2. **Empirical state**: check the workflow tree + scripts + reusable workflows for the mechanism. Which of (a)-(d) is actually wired? 3. **Gap analysis**: if design intent ≠ empirical state, what's the specific failure mode? 4. **Fix path decision**: - If (a)/(b) intended but broken: fix the mechanism + verify on next cut - If (c) intended: verify this class of drift is self-healing on next rolling PR (test: check whether v0.35.0's rolling PR will regenerate manifest correctly given v0.34.0 shipped state) - If gap is by-design: document the expected hand-update pattern as steady-state + codify as chamber discipline ## Verification AC - [x] Design intent named (which of (a)-(d)) — **(a)**: the cut workflow owns post-cut bookkeeping. Not a downstream release-published job, not next-cycle self-heal. - [x] Empirical state named — main `1bce291` wires `rt release` immediately followed by `rt post-cut`, with a separate manifest post-condition. **(a) is wired.** - [x] Gap analysis complete — root cause source- and run-proven on the v0.34.0 cut (task 19682, `eff2d9d1`): `rt release`'s Cutter wrote the cut manifest into the LOCAL checkout, the old manifest-precheck read that local `last_released_sha` as evidence of a prior REMOTE landing, and logged `idempotent skip`. **A false local skip — not a token gap and not a wiring omission.** - [x] Fix path decided — **already landed on main as `a74a4c9c`**, which makes the post-cut boundary remote-authoritative: only matching manifest state on remote `main` may skip; absent, unreadable or mismatching remote proceeds. Its mutation controls cover the local-current-run-write case. **No implementation made in this investigation.** ## Related - [PR#601](https://git.frankenbit.de/frankenbit/release-toolkit/pulls/601) — v0.34.0 manifest catch-up (this investigation's motivation) - [tmux-tell PR#871](https://git.frankenbit.de/frankenbit/tmux-tell/pulls/871), [PR#876](https://git.frankenbit.de/frankenbit/tmux-tell/pulls/876) — same-shape hand-updates for Option A cuts (excluded from this investigation's scope; those had a clear cause) - ADR-0009 (Go port architecture) — release-toolkit's own release-cut path is what shipped in Phase 7 - Phase 8 milestone #81 (Prerelease cutting write-side) — this investigation may surface something worth folding into the pre-1.0.0 arc ## Anchor Filed 2026-07-30 by Bosun per operator direction (post-PR#601 merge). Three empirical instances suggest something is genuinely non-firing at the substrate layer; investigation-shape rather than fix-shape until the root cause is named. Priority/low because manifest catch-up is small + not user-facing; but worth resolving before v1.0.0 arc so the release process is substrate-honest for the stability commitment.

bosun added the

priority/low

size/S

kind/chore

labels

2026-07-30 13:04:43 +02:00

sentry was assigned by bosun

2026-08-27 11:30:22 +02:00

bosun commented

2026-08-27 11:46:43 +02:00

Author

Owner

Closing — diagnosis complete, and the fix was already on main before this tracker was picked up.

All four investigative ACs ticked from @sentry's source- and run-proven diagnosis. No repo edits were made; the tracker asked for an investigation and the remedy had landed independently as a74a4c9c.

Root cause, and why it looked like a wiring gap

v0.34.0 cut   task 19682, commit eff2d9d1
  rt release's Cutter wrote the cut manifest into the LOCAL checkout
  the old manifest-precheck read that local last_released_sha
  treated it as evidence of a prior REMOTE landing
  logged "idempotent skip"
  no manifest commit, no push, no PR

🔑 A false LOCAL skip. The check read state its own run had just written, one step earlier, in the same working tree — and could not distinguish it from state a previous run had landed remotely. The two are byte-identical at the point of reading; only the remote can tell them apart.

a74a4c9c makes the boundary remote-authoritative: only matching state on remote main may skip; absent, unreadable or mismatching proceeds.

⚠️ `purser#60` is a DIFFERENT case and does not belong to this tracker

@sentry checked rather than assuming, and it is not a second post-cut non-fire:

cut task 21218   DID run, created PR#60 at 41cd0d33
manifest-check   task 21263, pinned rt v0.36.0
                 correctly FAILED 2026-08-17 — the draft release had no v0.4.0 tag
release published 2026-08-21; v0.4.0 now resolves byte-identically to d127f2ee

The red is stale historical evidence, and a current rerun pass is NOT established — the workflow is pull_request-only and Forgejo exposes no rerun endpoint. Main's 7fd9ed53 records draft/unpublished state so a missing tag is classifiable going forward.

📌 Residual, and it is operational rather than a defect in #602's wiring: gamma bookkeeping PRs whose CI is stale have no rerun path. They need a push or the web UI button. Not this tracker's scope; recorded so it is not rediscovered.

Investigation by @sentry, who explicitly did not close it. Verified and closed by @bosun.

**Closing — diagnosis complete, and the fix was already on main before this tracker was picked up.** All four investigative ACs ticked from @sentry's source- and run-proven diagnosis. **No repo edits were made**; the tracker asked for an investigation and the remedy had landed independently as `a74a4c9c`. ## Root cause, and why it looked like a wiring gap ``` v0.34.0 cut task 19682, commit eff2d9d1 rt release's Cutter wrote the cut manifest into the LOCAL checkout the old manifest-precheck read that local last_released_sha treated it as evidence of a prior REMOTE landing logged "idempotent skip" no manifest commit, no push, no PR ``` 🔑 **A false LOCAL skip.** The check read state its own run had just written, one step earlier, in the same working tree — and could not distinguish it from state a previous run had landed remotely. *The two are byte-identical at the point of reading; only the remote can tell them apart.* **`a74a4c9c` makes the boundary remote-authoritative**: only matching state on remote `main` may skip; absent, unreadable or mismatching proceeds. ## ⚠️ `purser#60` is a DIFFERENT case and does not belong to this tracker @sentry checked rather than assuming, and it is not a second post-cut non-fire: ``` cut task 21218 DID run, created PR#60 at 41cd0d33 manifest-check task 21263, pinned rt v0.36.0 correctly FAILED 2026-08-17 — the draft release had no v0.4.0 tag release published 2026-08-21; v0.4.0 now resolves byte-identically to d127f2ee ``` **The red is stale historical evidence, and a current rerun pass is NOT established** — the workflow is `pull_request`-only and Forgejo exposes no rerun endpoint. Main's `7fd9ed53` records draft/unpublished state so a missing tag is classifiable going forward. 📌 **Residual, and it is operational rather than a defect in `#602`'s wiring: gamma bookkeeping PRs whose CI is stale have no rerun path.** They need a push or the web UI button. *Not this tracker's scope; recorded so it is not rediscovered.* **Investigation by @sentry, who explicitly did not close it. Verified and closed by @bosun.**

bosun closed this issue

2026-08-27 11:46:43 +02:00