feat: idempotent cut pipeline — pre-check existing release + manifest state before re-doing work #128

New issue

Closed

opened 2026-06-26 18:47:28 +02:00 by quartermaster · 0 comments

quartermaster commented

2026-06-26 18:47:28 +02:00

Owner

Why

Surveyor df03 + operator question 2026-06-26 surfaced this jointly:

v0.10.0 cut: task 13307 substantively completed the cut at 18:33 (manifest direct-pushed, release published) but got cancelled mid-flight when its own manifest push triggered a new push:main event. Operator manually re-ran the cancelled task at 18:41 (task 13310) → release-decide correctly saw cut conditions → invoked draft-release.sh → release-create API returned 409 conflict (v0.10.0 already exists) → exit 1.
Same shape would fire on any workflow re-trigger AFTER a cut completed: tag already exists, manifest already at post-cut state, but the pipeline tries to re-do the work.

The cut work itself is irreversible by design (you can't un-publish a release without manual operator action). But the pipeline could be idempotent at the pre-check layer: see existing state, recognize "this cut already ran," exit clean.

Current idempotency state

Step	Idempotent?	Failure mode on re-run
`release-decide.sh`	✓	Reads manifest + git log; same input → same mode output
`release-prep.sh --rolling-mode`	✓-ish	Force-pushes rolling branch; safe to re-run
`draft-release.sh` (release-create)	✗	Forgejo API returns 409 conflict on existing tag/release
Manifest direct-push (path-α)	✗	`git push HEAD:main` rejected if already pushed
Manifest-via-PR (path-γ)	✗-ish	Force-pushed manifest branch can collide
Stale rolling PR cleanup (#87)	✓	Skips if nothing to clean

Proposed fixes

`draft-release.sh`: pre-check existing release

Before calling release-create, GET /repos/{owner}/{repo}/releases/tags/{tag}:

If 404 (not present): create release as today
If 200 + matches intended state (same draft value, same target_commitish, same body — or a documented subset thereof): log "release already exists at expected state — idempotent skip" + exit 0
If 200 + differs: log diagnostic + exit non-zero (substrate-state mismatch; operator must reconcile manually)

Manifest commit step (path-α + path-γ)

Before committing, read last_released_sha from the manifest on disk vs the cut's $HEAD_SHA:

If last_released_sha == $HEAD_SHA: manifest is already at post-cut state → skip the commit + push entirely
Else: proceed with the commit + push (this PR's substantive work for the cut)

Tag creation (implicit via release publish)

Forgejo creates the tag when the release is published. The release-pre-check above naturally covers this — if the release exists, the tag exists.

Composition with the multi-stage gates

The idempotency layer respects ADR-0003 + path-α: it just adds "if work is already done, recognize that + skip cleanly." No new gates, no behavior change on the first-run path.

Test seam

tests/draft-release.bats + a new tests/manifest-update.bats (or extend existing). Test cases:

First-run create: works as today (regression guard)
Re-run with release already present + matching state: exits 0 with idempotent-skip log
Re-run with release present + mismatching state: fails loud with diagnostic
Manifest already at post-cut state: skip commit + push, exit clean

What this PR does NOT do

Does NOT change first-run cut behavior (regression: ensure existing tests pass)
Does NOT make the cut UNDOABLE (idempotency is "if re-run, recognize already-done" — not "if mistake, undo")
Does NOT cover the cancel-mid-flight race that PRODUCED the v0.10.0 artifact (that's a Forgejo Actions concurrency question; idempotency makes the re-run after such a cancel safe)

Sequencing

Substantive but bounded. Estimated ~50 lines of bash + ~50 lines of bats. Engageable as a v0.10.x sprint, or bundled with #122 (test seam for read_rolling_pr_bump_label) since both are operator-paced post-v1.0-readiness work.

Refs

Empirical artifact: v0.10.0 cut run 253 (task 13310) failure 2026-06-26
Surveyor df03: reconciliation that surfaced the artifact + classified it as re-run-after-completion
Operator engagement 2026-06-26: "are the pipelines idempotent? Would it be possible to become them idempotent, if not already so?"
Composition refs: ADR-0003 (Gate-3 framework — the idempotency layer respects this); ADR-0007 (path-α direct-push, which gets its own idempotency pre-check at the manifest layer)

## Why Surveyor df03 + operator question 2026-06-26 surfaced this jointly: - **v0.10.0 cut**: task 13307 substantively completed the cut at 18:33 (manifest direct-pushed, release published) but got cancelled mid-flight when its own manifest push triggered a new push:main event. Operator manually re-ran the cancelled task at 18:41 (task 13310) → release-decide correctly saw cut conditions → invoked draft-release.sh → release-create API returned 409 conflict (v0.10.0 already exists) → exit 1. - Same shape would fire on any workflow re-trigger AFTER a cut completed: tag already exists, manifest already at post-cut state, but the pipeline tries to re-do the work. The cut work itself is irreversible by design (you can't un-publish a release without manual operator action). But the pipeline could be **idempotent at the pre-check layer**: see existing state, recognize "this cut already ran," exit clean. ## Current idempotency state | Step | Idempotent? | Failure mode on re-run | |---|---|---| | `release-decide.sh` | ✓ | Reads manifest + git log; same input → same mode output | | `release-prep.sh --rolling-mode` | ✓-ish | Force-pushes rolling branch; safe to re-run | | `draft-release.sh` (release-create) | ✗ | Forgejo API returns 409 conflict on existing tag/release | | Manifest direct-push (path-α) | ✗ | `git push HEAD:main` rejected if already pushed | | Manifest-via-PR (path-γ) | ✗-ish | Force-pushed manifest branch can collide | | Stale rolling PR cleanup (#87) | ✓ | Skips if nothing to clean | ## Proposed fixes ### `draft-release.sh`: pre-check existing release Before calling release-create, GET `/repos/{owner}/{repo}/releases/tags/{tag}`: - If 404 (not present): create release as today - If 200 + matches intended state (same `draft` value, same `target_commitish`, same `body` — or a documented subset thereof): log "release already exists at expected state — idempotent skip" + exit 0 - If 200 + differs: log diagnostic + exit non-zero (substrate-state mismatch; operator must reconcile manually) ### Manifest commit step (path-α + path-γ) Before committing, read `last_released_sha` from the manifest on disk vs the cut's `$HEAD_SHA`: - If `last_released_sha == $HEAD_SHA`: manifest is already at post-cut state → skip the commit + push entirely - Else: proceed with the commit + push (this PR's substantive work for the cut) ### Tag creation (implicit via release publish) Forgejo creates the tag when the release is published. The release-pre-check above naturally covers this — if the release exists, the tag exists. ## Composition with the multi-stage gates The idempotency layer respects ADR-0003 + path-α: it just adds "if work is already done, recognize that + skip cleanly." No new gates, no behavior change on the first-run path. ## Test seam `tests/draft-release.bats` + a new `tests/manifest-update.bats` (or extend existing). Test cases: 1. First-run create: works as today (regression guard) 2. Re-run with release already present + matching state: exits 0 with idempotent-skip log 3. Re-run with release present + mismatching state: fails loud with diagnostic 4. Manifest already at post-cut state: skip commit + push, exit clean ## What this PR does NOT do - Does NOT change first-run cut behavior (regression: ensure existing tests pass) - Does NOT make the cut UNDOABLE (idempotency is "if re-run, recognize already-done" — not "if mistake, undo") - Does NOT cover the cancel-mid-flight race that PRODUCED the v0.10.0 artifact (that's a Forgejo Actions concurrency question; idempotency makes the re-run after such a cancel safe) ## Sequencing Substantive but bounded. Estimated ~50 lines of bash + ~50 lines of bats. Engageable as a v0.10.x sprint, or bundled with #122 (test seam for read_rolling_pr_bump_label) since both are operator-paced post-v1.0-readiness work. ## Refs - **Empirical artifact**: v0.10.0 cut run 253 (task 13310) failure 2026-06-26 - **Surveyor df03**: reconciliation that surfaced the artifact + classified it as re-run-after-completion - **Operator engagement 2026-06-26**: *"are the pipelines idempotent? Would it be possible to become them idempotent, if not already so?"* - **Composition refs**: ADR-0003 (Gate-3 framework — the idempotency layer respects this); ADR-0007 (path-α direct-push, which gets its own idempotency pre-check at the manifest layer)

quartermaster referenced this issue from a commit

2026-06-26 22:05:14 +02:00

feat: idempotent cut pipeline — pre-check existing release + manifest state (closes #128)

quartermaster referenced this issue from a pull request that will close it,

2026-06-26 22:05:36 +02:00

feat: idempotent cut pipeline — pre-check existing release + manifest state (closes #128) #131

quartermaster closed this issue

2026-06-26 22:11:50 +02:00

quartermaster referenced this issue from a commit

2026-06-26 22:11:51 +02:00

chore(workflows): self-bootstrap release.yml @v0.10.1-rc.1 (idempotent cut pipeline dogfood)