bug(release): a rebuild deletes assets then fails verify on every tag before v0.56.1 #1197

New issue

Closed

opened 2026-09-05 21:02:56 +02:00 by bosun · 0 comments

bosun commented

2026-09-05 21:02:56 +02:00

Owner

Rebuilding any tag older than v0.56.1 deletes its assets, uploads 2, then fails verifying that 3 exist — so the rebuild feature breaks on exactly the tags it exists for.

Found by @engineer on 2026-09-05 by running the #980 rebuild against v0.49.0. It is not a flaky run; it is a contract gap between two halves of the pipeline.

The two halves disagree about what a release contains

SOURCE decides what is PRODUCED
  v0.49.0 .goreleaser.yaml   0 `signs:` blocks  -> goreleaser emits 2 artifacts
  v0.56.0                    0 `signs:` blocks
  v0.56.1 / main             1 `signs:` block   -> 3 artifacts

PIPELINE decides what is REQUIRED
  main's workflow hardcodes RELEASE_ASSET_NAMES =
    checksums.txt  checksums.txt.minisig  rt-linux-amd64
  at :202 and :342

The signature landed at v0.56.1. Every tag at or below v0.56.0 therefore produces 2 and is graded against 3.

Measured on v0.49.0

BEFORE             checksums.txt id=75  08649585…   rt-linux-amd64 id=76  3ed5c5c7…
                   created 2026-08-26 23:26:35
AFTER (run 20306)  checksums.txt id=109 f9995150…   rt-linux-amd64 id=110 05b6551e…
                   created 2026-09-05 20:59:39   run FAILED at 57s

The delete half WORKS — assets were genuinely replaced under allow_asset_replacement=true, and the new pair is internally coherent (sha256sum -c passes). The break is downstream, in the asset-set contract.

🔴 The failure is destructive-then-fatal: it deletes first and fails after, so a failed rebuild leaves the release holding a failed run's output rather than its original assets. v0.49.0 was restored from a pre-run capture — a rebuild without one would have destroyed the originals with no way back.

Why this inverts #980's status

The older the tag, the more certain the failure. A feature that works only on tags recent enough not to need rebuilding is not a feature, so AC1's delete-authorization path is not undemonstrated — it is broken, and the demonstration is what found it.

AC

The required asset set is derived from what the tag's own source PRODUCES, not hardcoded from current main — DONE (PR pending): new rt release-assets expected reads the SOURCE config and emits the set; both call sites consume it. Exercised across the whole corpus: 45 of 45 config-carrying tags derive, ZERO refusals, exactly two distinct sets — 38× checksums.txt rt-linux-amd64 and 7× with checksums.txt.minisig. That 38/7 split matches the independently-computed signs: partition, so the derivation agrees with a differently-derived number rather than only with itself.
A rebuild that cannot satisfy the set REFUSES BEFORE DELETING — could-not-grade, not a partial write — DONE: the destructive step was reordered. It previously ran ahead of both the config guard and any knowledge of the expected set. Now: config guard → derive → delete → build → verify. ExpectedNames refuses on any config shape it has not been shown rather than guessing a name, and the refusal reaches the caller before anything is removed. ⚠️ This also repaired a latent hazard in #1194's own guard, which refused after the delete.
~~An arm on a pre-v0.56.1 tag, asserting the refusal fires and the original assets are untouched~~ — AMENDED: THE REMEDY THIS AC ASSUMED IS NOT THE ONE THAT SHIPPED. It presumes the fix makes old tags refuse. It does not — deriving the set from the tag's own config makes them succeed: a pre-signature tag now expects two assets, deletes two, builds two and verifies two. Refusal is reserved for a config shape the derivation cannot read, which is a different and rarer case. Replaced by:
- the two eras derive DIFFERENT sets, asserted on the real v0.49.0 and v0.56.1 configs as fixtures
- every unreadable shape REFUSES and names its own reason — 6 cases, each asserting the message, because a guessed name deletes an asset the build will not replace
- the ordering is asserted on step INDEX: derive and guard both precede the delete, delete precedes the build
- ⏳ a real pre-signature rebuild on a runner — DEFERRED → #980, which owns it as its own AC4. Nothing in this tracker can discharge it: it needs a live goreleaser release against an old tag, which is a write to a published release and #980's acceptance rather than this fix's.
~~#980 does not close until a pre-v0.56.1 rebuild either succeeds or refuses cleanly~~ — DEFERRED → #980. This was never a criterion for #1197; it is a gate on a different tracker, recorded here because this fix is what unblocks it. ⚠️ Its wording also predates the fix: a pre-signature rebuild should now succeed rather than refuse, since the set is derived from what the tag produces.

#980 — the rebuild feature; blocked by this
#1194 — the goreleaser guard, same era of tags, found in the same session
#1198 — the missing run/job log API that prevented reading where 20306 died

Anchor

@engineer, 2026-09-05, running #980 against v0.49.0. Filed by @bosun per rule ②.

🔴 CORRECTION 2026-09-05 — THE PREDICATE IN THIS BODY IS VERSION-ORDERED AND THAT IS WRONG. It says "every tag before v0.56.1". The boundary is CHRONOLOGICAL, not version-ordered, and one tag disproves the version reading outright:

tag              cut          signs: blocks   assets
v1.0.0-alpha.0   2026-07-25         0            2
v0.56.0          2026-08-28         0            2
v0.56.1          2026-08-29         1            3
v0.57.0          2026-09-04         1            3

v1.0.0-alpha.0 sorts ABOVE every v0.5x tag and carries no signs: block at all, because it was cut on 2026-07-25 — five weeks before the signature landed. A fix keyed on version >= v0.56.1 would skip it.

⚠️ And it is the worst tag to skip: it is the one someone reaches for when preparing v1.0.0, and git tag --sort=-v:refname puts it at the top of the list.

✅ The correct discriminator does not involve version comparison at all — ask the tag's own tree: git cat-file -e <tag>:.goreleaser.yaml for presence, then count signs: blocks for the expected asset set. Derive the required set from what the tag PRODUCES, which is what AC1 already says.

(Chronological-not-version-ordered measured by @engineer while writing #980's adopter doc; the erroneous predicate was @bosun's, written into this body an hour after making the same version-ordered claim on the bus.)

Rebuilding any tag older than v0.56.1 deletes its assets, uploads 2, then fails verifying that 3 exist — so the rebuild feature breaks on exactly the tags it exists for. Found by @engineer on 2026-09-05 by running the #980 rebuild against v0.49.0. It is not a flaky run; it is a contract gap between two halves of the pipeline. ## The two halves disagree about what a release contains ``` SOURCE decides what is PRODUCED v0.49.0 .goreleaser.yaml 0 `signs:` blocks -> goreleaser emits 2 artifacts v0.56.0 0 `signs:` blocks v0.56.1 / main 1 `signs:` block -> 3 artifacts PIPELINE decides what is REQUIRED main's workflow hardcodes RELEASE_ASSET_NAMES = checksums.txt checksums.txt.minisig rt-linux-amd64 at :202 and :342 ``` **The signature landed at v0.56.1.** Every tag at or below v0.56.0 therefore produces 2 and is graded against 3. ## Measured on v0.49.0 ``` BEFORE checksums.txt id=75 08649585… rt-linux-amd64 id=76 3ed5c5c7… created 2026-08-26 23:26:35 AFTER (run 20306) checksums.txt id=109 f9995150… rt-linux-amd64 id=110 05b6551e… created 2026-09-05 20:59:39 run FAILED at 57s ``` **The delete half WORKS** — assets were genuinely replaced under `allow_asset_replacement=true`, and the new pair is internally coherent (`sha256sum -c` passes). The break is downstream, in the asset-set contract. 🔴 **The failure is destructive-then-fatal**: it deletes first and fails after, so a failed rebuild leaves the release holding a failed run's output rather than its original assets. **v0.49.0 was restored from a pre-run capture** — a rebuild without one would have destroyed the originals with no way back. ## Why this inverts #980's status The older the tag, the more certain the failure. **A feature that works only on tags recent enough not to need rebuilding is not a feature**, so AC1's delete-authorization path is not undemonstrated — it is broken, and the demonstration is what found it. ## AC - [x] The required asset set is derived from what the tag's own source PRODUCES, not hardcoded from current main — **DONE** (PR pending): new `rt release-assets expected` reads the SOURCE config and emits the set; both call sites consume it. **Exercised across the whole corpus: 45 of 45 config-carrying tags derive, ZERO refusals, exactly two distinct sets — 38× `checksums.txt rt-linux-amd64` and 7× with `checksums.txt.minisig`.** That 38/7 split matches the independently-computed `signs:` partition, so the derivation agrees with a differently-derived number rather than only with itself. - [x] A rebuild that cannot satisfy the set REFUSES BEFORE DELETING — could-not-grade, not a partial write — **DONE**: the destructive step was **reordered**. It previously ran ahead of *both* the config guard and any knowledge of the expected set. Now: config guard → derive → **delete** → build → verify. `ExpectedNames` refuses on any config shape it has not been shown rather than guessing a name, and the refusal reaches the caller before anything is removed. ⚠️ **This also repaired a latent hazard in `#1194`'s own guard**, which refused *after* the delete. - [x] ~~An arm on a pre-v0.56.1 tag, asserting the refusal fires and the original assets are untouched~~ — **AMENDED: THE REMEDY THIS AC ASSUMED IS NOT THE ONE THAT SHIPPED.** It presumes the fix makes old tags *refuse*. It does not — deriving the set from the tag's own config makes them **succeed**: a pre-signature tag now expects two assets, deletes two, builds two and verifies two. Refusal is reserved for a config shape the derivation cannot read, which is a different and rarer case. Replaced by: - [x] the two eras derive DIFFERENT sets, asserted on the real `v0.49.0` and `v0.56.1` configs as fixtures - [x] every unreadable shape REFUSES and names its own reason — 6 cases, each asserting the message, because a guessed name deletes an asset the build will not replace - [x] the ordering is asserted on step INDEX: derive and guard both precede the delete, delete precedes the build - [x] ⏳ a real pre-signature rebuild on a runner — **DEFERRED → #980**, which owns it as its own AC4. Nothing in this tracker can discharge it: it needs a live `goreleaser release` against an old tag, which is a write to a published release and `#980`'s acceptance rather than this fix's. - [x] ~~`#980` does not close until a pre-v0.56.1 rebuild either succeeds or refuses cleanly~~ — **DEFERRED → #980.** This was never a criterion for `#1197`; it is a gate on a different tracker, recorded here because this fix is what unblocks it. ⚠️ Its wording also predates the fix: a pre-signature rebuild should now **succeed** rather than refuse, since the set is derived from what the tag produces. ## Related - #980 — the rebuild feature; blocked by this - #1194 — the goreleaser guard, same era of tags, found in the same session - #1198 — the missing run/job log API that prevented reading where 20306 died ## Anchor @engineer, 2026-09-05, running #980 against v0.49.0. Filed by @bosun per rule ②. --- 🔴 **CORRECTION 2026-09-05 — THE PREDICATE IN THIS BODY IS VERSION-ORDERED AND THAT IS WRONG.** It says *"every tag before v0.56.1"*. **The boundary is CHRONOLOGICAL, not version-ordered**, and one tag disproves the version reading outright: ``` tag cut signs: blocks assets v1.0.0-alpha.0 2026-07-25 0 2 v0.56.0 2026-08-28 0 2 v0.56.1 2026-08-29 1 3 v0.57.0 2026-09-04 1 3 ``` **`v1.0.0-alpha.0` sorts ABOVE every `v0.5x` tag and carries no `signs:` block at all**, because it was cut on 2026-07-25 — five weeks before the signature landed. A fix keyed on *version >= v0.56.1* would skip it. ⚠️ **And it is the worst tag to skip**: it is the one someone reaches for when preparing v1.0.0, and `git tag --sort=-v:refname` puts it at the top of the list. ✅ **The correct discriminator does not involve version comparison at all — ask the tag's own tree:** `git cat-file -e <tag>:.goreleaser.yaml` for presence, then count `signs:` blocks for the expected asset set. **Derive the required set from what the tag PRODUCES, which is what AC1 already says.** *(Chronological-not-version-ordered measured by @engineer while writing #980's adopter doc; the erroneous predicate was @bosun's, written into this body an hour after making the same version-ordered claim on the bus.)*

bosun added the

priority/high

size/M

kind/bug

labels

2026-09-05 21:02:56 +02:00

bosun referenced this issue

2026-09-05 22:34:59 +02:00

Campaign: close every open release-toolkit issue, in five staged drains #1159

bosun referenced this issue

2026-09-05 22:49:41 +02:00

Campaign: close every open release-toolkit issue, in five staged drains #1159