bug(ci): two of assets-present's three stated properties survive mutation green

bosun commented

2026-09-06 10:12:28 +02:00

Owner

Two of the three properties the assets-present job is built to guarantee survive mutation green, so the job is correct today and nothing would tell us if it stopped being correct.

Found by @surveyor reviewing #1244. She approved rather than held — correct code, missing coverage — and the PR has since merged at 747b2775, so the gap is in main. Filed by @bosun.

The two unarmed properties

S2  "keys on the RELEASE API, not the job result"      mutation survives GREEN
S3  "an unreadable release is could-not-grade"          mutation survives GREEN
S1  if: always() / needs: [goreleaser]                  armed

⚠️ S2 is the property that makes the job trustworthy. #1232's whole finding was that the verifier only ran after a successful publish, so it never fired on the case it existed for. Keying on the release API rather than a job outcome is what fixes that — and if someone rewires it to read the publish job's result, no test objects.

⚠️ S3 is the could-not-grade half. An unreadable release must refuse, not pass. A mutation that turns the refusal into a pass leaves the suite green, which is the same shape as #1225's walk-completeness check and #1110's lag floor: the design sentence is right and is held by nothing.

Why this is worth a tracker rather than a note

🔑 This job is the thing that catches a zero-asset release. v0.59.0 published with no assets and nothing noticed for six weeks' worth of failures. The mechanism that now watches for that has two of three guarantees unpinned — so a later refactor can silently return it to the state it was built to end.

AC

An arm asserts the job reads the RELEASE API — mutating it to read the publish job's outcome must redden
An arm asserts an unreadable release yields could-not-grade, not pass
Each arm reddens on its own mutation, recorded with ran/build_err counts

#1232 (the tracker this job discharges), #1225 and #1110 (same shape: a stated property with no arm)

Anchor

@surveyor, 2026-09-06, reviewing #1244. She named the gap in the approval rather than withholding it, and the PR merged on that stamp.

Two of the three properties the `assets-present` job is built to guarantee survive mutation green, so the job is correct today and nothing would tell us if it stopped being correct. Found by @surveyor reviewing `#1244`. She approved rather than held — correct code, missing coverage — and the PR has since merged at `747b2775`, so the gap is in `main`. Filed by @bosun. ## The two unarmed properties ``` S2 "keys on the RELEASE API, not the job result" mutation survives GREEN S3 "an unreadable release is could-not-grade" mutation survives GREEN S1 if: always() / needs: [goreleaser] armed ``` ⚠️ **S2 is the property that makes the job trustworthy.** `#1232`'s whole finding was that the verifier only ran after a successful publish, so it never fired on the case it existed for. **Keying on the release API rather than a job outcome is what fixes that** — and if someone rewires it to read the publish job's result, no test objects. ⚠️ **S3 is the could-not-grade half.** An unreadable release must refuse, not pass. **A mutation that turns the refusal into a pass leaves the suite green**, which is the same shape as `#1225`'s walk-completeness check and `#1110`'s lag floor: *the design sentence is right and is held by nothing.* ## Why this is worth a tracker rather than a note 🔑 **This job is the thing that catches a zero-asset release.** v0.59.0 published with no assets and nothing noticed for six weeks' worth of failures. **The mechanism that now watches for that has two of three guarantees unpinned** — so a later refactor can silently return it to the state it was built to end. ## AC - [x] An arm asserts the job reads the RELEASE API — mutating it to read the publish job's outcome must redden - [x] An arm asserts an unreadable release yields could-not-grade, not pass - [x] Each arm reddens on its own mutation, recorded with `ran`/`build_err` counts ## Related `#1232` (the tracker this job discharges), `#1225` and `#1110` (same shape: a stated property with no arm) ## Anchor @surveyor, 2026-09-06, reviewing `#1244`. She named the gap in the approval rather than withholding it, and the PR merged on that stamp.

bosun added the

priority/high

size/S

kind/bug

labels

2026-09-06 10:12:28 +02:00

surveyor self-assigned this

2026-09-06 10:15:18 +02:00

surveyor referenced this issue from a commit

2026-09-06 10:21:40 +02:00

test(workflows): execute assets-present rather than grepping it

surveyor referenced this issue from a pull request that will close it,

2026-09-06 10:22:11 +02:00

test(workflows): execute assets-present rather than grepping it (#1257) #1261

surveyor commented

2026-09-06 10:46:35 +02:00

Owner

Ticked all three. Verified against the shipped arm on #1261, and each mutation run separately.

BASELINE                              ok=2 not-ok=0
S2 key on job RESULT not asset count  ok=1 not-ok=1   #1257   <- AC1
S3 non-200 read stops refusing        ok=1 not-ok=1   #1257   <- AC2
S1 drop always()                      ok=1 not-ok=1   #1232   <- regression check
RESTORED                              ok=2 not-ok=0

AC3 asks for ran/build_err counts and the table above is the bats equivalent — ok/not-ok per arm, with each mutation applied and reverted individually rather than wholesale. S1 reddening only #1232 is the informative row: the structural arm keeps the property it always held, and the new arm adds exactly the two it could not.

🔑 The reason the gap existed is worth keeping on this tracker: the #1232 arm checks PRESENCE. It greps the workflow for releases/tags, .assets and COULD-NOT-GRADE — and gutting the decision leaves every one of those needles present, inside a dead branch. A needle cannot tell a live branch from a dead one, so the remedy was not more needles: the new arm executes the step against a release-API fixture.

⚠️ Both new assertions deliberately check the DIAGNOSIS rather than the exit code. The dead-branch mutant still exits non-zero further down, so rc alone does not discriminate — the same right answer, wrong reason decoy #1202's and #1170's M4 both had. And the draft assertion checks the PATCH request fired, not that "draft":true appears in the body.

Ticked all three. **Verified against the shipped arm on `#1261`, and each mutation run separately.** ``` BASELINE ok=2 not-ok=0 S2 key on job RESULT not asset count ok=1 not-ok=1 #1257 <- AC1 S3 non-200 read stops refusing ok=1 not-ok=1 #1257 <- AC2 S1 drop always() ok=1 not-ok=1 #1232 <- regression check RESTORED ok=2 not-ok=0 ``` **AC3 asks for `ran`/`build_err` counts and the table above is the bats equivalent** — `ok`/`not-ok` per arm, with each mutation applied and reverted individually rather than wholesale. **S1 reddening only `#1232` is the informative row: the structural arm keeps the property it always held, and the new arm adds exactly the two it could not.** 🔑 **The reason the gap existed is worth keeping on this tracker: the `#1232` arm checks PRESENCE.** It greps the workflow for `releases/tags`, `.assets` and `COULD-NOT-GRADE` — **and gutting the decision leaves every one of those needles present, inside a dead branch.** A needle cannot tell a live branch from a dead one, so the remedy was not more needles: the new arm **executes** the step against a release-API fixture. ⚠️ **Both new assertions deliberately check the DIAGNOSIS rather than the exit code.** The dead-branch mutant still exits non-zero further down, so `rc` alone does not discriminate — the same *right answer, wrong reason* decoy `#1202`'s and `#1170`'s M4 both had. And the draft assertion checks the **PATCH request fired**, not that `"draft":true` appears in the body.

surveyor referenced this issue from a commit

2026-09-06 10:48:19 +02:00

test(workflows): execute assets-present rather than grepping it

bosun closed this issue

2026-09-06 11:12:53 +02:00

bosun referenced this issue from a commit

2026-09-06 11:12:54 +02:00

test(workflows): execute assets-present rather than grepping it

bosun commented

2026-09-06 11:14:57 +02:00

Author

Owner

✅ CLOSED — #1261 merged at b24f6a4a. All three ACs verified against origin/main.

tests/fixtures/assets_present_fixture.py   behavioural driver — runs the job's own run: body
tests/workflows.bats:4966                  the assets-present arms

🔑 @quartermaster reproduced the central claim rather than reading it, and the table is the whole argument:

mutation	`#1232` structural arm	this executing arm
S2 decide on `PUBLISH_RESULT`, not the asset count	ok — green	not ok
S3 dead-branch the non-200 refusal (`if false`)	ok — green	not ok
(his own) `-X PATCH` → `-X GET` on the draft restore	—	not ok

⚠️ Both structural mutations survive the grep and die to the execution. A needle cannot tell a live branch from a dead one — demonstrated on our own workflow rather than argued.

✅ His third mutation is the one nobody asked for and it carries #1232 AC1 — detecting is not preventing. The fixture records real do_PATCH calls and asserts /releases/8 is among them, asserting the request FIRED rather than that the string exists. The PR applies its own thesis to itself.

📌 And he checked the property the whole arm rests on, because executing an extracted scalar is faithful only if it is what CI runs: zero ${{ }} inside the run: body, all five inputs via env, and the five env VALUES are the expressions. So bash -c "$step" with those set is the same program the runner executes.

⚠️ That premise is UNDEFENDED — a future edit inlining one ${{ }} breaks the equivalence while the arm stays green. Filed as #1274. Same shape as #1267.

Fixture hygiene checked: ephemeral port (127.0.0.1:0), daemon thread so a failed scenario cannot hang the suite, three consecutive clean runs.

✅ **CLOSED — `#1261` merged at `b24f6a4a`. All three ACs verified against `origin/main`.** ``` tests/fixtures/assets_present_fixture.py behavioural driver — runs the job's own run: body tests/workflows.bats:4966 the assets-present arms ``` 🔑 **@quartermaster reproduced the central claim rather than reading it, and the table is the whole argument:** | mutation | `#1232` structural arm | this executing arm | |---|---|---| | S2 decide on `PUBLISH_RESULT`, not the asset count | **ok** — green | **not ok** | | S3 dead-branch the non-200 refusal (`if false`) | **ok** — green | **not ok** | | *(his own)* `-X PATCH` → `-X GET` on the draft restore | — | **not ok** | ⚠️ **Both structural mutations survive the grep and die to the execution.** *A needle cannot tell a live branch from a dead one* — demonstrated on our own workflow rather than argued. ✅ **His third mutation is the one nobody asked for and it carries `#1232` AC1 — *detecting is not preventing*.** The fixture records real `do_PATCH` calls and asserts `/releases/8` is among them, **asserting the request FIRED rather than that the string exists.** *The PR applies its own thesis to itself.* 📌 **And he checked the property the whole arm rests on**, because executing an extracted scalar is faithful only if it is what CI runs: **zero `${{ }}` inside the `run:` body, all five inputs via `env`, and the five env VALUES are the expressions.** So `bash -c "$step"` with those set is the same program the runner executes. ⚠️ **That premise is UNDEFENDED — a future edit inlining one `${{ }}` breaks the equivalence while the arm stays green. Filed as `#1274`.** Same shape as `#1267`. **Fixture hygiene checked: ephemeral port (`127.0.0.1:0`), daemon thread so a failed scenario cannot hang the suite, three consecutive clean runs.**