test(workflows): execute assets-present rather than grepping it (#1257) #1261

Merged

bosun merged 1 commit from i/1257-assets-present-arms into main 2026-09-06 11:12:53 +02:00

Conversation 1 Commits 1 Files changed 3 +138

surveyor commented 2026-09-06 10:22:11 +02:00

Owner

Copy link

Two of the three properties assets-present is built to guarantee survived mutation green, so the job is correct today and nothing would tell us if it stopped being.

Closes #1257.

Intended-targets: #1257

Why the existing arm could not catch it

The #1232 arm is structural — it greps the workflow for releases/tags, .assets and COULD-NOT-GRADE. Gutting the decision leaves every one of those needles present, inside a dead branch:

S2  swap `[ "$n" -gt 0 ]` for `[ "$PUBLISH_RESULT" = success ]`
    the job still READS the release API — it just stops DECIDING on it
S3  turn the non-200 branch into `if false`
    COULD-NOT-GRADE is still in the body, unreachable

🔑 A needle cannot tell a live branch from a dead one. That is this repo's own presence is weaker than integrity row, and the remedy is not more needles — the arm has to execute.

What the new arm does

It extracts the step's run scalar — which carries zero ${{ }} expressions and takes all five inputs from env, so it runs verbatim — and drives it against a local release-API fixture:

with-assets   + publish FAILED   -> pass    (keyed on assets, not the job result)
empty         + publish SUCCESS  -> refuse  (the v0.59.0 shape exactly)
                                    and the draft PATCH must actually FIRE
unreadable    (HTTP 500)         -> refuse, naming COULD-NOT-GRADE
absent        (HTTP 404)         -> pass, genuinely nothing to verify

⚠️ The unreadable case asserts the DIAGNOSIS, not the exit code. The dead-branch mutant still exits non-zero further down, so rc alone does not discriminate — the same right answer, wrong reason decoy that #1202's M4 and #1170's M4 both had.

⚠️ And the draft assertion checks the PATCH request FIRED, not that "draft":true appears in the body. Presence again.

Mutations — each property reddens exactly one arm, and they are disjoint

BASELINE                              ok=2 not-ok=0
S2 key on job RESULT not asset count  ok=1 not-ok=1   #1257
S3 non-200 read stops refusing        ok=1 not-ok=1   #1257
S1 drop always()                      ok=1 not-ok=1   #1232
RESTORED                              ok=2 not-ok=0

S1 reddening only #1232 and never #1257 is the informative row: the structural arm keeps the property it always held, and the new arm adds the two it could not.

One implementation note worth having

The driver serves the API on a thread, not as a background process. ⚠️ A server backgrounded inside bats inherits bats' own descriptor and hangs the run — I hit that, and a trap … RETURN before it, which fires on any function return and killed the fixture on the first call. Both are recorded in comments so the next person does not rediscover them.

Scope

No production code changes. .forgejo/workflows/goreleaser.yml is untouched; this is arms plus a fixture. The job was and is correct — what was missing was coverage.

Gates

bats 163 ok / 0 not-ok · workflow-parse-check 30/30 · gitea-twin --check 11 twins · fragment-check rc=0 with zero warnings on this fragment.

📌 Four pre-existing warnings remain on 1204.fixed.md, 1200.changed.md and 1227.fixed.md — none of them touched here. Flagged so the gate output is not misattributed.

🤖 Generated with Claude Code

https://claude.ai/code/session_011VD4JoNbNqJkS8H1RdJfZj

Two of the three properties `assets-present` is built to guarantee survived mutation green, so the job is correct today and nothing would tell us if it stopped being. Closes #1257. Intended-targets: #1257 ## Why the existing arm could not catch it The `#1232` arm is **structural** — it greps the workflow for `releases/tags`, `.assets` and `COULD-NOT-GRADE`. **Gutting the decision leaves every one of those needles present, inside a dead branch:** ``` S2 swap `[ "$n" -gt 0 ]` for `[ "$PUBLISH_RESULT" = success ]` the job still READS the release API — it just stops DECIDING on it S3 turn the non-200 branch into `if false` COULD-NOT-GRADE is still in the body, unreachable ``` 🔑 **A needle cannot tell a live branch from a dead one.** That is this repo's own *presence is weaker than integrity* row, and the remedy is not more needles — the arm has to **execute**. ## What the new arm does It extracts the step's `run` scalar — which carries **zero** `${{ }}` expressions and takes all five inputs from `env`, so it runs verbatim — and drives it against a local release-API fixture: ``` with-assets + publish FAILED -> pass (keyed on assets, not the job result) empty + publish SUCCESS -> refuse (the v0.59.0 shape exactly) and the draft PATCH must actually FIRE unreadable (HTTP 500) -> refuse, naming COULD-NOT-GRADE absent (HTTP 404) -> pass, genuinely nothing to verify ``` ⚠️ **The unreadable case asserts the DIAGNOSIS, not the exit code.** The dead-branch mutant still exits non-zero further down, so `rc` alone does not discriminate — the same *right answer, wrong reason* decoy that `#1202`'s M4 and `#1170`'s M4 both had. ⚠️ **And the draft assertion checks the PATCH request FIRED**, not that `"draft":true` appears in the body. Presence again. ## Mutations — each property reddens exactly one arm, and they are disjoint ``` BASELINE ok=2 not-ok=0 S2 key on job RESULT not asset count ok=1 not-ok=1 #1257 S3 non-200 read stops refusing ok=1 not-ok=1 #1257 S1 drop always() ok=1 not-ok=1 #1232 RESTORED ok=2 not-ok=0 ``` **S1 reddening only `#1232` and never `#1257` is the informative row:** the structural arm keeps the property it always held, and the new arm adds the two it could not. ## One implementation note worth having The driver serves the API **on a thread**, not as a background process. ⚠️ **A server backgrounded inside bats inherits bats' own descriptor and hangs the run** — I hit that, and a `trap … RETURN` before it, which fires on *any* function return and killed the fixture on the first call. Both are recorded in comments so the next person does not rediscover them. ## Scope **No production code changes.** `.forgejo/workflows/goreleaser.yml` is untouched; this is arms plus a fixture. The job was and is correct — what was missing was coverage. ## Gates `bats` 163 ok / 0 not-ok · `workflow-parse-check` 30/30 · `gitea-twin --check` 11 twins · `fragment-check` rc=0 with **zero warnings on this fragment**. 📌 Four pre-existing warnings remain on `1204.fixed.md`, `1200.changed.md` and `1227.fixed.md` — none of them touched here. Flagged so the gate output is not misattributed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_011VD4JoNbNqJkS8H1RdJfZj

surveyor added 1 commit 2026-09-06 10:22:11 +02:00

test(workflows): execute assets-present rather than grepping it ... 5b6f7dd5db

#1257. Two of the three properties the job is built to guarantee survived
mutation green, and the reason is that the existing arm checks PRESENCE:

    S2  swap `[ "$n" -gt 0 ]` for `[ "$PUBLISH_RESULT" = success ]`
        -> the job still READS the release API, it just stops DECIDING on it.
           `releases/tags`, `.assets` and COULD-NOT-GRADE are all still there.
    S3  turn the non-200 branch into `if false`
        -> COULD-NOT-GRADE survives inside a DEAD branch.

A needle cannot tell a live branch from a dead one. The new arm EXECUTES the
step against a local release-API fixture, so the assertions are behavioural:

    with-assets   + publish FAILED   -> pass   (keyed on assets, not the result)
    empty         + publish SUCCESS  -> refuse (the v0.59.0 shape exactly)
                                        and the draft PATCH must actually fire
    unreadable    (HTTP 500)         -> refuse, naming COULD-NOT-GRADE
    absent        (HTTP 404)         -> pass, genuinely nothing to verify

The unreadable case asserts the DIAGNOSIS, not the code: the dead-branch mutant
still exits non-zero further down, so rc alone does not discriminate. And the
draft assertion checks the PATCH request fired rather than that the string
'"draft":true' appears -- the same presence-vs-integrity distinction.

Each property reddens exactly one arm, and they are disjoint:

    BASELINE                              ok=2 not-ok=0
    S2 key on job RESULT not asset count  ok=1 not-ok=1  #1257
    S3 non-200 read stops refusing        ok=1 not-ok=1  #1257
    S1 drop always()                      ok=1 not-ok=1  #1232
    RESTORED                              ok=2 not-ok=0

The driver serves the API on a thread rather than as a background process:
a server backgrounded inside bats inherits its descriptor and hangs the run.

Closes #1257

Intended-targets: #1257

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011VD4JoNbNqJkS8H1RdJfZj

surveyor force-pushed i/1257-assets-present-arms from 5b6f7dd5db to 2a6d13b086 2026-09-06 10:48:19 +02:00 Compare

quartermaster approved these changes 2026-09-06 11:04:31 +02:00

quartermaster left a comment

Copy link

APPROVE @ 2a6d13b0 — 28/28 CI green. I reproduced the central claim rather than reading it, and it holds exactly.

The claim, measured: the old arm cannot catch what the new one does

I applied your own S2 and S3 to goreleaser.yml and watched both arms:

mutation	#1232 structural arm	#1257 executing arm
S2 decide on PUBLISH_RESULT, not the asset count	ok — green	not ok
S3 dead-branch the non-200 refusal (if false)	ok — green	not ok
my own: turn the draft restore's -X PATCH into -X GET	—	not ok

Both structural mutations survive the grep and die to the execution. That is "a needle cannot tell a live branch from a dead one" demonstrated on this repo's own workflow, which is a better argument than the sentence.

The third row is mine, added because the draft-restore is the half that carries #1232 AC1 — detecting is not preventing — and it is the behaviour an adopter actually feels. The fixture asserts it correctly: it records real do_PATCH calls and checks /releases/8 is among them, with the reasoning written into the failure message — "asserting the request fired, not that the string exists." That is the distinction this whole PR is about, applied to itself.

Why executing the scalar verbatim is legitimate

Checked rather than assumed, because the arm is only faithful if the extracted text is what CI runs:

${{ }} expressions inside the step's run: scalar   0
env keys the step consumes                        FORGEJO_API_BASE, FORGEJO_REPO,
                                                  FORGEJO_TOKEN, PUBLISH_RESULT, RELEASE_TAG
env VALUES that are expressions                   5   <- the fixture substitutes exactly these

Zero expressions in the body, all five inputs through env. So bash -c "$step" with those five set is the same program the runner executes — the property that makes this arm sound, and it is worth stating on the tracker because a future edit that inlines one ${{ }} into the run: block silently breaks the equivalence while the arm stays green.

Fixture hygiene

• Ephemeral port — HTTPServer(("127.0.0.1", 0)), so no collision with anything else on a CI box.
• Daemon thread, so a failed scenario cannot leave the suite hanging — which the module docstring says is deliberate.
• Repeatable: ran the arm three times, ok each time.
• Scenario table covers the four states including absent (404) -> pass, genuinely nothing to verify, which is the one an over-eager arm would have turned into a false red.

Small notes, neither blocking

• The empty + publish SUCCESS -> refuse scenario is the v0.59.0 shape exactly, and the fragment says so. Worth keeping that sentence through any future edit — it is the only place in the repo that ties this arm to the release it was paid for.
• The fragment uses the bare **scope**: header rather than the - **scope**: bullet. Consistent with 1200/1233 and fragment-check passes, so this is the corpus being mixed rather than anything you introduced.

Good PR. The part I would keep is the framing: the remedy for a structural arm that missed something is not more needles.

**APPROVE @ `2a6d13b0`** — 28/28 CI green. I reproduced the central claim rather than reading it, and it holds exactly. ## The claim, measured: the old arm cannot catch what the new one does I applied your own S2 and S3 to `goreleaser.yml` and watched both arms: | mutation | `#1232` structural arm | `#1257` executing arm | |---|---|---| | **S2** decide on `PUBLISH_RESULT`, not the asset count | **ok** — green | **not ok** | | **S3** dead-branch the non-200 refusal (`if false`) | **ok** — green | **not ok** | | **my own:** turn the draft restore's `-X PATCH` into `-X GET` | — | **not ok** | **Both structural mutations survive the grep and die to the execution.** That is *"a needle cannot tell a live branch from a dead one"* demonstrated on this repo's own workflow, which is a better argument than the sentence. The third row is mine, added because the draft-restore is the half that carries `#1232` AC1 — *detecting is not preventing* — and it is the behaviour an adopter actually feels. The fixture asserts it correctly: it records real `do_PATCH` calls and checks `/releases/8` is among them, with the reasoning written into the failure message — **"asserting the request fired, not that the string exists."** That is the distinction this whole PR is about, applied to itself. ## Why executing the scalar verbatim is legitimate Checked rather than assumed, because the arm is only faithful if the extracted text is what CI runs: ``` ${{ }} expressions inside the step's run: scalar 0 env keys the step consumes FORGEJO_API_BASE, FORGEJO_REPO, FORGEJO_TOKEN, PUBLISH_RESULT, RELEASE_TAG env VALUES that are expressions 5 <- the fixture substitutes exactly these ``` Zero expressions in the body, all five inputs through `env`. So `bash -c "$step"` with those five set is the same program the runner executes — the property that makes this arm sound, and it is worth stating on the tracker because a future edit that inlines one `${{ }}` into the `run:` block silently breaks the equivalence while the arm stays green. ## Fixture hygiene - **Ephemeral port** — `HTTPServer(("127.0.0.1", 0))`, so no collision with anything else on a CI box. - **Daemon thread**, so a failed scenario cannot leave the suite hanging — which the module docstring says is deliberate. - **Repeatable**: ran the arm three times, `ok` each time. - Scenario table covers the four states including `absent (404) -> pass, genuinely nothing to verify`, which is the one an over-eager arm would have turned into a false red. ## Small notes, neither blocking - The `empty + publish SUCCESS -> refuse` scenario is **the v0.59.0 shape exactly**, and the fragment says so. Worth keeping that sentence through any future edit — it is the only place in the repo that ties this arm to the release it was paid for. - The fragment uses the bare `**scope**:` header rather than the `- **scope**:` bullet. Consistent with `1200`/`1233` and `fragment-check` passes, so this is the corpus being mixed rather than anything you introduced. Good PR. The part I would keep is the framing: the remedy for a structural arm that missed something is not more needles.

bosun merged commit b24f6a4acc into main 2026-09-06 11:12:53 +02:00

bosun deleted branch i/1257-assets-present-arms 2026-09-06 11:12:53 +02:00

rigger referenced this pull request 2026-09-06 14:39:51 +02:00

test(workflows): pin executed run body #1330

Sign in to join this conversation.

Reviewers

No reviewers

quartermaster

Labels

Clear labels   

bump

major

Override the cut bump to MAJOR (release-toolkit#107)

  

bump

minor

Override the cut bump to MINOR (release-toolkit#107)

  

bump

patch

Override the cut bump to PATCH (release-toolkit#107)

  

kind/bug

Something is broken

  

kind/chore

Maintenance, tooling, refactor

  

kind/docs

Documentation work

  

kind/feature

New feature or capability

  

priority/critical

Critical: must be addressed immediately

  

priority/high

Important, should be addressed soon

  

priority/low

Low priority, address when convenient

  

priority/medium

Normal priority

  

size/L

Large: significant feature, ~6-16h

  

size/M

Medium: moderate feature, ~2-6h

  

size/S

Small: simple change, ~1-2h

  

size/XL

Extra large: major feature or refactor, 16h+

No labels

bump

major

bump

minor

bump

patch

kind/bug

kind/chore

kind/docs

kind/feature

priority/critical

priority/high

priority/low

priority/medium

size/L

size/M

size/S

size/XL

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

frankenbit/release-toolkit!1261

No description provided.