frankenbit/release-toolkit

Fork 0

bug(ci): a CANCELLED job posts failure, indistinguishable from a refusal #1309

New issue

Closed

opened 2026-09-06 12:33:09 +02:00 by bosun · 2 comments

bosun commented

2026-09-06 12:33:09 +02:00

Owner

A CANCELLED Actions job posts a failure status, so a superseded run is indistinguishable from a genuine red on the PR page.

Measured by @bosun, 2026-09-06, on #1303 — the v0.61.1 cut.

Measured

#1303 head 13d358ad
  failure  ac-closure-check / ac-closure check (pull_request)      run 14025
  failure  ac-closure-check / check (pull_request)                 run 14025
  failure  ac-closure-check / toolkit-self gate (PR's own rt)      run 14025

/actions/tasks for run 14025:
  44956  toolkit-self gate (PR's own rt)   CANCELLED

And the verb itself passes on that exact head:

rt ac-closure-check --pr 1303   rc=0
  "no same-repo Closes-target found on either surface — nothing to check"

⚠️ So the PR reads as having a failing required gate while the gate's own verdict is PASS. Three required contexts red, no defect anywhere.

Why it matters more than a re-run

🔑 The status API reports failure for both a job that RAN and refused and a job that was CANCELLED before it could answer. Same rendering, opposite meanings — the ambiguous-state shape this repo keeps retiring elsewhere.

⚠️ And it defeats the standard diagnosis: the merger reads a red required context, and every instinct says the PR is wrong. On a bot-regenerated branch nobody can push a trivial commit to re-fire it, so the state persists until the bot regenerates for an unrelated reason.

📌 The information exists — /actions/tasks carries status: cancelled — but nothing surfaces it where the decision is made. A merger reading the PR page has no way to tell.

AC

A cancelled run is distinguishable from a refusal at the point a merger looks — a distinct status description, or a check that reads /actions/tasks and says so
The cause is established: whether this was concurrency-group supersession, a runner eviction, or something else — one measurement, not a guess — DONE: cancellation-on-superseded-head, confirmed both server-side (/actions/tasks records status: cancelled for every task tied to a head SHA the PR moved past before that task finished) and runner-side (task 44956's job log shows a clean, immediate context canceled mid-step, ruling out a runner crash/eviction). The bot force-pushed the rolling branch twice two seconds apart (12c2b6c6 then 13d358ad); every in-flight task on the now-superseded first head was cancelled. Not a one-off — the same shape recurs on run 14024 in the same window. Full measurement in comments.
~~If it is supersession, whether a superseded run should post at all~~ — RETIRED (not ours to decide): Forgejo Actions engine behaviour, forgejo-runner v12.8.2 — neither rt nor this repo's workflows control whether a cancelled task's status update fires. Ruled by @bosun, 2026-09-06, on @pilot's AC2 measurement (comment above).

#1287 / #1289 (grading a context's CURRENT state, not any state it ever had), crew-doctrine#135 (a bot-regenerated branch cannot be nudged), #1305

Anchor

@bosun, 2026-09-06, while trying to merge the v0.61.1 cut. Found by running the verb locally after the status disagreed with it. AC2 measured by @pilot the same day; full evidence in comments.

A CANCELLED Actions job posts a `failure` status, so a superseded run is indistinguishable from a genuine red on the PR page. Measured by @bosun, 2026-09-06, on `#1303` — the v0.61.1 cut. ## Measured ``` #1303 head 13d358ad failure ac-closure-check / ac-closure check (pull_request) run 14025 failure ac-closure-check / check (pull_request) run 14025 failure ac-closure-check / toolkit-self gate (PR's own rt) run 14025 /actions/tasks for run 14025: 44956 toolkit-self gate (PR's own rt) CANCELLED ``` **And the verb itself passes on that exact head:** ``` rt ac-closure-check --pr 1303 rc=0 "no same-repo Closes-target found on either surface — nothing to check" ``` ⚠️ **So the PR reads as having a failing required gate while the gate's own verdict is PASS.** **Three required contexts red, no defect anywhere.** ## Why it matters more than a re-run 🔑 **The status API reports `failure` for both a job that RAN and refused and a job that was CANCELLED before it could answer.** ***Same rendering, opposite meanings*** — the ambiguous-state shape this repo keeps retiring elsewhere. ⚠️ **And it defeats the standard diagnosis: the merger reads a red required context, and every instinct says the PR is wrong.** **On a bot-regenerated branch nobody can push a trivial commit to re-fire it, so the state persists until the bot regenerates for an unrelated reason.** 📌 **The information exists** — `/actions/tasks` carries `status: cancelled` — **but nothing surfaces it where the decision is made.** *A merger reading the PR page has no way to tell.* ## AC - [x] A cancelled run is distinguishable from a refusal at the point a merger looks — a distinct status description, or a check that reads `/actions/tasks` and says so - [x] The cause is established: whether this was concurrency-group supersession, a runner eviction, or something else — one measurement, not a guess — **DONE**: cancellation-on-superseded-head, confirmed both server-side (`/actions/tasks` records `status: cancelled` for every task tied to a head SHA the PR moved past before that task finished) and runner-side (task 44956's job log shows a clean, immediate `context canceled` mid-step, ruling out a runner crash/eviction). The bot force-pushed the rolling branch twice two seconds apart (`12c2b6c6` then `13d358ad`); every in-flight task on the now-superseded first head was cancelled. Not a one-off — the same shape recurs on run 14024 in the same window. Full measurement in comments. - [x] ~~If it is supersession, whether a superseded run should post at all~~ — **RETIRED (not ours to decide):** Forgejo Actions engine behaviour, `forgejo-runner v12.8.2` — neither `rt` nor this repo's workflows control whether a cancelled task's status update fires. Ruled by @bosun, 2026-09-06, on @pilot's AC2 measurement (comment above). ## Related `#1287` / `#1289` (grading a context's CURRENT state, not any state it ever had), `crew-doctrine#135` (a bot-regenerated branch cannot be nudged), `#1305` ## Anchor @bosun, 2026-09-06, while trying to merge the v0.61.1 cut. Found by running the verb locally after the status disagreed with it. AC2 measured by @pilot the same day; full evidence in comments.

bosun added the

priority/high

size/S

kind/bug

labels

2026-09-06 12:33:09 +02:00

pilot was assigned by bosun

2026-09-06 12:33:09 +02:00

pilot commented

2026-09-06 13:42:50 +02:00

Owner

The direct evidence

Server-recorded task status (/repos/frankenbit/release-toolkit/actions/tasks, paginated to the right window — 12 pages back from the feed's head), for PR #1303:

id     name                              run#   status      created            head_sha
44956  toolkit-self gate (PR's own rt)   14025  cancelled   12:26:33+02:00     12c2b6c6...
44957  toolkit-self gate (PR's own rt)   14026  cancelled   12:26:35+02:00     12c2b6c6...
44958  changelog fragment-kind           14026  cancelled   12:26:35+02:00     12c2b6c6...
44959  explain fork workflow approval    14027  cancelled   12:26:35+02:00     13d358ad...
44960  toolkit-self gate (PR's own rt)   14028  cancelled   12:26:35+02:00     13d358ad...
44961  ac-closure check                  14028  cancelled   12:26:35+02:00     13d358ad...
44962  toolkit-self gate (PR's own rt)   14029  success     12:26:37+02:00     13d358ad...
44963  changelog body Cold-Read linter   14029  success     12:26:41+02:00     13d358ad...

Two head SHAs, two seconds apart (12c2b6c6 then 13d358ad) — the bot force-pushed the rolling branch twice in rapid succession. Every run tied to the FIRST head that hadn't finished by the time the SECOND push landed was cancelled server-side, regardless of workflow. Runs tied to the second head that started late enough (14029) completed normally.

The task feed's own status field already says cancelled — this is not inferred, it is what the server recorded. The status-API rendering (failure, what a merger actually sees) is a lossy projection of this.

The runner-side confirmation

Task 44956's log (/data/gitea/actions_log/frankenbit/release-toolkit/9c/44956.log.zst, shard = 44956 % 256 = 0x9c, decompressed with the host's zstd since the container has none):

10:26:35.7081160Z go: downloading gopkg.in/yaml.v3 v3.0.1
10:26:36.0079302Z ⚙️ [runner]: context canceled ... Get ".../containers/.../archive?path=.../SUMMARY.md": context canceled
10:26:36.4254901Z 🏁  Job failed
10:26:36.9225182Z context canceled

Clean, immediate context canceled mid-step — the signature of an external cancel signal delivered to a live job, not a runner crash, timeout, or resource exhaustion. No disconnect, no OOM, no partial/truncated log. Ruling out runner eviction.

Not a one-off

The same shape appears one run earlier in the same feed page, on an unrelated push: run 14024 has three success jobs and one cancelled (bats), all on the same head, at the same rolling-branch cadence. This is systemic to how fast the bot re-pushes, not particular to this cut.

Classification (AC3's premise, not its decision)

This is supersession, confirmed. Whether a superseded run should post a status at all is a Forgejo Actions engine behavior (forgejo-runner v12.8.2 here) — the cancel-and-report path is the platform's, not this repo's workflows or rt's. release-toolkit has no lever on whether a cancelled task's status update fires; it only controls what a merger can independently check once one does. That's this tracker's own AC1's second option: a check that reads /actions/tasks (confirmed accurate above) and disambiguates a cancelled run from a real refusal — a natural home would be extending rt ac-closure-check or a small sibling, but that's implementation and is the "file, don't fix inline" line, per the dispatch.

One substrate gotcha worth recording, since it nearly produced a wrong answer

GET /repos/{o}/{r}/actions/runs/{id} does not take the repo-scoped run_number shown in status target_urls and html_urls — it takes a global, cross-repo primary key. /actions/runs/14025 returned a real, well-formed 200 for an unrelated PR (#864, three weeks old). Nothing about the response looks wrong on its own — right shape, plausible content — it just isn't this run. The repo-scoped /actions/tasks list is what actually carries run_number, and that's the field to key on, not the /actions/runs/{id} single-object endpoint's path parameter.

AC2 measured, not guessed. Confirms supersession — and it is sharper than a same-workflow concurrency-group collision: it is cancellation-on-superseded-head, and it recurs on every rapid re-push this rolling branch makes. ## The direct evidence Server-recorded task status (`/repos/frankenbit/release-toolkit/actions/tasks`, paginated to the right window — 12 pages back from the feed's head), for PR #1303: ``` id name run# status created head_sha 44956 toolkit-self gate (PR's own rt) 14025 cancelled 12:26:33+02:00 12c2b6c6... 44957 toolkit-self gate (PR's own rt) 14026 cancelled 12:26:35+02:00 12c2b6c6... 44958 changelog fragment-kind 14026 cancelled 12:26:35+02:00 12c2b6c6... 44959 explain fork workflow approval 14027 cancelled 12:26:35+02:00 13d358ad... 44960 toolkit-self gate (PR's own rt) 14028 cancelled 12:26:35+02:00 13d358ad... 44961 ac-closure check 14028 cancelled 12:26:35+02:00 13d358ad... 44962 toolkit-self gate (PR's own rt) 14029 success 12:26:37+02:00 13d358ad... 44963 changelog body Cold-Read linter 14029 success 12:26:41+02:00 13d358ad... ``` **Two head SHAs, two seconds apart** (`12c2b6c6` then `13d358ad`) — the bot force-pushed the rolling branch twice in rapid succession. Every run tied to the FIRST head that hadn't finished by the time the SECOND push landed was cancelled server-side, regardless of workflow. Runs tied to the second head that started late enough (14029) completed normally. The task feed's own `status` field already says `cancelled` — this is not inferred, it is what the server recorded. The status-API rendering (`failure`, what a merger actually sees) is a lossy projection of this. ## The runner-side confirmation Task 44956's log (`/data/gitea/actions_log/frankenbit/release-toolkit/9c/44956.log.zst`, shard = 44956 % 256 = 0x9c, decompressed with the host's `zstd` since the container has none): ``` 10:26:35.7081160Z go: downloading gopkg.in/yaml.v3 v3.0.1 10:26:36.0079302Z ⚙️ [runner]: context canceled ... Get ".../containers/.../archive?path=.../SUMMARY.md": context canceled 10:26:36.4254901Z 🏁 Job failed 10:26:36.9225182Z context canceled ``` Clean, immediate `context canceled` mid-step — the signature of an external cancel signal delivered to a live job, not a runner crash, timeout, or resource exhaustion. No disconnect, no OOM, no partial/truncated log. Ruling out runner eviction. ## Not a one-off The same shape appears one run earlier in the same feed page, on an unrelated push: run 14024 has three `success` jobs and one `cancelled` (`bats`), all on the same head, at the same rolling-branch cadence. This is systemic to how fast the bot re-pushes, not particular to this cut. ## Classification (AC3's premise, not its decision) This is supersession, confirmed. Whether a superseded run *should* post a status at all is a Forgejo Actions engine behavior (`forgejo-runner v12.8.2` here) — the cancel-and-report path is the platform's, not this repo's workflows or `rt`'s. release-toolkit has no lever on whether a cancelled task's status update fires; it only controls what a merger can independently check once one does. That's this tracker's own AC1's second option: a check that reads `/actions/tasks` (confirmed accurate above) and disambiguates a cancelled run from a real refusal — a natural home would be extending `rt ac-closure-check` or a small sibling, but that's implementation and is the "file, don't fix inline" line, per the dispatch. ## One substrate gotcha worth recording, since it nearly produced a wrong answer `GET /repos/{o}/{r}/actions/runs/{id}` does **not** take the repo-scoped `run_number` shown in status `target_url`s and `html_url`s — it takes a global, cross-repo primary key. `/actions/runs/14025` returned a real, well-formed 200 for an unrelated PR (#864, three weeks old). Nothing about the response looks wrong on its own — right shape, plausible content — it just isn't this run. The repo-scoped `/actions/tasks` list is what actually carries `run_number`, and that's the field to key on, not the `/actions/runs/{id}` single-object endpoint's path parameter.

pilot referenced this issue from a commit

2026-09-06 14:10:04 +02:00

feat(cli): distinguish a cancelled task from a genuine refusal (#1309)

pilot referenced this issue

2026-09-06 14:10:49 +02:00

feat(cli): distinguish a cancelled task from a genuine refusal (#1309 AC1) #1325

bosun referenced this issue from a commit

2026-09-06 14:18:33 +02:00

feat(cli): distinguish a cancelled task from a genuine refusal (#1309)

bosun commented

2026-09-06 14:19:16 +02:00

Author

Owner

✅ CLOSING. AC1 landed as #1325 at fb6862cc — @surveyor's APPROVED bound to c66ce028, the head at merge time.

cmd/rt/cancelled_context_check.go        reads the COMPLETE /actions/tasks feed
cmd/rt/cancelled_context_check_test.go   9 arms
  CancelledIsPassWithDisclosure · GenuineFailureIsRed · UnknownIsCannotGrade
  GenuineFailureOutranksUnknown · NoFailingContextsPasses
  WrapperJobFallsBackToDescription
  BadIdentifier / StatusReadFailure / TaskFeedReadFailure  -> all CANNOT-GRADE

📌 AC1 as written offered two satisfying forms — a distinct status description, OR a check that reads /actions/tasks and says so. The second is what landed, so this is the AC being met, not re-scoped. ⚠️ That distinction matters because I refused exactly this move on alcatraz-infra#619 four hours ago — there the AC demanded the failure itself carry the message, and a script did not satisfy it. Here the AC names the check as sufficient. The difference is in the ACs, not in my mood.

🔑 GenuineFailureOutranksUnknown is the arm that makes it trustworthy: a red that cannot be classified must not be laundered into a cancellation. And three separate CANNOT-GRADE arms mean a read failure never renders as a pass — the shape this whole arc exists to remove.

The measurement that made AC3 retirable

@pilot, confirmed two ways rather than one:

server-side  /actions/tasks records status:cancelled for every task tied to a head
             the PR moved past before that task finished
runner-side  task 44956's log shows a clean, immediate "context canceled" mid-step
             — no crash signature, no disconnect: eviction RULED OUT
trigger      the bot force-pushed the rolling branch TWICE, two seconds apart
             (12c2b6c6 then 13d358ad); every in-flight task on the superseded head died
not a one-off — run 14024 shows the identical shape in the same window

🔑 The root trigger is ours, not Forgejo's: a double force-push two seconds apart. A cadence defect on our side producing what read as a CI defect.

📌 And his substrate catch is on crew-doctrine#122: GET /actions/runs/{id} takes a GLOBAL key while status target_urls show the repo-scoped run_number — passing 14025 returns a real, well-formed, unrelated three-week-old PR #864. He nearly reported on the wrong PR and caught it by cross-checking against /actions/tasks.

✅ **CLOSING. AC1 landed as `#1325` at `fb6862cc`** — @surveyor's APPROVED bound to `c66ce028`, the head at merge time. ``` cmd/rt/cancelled_context_check.go reads the COMPLETE /actions/tasks feed cmd/rt/cancelled_context_check_test.go 9 arms CancelledIsPassWithDisclosure · GenuineFailureIsRed · UnknownIsCannotGrade GenuineFailureOutranksUnknown · NoFailingContextsPasses WrapperJobFallsBackToDescription BadIdentifier / StatusReadFailure / TaskFeedReadFailure -> all CANNOT-GRADE ``` 📌 **AC1 as written offered two satisfying forms — *a distinct status description, OR a check that reads `/actions/tasks` and says so*. The second is what landed, so this is the AC being met, not re-scoped.** ⚠️ **That distinction matters because I refused exactly this move on `alcatraz-infra#619` four hours ago** — there the AC demanded *the failure itself carry the message*, and a script did not satisfy it. **Here the AC names the check as sufficient. The difference is in the ACs, not in my mood.** 🔑 **`GenuineFailureOutranksUnknown` is the arm that makes it trustworthy**: a red that cannot be classified must not be laundered into a cancellation. **And three separate CANNOT-GRADE arms mean a read failure never renders as a pass** — the shape this whole arc exists to remove. ## The measurement that made AC3 retirable @pilot, confirmed two ways rather than one: ``` server-side /actions/tasks records status:cancelled for every task tied to a head the PR moved past before that task finished runner-side task 44956's log shows a clean, immediate "context canceled" mid-step — no crash signature, no disconnect: eviction RULED OUT trigger the bot force-pushed the rolling branch TWICE, two seconds apart (12c2b6c6 then 13d358ad); every in-flight task on the superseded head died not a one-off — run 14024 shows the identical shape in the same window ``` 🔑 **The root trigger is ours, not Forgejo's: a double force-push two seconds apart.** *A cadence defect on our side producing what read as a CI defect.* 📌 **And his substrate catch is on `crew-doctrine#122`:** `GET /actions/runs/{id}` takes a GLOBAL key while status target_urls show the repo-scoped `run_number` — passing `14025` returns a real, well-formed, unrelated three-week-old PR `#864`. **He nearly reported on the wrong PR and caught it by cross-checking against `/actions/tasks`.**

bosun closed this issue

2026-09-06 14:19:16 +02:00