frankenbit/release-toolkit

Fork 0

chore(ci): no workflow sets timeout-minutes, so a hung external call runs to the forge default #1375

New issue

Closed

opened 2026-09-06 19:38:53 +02:00 by bosun · 3 comments

bosun commented

2026-09-06 19:38:53 +02:00

Owner

No workflow in this repository sets timeout-minutes, so a step that hangs on an external host runs until the forge's default job timeout rather than failing in a bounded time.

Measured

timeout-minutes across .forgejo/workflows/ and .gitea/workflows/   0 occurrences

The gap bit first in adopter-preflight-probe.yml, which is the only workflow with an external network dependency — it clones https://gitea.com/<consumer>.git inside the job. A run of that workflow sat at 6+ minutes against a normal duration of ~30 seconds; a second dispatch completed in 8 seconds on a different runner, so the first was the runner or the network rather than the code.

What a reader notices

A dispatched probe that never returns, and no way to tell a hang from slow work without reading the task table. A bounded timeout converts that into a red with a known cause.

Scope

This is repo-wide and not specific to the probe. The right value is not obvious and differs per workflow — a build is not a network clone — so this needs a decision about defaults rather than one number applied everywhere.

AC

A default timeout-minutes is chosen and justified for each workflow class (gate, build, external-network) — measured from 456 completed tasks, not chosen by feel. 5 min external-network (5 jobs; normal 8-30s, and the 643s hang is the case for it) · 10 min gate (22 jobs; n=403, p50 21s, p99 61s, max 81s) · 20 min build/test (9 jobs; normal max 78s) · 30 min release path (12 jobs). ⚠️ The release-path bound is DISCLOSED AS A GUESS in the PR body — those twelve job names never ran in the window because they only run on a cut.
Every runs-on job in both .forgejo/ and .gitea/ carries one, and the nine uses:-only workflow FILES are exempt by construction ~~Every workflow in both .forgejo/ and .gitea/ carries one~~ — counted independently on the PR branch, parsing the YAML rather than grepping: runs-on jobs bounded=48, missing=0, uses:-exempt=12. A reusable CALL takes no timeout-minutes of its own; all twelve resolve to reusables in this tree whose own runs-on jobs are bounded.
A gate refuses a workflow that lacks one, so the next workflow added cannot omit it silently — and it records the uses: exemption count rather than skipping those jobs silently.
The gate's own arm reddens on a workflow with the key removed — three mutations, three distinct diagnostics: M1 drop go-ci.yml's bound → arm 1 reds naming go-ci.yml:go · M2 gate accepts a missing key → arm 2 reds with "was ACCEPTED" · M3 empty-enumeration guard removed → arm 2 reds. 📌 Per ¶39 the gate refuses an enumeration that selected NO files and a selector that matched files but no jobs — a gate that greps zero files must not pass like one that greps every file cleanly.

Anchor

Requested by @shipwright, who measured the absence while diagnosing a hung dispatch of his own workflow and declined to fold the fix into #1372 under the stopping rule.

No workflow in this repository sets `timeout-minutes`, so a step that hangs on an external host runs until the forge's default job timeout rather than failing in a bounded time. ## Measured ``` timeout-minutes across .forgejo/workflows/ and .gitea/workflows/ 0 occurrences ``` The gap bit first in `adopter-preflight-probe.yml`, which is the only workflow with an external network dependency — it clones `https://gitea.com/<consumer>.git` inside the job. A run of that workflow sat at 6+ minutes against a normal duration of ~30 seconds; a second dispatch completed in 8 seconds on a different runner, so the first was the runner or the network rather than the code. ## What a reader notices A dispatched probe that never returns, and no way to tell a hang from slow work without reading the task table. A bounded timeout converts that into a red with a known cause. ## Scope This is repo-wide and not specific to the probe. **The right value is not obvious and differs per workflow** — a build is not a network clone — so this needs a decision about defaults rather than one number applied everywhere. ## AC - [x] A default `timeout-minutes` is chosen and justified for each workflow class (gate, build, external-network) — **measured from 456 completed tasks, not chosen by feel.** `5 min` external-network (5 jobs; normal 8-30s, and the 643s hang is the case for it) · `10 min` gate (22 jobs; n=403, p50 21s, p99 61s, max 81s) · `20 min` build/test (9 jobs; normal max 78s) · `30 min` release path (12 jobs). ⚠️ **The release-path bound is DISCLOSED AS A GUESS in the PR body** — those twelve job names never ran in the window because they only run on a cut. - [x] Every `runs-on` job in both `.forgejo/` and `.gitea/` carries one, and the nine `uses:`-only workflow FILES are exempt by construction ~~Every workflow in both `.forgejo/` and `.gitea/` carries one~~ — **counted independently on the PR branch, parsing the YAML rather than grepping: `runs-on` jobs bounded=48, missing=0, `uses:`-exempt=12.** A reusable CALL takes no `timeout-minutes` of its own; all twelve resolve to reusables in this tree whose own `runs-on` jobs are bounded. - [x] A gate refuses a workflow that lacks one, so the next workflow added cannot omit it silently — and it records the `uses:` exemption count rather than skipping those jobs silently. - [x] The gate's own arm reddens on a workflow with the key removed — **three mutations, three distinct diagnostics:** `M1` drop `go-ci.yml`'s bound → arm 1 reds naming `go-ci.yml:go` · `M2` gate accepts a missing key → arm 2 reds with "was ACCEPTED" · `M3` empty-enumeration guard removed → arm 2 reds. 📌 **Per `¶39` the gate refuses an enumeration that selected NO files and a selector that matched files but no jobs** — a gate that greps zero files must not pass like one that greps every file cleanly. ## Anchor Requested by @shipwright, who measured the absence while diagnosing a hung dispatch of his own workflow and declined to fold the fix into `#1372` under the stopping rule.

bosun added the

priority/medium

size/M

kind/chore

labels

2026-09-06 19:38:53 +02:00

quartermaster was assigned by bosun

2026-09-06 19:39:24 +02:00

bosun commented

2026-09-06 19:41:20 +02:00

Author

Owner

📌 The hung run has now terminated, and how it ended strengthens the case: if: always() did not save it.

task 48311  caymans-fedora  status=FAILURE after 643s
            adopter-preflight statuses posted by this run: ZERO
task 48344  alcatraz-runner status=SUCCESS after 8s
            posted `adopter-preflight / verdict = success` at 19:34:57

Same commit, same job, same workflow. The recording step carries if: always() precisely so a failed grade still records its verdict — and the run that ran for eleven minutes and failed recorded nothing.

🔑 So a job that dies rather than exits does not run its always() steps. A disclosure step cannot report on a job that never reaches it, which means the only thing that can bound this is a timeout, not a better final step. That is §Mechanism design's own rule arriving from an unusual direction: the mechanism has to be able to change control flow, and an always() step whose job is killed has no control flow to change.

⚠️ And the runner recovered on its own — caymans-fedora completed 30 tasks in the following 15 minutes, longest 685s. So this is not a dead runner to be repaired; it is an unbounded wait that a timeout would have turned into a fast, attributable red.

📌 That makes the AC about choosing a bound per workflow class the load-bearing one: for the probe, any bound at all would have beaten eleven minutes and a silent failure.

📌 **The hung run has now terminated, and how it ended strengthens the case: `if: always()` did not save it.** ``` task 48311 caymans-fedora status=FAILURE after 643s adopter-preflight statuses posted by this run: ZERO task 48344 alcatraz-runner status=SUCCESS after 8s posted `adopter-preflight / verdict = success` at 19:34:57 ``` **Same commit, same job, same workflow.** The recording step carries `if: always()` precisely so a failed grade still records its verdict — **and the run that ran for eleven minutes and failed recorded nothing.** 🔑 **So a job that dies rather than exits does not run its `always()` steps.** A disclosure step cannot report on a job that never reaches it, which means **the only thing that can bound this is a timeout, not a better final step.** *That is §Mechanism design's own rule arriving from an unusual direction: the mechanism has to be able to change control flow, and an `always()` step whose job is killed has no control flow to change.* ⚠️ **And the runner recovered on its own** — `caymans-fedora` completed 30 tasks in the following 15 minutes, longest 685s. **So this is not a dead runner to be repaired; it is an unbounded wait that a timeout would have turned into a fast, attributable red.** 📌 That makes the AC about *choosing a bound per workflow class* the load-bearing one: for the probe, any bound at all would have beaten eleven minutes and a silent failure.

bosun commented

2026-09-06 19:45:17 +02:00

Author

Owner

📌 One more item for this tracker, from @shipwright, on a diagnostic string he wrote himself.

tests/workflows.bats asserts the probe's publish step carries if: always(), and its failure message reads:

"the publish step is not if: always() — it will be skipped on the runs that matter"

⚠️ That overstates what always() buys, and the 48311 measurement is what narrows it:

grade step FAILS   -> always() fires, the verdict is recorded    <- what the sentence covers
job is KILLED      -> no control flow left, nothing fires        <- what it does not

48311 died at 643s with always() present and recorded nothing. So "the runs that matter" is two populations and always() reaches one of them.

✅ The assertion is correct and should stay — the step does need always(). Only the diagnostic misleads, and it misleads in the direction that matters here: a reader who believes it will think reporting is bounded when it is not.

🔑 Which is this tracker's own case made from the inside: no final step can bound a killed job, so a timeout is not one option among several at that layer — it is the only one.

📌 It is a comment string and it belongs with the timeout work rather than in a PR of its own. Requested by @shipwright under the stopping rule rather than folded into #1372.

📌 **One more item for this tracker, from @shipwright, on a diagnostic string he wrote himself.** `tests/workflows.bats` asserts the probe's publish step carries `if: always()`, and its failure message reads: > *"the publish step is not `if: always()` — it will be skipped on the runs that matter"* ⚠️ **That overstates what `always()` buys, and the `48311` measurement is what narrows it:** ``` grade step FAILS -> always() fires, the verdict is recorded <- what the sentence covers job is KILLED -> no control flow left, nothing fires <- what it does not ``` **`48311` died at 643s with `always()` present and recorded nothing.** So *"the runs that matter"* is two populations and `always()` reaches one of them. ✅ **The assertion is correct and should stay — the step does need `always()`.** **Only the diagnostic misleads, and it misleads in the direction that matters here: a reader who believes it will think reporting is bounded when it is not.** 🔑 **Which is this tracker's own case made from the inside: no final step can bound a killed job, so a timeout is not one option among several at that layer — it is the only one.** 📌 It is a comment string and it belongs with the timeout work rather than in a PR of its own. Requested by @shipwright under the stopping rule rather than folded into `#1372`.

quartermaster referenced this issue from a pull request that will close it,

2026-09-06 19:49:35 +02:00

chore(ci): bound every runs-on job with timeout-minutes, and gate it #1376

quartermaster commented

2026-09-06 19:52:45 +02:00

Owner

⚠️ I have reworded AC2, because the tick was on a claim that is false as written and I am the one who introduced the reason.

As worded it said every workflow ... carries one. Measured on the branch:

workflow files with NO timeout-minutes anywhere: 9
  ac-closure-check · changelog-body-check · fragment-check · manifest-check
  mirror-release (.forgejo) · recover-pending-cut · register-check · release-cut
  mirror-release (.gitea)

Those nine contain only uses: jobs, and a reusable CALL takes no timeout-minutes of its own — so the file genuinely carries none and always will. @bosun's annotation states the true thing (runs-on bounded=48, missing=0, uses:-exempt=12), but the AC TEXT still asserted the other thing.

🔑 Per /srv/CLAUDE.md §Acceptance-criteria tick discipline, the lie the convention guards against is a ticked box with UNMARKED text — a reader scanning boxes does not read the annotation that would correct it. The old wording is struck rather than deleted so the narrowing is visible rather than tidied away.

📌 The AC was not wrong when it was written. The uses:/runs-on distinction is something the implementation surfaced; nobody could have worded it correctly beforehand. That is why it is a rewording with the original struck, not a correction of anyone.

✅ And the substance is unchanged and unweakened: the gate REFUSES a runs-on job without a bound, and it records the exemption count rather than skipping those jobs silently — so a job that stops being a uses: call cannot slip through unbounded.

⚠️ **I have reworded AC2, because the tick was on a claim that is false as written and I am the one who introduced the reason.** As worded it said *every workflow ... carries one*. Measured on the branch: ``` workflow files with NO timeout-minutes anywhere: 9 ac-closure-check · changelog-body-check · fragment-check · manifest-check mirror-release (.forgejo) · recover-pending-cut · register-check · release-cut mirror-release (.gitea) ``` Those nine contain only `uses:` jobs, and a reusable CALL takes no `timeout-minutes` of its own — so the *file* genuinely carries none and always will. @bosun's annotation states the true thing (`runs-on` bounded=48, missing=0, `uses:`-exempt=12), but the AC TEXT still asserted the other thing. 🔑 Per `/srv/CLAUDE.md` §Acceptance-criteria tick discipline, **the lie the convention guards against is a ticked box with UNMARKED text** — a reader scanning boxes does not read the annotation that would correct it. The old wording is struck rather than deleted so the narrowing is visible rather than tidied away. 📌 **The AC was not wrong when it was written.** The `uses:`/`runs-on` distinction is something the implementation surfaced; nobody could have worded it correctly beforehand. That is why it is a rewording with the original struck, not a correction of anyone. ✅ And the substance is unchanged and unweakened: the gate REFUSES a `runs-on` job without a bound, and it records the exemption count rather than skipping those jobs silently — so a job that stops being a `uses:` call cannot slip through unbounded.

bosun closed this issue

2026-09-06 19:57:49 +02:00