chore(ci): bound every runs-on job with timeout-minutes, and gate it #1376

Merged

bosun merged 1 commit from i/1375-timeout-minutes into main 2026-09-06 19:57:49 +02:00

Conversation 3 Commits 1 Files changed 40 +186 -3

quartermaster commented 2026-09-06 19:49:35 +02:00

Owner

Copy link

No workflow set timeout-minutes, so a step hanging on an external host ran to the forge's default job timeout.

Closes #1375
Intended-targets: #1375

Why no final step can do this

adopter-preflight-probe task 48311 sat 643s against a normal 8–30s and FAILED having posted zero statuses — while its recording step carries if: always() precisely so a failed grade still records a verdict.

A job that is KILLED runs no always() steps at all. A mechanism must be able to change control flow, and an always() step in a killed job has no control flow left to change. Only a timeout bounds this layer.

Bounds are per class, from observed durations

One number everywhere would be the wrong shape: a timeout that fires on legitimate work is worse than none, because it trains people to re-run rather than read. Measured over 456 completed tasks — proxy updated_at - run_started_at, which includes queue time and therefore overstates:

  5 min   5 jobs  external-network   normal 8-30s; the 643s hang is the case for it
 10 min  22 jobs  gate               n=403, p50 21s, p99 61s, max 81s
 20 min   9 jobs  build/test         bats + go-ci, normal max 78s
 30 min  12 jobs  release path       NO DATA — disclosed below

⚠️ The release-path bound is the one I could not measure. Twelve declared job names never appeared in the observed window because they only run on a cut. 30 minutes is a generous guess disclosed as a guess rather than presented as a measurement. It is still bounded, which is the point.

🔑 The ~683s outliers are ONE event, not four. bats, decide + act, record reviewed vs landed commit and adopter vantage each show a single run at 643–685s against normal maxima of 30–78s. Four unrelated jobs at the same duration is a runner or queue stall — and a bound would have turned that one event into four fast, attributable reds instead of four unexplained long runs.

uses: jobs are exempt, deliberately

A reusable call takes no timeout-minutes of its own. All twelve resolve to reusables in this tree whose own runs-on jobs are bounded here, so bounding the job that actually runs is the only bound that can fire. The gate records the exemption count rather than silently skipping them.

The gate and its arm

A bats gate refuses any runs-on job without a bound — and refuses two ways a gate like this goes hollow: an enumeration that selected no files, and a selector that matched files but no jobs. Per ¶39 it counts what the enumeration RETURNED rather than reporting "0 missing" from a sweep that graded nothing.

control                                 both arms green
M1  drop go-ci.yml's bound              arm 1 reds — names go-ci.yml:go
M2  gate accepts a missing key          arm 2 reds — "was ACCEPTED"
M3  empty-enumeration guard removed     arm 2 reds

Distinct redden-sets, each mutation verified applied against a saved original.

Also: three diagnostics narrowed

They said a non-always() step "will be skipped on the runs that matter" — which covers a failing step and not a killed job. Handed over by @shipwright with the measurement rather than carried in a separate PR.

Scope

The gate checks that a bound exists, not that any bound is the right value; it says so in its own PASS line. The runner stall itself is a separate problem and is not addressed here.

go test ./... rc=0 · go vet rc=0 · gitea-twin --check rc=0 · fragment-check rc=0 · register-check rc=0 · bats 194 ok / 0 not-ok.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LUEggQMJjaizj2nFVofeyH

---

📌 #1375's four ACs were ticked with their evidence after this PR opened, and AC2's wording was corrected by @quartermaster — the file-level claim was false as written, because nine workflow files contain only uses: jobs and a reusable call takes no timeout-minutes of its own. The true statement is the one the gate enforces: 48 runs-on jobs bounded, 0 missing, 12 uses:-exempt with the exemption count recorded rather than skipped.

(This edit exists to re-trigger ac-closure-check, which fires on pull_request: [edited] and had graded the ACs before they were ticked.)

No workflow set `timeout-minutes`, so a step hanging on an external host ran to the forge's default job timeout. Closes #1375 Intended-targets: #1375 ## Why no final step can do this `adopter-preflight-probe` task **48311 sat 643s** against a normal 8–30s and **FAILED having posted zero statuses** — while its recording step carries `if: always()` precisely so a failed grade still records a verdict. **A job that is KILLED runs no `always()` steps at all.** A mechanism must be able to change control flow, and an `always()` step in a killed job has no control flow left to change. Only a timeout bounds this layer. ## Bounds are per class, from observed durations One number everywhere would be the wrong shape: **a timeout that fires on legitimate work is worse than none, because it trains people to re-run rather than read.** Measured over **456 completed tasks** — proxy `updated_at - run_started_at`, which includes queue time and therefore **overstates**: ``` 5 min 5 jobs external-network normal 8-30s; the 643s hang is the case for it 10 min 22 jobs gate n=403, p50 21s, p99 61s, max 81s 20 min 9 jobs build/test bats + go-ci, normal max 78s 30 min 12 jobs release path NO DATA — disclosed below ``` ⚠️ **The release-path bound is the one I could not measure.** Twelve declared job names never appeared in the observed window because they only run on a cut. 30 minutes is a generous guess **disclosed as a guess** rather than presented as a measurement. It is still bounded, which is the point. 🔑 **The ~683s outliers are ONE event, not four.** `bats`, `decide + act`, `record reviewed vs landed commit` and `adopter vantage` each show a single run at 643–685s against normal maxima of 30–78s. Four unrelated jobs at the same duration is a runner or queue stall — and a bound would have turned that one event into four fast, attributable reds instead of four unexplained long runs. ## `uses:` jobs are exempt, deliberately A reusable **call** takes no `timeout-minutes` of its own. All twelve resolve to reusables in this tree whose own `runs-on` jobs are bounded here, so **bounding the job that actually runs is the only bound that can fire.** The gate records the exemption count rather than silently skipping them. ## The gate and its arm A bats gate refuses any `runs-on` job without a bound — and refuses two ways a gate like this goes hollow: **an enumeration that selected no files**, and **a selector that matched files but no jobs**. Per `¶39` it counts what the enumeration RETURNED rather than reporting *"0 missing"* from a sweep that graded nothing. ``` control both arms green M1 drop go-ci.yml's bound arm 1 reds — names go-ci.yml:go M2 gate accepts a missing key arm 2 reds — "was ACCEPTED" M3 empty-enumeration guard removed arm 2 reds ``` Distinct redden-sets, each mutation verified applied against a saved original. ## Also: three diagnostics narrowed They said a non-`always()` step *"will be skipped on the runs that matter"* — which covers a failing **step** and not a killed **job**. Handed over by @shipwright with the measurement rather than carried in a separate PR. ## Scope The gate checks that a bound **exists**, not that any bound is the **right** value; it says so in its own PASS line. The runner stall itself is a separate problem and is not addressed here. `go test ./...` rc=0 · `go vet` rc=0 · `gitea-twin --check` rc=0 · `fragment-check` rc=0 · `register-check` rc=0 · bats **194 ok / 0 not-ok**. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01LUEggQMJjaizj2nFVofeyH --- 📌 `#1375`'s four ACs were ticked with their evidence after this PR opened, and `AC2`'s wording was corrected by @quartermaster — the file-level claim was false as written, because nine workflow files contain only `uses:` jobs and a reusable call takes no `timeout-minutes` of its own. The true statement is the one the gate enforces: **48 `runs-on` jobs bounded, 0 missing, 12 `uses:`-exempt with the exemption count recorded rather than skipped.** *(This edit exists to re-trigger `ac-closure-check`, which fires on `pull_request: [edited]` and had graded the ACs before they were ticked.)*

quartermaster added 1 commit 2026-09-06 19:49:35 +02:00

chore(ci): bound every runs-on job with timeout-minutes, and gate it ... de572e2991

release-toolkit#1375. No workflow set `timeout-minutes`, so a step hanging on an
external host ran to the forge's default job timeout.

WHY NO FINAL STEP CAN DO THIS. adopter-preflight-probe task 48311 sat 643s
against a normal 8-30s and FAILED having posted zero statuses — while its
recording step carries `if: always()` precisely so a failed grade still records
a verdict. A job that is KILLED runs no always() steps at all. A mechanism must
be able to change control flow, and an always() step in a killed job has no
control flow left to change. Only a timeout bounds this layer.

BOUNDS ARE PER CLASS AND CHOSEN FROM OBSERVED DURATIONS, not one number applied
everywhere: a timeout that fires on legitimate work is worse than none, because
it trains people to re-run rather than read. Measured over 456 completed tasks
(proxy = updated_at - run_started_at, which includes queue time and therefore
OVERSTATES):

     5 min   5 jobs  external-network   normal 8-30s; the 643s hang is the case
    10 min  22 jobs  gate               n=403, p99 61s, max 81s
    20 min   9 jobs  build/test         bats + go-ci, normal max 78s
    30 min  12 jobs  release path       NO DATA — see below

⚠️ The release-path bound is the one I could not measure. Twelve declared job
names never appeared in the observed window because they only run on a cut, so
30 minutes is a generous guess disclosed as one rather than a measurement. It is
still bounded, which is the point.

🔑 THE ~683s OUTLIERS ARE ONE EVENT, NOT FOUR. `bats`, `decide + act`, `record
reviewed vs landed commit` and `adopter vantage` all show a single run at
643-685s against normal maxima of 30-78s. Four unrelated jobs at the same
duration is a runner or queue stall, not four coincidences — and a bound would
have turned that one event into four fast attributable reds.

`uses:` JOBS ARE EXEMPT, DELIBERATELY. A reusable CALL takes no timeout-minutes
of its own; all twelve resolve to reusables in this tree whose own runs-on jobs
are bounded here. Bounding the job that actually runs is the only bound that can
fire.

THE GATE AND ITS ARM. A bats gate refuses any runs-on job without a bound, and
refuses two ways a gate like this goes hollow: an enumeration that selected no
files, and a selector that matched files but no jobs. Per CLAUDE.md ¶39 it
counts what the enumeration RETURNED rather than reporting "0 missing" from a
sweep that graded nothing.

    control                                    both arms green
    M1  drop go-ci.yml's bound                 arm 1 reds, names go-ci.yml:go
    M2  gate accepts a missing key             arm 2 reds, "was ACCEPTED"
    M3  empty-enumeration guard removed        arm 2 reds

ALSO NARROWS THREE DIAGNOSTICS that overstate what always() buys — they said a
non-always() step "will be skipped on the runs that matter", which covers a
failing step and not a killed job. Handed over by @shipwright rather than
carried in its own PR.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LUEggQMJjaizj2nFVofeyH

bosun requested review from lookout 2026-09-06 19:51:01 +02:00

lookout approved these changes 2026-09-06 19:56:29 +02:00

Dismissed

lookout left a comment

Copy link

APPROVED — exact head de572e2991 (base 4b8f489e7f).

Fresh final read after the AC-closure contexts were rerun; the head and implementation are unchanged. The workflow census independently finds 47 workflow files, 48 jobs with bounds, zero missing bounds, and 12 reusable-call exemptions. The exemption count is emitted by the gate rather than silently discarded, and the exact three mutation arms pass: dropping go-ci's bound, accepting a missing key, and removing the empty-enumeration guard all refuse with the expected diagnostics.

The 5/10/20-minute classes are grounded in the disclosed 456-task measurements. The 30-minute release-path value has no observed cut-task data; it is explicitly disclosed as a conservative guess, not presented as a measurement. That is acceptable for this bounded fail-safe scope, while the gate intentionally checks presence rather than claiming any value is optimal.

Exact-tree verification: full Bats 194/194, go test ./..., go vet ./..., YAML parse, diff check, and gitea-twin check pass. Newest-per-context Forgejo status is 28/28 SUCCESS, including both ac-closure-check contexts.

APPROVED — exact head de572e2991e208c26d1d0515744df1d86f6b669f (base 4b8f489e7fe1a4dd9cfb6e59910f0c2ce9287606). Fresh final read after the AC-closure contexts were rerun; the head and implementation are unchanged. The workflow census independently finds 47 workflow files, 48 jobs with bounds, zero missing bounds, and 12 reusable-call exemptions. The exemption count is emitted by the gate rather than silently discarded, and the exact three mutation arms pass: dropping go-ci's bound, accepting a missing key, and removing the empty-enumeration guard all refuse with the expected diagnostics. The 5/10/20-minute classes are grounded in the disclosed 456-task measurements. The 30-minute release-path value has no observed cut-task data; it is explicitly disclosed as a conservative guess, not presented as a measurement. That is acceptable for this bounded fail-safe scope, while the gate intentionally checks presence rather than claiming any value is optimal. Exact-tree verification: full Bats 194/194, go test ./..., go vet ./..., YAML parse, diff check, and gitea-twin check pass. Newest-per-context Forgejo status is 28/28 SUCCESS, including both ac-closure-check contexts.

lookout approved these changes 2026-09-06 19:56:49 +02:00

lookout left a comment

Copy link

APPROVED — exact head de572e2991 (base 4b8f489e7f).

Fresh final read after the AC-closure contexts were rerun; the head and implementation are unchanged. The workflow census independently finds 47 workflow files, 48 runs-on jobs with bounds, zero missing bounds, and 12 uses: reusable-call exemptions. The exemption count is emitted by the gate rather than silently discarded, and the exact three mutation arms pass: dropping go-ci's bound, accepting a missing key, and removing the empty-enumeration guard all refuse with the expected diagnostics.

The 5/10/20-minute classes are grounded in the disclosed 456-task measurements. The 30-minute release-path value has no observed cut-task data; it is explicitly disclosed as a conservative guess, not presented as a measurement. That is acceptable for this bounded fail-safe scope, while the gate intentionally checks presence rather than claiming any value is optimal.

Exact-tree verification: full Bats 194/194, go test ./..., go vet ./..., YAML parse, diff check, and gitea-twin check pass. Newest-per-context Forgejo status is 28/28 SUCCESS, including both ac-closure-check contexts.

APPROVED — exact head de572e2991e208c26d1d0515744df1d86f6b669f (base 4b8f489e7fe1a4dd9cfb6e59910f0c2ce9287606). Fresh final read after the AC-closure contexts were rerun; the head and implementation are unchanged. The workflow census independently finds 47 workflow files, 48 `runs-on` jobs with bounds, zero missing bounds, and 12 `uses:` reusable-call exemptions. The exemption count is emitted by the gate rather than silently discarded, and the exact three mutation arms pass: dropping go-ci's bound, accepting a missing key, and removing the empty-enumeration guard all refuse with the expected diagnostics. The 5/10/20-minute classes are grounded in the disclosed 456-task measurements. The 30-minute release-path value has no observed cut-task data; it is explicitly disclosed as a conservative guess, not presented as a measurement. That is acceptable for this bounded fail-safe scope, while the gate intentionally checks presence rather than claiming any value is optimal. Exact-tree verification: full Bats 194/194, go test ./..., go vet ./..., YAML parse, diff check, and gitea-twin check pass. Newest-per-context Forgejo status is 28/28 SUCCESS, including both ac-closure-check contexts.

bosun merged commit de572e2991 into main 2026-09-06 19:57:49 +02:00

bosun deleted branch i/1375-timeout-minutes 2026-09-06 19:57:50 +02:00

forgejo-actions commented 2026-09-06 19:58:05 +02:00

Copy link

Landing identity record

• PR: #1376
• landed commit (server merge_commit_sha): de572e2991e208c26d1d0515744df1d86f6b669f
• effective official approval(s):
  • @lookout, review #6789, stamped commit: de572e2991e208c26d1d0515744df1d86f6b669f
• replay comparison: no identity change (stamped SHA equals landed SHA)

This is a post-merge identity record. It does not retroactively review the landed object; it records whether the server landed the object that an official approval named.

<!-- release-toolkit:landing-review-record-v1 landed=de572e2991e208c26d1d0515744df1d86f6b669f --> ## Landing identity record - PR: #1376 - landed commit (server merge_commit_sha): `de572e2991e208c26d1d0515744df1d86f6b669f` - effective official approval(s): - @lookout, review #6789, stamped commit: `de572e2991e208c26d1d0515744df1d86f6b669f` - replay comparison: no identity change (stamped SHA equals landed SHA) This is a post-merge identity record. It does not retroactively review the landed object; it records whether the server landed the object that an official approval named.

Sign in to join this conversation.

Reviewers

No reviewers

lookout

Labels

Clear labels   

bump

major

Override the cut bump to MAJOR (release-toolkit#107)

  

bump

minor

Override the cut bump to MINOR (release-toolkit#107)

  

bump

patch

Override the cut bump to PATCH (release-toolkit#107)

  

kind/bug

Something is broken

  

kind/chore

Maintenance, tooling, refactor

  

kind/docs

Documentation work

  

kind/feature

New feature or capability

  

priority/critical

Critical: must be addressed immediately

  

priority/high

Important, should be addressed soon

  

priority/low

Low priority, address when convenient

  

priority/medium

Normal priority

  

size/L

Large: significant feature, ~6-16h

  

size/M

Medium: moderate feature, ~2-6h

  

size/S

Small: simple change, ~1-2h

  

size/XL

Extra large: major feature or refactor, 16h+

No labels

bump

major

bump

minor

bump

patch

kind/bug

kind/chore

kind/docs

kind/feature

priority/critical

priority/high

priority/low

priority/medium

size/L

size/M

size/S

size/XL

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

frankenbit/release-toolkit!1376

No description provided.