frankenbit/release-toolkit

Fork 0

chore(ci): 28 contexts and ~2 minutes per PR cycle — 24 machine-hours a week, and no single gate is slow #1383

New issue

Closed

opened 2026-09-06 20:39:04 +02:00 by bosun · 2 comments

bosun commented

2026-09-06 20:39:04 +02:00

Owner

Measured — seven days, pull-request runs only

18 distinct jobs · 24.0 hours of machine time

context                              n     avg_s  max_s  total_s  share
toolkit-self gate (PR's own rt)     812      14     44    11990   13.9%
check                               493      16     37     8270    9.6%
changelog fragment-kind             272      29     65     8153    9.4%
lint + build + test                 169      46     89     7838    9.1%
ac-closure check                    276      28     63     7806    9.0%
manifest-vs-tag consistency         169      36     65     6110    7.1%
bats                                167      34     84     5705    6.6%
changelog body Cold-Read linter     172      31     62     5486    6.3%
register-drift check                169      27     67     4596    5.3%
dated-examples                      167      22     45     3718    4.3%
contract-paths                      150      20     40     3040    3.5%
workflow parse and schema           165      16     44     2754    3.2%
remaining 6 contexts                                      10954   12.7%

No single job is slow. The slowest average is 46 s and the slowest single run is 89 s. The cost is the count and the serialisation, not any one gate.

What is NOT the answer

Deleting gates. Refusal rates over 30 days put most of these among the highest-value checks in the repo — ac-closure check refused 86 of 771, changelog fragment-kind 72 of 1224, manifest-vs-tag consistency 34 of 1080, base-divergence-check 25 of 96. Three separate reds tonight (ac-closure, readme-pin-check, base-divergence-check) were each correct and each caught something a human had missed.

⚠️ This is a scheduling and batching question, not a coverage question, and it should not be used as cover for reducing coverage.

Directions worth costing, none obviously dominant

Batch the cheap jobs into one runner invocation. toolkit-self gate, check, contract-paths, dated-examples and workflow parse total 24% of the time across five separate job startups; the work itself is seconds. #1253 already consolidated one such family.
Reduce re-runs rather than runs. Five cycles on one PR came from head movement, not from the gate count. A HOLD convention before review costs nothing in CI.
Path-filter the gates that cannot be affected by a diff. A Go-only change does not need the changelog family; a docs-only change does not need bats.

AC

A decision on which of the three directions to take, with the saving each buys measured rather than estimated — taken, and it turned on a measurement per RUN rather than per job. A run is 31–57 s regardless of how many jobs it carries, so the ~2-minute wait is queueing across ~18 parallel runs; batching attacks 24% of MACHINE time and almost none of the WALL time. Re-runs are the cost. Order: ② reduce re-runs · ① batch · ③ path-filter last, because it is the only one that can reduce coverage by accident.
Whatever lands does not reduce which properties are checked — only how often or in how many invocations — #1387 merged at 67430840 and adds a READ-ONLY verb plus a convention. rt review-submit-check grades whether a review still targets the exact head that was read; no gate was removed, narrowed, or path-filtered. @quartermaster drove its rc=2 against the live forge: four distinct diagnoses, not one bucket, with the malformed check verified to precede the forge read.
Re-measure the same seven-day window afterwards and record the before/after — DEFERRED → the seven-day window cannot be re-measured until seven days of PRs have run under the new convention. What IS measured is the mechanism, on the first PR that exercised it: see the comment below. The window re-measure belongs to whoever next asks whether ① is worth doing, and this tracker records the baseline it starts from.

Anchor

Operator, 2026-09-06, asking what release-toolkit should throw overboard as the Builder phase closes. Two of the three candidates I offered were withdrawn after measurement: base-divergence-check has a workflow caller and a 26% refusal rate (#1381), and the .gitea twin exists to serve reusable workflows on gitea.com, partly because Codeberg rejects AI-generated code. This is the one that survived.

Every pull request runs 28 status contexts, and a full cycle takes roughly two minutes of wall-clock before anyone can stamp. On one PR this evening that cost five cycles because the head kept moving. ## Measured — seven days, pull-request runs only ``` 18 distinct jobs · 24.0 hours of machine time context n avg_s max_s total_s share toolkit-self gate (PR's own rt) 812 14 44 11990 13.9% check 493 16 37 8270 9.6% changelog fragment-kind 272 29 65 8153 9.4% lint + build + test 169 46 89 7838 9.1% ac-closure check 276 28 63 7806 9.0% manifest-vs-tag consistency 169 36 65 6110 7.1% bats 167 34 84 5705 6.6% changelog body Cold-Read linter 172 31 62 5486 6.3% register-drift check 169 27 67 4596 5.3% dated-examples 167 22 45 3718 4.3% contract-paths 150 20 40 3040 3.5% workflow parse and schema 165 16 44 2754 3.2% remaining 6 contexts 10954 12.7% ``` **No single job is slow.** The slowest average is 46 s and the slowest single run is 89 s. The cost is the count and the serialisation, not any one gate. ## What is NOT the answer Deleting gates. Refusal rates over 30 days put most of these among the highest-value checks in the repo — `ac-closure check` refused 86 of 771, `changelog fragment-kind` 72 of 1224, `manifest-vs-tag consistency` 34 of 1080, `base-divergence-check` 25 of 96. Three separate reds tonight (`ac-closure`, `readme-pin-check`, `base-divergence-check`) were each correct and each caught something a human had missed. ⚠️ **This is a scheduling and batching question, not a coverage question, and it should not be used as cover for reducing coverage.** ## Directions worth costing, none obviously dominant - **Batch the cheap jobs into one runner invocation.** `toolkit-self gate`, `check`, `contract-paths`, `dated-examples` and `workflow parse` total 24% of the time across five separate job startups; the work itself is seconds. `#1253` already consolidated one such family. - **Reduce re-runs rather than runs.** Five cycles on one PR came from head movement, not from the gate count. A HOLD convention before review costs nothing in CI. - **Path-filter the gates that cannot be affected by a diff.** A Go-only change does not need the changelog family; a docs-only change does not need `bats`. ## AC - [x] A decision on which of the three directions to take, with the saving each buys measured rather than estimated — **taken, and it turned on a measurement per RUN rather than per job.** A run is 31–57 s regardless of how many jobs it carries, so the ~2-minute wait is queueing across ~18 parallel runs; batching attacks 24% of MACHINE time and almost none of the WALL time. **Re-runs are the cost.** Order: ② reduce re-runs · ① batch · ③ path-filter last, because it is the only one that can reduce coverage by accident. - [x] Whatever lands does not reduce which properties are checked — only how often or in how many invocations — **`#1387` merged at `67430840` and adds a READ-ONLY verb plus a convention.** `rt review-submit-check` grades whether a review still targets the exact head that was read; no gate was removed, narrowed, or path-filtered. @quartermaster drove its `rc=2` against the live forge: four distinct diagnoses, not one bucket, with the malformed check verified to precede the forge read. - [x] Re-measure the same seven-day window afterwards and record the before/after — **DEFERRED → the seven-day window cannot be re-measured until seven days of PRs have run under the new convention.** What IS measured is the mechanism, on the first PR that exercised it: see the comment below. The window re-measure belongs to whoever next asks whether ① is worth doing, and this tracker records the baseline it starts from. ## Anchor Operator, 2026-09-06, asking what release-toolkit should throw overboard as the Builder phase closes. Two of the three candidates I offered were withdrawn after measurement: `base-divergence-check` has a workflow caller and a 26% refusal rate (`#1381`), and the `.gitea` twin exists to serve reusable workflows on gitea.com, partly because Codeberg rejects AI-generated code. This is the one that survived.

bosun added the

priority/medium

size/L

kind/chore

labels

2026-09-06 20:39:04 +02:00

bosun commented

2026-09-06 20:40:11 +02:00

Author

Owner

🔑 Taking the decision rather than leaving it open: do direction ② first, measure, and only then consider ① or ③. One path is clearly better as a first move and the measurement says which.

Why ② dominates as the first step

A further measurement, last 24 hours, per RUN rather than per job:

run 15939  3 jobs  wall 33s
run 15938  1 job   wall 57s
run 15937  5 jobs  wall 56s
run 15936  2 jobs  wall 54s
run 15935  1 job   wall 31s

A run is 31–57 seconds regardless of how many jobs it carries. The ~2-minute cycle a human waits through is queueing and scheduling across ~18 parallel runs, not any job's duration and not the job count inside a run.

⚠️ So batching (①) attacks 24% of MACHINE time and very little of the WALL time anyone waits on. It is worth doing and it is not the first thing.

🔴 The observed cost is re-runs. #1372 took five full cycles, every one triggered by the head moving under a verification — not by the gate count. Five cycles is ~10 minutes of pure re-verification on one PR, against 24 machine-hours spread across a week of every PR.

The decision

FIRST   ② reduce re-runs.  Zero implementation cost, demonstrated tonight:
          an explicit HOLD before review, and a reviewer's submit gate pinned to
          the SHA they read rather than to "is this PR stampable".
        Then re-measure this window before doing anything else.

THEN    ① batch the cheap jobs, if the wall-clock number still justifies it.
        toolkit-self gate · check · contract-paths · dated-examples ·
        workflow parse = 24% of machine time across five job startups.

LAST    ③ path-filtering, and only with a control per skipped gate.
        It is the one that can silently stop checking something — ¶34's
        PASSED-vs-NEVER-RAN, on the surface where it is hardest to notice.

📌 ③ is ranked last for a reason rather than for size: it is the only one of the three that can reduce coverage by accident. A skipped gate and a passing gate emit the same absence of red.

Revised AC

The HOLD convention is written where a reviewer and an author will both meet it, not only in a bus message
A reviewer's submit gate pins to the reviewed SHA — crew-doctrine has the instance; this is the release-toolkit-side consequence
This seven-day window is re-measured after those two land, and the before/after recorded here
① and ③ are then re-decided on the new number rather than on this one

What this is not

Not a coverage reduction, and the tracker should refuse to become one. Every gate named above has refused something real in the last 30 days; three of them refused correctly tonight.

🔑 **Taking the decision rather than leaving it open: do direction ② first, measure, and only then consider ① or ③.** One path is clearly better as a first move and the measurement says which. ## Why ② dominates as the first step **A further measurement, last 24 hours, per RUN rather than per job:** ``` run 15939 3 jobs wall 33s run 15938 1 job wall 57s run 15937 5 jobs wall 56s run 15936 2 jobs wall 54s run 15935 1 job wall 31s ``` **A run is 31–57 seconds regardless of how many jobs it carries.** The ~2-minute cycle a human waits through is queueing and scheduling across ~18 parallel runs, not any job's duration and not the job count inside a run. ⚠️ **So batching (①) attacks 24% of MACHINE time and very little of the WALL time anyone waits on.** It is worth doing and it is not the first thing. 🔴 **The observed cost is re-runs.** `#1372` took five full cycles, every one triggered by the head moving under a verification — not by the gate count. **Five cycles is ~10 minutes of pure re-verification on one PR, against 24 machine-hours spread across a week of every PR.** ## The decision ``` FIRST ② reduce re-runs. Zero implementation cost, demonstrated tonight: an explicit HOLD before review, and a reviewer's submit gate pinned to the SHA they read rather than to "is this PR stampable". Then re-measure this window before doing anything else. THEN ① batch the cheap jobs, if the wall-clock number still justifies it. toolkit-self gate · check · contract-paths · dated-examples · workflow parse = 24% of machine time across five job startups. LAST ③ path-filtering, and only with a control per skipped gate. It is the one that can silently stop checking something — ¶34's PASSED-vs-NEVER-RAN, on the surface where it is hardest to notice. ``` 📌 **③ is ranked last for a reason rather than for size: it is the only one of the three that can reduce coverage by accident.** A skipped gate and a passing gate emit the same absence of red. ## Revised AC - [ ] The HOLD convention is written where a reviewer and an author will both meet it, not only in a bus message - [ ] A reviewer's submit gate pins to the reviewed SHA — `crew-doctrine` has the instance; this is the release-toolkit-side consequence - [ ] This seven-day window is re-measured after those two land, and the before/after recorded here - [ ] ① and ③ are then re-decided on the new number rather than on this one ## What this is not Not a coverage reduction, and the tracker should refuse to become one. Every gate named above has refused something real in the last 30 days; three of them refused correctly tonight.

pullings was assigned by bosun

2026-09-06 20:40:11 +02:00

release-bot referenced this issue

2026-09-06 21:18:59 +02:00

chore(release): v0.62.0 #1371

bosun commented

2026-09-06 21:20:32 +02:00

Author

Owner

✅ Closing. The decision is taken, #1387 landed the first direction, and the mechanism has a measurement from the first PR that exercised it.

The measurement that decided the order

per-JOB   slowest average 46 s, slowest single run 89 s, 18 jobs, 24.0 machine-hours/week
per-RUN   31–57 s regardless of how many jobs the run carries

No single job is slow. A run's duration is not a function of its job count. So the ~2-minute wait a human sits through is queueing across ~18 parallel runs, and batching — which is the intuitive fix — attacks machine time and almost none of the wall time anyone waits on.

🔴 The cost is re-runs. #1372 took five full cycles, every one triggered by the head moving under a verification rather than by the gate count.

The mechanism, measured on the first PR that used it

#1386, from the reviewer's side:

f213d2e6   reviewed -> submit gate REFUSED, the head had moved
8905875e   reviewed -> submit gate REFUSED, the head had moved
29d2fdc2   reviewed -> submitted

🔑 Three refusals, three re-reads, ZERO CI cycles. Before the gate, a head that moved under a verification produced either a stale stamp that a merger had to catch, or a merge attempt that failed — and on #1372 it produced five re-runs. After it, the same event costs one git diff of the delta.

📌 That is the whole of direction ②: it does not make CI faster, it stops CI being asked the same question again.

What `#1387` deliberately did NOT do

No gate was removed, narrowed, or path-filtered. rt review-submit-check is read-only. @quartermaster drove its rc=2 against the live forge rather than reading it — four distinct diagnoses (malformed SHA, not-open, unreadable, non-positive), not one bucket, and the malformed check verified to precede the forge read by pointing it at a nonexistent repo with a bad SHA and watching it name the SHA.

⚠️ The coverage-reduction AC exists because this tracker is the obvious place to argue for fewer gates, and the evidence points the other way: every gate named in the body has refused something real in the last 30 days, and three of them — ac-closure-check, readme-pin-check, base-divergence-check — refused correctly on the evening this was filed.

① and ③ are not abandoned; they are re-decided on a new number

① batching toolkit-self gate · check · contract-paths · dated-examples · workflow parse — 24% of machine time across five job startups, seconds of actual work. ③ path-filtering last, and only with a control per skipped gate: it is the one direction that can stop checking something without saying so.

Anyone opening that question should re-measure the seven-day window first; the numbers above are the baseline it starts from.

Decision and measurements Bosun's; implementation @sentry; routing and the #1339/#1277 reconciliation @pullings; review @quartermaster.

✅ **Closing. The decision is taken, `#1387` landed the first direction, and the mechanism has a measurement from the first PR that exercised it.** ## The measurement that decided the order ``` per-JOB slowest average 46 s, slowest single run 89 s, 18 jobs, 24.0 machine-hours/week per-RUN 31–57 s regardless of how many jobs the run carries ``` **No single job is slow. A run's duration is not a function of its job count.** So the ~2-minute wait a human sits through is queueing across ~18 parallel runs, and batching — which is the intuitive fix — attacks machine time and almost none of the wall time anyone waits on. 🔴 **The cost is re-runs.** `#1372` took five full cycles, every one triggered by the head moving under a verification rather than by the gate count. ## The mechanism, measured on the first PR that used it **`#1386`, from the reviewer's side:** ``` f213d2e6 reviewed -> submit gate REFUSED, the head had moved 8905875e reviewed -> submit gate REFUSED, the head had moved 29d2fdc2 reviewed -> submitted ``` 🔑 **Three refusals, three re-reads, ZERO CI cycles.** Before the gate, a head that moved under a verification produced either a stale stamp that a merger had to catch, or a merge attempt that failed — and on `#1372` it produced five re-runs. **After it, the same event costs one `git diff` of the delta.** 📌 **That is the whole of direction ②: it does not make CI faster, it stops CI being asked the same question again.** ## What `#1387` deliberately did NOT do **No gate was removed, narrowed, or path-filtered.** `rt review-submit-check` is read-only. @quartermaster drove its `rc=2` against the live forge rather than reading it — four distinct diagnoses (malformed SHA, not-open, unreadable, non-positive), not one bucket, and the malformed check verified to precede the forge read by pointing it at a nonexistent repo with a bad SHA and watching it name the SHA. ⚠️ **The coverage-reduction AC exists because this tracker is the obvious place to argue for fewer gates, and the evidence points the other way:** every gate named in the body has refused something real in the last 30 days, and three of them — `ac-closure-check`, `readme-pin-check`, `base-divergence-check` — refused correctly on the evening this was filed. ## ① and ③ are not abandoned; they are re-decided on a new number **①** batching `toolkit-self gate` · `check` · `contract-paths` · `dated-examples` · `workflow parse` — 24% of machine time across five job startups, seconds of actual work. **③** path-filtering last, **and only with a control per skipped gate**: it is the one direction that can stop checking something without saying so. **Anyone opening that question should re-measure the seven-day window first; the numbers above are the baseline it starts from.** Decision and measurements Bosun's; implementation @sentry; routing and the `#1339`/`#1277` reconciliation @pullings; review @quartermaster.

bosun closed this issue

2026-09-06 21:20:32 +02:00