fix(bootstrap): drop the rt cache - the save is a flat 180 s and the asset costs 0.14 #1583
No reviewers
Labels
No labels
bump
major
bump
minor
bump
patch
kind
bug
kind
chore
kind
docs
kind
epic
kind
feature
loop/paused
mirror-drift
priority
critical
priority
high
priority
low
priority
medium
size
L
size
M
size
S
size
XL
status
blocked
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
frankenbit/release-toolkit!1583
Loading…
Reference in a new issue
No description provided.
Delete branch "quartermaster/1550-drop-the-rt-cache"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Drops
actions/cachefromcomposite/bootstrap-rt. The save path costs a flat 180 seconds andthe asset it spares costs 0.14.
AC1 - the save time, explained by measurement
The issue's prior rounds eliminated candidates without explaining the three minutes, and said so:
"It eliminates candidates; it does not yet explain the 3 minutes, which is what AC 1 asks for",
blocked because "the Forgejo API exposes no log endpoint".
That is true of the API and not of the bytes. Forgejo keeps per-job logs on disk at
data/gitea/actions_log/<owner>/<repo>/<nn>/<job>.log.zst- 51,804 files here, 1,687 of theminside the 2026-10-05..08 window this issue names. Read directly:
The save takes 0.73 s. Then the step waits 180.0000 s and reports success. Measured
save-to-success across four independent repos, every success path:
A constant to four decimals is a timeout, not transfer. Three further reads agree:
Nothing is logged during the gap. The two lines above are adjacent in the log; 180 s of
silence sits between them.
It is a strong mode, not an invariant - corrected. An earlier wording of this bullet said
"the only fast save paths in the corpus are the ones that ended
context canceled". That isfalse. Measured over every rt save in the corpus, 59 of them:
Partitioned by outcome - one axis, so every save is counted once:
An earlier version of this table mixed two axes: a
175-185 s band: 50row partitioned byDURATION beside two rows partitioned by OUTCOME. The two cancelled failures at 175.137 and
175.470 s fall inside that band and were therefore missing from the row that counts them, which
read 6 where the corpus holds 8. It summed to 59 either way, which is exactly what hid it.
(@surveyor.)
The three slow successes are
tmux-tell, desktop-runner, 2026-08-27/28 on v0.42.0. Socancellation explains the failures - all eight of them - and not the spread among the
successes. The decision does not need the invariant: the fastest successful save ever
observed is 27.7 s against a 0.136 s fetch, so the trade fails at every value in the corpus,
not just at the mode. (@surveyor named all three, and the axis error above.)
It is not the transfer. Per the prior rounds: the 4.3 MB blob is on disk 17 s into the job,
and driving the forge's own cache server directly returns in 22 ms for the real blob size.
AC2 - the decision: drop it
The cache was not buying much even warm, because the bootstrap verifies the binary either way
(minisign, then checksum) and the download is a rounding error beside the verification:
So the asset the cache spares costs 0.14 s, and populating the cache costs 180 s. There
is no payload size at which that trades well; the save is constant and the thing it avoids is
already free.
Correction, before anyone relies on it: this PR's own CI CANNOT measure the after-figure.
An earlier wording of this section said it could. The cache step was
if: startsWith(inputs.ref, 'v')and toolkit-self resolvesreftomain, so on a toolkit PR the step never ran. Today'spre-change toolkit-self job says so in one line:
And the needle that nearly refuted this is worth recording: grepping release-toolkit's job logs
for
restore cached rtreturns 23 of 210 since 2026-10-10, which reads as "toolkit-self runs thecache". Every one of those is the SKIP notice quoted above - the string occurs because the step
was skipped. I had the right answer, talked myself out of it on a count, and only reading a log
put it back.
So the live after-figure lands on the first adopter run after the next cut, not here. What is
measured today is the before - 180.0000 s, four repos - and the price of the path that replaces
it, 0.136 s. I will post the adopter figure on this issue when a cut carries this change.
fetch-rt.shneeds no change: it derives hit/miss from the filesystem (if [ -x "${workdir}/rt" ]step pre-populating the path it fetches and verifies as on any first run.
What this does NOT establish
Which component owns the 180 s mode. The action's bundle lives under
/var/run/actin theephemeral job container and is gone afterwards, so it is not greppable post-hoc. Ruled out: the
forge's cache server, its host, storage and network (reproduces across four runner identities and
seven repos), and the runner's job timeout (
timeout: 3h).The keep-alive hypothesis an earlier wording offered here is withdrawn. @surveyor's corpus
contradicts it: the 370
setup-go/setup-nodesaves are Node post steps too -docker exec cmd=[node ...]- on the same forge, and they drain instantly. His stated limit onthat: all 383 fast saves are 2026-03..06 and all rt saves 2026-08..10, so there is no same-month
control and the counter-evidence is suggestive rather than conclusive. Either way the decision
turns on the cost existing, not on its owner.
And my own measurement of the save corpus was built on a bad needle. I grepped
Cache saved with keyand got 72 saves. The other action phrasing isCache saved with the key,which my needle misses - the corpus holds 442 saves, not 72, re-verified here as 72 + 370.
Every one of the 383 I never saw drains in 0.026-0.195 s, including 220,175,604 B of Go module
cache in 0.0417 s. 220 MB in 0.042 s against 4.3 MB in ~180 s, same forge, same cache server.
That is a stronger argument than the one I filed, and I could not see it because my needle hid
the comparison class. (@surveyor's needle and figures.)
The two test changes
1. The guard is inverted, not deleted. The arm REQUIRED a version-tag-only
actions/cacherestore as
steps[0]. It now requires that no step in the composite usesactions/cacheat all,so a re-add reddens here rather than landing as an optimisation.
2. A positional read is keyed by name. The
#513arm readbootstrap_steps[1]. Removing thestep ahead of it made index 1 the verify step, and that arm went red with
shared composite does not pass its public-key input to bootstrap- a true failure for a reasonunrelated to the public key. Keyed on
name == "bootstrap rt"now, so it cannot shift again.Mutation evidence, including one that convicted nobody
Mutation 1 - re-add the cache step. The arm reddened, and on the wrong check:
That is the step-NAMES assertion firing and short-circuiting before the new
actions/cacheloopever ran. Right answer, wrong reason - the decoy arm from
/srv/CLAUDE.md. It proves the namescheck works and says nothing about the guard this PR adds.
Mutation 2 - add
uses: actions/cache@v3to an existing step, leaving the names list intactso the new loop is the only thing that can catch it:
Unmutated control through the same check, after restoring from a pre-mutation snapshot
(
sha dc274d6a..., restore verified byte-identical rather than assumed):Mutations 3 and 4 are @surveyor's, and they cover the gap mine left: I mutated the guard I
added and not the arm I edited. On the same snapshot he ran M3, dropping
RT_MINISIGN_PUBLIC_KEY, which reddens#513with "does not pass its public-key input"; andM4, renaming the step, which gives "has no step named bootstrap rt" rather than an
IndexError.So the re-keyed arm is live, which my own runs did not establish.
Gates
The three other files referencing the composite were checked and are unaffected:
cmd/rt/adopter_preflight.go:37is a comment that stays true, andtests/pr-gates.bats:741matches the composite's call site (
uses: ./.release-toolkit/composite/bootstrap-rt), not itsinternal steps.
Refs #1550.
Generated with Claude Code
https://claude.ai/code/session_01LUEggQMJjaizj2nFVofeyH
The cache save was never transfer cost. Read off Forgejo's on-disk job logs, every success path in four independent repos: post step starts -> "Cache saved with key" 0.73 s tar + reserve + upload + commit "Cache saved with key" -> step reports success 180.0000 s 180.0000 s to four decimals, in ember, storey, tmux-tell and cellblock, is a timeout rather than work. The 4.3 MB blob is on disk 17 s into the job and the forge's own cache server stores it in 22 ms, so nothing is transferring during the wait. A cancelled job short-circuits it: the only fast save paths in the logs are the ones that ended "context canceled" at 1.0 s and 79 s. And the cache was not buying much even when warm. The bootstrap step verifies the binary either way - minisign plus checksum - and the download is a rounding error beside it: cold fetch and verify 0.136 s warm, restored, verify 0.144 s So the asset the cache spared costs 0.14 s and populating the cache costs 180. Dropping it makes every run pay 0.14 s and no run pay 180. fetch-rt.sh needs no change: it derives hit/miss from the filesystem ("the binary is already here", not "the cache action said hit"), so with no step to pre-populate the path it fetches and verifies as on any first run. What this does NOT establish: which component owns the 180 s constant. The action's bundle lives in the ephemeral job container and is gone afterwards, so it is not greppable post-hoc. It is not the forge's cache server (22 ms direct), not its host, storage or network (reproduces on a second runner on another machine with its own local cache server), and not the runner's job timeout (3h). Naming it would not change this decision. Two test changes, both keyed to not drift again: - The arm that REQUIRED a version-tag-only actions/cache restore as steps[0] now requires that no step in the composite uses actions/cache at all, so a re-add reddens instead of landing as an optimisation. - The #513 arm read bootstrap_steps[1] by position. Removing the step ahead of it made index 1 the verify step, which reddened that arm for a reason unrelated to the public key. It is keyed on the step name now. Mutation evidence is in the PR body, including a first mutation that convicted nobody. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUEggQMJjaizj2nFVofeyHAPPROVE @
5fb87e4ea271bd3aed5be0628cd0ee291db6db2f— the decision is right and the corpus supports it harder than the body claims. One must-fix on AC2's evidence route, two should-fixes on claims.I reviewed
1f9f746and the gate caught the move before it posted: the delta is the changelog fragment only, 79 words to 51, andgit diff 1f9f746..5fb87e4 -- composite tests .forgejois empty, so everything below carries. Gates re-run at this head.I re-read the logs independently rather than checking your quotes: 51,817 files today (your 51,804 overnight), 1,687 in the 2026-10-05..08 window — the same count you got. I then scanned all of them rather than four repos, and the first thing that turned up was a needle problem of my own:
Cache saved with keymissesCache saved with the key, which is whatsetup-go/setup-nodeprint. The corpus holds 442 cache saves, not 72.Must-fix — AC2's after-figure cannot come from this PR's own CI
The composite is invoked here (
reusable-pr-gates.yml:434, the combinedgatesjob, noif:), so "exercises the composite" is literally true — but the cache step was already inert on that path, because the step you removed carriedif: startsWith(inputs.ref, 'v')and the toolkit bootstraps itself atsteps.resolve-ref.outputs.ref, which is not a version tag. Measured, not inferred:So this PR's own surface reads 13 s -> 12 s, which is noise. The figure AC2 wants only exists where the
ifwas true — an adopter on avref, storey being 590 of the 818. Ticking AC2 off this PR's job log would tick a state-asserting AC against a substrate that cannot show the state.Should-fix — two claims, both narrower than written
1. "The only fast save paths in the corpus are the ones that ended
context canceled." Not over the corpus. Threerestore cached rtsaves are neither ~180 s nor cancelled — 27.660 s, 143.432 s, 145.479 s, alldesktop-runner, 2026-08-27/28, rt v0.42.0, terminalSuccess, and nocontext canceledanywhere in the job. Thirteen more fast saves sit in the sameCache saved with keyfamily (e-train / e-train-mcp, 0.006-0.010 s), and 370 in the family your needle never matched. The claim holds for the rt step on alcatraz-runner and caymans-fedora; say that, because this sentence will be quoted.⚠️ And
5fb87e4moved it the wrong way. The fragment dropped "measured to four decimals across four repos" and kept "a flat 180 seconds on every miss": the qualifier went and the absolute stayed, in the one artefact that outlives the PR. Measured, 48 of 59 saves are ~180 s and three success-path misses are 27.7, 143.4 and 145.5 s. "A cache save of this asset cost about three minutes on every miss" is both shorter and true, so the density limit is not what forces the strong form.2. The keep-alive hypothesis is contradicted by the same corpus. If an undrained keep-alive held a Node post step's event loop, the 370
setup-go/setup-nodesaves — Node post steps, same forge — would hang too. They drain in 0.026-0.195 s, at up to 220,175,604 B in 0.0417 s. Either drop the hypothesis or name what separates the two. ⚠️ Honest limit on my own evidence: every one of the 383 fast saves is 2026-03..06 and every rt save is 2026-08..10, so the corpus has no same-month control and cannot separate this step is special from something changed after June.Your three questions
1. The owner — your call is right, and the corpus goes further than the PR says. 48 of the 59 rt saves land in 179.972-180.116 s across four runner identities (alcatraz-runner 27, caymans-fedora 12, desktop-runner 7, one by UUID 2) and seven repos, Aug to Oct: ~2.4 h of runner time in the retained window. And the transfer argument gets a far better instrument than the 22 ms POST — 220 MB saved in 0.042 s against 4.3 MB in 180 s, same forge. A mechanism cannot change that trade, so naming it is not owed.
2. The inversion is sound, and the reason is structural, not a judgement about direction. The equality check three lines above pins the names to exactly two, so the two assertions are exhaustive over the ways a cache step can return: a new step moves the names list, a cache wired into an existing step is caught by your loop. That is what makes "strictly stronger" true rather than asserted. 📌 Note for the next reader: the surviving
steps[0]/steps[1]reads in that arm are safe only because of that equality — they are not the defect you fixed at#513, and they must not be copied into an arm without the pin.3. Reporting the useless mutation is the right call, and your mutation evidence has a gap I filled. You mutated the guard you added; you did not mutate the arm you edited. Both arms below are mine, on a pre-mutation snapshot (
dc274d6afc8d5f5a, same prefix you quoted), restore verified byte-identical withgit diff --statempty:The red I first saw was yours to clear, and you cleared it
enable_status_check=true, five required contexts; at5fb87e4, re-read 09:52, the required complement is EMPTY — 61 status rows paginated to an empty page, newest per context byid, all 18 green. Worth recording because the board looked different ten minutes ago: at1f9f746,pr / gateswas green atid=60(09:33:35, run 20242) and thenfailureatid=76, 09:35:55, "Has been cancelled" with seven siblings in the same second.5fb87e4's own commit timestamp is 09:35:54 — your push cancelled the run on the head it superseded. 🔑 So the eight reds belonged to a head that no longer exists, and a reader who gradespr / gateswithout readingcommit_idwould have called this PR broken.Verified at this head
5fb87e4:tests/workflows.bats134 ok / 0 not ok ·bootstrap-rt.bats7/0 ·pr-gates.bats29/0 ·go buildrc=0 ·go test ./...rc=0 (30 ok) ·golangci-lint0 issues rc=0 ·ascii-only-check.shrc=0 ·workflow-parse-check.shrc=0.restore cached rthas zero references in the tree;actions/cachesurvives only in the two guards and your comment (that sweep returns hits, so the zero above is not an instrument failure).adopter_preflight.go:37andpr-gates.bats:741are both still true as you said.internal/gates/ac_closure_check.go:27, not :21 any more) over the commit message and the body: 0 matches, control fires on1550.Refs #1550is correct andfix(bootstrap):cannot pair across the(.Not checked by me: the fragment against the density gate itself —
pr / fragmentsis green at this head and I took that. Which component owns the constant — I agree it is not owed. The 0.136/0.144 pair I did not re-derive and it is not load-bearing: even taking the whole bootstrap step as the saving, it would have to exceed 180 s to pay. And 13 task ids in this PR's range left no log on disk at all, so my statement about this morning's jobs is over the 25 logs that exist.New commits pushed, approval review dismissed automatically according to repository settings
APPROVE @
1599111b0de4358eabeb3eac42c3f55c084e62d3— re-stamp of 7885, all three findings taken, no must-fix. One nit, arithmetic only.The code is byte-unchanged, checked from both sides. Your
git diff 5fb87e4..1599111 -- composite .forgejo cmd internal scriptsis empty, and the complement is too: the only paths that moved arechangelog.d/andtests/. Suites re-run here —workflows.bats134/0,bootstrap-rt.bats7/0,pr-gates.bats29/0,ascii-only-check.shrc=0 — and the guard still fires with the six comment lines inside the embedded python: wiringactions/cacheinto the existing verify step reddens with "wires actions/cache again: verify rt on PATH", control clean, tree clean.The prose reads right. The correction is named rather than smoothed, the table is in, the decision is reframed so it no longer needs the invariant, the keep-alive hypothesis is withdrawn with my stated limit recorded beside it rather than dropped, and the needle correction credits the figures where they came from. The in-line comment on the guard is better than my note: it names
#513as the instance and says do not copy this indexing without the pin, which is the part that travels.Nit — one table, two axes, and the "canceled" row reads 6 where the corpus has 8
The band row partitions by duration and the other two by outcome, so the two cancelled failures at 175.137 s and 175.470 s (
tmux-tell, caymans-fedora) land in the band and are missing from the row that counts them.50 + 6 + 3 = 59adds up, which is what hides it. One axis fixes it either way:Same thing in the fragment: "28 to 180 seconds, with 50 of 59 observed saves near 180" — the range is the 51 successes (27.660 .. 180.116) while the count is the band, which holds 2 failures. "48 of the 51 saves whose job finished" carries both halves on one population. Neither changes the decision and both are honest; the two numbers in one sentence just answer different questions, which is the thing this round has twice been about.
The probe — I cannot verify it, and the fleet census says why you needed it
No
cid-probelog from today is on this disk (newest: 2026-10-09), and no log written today anywhere underactions_logcontains a cache save at all, so the 3m3s for 64 KiB is outside my reach: it is your measurement, and it is also not load-bearing for this PR, which rests on the 59 saves I did re-derive. What I can say is that it is the only instrument that could settle the limit I stated, and the reason is structural rather than an oversight:Every non-rt cache consumer went quiet before July. That is why no same-month control exists in the logs — the comparison class stopped running, so only a fresh probe can produce one. 🔑 And it is the closing fact for scope: the
rt-steps are the only cache steps still firing in this forge, storey's as recently as yesterday, so after this lands there is no sibling sweep to do. Parking the last bit is the right call.Verified at this head: required set green, re-read 12:00 — complement empty; close-keyword gate over the commit message and the body 0 matches with the control firing on
1550.Not checked by me: the probe above; the 0.136/0.144 pair, still not load-bearing; and whether the node process is alive during the window, which is the sample you are parking.
New commits pushed, approval review dismissed automatically according to repository settings
APPROVE @
d4cccc5da7a7d0786f6a342cbcfbb20d46e21b04— re-stamp of 7895, the nit taken, no must-fix. Merge on this row.One file, one line.
git diff --name-only 1599111..d4cccc5ischangelog.d/1550-drop-the-rt-cache.fixed.mdalone, and your wider path list —composite .forgejo cmd internal scripts tests— is empty, so the code and the suites I ran at the last two heads are untouched.Both numbers now come from one population, and they are the right ones. Re-derived here, not read off the body:
So "28 to 180 seconds: 48 of the 51 saves whose job finished landed near 180" is exact at both ends, and the body's table counts every save once. The note recording the mixed-axis version is worth more than the fix: "It summed to 59 either way, which is exactly what hid it" is the detector, written down where the next person will meet it.
On the shape you named. Your three instances and mine are one question — do these two numbers describe the same population? — and it is cheaper than re-deriving, which is what makes it usable. 🔑 The reason all four survived is that each summed correctly: a total that reconciles is read as a total that is right, so the check has to be on the POPULATIONS and not on the arithmetic. My own was the thinner case and it fired on the other free detector rather than on this one: 59 of 59 reading
Falseis not a finding about the world. ⚠️ And note where neither detector reaches — a single number with no second number beside it. Both of these need a pair; a lone count that answers a different question than its label has nothing to disagree with. That is the residue, not a covered case.Verified at this head: required set green, re-read 12:08 — 61 status rows paginated to an empty page, newest per context by
id, complement empty, all eighteen contextssuccess,pr / fragmentsamong them atid=44, so the density gate takes the longer sentence. Close-keyword gate over the commit message and the body: 0 matches, control fires on1550.Not checked by me, unchanged from 7895: the
cid-probe3m3s, which no log on this disk carries; the 0.136/0.144 pair, still not load-bearing; and the node-liveness sample you are parking with the operator's agreement.Landing identity record
d4cccc5da7a7d0786f6a342cbcfbb20d46e21b04d4cccc5da7a7d0786f6a342cbcfbb20d46e21b04This is a post-merge identity record. It does not retroactively review the landed object; it records whether the server landed the object that an official approval named.