test(repin): no test runs rt repin's realBake, so a re-pin that bakes the wrong ref passes go test #1563

New issue

Open

opened 2026-10-06 11:52:21 +02:00 by claude · 0 comments

claude commented

2026-10-06 11:52:21 +02:00

Member

#1561 found this while measuring what guards the bake. rt repin's production bake step, realBake, is never run by a test. A re-pin that passed the wrong ref into the bake would pass go test and push an rc tag baked for something else.

Measured

All reads are on live origin/main 6a36090, 2026-10-06.

What realBake does. cmd/rt/repin.go:64-74. It calls selfboot.CommitBake with ToolkitRef: ref and Phase: bake.PhaseRepin, and returns the detached commit's SHA. Its only reference is the production wiring in the cobra command, repinDeps{... bake: realBake} at cmd/rt/repin.go:180.

What the tests run instead. cmd/rt/repin_test.go calls runRepin seven times (lines 182, 225, 259, 277, 311, 355, 410). Each injects stubBake (line 291), which returns a fixed SHA without touching git. The TestValidateRepin_* tests do not bake. No test runs the repin command end to end: git grep '"repin"' over *_test.go finds only the C5 exclusion entry in cmd/rt/main_test.go:276.

The injection is deliberate. The repinDeps comment (cmd/rt/repin.go:52-56) says the bake does real git work that a scripted runner cannot fake. Injecting it made the orchestration testable, but no test exercises the real step on its own.

Mutants. Each was applied to an LF clone of 6a36090 and graded with go test ./cmd/rt/ ./internal/selfboot/ ./internal/bake/. The baseline is ok on all three packages.

Mutant	Where	Result
M1: `ToolkitRef: ref` becomes `ToolkitRef: "main"`	`cmd/rt/repin.go:67`	survives, 0 failures
M4: `ToolkitRef: ref` becomes `ToolkitRef: "v0.57.0"`	`cmd/rt/repin.go:67`	survives, 0 failures
M2: `CommitBake` bakes `'main'` instead of the requested ref	`internal/selfboot/selfboot.go:74`	killed by 5 tests, e.g. `TestCommitBake_TreeCarriesRewrittenRef`
M3: the bake kernel writes `'main'` whatever ref it is given	`internal/bake/marker.go`, `rewriteMarker`	killed by 16 tests, e.g. `TestRewriteMarker_Canonical`

M2 and M3 were re-run with byte-exact LF writes and gave the same kills. So the kernel and CommitBake are pinned, and rt repin's call into them is not.

Why it matters

rt repin is dormant for toolkit-self while the wrappers float at @main (AGENTS.md §2.2). It is still the tool for a deliberate tag pin, and the first run after a long dormancy is when an untested wiring line is most likely to have drifted.

If it regressed like M1 or M4, the re-pin would push an rc tag that does not carry its own ref. Only two mechanisms would then react:

build-ref-check would refuse the tag, but only after it exists. It has never graded an rc tag on the forge (#1561).
The #1222 fallback would defuse M1's 'main' at run time. It does not catch M4's stale value, which wins as baked. That is how v0.57.1 to v0.57.4 shipped a stale reusable-recover-pending-cut.yml (#1173).

AGENTS.md §2.5 says "The bake is pinned in Go tests", naming cmd/rt/build_bake_test.go, internal/bake and internal/selfboot. That holds for the kernel and CommitBake, but not for the step rt repin actually runs.

Acceptance criteria

A test runs realBake against a scratch repository carrying the canonical reusables. It asserts that the returned SHA is a commit, that its tree carries '<rc-tag>' at every marker-carrying canonical reusable, and that its parent is the scratch repository's HEAD.
The test fails on M1 and on M4 above. The PR body shows both runs.
AGENTS.md §2.5's list of tests that pin the bake names the new test.
arc42 §8.1's statement that the Go tests "do not reach rt repin's own call into the bake" is updated. That sentence lands with #1562, so update it there or after it.
go test ./... passes, and so does the CI linter (golangci-lint).

Out of scope

An end-to-end test of the whole rt repin command, which pushes a tag and a branch and opens a PR. The orchestration tests with stubBake already cover that sequence.
A pre-push refusal in rt repin for a tag whose tree does not carry its own ref. Today the only check on the baked value is build-ref-check, after the push. Whether to refuse earlier is a separate design question.

#1561 found this while measuring what guards the bake. `rt repin`'s production bake step, `realBake`, is never run by a test. A re-pin that passed the wrong ref into the bake would pass `go test` and push an rc tag baked for something else. ## Measured All reads are on live `origin/main` `6a36090`, 2026-10-06. **What `realBake` does.** `cmd/rt/repin.go:64-74`. It calls `selfboot.CommitBake` with `ToolkitRef: ref` and `Phase: bake.PhaseRepin`, and returns the detached commit's SHA. Its only reference is the production wiring in the cobra command, `repinDeps{... bake: realBake}` at `cmd/rt/repin.go:180`. **What the tests run instead.** `cmd/rt/repin_test.go` calls `runRepin` seven times (lines 182, 225, 259, 277, 311, 355, 410). Each injects `stubBake` (line 291), which returns a fixed SHA without touching git. The `TestValidateRepin_*` tests do not bake. No test runs the `repin` command end to end: `git grep '"repin"'` over `*_test.go` finds only the C5 exclusion entry in `cmd/rt/main_test.go:276`. The injection is deliberate. The `repinDeps` comment (`cmd/rt/repin.go:52-56`) says the bake does real git work that a scripted runner cannot fake. Injecting it made the orchestration testable, but no test exercises the real step on its own. **Mutants.** Each was applied to an LF clone of `6a36090` and graded with `go test ./cmd/rt/ ./internal/selfboot/ ./internal/bake/`. The baseline is ok on all three packages. | Mutant | Where | Result | |---|---|---| | M1: `ToolkitRef: ref` becomes `ToolkitRef: "main"` | `cmd/rt/repin.go:67` | **survives**, 0 failures | | M4: `ToolkitRef: ref` becomes `ToolkitRef: "v0.57.0"` | `cmd/rt/repin.go:67` | **survives**, 0 failures | | M2: `CommitBake` bakes `'main'` instead of the requested ref | `internal/selfboot/selfboot.go:74` | killed by 5 tests, e.g. `TestCommitBake_TreeCarriesRewrittenRef` | | M3: the bake kernel writes `'main'` whatever ref it is given | `internal/bake/marker.go`, `rewriteMarker` | killed by 16 tests, e.g. `TestRewriteMarker_Canonical` | M2 and M3 were re-run with byte-exact LF writes and gave the same kills. So the kernel and `CommitBake` are pinned, and `rt repin`'s call into them is not. ## Why it matters `rt repin` is dormant for toolkit-self while the wrappers float at `@main` (AGENTS.md §2.2). It is still the tool for a deliberate tag pin, and the first run after a long dormancy is when an untested wiring line is most likely to have drifted. If it regressed like M1 or M4, the re-pin would push an rc tag that does not carry its own ref. Only two mechanisms would then react: - `build-ref-check` would refuse the tag, but only after it exists. It has never graded an rc tag on the forge (#1561). - The #1222 fallback would defuse M1's `'main'` at run time. It does not catch M4's stale value, which wins as baked. That is how `v0.57.1` to `v0.57.4` shipped a stale `reusable-recover-pending-cut.yml` (#1173). AGENTS.md §2.5 says "The bake is pinned in Go tests", naming `cmd/rt/build_bake_test.go`, `internal/bake` and `internal/selfboot`. That holds for the kernel and `CommitBake`, but not for the step `rt repin` actually runs. ## Acceptance criteria - [ ] A test runs `realBake` against a scratch repository carrying the canonical reusables. It asserts that the returned SHA is a commit, that its tree carries `'<rc-tag>'` at every marker-carrying canonical reusable, and that its parent is the scratch repository's HEAD. - [ ] The test fails on M1 and on M4 above. The PR body shows both runs. - [ ] AGENTS.md §2.5's list of tests that pin the bake names the new test. - [ ] arc42 §8.1's statement that the Go tests "do not reach `rt repin`'s own call into the bake" is updated. That sentence lands with #1562, so update it there or after it. - [ ] `go test ./...` passes, and so does the CI linter (`golangci-lint`). ## Out of scope - An end-to-end test of the whole `rt repin` command, which pushes a tag and a branch and opens a PR. The orchestration tests with `stubBake` already cover that sequence. - A pre-push refusal in `rt repin` for a tag whose tree does not carry its own ref. Today the only check on the baked value is `build-ref-check`, after the push. Whether to refuse earlier is a separate design question.

claude added the

labels

2026-10-06 11:52:21 +02:00

claude referenced this issue

2026-10-06 11:52:33 +02:00

docs(arc42): nine sites credit the bake to check-self-bootstrap, which never grades the baked value #1561

claude referenced this issue