ci(gates): the landing-tree PASS names the base it graded #1340

Merged

bosun merged 2 commits from i/1195-ac2-pass-names-the-base into main 2026-09-06 16:36:57 +02:00

Conversation 4 Commits 2 Files changed 29 +223 -62

shipwright commented 2026-09-06 15:59:25 +02:00

Owner

Copy link

A green said the landing tree was green. It did not say which base that tree was built on.

Intended-targets: #1195

AC2 — the note goes in the PASS line, not in docs/

landing-tree replay: PASS — replayed onto origin/main at <sha> (N behind).
                     Every gate below grades THAT tree.
landing-tree replay: does NOT re-run when the base moves. A cut landing after
                     this point re-diverges it — compare <sha> against the base
                     tip before merging.

Mirrors #1307 on base-divergence-check. Per §Mechanism design, a scope note that cannot change an exit status is decoration and one nested in a success message is worse — so it lives in the line a reviewer actually reads. The landing tree is only the landing tree at the moment it was computed, and now the message says so.

Exercised against a real diverged repo with the step body extracted from the workflow rather than hand-copied, so what was tested is what ships. Both paths — replayed, and already-an-ancestor — print the SHA and the disclaimer.

🔴 The first version of the arm was INERT, and the control caught it

The needle was "base_sha" not in body. base_sha occurs five times in the step, so removing the capture still matched the four uses — a mutation that gutted the feature left the arm green.

🔑 A needle satisfied by a different occurrence of itself proves nothing. Split into two specific needles, and all three controls now redden separately:

break the capture      -> "PASS does not CAPTURE the base SHA"
break the print        -> "PASS does not PRINT the base SHA"
break the disclaimer   -> "does not disclose the residual window"

⚠️ Same class as the "rt " substring in #1135 — a needle matching more than it means. Second one today, and both were found by running the mutation rather than reading the check.

📌 I also nearly missed it twice over: the first control run passed because I had piped the mutation through 2>/dev/null, which hid an anchor assertion. The silenced-sweep row, on my own harness.

Verification

go build rc=0 · go test ./... rc=0 · golangci-lint rc=0 clean cache · register-check rc=0 · gitea-twin --check rc=0 · bats 119 arms, 0 not-ok. Rebased onto current main.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DbnWrAAh3iGuPAQF53nuXG

A green said the landing tree was green. It did not say which base that tree was built on. Intended-targets: #1195 ## AC2 — the note goes in the PASS line, not in `docs/` ``` landing-tree replay: PASS — replayed onto origin/main at <sha> (N behind). Every gate below grades THAT tree. landing-tree replay: does NOT re-run when the base moves. A cut landing after this point re-diverges it — compare <sha> against the base tip before merging. ``` **Mirrors `#1307` on `base-divergence-check`.** Per §Mechanism design, a scope note that cannot change an exit status is decoration and one nested in a success message is worse — so it lives in the line a reviewer actually reads. **The landing tree is only the landing tree at the moment it was computed, and now the message says so.** Exercised against a real diverged repo with the step body **extracted from the workflow rather than hand-copied**, so what was tested is what ships. Both paths — replayed, and already-an-ancestor — print the SHA and the disclaimer. ## 🔴 The first version of the arm was INERT, and the control caught it The needle was `"base_sha" not in body`. **`base_sha` occurs five times in the step**, so removing the *capture* still matched the four *uses* — a mutation that gutted the feature left the arm green. 🔑 **A needle satisfied by a different occurrence of itself proves nothing.** Split into two specific needles, and all three controls now redden separately: ``` break the capture -> "PASS does not CAPTURE the base SHA" break the print -> "PASS does not PRINT the base SHA" break the disclaimer -> "does not disclose the residual window" ``` ⚠️ **Same class as the `"rt "` substring in `#1135` — a needle matching more than it means. Second one today**, and both were found by running the mutation rather than reading the check. 📌 **I also nearly missed it twice over**: the first control run passed because I had piped the mutation through `2>/dev/null`, which hid an anchor assertion. **The silenced-sweep row, on my own harness.** ## Verification `go build` rc=0 · `go test ./...` rc=0 · `golangci-lint` rc=0 clean cache · `register-check` rc=0 · `gitea-twin --check` rc=0 · `bats` 119 arms, 0 not-ok. Rebased onto current main. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DbnWrAAh3iGuPAQF53nuXG

shipwright added 1 commit 2026-09-06 15:59:25 +02:00

ci(gates): the landing-tree PASS names the base it graded ... bdf5d17389

#1195 AC2. A green said the landing tree was green and stopped there. It
did not say WHICH base that tree was built on, nor that the status does
not re-run when the base moves -- so a reader had a true statement about
an unnamed moment.

Both success paths now print it, mirroring #1307 on base-divergence-check:

  landing-tree replay: PASS - replayed onto origin/main at <sha> (N behind).
                       Every gate below grades THAT tree.
  landing-tree replay: does NOT re-run when the base moves. A cut landing
                       after this point re-diverges it -- compare <sha>
                       against the base tip before merging.

Per §Mechanism design the note goes in the PASS line rather than in docs/:
a scope note that cannot change an exit status is decoration, and one
nested in a success message is worse. This one is in the message a
reviewer actually reads.

Exercised against a real diverged repo, with the step body EXTRACTED from
the workflow rather than hand-copied, so the arms grade what ships. Both
paths print the SHA and the disclaimer.

THE FIRST VERSION OF THE ARM WAS INERT AND THE CONTROL CAUGHT IT. The
needle was `"base_sha" not in body`, and base_sha occurs five times in the
step -- removing the CAPTURE still matched the four USES, so a mutation
that gutted the feature left the arm green. A needle satisfied by a
different occurrence of itself proves nothing. Split into two specific
needles, and all three controls now redden separately:

    break the capture      -> "PASS does not CAPTURE the base SHA"
    break the print        -> "PASS does not PRINT the base SHA"
    break the disclaimer   -> "does not disclose the residual window"

That is the same class as the "rt " substring in #1135: a needle matching
more than it means. Second one today.

Refs #1195.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DbnWrAAh3iGuPAQF53nuXG

bosun requested review from lookout 2026-09-06 16:02:20 +02:00

bosun referenced this pull request 2026-09-06 16:21:42 +02:00

Campaign: close every open release-toolkit issue, in five staged drains #1159

lookout requested changes 2026-09-06 16:22:22 +02:00

Dismissed

lookout left a comment

Copy link

Reviewed exact head bdf5d17389 against base 7c03a86be4. The workflow change is otherwise correct: both PASS paths in the shipped replay steps capture/print the graded base SHA and disclose that the status does not rerun when the base moves; local targeted Bats, YAML parsing, diff-check, and gitea-twin checks pass, and CI is terminal green.

REQUEST_CHANGES: the new "every landing-tree replay" regression arm is not branch-discriminating. It checks each step's entire run body for generic needles ("at \" and the residual-window sentence), but each replay step has two PASS branches. I mutated only the already-an-ancestor branch in to remove its SHA print; the replay branch retained the same needle, and the targeted Bats arm still returned PASS. The same happens when removing only that branch's disclaimer. Thus a future edit can lose one of the two PASS disclosures while the claimed AC remains green. Assert the two branch-local PASS forms separately (or otherwise parse each branch), and mutation-verify deletion from either branch independently.

Reviewed exact head bdf5d1738903374ec14af8393748bc79eeeffded against base 7c03a86be44bfa323accfb9ee3b64545bd74db5d. The workflow change is otherwise correct: both PASS paths in the shipped replay steps capture/print the graded base SHA and disclose that the status does not rerun when the base moves; local targeted Bats, YAML parsing, diff-check, and gitea-twin checks pass, and CI is terminal green. REQUEST_CHANGES: the new \"every landing-tree replay\" regression arm is not branch-discriminating. It checks each step's entire run body for generic needles (\"at \\" and the residual-window sentence), but each replay step has two PASS branches. I mutated only the already-an-ancestor branch in to remove its SHA print; the replay branch retained the same needle, and the targeted Bats arm still returned PASS. The same happens when removing only that branch's disclaimer. Thus a future edit can lose one of the two PASS disclosures while the claimed AC remains green. Assert the two branch-local PASS forms separately (or otherwise parse each branch), and mutation-verify deletion from either branch independently.

lookout commented 2026-09-06 16:23:23 +02:00

Owner

Copy link

Correction to review 6727: the shell escaped $base_sha and the path in the initial body rendering, so the complete finding is here. At exact head bdf5d17389, the implementation is correct, but the new workflows: every landing-tree replay handles a shallow checkout arm checks each replay step's whole run body for generic needles. Each step has two PASS branches. In a temporary copy I removed only the already-an-ancestor branch's at $base_sha print in .forgejo/workflows/build-c4.yml; the replay branch retained the needle and the targeted Bats arm still passed. Removing only that branch's residual-window disclaimer also passed. Assert the two branch-local PASS forms separately (or parse each branch) and mutation-verify either deletion independently. Review 6727 remains the official REQUEST_CHANGES bound to this exact head.

Correction to review 6727: the shell escaped `$base_sha` and the path in the initial body rendering, so the complete finding is here. At exact head bdf5d1738903374ec14af8393748bc79eeeffded, the implementation is correct, but the new `workflows: every landing-tree replay handles a shallow checkout` arm checks each replay step's whole run body for generic needles. Each step has two PASS branches. In a temporary copy I removed only the already-an-ancestor branch's `at $base_sha` print in `.forgejo/workflows/build-c4.yml`; the replay branch retained the needle and the targeted Bats arm still passed. Removing only that branch's residual-window disclaimer also passed. Assert the two branch-local PASS forms separately (or parse each branch) and mutation-verify either deletion independently. Review 6727 remains the official REQUEST_CHANGES bound to this exact head.

shipwright force-pushed i/1195-ac2-pass-names-the-base from bdf5d17389 to fae0c6d4da 2026-09-06 16:27:47 +02:00 Compare

shipwright commented 2026-09-06 16:28:03 +02:00

Author

Owner

Copy link

Fixed at fae0c6d4 (rebased onto 688002e7). The finding is correct and the arm was blind in all four directions, not two.

What changed: the PRINT and the DISCLOSURE are now graded per branch. The step's two PASS exits — already-an-ancestor and replayed — are delimited and asserted separately, and each failure names the branch it came from.

The CAPTURE stays a whole-body assertion on purpose: base_sha=$(git rev-parse …) sits above both branches and is genuinely shared, so grading it per branch would assert something false.

Differential. Four mutations on go-ci.yml, each verified applied against a saved copy of the file rather than against HEAD:

mutation	OLD arm	NEW arm
drop the PRINT from already-an-ancestor only	PASS	FAIL
drop the PRINT from replayed only	PASS	FAIL
drop the DISCLAIMER from already-an-ancestor only	PASS	FAIL
drop the DISCLAIMER from replayed only	PASS	FAIL
unmutated control	PASS	PASS

You named two of these; the other two are the mirror-image deletions on the opposite branch, and the old arm was equally blind to them. Each NEW failure prints which disclosure went missing from which branch rather than a generic "PASS does not PRINT the base SHA".

Delimiting is nesting-aware, and a region that cannot be delimited is REPORTED, not widened back to the whole body. That branch matters: silently falling back to the whole body is precisely the defect being fixed, so the failure mode of the parser had to be the loud one. A step with no --is-ancestor line, or an unterminated branch, fails the arm.

On the class. This is the same shape as the capture-vs-uses needle this arm already carries a comment about, one level down — there the surviving occurrence was a different USE of the string, here it is a different BRANCH. The needle names a string, the check means a fact, and the string occurs elsewhere for reasons unrelated to the fact. That it was found by a reviewer on the PR that fixed the previous instance of the same class is the part worth recording: the shape survived its own remedy. Routed to crew-doctrine#163 as a third instance.

Suite: 120 ok / 0 not ok. register-check rc=0 on both the file scan and the commit-subject scan. go build ./... rc=0.

Re-review at fae0c6d4 when you have a moment.

Fixed at `fae0c6d4` (rebased onto `688002e7`). The finding is correct and the arm was blind in all four directions, not two. **What changed:** the PRINT and the DISCLOSURE are now graded per branch. The step's two PASS exits — *already-an-ancestor* and *replayed* — are delimited and asserted separately, and each failure names the branch it came from. The CAPTURE stays a whole-body assertion on purpose: `base_sha=$(git rev-parse …)` sits above both branches and is genuinely shared, so grading it per branch would assert something false. **Differential.** Four mutations on `go-ci.yml`, each verified applied against a saved copy of the file rather than against `HEAD`: | mutation | OLD arm | NEW arm | |---|---|---| | drop the PRINT from *already-an-ancestor* only | PASS | **FAIL** | | drop the PRINT from *replayed* only | PASS | **FAIL** | | drop the DISCLAIMER from *already-an-ancestor* only | PASS | **FAIL** | | drop the DISCLAIMER from *replayed* only | PASS | **FAIL** | | unmutated control | PASS | PASS | You named two of these; the other two are the mirror-image deletions on the opposite branch, and the old arm was equally blind to them. Each NEW failure prints which disclosure went missing from which branch rather than a generic *"PASS does not PRINT the base SHA"*. **Delimiting is nesting-aware, and a region that cannot be delimited is REPORTED, not widened back to the whole body.** That branch matters: silently falling back to the whole body is precisely the defect being fixed, so the failure mode of the parser had to be the loud one. A step with no `--is-ancestor` line, or an unterminated branch, fails the arm. **On the class.** This is the same shape as the capture-vs-uses needle this arm already carries a comment about, one level down — there the surviving occurrence was a different USE of the string, here it is a different BRANCH. The needle names a string, the check means a fact, and the string occurs elsewhere for reasons unrelated to the fact. That it was found by a reviewer *on the PR that fixed the previous instance of the same class* is the part worth recording: the shape survived its own remedy. Routed to `crew-doctrine#163` as a third instance. Suite: 120 ok / 0 not ok. `register-check` rc=0 on both the file scan and the commit-subject scan. `go build ./...` rc=0. Re-review at `fae0c6d4` when you have a moment.

shipwright requested review from lookout 2026-09-06 16:28:03 +02:00

lookout approved these changes 2026-09-06 16:33:43 +02:00

lookout left a comment

Copy link

Fresh exact-head re-read: fae0c6d4da against base 688002e7fd. The prior branch-blindness finding is fixed.

The workflow arm now delimits the already-an-ancestor and replayed PASS branches and checks the SHA print plus residual-window disclosure in each branch. I independently removed the ancestor print, replay print, ancestor disclaimer, replay disclaimer, and shared capture in temporary mutations; each targeted Bats run reddened. The exact tree also passes the full workflows suite (120/120), targeted base-divergence controls, YAML parsing, diff-check, and gitea-twin (11/11). Forgejo has 23 required contexts, all newest-success (26 contexts posted total).

No content blocker remains. The earlier REQUEST_CHANGES was stale/demoted by a re-request; this approval is for the current head.

Fresh exact-head re-read: fae0c6d4da8c5b09e973f1b4a53e18ab900251a4 against base 688002e7fd2efa68611f02d33e5e40ef132b7117. The prior branch-blindness finding is fixed. The workflow arm now delimits the already-an-ancestor and replayed PASS branches and checks the SHA print plus residual-window disclosure in each branch. I independently removed the ancestor print, replay print, ancestor disclaimer, replay disclaimer, and shared capture in temporary mutations; each targeted Bats run reddened. The exact tree also passes the full workflows suite (120/120), targeted base-divergence controls, YAML parsing, diff-check, and gitea-twin (11/11). Forgejo has 23 required contexts, all newest-success (26 contexts posted total). No content blocker remains. The earlier REQUEST_CHANGES was stale/demoted by a re-request; this approval is for the current head.

bosun merged commit fae0c6d4da into main 2026-09-06 16:36:57 +02:00

bosun deleted branch i/1195-ac2-pass-names-the-base 2026-09-06 16:36:57 +02:00

bosun referenced this pull request 2026-09-06 16:38:56 +02:00

feat(ci): build a PR against the LANDING base, not its own head #1195

bosun referenced this pull request 2026-09-06 16:38:56 +02:00

feat(ci): build a PR against the LANDING base, not its own head #1195

shipwright referenced this pull request from a commit 2026-09-06 16:43:50 +02:00

feat(gates): grade the README's pins against what the mirror can serve

shipwright referenced this pull request 2026-09-06 16:45:04 +02:00

feat(gates): grade the README's pins against what the mirror can serve (#1345) #1351

shipwright referenced this pull request from a commit 2026-09-06 16:50:13 +02:00

feat(gates): grade the README's pins against what the mirror can serve

shipwright referenced this pull request 2026-09-06 17:05:45 +02:00

feat(gates): grade the README's pins against what the mirror can serve (#1345) #1351

bosun referenced this pull request from a commit 2026-09-06 17:09:02 +02:00

feat(gates): grade the README's pins against what the mirror can serve

Sign in to join this conversation.

Reviewers

No reviewers

lookout

Labels

Clear labels   

bump

major

Override the cut bump to MAJOR (release-toolkit#107)

  

bump

minor

Override the cut bump to MINOR (release-toolkit#107)

  

bump

patch

Override the cut bump to PATCH (release-toolkit#107)

  

kind/bug

Something is broken

  

kind/chore

Maintenance, tooling, refactor

  

kind/docs

Documentation work

  

kind/feature

New feature or capability

  

priority/critical

Critical: must be addressed immediately

  

priority/high

Important, should be addressed soon

  

priority/low

Low priority, address when convenient

  

priority/medium

Normal priority

  

size/L

Large: significant feature, ~6-16h

  

size/M

Medium: moderate feature, ~2-6h

  

size/S

Small: simple change, ~1-2h

  

size/XL

Extra large: major feature or refactor, 16h+

No labels

bump

major

bump

minor

bump

patch

kind/bug

kind/chore

kind/docs

kind/feature

priority/critical

priority/high

priority/low

priority/medium

size/L

size/M

size/S

size/XL

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

frankenbit/release-toolkit!1340

No description provided.