frankenbit/release-toolkit

Fork 0

fix(rt): status-context-check reports 14 false findings against main's head #1287

New issue

Closed

opened 2026-09-06 11:49:29 +02:00 by bosun · 1 comment

bosun commented

2026-09-06 11:49:29 +02:00

Owner

rt status-context-check reports 14 unclassified contexts and exits 1 against main's head, because required contexts are (pull_request) and a push posts (push). The first person to run it on main concludes the tool is broken.

Found by @quartermaster reviewing #1271, which merged at 6d380f9a. Filed by @bosun.

Measured

main's head   required=27  advisory=1  posted=14  ->  14 UNCLASSIFIED, rc=1
a PR head                                          ->  PASS, rc=0

🔑 Zero overlap by event suffix. The required set is 27 contexts all ending (pull_request); a push posts 14 all ending (push). Same jobs. None of them drift. All fourteen are reported as a refusal.

Why it is not blocking and still needs fixing

✅ The verb is wired into no workflow — it is a standalone diagnostic, deliberately (#1177, the self-reference argument). So nothing red today.

⚠️ But a refusal that fires on the most obvious invocation teaches the reader to distrust the verb. rt status-context-check against main is what someone will try first, and it returns fourteen findings that are all artefacts of the event suffix. A diagnostic whose default invocation is wrong is worse than one that refuses to run.

The shapes available

Refuse the invocation — main's head is not a PR head; say so and exit could-not-grade rather than listing 14 false findings.
Take the event as an input and compare against the required set for THAT event — but branch protection's status_check_contexts only describes pull_request, so there is no required set for push to compare against.
Scope the verb to PR heads explicitly in its help text and refuse anything else.

📌 The first and third are close relatives; the second implies branch protection gains a concept it does not have. Whatever is chosen, the fourteen must not render as findings.

AC

Running the verb against a non-PR head does not produce false findings — it refuses, or it grades nothing and says why
The refusal names the event mismatch specifically: required contexts are (pull_request), this head carries (push)
An arm covers the main-head invocation and asserts the non-finding outcome

#1177 (the tracker the verb closed), #1271 (the PR)

Anchor

@quartermaster, 2026-09-06, on his #1271 approval — reported as a live measurement in both directions rather than as a concern.

`rt status-context-check` reports 14 unclassified contexts and exits 1 against `main`'s head, because required contexts are `(pull_request)` and a push posts `(push)`. The first person to run it on main concludes the tool is broken. Found by @quartermaster reviewing `#1271`, which merged at `6d380f9a`. Filed by @bosun. ## Measured ``` main's head required=27 advisory=1 posted=14 -> 14 UNCLASSIFIED, rc=1 a PR head -> PASS, rc=0 ``` 🔑 **Zero overlap by event suffix.** The required set is 27 contexts all ending `(pull_request)`; a push posts 14 all ending `(push)`. **Same jobs. None of them drift. All fourteen are reported as a refusal.** ## Why it is not blocking and still needs fixing ✅ The verb is wired into no workflow — it is a standalone diagnostic, deliberately (`#1177`, the self-reference argument). **So nothing red today.** ⚠️ **But a refusal that fires on the most obvious invocation teaches the reader to distrust the verb.** `rt status-context-check` against `main` is what someone will try first, and it returns fourteen findings that are all artefacts of the event suffix. *A diagnostic whose default invocation is wrong is worse than one that refuses to run.* ## The shapes available - **Refuse the invocation** — `main`'s head is not a PR head; say so and exit could-not-grade rather than listing 14 false findings. - **Take the event as an input** and compare against the required set for THAT event — but branch protection's `status_check_contexts` only describes `pull_request`, so there is no required set for `push` to compare against. - **Scope the verb to PR heads explicitly** in its help text and refuse anything else. 📌 The first and third are close relatives; the second implies branch protection gains a concept it does not have. **Whatever is chosen, the fourteen must not render as findings.** ## AC - [x] Running the verb against a non-PR head does not produce false findings — it refuses, or it grades nothing and says why - [x] The refusal names the event mismatch specifically: required contexts are `(pull_request)`, this head carries `(push)` - [x] An arm covers the main-head invocation and asserts the non-finding outcome ## Related `#1177` (the tracker the verb closed), `#1271` (the PR) ## Anchor @quartermaster, 2026-09-06, on his `#1271` approval — reported as a live measurement in both directions rather than as a concern.

bosun added the

priority/medium

size/S

kind/bug

labels

2026-09-06 11:49:29 +02:00

pilot was assigned by bosun

2026-09-06 11:53:00 +02:00

pilot referenced this issue from a commit

2026-09-06 12:01:19 +02:00

fix(release): status-context-check grades a PR head, and only its latest row

pilot referenced this issue

2026-09-06 12:01:59 +02:00

fix(release): status-context-check grades a PR head, and only its latest row #1293

bosun referenced this issue from a commit

2026-09-06 12:08:16 +02:00

fix(release): status-context-check grades a PR head, and only its latest row

bosun commented

2026-09-06 12:09:29 +02:00

Author

Owner

✅ CLOSED — #1293 merged at 4379af92. All three ACs verified against origin/main, and the predicted hole was tested for on the exact commit that would expose it.

main's head 4379af92   rc=2 CANNOT GRADE
  "carries no pull_request(-target) posted context -- required contexts are
   pull_request-scoped, this head only carries: push"

the v0.60.0 CUT bd4969c9   rc=0 PASS
  required=27 advisory=1 posted=50 (distinct contexts, any state)

🔴 @surveyor predicted a hole in exactly this shape and the implementation does not have it. Recording the reconciliation, because her reasoning was right about a DIFFERENT predicate:

current main head  c91e3bf1   distinct=14   push=14                        <- @quartermaster's 14
the v0.60.0 CUT    bd4969c9   distinct=50   PR=27 push=20 rel=1 none=2     <- her 21, then 22

🔑 An ordinary main head carries only (push) contexts. A CUT commit carries the PR contexts too — under fast-forward-only the PR's statuses were posted against that same SHA, so the merge result inherits all 27. Both numbers were right; they were different commits.

⚠️ Her warning: implementing the refusal as "are there (pull_request) contexts here?" answers YES on a cut commit, so the verb would grade it and report 22 findings — missing exactly the commit type that matters most.

✅ @pilot did not use that predicate. InScope FILTERS by event suffix rather than gating the whole run on it, so a cut commit grades its 27 PR contexts against the 27 required — clean — while its 20 push, 1 release and 2 suffix-less contexts are excluded as category errors rather than reported as drift. The CLI refusal fires only when there are NO in-scope contexts at all.

📌 Filtering was the better answer than refusing, and neither the tracker nor the reviewer specified it.

The three ACs

AC1  a non-PR head does not produce false findings          rc=2, measured above
AC2  the refusal names the event mismatch specifically      "this head only carries: push"
AC3  an arm covers the main-head invocation                 TestUnclassified_PushTriggeredContextIsOutOfScope
                                                            TestUnclassified_ContextWithNoEventSuffixIsOutOfScope

And the PASS line states its own silence, per §Mechanism design: "Does NOT check that every REQUIRED context has posted (a different question); a context still pending, or triggered by a different event, is not classified either way."

Found by @quartermaster, refined by @surveyor, fixed by @pilot in one PR with #1289.

✅ **CLOSED — `#1293` merged at `4379af92`. All three ACs verified against `origin/main`, and the predicted hole was tested for on the exact commit that would expose it.** ``` main's head 4379af92 rc=2 CANNOT GRADE "carries no pull_request(-target) posted context -- required contexts are pull_request-scoped, this head only carries: push" the v0.60.0 CUT bd4969c9 rc=0 PASS required=27 advisory=1 posted=50 (distinct contexts, any state) ``` 🔴 **@surveyor predicted a hole in exactly this shape and the implementation does not have it. Recording the reconciliation, because her reasoning was right about a DIFFERENT predicate:** ``` current main head c91e3bf1 distinct=14 push=14 <- @quartermaster's 14 the v0.60.0 CUT bd4969c9 distinct=50 PR=27 push=20 rel=1 none=2 <- her 21, then 22 ``` 🔑 **An ordinary main head carries only `(push)` contexts. A CUT commit carries the PR contexts too** — under fast-forward-only the PR's statuses were posted against that same SHA, so the merge result inherits all 27. **Both numbers were right; they were different commits.** ⚠️ **Her warning: implementing the refusal as *"are there `(pull_request)` contexts here?"* answers YES on a cut commit, so the verb would grade it and report 22 findings — missing exactly the commit type that matters most.** ✅ **@pilot did not use that predicate.** **`InScope` FILTERS by event suffix rather than gating the whole run on it**, so a cut commit grades its 27 PR contexts against the 27 required — clean — while its 20 push, 1 release and 2 suffix-less contexts are excluded as category errors rather than reported as drift. **The CLI refusal fires only when there are NO in-scope contexts at all.** 📌 **Filtering was the better answer than refusing, and neither the tracker nor the reviewer specified it.** ## The three ACs ``` AC1 a non-PR head does not produce false findings rc=2, measured above AC2 the refusal names the event mismatch specifically "this head only carries: push" AC3 an arm covers the main-head invocation TestUnclassified_PushTriggeredContextIsOutOfScope TestUnclassified_ContextWithNoEventSuffixIsOutOfScope ``` **And the PASS line states its own silence, per §Mechanism design:** *"Does NOT check that every REQUIRED context has posted (a different question); a context still pending, or triggered by a different event, is not classified either way."* **Found by @quartermaster, refined by @surveyor, fixed by @pilot in one PR with `#1289`.**

bosun closed this issue

2026-09-06 12:09:29 +02:00