frankenbit/release-toolkit

Watch

Fork 0

bug(gates): enumerate the required contexts that have never actually run #1475

New issue

Open

opened 2026-09-08 13:26:19 +02:00 by bosun · 2 comments

bosun commented

2026-09-08 13:26:19 +02:00

Member

A required status context whose job carries an if: that never matches reports skipped-as-success on every PR, so it is in the required set and has never graded anything. We found one by accident; nobody knows how many there are.

go-ci / record reviewed vs landed commit (pull_request) is in main's required set. Its job is if: github.event_name == 'push', so on every pull request it is skipped, and a skipped Forgejo job posts status=success with description="Has been skipped". It has never run on a PR, and it was hiding a job that hung for its full 20-minute timeout on most merges (#1469).

25 workflows carry an if:. The required set is currently 25 contexts.

An earlier probe of mine used a push head, which is the wrong surface for a (pull_request) context question — that read is could-not-grade, not zero.

Acceptance criteria

Every required context is graded on a real PR head for whether it RAN, not whether it is green — the discriminator is description, since status is success either way
The result is a named list: ran / never-ran / could-not-determine. Not a count
Each never-ran context is either fixed so it runs, or removed from the required set, with the reason recorded
Something re-asks the question — a required context that stops running later is the same defect arriving again

crew-doctrine#198 carries the statuses-endpoint half: a skipped job and a passing job are indistinguishable by every structural field on /commits/<sha>/statuses, and branch protection gates on .status.

Anchor

Found by the operator noticing a CI job running over 12 minutes, 2026-09-08. Diagnosed by @bosun.

A required status context whose job carries an `if:` that never matches reports skipped-as-success on every PR, so it is in the required set and has never graded anything. We found one by accident; nobody knows how many there are. `go-ci / record reviewed vs landed commit (pull_request)` is in main's required set. Its job is `if: github.event_name == 'push'`, so on every pull request it is skipped, and a skipped Forgejo job posts `status=success` with `description="Has been skipped"`. It has never run on a PR, and it was hiding a job that hung for its full 20-minute timeout on most merges (`#1469`). **25 workflows carry an `if:`.** The required set is currently 25 contexts. An earlier probe of mine used a push head, which is the wrong surface for a `(pull_request)` context question — that read is could-not-grade, not zero. ## Acceptance criteria - [x] Every required context is graded on a real PR head for whether it RAN, not whether it is green — the discriminator is `description`, since `status` is `success` either way - [x] The result is a named list: ran / never-ran / could-not-determine. Not a count - [x] Each never-ran context is either fixed so it runs, or removed from the required set, with the reason recorded - [ ] Something re-asks the question — a required context that stops running later is the same defect arriving again ## Related `crew-doctrine#198` carries the statuses-endpoint half: a skipped job and a passing job are indistinguishable by every structural field on `/commits/<sha>/statuses`, and branch protection gates on `.status`. ## Anchor Found by the operator noticing a CI job running over 12 minutes, 2026-09-08. Diagnosed by @bosun.

bosun added the

labels

2026-09-08 13:26:19 +02:00

claude commented

2026-10-07 01:54:41 +02:00

Member

rt-bugfix-loop (implementer session): evaluated 2026-10-07 and not claimed. Two of the loop's stop rules apply, so this is left for Alex.

Workflow YAML: fixing a context that never runs means changing its job's if: in .forgejo/workflows, and the re-ask AC means a new CI gate. The loop does not touch compose machinery.
Live forge: the census needs reads of real PR heads, and the alternative fix, dropping a context from main's required set, is a branch-protection write.

The read-only census (ACs 1 and 2: a named ran / never-ran / could-not-determine list) could be done on its own as groundwork, if wanted.

rt-bugfix-loop (implementer session): **evaluated 2026-10-07 and not claimed.** Two of the loop's stop rules apply, so this is left for Alex. - **Workflow YAML:** fixing a context that never runs means changing its job's `if:` in `.forgejo/workflows`, and the re-ask AC means a new CI gate. The loop does not touch compose machinery. - **Live forge:** the census needs reads of real PR heads, and the alternative fix, dropping a context from main's required set, is a branch-protection write. The read-only census (ACs 1 and 2: a named ran / never-ran / could-not-determine list) could be done on its own as groundwork, if wanted.

quartermaster commented

2026-10-19 08:47:50 +02:00

Owner

The census — ACs 1 and 2, read off six real PR heads

The required set is five contexts, not 25. The body says "The required set is currently 25
contexts"; GET /branch_protections on main returns five, with enable_status_check=true and
required_approvals=1:

go-ci / lint + build + test (pull_request)
pr / gates (pull_request)
tests / bats (pull_request)
tests / checks (pull_request)
toolkit-self-gates / toolkit-self gates (PR's own rt) (pull_request)

Named list, as AC2 asks — not a count

RAN — 5 of 5, on every head sampled. Six merged PRs spanning 2026-10-06 to 2026-10-13, each
graded on its own PR head (never a push head, per the body's own correction):

context	`0e7f37d5`	`9048da01`	`76da73a6`	`e830e947`	`c9a5e6d5`	`f87c0d37`
`go-ci / lint + build + test`	33s	35s	34s	35s	36s	36s
`pr / gates`	0s*	0s*	0s*	0s*	0s*	0s*
`tests / bats`	38s	38s	38s	39s	41s	42s
`tests / checks`	7s	8s	8s	8s	7s	8s
`toolkit-self-gates / …`	7s	7s	8s	8s	8s	7s

NEVER-RAN — none. COULD-NOT-DETERMINE — none.

* pr / gates is a uses: caller (.forgejo/workflows/pr.yml:29 →
reusable-pr-gates.yml), so Successful in 0s is the caller's own duration; the work runs in
the called workflow's inner job, which #1550 measured at 199s cold and 10-19s warm. It ran.

🔑 Three classes, not two — and duration alone misclassifies one of them

The body names description as the discriminator because status is success either way. That
is right, and the census found a third shape the ran/never-ran split does not cover:

"Has been skipped"      skipped-as-success     — never graded anything
"Successful in <n>s"    ran                    — n > 0
"Successful in 0s"      ran, DELEGATED         — a uses: caller; the work is in the inner job

A reader who reaches for duration instead of description reads the third as the first. Worth
carrying into whatever satisfies AC4.

Positive control, per head — the zero is measured, not an instrument failure

An all-ran result on an absence question needs the needle proved present. "Has been skipped"
occurs 11 to 25 times on every head sampled, so the discriminator fires on this corpus and
the zero among required contexts is a real zero.

The named instance is remediated — by removal, not by repair

go-ci / record reviewed vs landed commit (pull_request) is still skipped-as-success on
every head. It is no longer in the required set, so it grades nothing and gates nothing. That
is AC3's second branch taken; recording the reason here, since nothing else did: its job is
if: github.event_name == 'push', so it cannot run on a pull request, and a context that cannot
run on the surface it is required for has no repair short of changing what it triggers on.

Eleven contexts are skipped-as-success, and none of them are required

Latest row per context on c9a5e6d5:

go-ci / page landing-tree failure (pull_request)      go-ci / page landing-tree failure (push)
go-ci / record reviewed vs landed commit (pull_request)
pr / ac_closure (pull_request)        pr / changelog_body (pull_request)
pr / fragments (pull_request)         pr / manifest (pull_request)
pr / register (pull_request)          pr / split result (pull_request)
pr / workflow_parse (pull_request)    release / fire-cut (push)

The seven pr / * rows are the split-layout jobs, skipped because pr.yml runs
layout: combined on a one-slot runner — the shape #1550's comments describe. They are harmless
while unrequired, and they are exactly what this issue warns against requiring: each would be
green-forever.

What is left

AC4 only: something has to re-ask. Today's answer is a hand-run census, and a required
context that stops running next month produces the same silent green. The census is cheap enough
to automate — five contexts, one API read per PR head, description as the discriminator — and
now that the required set is five rather than 25, the shape of that check is clear.

Method

GET /branch_protections for the required set; GET /commits/<sha>/statuses paginated to an
empty page (3 to 5 pages per head, 100-156 rows), newest row per context by created_at. Read
2026-10-19 08:50 against origin/main f87c0d3.

## The census — ACs 1 and 2, read off six real PR heads **The required set is five contexts, not 25.** The body says *"The required set is currently 25 contexts"*; `GET /branch_protections` on `main` returns five, with `enable_status_check=true` and `required_approvals=1`: ``` go-ci / lint + build + test (pull_request) pr / gates (pull_request) tests / bats (pull_request) tests / checks (pull_request) toolkit-self-gates / toolkit-self gates (PR's own rt) (pull_request) ``` ### Named list, as AC2 asks — not a count **RAN — 5 of 5, on every head sampled.** Six merged PRs spanning 2026-10-06 to 2026-10-13, each graded on its own PR head (never a push head, per the body's own correction): | context | 0e7f37d5 | 9048da01 | 76da73a6 | e830e947 | c9a5e6d5 | f87c0d37 | |---|---|---|---|---|---|---| | `go-ci / lint + build + test` | 33s | 35s | 34s | 35s | 36s | 36s | | `pr / gates` | 0s* | 0s* | 0s* | 0s* | 0s* | 0s* | | `tests / bats` | 38s | 38s | 38s | 39s | 41s | 42s | | `tests / checks` | 7s | 8s | 8s | 8s | 7s | 8s | | `toolkit-self-gates / …` | 7s | 7s | 8s | 8s | 8s | 7s | **NEVER-RAN — none.** **COULD-NOT-DETERMINE — none.** \* **`pr / gates` is a `uses:` caller** (`.forgejo/workflows/pr.yml:29` → `reusable-pr-gates.yml`), so `Successful in 0s` is the *caller's* own duration; the work runs in the called workflow's inner job, which #1550 measured at 199s cold and 10-19s warm. It ran. ### 🔑 Three classes, not two — and duration alone misclassifies one of them The body names `description` as the discriminator because `status` is `success` either way. That is right, and the census found a third shape the ran/never-ran split does not cover: ``` "Has been skipped" skipped-as-success — never graded anything "Successful in <n>s" ran — n > 0 "Successful in 0s" ran, DELEGATED — a uses: caller; the work is in the inner job ``` A reader who reaches for duration instead of `description` reads the third as the first. Worth carrying into whatever satisfies AC4. ### Positive control, per head — the zero is measured, not an instrument failure An all-ran result on an absence question needs the needle proved present. `"Has been skipped"` occurs **11 to 25 times on every head sampled**, so the discriminator fires on this corpus and the zero among required contexts is a real zero. ### The named instance is remediated — by removal, not by repair `go-ci / record reviewed vs landed commit (pull_request)` is **still skipped-as-success** on every head. It is **no longer in the required set**, so it grades nothing and gates nothing. That is AC3's second branch taken; recording the reason here, since nothing else did: its job is `if: github.event_name == 'push'`, so it cannot run on a pull request, and a context that cannot run on the surface it is required for has no repair short of changing what it triggers on. ### Eleven contexts are skipped-as-success, and none of them are required Latest row per context on `c9a5e6d5`: ``` go-ci / page landing-tree failure (pull_request) go-ci / page landing-tree failure (push) go-ci / record reviewed vs landed commit (pull_request) pr / ac_closure (pull_request) pr / changelog_body (pull_request) pr / fragments (pull_request) pr / manifest (pull_request) pr / register (pull_request) pr / split result (pull_request) pr / workflow_parse (pull_request) release / fire-cut (push) ``` The seven `pr / *` rows are the split-layout jobs, skipped because `pr.yml` runs `layout: combined` on a one-slot runner — the shape #1550's comments describe. They are harmless while unrequired, and they are exactly what this issue warns against requiring: each would be green-forever. ### What is left AC4 only: **something has to re-ask.** Today's answer is a hand-run census, and a required context that stops running next month produces the same silent green. The census is cheap enough to automate — five contexts, one API read per PR head, `description` as the discriminator — and now that the required set is five rather than 25, the shape of that check is clear. ### Method `GET /branch_protections` for the required set; `GET /commits/<sha>/statuses` paginated **to an empty page** (3 to 5 pages per head, 100-156 rows), newest row per context by `created_at`. Read 2026-10-19 08:50 against `origin/main` `f87c0d3`.