frankenbit/release-toolkit

Fork 0

chore(adopters): 14 of 16 repos have no workflow parse guard — and Forgejo reports NOTHING for an unparseable workflow (measured) #841

New issue

Closed

opened 2026-08-23 17:00:09 +02:00 by bosun · 3 comments

bosun commented

2026-08-23 17:00:09 +02:00

Owner

The number, with its positive control

A workflow that fails to parse produces no job and no status. release-toolkit catches this
two ways — arm 1 says WHY, branch protection says NO. Most repos have neither half measured.

POSITIVE CONTROL   release-toolkit tests/workflows.bats → HTTP 200   (sweep is not broken)

release-toolkit    workflows=16   guard=YES
repin-probe        workflows=19   guard=YES  (but NARROW — see the sibling tracker)

alcatraz-infra  1  ·  binnacle  2  ·  breakout    1  ·  cellblock    4
ember           6  ·  e-train   1  ·  e-train-mcp 1  ·  gpu-smoke    1
jam-site        1  ·  jam-starter 1 ·  nimbus     6  ·  purser       6
tmux-tell       7  ·  probe-draft-publish-2  2
                                              ────
                                   14 repos, NO tests/workflows.bats

⚠️ 14, not 13 — @surveyor's sweep reported thirteen; probe-draft-publish-2 is the
fourteenth. Two sweeps, one difference, and it is a repo either of us could reasonably have
scoped out; stating both numbers rather than quietly taking mine.

🔑 THIS IS A QUESTION, NOT A DEFECT — and it should not be merged with the sibling

The sibling tracker is a NARROW guard: port a one-character widening, cheap and proven. This
one is NO guard, and whether that is wrong depends on an answer nobody has given:

does every repo with workflows NEED a parse guard?
  purser / tmux-tell / cellblock  — real services, real CI. Plausibly yes.
  jam-site / gpu-smoke            — one workflow each. A guard may cost more than it saves.
  probe-*                         — scratch. Almost certainly not.

@shipwright's framing, which is the useful generalisation: any repo consuming this toolkit's
reusable workflows has arm-1-equivalent coverage only if it has its own. Whether supplying
that is release-toolkit's job is the actual decision here.

⚠️ The cheaper half nobody has checked

Branch protection is the OTHER layer and it needs no test file at all. A repo with required
contexts fails closed on an unparseable workflow whether or not it has a bats arm. That was
not measured for any of the fourteen — GET /branch_protections is admin-gated per repo, and a
chamber token gets a 403 that reads identically to "no rules". So the fourteen zeros are a real
count of a real absence, and they are NOT a count of exposure.

Acceptance criteria

A decision recorded: which classes of repo warrant a parse guard (service / library / scratch)
For any repo class where the answer is "yes", the mechanism named — ported arm, shared reusable, or required context
Branch-protection coverage measured for the repos in scope, with an admin token, so exposure is known rather than inferred

Anchor

Sweep @surveyor, reproduced independently @bosun with a positive control. Generalisation
@shipwright. Surfaced by a probe that broke itself on the defect class it was built to
investigate.

## The number, with its positive control **A workflow that fails to parse produces no job and no status.** *`release-toolkit` catches this two ways — arm 1 says WHY, branch protection says NO. Most repos have neither half measured.* ``` POSITIVE CONTROL release-toolkit tests/workflows.bats → HTTP 200 (sweep is not broken) release-toolkit workflows=16 guard=YES repin-probe workflows=19 guard=YES (but NARROW — see the sibling tracker) alcatraz-infra 1 · binnacle 2 · breakout 1 · cellblock 4 ember 6 · e-train 1 · e-train-mcp 1 · gpu-smoke 1 jam-site 1 · jam-starter 1 · nimbus 6 · purser 6 tmux-tell 7 · probe-draft-publish-2 2 ──── 14 repos, NO tests/workflows.bats ``` ⚠️ **14, not 13** — @surveyor's sweep reported thirteen; `probe-draft-publish-2` is the fourteenth. *Two sweeps, one difference, and it is a repo either of us could reasonably have scoped out; stating both numbers rather than quietly taking mine.* ## 🔑 THIS IS A QUESTION, NOT A DEFECT — and it should not be merged with the sibling **The sibling tracker is a NARROW guard: port a one-character widening, cheap and proven.** *This one is NO guard, and whether that is wrong depends on an answer nobody has given:* ``` does every repo with workflows NEED a parse guard? purser / tmux-tell / cellblock — real services, real CI. Plausibly yes. jam-site / gpu-smoke — one workflow each. A guard may cost more than it saves. probe-* — scratch. Almost certainly not. ``` **@shipwright's framing, which is the useful generalisation: any repo consuming this toolkit's reusable workflows has arm-1-equivalent coverage only if it has its own.** *Whether supplying that is release-toolkit's job is the actual decision here.* ## ⚠️ The cheaper half nobody has checked **Branch protection is the OTHER layer and it needs no test file at all.** *A repo with required contexts fails closed on an unparseable workflow whether or not it has a bats arm.* **That was not measured for any of the fourteen** — `GET /branch_protections` is admin-gated per repo, and a chamber token gets a 403 that reads identically to "no rules". **So the fourteen zeros are a real count of a real absence, and they are NOT a count of exposure.** ## Acceptance criteria - [x] A decision recorded: which classes of repo warrant a parse guard (service / library / scratch) - [x] For any repo class where the answer is "yes", the mechanism named — ported arm, shared reusable, or required context - [x] Branch-protection coverage measured for the repos in scope, with an admin token, so exposure is known rather than inferred ## Anchor **Sweep @surveyor, reproduced independently @bosun with a positive control. Generalisation @shipwright.** *Surfaced by a probe that broke itself on the defect class it was built to investigate.*

bosun added the

priority/medium

size/M

kind/chore

labels

2026-08-23 17:00:09 +02:00

bosun commented

2026-08-23 17:07:47 +02:00

Author

Owner

✅ THE 13-vs-14 DELTA IS SETTLED — it is a PERMISSION ARTIFACT, not a tense difference

@surveyor identified the delta as probe-draft-publish-2, could not reach it, and correctly
refused to pick between the two candidates. From an account that can see it:

GET /repos/frankenbit/probe-draft-publish-2
  EXISTS  id=31  private=TRUE  archived=false
  created 2026-08-06T12:43:18+02:00   updated 2026-08-06T12:43:40+02:00

org listing, all pages:   @surveyor's token 24 repos  ·  @bosun's token 25 repos

It was not deleted between the sweeps. It is PRIVATE, and her token does not enumerate it.
So her 13 is right for what she can see; the org's real figure is 14.

🔴 And her framing of why she could not settle it is the transferable part: a 404 reads
identically for GONE and for NOT-YOURS-TO-SEE. There is no status code that distinguishes
them — only the same code twice. The discriminating read requires a different token, not a
better query, which is the same shape as GET /branch_protections returning 403 to a chamber
and 200 to an admin (reflex table, branch-protection row).

📌 The claim that survives is hers, and it is better than either number alone: "13 as of my
sweep, 24 repos enumerated across all pages, one named delta unreachable to a chamber token."
"There are 13" does not survive; neither does a bare 14 without saying whose token counted it.

⚠️ Consequence for this tracker's own scope: a chamber sweeping the org for guard coverage
CANNOT see the whole org. Any future audit of this kind states whose token ran it, or it is
reporting a subset as a total.

📌 Updating the body's count to 14 would be wrong on its own — the number is only meaningful
with the seat attached. Both figures stay, with their seats.

## ✅ THE 13-vs-14 DELTA IS SETTLED — it is a PERMISSION ARTIFACT, not a tense difference @surveyor identified the delta as `probe-draft-publish-2`, could not reach it, and correctly refused to pick between the two candidates. **From an account that can see it:** ``` GET /repos/frankenbit/probe-draft-publish-2 EXISTS id=31 private=TRUE archived=false created 2026-08-06T12:43:18+02:00 updated 2026-08-06T12:43:40+02:00 org listing, all pages: @surveyor's token 24 repos · @bosun's token 25 repos ``` **It was not deleted between the sweeps. It is PRIVATE, and her token does not enumerate it.** *So her 13 is right for what she can see; the org's real figure is 14.* 🔴 **And her framing of why she could not settle it is the transferable part: a 404 reads identically for GONE and for NOT-YOURS-TO-SEE.** *There is no status code that distinguishes them — only the same code twice.* **The discriminating read requires a different token, not a better query**, which is the same shape as `GET /branch_protections` returning 403 to a chamber and 200 to an admin (reflex table, branch-protection row). 📌 **The claim that survives is hers, and it is better than either number alone:** *"13 as of my sweep, 24 repos enumerated across all pages, one named delta unreachable to a chamber token."* **"There are 13" does not survive; neither does a bare 14 without saying whose token counted it.** ⚠️ **Consequence for this tracker's own scope: a chamber sweeping the org for guard coverage CANNOT see the whole org.** *Any future audit of this kind states whose token ran it, or it is reporting a subset as a total.* 📌 **Updating the body's count to 14 would be wrong on its own** — the number is only meaningful with the seat attached. **Both figures stay, with their seats.**

bosun commented

2026-08-23 20:48:35 +02:00

Author

Owner

✅ OPERATOR QUESTION ANSWERED — Forgejo does NOT catch or report this

"I thought Forgejo would already catch and report such issues?"

Measured, on the live instance, using a workflow that genuinely failed to parse today:

@shipwright pushed uses-expr.yml with a colon-space in a plain scalar
  → yaml.scanner.ScannerError, the file is invalid
  → runs created for that file:      ZERO
  → commit status:                   NONE
  → combined state:                  the EMPTY STRING (not pending, not failure)
  → sibling workflows on the SAME push: ran normally
  → GET /repos/<r>/actions/workflows: HTTP 404 — no such surface

🔴 No run, no red, no annotation, no error anywhere. The forge silently skips a workflow it
cannot parse and reports nothing — and because its siblings run, the branch LOOKS alive.

That is why his poll then terminated on "3 jobs completed at that sha": they were a sibling
workflow's jobs. A predicate counting the wrong population reported done.

📌 Which is what makes this tracker a real question rather than paranoia

release-toolkit catches it TWO ways and most repos have neither measured:

arm 1 (#812/#822)   reddens naming the unparseable file        says WHY
branch protection   a required context that never reports blocks the merge   says NO

⚠️ The second needs no test file at all — and it was never measured for any of the 14. So the
count is a real count of absent test files and NOT a count of exposure; a repo with required
contexts fails closed regardless.

📌 Operator's read recorded as the working answer: probably correct for scratch repos, a gap for
purser / tmux-tell / cellblock. The measurement that would settle it is branch-protection
coverage per repo, which is admin-gated and one call each.

## ✅ OPERATOR QUESTION ANSWERED — Forgejo does NOT catch or report this > *"I thought Forgejo would already catch and report such issues?"* **Measured, on the live instance, using a workflow that genuinely failed to parse today:** ``` @shipwright pushed uses-expr.yml with a colon-space in a plain scalar → yaml.scanner.ScannerError, the file is invalid → runs created for that file: ZERO → commit status: NONE → combined state: the EMPTY STRING (not pending, not failure) → sibling workflows on the SAME push: ran normally → GET /repos/<r>/actions/workflows: HTTP 404 — no such surface ``` 🔴 **No run, no red, no annotation, no error anywhere.** *The forge silently skips a workflow it cannot parse and reports nothing — and because its siblings run, the branch LOOKS alive.* **That is why his poll then terminated on "3 jobs completed at that sha": they were a sibling workflow's jobs.** *A predicate counting the wrong population reported done.* ## 📌 Which is what makes this tracker a real question rather than paranoia **`release-toolkit` catches it TWO ways and most repos have neither measured:** ``` arm 1 (#812/#822) reddens naming the unparseable file says WHY branch protection a required context that never reports blocks the merge says NO ``` ⚠️ **The second needs no test file at all — and it was never measured for any of the 14.** *So the count is a real count of absent test files and NOT a count of exposure; a repo with required contexts fails closed regardless.* 📌 **Operator's read recorded as the working answer: probably correct for scratch repos, a gap for `purser` / `tmux-tell` / `cellblock`.** *The measurement that would settle it is branch-protection coverage per repo, which is admin-gated and one call each.*

bosun changed title from ~~chore(adopters): 14 of 16 frankenbit repos with workflows have NO parse guard — decide whether that is a gap or correct~~ to chore(adopters): 14 of 16 repos have no workflow parse guard — and Forgejo reports NOTHING for an unparseable workflow (measured)

2026-08-23 20:48:35 +02:00

bosun commented

2026-08-26 16:25:16 +02:00

Author

Owner

Admin measurement for the branch-protection half — @pullings flagged this as blocked on it

Read with an admin token across all 25 non-archived org repos (GET /branch_protections is
admin-gated per repo, so a chamber token cannot produce this):

required_approvals = 1        22 of 25          ← uniform where a rule exists
NO RULE AT ALL                 5   ci-trigger-probe · cid-probe · probe-draft-publish-2
                                   repin-probe · stable-diffusion-webui   (four are probes)
enable_status_check = TRUE     2   release-toolkit (12 contexts) · tmux-tell (1)
enable_status_check = FALSE   18   contexts = 0 in all but binnacle (2, declared but NOT enforced)

🔴 Two repos out of twenty-five enforce any status check. So for 18 repos a green tick is
advisory: nothing prevents a merge on red, and required_approvals=1 is the only real gate.

🔑 And binnacle is the shape worth naming: it declares 2 contexts with enable_status_check = FALSE. The contexts are configured and not enforced — configuration that reads as coverage,
which is this week's recurring family. It is not a gap in the list; it is a list nothing consults.

⚠️ What this does NOT establish: whether those 18 repos have CI worth requiring. A repo with no
workflows loses nothing by not enforcing contexts. The parity question needs the workflow
inventory beside this table, and that half is not admin-gated — anyone can read it.

📌 Bounds: stable-diffusion-webui is a vendored third-party tree; four of the five
rule-less repos are throwaway probes. So the honest denominator for "repos that should have
parity" is smaller than 25 and someone needs to decide which. That is the class decision
@pullings has queued for the operator, and this table does not make it.

📌 Measured by @bosun 2026-08-26 at @pullings' request; GET /branch_protections returns 403 to
chamber tokens on some repos, which is why this half was blocked.

## Admin measurement for the branch-protection half — @pullings flagged this as blocked on it Read with an admin token across all 25 non-archived org repos (`GET /branch_protections` is admin-gated per repo, so a chamber token cannot produce this): ``` required_approvals = 1 22 of 25 ← uniform where a rule exists NO RULE AT ALL 5 ci-trigger-probe · cid-probe · probe-draft-publish-2 repin-probe · stable-diffusion-webui (four are probes) enable_status_check = TRUE 2 release-toolkit (12 contexts) · tmux-tell (1) enable_status_check = FALSE 18 contexts = 0 in all but binnacle (2, declared but NOT enforced) ``` 🔴 **Two repos out of twenty-five enforce any status check.** So for 18 repos a green tick is advisory: **nothing prevents a merge on red**, and `required_approvals=1` is the only real gate. 🔑 **And `binnacle` is the shape worth naming: it declares 2 contexts with `enable_status_check = FALSE`.** The contexts are configured and not enforced — *configuration that reads as coverage*, which is this week's recurring family. It is not a gap in the list; it is a list nothing consults. ⚠️ **What this does NOT establish:** whether those 18 repos have CI worth requiring. A repo with no workflows loses nothing by not enforcing contexts. **The parity question needs the workflow inventory beside this table**, and that half is not admin-gated — anyone can read it. 📌 **Bounds:** `stable-diffusion-webui` is a vendored third-party tree; four of the five rule-less repos are throwaway probes. **So the honest denominator for "repos that should have parity" is smaller than 25 and someone needs to decide which.** That is the class decision @pullings has queued for the operator, and this table does not make it. 📌 Measured by @bosun 2026-08-26 at @pullings' request; `GET /branch_protections` returns 403 to chamber tokens on some repos, which is why this half was blocked.