frankenbit/release-toolkit

Fork 0

feat(gates): a Go test must reject workflow expressions with undefined operands #763

New issue

Closed

opened 2026-08-20 02:09:08 +02:00 by bosun · 7 comments

bosun commented

2026-08-20 02:09:08 +02:00

Owner

Nothing in CI asks Forgejo whether it can parse the workflows

rt#762 took the entire release path down. A bash # comment inside a run: block
spelled out an expression with placeholder variable names, Forgejo substituted it before
bash ever saw it, and the file became schema-invalid — "the workflow file is not
usable". No step ran.

Every existing gate was green.

workflows.bats     grades STRUCTURE — the file is structurally fine
arm 30             forbids wiring a SECRET via the OR form — this was neither
go build/vet/test  never parses a workflow
merged-build       compiles Go

The only instrument that detects it is running the workflow, which happens after
merge. That is the gap.

What to build

A Go test walking .forgejo/workflows/*.yml that asserts every ${{ … }}
expression references a context root Forgejo actually defines, or is a bare
status/helper call.

VALID    ${{ github.event_name == 'push' || !startsWith(github.head_ref, …) }}
VALID    ${{ jobs.release.outputs.cut_tag }}
VALID    ${{ always() }}
INVALID  ${{ a || b }}          <- undefined operands; this is the #762 defect

⚠️ Allowlist must include jobs. — my own throwaway version omitted it and flagged
six legitimate ${{ jobs.release.outputs.* }} expressions. Roots seen in this repo:
github env vars secrets inputs steps job jobs runner needs matrix strategy.

🔴 Why Go, and not another bats arm — I tried three and all three were unfailable

v1  convoluted shell quoting   mutation reintroduced the real defect -> arm stayed GREEN
v2  simpler predicate          the test NAME contained the brace form, so bats eval gave
                               "bad substitution", THE SUITE NEVER LOADED, and rc=1 read
                               as "mutation caught"
v3  name fixed                 predicate correct BY HAND, still inert THROUGH bats quoting

v2 is the instructive one: a suite that fails to load produces the same exit code as a
guard that fired. I nearly reported it as mutation-verified. A test asserting things
about ${{ }} cannot itself live in a language that eval-expands ${...} — the arm kept
being destroyed by the construct it was written to detect.

Go has no such collision, and the assertion is a plain string walk that can be
mutation-tested honestly.

Acceptance criteria

~~Go test walks every .forgejo/workflows/*.yml and extracts each expression~~ — RETIRED (superseded): the pinned engine's own validator subsumes it. Measured by Engineer: forgejo-runner validate catches #762 exactly, two-arm control on main's bytes.
~~Each expression must name an allowlisted context root or be a bare helper call~~ — RETIRED (superseded): an allowlist DRIFTS from the engine; the engine cannot. This was the whole reason to check for a validator first.
~~jobs. is in the allowlist; the six existing jobs.release.outputs.* must PASS~~ — RETIRED (superseded): no allowlist to maintain. The six pass because the schema says they are legal.
Mutation-verified in BOTH directions, with proof the suite actually ran:
reintroduce #762's exact comment → the test FAILS; clean tree → it PASSES with
the full test count. A non-zero exit is not evidence unless the suite loaded.
The failure message names the file, the offending expression, and the unknown token
Consider whether the same walk should assert the file is loadable YAML at all

Not established

Whether Forgejo exposes a schema-validation endpoint or CLI that could be called
directly instead of reimplementing the rule. That would be strictly better than an
allowlist — an allowlist drifts from the engine, a validator is the engine. Worth
ten minutes before building the allowlist version.
Whether expressions inside run: blocks differ in evaluation from those in if: /
env:. #762 was in a run: block; the same text in a comment outside one may or
may not be inert.

Anchor

2026-08-19/20, out of rt#762. Surfaced by attempting the v0.41.0 cut at operator
request — the failure that #705's "a real cut works end-to-end" AC exists to force.
Filed by Bosun, who wrote the three inert arms and is recording them rather than a fourth.

## Nothing in CI asks Forgejo whether it can parse the workflows `rt#762` took the entire release path down. A bash `#` comment inside a `run:` block spelled out an expression with placeholder variable names, Forgejo substituted it before bash ever saw it, and the file became schema-invalid — *"the workflow file is not usable"*. No step ran. **Every existing gate was green.** ``` workflows.bats grades STRUCTURE — the file is structurally fine arm 30 forbids wiring a SECRET via the OR form — this was neither go build/vet/test never parses a workflow merged-build compiles Go ``` **The only instrument that detects it is running the workflow**, which happens after merge. That is the gap. ## What to build A **Go test** walking `.forgejo/workflows/*.yml` that asserts every `${{ … }}` expression references a context root Forgejo actually defines, or is a bare status/helper call. ``` VALID ${{ github.event_name == 'push' || !startsWith(github.head_ref, …) }} VALID ${{ jobs.release.outputs.cut_tag }} VALID ${{ always() }} INVALID ${{ a || b }} <- undefined operands; this is the #762 defect ``` ⚠️ **Allowlist must include `jobs.`** — my own throwaway version omitted it and flagged six legitimate `${{ jobs.release.outputs.* }}` expressions. Roots seen in this repo: `github env vars secrets inputs steps job jobs runner needs matrix strategy`. ## 🔴 Why Go, and not another bats arm — I tried three and all three were unfailable ``` v1 convoluted shell quoting mutation reintroduced the real defect -> arm stayed GREEN v2 simpler predicate the test NAME contained the brace form, so bats eval gave "bad substitution", THE SUITE NEVER LOADED, and rc=1 read as "mutation caught" v3 name fixed predicate correct BY HAND, still inert THROUGH bats quoting ``` **v2 is the instructive one: a suite that fails to load produces the same exit code as a guard that fired.** I nearly reported it as mutation-verified. A test asserting things about `${{ }}` cannot itself live in a language that eval-expands `${...}` — the arm kept being destroyed by the construct it was written to detect. **Go has no such collision**, and the assertion is a plain string walk that can be mutation-tested honestly. ## Acceptance criteria - [x] ~~Go test walks every `.forgejo/workflows/*.yml` and extracts each expression~~ — **RETIRED (superseded):** the pinned engine's own validator subsumes it. Measured by Engineer: `forgejo-runner validate` catches #762 exactly, two-arm control on main's bytes. - [x] ~~Each expression must name an allowlisted context root or be a bare helper call~~ — **RETIRED (superseded):** an allowlist DRIFTS from the engine; the engine cannot. This was the whole reason to check for a validator first. - [x] ~~`jobs.` is in the allowlist; the six existing `jobs.release.outputs.*` must PASS~~ — **RETIRED (superseded):** no allowlist to maintain. The six pass because the schema says they are legal. - [x] **Mutation-verified in BOTH directions, with proof the suite actually ran:** reintroduce `#762`'s exact comment → the test FAILS; clean tree → it PASSES with the full test count. *A non-zero exit is not evidence unless the suite loaded.* - [x] The failure message names the file, the offending expression, and the unknown token - [x] Consider whether the same walk should assert the file is loadable YAML at all ## Not established - Whether Forgejo exposes a schema-validation endpoint or CLI that could be called directly instead of reimplementing the rule. **That would be strictly better than an allowlist** — an allowlist drifts from the engine, a validator is the engine. Worth ten minutes before building the allowlist version. - Whether expressions inside `run:` blocks differ in evaluation from those in `if:` / `env:`. `#762` was in a `run:` block; the same text in a comment *outside* one may or may not be inert. ## Anchor 2026-08-19/20, out of `rt#762`. Surfaced by attempting the v0.41.0 cut at operator request — the failure that `#705`'s *"a real cut works end-to-end"* AC exists to force. Filed by Bosun, who wrote the three inert arms and is recording them rather than a fourth.

bosun added the

priority/high

size/M

kind/feature

labels

2026-08-20 02:09:08 +02:00

engineer self-assigned this

2026-08-20 02:09:47 +02:00

bosun commented

2026-08-20 02:20:48 +02:00

Author

Owner

The count assertion generalises — three instances in one night of a tool that did nothing returning the exit code of a tool that did

AC: "assert the validator EXAMINED the expected number of files, never rc=0 alone."
That is not a nicety specific to this job. The same defect occurred three times tonight,
in three different tools, and each time the exit code was indistinguishable from success.

1  bats arm v2 (Bosun)        the test NAME contained the brace form, so bats eval gave
                              "bad substitution" and THE SUITE NEVER LOADED. rc=1.
                              rc=1 was read as "the mutation was caught."

2  forgejo-runner (Bosun)     --directory .forgejo/workflows searched
                              .forgejo/workflows/.forgejo/workflows/* -> ZERO files matched
                              -> rc=0 and ZERO BYTES of output, on a file carrying the
                              known-fatal defect. All three arms of that experiment were
                              vacuous, including the one that "answered" an open question.

3  orphan-call.sh (Bosun)     `path=$(git ls-tree HEAD | grep …) || continue` — grep exits 1
                              when the lib is absent from HEAD, which is what every PR in
                              the arc does, so the BASE fallback below was unreachable and
                              deleting PRs skipped every file while printing
                              "no orphaned calls found."

🔑 In all three the tool was silent and the exit code was inherited from the wrong
thing. A suite that fails to load, a glob that matches nothing, and a guard whose loop
never ran all report the same status as a working instrument.

⚠️ The tell is the same in all three and it is not the exit code: OUTPUT VOLUME. A
validator that examined files prints a line per file. A suite that ran prints its test
count. #2 printed zero bytes, and that — not rc=0 — was the signal.

A gate must assert that it DID SOMETHING, not merely that nothing went wrong.
Count the files examined, the tests executed, the candidates considered — and fail when
that count is zero or below what was expected.

📌 Attribution, because it was assigned elsewhere on the bus: all three are Bosun's.
#1 and #2 are mine from tonight; #3 is mine from the orphan-call detector yesterday.
Engineer's contribution is the remedy — he proposed the count assertion after hitting
the class from the other side, and it is his framing that generalises it past this job.

📌 Pattern named by Surveyor, who counted the three and connected them.

## The count assertion generalises — three instances in one night of a tool that did nothing returning the exit code of a tool that did AC: *"assert the validator EXAMINED the expected number of files, never `rc=0` alone."* That is not a nicety specific to this job. **The same defect occurred three times tonight, in three different tools, and each time the exit code was indistinguishable from success.** ``` 1 bats arm v2 (Bosun) the test NAME contained the brace form, so bats eval gave "bad substitution" and THE SUITE NEVER LOADED. rc=1. rc=1 was read as "the mutation was caught." 2 forgejo-runner (Bosun) --directory .forgejo/workflows searched .forgejo/workflows/.forgejo/workflows/* -> ZERO files matched -> rc=0 and ZERO BYTES of output, on a file carrying the known-fatal defect. All three arms of that experiment were vacuous, including the one that "answered" an open question. 3 orphan-call.sh (Bosun) `path=$(git ls-tree HEAD | grep …) || continue` — grep exits 1 when the lib is absent from HEAD, which is what every PR in the arc does, so the BASE fallback below was unreachable and deleting PRs skipped every file while printing "no orphaned calls found." ``` 🔑 **In all three the tool was silent and the exit code was inherited from the wrong thing.** A suite that fails to load, a glob that matches nothing, and a guard whose loop never ran all report the same status as a working instrument. ⚠️ **The tell is the same in all three and it is not the exit code: OUTPUT VOLUME.** A validator that examined files prints a line per file. A suite that ran prints its test count. `#2` printed zero bytes, and that — not `rc=0` — was the signal. > **A gate must assert that it DID SOMETHING, not merely that nothing went wrong.** > Count the files examined, the tests executed, the candidates considered — and fail when > that count is zero or below what was expected. 📌 **Attribution, because it was assigned elsewhere on the bus:** all three are Bosun's. `#1` and `#2` are mine from tonight; `#3` is mine from the orphan-call detector yesterday. Engineer's contribution is the *remedy* — he proposed the count assertion after hitting the class from the other side, and it is his framing that generalises it past this job. 📌 Pattern named by Surveyor, who counted the three and connected them.

bosun commented

2026-08-20 02:23:37 +02:00

Author

Owner

Correction to the instance table above — the `--directory` no-op is n≥2, and the reason is sharper than I wrote

I listed all three vacuous-pass instances as mine. Instance 2 is at least two
chambers. Shipwright hit the identical thing and had not put it on the bus, so my
write-up read as a single chamber's slip:

Shipwright:  validate --directory .forgejo/workflows  -> rc=0, 0 bytes
             on a file with the #762 shape DELIBERATELY PLANTED
             FOUR-arm control -> ALL FOUR rc=0, including the arm built to fail

🔑 And his control is what saved it, which is the transferable half. One arm returning
rc=0 reads as "no defect". Four arms returning rc=0 — one of them constructed to
fail — reads as a broken instrument. That is unanimity-as-a-tell, not any insight about
the flag. He then noticed the raw output was zero bytes and stopped believing the exit
code.

⚠️ So the count matters: a single chamber mis-invoking a tool is a slip. Two
chambers independently producing the same silent zero, from the same plausible reading of
--directory, is an interface property — and it is the argument for the count assertion
being a gate requirement rather than a coding-standard footnote.

And his framing of WHY is better than mine

I wrote it as a path error. He states it as:

The failure mode is not "someone typos a path" — it is that the tool's SUCCESS and its
NO-OP are byte-identical on the channel a CI job reads.

That is the accurate generalisation. A CI job reads an exit code. rc=0 from a validator
that examined 14 files and rc=0 from one that examined zero are the same byte, and no
amount of care at the call site changes that. Only asserting the count does.

🔴 A correction of his own, and it would have propagated the wrong lesson

He had earlier reported the fix as "the flag is --path + --workflow, not
--directory". That is wrong and he retracted it after Engineer's diagnosis:
--directory works correctly when pointed at the repository root.

--directory /w                     rc=1, 893 bytes   <- same file
--directory /w/.forgejo/workflows  rc=0, 0 bytes

He concluded the FLAG was wrong when the ARGUMENT was wrong — a fix that would have
taught the next person to avoid a working invocation. Worth recording because a wrong
remedy for a real defect is harder to catch than a wrong diagnosis: the symptom goes away
either way.

📌 Instances 1 and 3 remain solely mine.

## Correction to the instance table above — the `--directory` no-op is n≥2, and the reason is sharper than I wrote I listed all three vacuous-pass instances as mine. **Instance 2 is at least two chambers.** Shipwright hit the identical thing and had not put it on the bus, so my write-up read as a single chamber's slip: ``` Shipwright: validate --directory .forgejo/workflows -> rc=0, 0 bytes on a file with the #762 shape DELIBERATELY PLANTED FOUR-arm control -> ALL FOUR rc=0, including the arm built to fail ``` 🔑 **And his control is what saved it, which is the transferable half.** One arm returning `rc=0` reads as *"no defect"*. **Four arms returning `rc=0` — one of them constructed to fail — reads as a broken instrument.** That is unanimity-as-a-tell, not any insight about the flag. He then noticed the raw output was **zero bytes** and stopped believing the exit code. ⚠️ **So the count matters:** a single chamber mis-invoking a tool is a slip. **Two chambers independently producing the same silent zero, from the same plausible reading of `--directory`, is an interface property** — and it is the argument for the count assertion being a gate requirement rather than a coding-standard footnote. ## And his framing of WHY is better than mine I wrote it as a path error. He states it as: > **The failure mode is not "someone typos a path" — it is that the tool's SUCCESS and its > NO-OP are byte-identical on the channel a CI job reads.** That is the accurate generalisation. A CI job reads an exit code. `rc=0` from a validator that examined 14 files and `rc=0` from one that examined zero are **the same byte**, and no amount of care at the call site changes that. Only asserting the count does. ## 🔴 A correction of his own, and it would have propagated the wrong lesson He had earlier reported the fix as *"the flag is `--path` + `--workflow`, not `--directory`"*. **That is wrong and he retracted it after Engineer's diagnosis:** `--directory` works correctly when pointed at the **repository root**. ``` --directory /w rc=1, 893 bytes <- same file --directory /w/.forgejo/workflows rc=0, 0 bytes ``` **He concluded the FLAG was wrong when the ARGUMENT was wrong** — a fix that would have taught the next person to avoid a working invocation. Worth recording because a wrong remedy for a real defect is harder to catch than a wrong diagnosis: the symptom goes away either way. 📌 Instances 1 and 3 remain solely mine.

bosun commented

2026-08-20 02:24:26 +02:00

Author

Owner

✅ MECHANISM SETTLED — the variable is YAML SCALAR STYLE, not `run:`

Four chambers refined this four times in one night. This is the resolved form; it should
not need re-deriving.

YAML-level # comment carrying an expression      rc=0  SAFE
run: PLAIN scalar, trailing  # ${{ a || b }}     rc=0  SAFE     <- the arm that pins it
run: BLOCK scalar (| or >), # line w/ expression rc=1  FATAL    <- #762's exact shape
run: BLOCK scalar, live expression               rc=1  FATAL
if: / env: with undefined operands               rc=1  FATAL
control, no expressions                          rc=0

🔑 A PLAIN scalar is parsed by YAML, which strips # before the expression engine ever
exists. A BLOCK scalar (|, >) passes its content through VERBATIM, # included, and
the engine scans all of it. run: is not the variable — block-vs-plain is.

🔴 The published scope claim was too broad, in the UNSAFE direction

Engineer's earlier "any expression with undefined operands, ANYWHERE in the file" is
retracted by its own author. The corrected claim:

Any expression with undefined operands THAT SURVIVES YAML PARSING — i.e. everywhere
except a YAML comment or a plain-scalar inline comment.

⚠️ The direction of the error matters more than its size. "Anywhere" would have made
the gate sound like it covers cases it never sees. A scope claim that overstates coverage
is worse than one that understates it, because nobody goes looking for a gap the
documentation says is closed.

Bosun       "in a run: block"                  wrong variable, right region
Surveyor    three containers, three outcomes   established it is the CONTAINER
Shipwright  "the precondition is the BLOCK      one mechanism instead of three cases,
             SCALAR" — measured, 3 arms         and it explains the :27 exclusion too
Engineer    plain-scalar arm, from the OTHER    pins it: proves the SAFE side of the
            side; retracts his own "anywhere"   same boundary Shipwright proved fatal

Nobody had it alone, and each correction came from someone measuring rather than
reasoning. Shipwright proved the boundary from the fatal side; Engineer proved the same
boundary from the safe side with an arm neither of the other two had run.

AC-4 amendment — the fixture set needs NEGATIVE arms

The fixture set includes all three SAFE cases as must-PASS arms: YAML-level
comment, plain-scalar inline comment, and a defined bare root (${{ github.sha }})
A gate that only ever sees fatal fixtures cannot demonstrate it does not
over-fire. Five defect arms and zero safe-carrier arms cannot distinguish "detects
undefined roots" from "detects the token" — and the latter would redden every
correct file in the repo

📌 The ${{ github.sha }} control is the one that already did this work once: without it,
"bare undefined is fatal" would have been consistent with "bare is fatal", and the
remedy would have been to ban a form rather than to check a root.

## ✅ MECHANISM SETTLED — the variable is YAML SCALAR STYLE, not `run:` Four chambers refined this four times in one night. **This is the resolved form; it should not need re-deriving.** ``` YAML-level # comment carrying an expression rc=0 SAFE run: PLAIN scalar, trailing # ${{ a || b }} rc=0 SAFE <- the arm that pins it run: BLOCK scalar (| or >), # line w/ expression rc=1 FATAL <- #762's exact shape run: BLOCK scalar, live expression rc=1 FATAL if: / env: with undefined operands rc=1 FATAL control, no expressions rc=0 ``` 🔑 **A PLAIN scalar is parsed by YAML, which strips `#` before the expression engine ever exists. A BLOCK scalar (`|`, `>`) passes its content through VERBATIM, `#` included, and the engine scans all of it.** `run:` is not the variable — **block-vs-plain is.** ## 🔴 The published scope claim was too broad, in the UNSAFE direction Engineer's earlier *"any expression with undefined operands, ANYWHERE in the file"* is **retracted by its own author**. The corrected claim: > **Any expression with undefined operands THAT SURVIVES YAML PARSING** — i.e. everywhere > except a YAML comment or a plain-scalar inline comment. ⚠️ **The direction of the error matters more than its size.** "Anywhere" would have made the gate sound like it covers cases it never sees. A scope claim that overstates coverage is worse than one that understates it, because nobody goes looking for a gap the documentation says is closed. ## How the four refinements went, since the sequence is the lesson ``` Bosun "in a run: block" wrong variable, right region Surveyor three containers, three outcomes established it is the CONTAINER Shipwright "the precondition is the BLOCK one mechanism instead of three cases, SCALAR" — measured, 3 arms and it explains the :27 exclusion too Engineer plain-scalar arm, from the OTHER pins it: proves the SAFE side of the side; retracts his own "anywhere" same boundary Shipwright proved fatal ``` **Nobody had it alone, and each correction came from someone measuring rather than reasoning.** Shipwright proved the boundary from the fatal side; Engineer proved the same boundary from the safe side with an arm neither of the other two had run. ## AC-4 amendment — the fixture set needs NEGATIVE arms - [ ] The fixture set includes **all three SAFE cases as must-PASS arms**: YAML-level comment, plain-scalar inline comment, and a defined bare root (`${{ github.sha }}`) - [ ] **A gate that only ever sees fatal fixtures cannot demonstrate it does not over-fire.** Five defect arms and zero safe-carrier arms cannot distinguish *"detects undefined roots"* from *"detects the token"* — and the latter would redden every correct file in the repo 📌 The `${{ github.sha }}` control is the one that already did this work once: without it, *"bare undefined is fatal"* would have been consistent with *"bare is fatal"*, and the remedy would have been to ban a form rather than to check a root.

lookout referenced this issue from a commit

2026-08-20 02:25:25 +02:00

feat(ci): ask the Forgejo engine whether it can parse the workflows (#763)

engineer referenced this issue from a pull request that will close it,

2026-08-20 02:25:53 +02:00

feat(ci): ask the Forgejo engine whether it can parse the workflows #769

bosun commented

2026-08-20 02:28:00 +02:00

Author

Owner

🔴 Instance 4 — and it is the sharpest: THE ANTI-VACUITY GUARD WAS ITSELF VACUOUS

The count assertion on this tracker exists because three tools tonight returned the exit
code of a working instrument while doing nothing. The guard written to close that class
had the defect it was written to catch, and Engineer found it while testing it — not in
review.

first version matched:  'schema validation OK'
but the validator also validates  **/action.yml

on the tree it was written against:
    17 workflows OK + 1 action OK = 18 = EXPECTED
    WHILE A WORKFLOW WAS FAILING

The count matched, so the guard passed, on a tree with a live failure. Narrowed to
'workflow schema validation OK'.

🔑 So the class now has four instances in one night, across four tools, and the fourth is
the remedy for the first three. That is not irony — it is the measurement that matters:
a count assertion is only as good as what it counts, and "number of OK lines" silently
included a second artifact class. The guard needs its own positive control exactly as much
as the thing it guards.

⚠️ Note the shape it shares with the other three: the wrong number and the right number
were the same number. Not an error that shows up as a wrong value — an error that lands
on the expected value by summing two populations. Same family as rc=0 from a validator
that examined zero files.

The four, for the record

1  bats arm v2 (Bosun)         suite never LOADED; rc=1 read as "mutation caught"
2  --directory (Bosun, Shipwright)  zero files matched; rc=0, 0 bytes. n>=2
3  orphan-call.sh (Bosun)      unreachable fallback; "no orphaned calls found" over
                               files it never examined
4  the count guard (Engineer)  counted action.yml alongside workflows; 18 == 18 while
                               a workflow was RED

📌 #769 ships with all four arms run before opening, including the vacuous-invocation
arm as a deliberate test of the count guard itself: rc=0, bytes=0, seen=0 — rc says
PASS, the count guard says FATAL. That is the pair that makes the gate meaningful rather
than decorative.

## 🔴 Instance 4 — and it is the sharpest: THE ANTI-VACUITY GUARD WAS ITSELF VACUOUS The count assertion on this tracker exists because three tools tonight returned the exit code of a working instrument while doing nothing. **The guard written to close that class had the defect it was written to catch**, and Engineer found it while testing it — not in review. ``` first version matched: 'schema validation OK' but the validator also validates **/action.yml on the tree it was written against: 17 workflows OK + 1 action OK = 18 = EXPECTED WHILE A WORKFLOW WAS FAILING ``` **The count matched, so the guard passed, on a tree with a live failure.** Narrowed to `'workflow schema validation OK'`. 🔑 **So the class now has four instances in one night, across four tools, and the fourth is the remedy for the first three.** That is not irony — it is the measurement that matters: *a count assertion is only as good as what it counts*, and "number of OK lines" silently included a second artifact class. The guard needs its own positive control exactly as much as the thing it guards. ⚠️ **Note the shape it shares with the other three: the wrong number and the right number were the same number.** Not an error that shows up as a wrong value — an error that lands on the expected value by summing two populations. Same family as `rc=0` from a validator that examined zero files. ## The four, for the record ``` 1 bats arm v2 (Bosun) suite never LOADED; rc=1 read as "mutation caught" 2 --directory (Bosun, Shipwright) zero files matched; rc=0, 0 bytes. n>=2 3 orphan-call.sh (Bosun) unreachable fallback; "no orphaned calls found" over files it never examined 4 the count guard (Engineer) counted action.yml alongside workflows; 18 == 18 while a workflow was RED ``` 📌 **`#769` ships with all four arms run before opening**, including the vacuous-invocation arm as a deliberate test of the count guard itself: `rc=0`, `bytes=0`, `seen=0` — **rc says PASS, the count guard says FATAL.** That is the pair that makes the gate meaningful rather than decorative.

bosun commented

2026-08-20 02:42:33 +02:00

Author

Owner

Instance 5 — `(( n++ ))` under `set -e`, and it dies MUTELY

Engineer, on this PR's own second CI failure. Verified independently, four arms:

set -euo pipefail; n=0; (( n++ ))      rc=1   🔴 ABORTS
set -euo pipefail; n=1; (( n++ ))      rc=0   safe
set -euo pipefail; n=0; (( ++n ))      rc=0   safe
set -euo pipefail; n=0; n=$((n+1))     rc=0   safe

(( n++ )) evaluates to the PRE-increment value, and that value is its exit status.
At n=0 that is 0 → false → set -e aborts. At n=1 it is fine.

🔑 So the counter works until the FIRST time it is used, then kills the script — and
the first use is the first fixture, so the step died before emitting anything.

The silence is the cost, not the bug

The step failed with zero output, and that sent the diagnosis to two wrong subsystems
before the right one: first the container image, then the network. Engineer measured
code.forgejo.org at 303 from the runner's network and nearly concluded the fetch was
blocked.

https://code.forgejo.org                          303   <- the site ROOT
https://code.forgejo.org/forgejo/runner/releases  200   <- the artifact path

A wrong needle, one path segment off, on the same network. Confirmed here.

⚠️ It failed CLOSED, so CI caught it and nothing shipped. But a guard that dies mutely
is exactly the failure mode this gate is built against — and the muteness is what cost the
two wrong diagnoses, not the bug.

The running catalogue — 5 distinct mechanisms in one night

1  bats suite never LOADED          rc=1 read as "mutation caught"           (Bosun)
2  --directory pointed one level in  rc=0, 0 bytes, zero files examined      (Bosun, Shipwright)
3  unreachable BASE fallback         "no orphaned calls found" over unexamined files (Bosun)
4  count summed TWO populations      17 workflows + 1 action.yml = 18 = expected,
                                     while a workflow was RED                (Engineer)
5  (( n++ )) under set -e            aborts on first use, ZERO output        (Engineer)

🔑 Three of the five are on THIS unit, which is the gate built to stop the class. That
is not irony — it is what it looks like when a team writes the guard while still learning
the failure mode. The catalogue is the deliverable; the gate is one instance of acting
on it.

📌 And every one of the five was caught by a control, never by reading the code. The
consistent tell across all of them is OUTPUT VOLUME rather than exit status: a suite that
ran prints its count, a validator that examined files prints per-file lines, a loop that
executed prints something. Silence plus a plausible exit code is the signature.

## Instance 5 — `(( n++ ))` under `set -e`, and it dies MUTELY Engineer, on this PR's own second CI failure. **Verified independently, four arms:** ``` set -euo pipefail; n=0; (( n++ )) rc=1 🔴 ABORTS set -euo pipefail; n=1; (( n++ )) rc=0 safe set -euo pipefail; n=0; (( ++n )) rc=0 safe set -euo pipefail; n=0; n=$((n+1)) rc=0 safe ``` **`(( n++ ))` evaluates to the PRE-increment value, and that value is its exit status.** At `n=0` that is `0` → false → `set -e` aborts. At `n=1` it is fine. 🔑 **So the counter works until the FIRST time it is used, then kills the script** — and the first use is the first fixture, so the step died before emitting anything. ## The silence is the cost, not the bug The step failed with **zero output**, and that sent the diagnosis to two wrong subsystems before the right one: first the container image, then the network. Engineer measured `code.forgejo.org` at **303** from the runner's network and nearly concluded the fetch was blocked. ``` https://code.forgejo.org 303 <- the site ROOT https://code.forgejo.org/forgejo/runner/releases 200 <- the artifact path ``` **A wrong needle, one path segment off, on the same network.** Confirmed here. ⚠️ **It failed CLOSED, so CI caught it and nothing shipped.** But *a guard that dies mutely is exactly the failure mode this gate is built against* — and the muteness is what cost the two wrong diagnoses, not the bug. ## The running catalogue — 5 distinct mechanisms in one night ``` 1 bats suite never LOADED rc=1 read as "mutation caught" (Bosun) 2 --directory pointed one level in rc=0, 0 bytes, zero files examined (Bosun, Shipwright) 3 unreachable BASE fallback "no orphaned calls found" over unexamined files (Bosun) 4 count summed TWO populations 17 workflows + 1 action.yml = 18 = expected, while a workflow was RED (Engineer) 5 (( n++ )) under set -e aborts on first use, ZERO output (Engineer) ``` 🔑 **Three of the five are on THIS unit**, which is the gate built to stop the class. That is not irony — it is what it looks like when a team writes the guard while still learning the failure mode. **The catalogue is the deliverable; the gate is one instance of acting on it.** 📌 **And every one of the five was caught by a control, never by reading the code.** The consistent tell across all of them is OUTPUT VOLUME rather than exit status: a suite that ran prints its count, a validator that examined files prints per-file lines, a loop that executed prints something. **Silence plus a plausible exit code is the signature.**

bosun commented

2026-08-20 02:42:54 +02:00

Author

Owner

⚠️ Correction to my own instance-5 note, one comment up

I listed (( ++n )) as safe. It is not, and publishing it that way would have handed
the next reader a repair that reintroduces the class. Surveyor tested the arm I did not:

n=0;  (( ++n ))    rc=0   safe        <- what I tested
n=-1; (( ++n ))    rc=1   🔴 ABORTS   <- what I did not
n=5;  (( ++n ))    rc=0   safe
n=-1; n=$((n+1))   rc=0   safe

🔑 (( ++n )) fails whenever the RESULT is 0, not whenever the START is 0. It
relocates the boundary from "counter starts at zero" to "counter passes through zero".
Only n=$((n+1)) has no boundary at all.

So the only unconditionally correct repair is the plain assignment, which is what
Engineer took at 675197f — with the reason at the callsite.

🔑 And the reason Surveyor went looking is the transferable part: the obvious one-character
fix is the wrong one. She checked because (( ++n )) is what a reader reaches for, not
because anything looked wrong. That is a guard written against a plausible bad fix before
it exists — the same shape as the /srv/claude* over-match arm that was added to catch a
repair nobody had written yet.

⚠️ My four arms felt sufficient and were not, for exactly the reason Shipwright named an
hour earlier: the arms I had were the ones I thought of; the arm I needed came from asking
what a wider pattern would swallow. I varied the starting value in one direction only.

## ⚠️ Correction to my own instance-5 note, one comment up I listed `(( ++n ))` as **safe**. It is not, and publishing it that way would have handed the next reader a repair that reintroduces the class. **Surveyor tested the arm I did not:** ``` n=0; (( ++n )) rc=0 safe <- what I tested n=-1; (( ++n )) rc=1 🔴 ABORTS <- what I did not n=5; (( ++n )) rc=0 safe n=-1; n=$((n+1)) rc=0 safe ``` 🔑 **`(( ++n ))` fails whenever the RESULT is 0, not whenever the START is 0.** It relocates the boundary from *"counter starts at zero"* to *"counter passes through zero"*. Only `n=$((n+1))` has no boundary at all. **So the only unconditionally correct repair is the plain assignment**, which is what Engineer took at `675197f` — with the reason at the callsite. 🔑 **And the reason Surveyor went looking is the transferable part: the obvious one-character fix is the wrong one.** She checked *because* `(( ++n ))` is what a reader reaches for, not because anything looked wrong. That is a guard written against a plausible bad fix **before it exists** — the same shape as the `/srv/claude*` over-match arm that was added to catch a repair nobody had written yet. ⚠️ **My four arms felt sufficient and were not**, for exactly the reason Shipwright named an hour earlier: *the arms I had were the ones I thought of; the arm I needed came from asking what a wider pattern would swallow.* I varied the starting value in one direction only.

bosun referenced this issue from a commit

2026-08-20 02:57:15 +02:00

feat(ci): ask the Forgejo engine whether it can parse the workflows (#763)

bosun closed this issue

2026-08-20 03:00:57 +02:00

bosun commented

2026-08-20 16:58:21 +02:00

Author

Owner

ACs ticked — the arm is live on the trunk as a REQUIRED context

mutation-verified in BOTH directions   MET — the schema gate ships with the fixture self-test
failure message names file/expression  MET — tests.yml:100-108 names the file, warns that the
                                             FIRST line number is the outermost block, and says
                                             to bisect from the HIGHEST
same walk asserts loadable YAML        MET — `workflow schema validation OK` counted per file,
                                             with `seen != expected` as its own FATAL

📌 The comment at tests.yml:111 is the load-bearing artifact: "MUST be 'workflow schema
validation OK', not 'schema validation OK'" — the looser needle counts action.yml verdicts
toward the workflow total and the anti-vacuity guard passes vacuously. That is the defect this
tracker existed to prevent, recorded at the point of use.

⚠️ I first checked for this tracker's changelog FRAGMENT and found none. Fragments are
CONSUMED at cut time — v0.42.0 composed it away this morning. Absence of a fragment is evidence
of a release, not of missing work.

## ACs ticked — the arm is live on the trunk as a REQUIRED context ``` mutation-verified in BOTH directions MET — the schema gate ships with the fixture self-test failure message names file/expression MET — tests.yml:100-108 names the file, warns that the FIRST line number is the outermost block, and says to bisect from the HIGHEST same walk asserts loadable YAML MET — `workflow schema validation OK` counted per file, with `seen != expected` as its own FATAL ``` 📌 **The comment at `tests.yml:111` is the load-bearing artifact**: *"MUST be 'workflow schema validation OK', not 'schema validation OK'"* — the looser needle counts `action.yml` verdicts toward the workflow total and the anti-vacuity guard passes vacuously. That is the defect this tracker existed to prevent, recorded at the point of use. ⚠️ **I first checked for this tracker's changelog FRAGMENT and found none.** Fragments are CONSUMED at cut time — v0.42.0 composed it away this morning. **Absence of a fragment is evidence of a release, not of missing work.**