ci(tests): bats runs without --print-output-on-failure, so a red arm gives an assertion line and no $output — three wrong mechanisms cost four chambers an hour #686

New issue

Closed

opened 2026-08-17 23:08:46 +02:00 by bosun · 0 comments

bosun commented

2026-08-17 23:08:46 +02:00

Owner

The gap

CI runs the suite as:

# .forgejo/workflows/tests.yml
- name: run bats suite
  run: bats tests/

No --print-output-on-failure. So a red arm yields exactly this and nothing more:

not ok 694 #663 arm 2 (keying direction): stray prepare from a feature branch is REFUSED …
# (in test file tests/release-decide.bats, line 1182)
#   `[[ "$output" == *"mode=update"* ]]' failed

We learn which assertion failed. We do not learn what $output was — so we cannot see which
branch the script actually took, which is the only thing that identifies the cause.

What it cost, measured

release-toolkit#680's red at that arm consumed roughly an hour of four chambers. Three
mechanisms were proposed and all three refuted:

"layer 2 depends on credential/network timing"   refuted from the source — the stub returns
                                                 in every path once the fixture exists
"cross-test dependency at :1110"                 refuted by probe — :1110 is a HELPER, and the
                                                 variable is set inside the arm's own scope
"the seam fails open" (#685)                     REAL, but reddens at :1189 (the discriminator)
                                                 not :1182 — wrong signature, so not this cause

Every one of those could have been settled in seconds by seeing $output. The red is still
unexplained and the job cannot be re-run — every rerun endpoint 404s for a chamber token, so the
single most informative artifact is gone until someone pushes.

Fix

bats --print-output-on-failure tests/ in tests.yml — verified on merged main: present, 1 occurrence (PR#732, f6075cd4)
--show-output-of-passing-tests deliberately NOT added — verified absent on merged main (0 occurrences), so passing-test output stays hidden. This is only about failures,
and the suite is 878 arms, so unconditional output would drown the log

⚠️ Scope note: this changes nothing about whether tests pass. It changes only what a failure
tells you. A green suite is byte-identical before and after.

Why this is not a nit

🔑 A failure that cannot be diagnosed from its own log is a failure that gets re-run until it
goes away. That is the route-around-versus-fix shape in CLAUDE.md: the blockage clears, the
cause survives, and the next occurrence starts the investigation from zero. Tonight it produced
three confident wrong mechanisms from three careful people, which is the predictable output of
asking four chambers to infer a cause from an assertion line.

Anchor

2026-08-17, release-toolkit#680 task 21369. Filed by @bosun after @engineer and @surveyor each
refuted their own mechanisms by measurement and the red remained unexplained with the evidence
unavailable.

AC sweep 2026-08-19 (Bosun, operator request). Both are state-asserting and both were re-derived from merged main rather than from the PR description — grep -c on .forgejo/workflows/tests.yml for each flag. Landed via PR#732.

## The gap CI runs the suite as: ```yaml # .forgejo/workflows/tests.yml - name: run bats suite run: bats tests/ ``` **No `--print-output-on-failure`.** So a red arm yields exactly this and nothing more: ``` not ok 694 #663 arm 2 (keying direction): stray prepare from a feature branch is REFUSED … # (in test file tests/release-decide.bats, line 1182) # `[[ "$output" == *"mode=update"* ]]' failed ``` **We learn which assertion failed. We do not learn what `$output` was** — so we cannot see which branch the script actually took, which is the only thing that identifies the cause. ## What it cost, measured release-toolkit#680's red at that arm consumed roughly an hour of **four chambers**. Three mechanisms were proposed and all three refuted: ``` "layer 2 depends on credential/network timing" refuted from the source — the stub returns in every path once the fixture exists "cross-test dependency at :1110" refuted by probe — :1110 is a HELPER, and the variable is set inside the arm's own scope "the seam fails open" (#685) REAL, but reddens at :1189 (the discriminator) not :1182 — wrong signature, so not this cause ``` **Every one of those could have been settled in seconds by seeing `$output`.** The red is still unexplained and the job cannot be re-run — every rerun endpoint 404s for a chamber token, so the single most informative artifact is gone until someone pushes. ## Fix - [x] `bats --print-output-on-failure tests/` in `tests.yml` — verified on merged `main`: present, 1 occurrence (PR#732, `f6075cd4`) - [x] `--show-output-of-passing-tests` deliberately NOT added — verified absent on merged `main` (0 occurrences), so passing-test output stays hidden. This is only about failures, and the suite is 878 arms, so unconditional output would drown the log ⚠️ **Scope note: this changes nothing about whether tests pass.** It changes only what a failure tells you. A green suite is byte-identical before and after. ## Why this is not a nit 🔑 **A failure that cannot be diagnosed from its own log is a failure that gets re-run until it goes away.** That is the route-around-versus-fix shape in CLAUDE.md: the blockage clears, the cause survives, and the next occurrence starts the investigation from zero. Tonight it produced three confident wrong mechanisms from three careful people, which is the predictable output of asking four chambers to infer a cause from an assertion line. ## Anchor 2026-08-17, release-toolkit#680 task 21369. Filed by @bosun after @engineer and @surveyor each refuted their own mechanisms by measurement and the red remained unexplained with the evidence unavailable. --- **AC sweep 2026-08-19 (Bosun, operator request).** Both are state-asserting and both were re-derived from merged `main` rather than from the PR description — `grep -c` on `.forgejo/workflows/tests.yml` for each flag. Landed via PR#732.

bosun added the

priority/high

size/S

kind/chore

labels

2026-08-17 23:08:46 +02:00

bosun referenced this issue

2026-08-17 23:12:37 +02:00

fix(release-decide): discriminate cut plumbing from adopter work by file scope (#650) #680

bosun referenced this issue

2026-08-18 11:12:36 +02:00

ci(release): the cut-safeguard cannot run pre-merge — it triggers on push:main only, so it can only refuse after the merge is irreversible #688