test: Layer-2/3 cut-safeguard FAIL-path coverage (the safeguard's protective behavior is currently untested) #64

New issue

Closed

opened 2026-06-25 17:03:14 +02:00 by surveyor · 0 comments

surveyor commented

2026-06-25 17:03:14 +02:00

Owner

Why this exists now

Slice 1b (PR #63) added the three-layer cut-safeguard to release-decide.sh. The whole point of the slice (Surveyor a315 catch on slice 1): subject-regex-alone is insufficient once act on decision is live — a false-subject-match must NOT false-CUT. Layer 2 (branch-source-check) + Layer 3 (author-identity) exist precisely to flip cut→update when a non-rolling merge happens to carry a chore(release): prepare vX.Y.Z subject.

The slice-1b tests verify the safeguard is wired, but not that it works:

All 3 release-decide.bats slice-1b tests run --dry-run, which sets FORGEJO_API_DRY_RUN → Layer 2 + Layer 3 both return skip → skip is treated as pass → mode=cut.
So the tests exercise the cut-anyway path (skip→cut) and the log framing — NOT the protective path (fail→mode=update with safeguard_fail=layer{2,3} surfaced).

In other words: the safeguard's actual protective behavior — the reason the slice exists — has no unit coverage. A regression that broke Layer 2's fail branch would not be caught by the current suite.

Why it was deferred (and why that's defensible, not blocking)

The wiring is verifiable by reading and was verified at source in the #63 review — the LAYER2 != fail && LAYER3 != fail → cut gate and the SAFEGUARD_FAIL emit are correct.
The fail-direction is safe: a Layer-2 regression either fails-closed (→ cut never fires → slice-4 dogfood cut catches it immediately) or fails-open (→ degrades to v0.3.x cut-anyway, no worse than before the safeguard existed).
Slice 4 (dogfood cut) is the empirical end-to-end gate that WILL exercise the real Layer-2 pass-path against the live Forgejo API.

So this is a unit-coverage gap with a downstream empirical backstop — not a correctness hole. Filing per deferral-discipline so the gap is substrate-recorded rather than living only as a PR-body bullet.

What a fix covers

Layer 2 FAIL: merged PR's head.ref != release-prep/rolling → mode=update, safeguard_fail=layer2 emitted
Layer 2 FAIL: no PR found for merge_commit_sha (direct push / window-miss) → mode=update, safeguard_fail=layer2
Layer 3 FAIL: release_author configured + merged PR author mismatch → mode=update, safeguard_fail=layer3
The safeguard_fail workflow output reaches the act on decision update branch's log line

Mechanism note (non-prescriptive)

PR #63 §"What this PR does NOT do" scopes this to "API mock infrastructure," which may be heavier than needed. The tests already run "$SCRIPT" as a subprocess; a lighter path could be a function-stub of forgejo_find_pr_by_merge_sha (or a PATH-shadowed curl / forgejo_api_call) returning canned JSON for the non-dry-run branch — if the script's sourcing order allows the override to stick. Whichever is cleaner in the harness is the implementer's call; the AC is fail-path coverage, not a specific mock layer.

Refs

Slice 1b: PR #63 §"What this PR does NOT do" (honest deferral; Surveyor review filed this tracker to record it)
Originating catch: Surveyor a315 (slice 1 review — subject-regex-alone insufficient)
Empirical backstop: slice 4 dogfood cut (arc #52)

Filed: 2026-06-25 from Surveyor's #63 review.

## Why this exists now Slice 1b ([PR #63](https://git.frankenbit.de/frankenbit/release-toolkit/pulls/63)) added the three-layer cut-safeguard to `release-decide.sh`. The whole point of the slice (Surveyor a315 catch on slice 1): **subject-regex-alone is insufficient once `act on decision` is live — a false-subject-match must NOT false-CUT.** Layer 2 (branch-source-check) + Layer 3 (author-identity) exist precisely to flip cut→update when a non-rolling merge happens to carry a `chore(release): prepare vX.Y.Z` subject. The slice-1b tests verify the safeguard is **wired**, but not that it **works**: - All 3 `release-decide.bats` slice-1b tests run `--dry-run`, which sets `FORGEJO_API_DRY_RUN` → Layer 2 + Layer 3 both return `skip` → `skip` is treated as pass → `mode=cut`. - So the tests exercise the **cut-anyway** path (skip→cut) and the log framing — NOT the protective path (`fail`→`mode=update` with `safeguard_fail=layer{2,3}` surfaced). In other words: the safeguard's actual protective behavior — the reason the slice exists — has no unit coverage. A regression that broke Layer 2's `fail` branch would not be caught by the current suite. ## Why it was deferred (and why that's defensible, not blocking) - The wiring is verifiable by reading and was verified at source in the #63 review — the `LAYER2 != fail && LAYER3 != fail → cut` gate and the `SAFEGUARD_FAIL` emit are correct. - The fail-direction is **safe**: a Layer-2 regression either fails-closed (→ cut never fires → slice-4 dogfood cut catches it immediately) or fails-open (→ degrades to v0.3.x cut-anyway, no worse than before the safeguard existed). - **Slice 4 (dogfood cut)** is the empirical end-to-end gate that WILL exercise the real Layer-2 pass-path against the live Forgejo API. So this is a unit-coverage gap with a downstream empirical backstop — not a correctness hole. Filing per deferral-discipline so the gap is substrate-recorded rather than living only as a PR-body bullet. ## What a fix covers - Layer 2 FAIL: merged PR's `head.ref != release-prep/rolling` → `mode=update`, `safeguard_fail=layer2` emitted - Layer 2 FAIL: no PR found for `merge_commit_sha` (direct push / window-miss) → `mode=update`, `safeguard_fail=layer2` - Layer 3 FAIL: `release_author` configured + merged PR author mismatch → `mode=update`, `safeguard_fail=layer3` - The `safeguard_fail` workflow output reaches the `act on decision` update branch's log line ## Mechanism note (non-prescriptive) PR #63 §"What this PR does NOT do" scopes this to "API mock infrastructure," which may be heavier than needed. The tests already `run "$SCRIPT"` as a subprocess; a lighter path could be a **function-stub** of `forgejo_find_pr_by_merge_sha` (or a PATH-shadowed `curl` / `forgejo_api_call`) returning canned JSON for the non-dry-run branch — if the script's sourcing order allows the override to stick. Whichever is cleaner in the harness is the implementer's call; the AC is fail-path coverage, not a specific mock layer. ## Refs - Slice 1b: [PR #63](https://git.frankenbit.de/frankenbit/release-toolkit/pulls/63) §"What this PR does NOT do" (honest deferral; Surveyor review filed this tracker to record it) - Originating catch: Surveyor a315 (slice 1 review — subject-regex-alone insufficient) - Empirical backstop: slice 4 dogfood cut (arc [#52](https://git.frankenbit.de/frankenbit/release-toolkit/issues/52)) Filed: 2026-06-25 from Surveyor's #63 review.