frankenbit/release-toolkit

Fork 0

bug(fragment-check): the local verb passes where the CI context of the same name fails #1456

New issue

Closed

opened 2026-09-07 21:38:05 +02:00 by bosun · 2 comments

bosun commented

2026-09-07 21:38:05 +02:00

Owner

rt fragment-check run locally passes on a branch whose fragment-check CI context is red, because the coverage half is skipped when FRAGMENT_CHECK_COVERAGE is unset — and the skip is disclosed inside a PASS message with exit 0.

Measured — 2026-09-07 on `#1455`

local:  rt fragment-check                                        rc=0
        "all fragments in changelog.d carry a supported kind…"
        "density PASS — 11 fragment(s) graded"
        …and, mid-paragraph:
        "fragment-coverage: SKIPPED - PR fragment coverage was not graded
         because FRAGMENT_CHECK_COVERAGE is unset"

CI:     fragment-check / check           FAILURE
        fragment-check / changelog fragment-kind   FAILURE
        cause: 5 files, none under changelog.d/, and no No-Changelog declaration

The verb and the context share a name and grade different predicates. The local one is weaker, and it is the one an author runs before pushing.

Why the disclosure does not save it

The skip line is correct, present, and sits in the middle of a long success message that also carries a density PASS and a paragraph about what is graded at cut time. A scope note inside a success message is decoration — §Mechanism design's own rule — and here it cannot change the exit status, which is 0.

The neighbouring cost, same context, one hour apart

#1449   fragments PRESENT, density failed (two >30-word sentences)
#1455   fragments ABSENT, coverage failed

Same context name, same red, opposite remedies — and a relayed diagnosis of the first sent an author to add a file he already had. Both times the first line of the output explained nothing and the answer was forty lines down.

AC

Running the verb locally either grades coverage or REFUSES to report a pass it did not grade — could-not-grade is its own state and this command already uses it elsewhere
The two failure modes are distinguishable from the context alone, or the context names which half failed — DEFERRED → #1465, which owns splitting fragment coverage and density into independently named contexts so the failing half remains visible without job logs
Whatever is chosen, an author who runs the verb before pushing is not told PASS about a predicate that was skipped

Anchor

@bosun, verifying @pullings' diagnosis of #1455 rather than relaying it — his previous one on #1449 was correct about the red and wrong about the cause, in the opposite direction.

`rt fragment-check` run locally passes on a branch whose `fragment-check` CI context is red, because the coverage half is skipped when `FRAGMENT_CHECK_COVERAGE` is unset — and the skip is disclosed inside a PASS message with exit 0. ## Measured — 2026-09-07 on `#1455` ``` local: rt fragment-check rc=0 "all fragments in changelog.d carry a supported kind…" "density PASS — 11 fragment(s) graded" …and, mid-paragraph: "fragment-coverage: SKIPPED - PR fragment coverage was not graded because FRAGMENT_CHECK_COVERAGE is unset" CI: fragment-check / check FAILURE fragment-check / changelog fragment-kind FAILURE cause: 5 files, none under changelog.d/, and no No-Changelog declaration ``` **The verb and the context share a name and grade different predicates. The local one is weaker, and it is the one an author runs before pushing.** ## Why the disclosure does not save it The skip line is correct, present, and sits in the middle of a long success message that also carries a `density PASS` and a paragraph about what is graded at cut time. **A scope note inside a success message is decoration** — §Mechanism design's own rule — and here it cannot change the exit status, which is `0`. ## The neighbouring cost, same context, one hour apart ``` #1449 fragments PRESENT, density failed (two >30-word sentences) #1455 fragments ABSENT, coverage failed ``` **Same context name, same red, opposite remedies** — and a relayed diagnosis of the first sent an author to add a file he already had. Both times the first line of the output explained nothing and the answer was forty lines down. ## AC - [x] Running the verb locally either grades coverage or REFUSES to report a pass it did not grade — could-not-grade is its own state and this command already uses it elsewhere - [x] The two failure modes are distinguishable from the context alone, or the context names which half failed — **DEFERRED → #1465**, which owns splitting fragment coverage and density into independently named contexts so the failing half remains visible without job logs - [x] Whatever is chosen, an author who runs the verb before pushing is not told PASS about a predicate that was skipped ## Anchor @bosun, verifying @pullings' diagnosis of `#1455` rather than relaying it — his previous one on `#1449` was correct about the red and wrong about the cause, in the opposite direction.

bosun added the

priority/high

size/S

kind/bug

labels

2026-09-07 21:38:05 +02:00

sentry was assigned by pullings

2026-09-07 21:54:43 +02:00

sentry referenced this issue

2026-09-07 22:08:33 +02:00

fix(fragment-check): refuse ungraded local coverage #1460

release-bot referenced this issue

2026-09-07 22:28:36 +02:00

chore(release): v0.63.0 #1448

bosun commented

2026-09-07 22:29:44 +02:00

Author

Owner

#1460 merged at 68a0b4fb. AC1 and AC3 ticked; AC2 is NOT satisfied and this stays open.

env -u FRAGMENT_CHECK_COVERAGE  rt fragment-check
  rc=2 · 207 bytes · ZERO occurrences of "PASS"
  "fragment-coverage: COULD-NOT-GRADE (rc=2) - PR coverage was not graded because
   FRAGMENT_CHECK_COVERAGE is unset; set it to required in a PR workflow, or set it
   to content for an explicit content-only check."

FRAGMENT_CHECK_COVERAGE=content   rc=0 · the SKIPPED half disclosed · density PASS
reusable-changelog-fragment-check.yml:318   FRAGMENT_CHECK_COVERAGE: required

AC1/AC3 are the substance: the verb no longer says PASS about a predicate it did not grade. An author running it bare before pushing now gets rc=2 and no PASS anywhere in the output — which is the exact case that let #1455 reach a reviewer with a red it could not reproduce locally.

AC2 is open, and it is mine to have written loosely

the two failure modes are distinguishable from the context alone, or the context names which half failed

They still are not. A coverage failure and a density failure both arrive as fragment-check / check red, and only the job output separates them — which this forge does not expose. #1449 failed one half and #1455 the other, an hour apart, and a relayed diagnosis of the first sent its author to add a file he already had.

📌 That half is not really about the local verb at all — it is the no-readable-job-log problem (#1348's family) narrowed to one context. ⚠️ I am flagging it rather than quietly dropping it: the AC as written asks for something #1460 did not attempt and was not asked to.

@sentry — your call whether AC2 is worth doing here, or should be retired as out of scope with a pointer. I am not closing it for you and I am not leaving it unremarked.

`#1460` merged at `68a0b4fb`. **AC1 and AC3 ticked; AC2 is NOT satisfied and this stays open.** ``` env -u FRAGMENT_CHECK_COVERAGE rt fragment-check rc=2 · 207 bytes · ZERO occurrences of "PASS" "fragment-coverage: COULD-NOT-GRADE (rc=2) - PR coverage was not graded because FRAGMENT_CHECK_COVERAGE is unset; set it to required in a PR workflow, or set it to content for an explicit content-only check." FRAGMENT_CHECK_COVERAGE=content rc=0 · the SKIPPED half disclosed · density PASS reusable-changelog-fragment-check.yml:318 FRAGMENT_CHECK_COVERAGE: required ``` **AC1/AC3 are the substance: the verb no longer says PASS about a predicate it did not grade.** An author running it bare before pushing now gets `rc=2` and no PASS anywhere in the output — which is the exact case that let `#1455` reach a reviewer with a red it could not reproduce locally. ## AC2 is open, and it is mine to have written loosely > *the two failure modes are distinguishable from the context alone, or the context names which half failed* **They still are not.** A coverage failure and a density failure both arrive as `fragment-check / check` red, and only the job output separates them — which this forge does not expose. **`#1449` failed one half and `#1455` the other, an hour apart, and a relayed diagnosis of the first sent its author to add a file he already had.** 📌 **That half is not really about the local verb at all — it is the no-readable-job-log problem (`#1348`'s family) narrowed to one context.** ⚠️ **I am flagging it rather than quietly dropping it: the AC as written asks for something `#1460` did not attempt and was not asked to.** @sentry — **your call whether AC2 is worth doing here, or should be retired as out of scope with a pointer.** *I am not closing it for you and I am not leaving it unremarked.*

bosun commented

2026-09-08 00:08:58 +02:00

Author

Owner

Closing on @sentry's decision, 2026-09-08 00:07 CEST.

AC1 and AC3 are satisfied by #1460 at 68a0b4fb — the verb refuses rather than reporting a pass it did not grade:

env -u FRAGMENT_CHECK_COVERAGE  rt fragment-check
  rc=2 · ZERO occurrences of "PASS" in the output
FRAGMENT_CHECK_COVERAGE=content   rc=0 · the skipped half disclosed

AC2 is DEFERRED to #1465, not retired. @sentry's ruling, and his reasoning is the right one: the criterion is valid — a reader must be able to tell which half failed — but delivering it means splitting the job into two independently named contexts, which is separate work. #1460 was never asked to do that.

The AC was written loosely by me. It asked for something the change did not attempt, and I flagged it rather than quietly grading against it; @sentry decided the disposition, which was his to make.

#1465 carries it, and records the sibling instance: rt#1463 is the same shape in readme-pin-check.yml — one job, one context, two graders that fail for unrelated reasons. There the bundling prevents a safe half from being required; here it costs diagnosis. Two instances is a class worth catching at design time.

Closing on @sentry's decision, 2026-09-08 00:07 CEST. **AC1 and AC3 are satisfied by `#1460` at `68a0b4fb`** — the verb refuses rather than reporting a pass it did not grade: ``` env -u FRAGMENT_CHECK_COVERAGE rt fragment-check rc=2 · ZERO occurrences of "PASS" in the output FRAGMENT_CHECK_COVERAGE=content rc=0 · the skipped half disclosed ``` **AC2 is DEFERRED to #1465, not retired.** @sentry's ruling, and his reasoning is the right one: the criterion is valid — a reader must be able to tell which half failed — but delivering it means splitting the job into two independently named contexts, which is separate work. `#1460` was never asked to do that. The AC was written loosely by me. It asked for something the change did not attempt, and I flagged it rather than quietly grading against it; @sentry decided the disposition, which was his to make. #1465 carries it, and records the sibling instance: `rt#1463` is the same shape in `readme-pin-check.yml` — one job, one context, two graders that fail for unrelated reasons. There the bundling prevents a safe half from being *required*; here it costs *diagnosis*. Two instances is a class worth catching at design time.

bosun closed this issue

2026-09-08 00:08:58 +02:00