frankenbit/release-toolkit

Fork 0

docs: the dry-run fidelity guarantee is one-directional but reads as two — it cost a merge hold #662

New issue

Closed

opened 2026-08-06 11:29:47 +02:00 by engineer · 3 comments

engineer commented

2026-08-06 11:29:47 +02:00

Owner

The claim

docs/integration.md, in the dry-run section (grep still fires in dry-run — line numbers have
moved since v0.35.0):

release-decide.sh: computes mode=noop|update|cut from the local git history. Layer-2
(branch-source-check) and Layer-3 (author-identity) cut-safeguards return skip in dry-run — a
mode=cut decision that would fire live still fires in dry-run

Correct as written. It states live cut ⇒ dry-run cut — dry-run does not suppress a real cut.
A guarantee against false negatives.

The problem

It sits four lines above:

You can verify "this is what would ship" before firing the real run.

That promises fidelity, which reads as bidirectional. So the natural reading of the earlier sentence
becomes dry-run cut ⇒ live cut — the converse, which is false, and false via the very layer the
sentence names.

dry-run   Layer 2 returns skip (:493), and skip is treated as not-fail (:645)  => mode=cut
live      Layer 2 compares head.label to ROLLING_BRANCH_NAME; mismatch => fail => mode=update

A dry-run mode=cut is not evidence of a live cut whenever the merged PR's head branch is not the
rolling branch. The doc gives readers no way to see that, because the one sentence that bounds the
guarantee is phrased as reassurance.

It cost a real merge hold today

frankenbit/purser#35 (the v0.2.0 unwind). A chamber ran decide --dry-run against the candidate
tree, got mode=cut, cited this line as establishing that the decision reproduces live, and held the
merge on the belief that merging would cut, publish (that repo resolves publish_mode: immediate),
and fire its deploy-on-release workflow.

Live, the PR's head.label is chore/unwind-v0.2.0-prep, which cannot match the
release-prep/rolling default, so Layer 2 returns fail and the result is mode=update. The hold
was unnecessary and the reasoning was sound given what the doc says.

Suggested change

State the direction explicitly, and name the false-positive case:

Layer-2 and Layer-3 return skip in dry-run, and skip is treated as not-fail. So dry-run never
suppresses a cut that would fire live — but it can report mode=cut for a run that would
resolve to mode=update live, because the live Layer-2 branch-source check is not exercised. To
predict a cut for a merge whose head branch is not the rolling branch, read the PR's head.label;
dry-run cannot answer it.

And reword or qualify the "verify what would ship" line, which is the half that makes the
one-directional guarantee read as two-directional.

The same PR surfaced #660 and #661.
Class note: a guarantee that holds in one direction, placed under a promise that sounds
bidirectional, is read as bidirectional. The fix is stating the direction, not adding emphasis.

## The claim `docs/integration.md`, in the dry-run section (grep `still fires in dry-run` — line numbers have moved since v0.35.0): > `release-decide.sh`: computes `mode=noop|update|cut` from the local git history. Layer-2 > (branch-source-check) and Layer-3 (author-identity) cut-safeguards return `skip` in dry-run — a > `mode=cut` decision that would fire live still fires in dry-run Correct as written. It states `live cut ⇒ dry-run cut` — dry-run does not **suppress** a real cut. A guarantee against false negatives. ## The problem It sits four lines above: > You can verify "this is what would ship" before firing the real run. That promises fidelity, which reads as bidirectional. So the natural reading of the earlier sentence becomes `dry-run cut ⇒ live cut` — the converse, which is **false**, and false via the very layer the sentence names. ``` dry-run Layer 2 returns skip (:493), and skip is treated as not-fail (:645) => mode=cut live Layer 2 compares head.label to ROLLING_BRANCH_NAME; mismatch => fail => mode=update ``` A dry-run `mode=cut` is not evidence of a live cut whenever the merged PR's head branch is not the rolling branch. The doc gives readers no way to see that, because the one sentence that bounds the guarantee is phrased as reassurance. ## It cost a real merge hold today `frankenbit/purser#35` (the v0.2.0 unwind). A chamber ran `decide --dry-run` against the candidate tree, got `mode=cut`, cited this line as establishing that the decision reproduces live, and held the merge on the belief that merging would cut, publish (that repo resolves `publish_mode: immediate`), and fire its deploy-on-release workflow. Live, the PR's `head.label` is `chore/unwind-v0.2.0-prep`, which cannot match the `release-prep/rolling` default, so Layer 2 returns `fail` and the result is `mode=update`. The hold was unnecessary and the reasoning was sound given what the doc says. ## Suggested change State the direction explicitly, and name the false-positive case: > Layer-2 and Layer-3 return `skip` in dry-run, and `skip` is treated as not-fail. So dry-run never > **suppresses** a cut that would fire live — but it **can report `mode=cut` for a run that would > resolve to `mode=update` live**, because the live Layer-2 branch-source check is not exercised. To > predict a cut for a merge whose head branch is not the rolling branch, read the PR's `head.label`; > dry-run cannot answer it. And reword or qualify the "verify what would ship" line, which is the half that makes the one-directional guarantee read as two-directional. ## Related - The same PR surfaced #660 and #661. - Class note: a guarantee that holds in one direction, placed under a promise that sounds bidirectional, is read as bidirectional. The fix is stating the direction, not adding emphasis.

engineer commented

2026-08-06 11:32:22 +02:00

Author

Owner

Second instance, same family — a hand-run CLI resolves defaults the workflow overrides

This issue was filed about a dry-run mode=cut being read as a live prediction. The same morning
produced two more instances of the same family, on different fields, and both misled careful readers.
Widening the issue.

The shape

A hand-run of the toolkit's CLI supplies none of the workflow's flags, so layered options fall
through to defaults the workflow overrides. The output is correct for the invocation and wrong for
the question being asked. Nothing in the output distinguishes those.

Instance 2 — `publish_mode`

A hand-run rt release --dry-run printed publish_mode: draft and "draft":true. The consumer
repo resolves immediate via layer 3 (the with: block). A hand-run has no with: block, so layer
3 is absent and it falls through to the default.

Self-caught by the chamber who ran it, who stated the rule exactly: "reading draft out of a
shell-run rt release would be reading an artifact of the invocation."

Instance 3 — the rolling branch name, and this one cost the most

release-prep.sh:328   --rolling-mode (no --branch)  -> release-prep/rolling
release-prep.sh:330   default, no flags             -> release-prep/$NEW_TAG   (v0.3.x one-shot)

The workflow runs prep in rolling mode. A hand-run without --rolling-mode names the branch
release-prep/v0.2.0.

That name was then used as the premise for what #25's branch had been. The real value was
release-prep/rolling. Consequence chain: wrong branch name → Layer 2's pass at the live run became
unexplainable → the model that could not explain the run that already happened was used to predict
the next one → a false mode=cut went into a PR body as fact, and a merge was held on it.

Three chambers spent roughly forty minutes on it.

Why these belong together

All three are the same failure: an instrument answering correctly about a configuration that is not
the one under test. The dry-run case removes a layer (Layer 2 → skip); the other two substitute a
default for a layer the workflow supplies. In each case the output carries no marker that the
resolved configuration differs from the live one.

Suggested change

Beyond the doc wording already proposed above:

Have release-prep.sh and rt release print the resolved configuration and where each value
came from — e.g. publish_mode=draft (layer: default; workflow supplies layer 3),
branch=release-prep/v0.2.0 (one-shot default; --rolling-mode gives release-prep/rolling). A
value printed without its provenance is what makes these unfalsifiable from the output.
Have --dry-run name the layers it did not exercise, rather than only the ones it did. The
existing doc sentence does this in prose for Layer 2/3; the tool itself does not.

That is the same principle as the scope-at-point-of-use rule: the pass message should name what it
did not check. Here the tool should name which configuration it resolved and which one it could not
see.

## Second instance, same family — a hand-run CLI resolves defaults the workflow overrides This issue was filed about a dry-run `mode=cut` being read as a live prediction. The same morning produced two more instances of the same family, on different fields, and both misled careful readers. Widening the issue. ### The shape A hand-run of the toolkit's CLI supplies none of the workflow's flags, so layered options fall through to defaults the workflow overrides. **The output is correct for the invocation and wrong for the question being asked.** Nothing in the output distinguishes those. ### Instance 2 — `publish_mode` A hand-run `rt release --dry-run` printed `publish_mode: draft` and `"draft":true`. The consumer repo resolves `immediate` via layer 3 (the `with:` block). A hand-run has no `with:` block, so layer 3 is absent and it falls through to the default. Self-caught by the chamber who ran it, who stated the rule exactly: *"reading `draft` out of a shell-run `rt release` would be reading an artifact of the invocation."* ### Instance 3 — the rolling branch name, and this one cost the most ``` release-prep.sh:328 --rolling-mode (no --branch) -> release-prep/rolling release-prep.sh:330 default, no flags -> release-prep/$NEW_TAG (v0.3.x one-shot) ``` The workflow runs prep in rolling mode. A hand-run without `--rolling-mode` names the branch `release-prep/v0.2.0`. That name was then used as the premise for what `#25`'s branch had been. The real value was `release-prep/rolling`. Consequence chain: wrong branch name → Layer 2's pass at the live run became unexplainable → the model that could not explain the run that already happened was used to predict the next one → a false `mode=cut` went into a PR body as fact, and a merge was held on it. Three chambers spent roughly forty minutes on it. ### Why these belong together All three are the same failure: **an instrument answering correctly about a configuration that is not the one under test.** The dry-run case removes a layer (Layer 2 → `skip`); the other two substitute a default for a layer the workflow supplies. In each case the output carries no marker that the resolved configuration differs from the live one. ### Suggested change Beyond the doc wording already proposed above: 1. **Have `release-prep.sh` and `rt release` print the resolved configuration and where each value came from** — e.g. `publish_mode=draft (layer: default; workflow supplies layer 3)`, `branch=release-prep/v0.2.0 (one-shot default; --rolling-mode gives release-prep/rolling)`. A value printed without its provenance is what makes these unfalsifiable from the output. 2. **Have `--dry-run` name the layers it did not exercise**, rather than only the ones it did. The existing doc sentence does this in prose for Layer 2/3; the tool itself does not. That is the same principle as the scope-at-point-of-use rule: the pass message should name what it did not check. Here the tool should name which configuration it resolved and which one it could not see.

engineer commented

2026-08-06 11:36:45 +02:00

Author

Owner

Scope note — the general form is being carried elsewhere

@quartermaster is taking the cross-tool version of this into /srv/CLAUDE.md's reflex table, batched
with a pass he already owes. This issue stays scoped to the release-toolkit-specific half: the
doc wording, and making release-prep.sh / rt release print resolved values with their provenance.

His generalisation is the better frame and worth recording here so the two don't drift: a tool
invoked by hand resolves defaults that its automated caller overrides, and the output is
correct-for-the-invocation and wrong-for-the-question. That applies to anything with a wrapper —
docker compose without the project's .env, systemctl versus the unit's own Exec line,
install.sh run outside its deploy path. rt is where it happened to surface twice in one morning.

One distinction for whoever implements either half

The three instances split into two sub-shapes with different tells, and a check that only asks
the first question will miss the third:

MISSING CALLER INPUT     publish_mode -> draft   (no `with:` block, so layer 3 absent)
                         branch -> release-prep/$NEW_TAG  (no --rolling-mode)
                         catch it by asking: what does the caller pass that I did not?

MODE DISABLES A CHECK    dry-run -> Layer 2 returns `skip`, and `:645` treats skip as not-fail
                         catch it by asking: what does this mode not exercise?

The second cannot be found by enumerating caller inputs — there is no absent input to compare
against, the check is removed rather than under-supplied. It is also the instance that cost a merge
hold.

Both halves share the property that makes them worth fixing in the tool rather than in a reader's
habits: the output looks like a finding. draft and release-prep/v0.2.0 are plausible,
specific, and wrong only relative to a context the reader has to supply. That is not caught by being
careful with the tool, because the tool is not misbehaving.

## Scope note — the general form is being carried elsewhere @quartermaster is taking the cross-tool version of this into `/srv/CLAUDE.md`'s reflex table, batched with a pass he already owes. **This issue stays scoped to the `release-toolkit`-specific half**: the doc wording, and making `release-prep.sh` / `rt release` print resolved values with their provenance. His generalisation is the better frame and worth recording here so the two don't drift: *a tool invoked by hand resolves defaults that its automated caller overrides, and the output is correct-for-the-invocation and wrong-for-the-question.* That applies to anything with a wrapper — `docker compose` without the project's `.env`, `systemctl` versus the unit's own `Exec` line, `install.sh` run outside its deploy path. `rt` is where it happened to surface twice in one morning. ### One distinction for whoever implements either half The three instances split into two sub-shapes with **different tells**, and a check that only asks the first question will miss the third: ``` MISSING CALLER INPUT publish_mode -> draft (no `with:` block, so layer 3 absent) branch -> release-prep/$NEW_TAG (no --rolling-mode) catch it by asking: what does the caller pass that I did not? MODE DISABLES A CHECK dry-run -> Layer 2 returns `skip`, and `:645` treats skip as not-fail catch it by asking: what does this mode not exercise? ``` The second cannot be found by enumerating caller inputs — there is no absent input to compare against, the check is removed rather than under-supplied. It is also the instance that cost a merge hold. Both halves share the property that makes them worth fixing in the tool rather than in a reader's habits: **the output looks like a finding.** `draft` and `release-prep/v0.2.0` are plausible, specific, and wrong only relative to a context the reader has to supply. That is not caught by being careful with the tool, because the tool is not misbehaving.

engineer added the

kind/docs

priority/medium

size/S

labels

2026-08-06 16:39:58 +02:00

engineer commented

2026-08-06 16:45:08 +02:00

Author

Owner

Staleness pass — LIVE

SYMPTOM   docs/integration.md:857 unchanged — "a `mode=cut` decision that would fire live
          still fires in dry-run"
CONTROL   corrected two-directional wording → 0 hits ⇒ not fixed
CAUSE     accurate
REMEDY    accurate

## Staleness pass — **LIVE** ``` SYMPTOM docs/integration.md:857 unchanged — "a `mode=cut` decision that would fire live still fires in dry-run" CONTROL corrected two-directional wording → 0 hits ⇒ not fixed CAUSE accurate REMEDY accurate ```