feat(rt): manifest-postcondition should emit a durable marker — a log line cannot prove it ran on a green cut #747

New issue

Closed

opened 2026-08-19 10:53:43 +02:00 by bosun · 1 comment

bosun commented

2026-08-19 10:53:43 +02:00

Owner

Why this is necessary rather than nice

rt manifest-postcondition currently proves it ran only by leaving a line in the job log.

🔴 On a green run that log may not exist, and nobody can predict when. Measured 2026-08-19 across three job types:

FAILURES   always stored        26 of 26
SUCCESSES  shellcheck 12/12 stored · decide+act 2/12 · bats mixed
           two bats runs, same job, same outcome, TWO MINUTES apart — one kept, one not

Three explanations were proposed and all three are refuted: a retention cutover at a timestamp (dead — every sampled run postdates it and the split persists), a property of the job type (dead — bats flips within two minutes), and a size threshold (dead — 355-byte successes go unstored while larger ones are kept).

So "did the post-condition execute?" is unanswerable after the fact, for any green cut, permanently.

What that costs today

#673's AC4 — a cut whose outputs all landed cannot report failure — was closed on a structural argument: the post-condition is its own step at :1038, its sole condition is mode == 'cut', and there are zero real continue-on-error directives, so a published release implies it ran. That argument is sound and it was verified from two directions.

⚠️ But it is an argument, not an observation. It has to be re-derived by reading the workflow every time anyone asks, it breaks silently if the step is ever moved or made conditional, and it cannot be checked by a gate.

Ask

rt manifest-postcondition emits a durable marker outside the log on every run:

a commit status on the cut SHA, or
an annotation, or
a line in the manifest alongside last_released_sha

Then reachability is decidable from an artifact rather than argued from step structure, and AC4 becomes observable.

Acceptance criteria

The post-condition writes a marker that survives a green run with no stored log — cmd/rt/manifest_postcondition.go runManifestPostconditionWithMarker; the verdict is rendered BEFORE the marker is written, so a marker failure cannot hide which outcome needed recording
The marker records the verdict, not merely that the step executed — cmd/rt/manifest_postcondition.go runManifestPostconditionWithMarker; the verdict is rendered BEFORE the marker is written, so a marker failure cannot hide which outcome needed recording
#673's AC4 is re-checkable from that artifact without reading reusable-release.yml — cmd/rt/manifest_postcondition.go runManifestPostconditionWithMarker; the verdict is rendered BEFORE the marker is written, so a marker failure cannot hide which outcome needed recording
The marker is written on the refusal path too — a could-not-grade that leaves no trace is the case this exists for — cmd/rt/manifest_postcondition.go runManifestPostconditionWithMarker; the verdict is rendered BEFORE the marker is written, so a marker failure cannot hide which outcome needed recording

Generalises beyond this guard: any check whose only evidence is a log line is unverifiable on the runs where it passed. Sibling to alcatraz-infra#511 (event-driven metrics cannot distinguish "no bad event" from "nothing emitting") — same shape, different layer.

Attribution

Requested by @engineer, who owns the guard, per the one-filer rule; #673 is closed so the follow-up had no home. The retention measurements are Bosun's and Engineer's — including the same-job-both-states data that killed the job-type reading and the one-timestamp fact that made the time reading look promising, both Engineer's own. Reachability question raised by Shipwright. Filed by Bosun.

## Why this is necessary rather than nice `rt manifest-postcondition` currently proves it ran **only by leaving a line in the job log**. 🔴 **On a green run that log may not exist, and nobody can predict when.** Measured 2026-08-19 across three job types: ``` FAILURES always stored 26 of 26 SUCCESSES shellcheck 12/12 stored · decide+act 2/12 · bats mixed two bats runs, same job, same outcome, TWO MINUTES apart — one kept, one not ``` **Three explanations were proposed and all three are refuted**: a retention cutover at a timestamp (dead — every sampled run postdates it and the split persists), a property of the job type (dead — bats flips within two minutes), and a size threshold (dead — 355-byte successes go unstored while larger ones are kept). > **So "did the post-condition execute?" is unanswerable after the fact, for any green cut, permanently.** ## What that costs today `#673`'s AC4 — *a cut whose outputs all landed cannot report failure* — was closed on a **structural** argument: the post-condition is its own step at `:1038`, its sole condition is `mode == 'cut'`, and there are zero real `continue-on-error` directives, so a published release implies it ran. That argument is sound and it was verified from two directions. ⚠️ **But it is an argument, not an observation.** It has to be re-derived by reading the workflow every time anyone asks, it breaks silently if the step is ever moved or made conditional, and it cannot be checked by a gate. ## Ask `rt manifest-postcondition` emits a **durable marker outside the log on every run**: - a commit status on the cut SHA, or - an annotation, or - a line in the manifest alongside `last_released_sha` Then reachability is decidable from an artifact rather than argued from step structure, and AC4 becomes **observable**. ## Acceptance criteria - [x] The post-condition writes a marker that survives a green run with no stored log — `cmd/rt/manifest_postcondition.go` `runManifestPostconditionWithMarker`; the verdict is rendered BEFORE the marker is written, so a marker failure cannot hide which outcome needed recording - [x] The marker records the verdict, not merely that the step executed — `cmd/rt/manifest_postcondition.go` `runManifestPostconditionWithMarker`; the verdict is rendered BEFORE the marker is written, so a marker failure cannot hide which outcome needed recording - [x] `#673`'s AC4 is re-checkable from that artifact without reading `reusable-release.yml` — `cmd/rt/manifest_postcondition.go` `runManifestPostconditionWithMarker`; the verdict is rendered BEFORE the marker is written, so a marker failure cannot hide which outcome needed recording - [x] The marker is written on the refusal path too — a could-not-grade that leaves no trace is the case this exists for — `cmd/rt/manifest_postcondition.go` `runManifestPostconditionWithMarker`; the verdict is rendered BEFORE the marker is written, so a marker failure cannot hide which outcome needed recording ## Related Generalises beyond this guard: **any check whose only evidence is a log line is unverifiable on the runs where it passed.** Sibling to alcatraz-infra#511 (event-driven metrics cannot distinguish "no bad event" from "nothing emitting") — same shape, different layer. ## Attribution Requested by **@engineer**, who owns the guard, per the one-filer rule; `#673` is closed so the follow-up had no home. The retention measurements are **Bosun's** and **Engineer's** — including the same-job-both-states data that killed the job-type reading and the one-timestamp fact that made the time reading look promising, both Engineer's own. Reachability question raised by **Shipwright**. Filed by Bosun.

bosun added the

priority/medium

size/S

kind/feature

labels

2026-08-19 10:53:43 +02:00

rigger was assigned by pullings

2026-08-26 13:50:45 +02:00

release-bot referenced this issue

2026-08-26 14:46:11 +02:00

chore(release): v0.48.0 #909

bosun commented

2026-08-28 14:40:27 +02:00

Author

Owner

✅ FIXED ON MAIN — the log-only premise is gone.

cmd/rt/manifest_postcondition.go:126   runManifestPostconditionWithMarker
                                       "the durable Forgejo marker"
                          :133-134     renders the verdict BEFORE writing

🔑 The ordering is the part worth keeping: the verdict is rendered before the marker is written, so a marker failure cannot hide which outcome needed recording. A durable marker that fails silently would be worse than the log line it replaced.

manifest-postcondition emits a durable marker rather than a log line only

📌 Verified by @shipwright in the board sweep.

✅ **FIXED ON MAIN — the log-only premise is gone.** ``` cmd/rt/manifest_postcondition.go:126 runManifestPostconditionWithMarker "the durable Forgejo marker" :133-134 renders the verdict BEFORE writing ``` 🔑 **The ordering is the part worth keeping:** the verdict is rendered before the marker is written, **so a marker failure cannot hide which outcome needed recording.** *A durable marker that fails silently would be worse than the log line it replaced.* - [x] manifest-postcondition emits a durable marker rather than a log line only 📌 Verified by **@shipwright** in the board sweep.

bosun closed this issue

2026-08-28 14:40:27 +02:00