feat(mirror): mirror on `release: published` again, and alert instead of commenting on a tracker #1586

New issue

Open

opened 2026-10-21 23:29:24 +02:00 by quartermaster · 1 comment

quartermaster commented

2026-10-21 23:29:24 +02:00

Owner

Successor to #1348, which closed having met its ACs. #1348 made the omission detectable; this makes it self-clearing and moves the report off a tracker nobody is reading.

Operator ruling, 2026-10-21, on both halves:

We should change the reporting shape in any case. Reopening an issue and posting comments on it will only work if there are a dozen of other issues at most and the project is being actively worked on. I focused on Storey for the most part, so the issue #1348 on rt just rotted for the last days.

I assume that we may automate the mirroring while still keeping the release cut independent from the availability of a 3rd party.

The enabler is not new — it was built, it worked, and it was deleted for an unrelated reason

#427   2026-07-06  mirror-on-cut.yml — a SEPARATE workflow on `release: published`,
                   built so "mirror-failure never touches the cut workflow's job graph"
#793   2026-08-21  retired, because CODEBERG was retired over its AI-code policy.
                   The pattern did not fail; its target went away.
#1030              the gitea.com mirror arrives, workflow_dispatch-only
#1212              ruled manual by design, on cut-dependency grounds #427 had answered

⚠️ #1212's ruling is mine and it re-solved a solved problem. Its reasoning about not taking a hard dependency on a third party inside the cut stands; what it missed is that a separate workflow on release: published already gave exactly that isolation, and had done for six weeks. mirror-drift-check.yml relies on the same isolation today — it fires on that event to DETECT and does not mirror on it.

Three layers

1  INSTANT      a separate workflow on `release: published` dispatches the mirror.
                Re-wires #427's pattern at gitea.com. A failure here cannot redden or
                block the cut — that is the whole property #1212 bought, kept intact.
2  RECONCILER   the existing nightly (cron '47 6 * * *') stays, and becomes the safety
                net for whatever layer 1 missed: an outage at publish time, a dropped
                event, a backfill. With layer 1 working it should normally find nothing.
3  ALERT        Prometheus + Grafana. Drift found by the NIGHTLY means layer 1 failed,
                which is abnormal and worth knowing. Drift that survives the nightly's
                own dispatch is the page.

🔑 No quarantine window. The operator's reason, which corrects the premise my suggestion rested on: "The development of rt on our local forge is not for the sake of isolation / quarantine of other users, but rather because of convenience. We had the instant mirroring, but dropped it only because of the dependency that could break a cut." The local forge is not a staging gate, so there is nothing a delay would buy.

What must be got right

rc=2 NEVER dispatches. Could-not-grade means the comparison was not made. Dispatching on it acts on no information; treating it as clean is #1259 again — the token 401'd while four releases were genuinely absent, and the check correctly refused before it could compare. Three states, three actions: 0 nothing · 1 dispatch, then re-check · 2 alert the instrument, dispatch nothing.
A VALUE rule AND a STALENESS rule, per /srv/CLAUDE.md's system_configs_drift pattern: a stopped checker leaves the last value served, including a reassuring zero. This matters MORE here than in #1348, because once drift self-clears, silence becomes the normal state — and silence is also what a dead nightly produces.
force_recreate must be unreachable from automation. It is destructive, and mirror-release.yml's header already confines it to the dispatch surface. Layer 1 and 2 pass since_tag/tag_name only.
Drop the comment channel. The alert annotation carries the oldest drifting tag. A durable record of a failed mirror lives in the run log; if a human needs a tracker they file one.
One attempt per run. A mirror that fails every night should produce a nightly alert, not a retry loop.

The dispatch mechanics are already proven in-repo

release.yml:184 POSTs .../actions/workflows/<file>/dispatches with secrets.RELEASE_TOOLKIT_TOKEN. ⚠️ And it falls back in SHELL on purpose — their TC-1 note (#41/#47) records that ${{ secrets.A || secrets.B }} does not fall back when A is unset. Copy that shape rather than the obvious one.

Open, for whoever picks this up

Where the metric is pushed from: remote_write out of the Actions run (precedent: storey's bench pushes its bars that way), or a host systemd timer plus node_exporter's textfile collector (precedent: system-configs-drift-check.timer). The second matches the existing drift-alert pattern exactly but moves MIRROR_TOKEN_GITEA_COM onto the host, which is a credential-placement decision for the operator.
Whether layer 2's dispatch needs a distinct alert from layer 1's silent failure, or one counter covers both.

AC

A release cut on this forge appears on gitea.com without a human dispatching anything, and the cut's own status is unaffected when the mirror fails
A mirror failure reaches a person who is not reading this tracker
rc=1, rc=2 and clean are distinguishable from the alerting surface, and rc=2 never triggers a mirror
A stopped checker is alertable — not a reassuring zero
The automated comment stream on #1348 stops

Successor to #1348, which closed having met its ACs. **#1348 made the omission detectable; this makes it self-clearing and moves the report off a tracker nobody is reading.** Operator ruling, 2026-10-21, on both halves: > We should change the reporting shape in any case. Reopening an issue and posting comments on it will only work if there are a dozen of other issues at most and the project is being actively worked on. I focused on Storey for the most part, so the issue #1348 on rt just rotted for the last days. > I assume that we may automate the mirroring while still keeping the release cut independent from the availability of a 3rd party. ## The enabler is not new — it was built, it worked, and it was deleted for an unrelated reason ``` #427 2026-07-06 mirror-on-cut.yml — a SEPARATE workflow on `release: published`, built so "mirror-failure never touches the cut workflow's job graph" #793 2026-08-21 retired, because CODEBERG was retired over its AI-code policy. The pattern did not fail; its target went away. #1030 the gitea.com mirror arrives, workflow_dispatch-only #1212 ruled manual by design, on cut-dependency grounds #427 had answered ``` ⚠️ **#1212's ruling is mine and it re-solved a solved problem.** Its reasoning about not taking a hard dependency on a third party inside the cut stands; what it missed is that a separate workflow on `release: published` already gave exactly that isolation, and had done for six weeks. `mirror-drift-check.yml` relies on the same isolation today — it fires on that event to DETECT and does not mirror on it. ## Three layers ``` 1 INSTANT a separate workflow on `release: published` dispatches the mirror. Re-wires #427's pattern at gitea.com. A failure here cannot redden or block the cut — that is the whole property #1212 bought, kept intact. 2 RECONCILER the existing nightly (cron '47 6 * * *') stays, and becomes the safety net for whatever layer 1 missed: an outage at publish time, a dropped event, a backfill. With layer 1 working it should normally find nothing. 3 ALERT Prometheus + Grafana. Drift found by the NIGHTLY means layer 1 failed, which is abnormal and worth knowing. Drift that survives the nightly's own dispatch is the page. ``` 🔑 **No quarantine window. The operator's reason, which corrects the premise my suggestion rested on:** *"The development of rt on our local forge is not for the sake of isolation / quarantine of other users, but rather because of convenience. We had the instant mirroring, but dropped it only because of the dependency that could break a cut."* The local forge is not a staging gate, so there is nothing a delay would buy. ## What must be got right - **`rc=2` NEVER dispatches.** Could-not-grade means the comparison was not made. Dispatching on it acts on no information; treating it as clean is #1259 again — the token 401'd while four releases were genuinely absent, and the check correctly refused before it could compare. Three states, three actions: `0` nothing · `1` dispatch, then re-check · `2` alert the instrument, dispatch nothing. - **A VALUE rule AND a STALENESS rule**, per `/srv/CLAUDE.md`'s `system_configs_drift` pattern: a stopped checker leaves the last value served, including a reassuring zero. This matters MORE here than in #1348, because once drift self-clears, silence becomes the normal state — and silence is also what a dead nightly produces. - **`force_recreate` must be unreachable from automation.** It is destructive, and `mirror-release.yml`'s header already confines it to the dispatch surface. Layer 1 and 2 pass `since_tag`/`tag_name` only. - **Drop the comment channel.** The alert annotation carries the oldest drifting tag. A durable record of a failed mirror lives in the run log; if a human needs a tracker they file one. - **One attempt per run.** A mirror that fails every night should produce a nightly alert, not a retry loop. ## The dispatch mechanics are already proven in-repo `release.yml:184` POSTs `.../actions/workflows/<file>/dispatches` with `secrets.RELEASE_TOOLKIT_TOKEN`. ⚠️ **And it falls back in SHELL on purpose** — their TC-1 note (#41/#47) records that `${{ secrets.A || secrets.B }}` does **not** fall back when A is unset. Copy that shape rather than the obvious one. ## Open, for whoever picks this up - Where the metric is pushed from: `remote_write` out of the Actions run (precedent: storey's bench pushes its bars that way), or a host systemd timer plus node_exporter's textfile collector (precedent: `system-configs-drift-check.timer`). The second matches the existing drift-alert pattern exactly but moves `MIRROR_TOKEN_GITEA_COM` onto the host, which is a credential-placement decision for the operator. - Whether layer 2's dispatch needs a distinct alert from layer 1's silent failure, or one counter covers both. ## AC - [ ] A release cut on this forge appears on gitea.com without a human dispatching anything, and the cut's own status is unaffected when the mirror fails - [ ] A mirror failure reaches a person who is not reading this tracker - [ ] `rc=1`, `rc=2` and clean are distinguishable from the alerting surface, and `rc=2` never triggers a mirror - [ ] A stopped checker is alertable — not a reassuring zero - [ ] The automated comment stream on #1348 stops

quartermaster added the

labels

2026-10-21 23:29:24 +02:00

quartermaster referenced this issue

2026-10-21 23:29:24 +02:00

bug(ci): mirror-drift-check has failed twice today and its red reaches nobody #1348