frankenbit/release-toolkit

Watch

Fork 0

bug(mirror-drift-check): v0.63.1 sat unmirrored for 24 days under 26 DRIFT comments on #1348 that reached nobody #1520

New issue

Closed

opened 2026-10-02 20:52:23 +02:00 by claude · 1 comment

claude commented

2026-10-02 20:52:23 +02:00

Member

v0.63.1 sat unmirrored on gitea.com for 24 days and 19 hours. Every morning, mirror-drift-check posted a comment to the standing tracker #1348 naming it: 26 comments, and none of them got a reply. The gap surfaced only on 2026-10-02, when the v0.64.0 cut's release-event run put a red status on the prep commit that the operator happened to see. #1348's shipped AC says the drift report goes to "a surface a reader actually looks at". This measurement says it does not.

Measured

Measured 2026-10-02.

Fact	Evidence
v0.63.1 published here	`2026-09-08T01:21:51+02:00` (`GET /releases/tags/v0.63.1`)
v0.63.1 published on the mirror	`2026-10-02T18:48:59Z`, by `mirror-release.yml` run 19189, dispatched on operator request
last `mirror-release` dispatch before that	run 18097 (alex, 2026-09-07), `tag_name=v0.63.0`
#1348 DRIFT comments naming v0.63.1	26, from `2026-09-08T08:48:01+02:00` to `2026-10-02T20:37:17+02:00`, all by `forgejo-actions`
non-DRIFT comments on #1348 after the first one	0. The last human comment is herald's, `2026-09-07T23:20`.
#1348's state	open, `priority/high`, assigned to `engineer`, every AC ticked
how the gap surfaced	run 19185 (`release: published` for v0.64.0) posted `mirror-drift-check / drift = failure` on `eefff94`, and the operator saw it on the cut

Run 19185's log: FAIL: 1 release(s) carry assets here and do not on the mirror: v0.63.1. After run 19189, both v0.63.1 and v0.64.0 are on the mirror. Their checksums.txt, checksums.txt.minisig and rt-linux-amd64 match the origin's byte for byte (sha256 compared).

Settled, and not reopened here

Mirroring is manual by design. See #1212, #1392 and mirror-release.yml:23-27: "A mirror failure must not redden or block a cut that has already published correctly here." #1212's own words name the gap this issue measures: "manual is fine; manual with nothing watching produced the gap". Something is watching now. Nothing the watcher says reaches a person.

Why it matters

Third time for the same failure. #1212: six consecutive cuts unmirrored before anyone noticed. #1348: "its red reaches nobody". Now 26 reports into a tracker nobody reads.
Public adopters fall behind without anyone saying so. gitea.com is the public adoption surface (#1021). Until run 19189, the newest release the mirror served was v0.63.0, while origin had v0.63.1 and v0.64.0. Run 19189's README job has now opened #1519 to move the adopter pin.
The check works, so the gap is in the delivery. rt mirror-drift-check named the right tag with the right remedy on every one of the 26 runs.

Not established

Whether engineer, the assignee, receives or reads Forgejo notifications for comments on an assigned issue. It was not probed. The outcome (0 replies in 24 days) is measured; the cause is not.
Whether an @-mention in a bot comment notifies the mentioned user on this instance. Not probed. Option (B) depends on it.

Decision tree

(A) Show unmirrored tags where the operator already is during a cut. For example, the rolling release PR body or rt decide's log line could name the unmirrored tags, advisory and never blocking.

Right when cuts are frequent enough that "by the next cut" is soon enough.
Leaves: latency up to one cut interval. Here that would have been 24 days, because v0.64.0 was the next cut after v0.63.1.

(B) Notify a named person. The DRIFT comment @-mentions the operator, or the check assigns the tracker to them.

Right when that person reads mention notifications. That has to be measured first (see Not established).
Leaves: it is still a comment stream. It only works if the notification is read.

(C) Escalate with age. Day 1 is a tracker comment. After N days, open a fresh issue assigned to the operator, or post the red where (A) would.

Right when a short gap is tolerable but a long one is not.
Leaves: more machinery for the same signal.

(D) Mirror on release: published in a separate workflow, outside the cut's graph, and keep the drift check as the backstop.

Right when the recorded reason for staying manual is the only one. That reason is that a mirror failure must not redden or block the cut. A separate release-triggered workflow meets it the same way mirror-drift-check.yml already does: its red lands as its own status context, and the cut's jobs are untouched.
Leaves: this reverses #1212's "manual by design", so it is the operator's call, not a fix. A gitea.com outage would then show red on the released commit, which mirror-drift-check already does today.

Lean, for the operator to overrule: (D). It removes the step whose omission has now happened three times, and the stated reason for keeping it manual is satisfied by the same separation mirror-drift-check.yml already uses. If (D) is declined, (B) is the cheapest delivery fix, once mention notifications are measured.

Acceptance criteria

A decision is recorded on this issue among (A)–(D) or a combination.
An unmirrored release reaches a named person within one day of publication. This is demonstrated with a live or dispatched run whose delivery is read back on the receiving side, not just shipped (AGENTS.md §8 Axis C).
If (D): a forced mirror failure leaves the cut's own jobs and required contexts unaffected, measured.
#1348's "surface a reader actually looks at" AC is re-graded against this measurement, in a comment on #1348.

v0.63.1 sat unmirrored on gitea.com for **24 days and 19 hours**. Every morning, `mirror-drift-check` posted a comment to the standing tracker #1348 naming it: **26 comments, and none of them got a reply**. The gap surfaced only on 2026-10-02, when the v0.64.0 cut's release-event run put a red status on the prep commit that the operator happened to see. #1348's shipped AC says the drift report goes to *"a surface a reader actually looks at"*. This measurement says it does not. ## Measured Measured 2026-10-02. | Fact | Evidence | |---|---| | v0.63.1 published here | `2026-09-08T01:21:51+02:00` (`GET /releases/tags/v0.63.1`) | | v0.63.1 published on the mirror | `2026-10-02T18:48:59Z`, by `mirror-release.yml` run 19189, dispatched on operator request | | last `mirror-release` dispatch before that | run 18097 (alex, 2026-09-07), `tag_name=v0.63.0` | | #1348 DRIFT comments naming v0.63.1 | **26**, from `2026-09-08T08:48:01+02:00` to `2026-10-02T20:37:17+02:00`, all by `forgejo-actions` | | non-DRIFT comments on #1348 after the first one | **0**. The last human comment is herald's, `2026-09-07T23:20`. | | #1348's state | open, `priority/high`, assigned to `engineer`, every AC ticked | | how the gap surfaced | run 19185 (`release: published` for v0.64.0) posted `mirror-drift-check / drift = failure` on `eefff94`, and the operator saw it on the cut | Run 19185's log: `FAIL: 1 release(s) carry assets here and do not on the mirror: v0.63.1`. After run 19189, both v0.63.1 and v0.64.0 are on the mirror. Their `checksums.txt`, `checksums.txt.minisig` and `rt-linux-amd64` match the origin's byte for byte (sha256 compared). ## Settled, and not reopened here **Mirroring is manual by design.** See #1212, #1392 and `mirror-release.yml:23-27`: *"A mirror failure must not redden or block a cut that has already published correctly here."* #1212's own words name the gap this issue measures: *"manual is fine; manual with nothing watching produced the gap"*. Something is watching now. **Nothing the watcher says reaches a person.** ## Why it matters - **Third time for the same failure.** #1212: six consecutive cuts unmirrored before anyone noticed. #1348: *"its red reaches nobody"*. Now 26 reports into a tracker nobody reads. - **Public adopters fall behind without anyone saying so.** gitea.com is the public adoption surface (#1021). Until run 19189, the newest release the mirror served was v0.63.0, while origin had v0.63.1 and v0.64.0. Run 19189's README job has now opened #1519 to move the adopter pin. - **The check works, so the gap is in the delivery.** `rt mirror-drift-check` named the right tag with the right remedy on every one of the 26 runs. ## Not established - **Whether `engineer`, the assignee, receives or reads Forgejo notifications** for comments on an assigned issue. It was not probed. The outcome (0 replies in 24 days) is measured; the cause is not. - **Whether an @-mention in a bot comment notifies the mentioned user on this instance.** Not probed. Option (B) depends on it. ## Decision tree **(A) Show unmirrored tags where the operator already is during a cut.** For example, the rolling release PR body or `rt decide`'s log line could name the unmirrored tags, advisory and never blocking. - *Right when* cuts are frequent enough that "by the next cut" is soon enough. - *Leaves:* latency up to one cut interval. Here that would have been 24 days, because v0.64.0 was the next cut after v0.63.1. **(B) Notify a named person.** The DRIFT comment @-mentions the operator, or the check assigns the tracker to them. - *Right when* that person reads mention notifications. That has to be measured first (see Not established). - *Leaves:* it is still a comment stream. It only works if the notification is read. **(C) Escalate with age.** Day 1 is a tracker comment. After N days, open a fresh issue assigned to the operator, or post the red where (A) would. - *Right when* a short gap is tolerable but a long one is not. - *Leaves:* more machinery for the same signal. **(D) Mirror on `release: published` in a separate workflow, outside the cut's graph, and keep the drift check as the backstop.** - *Right when* the recorded reason for staying manual is the only one. That reason is that a mirror failure must not redden or block the cut. A separate release-triggered workflow meets it the same way `mirror-drift-check.yml` already does: its red lands as its own status context, and the cut's jobs are untouched. - *Leaves:* this reverses #1212's "manual by design", so it is the operator's call, not a fix. A gitea.com outage would then show red on the released commit, which `mirror-drift-check` already does today. **Lean, for the operator to overrule:** (D). It removes the step whose omission has now happened three times, and the stated reason for keeping it manual is satisfied by the same separation `mirror-drift-check.yml` already uses. If (D) is declined, (B) is the cheapest delivery fix, once mention notifications are measured. ## Acceptance criteria - [ ] **A decision is recorded on this issue** among (A)–(D) or a combination. - [ ] **An unmirrored release reaches a named person within one day of publication.** This is demonstrated with a live or dispatched run whose delivery is read back on the receiving side, not just shipped (AGENTS.md §8 Axis C). - [ ] If (D): a forced mirror failure leaves the cut's own jobs and required contexts unaffected, measured. - [ ] #1348's *"surface a reader actually looks at"* AC is re-graded against this measurement, in a comment on #1348.

claude added the

labels

2026-10-02 20:52:23 +02:00

claude referenced this issue

2026-10-02 20:52:23 +02:00

bug(mirror-drift-check): the DRIFT comment links its run at http://forgejo:3000, the runner's internal host #1521

claude referenced this issue

2026-10-02 20:52:36 +02:00

bug(ci): mirror-drift-check has failed twice today and its red reaches nobody #1348

quartermaster commented

2026-10-22 01:16:55 +02:00

Owner

Closing as superseded by #1586 / PR #1587, which took an option this issue's decision tree does not name — and the residue is named below rather than closed with it.

This issue's finding is that the delivery reaches nobody: 26 correct DRIFT reports naming v0.63.1, 0 replies, 24 days and 19 hours unmirrored. Options (A) show it during a cut, (B) notify a named person, (C) escalate with age — all three keep the report as the mechanism and try to route it better.

(D), which is what landed: fix the cause so the report is rarely needed.

mirror-on-publish.yml           on origin/main at 05bd59ca — dispatches the mirror on
                                `release: published`, a SEPARATE workflow, so its red
                                still cannot redden or block a cut (#1212 intact)
mirror-drift-check.yml          one tracker per drift EPISODE, filed when none is open;
                                it never reopens a closed one.  reopen idiom: 0 hits.
                                hardcoded `issues/1348`: 0 hits.
drift right now                 v0.62.3 .. v0.67.0, all 8 asset-bearing releases:
                                assets=3 here and assets=3 on the mirror

So the 24-day shape cannot recur the same way: the mirror no longer waits for a human to dispatch it, and a report that does appear lands on a tracker that shows up on the open board as new rather than as comment number 27 on a standing one.

What this does NOT fix, and where it now lives

🔑 The delivery question itself is not answered, it is made rarer. A filed tracker is still an issue on a board, and this issue's own evidence is that a board entry can go 24 days unread while the project's attention is elsewhere. The operator's words on 2026-10-21, closing #1348: "Reopening an issue and posting comments on it will only work if there are a dozen of other issues at most and the project is being actively worked on."

#1586 owns the real answer and it is unbuilt: a Prometheus gauge plus one Grafana alert, with a staleness partner so a stopped checker is alertable rather than serving a reassuring zero. Its open ACs carry it, including the push-path decision that blocks it — remote_write needs protobuf+snappy from a shell step or new dependencies in a two-dependency Go module, and the textfile-collector route would be a second implementation of a predicate rt mirror-drift-check already owns.

⚠️ And one thing in this issue's "Not established" is still not established, so it is not being quietly dropped: whether the assignee receives or reads Forgejo notifications for an assigned issue, and whether an @-mention in a bot comment notifies on this instance. Neither was probed then and neither has been since. Option (B) depended on both. Also measured since, on the same instance: the forge has not successfully sent any notification email for at least two weeks — alcatraz-infra#940, the relay's cert names only localhost while SMTP_ADDR=smtp-relay, 0 deliveries from git@frankenbit.de against 10,852 from other senders. So for that window the answer to (B) was nobody receives anything, which is worth knowing before anyone reaches for a notification-based remedy again.

Why it is closed rather than left open behind #1586

The measured instance is remediated (all 8 releases mirrored, byte-identical assets), the cause is addressed on main, and what remains is one AC on a tracker that already owns it. Leaving this open would put the same residue on two boards, which is this issue's own complaint in miniature.

**Closing as superseded by #1586 / PR #1587, which took an option this issue's decision tree does not name — and the residue is named below rather than closed with it.** This issue's finding is that the delivery reaches nobody: 26 correct DRIFT reports naming v0.63.1, 0 replies, 24 days and 19 hours unmirrored. Options **(A)** show it during a cut, **(B)** notify a named person, **(C)** escalate with age — all three keep the report as the mechanism and try to route it better. **(D), which is what landed: fix the cause so the report is rarely needed.** ``` mirror-on-publish.yml on origin/main at 05bd59ca — dispatches the mirror on `release: published`, a SEPARATE workflow, so its red still cannot redden or block a cut (#1212 intact) mirror-drift-check.yml one tracker per drift EPISODE, filed when none is open; it never reopens a closed one. reopen idiom: 0 hits. hardcoded `issues/1348`: 0 hits. drift right now v0.62.3 .. v0.67.0, all 8 asset-bearing releases: assets=3 here and assets=3 on the mirror ``` So the 24-day shape cannot recur the same way: the mirror no longer waits for a human to dispatch it, and a report that does appear lands on a tracker that shows up on the open board as new rather than as comment number 27 on a standing one. ## What this does NOT fix, and where it now lives 🔑 **The delivery question itself is not answered, it is made rarer.** A filed tracker is still an issue on a board, and this issue's own evidence is that a board entry can go 24 days unread while the project's attention is elsewhere. The operator's words on 2026-10-21, closing #1348: *"Reopening an issue and posting comments on it will only work if there are a dozen of other issues at most and the project is being actively worked on."* **#1586 owns the real answer and it is unbuilt:** a Prometheus gauge plus one Grafana alert, with a staleness partner so a stopped checker is alertable rather than serving a reassuring zero. Its open ACs carry it, including the push-path decision that blocks it — `remote_write` needs protobuf+snappy from a shell step or new dependencies in a two-dependency Go module, and the textfile-collector route would be a second implementation of a predicate `rt mirror-drift-check` already owns. ⚠️ **And one thing in this issue's "Not established" is still not established**, so it is not being quietly dropped: whether the assignee receives or reads Forgejo notifications for an assigned issue, and whether an @-mention in a bot comment notifies on this instance. Neither was probed then and neither has been since. Option (B) depended on both. **Also measured since, on the same instance: the forge has not successfully sent any notification email for at least two weeks** — `alcatraz-infra#940`, the relay's cert names only `localhost` while `SMTP_ADDR=smtp-relay`, 0 deliveries from `git@frankenbit.de` against 10,852 from other senders. So for that window the answer to (B) was *nobody receives anything*, which is worth knowing before anyone reaches for a notification-based remedy again. ## Why it is closed rather than left open behind #1586 The measured instance is remediated (all 8 releases mirrored, byte-identical assets), the cause is addressed on main, and what remains is one AC on a tracker that already owns it. Leaving this open would put the same residue on two boards, which is this issue's own complaint in miniature.

quartermaster closed this issue

2026-10-22 01:16:55 +02:00