feat: structured JSON logging substrate (observability foundation per operator engagement)

quartermaster commented

2026-06-27 02:05:09 +02:00

Owner

Why

The toolkit emits naturally-metric-able events during the cut path (cut decision, mode, idempotency-skip, failure step + cause, token-fallback usage, path-α-vs-γ disposition, re-pin discipline timing). Today they're scattered across unstructured runner logs; root-causing intermittent failures means log-scrolling instead of querying.

Structured JSON logging would:

Let Loki/Promtail (already running on alcatraz) ingest cut events directly → Grafana dashboards on top
Make secrets-leak auditing (#156) easier (one place to check structured-output sanitization)
Surface user-experience metrics (time-to-first-cut from #157 walkthrough)
Enable future Prometheus pushgateway / pre-built dashboards / SLO substrate IF scale grows (out of THIS PR's scope)

The "Grafana integration" idea operator surfaced was substantively larger; this tracker is the foundation layer that the larger substrate would compose ON.

What it looks like

A new helper in scripts/lib/:

# scripts/lib/events.sh
event_emit() {
    local kind="$1"; shift
    local fields=("$@")
    # Emits to stderr (so it doesn't pollute stdout for callers that
    # capture output for parsing) as a single JSON line per event.
    {
        printf '{"ts":"%s","kind":"%s"' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$kind"
        for kv in "${fields[@]}"; do
            local key="${kv%%=*}"; local val="${kv#*=}"
            printf ',"%s":' "$key"
            printf '%s' "$val" | jq -Rs .
        done
        printf '}\n'
    } >&2
}

Used like:

event_emit cut_decided mode=update bump=minor reason="conventional commits + fragments imply minor"
event_emit manifest_skip head_sha="$HEAD_SHA" last_released="$LAST_SHA"
event_emit api_call endpoint=releases method=POST status=201 ms=320
event_emit error step="release-decide" cause="OWNER unbound" recoverable=false

Output (one event per JSON line on stderr):

{"ts":"2026-06-27T01:30:00Z","kind":"cut_decided","mode":"update","bump":"minor","reason":"conventional commits + fragments imply minor"}

Loki ingest

The Forgejo-runner already runs in docker; Promtail (or Alloy) can scrape its container stdout/stderr. The JSON lines parse natively in LogQL:

{container="forgejo-runner"} |= "release-toolkit" | json | kind="cut_decided" | rate(5m)

No new pushgateway / sidecar / instrumentation surface needed for the consumer.

Scope

In scope

scripts/lib/events.sh — the helper
Instrument the high-value sites in release-decide.sh, release-prep.sh, manifest-precheck.sh, draft-release.sh (decision points, failure paths, API calls)
bats coverage for the helper (output is valid JSON per event class)
AGENTS.md / integration.md note on the events schema + Loki ingest example
A simple Grafana dashboard JSON ships in examples/grafana/ (basic cut-cadence + decision-mode-mix)

Out of scope (explicit)

NO Prometheus pushgateway — JSON-on-stderr + Loki is the substrate
NO consumer-facing config knobs — emit always (security audit may surface what gets sanitized; that's #156's surface)
NO alerting rules / SLO templates — that's substrate-evolution at higher scale; not pre-1.0
NO cross-consumer aggregation framework — single Loki queryable across consumers if the operator wants; no toolkit-side aggregation

Event taxonomy (initial)

Event	Fields	Site
`cut_decided`	mode, bump, reason	release-decide.sh exit
`manifest_skip`	head_sha, last_released_sha	manifest-precheck.sh skip path
`manifest_proceed`	reason	manifest-precheck.sh proceed path
`api_call`	endpoint, method, status, ms	forgejo-api.sh wrapper
`error`	step, cause, recoverable	log() at fatal exits
`repin_check`	head_match, pinned_ref	check-self-bootstrap.sh result
`cut_complete`	version, path, ms_total	_release.yml cut path success

Pre-1.0 release should ship the first ~10 events; more can land post-1.0.

Composition

#156 security audit: surface for token-leak audit; this gives a SINGLE place to check sanitization (all events flow through event_emit); cleaner audit boundary
#157 walkthrough audit: Herald can read structured logs to assess adoption friction — events show WHERE adopters stall
#148 build-bake: parallel; doesn't compose directly with this
#149/#150 Unicode cleanups: emit log strings would benefit from ASCII-only (Loki parsing is more reliable with ASCII)

What this PR does NOT do

Does NOT add metric-emission via Prometheus pushgateway — JSON-on-stderr is the surface
Does NOT add consumer-facing dashboards beyond a basic example — basic Grafana JSON only
Does NOT change runtime behavior of any cut step — pure observability layer
Does NOT add ALERTING substrate — out of pre-1.0 scope

Refs

Operator engagement 2026-06-27: agreed to (A) JSON logging substrate over (B) Prometheus pushgateway / (C) full observability platform — current scale doesn't justify (B)/(C); (A) is the foundation that enables future evolution
Composition: #156 security, #157 walkthrough, #149/#150 (Unicode cleanups)
Substrate context: alcatraz Loki already running; ingestion path exists

## Why The toolkit emits naturally-metric-able events during the cut path (cut decision, mode, idempotency-skip, failure step + cause, token-fallback usage, path-α-vs-γ disposition, re-pin discipline timing). Today they're scattered across unstructured runner logs; root-causing intermittent failures means log-scrolling instead of querying. Structured JSON logging would: - Let Loki/Promtail (already running on alcatraz) ingest cut events directly → Grafana dashboards on top - Make secrets-leak auditing ([#156](https://git.frankenbit.de/frankenbit/release-toolkit/issues/156)) easier (one place to check structured-output sanitization) - Surface user-experience metrics (time-to-first-cut from [#157](https://git.frankenbit.de/frankenbit/release-toolkit/issues/157) walkthrough) - Enable future Prometheus pushgateway / pre-built dashboards / SLO substrate IF scale grows (out of THIS PR's scope) The "Grafana integration" idea operator surfaced was substantively larger; this tracker is the foundation layer that the larger substrate would compose ON. ## What it looks like A new helper in `scripts/lib/`: ```bash # scripts/lib/events.sh event_emit() { local kind="$1"; shift local fields=("$@") # Emits to stderr (so it doesn't pollute stdout for callers that # capture output for parsing) as a single JSON line per event. { printf '{"ts":"%s","kind":"%s"' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$kind" for kv in "${fields[@]}"; do local key="${kv%%=*}"; local val="${kv#*=}" printf ',"%s":' "$key" printf '%s' "$val" | jq -Rs . done printf '}\n' } >&2 } ``` Used like: ```bash event_emit cut_decided mode=update bump=minor reason="conventional commits + fragments imply minor" event_emit manifest_skip head_sha="$HEAD_SHA" last_released="$LAST_SHA" event_emit api_call endpoint=releases method=POST status=201 ms=320 event_emit error step="release-decide" cause="OWNER unbound" recoverable=false ``` Output (one event per JSON line on stderr): ```json {"ts":"2026-06-27T01:30:00Z","kind":"cut_decided","mode":"update","bump":"minor","reason":"conventional commits + fragments imply minor"} ``` ## Loki ingest The Forgejo-runner already runs in docker; Promtail (or Alloy) can scrape its container stdout/stderr. The JSON lines parse natively in LogQL: ```logql {container="forgejo-runner"} |= "release-toolkit" | json | kind="cut_decided" | rate(5m) ``` No new pushgateway / sidecar / instrumentation surface needed for the consumer. ## Scope ### In scope - `scripts/lib/events.sh` — the helper - Instrument the high-value sites in `release-decide.sh`, `release-prep.sh`, `manifest-precheck.sh`, `draft-release.sh` (decision points, failure paths, API calls) - bats coverage for the helper (output is valid JSON per event class) - AGENTS.md / integration.md note on the events schema + Loki ingest example - A simple Grafana dashboard JSON ships in `examples/grafana/` (basic cut-cadence + decision-mode-mix) ### Out of scope (explicit) - **NO Prometheus pushgateway** — JSON-on-stderr + Loki is the substrate - **NO consumer-facing config knobs** — emit always (security audit may surface what gets sanitized; that's #156's surface) - **NO alerting rules / SLO templates** — that's substrate-evolution at higher scale; not pre-1.0 - **NO cross-consumer aggregation framework** — single Loki queryable across consumers if the operator wants; no toolkit-side aggregation ## Event taxonomy (initial) | Event | Fields | Site | |---|---|---| | `cut_decided` | mode, bump, reason | release-decide.sh exit | | `manifest_skip` | head_sha, last_released_sha | manifest-precheck.sh skip path | | `manifest_proceed` | reason | manifest-precheck.sh proceed path | | `api_call` | endpoint, method, status, ms | forgejo-api.sh wrapper | | `error` | step, cause, recoverable | log() at fatal exits | | `repin_check` | head_match, pinned_ref | check-self-bootstrap.sh result | | `cut_complete` | version, path, ms_total | _release.yml cut path success | Pre-1.0 release should ship the first ~10 events; more can land post-1.0. ## Composition - **[#156 security audit](https://git.frankenbit.de/frankenbit/release-toolkit/issues/156)**: surface for token-leak audit; this gives a SINGLE place to check sanitization (all events flow through `event_emit`); cleaner audit boundary - **[#157 walkthrough audit](https://git.frankenbit.de/frankenbit/release-toolkit/issues/157)**: Herald can read structured logs to assess adoption friction — events show WHERE adopters stall - **#148 build-bake**: parallel; doesn't compose directly with this - **#149/#150 Unicode cleanups**: emit log strings would benefit from ASCII-only (Loki parsing is more reliable with ASCII) ## What this PR does NOT do - **Does NOT add metric-emission via Prometheus pushgateway** — JSON-on-stderr is the surface - **Does NOT add consumer-facing dashboards beyond a basic example** — basic Grafana JSON only - **Does NOT change runtime behavior of any cut step** — pure observability layer - **Does NOT add ALERTING substrate** — out of pre-1.0 scope ## Refs - **Operator engagement 2026-06-27**: agreed to (A) JSON logging substrate over (B) Prometheus pushgateway / (C) full observability platform — current scale doesn't justify (B)/(C); (A) is the foundation that enables future evolution - **Composition**: [#156 security](https://git.frankenbit.de/frankenbit/release-toolkit/issues/156), [#157 walkthrough](https://git.frankenbit.de/frankenbit/release-toolkit/issues/157), [#149](https://git.frankenbit.de/frankenbit/release-toolkit/issues/149)/[#150](https://git.frankenbit.de/frankenbit/release-toolkit/issues/150) (Unicode cleanups) - **Substrate context**: alcatraz Loki already running; ingestion path exists