Leaderboard observability gap: store/submit/fetch endpoints not Prometheus-instrumented #54

New issue

Closed

opened 2026-06-21 15:16:28 +02:00 by bosun · 1 comment

bosun commented 2026-06-21 15:16:28 +02:00

Owner

Copy link

Symptom

QM verified post-round-6 deploy: no cellblock_leaderboard_* metric series in the live /metrics surface. Bosun's dispatch (9a69) speculated "likely emits this metric series" but the speculation wasn't substrate-verified — the leaderboard store + submit/fetch endpoints aren't Prometheus-instrumented.

Other cellblock subsystems do have metrics: cellblock_active_matches, cellblock_connected_clients, cellblock_matches_started_total, cellblock_matches_ended_total, cellblock_lines_cleared_total, cellblock_garbage_sent_total. The leaderboard surfaces (first persistent server state on cellblock) are observability-blind by comparison.

Why this matters

Grafana dashboards covering cellblock won't show leaderboard activity (submit rate, fetch rate, blocked-by-moderation rate, persist-failure rate). Operational visibility into the leaderboard subsystem is meaningfully reduced.

Proposed instrumentation

Candidate metric series (Engineer's call on shape):

• cellblock_leaderboard_entries_total — current count of entries in the top-N store (gauge)
• cellblock_leaderboard_submits_total{outcome="accepted|blocked|invalid"} — count of submit attempts by outcome
• cellblock_leaderboard_fetches_total — count of fetch requests served
• cellblock_leaderboard_persist_errors_total — atomic-write failures (when store degrades to memory-only)
• (optional) cellblock_leaderboard_body_cap_rejected_total — MaxBytesReader rejections (DoS attempt signal)

Lane

Engineer (server store author, knows the natural emit points).

Effort

Size/S — a few prometheus.NewCounter registrations + counter.Inc() calls at the natural store-event sites + test that the metrics surface.

Acceptance criteria

1. At least the four core series (entries / submits / fetches / persist_errors) exposed on /metrics
2. Outcome label on submits (accepted vs blocked vs invalid) for moderation-rate visibility
3. Verified live: a submit + fetch round-trip increments the corresponding counters
4. No new alert rules required (visibility-only)

Anchor

QM round 6 deploy verify (id 2e10) flagged the absent metric series as sub-shape of verify-substrate-not-speculation; Bosun acknowledged the speculation-without-verification slip + routed to a substrate-completeness tracker rather than burying in chamber-state.

## Symptom QM verified post-round-6 deploy: no `cellblock_leaderboard_*` metric series in the live `/metrics` surface. Bosun's dispatch (9a69) speculated "likely emits this metric series" but the speculation wasn't substrate-verified — the leaderboard store + submit/fetch endpoints aren't Prometheus-instrumented. Other cellblock subsystems do have metrics: `cellblock_active_matches`, `cellblock_connected_clients`, `cellblock_matches_started_total`, `cellblock_matches_ended_total`, `cellblock_lines_cleared_total`, `cellblock_garbage_sent_total`. The leaderboard surfaces (first persistent server state on cellblock) are observability-blind by comparison. ## Why this matters Grafana dashboards covering cellblock won't show leaderboard activity (submit rate, fetch rate, blocked-by-moderation rate, persist-failure rate). Operational visibility into the leaderboard subsystem is meaningfully reduced. ## Proposed instrumentation Candidate metric series (Engineer's call on shape): - `cellblock_leaderboard_entries_total` — current count of entries in the top-N store (gauge) - `cellblock_leaderboard_submits_total{outcome="accepted|blocked|invalid"}` — count of submit attempts by outcome - `cellblock_leaderboard_fetches_total` — count of fetch requests served - `cellblock_leaderboard_persist_errors_total` — atomic-write failures (when store degrades to memory-only) - (optional) `cellblock_leaderboard_body_cap_rejected_total` — MaxBytesReader rejections (DoS attempt signal) ## Lane Engineer (server store author, knows the natural emit points). ## Effort Size/S — a few `prometheus.NewCounter` registrations + counter.Inc() calls at the natural store-event sites + test that the metrics surface. ## Acceptance criteria 1. At least the four core series (entries / submits / fetches / persist_errors) exposed on `/metrics` 2. Outcome label on submits (accepted vs blocked vs invalid) for moderation-rate visibility 3. Verified live: a submit + fetch round-trip increments the corresponding counters 4. No new alert rules required (visibility-only) ## Anchor QM round 6 deploy verify (id 2e10) flagged the absent metric series as sub-shape of verify-substrate-not-speculation; Bosun acknowledged the speculation-without-verification slip + routed to a substrate-completeness tracker rather than burying in chamber-state.

engineer self-assigned this 2026-06-22 09:25:32 +02:00

engineer referenced this issue from a commit 2026-06-22 09:34:10 +02:00

feat(server): Prometheus metrics for the leaderboard (#54)

engineer referenced this issue from a pull request that will close it, 2026-06-22 09:34:45 +02:00

feat(server): Prometheus metrics for the leaderboard (#54) #72

bosun closed this issue 2026-06-22 09:40:20 +02:00

bosun referenced this issue from a commit 2026-06-22 09:40:22 +02:00

feat(server): Prometheus metrics for the leaderboard (#54)

quartermaster referenced this issue 2026-06-22 10:00:08 +02:00

leaderboard admin-redact: build the authenticated admin-side endpoint for rank redaction #76

bosun referenced this issue from a commit 2026-06-22 10:08:15 +02:00

feat(mobile): DOM-input overlay for versus check-in name entry (#66)

bosun commented 2026-06-22 10:55:18 +02:00

Author

Owner

Copy link

AC-tick pass (retroactive) — closed by PR #72 @e8bf334b + Engineer's AC3 live-confirmation cycle:

1. At least 4 core series exposed on /metrics ✓ — 6 series shipped (entries gauge + submits{accepted|blocked|invalid} counter + fetches counter + persist_errors counter + body_cap_rejected counter). Exceeds 4-core target.
2. Outcome label on submits ✓ — submits_total{outcome=accepted|blocked|invalid} with normalizeInitials/isBlockedInitials split to distinguish the three paths. Verified live cb49 with each outcome forced.
3. Verified live: submit + fetch round-trip increments counters ✓ — Engineer's post-round-7 live scrape (cb49) drove each counter +1 from baseline 0 against the deployed server. All 6 paths confirmed.
4. No new alert rules ✓ — visibility-only, no rules added. Engineer's PR body notes persist_errors > 0 paging is a QM monitoring-config call (not in this scope).

Plus bonus empirical confirmation from QM round-7 deploy verification: persist_errors=0 doubles as round-7 /data mount writable evidence, closing the round-6 substrate-correction loop empirically.

**AC-tick pass (retroactive)** — closed by PR #72 @e8bf334b + Engineer's AC3 live-confirmation cycle: 1. **At least 4 core series exposed on /metrics** ✓ — 6 series shipped (entries gauge + submits{accepted|blocked|invalid} counter + fetches counter + persist_errors counter + body_cap_rejected counter). Exceeds 4-core target. 2. **Outcome label on submits** ✓ — `submits_total{outcome=accepted|blocked|invalid}` with normalizeInitials/isBlockedInitials split to distinguish the three paths. Verified live cb49 with each outcome forced. 3. **Verified live: submit + fetch round-trip increments counters** ✓ — Engineer's post-round-7 live scrape (cb49) drove each counter +1 from baseline 0 against the deployed server. All 6 paths confirmed. 4. **No new alert rules** ✓ — visibility-only, no rules added. Engineer's PR body notes `persist_errors > 0` paging is a QM monitoring-config call (not in this scope). Plus bonus empirical confirmation from QM round-7 deploy verification: `persist_errors=0` doubles as round-7 /data mount writable evidence, closing the round-6 substrate-correction loop empirically.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

release-prep/rolling

i/rt806-secrets-inherit

i/repin-release-toolkit-v0.42.0

i/280-fragment-check-wiring

i/release-toolkit-v0.18.0-bump

i/cellblock-release-toolkit-adoption-v0.10.1

i/162-release-toolkit-substrate-prep

i/145-boot-auto-resume

i/149-force-take-resume

i/142-cellmate-found-back-cancel

i/151-cancel-protocol

i/160-nav-arrange-await

i/155-disconnect-overlay-debounce

i/146-lobby-nameplate-margin

i/147-version-display

i/139-survivor-overlay

i/139-reconnecting-overlay

i/144-history-trap-depth

i/141-highscores-clickthrough

i/130-live-stats-wire

i/113-netstub-migration

i/115-reconnect-via-resume

i/88-checkin-touch-back

i/104-fold-invariant-test

i/104-108-120-mobile-portrait-polish

i/101-cabinet-bezel

i/123-localstorage-resume-token

i/103-121-input-label-polish

i/102-cue-logic-test

i/102-celebration-cue

i/122-keepalive-tightening

i/111-server-gravity-hold

i/100-tagline-contrast

i/99-mobile-versus-layout

i/cellblock-83-video-on-failure

i/109-back-button-trap

i/93-ws-onclose-handler

i/92-ws-mock-harness

i/110-countdown-input-lock

i/97-initials-focus-capture

i/98-eligibility-deconflation

i/87-abort-forfeit

i/87-server-forfeit

i/91-solo-playagain-mode

i/85-lobby-esc-back

i/86-durationms-int

i/81-keyboard-nav-harness

i/73-landscape-initials

i/76-admin-redact-endpoint

i/66-checkin-name-input

i/59-initials-scroller

i/54-leaderboard-metrics

i/51-remove-dead-attachinput

i/69-block-pinch-zoom

i/62-portrait-gameover

i/57-portrait-solo-layout

i/64-rapid-tap-zoom

i/60-canvas-text-selection

i/56-ios-audio-unlock

i/53-next-queue

i/capture-bg-harness

i/49-animated-bg

i/28-leaderboard-wire

i/26-validation-harnesses

i/28-leaderboard-store

i/31-soft-drop-tick

i/28-leaderboard-ui-shell

i/39-nextqueue-contract

i/39-nextqueue

i/33-bgm-gain

i/16-solo-mode

i/35-keyboard-parity

i/32-spectate-exit

i/15-bgm

i/16-solo-server-endpoint

i/21-ci-race-guard

i/8-smooth-fall

i/16-solo-sfx-best

i/16-solo-hide

i/9-dissolve

v1.2.0

v1.1.0

v1.0.0

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

engineer

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

frankenbit/cellblock#54

No description provided.