feat: surface circuit-breaker state and per-domain cooldown counts as Prometheus metrics

[oten91]

Summary

Two metrics that were previously only observable via /ready introspection or direct Redis access are now first-class Prometheus gauges:

• path_circuit_breaker_state{service_id, domain} — 1 if domain is currently locked out, 0 otherwise
• path_endpoints_in_cooldown{domain, rpc_type, service_id} — count of endpoints currently in strike cooldown

Both metrics fill real operational visibility gaps that came up during PR #512 work — operators could see broken-domain effects in error-rate graphs but couldn't directly answer "which domains are circuit-broken right now?" without Redis access.

What this enables on dashboards

Once Grafana picks these up:

• "Currently Broken Domains" table — list-of-domains view with service_id + domain + state, sortable by service.
• "Broken Domains" stat tile — single number for at-a-glance health (0 = healthy, >5 = widespread infra issues).
• "Cooldown" column on the Supplier Quality table — operators can see "how many of my endpoints are in cooldown right now?" alongside RPS, success%, etc.

These dashboard panels are not in this PR (kept to the metric-only diff for review clarity); they'll go in a small follow-up commit that updates local/observability/dashboards/*.json once this metric is deployed.

Implementation notes

path_circuit_breaker_state

State transitions in gateway/domain_circuit_breaker.go:

Trigger	Gauge becomes
MarkBroken	1
ClearService	0 (for every cleared domain)
refreshLocal finds expired entries	0 (for each expired domain)
refreshFromRedis finds expired entries	0 (for each), and re-asserts 1 for currently-broken domains so fresh pods that lazily pick up Redis state stay consistent without going through MarkBroken locally

All gauge sets happen outside cb.mu to avoid taking the metrics lock under the cache mutex.

There is a small inherent staleness window: a circuit-breaker entry whose TTL just expired remains at gauge=1 until the cache TTL elapses (5s default) and refreshLocal runs. That's bounded and fine for a dashboard.

path_endpoints_in_cooldown

New LeaderboardDataProvider.GetCooldownCountData(ctx) method, implemented on Shannon's Protocol. Walks active sessions, fetches each endpoint's reputation score, increments per-(domain, service_id, rpc_type) when score.IsInCooldown() returns true.

Published every 10s alongside the existing leaderboard / mean score / supplier score metrics. Resets between snapshots so a domain dropping to zero cooldown'd endpoints shows zero (rather than sticking at its last value via Prometheus' 5-min staleness window).

Test plan

• go test ./... — all green
• go vet ./... — clean
• New test: TestDomainCircuitBreaker_MetricGaugeTransitions covers MarkBroken → 1, ClearService → 0, TTL expiry + refresh → 0
• Canary deploy: verify path_circuit_breaker_state and path_endpoints_in_cooldown series appear in Prometheus
• Trigger a circuit break (or use /admin/circuit-breaker/clear/{serviceId} to test the clear path) and confirm the gauge transitions
• Verify staleness behavior: entry expires → gauge drops to 0 within ~5s

Cardinality

Both new metrics are bounded by services × domains — already low cardinality (~50-200 unique domains × ~80 services on mainnet). No risk of cardinality explosion.

🤖 Generated with Claude Code

[oten91]

Update: added break/recovery reason visibility

Building on the gauge-only first commit, this commit (06f3767) adds:

New metric path_circuit_breaker_events_total{service_id, domain, reason_category, event}

• Counter incremented on each broken/recovered transition
• reason_category is a bounded bucket: retry, batch_transport, batch_heuristic, parallel_retry, heuristic, unknown
• Extracted via classifyCircuitBreakReason from the free-text reason that already flows into MarkBroken (no new logging or call-site changes — same reasons, just bucketed for label use)

Wiring

• MarkBroken → emits broken event with classified reason
• refreshLocal / refreshFromRedis → emits recovered for natural TTL-driven recovery
• ClearService → emits recovered for admin-cleared transitions
• refreshFromRedis resync path deliberately does NOT emit broken events (those would be double-counted with the originating pod's MarkBroken call)

Tests

• TestClassifyCircuitBreakReason — prefix-mapping correctness, order-sensitive
• TestCircuitBreakerEventsCounter — verifies counter increments on MarkBroken → ClearService transitions
• All existing tests still pass

Dashboard panels added (local files only, not in this PR)

1. "Circuit-Breaker Breaks per Second (by Reason)" — stacked timeseries showing rate of breaks decomposed by reason_category. Spike in batch_transport = upstream connection issues; spike in heuristic = bad response patterns from suppliers.
2. "Top Broken Domains (last 1h, by reason)" — topk(15) table over increase(events_total[1h]) for diagnostic attribution.

Cardinality ≈ services × domains × 6 reasons × 2 events ≈ 50K series upper bound. Bounded.

Promotion checklist from earlier still applies; this just adds the events counter alongside the gauge. Same risk profile, same canary soak gives equal coverage. Tests are green, vet clean.