Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,15 @@ _Avoid_: Data point, record, check result
A Monitoring Sample produced by a check that actually ran against the target, whatever the outcome: a clean evaluation (`REALTIME`), a timed-out check (`TIMEOUT`), or a check that errored (`ERROR`). A check failing to reach the target is itself an observation — for most monitor types that is exactly what "down" looks like.

**Synthetic Sample**:
A Monitoring Sample written by the system or an admin rather than by a check: a raw heartbeat receipt (`SIGNAL`), a status pushed through the data API (`MANUAL`), a default-status fill (`DEFAULT_STATUS`), or an incident/maintenance overlay (`INCIDENT`, `MAINTENANCE`).
A Monitoring Sample written by the system or an admin rather than by a check: a raw heartbeat receipt (`SIGNAL`), a status pushed through the data API (`MANUAL`), a default-status fill (`DEFAULT_STATUS`), a last-known-status fill (`CARRIED`), or an incident/maintenance overlay (`INCIDENT`, `MAINTENANCE`).

Comment on lines 45 to +47
**Default Status**:
A monitor's answer to what a minute without a Monitoring Sample means. Exactly one choice from a closed set: nothing (`NONE` — the minute shows no data), a fixed status (`UP`, `DOWN`, `DEGRADED`) written as default-status fill, or Last Known Status. `MAINTENANCE` is not a Default Status (a maintenance overlay is an event, not a fill).
_Avoid_: Fallback status, fill status

**Last Known Status**:
A Default Status choice where a minute without a sample repeats the most recent Alert-Visible Sample — status and latency alike — written as a Carried Sample (`CARRIED`). Carried Samples are themselves alert-visible, so the chain continues from the last live statement: overlays and heartbeat receipts never become sticky, backdated corrections do not change the present, and alerts trigger and resolve on carried minutes like any other. Carry never expires and never backfills: it starts at the tick after the choice is made, and Carried Samples persist as history if the choice is later changed. A monitor with no Alert-Visible Sample yet has nothing to carry — its minutes show no data. Only None-type monitors may choose it; changing the monitor's type away from None resets the Default Status to UP.
_Avoid_: Sticky status, carry-forward mode

**Stale Member**:
A Member whose monitor is no longer an Eligible Monitor (paused or deleted after being added). It remains a Member until explicitly removed, but is excluded from the group score.
Expand All @@ -63,7 +71,7 @@ A notification channel (email, webhook, Discord, Slack) that Alert Configuration
_Avoid_: Notifier, channel

**Alert-Visible Sample**:
A Monitoring Sample that alert evaluation can see: every Observed Sample, plus data-API pushes (`MANUAL`) and default-status fill (`DEFAULT_STATUS`). Raw heartbeat receipts (`SIGNAL`) and incident/maintenance overlays are never alert-visible — while an overlay is active the alert window freezes (alerts neither trigger nor resolve). All alert conditions (status and latency alike) evaluate the same alert-visible timeline.
A Monitoring Sample that alert evaluation can see: every Observed Sample, plus data-API pushes (`MANUAL`), default-status fill (`DEFAULT_STATUS`), and last-known-status fill (`CARRIED`). Raw heartbeat receipts (`SIGNAL`) and incident/maintenance overlays are never alert-visible — while an overlay is active the alert window freezes (alerts neither trigger nor resolve). All alert conditions (status and latency alike) evaluate the same alert-visible timeline.

Comment on lines 73 to 75
**Failure Threshold**:
The number of consecutive Alert-Visible Samples matching the condition required to trigger an Alert.
Expand Down
2 changes: 1 addition & 1 deletion docs/adr/0005-alerts-evaluate-alert-visible-samples.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Alerts evaluate alert-visible samples, not just REALTIME ones

The consecutive-sample checks behind alert evaluation (`consecutivelyStatusFor`, `consecutivelyLatencyGreaterThan`, `consecutivelyLatencyLessThan` in `src/lib/server/db/repositories/monitoring.ts`) consider samples whose type is `REALTIME`, `ERROR`, `TIMEOUT`, `MANUAL`, or `DEFAULT_STATUS` — the "alert-visible" set — instead of `REALTIME` only. Both data-API PATCH endpoints (single timestamp and range) enqueue one alert evaluation after writing `MANUAL` rows. `SIGNAL` rows and `INCIDENT`/`MAINTENANCE` overlay rows remain invisible to alerting.
The consecutive-sample checks behind alert evaluation (`consecutivelyStatusFor`, `consecutivelyLatencyGreaterThan`, `consecutivelyLatencyLessThan` in `src/lib/server/db/repositories/monitoring.ts`) consider samples whose type is `REALTIME`, `ERROR`, `TIMEOUT`, `MANUAL`, or `DEFAULT_STATUS` — the "alert-visible" set — instead of `REALTIME` only. Both data-API PATCH endpoints (single timestamp and range) enqueue one alert evaluation after writing `MANUAL` rows. `SIGNAL` rows and `INCIDENT`/`MAINTENANCE` overlay rows remain invisible to alerting. Amended by ADR 0006: last-known-status fill (`CARRIED`) later joined the alert-visible set under the same invariant.

Two issues drove this. In #633, a GameDig monitor showed DOWN on the status page but never alerted: a down game server makes `GameDig.query` throw, so every down-sample is recorded as `ERROR`, which the old `type = REALTIME` filter excluded — the "N consecutive DOWN" condition could never become true. The same failure mode silently broke gRPC, SQL, and SSL monitors (hard-down records `ERROR`) and API monitors whose outage manifests as timeouts (`TIMEOUT`). In #720, a NONE monitor driven by the data API never alerted for two stacked reasons: PATCH writes `MANUAL` rows the filter excluded, and the endpoint never enqueued evaluation at all. The status page and UPTIME alerts have no type filter, which is why users saw DOWN while alerts stayed silent.

Expand Down
7 changes: 7 additions & 0 deletions docs/adr/0006-last-known-status-fill.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Last Known Status is a Default Status choice that repeats the latest alert-visible sample

Issue #721: push-driven NONE monitors lost their status between pushes in v4 — one red minute, then gray forever — because the per-minute sample model only fills gaps with a static `default_status`. We added a fifth Default Status choice, `LAST_KNOWN`, where each tick with no observed sample writes a `CARRIED` row repeating the most recent alert-visible sample — status and latency alike. It is modeled as a dropdown value rather than a separate "sticky" checkbox so that "what does a minute without a sample mean" stays a single dimension with no conflicting combinations; the same cleanup removed the `MAINTENANCE` option, which the UI offered but the fill engine had always silently ignored (stored `MAINTENANCE`/unknown values migrate to `NONE`, preserving behavior).

The carry source is the most recent **Alert-Visible Sample** by timestamp, and `CARRIED` itself joins the alert-visible set. That one rule does three jobs: incident/maintenance overlays and raw heartbeat `SIGNAL` receipts can never become sticky; backdated data-API corrections cannot rewrite the present (newer carried rows outrank them); and ADR 0005's invariant — every flow that enqueues alert evaluation contributes a row the evaluator can see — keeps holding. The consequences were accepted deliberately: a single DOWN push triggers status alerts once carried minutes meet the failure threshold, alerts never auto-resolve (recovery must be pushed — unlike a fixed default, nothing "resumes"), and enabling the option on a monitor whose last push was DOWN fires the alert shortly after — the bug report, inverted, same as ADR 0005.

Rejected alternatives: a staleness cap ("carry for at most X, then gray") re-introduces "absence means unknown" — the opposite of what the admin just selected — and dead-integration detection is what Heartbeat monitors are for; reusing the `DEFAULT` sample type for carried rows loses the stored distinction between "admin declared absence means X" and "system repeated the last live statement"; backfilling the gap on enable would mutate historical uptime, so carry is tick-forward only and `CARRIED` rows persist as history if the setting is later changed. `LAST_KNOWN` is selectable only on NONE-type monitors — for polled types an observed sample wins the merge every tick, so offering it would be a dormant knob; changing a monitor's type away from NONE auto-resets the Default Status to UP so the invalid combination never persists.
Loading