Skip to content

feat(observability): add managed monitoring stack and diagnostics - #2388

Merged
AlexStocks merged 25 commits into
oceanbase:masterfrom
happy-v587:feat/1985-observability-stack
Nov 5, 2026
Merged

AlexStocks merged 25 commits into
oceanbase:masterfrom
happy-v587:feat/1985-observability-stack

Conversation

@happy-v587

@happy-v587 happy-v587 commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Which issue or RFC does this PR close?

Closes #1985.

Implements the optional deployment described in RFC 0046.

Rationale for this change

Operators currently have to assemble collection, storage, dashboards and alert diagnostics themselves. Provide an opt-in stack and external-platform connections to inspect service health, processing progress and dependencies through correlated metrics, logs and traces.

What changes are included in this PR?

  • Package the Docker Compose stack, pinned component configuration, storage, authentication, retention, dashboards, alerts and runbooks. Include Prometheus, Grafana, Loki, Tempo, OpenTelemetry Collector and Phoenix. The demo and managed deployment share a bounded trace/log profile. Server installations declare the process-metrics dependency.
  • Add powercontext observability up|status|check|down|connect and server run --observability for existing Servers, containers, split API/background deployments and native personal services. Reject conflicting signal-specific trace/log exporter overrides from environment files and process environments.
  • Add bounded Provider and Worker observations, HTTP status, retrieval/exact-read, processing and Bootstrap delivery metadata. Preserve worker cleanup, cancellation and failure evidence through policy blocks and resumed execution until acknowledgement. Serialize retained log timestamps as ISO-8601 UTC. Observe the wire delegate inside inference privacy protection, preserving admission and response-release boundaries.
  • Connect service, Scope and Provider dashboards to matching logs and traces, with sampled exemplars where available. Keep the operational field allowlist and exclude content, credentials and execution identities from metric labels. Update both documentation locales.
  • Regenerate HTTP/Desktop contracts and qualify the combined twelve-operation Desktop protocol. The exact-Scope statistics adapter accepts declared optional observation fields. Preserve every historical qualification identity and add a separate statistics record. Receipt statistics read existing records; this adds no database migration. Keep macOS image-busy recovery bounded and backend acceptance budgets inclusive of cleanup.
Server /metrics -> Prometheus -> Grafana dashboards and alerts
Server/Worker logs and traces -> Collector -> Loki / Tempo / Phoenix
Grafana diagnostics -> related logs and traces when evidence is available

Are there any user-facing changes?

  • Users can explicitly install the monitoring stack or connect an existing OTLP/Prometheus platform. Normal SDK use does not configure logging or require a monitoring backend.
  • Grafana provides service, Scope, processing and Provider diagnostics with evidence links and documented missing/stale-data semantics. Exercise probes are explicit.
  • Statistics fields and monitoring settings are additive. Desktop statistics remain available with the combined contract through its separately qualified profile. No intentionally breaking API or stored-data change is included. Worker cleanup, privacy, cancellation and export failure isolation are the main runtime risks.

How was this change tested?

  • make check passes, including hooks, lock validation, typing and all 61 integration-manifest cases. The OceanBase lifecycle-budget repair passes all 33 CI pipeline cases on real Linux/GNU timeout without skips. The exact failed factory node completes against an isolated real OceanBase instance on macOS in 58.91 seconds with zero exit and complete, unskipped JUnit; the hosted Linux slowdown was not reproduced locally. make contract-test passes all 62 cases. Desktop generated-contract and IPC drift checks, Rust formatting and Clippy with --all-targets -- -D warnings pass.
  • pnpm typecheck, pnpm test and pnpm lint pass; all 183 frontend tests succeed. Local Node is 25.2.1; CI's specified Node 24.14.1 remains a separate environment.
  • cargo +1.95.0 test --locked --manifest-path desktop/src-tauri/Cargo.toml --lib --tests passes 133 tests with two explicit ignores. The public statistics regression first reproduced InvalidResponse for the Server's legal processing/bootstrap_delivery fields, then passed after the typed adapter repair. Existing exact-Scope, null-estimator, revocation, cancellation and deployment-identity tests pass. The matching-contract product gate failed before the separate twelve-operation qualification was added and now passes.
  • The real built-wheel SQLite harness passes anonymous loopback, static Bearer, HTTPS with explicit CA/base path, and enforced Provider modes. A default native probe at clean Desktop commit e4d5e873 uses ordinary product-profile selection and reports bundled product records for sqlite-observability-stats-4568b499-v1. It covers empty/persisted inventory, UTC today/7d/30d, partial usage, independent Memory operations, Scope isolation, revocation, uncertain writes without replay and Server survival. The wheel's 736 runtime files match Server source 4568b499; source, wheel and contract digests are recorded in the qualification. Partial usage is seeded Runtime-recorder input, not a real model call. All 46 master and 42 prior feature records retain their exact identities after rebasing onto master 2615f7fc.
  • Main run 41877415157 at head 01dd266b completed 103 jobs: 87 successful, twelve conditional skips, one cancelled and three failed. All Python 3.11–3.14 unit/e2e and four Desktop platform jobs passed. The direct failure was OceanBase clean migration: a passing factory test body reached the 180-second process cutoff before complete JUnit; two aggregate gates then failed. Adjacent complete grant cases took 158–169 seconds before interpreter overhead. The clean-migration invocation now explicitly selects 300 seconds, with a 90-minute step and a 175-minute enclosing job. The macOS ARM candidate was cancelled before acquiring a hosted runner; GitHub's annotations report capacity constraints. Fresh CI must verify this head. Earlier isolated Docker execution verified authenticated collection, provisioning and matching Tempo/Loki/Prometheus evidence; that full stack was not rerun for this workflow-only repair.

Local native/SQLite protocol results do not establish installed UI, native credential persistence, Windows execution, external production acceptance or release readiness. Discovery does not attest a remote binary. Backend acceptance retains business assertions and complete unskipped JUnit checks; SeekDB has a 40-minute migration step within an 80-minute job. OceanBase clean-migration and long-query acceptance select the 300-second process budget; other isolated calls retain the 180-second default. No retry, skip-policy change or relaxed assertion is included.

AI usage statement

OpenAI Codex assisted with implementation, conflict resolution, tests, documentation and this PR description. Validation distinguishes local execution from GitHub Actions results.

Comment thread deploy/observability/init.py Resolved
@happy-v587
happy-v587 force-pushed the feat/1985-observability-stack branch 2 times, most recently from 0c976ea to 2e96771 Compare November 3, 2026 01:40
@AlexStocks AlexStocks added the 1.5.1 v1.5.1 release tracking label Nov 3, 2026
@happy-v587
happy-v587 force-pushed the feat/1985-observability-stack branch from 69a82da to 6e07f28 Compare November 3, 2026 08:22
@happy-v587
happy-v587 force-pushed the feat/1985-observability-stack branch from 6e07f28 to 8dfc9c8 Compare November 3, 2026 13:00

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the managed logging configuration conflict below before merging.

@happy-v587
happy-v587 force-pushed the feat/1985-observability-stack branch from 6aa1e1d to b37639c Compare November 3, 2026 18:12

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@happy-v587
happy-v587 force-pushed the feat/1985-observability-stack branch from 9388e3e to d185090 Compare November 4, 2026 06:38
@happy-v587
happy-v587 requested a review from AlexStocks November 4, 2026 08:06
@happy-v587

Copy link
Copy Markdown
Collaborator Author

relate #2371

Add an opt-in monitoring stack with Grafana, Prometheus, Loki, Tempo,
OpenTelemetry Collector and Phoenix, plus external-platform connections.
Provide CLI lifecycle commands and native service opt-in profiles.

Expose correlated service, Scope, processing and Provider diagnostics.
Observe HTTP responses, model calls, Bootstrap delivery and Artifact reads
without exporting content, credentials or unbounded identity labels.
Preserve Worker cleanup, failure diagnostics and managed logging behavior.

Update generated API contracts, bilingual operating guides and focused
coverage for signal correlation, failure isolation and spawned Workers.

Refs oceanbase#1985
Skip the four deployment directory symlink aliases that license-eye reads
as files, while retaining checks on the real packaged resources. Add the
required Apache-2.0 headers to 54 source and configuration files, keeping
the Docker syntax directive first and all original source bytes intact.

Validate with the pinned license-eye v0.9.0 checker, make check and the
62 API contract tests.
Run exporter-dependent configuration coverage in the OTLP job and require zero skips. Keep CLI defaults, real tracing identity and collector failure recovery in sync with their public contracts. Accept the provider_factory keyword in controlled service-chain models. Classify generic timeouts at worker boundaries so synchronous flush retains processing_failed.
Scope revalidation shares the 800 ms Stop budget, so a retry can time out before it reads its existing pause. Reproduce that path with a delayed real Scope response and verify persisted receipt identity, target position, retained pause and no duplicate flush instead of requiring one timing-dependent diagnostic.
Server process inventory imports psutil independently of the optional code integration. Include it in the server dependency closure and refresh the root, harness and evaluation locks so minimal server installations, native services and managed packages can start without the code extra.
@happy-v587
happy-v587 force-pushed the feat/1985-observability-stack branch from 3157985 to 01dd266 Compare November 4, 2026 17:25
happy-v587 and others added 4 commits November 5, 2026 07:21
Co-Authored-By: Claude Code <noreply@anthropic.com>
Handle pnpm version diagnostics without consuming simulated build attempts and isolate GitHub step output. Include the latest master seekdb migration scheduling update.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Keep the managed logging and observability declarations while adopting the shared Skill layout, SQLite worker cleanup, and the tested migration timeout contract.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@AlexStocks
AlexStocks merged commit 20f64d7 into oceanbase:master Nov 5, 2026
12 of 13 checks passed
jackie-cqz added a commit to jackie-cqz/powercontext that referenced this pull request Nov 6, 2026
Resolve conflicts with the managed observability stack (oceanbase#2388) and the 60
other commits that landed since the previous merge.

- `server/cli.py`: keep `powercontext.env_file` and the relocated
  `server.inference_notice`, and add the new observability notice import.
- `cli/observability_notice.py` is new upstream and only the Server's
  `serve` command uses it. It moves to `server/observability_notice.py`, as
  `inference_notice` did, so the Server does not import the CLI package.
- `release/upgrade-support.json`: keep the rationale added upstream for
  observability and the SQLite connection-worker drain, append the sentence
  about this branch's code-organization changes, and recompute the persistence
  digest after formatting. The generated table and index DDL for all 75
  tables is identical to master on the SQLite and MySQL dialects.

Follow the moves into code that arrived with the merge:

- The observability profile, the service launcher, the demo and host scripts
  under `deploy/observability`, and the new observability tests import
  `powercontext.env_file` and `powercontext.builtin.authz`.
- Five new provider-observability test calls stop passing the removed
  `scheduler_path` argument.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.5.1 v1.5.1 release tracking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: ship a reference observability stack with dashboards and alerts

6 participants