Skip to content

Latest commit

 

History

History
121 lines (101 loc) · 9.49 KB

testing.md

File metadata and controls

121 lines (101 loc) · 9.49 KB

Tests as product documentation

Code map · Desktop scenarios · Agent/Worker scenarios

Choose a user promise, then the layer that can observe its failure. Keep tests beside that owner and preserve independent assertions. A focused filesystem conflict test can document a product guarantee more precisely than another broad happy-path UI journey.

What each layer proves

Layer Appropriate assertion Evidence limit
Pure state / policy Intent, thresholds, branch isolation and request construction No service dispatch or production wiring
Component with real filesystem/database/controlled service Ownership, rollback, authorization, cancellation and recovery Not every production entry or OS
Desktop React + mock IPC Controls, edits, pending/error feedback and emitted commands No Tauri effects, native clipboard or actual daemon mutation
Desktop command authority / native compilation ACL check, configured Mac desktop-pilot --all-targets consumer compilation An allowed/compiled command is not a WebView invocation or native scenario
Real CLI/daemon or Gateway listener Published plans, actual upstream attempts and durable effects Controlled upstreams do not establish live-model answer quality
Native Desktop / installed Agent / live provider The selected real integration path and independently observed outcome Bound to its exact environment, revision and executed scenarios
Frozen compatibility / expected-red / oracle fixture A historical reader, intended rejection or test oracle Not current product completion

Start with the representative cases in the capability map. A file named product, a golden containing green, or a zero process exit does not prove execution. In tools/product-e2e, transaction recovery contains expected-red composition, adapter contracts are component evidence, and oracle tests validate the oracle. The real installed management loop is in publication_process.rs.

Reuse the fixture at the owning boundary

Need Reuse / preserve
Prepare a daemon's current sources, grants and plans Publication Product: its Product.cli is internal Local Control notation, not the standalone CLI; protected input belongs to that fixture's daemon
Configure a real native Agent Agent product support: shared Preview → Apply → status and bounded process capture; native argv/ownership/result interpretation remain in the leaf
Prove saved model configuration Additional-model journey: ordinary native startup reads saved settings, without injected provider/endpoint/credential overrides
Observe native routing and rejected attempts NativeContextUpstream: count every model POST, including rejected endpoints/models/credentials; keep the ledger for positive route/history assertions
Prove exact Continue, concurrent routes and cancellation Native context, boundary journey, boundary oracle guide
Prove restoration of old durable state Frozen producer fixtures, crash-boundary tests: old bytes through the current consumer, not a new serializer pretending to be history
Preserve native probe failure classification Pi probe tests, immutable launcher: mutable test bodies are interpreter inputs; executable availability and version/output failures have separate assertions
Exercise actual Desktop controls Browser scenario map and native acceptance: select stable scenario IDs and require every declared outcome

The publication fixture leases its listener address across daemon restarts; do not substitute a released bind(0) probe. Its protected-input channel cannot be attached to an existing Desktop. Prepare that instance first, then use real UI mutations. Main-Agent delegation must separately invoke the actual public CLI.

Keep borrowed user resources, writable native acceptance configuration, fresh product storage and per-run materials separate. Each product instance needs a private runtime/storage namespace. CLI submissions have no local receipt directory; recovery uses the original key and request, with title/detail lookup only as an aid. CLI submission regressions cover denied local writes and uncertain responses; durable admission and native Main journeys remain separate. Expose the selected CLI and Pi's selected Node consistently to main-Agent checks and Worker launch. Never repair permissions on pre-existing user directories to satisfy a fixture; cleanup verifies the owned object's identity and preserves foreign files.

For a native run, read the ecosystem's acceptance entry for explicit inputs, writable roots, required outcomes and cleanup. Report a preparation/cleanup failure separately while retaining the first product failure. Process exit, successful setup or the model's success text cannot replace a Gateway receipt and an independent tool artifact.

Keep assertions independent of implementation layout

Build legal current state through current producers; compute expected outcomes independently. A historical journal fixture must retain its original producer bytes, including relevant feature combinations such as JSON ordering. Register recovery readers and removal conditions in the compatibility registry.

Require the capability under test rather than a fixed total number of Agents. Keep inventory completeness at the registry boundary: registry, bundle source and bundle producer must agree. Changed deterministic generators run twice; the second run must leave zero diff.

Use shared enable/adjust/disable and blocked-action recovery scenarios from the Agent feature guide. Native protocol differences alone do not justify an extra verification control. Browser fixtures, including standalone editors, select stable IDs rather than positional slices or translated labels; require nonempty selection and all expected outcomes.

Before consolidating tests, list their unique failure conditions. Preserve crash, stale-preview, cancellation, foreign-file, authorization, timeout and compatibility cases even when happy paths overlap. For a broken gate, demonstrate that a realistic bad case fails. For managed native overrides, retain a positive control showing that the conflicting native setting would otherwise be consumed.

Protocol changes reuse saved-route and native-context journeys across Responses and Messages. Assert actual request paths, trusted receipt identity, rotation and independent restore. A removed-route witness uses its previously working protocol and proves a typed authorization rejection plus zero upstream requests; an unknown HTTP path or old restore fixture does not prove route revocation or native execution.

Select and report validation

  1. Run python3 scripts/test-plan.py --base <exact-base>; use --integration for branch integration. Review affected consumers and explain semantic adjustments.
  2. Follow the selected commands and CI definitions. Use scripts/validation.py for configured hosts, exact committed Rust candidates, feature plan, phase and related runs. Compile affected native consumers early; featureless Desktop checks and zero-selected tests do not satisfy that gate.
  3. Diagnose focused failures, freeze after affected checks pass, then complete the selected final checks once. Missing required cases remain unexecuted, not green.
  4. Record revision, command, platform, cases, result and evidence location. Preserve reused evidence at its original SHA; assess the three slowest measured modules once without automatically refactoring the frozen candidate.

Final backend DAG admission and preparation are owned by validation-schedule.py, with regressions. CPA bridge process fixtures run in an isolated lane before other listeners; Gateway runtime consumers wait for prepared product roles.

Code-map and local ownership-guide edits require link/owner review and the selected contract checks, not new model calls. Runtime-embedded Markdown and shipped Skills can change behavior and need their actual consumers. Instructions and compilation remain distinct from executed business evidence.