Skip to content

fix(process): observe already-due trailing callbacks before settling the capture journal - #1700

Merged
morluto merged 1 commit into
morluto:mainfrom
JasonBuildAI:fix/process-capture-journal-settle-6f2c
Oct 11, 2026
Merged

morluto merged 1 commit into
morluto:mainfrom
JasonBuildAI:fix/process-capture-journal-settle-6f2c

Conversation

@JasonBuildAI

Copy link
Copy Markdown
Contributor

Summary

Fixes #1697. Process-capture Evidence can silently drop the terminal tail when the host stalls the event loop around exit: settleProcessCaptureJournal judged the journal quiet by wall-clock time alone, so a trailing observation callback that was already due ran after the settle returned and never reached the immutable capture. PTY capture results now retain the trailing observations they were always meant to keep.

Problem and expected behavior

When the event loop stalls longer than the quiet window (quietMs, 25 ms by default) and then resumes, the settle's own wake-up timer and a trailing observation callback are both overdue. Due timers run in expiry order, so the earlier wake-up evaluates the journal before the trailing callback has appended its observation, decides the journal is quiet, and finalizes the capture. The trailing observation is dropped without any truncation flag.

Smallest trigger (unit level, deterministic): a trailing 12 ms timer that appends to the journal, plus a 40 ms synchronous stall right after starting settleProcessCaptureJournal(journal, 20, 200).

Expected: the settle does not declare silence while callbacks that were already due are still queued; trailing observations delivered before the capture becomes immutable are retained.

Change and scope

  • settleProcessCaptureJournal now awaits setImmediate() after each quiet-window sleep, so timer and poll callbacks that were already due run before silence is judged.
  • Regression test: does not settle while a trailing observation is already due behind a stalled event loop in src/process/capture/ProcessHarness.test.ts.

Intentionally not included: no change to quiet/max-wait defaults, no rework of capture finalization, no new APIs.

Contract and boundary impact

  • Semantic owner and earliest changed stage: process capture lifecycle; the earliest affected stage is finalization after process exit.
  • CLI and MCP/tool-catalog contract: none; tool schemas and results are unchanged.
  • Provider, bridge, target-format, or platform compatibility: none; the logic is platform-independent.
  • Evidence, artifact, provenance, or reconstruction contract: capture Evidence keeps trailing observations that it could previously drop; no format change.
  • Process execution, authorization, cleanup, or containment impact: none.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact: none.

Evidence and regression coverage

Evidence is executed on the change branch and on the base revision.

  • Tests added or updated: new deterministic regression in src/process/capture/ProcessHarness.test.ts. The existing waits for terminal observations delivered after the exit callback started failing 1 in 5 standalone runs (and reliably in a 65-file run) on Windows; it passes 6/6 runs after this change.
  • Base reproduction (base ee46323 with only the new test applied, fix reverted):
FAIL  |adapters| src/process/capture/ProcessHarness.test.ts > does not settle while a trailing observation is already due behind a stalled event loop
AssertionError: expected [ { capture_order: +0, …(2) } ] to have a length of 2 but got 1
 Test Files  1 failed (1)
      Tests  1 failed | 30 skipped (31)
  • After the change:
 Test Files  1 passed (1)
      Tests  1 passed | 30 skipped (31)
  • Project focused lane after the change:
npm run test:focused -- src/process/capture/ProcessHarness.test.ts src/process/capture/ProcessCaptureLifecycle.test.ts
 Test Files  2 passed (2)
      Tests  55 passed (55)
  • User-visible CLI/MCP output (if applicable): none; result shape is unchanged.
  • Remaining proof gaps: no whole-harness PTY stall repro (would need a controlled host stall during a real capture); the unit-level regression pins the settle contract directly.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

  • npm run test:focused -- src/process/capture/ProcessHarness.test.ts src/process/capture/ProcessCaptureLifecycle.test.ts — 55 passed
  • npm run check:fast — typecheck and lint passed (also ran in the pre-push hook)
  • npx oxfmt --check src/process/capture/ProcessHarness.test.ts src/process/capture/ProcessCaptureLifecycle.ts — clean
  • Note: a broader Windows source-test run shows 12 pre-existing platform failures unrelated to this change (symlink creation EPERM, simulated win32 platform paths, missing locally built native artifact). This change removes the one failure it targets.

Compatibility, safety, and release

  • Breaking changes or migration steps: none.
  • Real Hopper/Ghidra, browser, or OS coverage: not applicable; the changed code is the platform-independent capture settle path.
  • Package or release metadata impact: none.
  • Security, privacy, process, or containment review: none; the change adds one event-loop turn per quiet-window iteration.

Review checklist

  • This PR addresses a concrete problem or an agreed enhancement.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed (none needed).
  • User-visible CLI/MCP changes include representative output (not applicable).
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

AI disclosure: this change was drafted with AI assistance at my request, under my direction and review.

…the capture journal

A stalled event loop can leave the settle wake-up and a trailing observation
callback due at the same resume. The earlier wake-up runs first, declares
silence, and the capture becomes immutable before the trailing callback
appends its observation.

Drain the loop turn after the quiet sleep so timer and poll callbacks that
were already due run before silence is judged, and cover the stalled-loop
case with a deterministic regression test.

Fixes morluto#1697
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@morluto

morluto commented Oct 11, 2026

Copy link
Copy Markdown
Owner

@codex

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-11T11:04:34.948567Z 1658564 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 1658564d90

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@morluto
morluto merged commit ed084b3 into morluto:main Oct 11, 2026
31 checks passed
@github-actions github-actions Bot mentioned this pull request Oct 11, 2026
3 tasks done
@CassieFox23

Copy link
Copy Markdown
Contributor

TL;DR: The PR's new test fails on the base and passes on the PR, on Node 22, 24 and 26, and the patch applies to current main. I could not make a real PTY capture lose its tail on Linux, on the base or on the PR, in 600 stalled runs. So the fix is proven at the unit level only.

Results

Base is ee46323, the merge base. Head is 1658564, the head that merged as ed084b3 after my runs. Main is 2165045 (#1733), with the patch applied.

Check Node 22.23.1 Node 24.18.0 Node 26.11.1
New test on base (test file copied in) fails fails fails
New test on head passes passes passes
Both capture test files on base (55 tests) 54 pass, 1 fails 54 pass, 1 fails 54 pass, 1 fails
Both capture test files on head 55 pass 55 pass 55 pass
Both capture test files on main plus the patch 55 pass 55 pass 55 pass

The base failure is the one in the PR body: "expected ... to have a length of 2 but got 1".

Whole-capture runs, Node 24 only. Each run was a real PTY capture of a small shell command whose last line is FINAL-TAIL-LINE. I called REA's built API in-process and blocked the event loop for 60 ms with a synchronous wait. The last stall row blocked it three times per run. The big-output rows print 15,000 lines first.

Stall point Runs per tree Final line in raw output, base Final line in raw output, head
None 50 50 of 50 50 of 50
Inside the settle loop 50 50 of 50 50 of 50
Just before the exit event 50 50 of 50 50 of 50
When the last output chunk arrives 50 50 of 50 50 of 50
All three together 50 50 of 50 50 of 50
Big output, no stall 50 50 of 50 50 of 50
Big output, settle stall 50 50 of 50 50 of 50
Big output, all three stalls 50 50 of 50 50 of 50

No capture failed validation in any run. In two big-output runs on head, the screen text missed the final line. More checks has the details.

Why I think it did not reproduce, from reading the code. On Linux, node-pty sends the exit event only after the PTY socket closes, so no raw output arrives after exit. REA cancels its timed inputs at exit. It also waits for settlement and stops the sampler before the settle loop starts. The screen renderer is the one late producer, and REA waits for it again before it builds the capture.

One finding on both trees: the settle loop still returns early. With big output and a stall, some screen updates were logged after the final file snapshot. Base: 3 of 50 with the settle stall and 4 of 50 with all three stalls. Head: 3 of 50 and 3 of 50. With no stall, 0 of 50 on both. Nothing was lost, because REA's own renderer wait caught them. By the code and these counts, the PR waits for due timers but not for slow renderer work. I did not look into why the renderer was late.

Not run

  • Windows and macOS. The issue came from Windows, where capture is not supported.
  • Stalls inside the child process. My stalls hit REA's process only.
  • A whole-capture run where the PR changes the result. I found none.
  • Main without the fix, for the new test.
  • The full test suite and the lint and type checks.

Thanks, JasonBuildAI, for the clear issue and the small, focused fix.

More checks
Check Base Head
Capture and 6 busy loops all pinned to one CPU core, 50 small and 50 big runs 100 of 100 kept 100 of 100 kept
Older exit-callback test, 20 runs on an idle box 0 failures 0 failures
Same test, 20 runs with 4 busy loops on its core 0 failures 0 failures
Two no-stall captures, ids and times removed, compared by diff same as head same as base
Time per capture, small command, in-process, median of 50 396 ms 398 ms
Time per capture, CLI wall clock, median of 20 1,477 ms 1,482 ms

The timings are rough. Other test jobs shared the box. The change adds one event-loop turn per 4 ms wait, and I saw no cost from it.

I removed ids, timestamps and elapsed times with jq, then ran diff. Base against head gave no differences. Two base captures against each other gave none, and so did two head captures. REA's own compare command said "changed" for base against head. The first difference it named was a settle time of 99 ms against 101 ms.

Two of the 150 big-output runs on head were marked truncated. They had 170 and 168 raw frames, and no other run had more than 149. Base had none. In both, the raw output kept the final line, but the screen text did not. One had no stall at all. I think output size caused it, not the fix, but I did not check that. I did not read the truncation details.

Commands
# trees from the read-only mirror, built in /tmp (Node 24, npm ci, npm run build, npm run test:prepare)
git -C ~/code/rea-mirror/rea.git archive 1658564d9 | tar -x -C /tmp/pr1700/head
git -C ~/code/rea-mirror/rea.git archive ee46323b4 | tar -x -C /tmp/pr1700/base
git -C ~/code/rea-mirror/rea.git archive 21650455a | tar -x -C /tmp/pr1700/main
git apply --check pr1700.patch          # in the main tree, exit 0; then git apply

# unit tests, per Node (22 = /usr/bin/node, 24, 26)
node node_modules/vitest/vitest.mjs run src/process/capture/ProcessHarness.test.ts \
  -t "does not settle while a trailing observation"
node node_modules/vitest/vitest.mjs run src/process/capture/ProcessHarness.test.ts \
  src/process/capture/ProcessCaptureLifecycle.test.ts

# whole-capture runs: calls captureProcessScenario from dist, 60 ms Atomics.wait stalls
PROG='echo first-line; echo second-line; echo FINAL-TAIL-LINE' \
  node stall-runner.mjs /tmp/pr1700/<base|head> <none|settle|exit|data|all> 10 out.jsonl
PROG='seq 1 15000; echo FINAL-TAIL-LINE' node stall-runner.mjs ...

# CPU contention: busy loops and the runner all pinned to one core
taskset -c 11 node -e 'for(;;){}' &     # 6 of these, killed by PID afterwards
taskset -c 11 node stall-runner.mjs /tmp/pr1700/<base|head> none 10 out.jsonl

# CLI
node scripts/rea.mjs capture-process scenario.json --json > cap.json
node scripts/rea.mjs compare-process-captures a.json b.json --json
Test setup

🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

@morluto morluto added bug Something isn't working area: runtime Process capture and runtime observation labels Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: runtime Process capture and runtime observation bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Process capture can settle its event journal while already-due trailing observations are still queued

3 participants