Repository navigation
fix(process): observe already-due trailing callbacks before settling the capture journal - #1700
Conversation
…the capture journal A stalled event loop can leave the settle wake-up and a trailing observation callback due at the same resume. The earlier wake-up runs first, declares silence, and the capture becomes immutable before the trailing callback appends its observation. Drain the loop turn after the quiet sleep so timer and poll callbacks that were already due run before silence is judged, and cover the stalled-loop case with a deterministic regression test. Fixes morluto#1697
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
TL;DR: The PR's new test fails on the base and passes on the PR, on Node 22, 24 and 26, and the patch applies to current main. I could not make a real PTY capture lose its tail on Linux, on the base or on the PR, in 600 stalled runs. So the fix is proven at the unit level only. ResultsBase is ee46323, the merge base. Head is 1658564, the head that merged as ed084b3 after my runs. Main is 2165045 (#1733), with the patch applied.
The base failure is the one in the PR body: "expected ... to have a length of 2 but got 1". Whole-capture runs, Node 24 only. Each run was a real PTY capture of a small shell command whose last line is FINAL-TAIL-LINE. I called REA's built API in-process and blocked the event loop for 60 ms with a synchronous wait. The last stall row blocked it three times per run. The big-output rows print 15,000 lines first.
No capture failed validation in any run. In two big-output runs on head, the screen text missed the final line. More checks has the details. Why I think it did not reproduce, from reading the code. On Linux, node-pty sends the exit event only after the PTY socket closes, so no raw output arrives after exit. REA cancels its timed inputs at exit. It also waits for settlement and stops the sampler before the settle loop starts. The screen renderer is the one late producer, and REA waits for it again before it builds the capture. One finding on both trees: the settle loop still returns early. With big output and a stall, some screen updates were logged after the final file snapshot. Base: 3 of 50 with the settle stall and 4 of 50 with all three stalls. Head: 3 of 50 and 3 of 50. With no stall, 0 of 50 on both. Nothing was lost, because REA's own renderer wait caught them. By the code and these counts, the PR waits for due timers but not for slow renderer work. I did not look into why the renderer was late. Not run
Thanks, JasonBuildAI, for the clear issue and the small, focused fix. More checks
The timings are rough. Other test jobs shared the box. The change adds one event-loop turn per 4 ms wait, and I saw no cost from it. I removed ids, timestamps and elapsed times with jq, then ran diff. Base against head gave no differences. Two base captures against each other gave none, and so did two head captures. REA's own compare command said "changed" for base against head. The first difference it named was a settle time of 99 ms against 101 ms. Two of the 150 big-output runs on head were marked truncated. They had 170 and 168 raw frames, and no other run had more than 149. Base had none. In both, the raw output kept the final line, but the screen text did not. One had no stall at all. I think output size caused it, not the fix, but I did not check that. I did not read the truncation details. Commands# trees from the read-only mirror, built in /tmp (Node 24, npm ci, npm run build, npm run test:prepare)
git -C ~/code/rea-mirror/rea.git archive 1658564d9 | tar -x -C /tmp/pr1700/head
git -C ~/code/rea-mirror/rea.git archive ee46323b4 | tar -x -C /tmp/pr1700/base
git -C ~/code/rea-mirror/rea.git archive 21650455a | tar -x -C /tmp/pr1700/main
git apply --check pr1700.patch # in the main tree, exit 0; then git apply
# unit tests, per Node (22 = /usr/bin/node, 24, 26)
node node_modules/vitest/vitest.mjs run src/process/capture/ProcessHarness.test.ts \
-t "does not settle while a trailing observation"
node node_modules/vitest/vitest.mjs run src/process/capture/ProcessHarness.test.ts \
src/process/capture/ProcessCaptureLifecycle.test.ts
# whole-capture runs: calls captureProcessScenario from dist, 60 ms Atomics.wait stalls
PROG='echo first-line; echo second-line; echo FINAL-TAIL-LINE' \
node stall-runner.mjs /tmp/pr1700/<base|head> <none|settle|exit|data|all> 10 out.jsonl
PROG='seq 1 15000; echo FINAL-TAIL-LINE' node stall-runner.mjs ...
# CPU contention: busy loops and the runner all pinned to one core
taskset -c 11 node -e 'for(;;){}' & # 6 of these, killed by PID afterwards
taskset -c 11 node stall-runner.mjs /tmp/pr1700/<base|head> none 10 out.jsonl
# CLI
node scripts/rea.mjs capture-process scenario.json --json > cap.json
node scripts/rea.mjs compare-process-captures a.json b.json --jsonTest setup
🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊 |
Summary
Fixes #1697. Process-capture Evidence can silently drop the terminal tail when the host stalls the event loop around exit:
settleProcessCaptureJournaljudged the journal quiet by wall-clock time alone, so a trailing observation callback that was already due ran after the settle returned and never reached the immutable capture. PTY capture results now retain the trailing observations they were always meant to keep.Problem and expected behavior
When the event loop stalls longer than the quiet window (
quietMs, 25 ms by default) and then resumes, the settle's own wake-up timer and a trailing observation callback are both overdue. Due timers run in expiry order, so the earlier wake-up evaluates the journal before the trailing callback has appended its observation, decides the journal is quiet, and finalizes the capture. The trailing observation is dropped without any truncation flag.Smallest trigger (unit level, deterministic): a trailing 12 ms timer that appends to the journal, plus a 40 ms synchronous stall right after starting
settleProcessCaptureJournal(journal, 20, 200).Expected: the settle does not declare silence while callbacks that were already due are still queued; trailing observations delivered before the capture becomes immutable are retained.
Change and scope
settleProcessCaptureJournalnow awaitssetImmediate()after each quiet-window sleep, so timer and poll callbacks that were already due run before silence is judged.does not settle while a trailing observation is already due behind a stalled event loopinsrc/process/capture/ProcessHarness.test.ts.Intentionally not included: no change to quiet/max-wait defaults, no rework of capture finalization, no new APIs.
Contract and boundary impact
docs/public/product-catalog.json), package, or installation impact: none.Evidence and regression coverage
Evidence is executed on the change branch and on the base revision.
src/process/capture/ProcessHarness.test.ts. The existingwaits for terminal observations delivered after the exit callbackstarted failing 1 in 5 standalone runs (and reliably in a 65-file run) on Windows; it passes 6/6 runs after this change.For evidence-bearing changes:
Validation performed
npm run test:focused -- src/process/capture/ProcessHarness.test.ts src/process/capture/ProcessCaptureLifecycle.test.ts— 55 passednpm run check:fast— typecheck and lint passed (also ran in the pre-push hook)npx oxfmt --check src/process/capture/ProcessHarness.test.ts src/process/capture/ProcessCaptureLifecycle.ts— cleanwin32platform paths, missing locally built native artifact). This change removes the one failure it targets.Compatibility, safety, and release
Review checklist
type(scope): outcome.AI disclosure: this change was drafted with AI assistance at my request, under my direction and review.