Repository navigation
FIX-2269: continue a worker's call that dies after sending, before it parks - #4078
Conversation
…t parks
A wait that starts its own work (dispatcher({ wait: true }), a worker's
call) now owes its park through the new optional RequestWaits.owingPark
verb. The engine keeps the obligation as an engine-owned wake on the
request's session; the durability sweep continues an interrupted request
that still owes one, and its replay parks on the work it already started.
The park, the wait's return or throw, the request's end and a stop clear
it; past 10 minutes from the first owe the request is left interrupted.
History, retention and waits.ended read an owed request as parked.
A call's repeated send now dispatches itself on a flow with a pass, so a
call continued after dying between its row and its dispatch has a request
to wait on and is answered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Second look + architecture passOverall: the shape is sound. An engine-owned wake with a sweep branch reuses the existing wake store, lease and continue path, and adds no new storage. The tests show the red→green and include controls. Findings below are ranked; none is a blocker I can prove from reading alone. Worth changing in this PR
Worth a decision or a note (not necessarily this PR)
Architecture notes (opportunistic, scoped to this change)
Test gaps worth one case each
Generated by Claude Code |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c5179bf54c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Simplify / approach review (FIX-2269)
Verdict: The durability shape here looks like the right tradeoff for the spec, not an over-built parallel recovery stack. The gap is real (work started, gate not written, request stuck interrupted), and the fix reuses what already exists: wake store + step 2c + continueUnderLease, extended through drivenAgain / retention / stop. A request-record flag would shrink some read paths but was rejected for adapter surface; Workforce-only recovery would miss dispatcher({ wait: true }). I would not push for a different architecture.
Where complexity is avoidable (follow-ups, not blockers):
-
Duplicated caller wrapper —
sendAndWaitandcallWorkerboth do the sameowingPark ? owingPark(work) : work()dance. Spec review already pointed at a single core helper so dispatch-wait and call cannot drift; that still seems worth doing in this PR or immediately after. -
Optional
RequestWaits.owingPark— Durable hosts wired increate-request-hostalways supply it when wakes exist, so the!== undefinedguards at every call site add noise. Tighter typing (or a smallwithOwingParkin core that no-ops when absent) would read cleaner. -
PR scope — FIX-2269 is joined by FIX-1613 (
taskIssueSegment+ routed-coder e2e) and retained specs for FIX-1613/2268. The harness segment fix is logically tied to proving routed→coder (and FIX-2242 closure), but the diff is ~2.8k lines and mixes three intents. If anything here slips, review gets harder; splitting harness/1613 from engine durability would have been easier to review (process note, not a design flaw). -
Small implementation nits —
createOwingParkusesparkOwedKey(...)!;clearParkOwedre-gets afterowealready returned a version (extra wake read on happy path). Both are minor.
Performance: Extra wake I/O per dispatch-wait/call-send is the intended durability cost. Sweeper continueOwedPark is edge-case only; drivenAgain → owesPark adds wake reads for gate-less interrupted rows when classifying parked vs dead — acceptable unless retention scans huge sessions (worth watching, not a merge blocker).
Verification: @flow-state-dev/engine test suite including park-owed.test.ts passes locally on this branch.
Alternative approaches considered (and why I am not asking for them):
| Alternative | Why not |
|---|---|
| Request-row "owes park" field | Spec dropped E5; every store adapter |
| Engine-only owe inside dispatch seam | Misses call path; hides durable-action contract |
Continue all interrupted |
Unsafe replay |
Inline notes on the duplicated wrapper, bundled harness change, and one engine nit.
Sent by Cursor Automation: Review For Best Approach
There was a problem hiding this comment.
Code Snob. I tried one rewrite. It tied, so there is no POC.
createOwingPark writes a wake with the same fields RequestWakes.owe already writes (flow, user, org, fence, and the sweep arm). I pointed the owe at that verb and kept the clear on the store, because a wake's settle will not clear while its fence is open. Against this head that was −3 lines in park-owed.ts, create-request-host.ts, and request-wakes.ts. The caller still passes the store, the session, the tenant and the fence, and landBy becomes a second now() instead of the same timestamp as dueAt. I threw it out.
A helper for the two clears (runAction, record-request-stop) is longer than the two call sites. continueOwedPark is the four outcomes already on this head (continue, move, clear, give up). sendAndWait and callWorker are each the work plus one ternary. The engine test is one script and a freeze point per case; the workforce test's store proxy is the kill after the row.
Files: packages/engine/src/durability/park-owed.ts, durability-sweeper.ts, stop-suspended.ts, wait-chain.ts, request-wakes.ts, packages/engine/src/context/create-request-host.ts, packages/engine/src/execution/runAction.ts, record-request-stop.ts, packages/core/src/blocks/dispatcher.ts, packages/core/src/types/request-host.ts, packages/workforce/src/messages/call.ts, send-message.ts, packages/engine/test/park-owed.test.ts, packages/workforce/test/worker-calls.test.ts.
Head c5179bf54c417b68c55fa023809868d01b199e8e.
Sent by Cursor Automation: Code Snob
…dispatcher out Codex review on #4078: - P2: a stop could land between the sweep's read of the owe and its continuation. The stop now lets the owe go under the request's lease, and the sweep's admission reads the owe again under it; a stop that finds the lease held answers already-resolved. - P1: runOnce records only after its function returns, so a waiting dispatcher that died after its child was accepted would dispatch a second child on the continuation (the seam takes no stable request id). The dispatcher no longer owes a park; only the call does, whose message id is kept first and taken once. Recorded in DECISIONS for Jake's call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7baa32f455
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex review on #4078 (P2): drivenAgain checked the owed park only when no gate was logged, so a request whose earlier wait's gate was answered and pruned, and which then died in a later call before parking, read as ended to waits.ended and retention. The owe is now read first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c12b48ec1f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex review on #4078 (P2): a call continued after dying between paying its budget and writing its row repeated payPlace with the same message id, counting it twice against the request (refused under perTurn: 1) and appending it twice to the chain. The request now names the ids it paid for and the chain appends an id once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f
Codex review on #4078 (P2): runAction cleared the owe right after the gate was written, before the suspension item reached the request's log. A death in between left an interrupted request with an unlogged pending gate (which the resume refuses as busy) and no owe to continue it. The owe is now cleared after the suspension item is flushed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 96e38d90c5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…owes it Parallel waits in one request share the request's owed-park wake. The wrapper now counts them and clears the wake only when the last settles, and never once one has parked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f
|
Architect: not approving yet. The Codex P1 on — FSD Architect |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 160096750c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A host dies after the turn's send is recorded and before its gate is written; the next host's sweeps continue the request, which parks on the send it already made and is answered in the same request. The control drops the owe and stays interrupted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f
|
Decision: accept the call-only scope in this PR. Replay covers only the worker's call (the 7baa32f narrowing). Building a stable dispatch identity, so a replay can't start a second child, becomes a follow-up issue. Please amend the FIX-2269 spec in this PR so it records the narrower scope and points to the follow-up, then resolve the conflicts with main. |
…-d2ex6z # Conflicts: # packages/engine/src/context/createExecutionContext.ts # packages/workforce/src/messages/send-message.ts
|
Done in d737a24:
Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d737a24da4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f
Red on main since #4078: both hook-timeout failures in worker-calls.test.ts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiGfGi2RaecD1vUwy8K
…sh tests (FIX-2269) (#4204) * FIX-2269: continue a wait that dies after starting its work, before it parks A wait that starts its own work (dispatcher({ wait: true }), a worker's call) now owes its park through the new optional RequestWaits.owingPark verb. The engine keeps the obligation as an engine-owned wake on the request's session; the durability sweep continues an interrupted request that still owes one, and its replay parks on the work it already started. The park, the wait's return or throw, the request's end and a stop clear it; past 10 minutes from the first owe the request is left interrupted. History, retention and waits.ended read an owed request as parked. A call's repeated send now dispatches itself on a flow with a pass, so a call continued after dying between its row and its dispatch has a request to wait on and is answered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f * FIX-2269: keep parkOwedEntry module-private (fallow: unused export) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f * FIX-2269: fence a stop against the owed-park continuation; leave the dispatcher out Codex review on #4078: - P2: a stop could land between the sweep's read of the owe and its continuation. The stop now lets the owe go under the request's lease, and the sweep's admission reads the owe again under it; a stop that finds the lease held answers already-resolved. - P1: runOnce records only after its function returns, so a waiting dispatcher that died after its child was accepted would dispatch a second child on the continuation (the seam takes no stable request id). The dispatcher no longer owes a park; only the call does, whose message id is kept first and taken once. Recorded in DECISIONS for Jake's call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f * FIX-2269: read the owe before the logged gate when classifying a request Codex review on #4078 (P2): drivenAgain checked the owed park only when no gate was logged, so a request whose earlier wait's gate was answered and pruned, and which then died in a later call before parking, read as ended to waits.ended and retention. The owe is now read first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f * FIX-2269: a repeated send pays its message budget once Codex review on #4078 (P2): a call continued after dying between paying its budget and writing its row repeated payPlace with the same message id, counting it twice against the request (refused under perTurn: 1) and appending it twice to the chain. The request now names the ids it paid for and the chain appends an id once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f * FIX-2269: pay the owed park only once its gate is logged Codex review on #4078 (P2): runAction cleared the owe right after the gate was written, before the suspension item reached the request's log. A death in between left an interrupted request with an unlogged pending gate (which the resume refuses as busy) and no owe to continue it. The owe is now cleared after the suspension item is flushed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f * FIX-2269: keep the owed park while another wait in the request still owes it Parallel waits in one request share the request's owed-park wake. The wrapper now counts them and clears the wake only when the last settles, and never once one has parked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f * FIX-2269: integration scenario for a wait killed before it parks A host dies after the turn's send is recorded and before its gate is written; the next host's sweeps continue the request, which parks on the send it already made and is answered in the same request. The control drops the owe and stays interrupted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f * FIX-2269: document owed parks in the execution architecture doc Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f * test(workforce): bound the next host's shutdown drain in the call crash tests (FIX-2269) Since per-worker token usage, a worker's request owes a request-end hook from its start. The dead host's frozen request never pays it, and the debt is kept per process, which both test hosts share, so the next host's dispose waited out its 10 s drain and the afterEach hook timed out. Dispose its router with a short drain first, as the orchestration crash test does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f * test(workforce): bound the drain in a suite afterEach instead of a host clone (FIX-2269) Folds Cursor's review: the crash suite's afterEach disposes the router with a short drain before the file hook disposes the host, and callThenDie keeps open = next. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f * chore(claude-code-style): regenerate the lab inventory table Main's README table went stale with #4183 and turns Test (3/3) red on every PR. Written by the lab's own scripts/inventory.mjs --write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f --------- Co-authored-by: Claude <noreply@anthropic.com>
…nfig (#4206) * FIX-2335: Teammates lab drops projects for one working folder from config The lab's host now runs on the existing constant run source, a git checkout the operator names in TEAMMATES_WORKDIR, listed in localRepositories. Unset, missing or not a checkout, the lab fails at start; it never falls back to the current directory or its own checkout. Project tools, TEAMMATES_PROJECTS and the start-up project writes are gone, and so are the header project select, the New project form, the needs-project card and the repository approval card. Task cards say running and done. No WORKER.md mentions a project. ER-8 and FIX-2154 D3 get dated amendments scoped to this lab; packages/ is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiCsRqCenZsFQRpJBHi * FIX-2335: workingFolder stays private to host.mts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiGfGi2RaecD1vUwy8K * FIX-2335: test the working folder check; drop stale header lines and unused import Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiGfGi2RaecD1vUwy8K * FIX-2335: keep the approval card for any worker's waitForUser approval The project tools were the only thing asking for approval, but waitForUser is a control tool every agent worker has. Without the entry and answerApproval a worker that asked would stop with nothing on the page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiGfGi2RaecD1vUwy8K * Claude Code-style lab: regenerate the README inventory main left stale #4183 (FIX-2288) changed the lab's files without rewriting the table, so the lab's inventory test is red on main and on every PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiGfGi2RaecD1vUwy8K * Port #4204: bound the next host's shutdown drain in the call crash tests Red on main since #4078: both hook-timeout failures in worker-calls.test.ts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiGfGi2RaecD1vUwy8K * Teammates lab: regenerate the README inventory after the merge Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NczDiGfGi2RaecD1vUwy8K --------- Co-authored-by: Claude <noreply@anthropic.com>


Requested by Jake · project thread
Implements specs/issues/FIX-2269 (spec #4058, merged). Blocks the FIX-2242 closure of the FIX-2070 epic.
The problem
A worker's call (
sendMessage({ wait: true })) sends its message and then parks on the reply. If the process died after the send and before the gate was written, the request was leftinterruptedwith no gate. Nothing continued it, so the call never answered the request that made it.What changes
RequestWaitsgets an optionalowingPark(work)verb. Its JSDoc says the work it wraps must be safe to repeat.durability/park-owed.ts):parkOwed:<requestId>, on the request's session. It is due one stale threshold out.runAction), when the wrapped work returns or throws (except aSuspensionError), and when the sweep finds the request parked or ended.interruptedrequest that still owes a park, under its lease and with no resume context, the same continuation the continue route uses. It moves the wake on while the request isin_progress, and gives up 10 minutes after the first owe, leaving the requestinterrupted.parkedInterruptedIdsanddrivenAgainnow read the wake store, so history, retention andwaits.endedtreat an owed request as one that will run again.callWorkerwraps its send andawaitReplyinowingPark.dispatcher({ wait: true }). After the Codex P1 finding it no longer owes a park.runOncerecords only aftersend()returns, and the dispatch seam takes no stable request id, so a continuation could start a second child. The spec had included the dispatcher, so the scope change is Jake's call (see DECISIONS.md).durable-execution.md, one exception sentence inbackground-work.md, and the wait-gate header.Verification
packages/engine/test/park-owed.test.ts, 10 tests:already-resolved.in_progressis not continued.packages/workforce/test/worker-calls.test.ts, "a call whose process dies after its row, before it parks":running.🤖 Generated with Claude Code
https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f