Skip to content

spec(FIX-2269): a wait killed before it parks still answers - #4058

Merged
jhoffner merged 3 commits into
mainfrom
claude/project-thread-d2ex6z
Oct 28, 2026
Merged

jhoffner merged 3 commits into
mainfrom
claude/project-thread-d2ex6z

Conversation

@jhoffner

Copy link
Copy Markdown
Collaborator

Requested by Jake · project thread

Someone who… Today After
asks a worker a question that needs a call, and the server dies between the call's row and its park The message lands once. The asking request stays interrupted. The reply is lost The engine continues the request. The call reads answered with the same call id
uses dispatcher({ wait: true }), and the server dies between the dispatch and the park The parent stays interrupted The parent continues and gets the child's output
operates the host Must press Continue by hand Nothing to do
has a request that dies with no wait in progress Stays interrupted Unchanged

Goal: a request that starts work it will wait on, and dies before it parks, is continued by the engine and gets the answer once.

How we'll know: an engine test (dispatcher({ wait: true }), fault after the dispatch) and a Workforce test (a call with exit-after-row) each restart on the same store and sweep, and the request completes with the answer, sent once. Control: remove the "owe a park" call, and both requests stay interrupted. A request that owes nothing is not continued.

How: a new RequestWaits verb. A wait calls it before it starts its work. The durability sweep (step 2b) continues an interrupted request that owes a park and has no gate, with the same continuation as the continue route.

Sign off

D1 · The fix is in the engine: it offers "owe a park", and the wait callers use it

This decision came down to one row: it also closes the same gap in core's dispatcher({ wait: true }), and it keeps Workforce concepts out of Layer 1. If wrong: the fix moves into Workforce, and the dispatcher gap stays open. See DECISIONS.md.

The debt: one verb, one obligation per request, one sweep branch. Nothing is removed. A caller that owes a park says its request can be replayed safely up to the park. If wrong: a careless new caller replays effects that are not under runOnce. The verb's doc tells callers this.

Reviewers · look here: D1's layer split. Also E2: a call that answers without parking clears the obligation.

Spec · Decisions · FIX-2269 · epic FIX-2070 · blocks FIX-2242. Merging is the approval.

How to review this

This is a direction set, not an implementation and not a build plan. Is this the right approach? In scope: the goal, the approach, the numbered decision, the debt. Out of scope (the implementer owns it): the build plan, names, file paths, edge cases that don't change the approach, docs wording. Feedback below the line is recorded in DECISIONS.md → Notes from review for the implementer.


Generated by Claude Code

@jhoffner jhoffner self-assigned this Oct 28, 2026
@vercel

vercel Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
flow-state-dev-docs Ignored Ignored Oct 28, 2026 5:16pm UTC
flow-state-dev-kitchen-sink Ignored Ignored Oct 28, 2026 5:16pm UTC
flow-state-dev-ui Ignored Ignored Preview Oct 28, 2026 5:16pm UTC

Request Review

@linear-code

linear-code Bot commented Oct 28, 2026

Copy link
Copy Markdown

FIX-2269

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T17:17:57.708023Z eb30af3 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copy link
Copy Markdown
Collaborator Author

Second look + architecture pass (direction check, not a build plan)

Verdict: D1 holds. Fixing this in the engine is better than a Workforce-side recovery path, and the dispatcher gap in sendAndWait (packages/core/src/blocks/dispatcher.ts) is real, so one verb covering both callers is the right size. Findings below are for the implementer, except F1 and F2, which change what the spec promises.

F1 · The "careless caller" debt is mostly the existing durable-action contract (reframe, shrinks the debt)

The spec says a caller that owes a park "says its request is safe to replay up to the park", and calls that new debt. But dispatcher({ wait: true }) and the call already require a durable action (ctx.runOnce and ctx.suspend, see the wait_unavailable guard). Resume after a park already replays the in-flight block from its start. So the block was already required to be replay-safe up to the park.

  • Say so in SPEC and in the verb's doc. The verb then adds no new promise; it only stops waiting for a human to act on one the block already made.
  • Worth checking and stating: is there any path where a wait is reachable from a non-replaying request (no runOnce)? The guard says no. If that is confirmed, "Locks in" in D1 can be dropped.

F2 · Where the replay starts is wider than the caller can vouch for

Related, and the real residual risk. The wait caller only knows its own send's runOnce keys. The surrounding block (user code before the dispatcher(...) call) is replayed too. E3 says "runs the in-flight block again", which is right, but D1's table row "Who says replay is safe: the caller, which knows its runOnce keys" overstates it. The accurate row is "the durable-action contract". Fix the wording so a reader does not think the dispatcher can certify the user's block.

F3 · E4's retry bound adds state; check whether an existing bound already covers it

E4 adds a "continued three times" counter on top of the obligation (E5's new field or row). That is a second new persisted field. Before adding it, look at whether the existing orphan/retention age-out of interrupted requests (retention.ts, step 5) plus the lease already bounds a crash loop. If a counter is needed, store it in the same record as the obligation (one nullable field holding { attempts }), not a separate one, and dual-read per BP-030.

F4 · Make "owe" and "clear" one idempotent pair, not two call sites

Today three places touch the obligation: the verb (owe), the park (clear, E2), and a no-park return (clear, E2). That is easy to leak: a call that throws between owe and either exit leaves the obligation set on a request that then dies elsewhere (a later, unrelated crash gets replayed). Prefer a shape where the caller cannot forget the clear, e.g. the verb takes the work as a callback (waits.owingPark(fn)) or returns a disposer, and the engine clears on any exit, success or throw. This is cheaper than documenting the rule for each future caller, and is the one place the spec's "documented in the verb's doc" mitigation is weakest.

F5 · Ordering across the two crash windows

There are two windows, and the spec only draws one:

  1. owe → row/dispatch written → die (covered).
  2. owe written → die before the work started (nothing was sent). The sweep continues the request, which replays and sends once. Fine, but add this case to the engine test (fault right after owe, before the send), since it is the case that proves the obligation does not need to know whether the work started.

F6 · Scope in the engine test

The control ("a request that owes nothing is not continued") is good. Add the third control the issue implies: a request that owed, parked, and then died after resume is not continued by this branch (E2 says the park clears it). Without it, a mistake in E2 would silently turn every post-wait crash into an auto-replay.

Architecture notes (opportunity only where this PR is already there)

  • Reuse is right. continueRequest, resumeUnderLease, step 2b's scan, and the FIX-2166 bring-forward are all reused. Keep the new branch inside the existing step 2b scan of interrupted requests rather than adding a fourth scan: step 2b already pages the parked set with items, so the owed-park check is one more predicate on records already loaded, not another query.
  • Storage choice (E5). Prefer a field on the request record over a durability-provider row: stores.request.list({ status: PARKED, withItems: true }) already returns it, so the sweep does zero extra reads. A provider row means a second lookup per interrupted request. Only pick the row if the request-store conformance suite (stores/testing/request-store-conformance.ts) would otherwise need a new required field on every store adapter (sqlite, postgres), which is a real cost; say which you picked and why in DECISIONS.
  • No broader refactor warranted. I looked for a consolidation opportunity in request-recovery.ts vs. the sweeper's re-drive path, and they already share continueRequest. Nothing to bundle here; leave it.

Docs

The spec lists the durability page. Also update packages/engine/README.md if RequestWaits is documented there, and the RequestWaits JSDoc in packages/core/src/types/request-host.ts (BP-007).

Not blocking merge on any of these. F1/F2 are wording fixes to the spec; F3/F4 are worth a decision note before build.


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/simplify review (spec-only PR)

Verdict: The chosen direction — engine RequestWaits “owe a park”, callers opt in before send/dispatch, step 2b continues interrupted + owes + no gate via existing continueRequest — is the right-sized fix. Rejected shapes (session wake, engine reading Workforce rows, gate-before-bind-target, continue-all-interrupted) would either duplicate recovery or guess replay safety. This is not an over-engineered architecture; it is a small obligation + one sweep branch on machinery you already have (FIX-811, FIX-1816/2b, FIX-2166, runOnce).

Where complexity shows up: mostly documentation surface, not the design. SPEC and DECISIONS repeat the same rejects, bring-forward timing, obligation lifecycle, and D1 in Sign off. Trimming ~30% by enforcing “SPEC = personas + goal + what changes + verify; DECISIONS = D1 table + considered/dropped” would make the spec easier to gate without changing direction.

Implementation PR (when this lands): keep the diff small with these reuse/simplify hooks (not blockers for this spec):

  1. One core helper for “owe → runOnce send → waitOnRequest” so dispatcher.sendAndWait and Workforce call.ts cannot drift on the owe call site.
  2. Clear obligation in the engine on successful parkOnWait (and on request end per E2); callers should not each clear.
  3. FIX-2166-style bring-forward on durability, not only stale interrupt: today wake bring-forward is wired through noteWakeOwed / sweeper arm(); owed-park needs the same durability tick hook or recovery tail becomes stale threshold plus sweep interval.
  4. Retention / parkedInterruptedIds: interrupted + owes park should behave like “will run again” (same family as parked-for-redrive), or history/retention may treat a recoverable turn as dead — worth one line in SPEC “What changes” or debt table.
  5. E4 (max 3 continues): either add a verify bullet/control or soften to “bounded cap (implementer, logged)” like E5 storage — right now verification only pins owe vs not-owe.
  6. E2 “answers without parking clears”: the subtlest invariant; one explicit lifecycle sentence in SPEC would reduce mis-implementation.

Performance (operational, not a reason to reject): every blocking call/dispatch pays one durable owe write; mass crash can fan out many async continueRequest replays in one 2b tick (E4 caps per request, not cluster-wide). Acceptable trade for closing FIX-2070 leg c / FIX-2242; optional implementer note on concurrency limits if fleet-restart SLOs matter.

Skipped (would need owner call): whether host/DevTool row stays a first-class persona in the table; whether Sign off “If wrong: Workforce-only” is required by your spec template.

Approving direction for merge; inline notes are doc tightening and implementer guardrails.

Open in Web View Automation 

Sent by Cursor Automation: Review For Best Approach

Comment thread specs/issues/FIX-2269/SPEC.md Resolved
Comment thread specs/issues/FIX-2269/SPEC.md Resolved
Comment thread specs/issues/FIX-2269/SPEC.md Resolved
Comment thread specs/issues/FIX-2269/DECISIONS.md Resolved
Comment thread specs/issues/FIX-2269/DECISIONS.md Resolved
Comment thread specs/issues/FIX-2269/SPEC.md Resolved
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NczDiEuGX4bfXGJoNxGQ6f

Copy link
Copy Markdown
Collaborator Author

Second-look pass folded in eb30af3:

  • F1 and F2: the debt is now stated as the durable-action contract a wait already needs. D1's table names that contract, not the caller.
  • F4: E2 keeps owe and clear as one wrapped pair. The engine clears the obligation on every exit.
  • F3: E4 keeps the count with the obligation, or drops it if retention already bounds a crash loop.
  • F5 and F6: both cases are added as controls.
  • E5: a request-record field is named as the lean choice.

The scan-placement and docs notes are recorded for the implementer.


Generated by Claude Code

@jhoffner
jhoffner merged commit 155f5b5 into main Oct 28, 2026
18 checks passed
@jhoffner
jhoffner deleted the claude/project-thread-d2ex6z branch October 28, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants