Skip to content

fix(engine): abort reaches every live run of a request (FIX-1665, part 1 of 3) - #2971

Merged
jhoffner merged 3 commits into
mainfrom
fix/FIX-1665-attempt-fence
Oct 9, 2026
Merged

jhoffner merged 3 commits into
mainfrom
fix/FIX-1665-attempt-fence

Conversation

@jhoffner

@jhoffner jhoffner commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

One request can have two runs live at once: the stale sweep marks a slow but live run interrupted, and /continue starts a second run under the same id. The abort registry kept one controller per request id, so the second run replaced the first's controller, and whichever run ended first removed the survivor's. After that, aborting the request did nothing to the run still going.

What this does

  • The abort registry now keeps one controller per live run under an id. An abort fires all of them, and a run that ends removes only its own.
  • deregisterAbortController(requestId, controller?) gains an optional controller. The id-only form still removes everything, so existing callers keep working.
  • When runAction declines a handed-over queued controller, it removes that controller instead of leaving it next to its own.

What's asked of you

Review the registry change. This PR covers one of the three holes in FIX-1665. The other two need a decision before they can be built, and that decision has been escalated to the coordinator rather than made here:

  • Cross-process finalize fence. Fixing it means changing the RequestStore and ActiveRequestRegistry contracts that custom stores implement. A repro is included, marked it.fails.
  • Filesystem sidecar namespace and migration. This changes the on-disk layout.

Parts worth reviewing closely

  • replaceAbortController now swaps the controller in place within the run's own entry. It no longer competes for a single slot.
  • The host's queued path: the queued controller is either kept by runAction (same object, deregistered by both, idempotently) or removed when runAction starts on a fresh one.
  • Two existing tests in abort.test.ts assumed that registering replaced the earlier controller. They now model a finished request the way the code does, by deregistering it.
Evidence

Red, on origin/main with only the new test file: the 3 new tests in test/request-finalization-overlap.test.ts fail (they reuse its parked-run harness). The in-process abort tests fail with expected false to be true because the survivor's controller is gone. The cross-process test fails with finalizedAtMs: <number>, registered: false while the other run is still in onFinished.

Green: the two in-process tests pass. The cross-process test stays red and is held as it.fails.

  • pnpm --filter @flow-state-dev/engine typecheck: clean
  • pnpm --filter @flow-state-dev/engine test: 273 files, 3624 passed
  • pnpm --filter @flow-state-dev/integration-tests test: 41 files, 153 passed
  • node scripts/validate-changeset-refs.mjs and node scripts/check-mailbox-rename.mjs: pass

Second-path checklist: no persisted field changes, so there is no legacy read path. The public API stays backward compatible. On the concurrent path, two runs register, one ends, and abort still reaches the other (tested). On the cancel and setup-error paths, each deregister passes the run's own controller. Multi-tenant fencing is unchanged, because it still goes through the incarnation match.

Changeset: abort-every-live-attempt.md (engine, patch).

Links: FIX-1665

🤖 Generated with Claude Code

https://claude.ai/code/session_01DME8pzA4e4Rc7fRLGsYavW


Generated by Claude Code


Note

Medium Risk
Changes in-process cancellation registry and all run teardown paths; behavior is well covered by new overlap tests but affects concurrent continuation/stale-sweep scenarios.

Overview
Fixes FIX-1665 where overlapping runs under one request id could not be aborted reliably: a second run used to replace the first run’s AbortController, and the first run to finish cleared the registry for everyone.

The in-process abort registry now stores one controller per live run attempt (array per requestId). abortRequest aborts all matching attempts (still respects incarnation fencing). deregisterAbortController(requestId, controller?) removes only that attempt’s controller; the id-only form still bulk-clears for tests.

runAction and the inbound host’s queued pre-start path pass the specific controller on every cleanup, and drop a rejected abort handoff so it does not linger as a phantom second attempt.

Tests cover multi-attempt abort/deregister and overlapping continuation scenarios; cross-process finalize overlap remains it.fails (documented follow-up).

Reviewed by Cursor Bugbot for commit 546ad31. Configure here.

…g keeps the other's controller (FIX-1665)

The abort registry held one controller per request id, so when the stale
sweep and /continue left two runs of one request live in a process, the
second registration replaced the first, and whichever run ended first
deregistered the survivor's controller. The survivor could then no longer
be aborted.

The registry now holds one controller per live run under the id: an abort
fires all of them, and a run deregisters only its own controller.
`deregisterAbortController` keeps its id-only form (removes all).

Also adds a repro of the cross-process half of FIX-1665 (a run in another
process finalizes and deregisters the id while this one still writes),
marked it.fails: fencing it needs a store-contract decision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DME8pzA4e4Rc7fRLGsYavW
@linear-code

linear-code Bot commented Oct 9, 2026

Copy link
Copy Markdown

FIX-1665

@vercel

vercel Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
flow-state-dev-docs Ignored Ignored Oct 9, 2026 8:42pm UTC
flow-state-dev-kitchen-sink Ignored Ignored Oct 9, 2026 8:42pm UTC
flow-state-dev-ui Ignored Ignored Preview Oct 9, 2026 8:42pm UTC

Request Review

@cursor

cursor Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: cd20964f-ad54-4844-8151-b9ae1a3e376c)

@jhoffner
jhoffner marked this pull request as ready for review October 9, 2026 19:44
@cursor

cursor Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 86ef412c-dc5c-4e1c-84f0-86a2585c4d43)

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplify / architecture read

Problem (plain terms): After stale sweep + /continue, two run attempts can share one request id in the same process. The old abort registry was last-writer-wins: one attempt finishing called deregisterAbortController(requestId) and removed the slot, so abort could miss the sibling still running.

Did this go too far? For the in-process slice (part 1 of 3), no. Map<string, Registered[]> plus deregisterAbortController(requestId, controller?) is the smallest API-stable fix: abort fans out to every live controller under the id; teardown removes only this attempt’s controller. Composite keys (requestId:attemptId) would still need a secondary index or fan-out for unfenced “abort this id” callers. Store-backed live-attempt counting is the right direction for cross-process finalize (your it.fails test says that explicitly) and is correctly out of scope here—not a simpler substitute for this bug.

What looks right

  • abortRequest filtering + per-controller firedBy preserves incarnation fencing when multiple slots share an id.
  • runAction passing registered on every exit path and deregistering a rejected abortHandoff controller avoids phantom second attempts.
  • Transport deregisterAbortController(requestId, queuedAbort) matches the new semantics.
  • Unit tests in abort.test.ts + integration in request-attempt-overlap.test.ts encode the regression; tests pass locally.

Simplify follow-ups (non-blocking)

  1. Test harness duplication: makeParkedFlow in request-attempt-overlap.test.ts is almost the same shape as overlap() in request-finalization-overlap.test.ts (gates, parked, onFinished hold when finishing === 2). Worth a shared packages/engine/test/fixtures/overlapping-runs.ts (or exporting one builder from the finalization file) with options like captureSignals and kind—would shrink this PR and keep overlap scenarios from drifting.
  2. Docs at the top of abort-registry.ts: Lines 3–6 still read like one controller per request until terminal state; the new paragraph at 14–18 is the accurate model—I'd align the opening blurb so readers don't infer single-slot semantics.
  3. deregisterAbortController(requestId) without controller: Still deletes all attempts under the id (fine for tests/afterEach). A one-line JSDoc nudge (“tests / bulk clear only”) would reduce the risk of a future production call during overlap.
  4. Imports in the new test file: Mixed ../src vs ../src/execution/abort-registry is a minor consistency nit; barrel-only is fine if those symbols are re-exported.
  5. Micro: deregister could splice in-place on the existing array instead of filter + set—irrelevant at k≈2, optional.

Skipped (larger than this diff)

  • Requiring controller on every deregister (breaking / follow-up).
  • Consolidating overlap + abort + finalization under store-level live-attempt records (parts 2–3).

Verdict: Direction is sound; complexity matches the failure mode. Approving—optional cleanups above are polish, not blockers.

Open in Web View Automation 

Sent by Cursor Automation: Review For Best Approach

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Snob

request-attempt-overlap.test.ts copies the parked-run harness already in request-finalization-overlap.test.ts (two gates, the second onFinished hold, interrupt, then /continue). Fold the abort checks onto that harness and delete the new file. About 100 lines come out. The assertions stay: one run ending leaves the other's controller, one abort reaches both live runs, and the cross-process case stays it.fails.

The registry is already the small shape. A nested map in place of the list does not get shorter, and the id-only deregister is the compatibility this PR kept.

POC, does not merge: #2982

Fold that subtraction into this PR. Close the POC unmerged after this lands.

View PR

Open in Web View Automation 

Sent by Cursor Automation: Code Snob

… harness; align abort-registry docs (FIX-1665)

The abort checks and the cross-process repro (still it.fails) now reuse the
parked-run harness in request-finalization-overlap.test.ts, and the duplicate
request-attempt-overlap.test.ts is gone. The abort-registry file header now
describes one controller per run attempt, and the id-only
deregisterAbortController form is documented as a bulk clear.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DME8pzA4e4Rc7fRLGsYavW
@cursor

cursor Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 689856eb-e90d-43fe-88ba-5df4f826ed0a)

@jhoffner

jhoffner commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

@claude CI failures

jhoffner commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

On it. The one real failure is Test (2/3), and Typecheck & Test is red only because it rolls that job up. The failing test is packages/workforce/test/coordinator-rounds.test.ts:602 ("closes an overdue round on the conversation's next wake…"). It expected the coder to receive the EM's answer after a cancelled delegate, and it got nothing.

This PR's source changed only in comments since the last green head (920adc3). Even so, that test cancels through the abort path this PR changes, so I'm not calling it a flake. The test is being run 20+ times on this branch and on main:

  • Fails only here: I'll fix the abort path, with a test that shows the failure, and push.
  • Fails on main too: I'll report the race and a proposed fix.

Generated by Claude Code

Brings in FIX-1840 (#2983): the coordinator conversation's CAS retry
budget, which fixes the coordinator-rounds.test.ts:602 failure that
turned Test (2/3) red on this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMg2dz8sMEdyzCQTRsaMTV
@cursor

cursor Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: aa93e2d6-8ea2-427c-a4c6-393d986a6922)

jhoffner commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up on the CI failures. Merged origin/main (705ba5d) into the branch with a merge commit; head is now 546ad31. None of the three red checks came from this PR.

Test (2/3) and Typecheck & Test. The failing assertion was at packages/workforce/test/coordinator-rounds.test.ts:602. It is main's bug FIX-1840, fixed in #2983, which merged after this branch was cut. A coordinator conversation's pass-on delivery lost its session write more than the default 3 CAS retries allowed. The request then failed after its round had closed, so the EM's answer never reached the coder. #2983 raises the coordinator session to cas.maxRetries: 8 and adds a test that forces exactly that write to lose four times in a row.

Changesets name their Linear issue. The fragment was already correct, and the latest run on 6dff11e passed. The red entries are cancelled runs: cancel-in-progress cancels a run when the push and the title edit land seconds apart. Run locally with main's guard, this branch and title pass. A control with a title naming another issue fails, which shows the guard does read the title. I left the fragment unchanged.

Also checked on the merged head: engine tests pass (275 files, 3649 passed, 6 skipped, with the cross-process it.fails held), and turbo typecheck --concurrency=2 passes 74/74.


Generated by Claude Code

jhoffner commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

Test (2/3) failed at packages/workforce/test/coordinator-rounds.test.ts:602. The failure isn't this PR's:

Root cause: the coordinator's internal routeOn action isn't queued, unlike its neighbours DELEGATE_ANSWER_ACTION and DELEGATE_MISSED_ACTION. It races the next post's writes on the same session state and fails with concurrent_modification after the round has already left openRounds. The delegate's hand-off is then lost. The abort path isn't involved: the failing request is never cancelled.

Fix: none exists yet. The likely fix is concurrency: "queue" on ROUTE_ON_ACTION in packages/workforce/src/coordinator/coordinator-flow.ts. It's outside this PR's scope and is proposed to the owner as its own bug. Re-running the failed job once.


Generated by Claude Code

@jhoffner

jhoffner commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

1 similar comment
@jhoffner

jhoffner commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-09T23:48:55.422400Z 546ad31 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@jhoffner
jhoffner merged commit 6b65ca5 into main Oct 9, 2026
19 of 20 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 546ad31e2c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +35 to +36
/** Controllers of the live attempts of each request id, in registration order. */
const controllers = new Map<string, Registered[]>();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Update the abort lifecycle contract for concurrent attempts

When a stale sweep and /continue leave two runs live—the exact scenario this patch supports—the authoritative architecture document still says each request has three controllers and diagrams a single registry controller (docs/architecture/execution-and-errors.md:186-220). That is now false because the registry holds one controller per attempt and abort fans out across them; leaving the contract unchanged risks subsequent execution work restoring the old replacement or bulk-cleanup semantics. Update the architecture documentation alongside this core execution behavior change.

AGENTS.md reference: AGENTS.md:L203-L208

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants