Repository navigation
fix(engine): fence cancels on the request's incarnation (FIX-1654) - #2430
Conversation
The abort route fenced its conditional write on the checked record's createdAt, which is wrong both ways: a same-owner hand-off rewrites createdAt (the owner's cancel 404s and the run keeps going), and two requests under one id can share a millisecond (another tenant's cancel lands). setFieldsIfStatus now takes expectedIncarnation, compared with the stored record's resolved incarnation inside each store's atomic step; resolveRequestIncarnation is exported for out-of-tree stores. The route's in-process fire is fenced the same way: registered abort controllers carry their request's incarnation, runAction registers with it, and the host tags its queued pre-registration once the record is claimed. Callers with no record to check (CLI, cascading child abort) still act on the id alone. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
|
Standing down on Process guards: it fails on every PR right now because of Generated by Claude Code |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Second look + architecture passVerdict: the approach is right; no blocking findings. I read the full source diff (not the tests line by line) against Second look: does it earn its scope?
Non-blocking notes (implementer's call)
improve-codebase-architectureThis PR is a legitimate opportunity to deepen one thing, and it already does: request identity now has one definition ( I would not widen further in this PR:
No BP conflicts found. BP-030 (tolerate the old shape) is honoured by the Generated by Claude Code |
There was a problem hiding this comment.
Code Snob — already tight, on 283fb2aac.
I read packages/engine/src/execution/abort-registry.ts, packages/engine/src/routes/abort-routes.ts, the claim block in packages/engine/src/execution/runAction.ts, materializeOwned and the tag site in packages/engine/src/transports/host/createInboundTransportHost.ts, resolveRequestIncarnation in packages/engine/src/stores/scope-keys.ts, the four setFieldsIfStatus adapters (memory, filesystem, sqlite, postgres), and the new route, conformance, and integration tests.
A rewrite would not shrink the contract. The fifth-argument swap stays inside each store's existing atomic step: memory and filesystem already call resolveRequestIncarnation, and sqlite's one-liner plus the postgres COALESCE are the package and SQL boundaries this PR recorded, not a helper waiting to be extracted. tagAbortController is the queued window — the controller has to exist before the record is claimed, and the incarnation is only known after — so registering later turns that cancel from a 204 into a 202. matches is the one predicate shared by the fire and the probe. Collapsing the route's hasActiveAbortController + abortRequest pair, or the runAction claim-or-adopt binding, saves a local line and leaves the same four compares and the same tag.
No POC.
Sent by Cursor Automation: Code Snob
There was a problem hiding this comment.
Simplify review (FIX-1654)
Verdict: The direction is right and I’m approving. Swapping the setFieldsIfStatus fence from createdAt to resolveRequestIncarnation(record) is the fix the types already called out as the follow-up; createdAt was never request identity under same-owner hand-off or same-millisecond reuse. Threading that same incarnation through the abort route and the in-process registry is necessary, not extra scope — the new route tests show a store write can succeed while a later controller under the same id would be the wrong in-process target without registry fencing.
Did this go too far?
Mostly no. The surface area looks large because tests and conformance correctly spell out legacy legacy_<createdAt> behavior and the hand-off case, not because production logic sprawled.
The one real complexity add is tagAbortController: queued dispatch registers a controller before materializeOwned / claimRequestRecord returns an incarnation (see host comment ~900–908). Fenced abort must not fire an untagged controller, so tagging after claim is coherent. Alternatives I considered:
- Store-only fix (no registry changes) — insufficient; in-process 204 path can abort the wrong run while the durable intent landed on the checked record.
- Defer
registerAbortControlleruntil incarnation is known — would reopen the window where the id is discoverable in the store but nothing is abortable locally; the host explicitly registers early to close that. - Keep fencing on
createdAtand patch edge cases — fights the existing incarnation model (isSameRequest, workspace scope identity); incarnation is the right key.
So tagAbortController is a reasonable tradeoff. Optional tighten: it’s only needed by the transport host today — if you want a smaller public API, keep it on the execution submodule export for tests/host and drop it from the changeset’s “callers should know” list unless you intend third-party hosts to tag too.
Low-cost simplifications (non-blocking)
| Area | Suggestion |
|---|---|
abort-routes.ts |
After a successful fenced write, if (abortRequest(requestId, incarnation)) → 204 else 202 avoids duplicate map lookups and duplicates matches() logic with hasActiveAbortController. |
runAction.ts |
Collapse admit vs claim into one const runIncarnation = resolveRequestIncarnation(admittedRequest ?? claimed) to drop let + duplicated resolve. |
store-sqlite |
Inline legacy_${createdAt} duplicates resolveRequestIncarnation; postgres already pulls runtime helpers from engine — importing the helper (or a tiny shared fence helper in stores/shared.ts for memory/filesystem) reduces drift risk. |
| Tests | abortWithInterleave / abortThroughRouter are the same harness in three places; a shared test helper would shrink the diff without weakening coverage. Integration hand-off leg (a) could use claimRequestRecord like engine tests instead of manual set + createdAt bump. |
| Conformance | stamped() overlaps makeRecord; the as string cast on nullable incarnation is a small smell — optional fold into makeRecord overrides. |
Performance / structure
No concerns on hot paths: heartbeat poll stays unfenced by id; fence work is abort-scale. Directory boundaries stay clean (route → store helper → adapters); no new cross-package smells beyond the existing sqlite “restate engine rule in SQL” pattern.
Skipped (larger than this PR)
- Indexed
incarnationcolumn for SQL stores. - Merging the two changeset fragments (fine as-is).
Nice work closing the loop on a fence that was documented as wrong before this landed.
Sent by Cursor Automation: Review For Best Approach
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 283fb2aac0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… claimed A cancel can land once the enqueue-time record is claimed, while the host is still materializing the rest of the request. Tagging only after materialization left the controller unfenced for that window, so the route answered 202 and, with heartbeats off, the cancelled request ran. materializeOwned now reports the claimed incarnation through a callback right after the claim. Review nits: the route branches 204/202 on abortRequest's return value; runAction resolves its incarnation in one const; tagAbortController stays off the public barrel and the changeset. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dd940f9b9a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…opts runAction tagged its controller from admission's read, but the execution context reads the record again and runs as whatever request holds the id then. When another request took the id in between, a cancel fenced on the adopted request missed this run's controller, and with heartbeats off the run kept going. The controller is now re-tagged with the adopted incarnation, and a cancel recorded while the tag was stale is delivered from one read of the stored intent. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c323625a75
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2b152c2520
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…tes as The catch-up read after a re-tag, the heartbeat intent poll and both abort classifications read the cancel flag by id alone, so a cancel recorded on a later request that took the id could stop or reclassify this run. Each now checks the flag first (the O(1) read) and, only when it is set, reads the record and requires its incarnation to match the one the run executes as. The poll's fire is fenced on it too. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d0aaad4c65
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
With heartbeats off, a run that adopts a record already on file (a queued stub, or a replacement that took the id while the request waited at the gate) never read the stored cancel, and a fenced fire aimed at that request could have missed the earlier controller under the id. The start-of-run intent read now also runs in that case, fenced on the incarnation the run executes as. The BR-12 route test now swaps the controller and the record after the write applies, so a route that decides before the write and fires by id fails it. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3cfc7da753
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ettled The start-of-run intent read ran before the execution context adopted the record, so a cancel stored on the request admission read could abort the controller (and set the delivered latch) of a different request that took the id before the context's read. The poll now waits until the context has settled which request the run executes as, and the start read runs right after that, against that request only. Heartbeat ticks before then do no reads. Docs now say that with heartbeatIntervalMs: 0 a request recorded before its run started is still checked once at start. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9e60233b8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…equest The start read ran only under a condition that three review rounds found gaps in; the latest was a claim that takes over a record already cancelled, with heartbeats off. It now runs for every run, once the context has settled which request the run executes as: one O(1) read before any block runs. The run's signal is now its own controller, fed by the registered one only after that settle point. A fire that landed earlier, for a request that then lost the id to the one the context adopted, is dropped with its controller instead of starting the adopted request cancelled. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 73c34a4b0a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… swap Round 7 replaced the registered controller whenever it had fired before the context adopted another request. An unfenced fire (shutdown cancelling a detached child, the CLI stopping its own turn) stores nothing, so the start read cannot redeliver it, and the swap threw it away. The registry now records whether a fire was fenced, and the swap drops only a fire fenced on the earlier incarnation. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b9e0d8a82c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
An unfenced fire that landed after a fenced one recorded nothing, since the controller had already fired, so the settle-time swap still dropped it. The registry now records fenced and unfenced fires separately, the unfenced one even on an already-fired controller, and the swap happens only when every pre-settle fire was fenced. A table test covers every pre-settle fire order against adopting the same or another request. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 35f52685b9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…d gate's terminal writes (FIX-1654) How a controller was fired is now recorded on the controller itself, so a run whose controller another run displaced still knows whether a pre-settle fire was fenced on the earlier request or unfenced. A fired controller is replaced at settle only if it still holds the registry slot. The host's queued path ends only the request it claimed: a cancel for A no longer terminates B when B has taken the id before the gate opens. The run adopts B on a fresh controller and B's start read settles B's own cancel. The queue-timeout and external-dispatch failure writes are fenced the same way. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8cc7ca6110
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| expectedIncarnation !== undefined && | ||
| resolveRequestIncarnation(record) !== expectedIncarnation | ||
| ) { | ||
| return; |
There was a problem hiding this comment.
Make queued terminal writes incarnation-atomic
Fresh evidence beyond the earlier gate-handoff fix is the read/write gap inside this helper: if cleanup reads queued request A, then retention deletes A and request B takes the ID before the subsequent set(..., "any"), this incarnation check has already passed and the stale A snapshot overwrites B as aborted or failed. This affects the newly fenced cancellation, timeout, and materialization-failure paths, so perform the status transition with an atomic incarnation predicate rather than a pre-read check.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed. I'm deferring this rather than fixing it in this PR. The gap is now down to one read and one write inside terminateUnenqueuedRequest: the incarnation check is done on the read, and the terminal write follows it. Closing it properly needs a conditional write that can set a terminal status under an incarnation predicate. That has to be added to the store contract and implemented in all four stores, because today's conditional write (setFieldsIfStatus) deliberately excludes status. It's tracked in FIX-1665. I'm leaving this thread open until then.
Generated by Claude Code
…IX-1654) The host's queued path now hands runAction the controller it registered while the request waited, with the incarnation it was tagged with, instead of forwarding its raw signal. The run registers that controller as its own and applies the same rule as at settle: a controller fired only by fenced fires for another incarnation is dropped for a fresh one. A cancel of the request the gate handed over no longer stops a later request that took the id before the run adopted it. Documents the registry/run-local/side-chain controller topology and the settle-gated forwarding in docs/architecture/execution-and-errors.md. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G


The problem
Cancelling a request first checks that the caller owns it, then records the cancel only on that same request. Until now, "that same request" was identified by the record's creation time, and that gets it wrong in two ways:
The POC merged with the spec reproduced both on
main.What this does
The cancel now names the request by its incarnation. That is the per-request identity every other request-scoped part of the engine already uses.
setFieldsIfStatus's fifth parameter is nowexpectedIncarnation. Each of the four stores compares it inside its existing atomic step. A record from before incarnations existed answers tolegacy_<createdAt>. Postgres needs no schema change.docs/architecture/execution-and-errors.mdnow describes three controllers: the registry controller, the run-local controller, and the side-chain controller. It also describes the settle step that gates forwarding between them.Fixes FIX-1654 · spec:
specs/issues/FIX-1654/, reviewed in #2415 · epic FIX-1635What's asked of you
Nothing to decide. The spec was approved and merged, and this builds it as written. Two items are deferred (below), and there is one coordination point with #2393.
Deferred
setFieldsIfStatusdeliberately excludesstatus. The review thread stays open for it.finallyremoves the in-memory abort controller by id alone. If another run in the same process registers under that id at that moment, its controller is removed. No request status is written.Parts worth reviewing closely
setFieldsIfStatus, and they don't agree on its shape:expectedIncarnationargument and keepsexpectedCreatedAtas the fifth.stores/types.ts, the four store adapters and the conformance file. Whichever PR merges second needs to:(…, now, undefined, incarnation)call sites inrunActionand retention to(…, now, incarnation);legacy_rule.runAction, and the handoff (abortHandoff). The pre-settle table inabort.test.tscrosses:createInboundTransportHost.ts). A cancelled request whose id another request has since taken is not terminated. The run adopts the new request on an unfired controller.COALESCE(data->>'incarnation', 'legacy_' || (data->>'createdAt'))under the existingFOR UPDATElock.How to review this
Evidence
Red on
main's code (the tests of this PR, before the fix):expected 202 to be 404(same-ms other tenant),expected 404 to be 202(owner hand-off),expected true to be false(a later request's controller fired).main's adapter: 5 failed, 34 passed. Postgres (PGlite): 7 failed, 32 passed.expected 404 to be 204). Leg (b) passes on both, as the spec predicted.expected true to be false);expected 'aborted' not to be 'aborted');expected [ 'started cancelled' ] to deeply equal [ 'started live' ]).Green on this branch (head 86d04f2):
packages/engine: 3119 passed, 6 skipped, 1 failed. The failure is the known filesystem session-stream BR-23 test, which times out under full-suite load.abort.test.tsalone: 98/98.packages/integration-tests: 39 files, 141 passed.packages/store-sqlite328 passed;packages/store-postgrespassed exceptsession-stream-indexes.test.ts, which timed out under full-suite load and passes alone;packages/cli278 passed.pnpm typecheckandvalidate-package-boundaries.mjswere clean.Docs. The engine README and
docs/architecture/state-and-scopes.mdapply the drafts approved in the spec's DOCS.md.docs/architecture/execution-and-errors.mddescribes the cancellation topology. Theapps/docspages say that each run checks for a stored cancel once as it starts.Changesets.
engineminor.store-sqliteandstore-postgrespatch.🤖 Generated with Claude Code
https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G