Repository navigation
fix(stores): delete a request's stream events with its record (FIX-1647) - #2393
Conversation
…ecord (FIX-1647) Session retention deleted a request's record but left its stream events (and runOnce results) in the memory, SQLite and Postgres stores. Request ids are caller-supplied, so a later request, from another user, could take the freed id and replay the previous run's events from sequence 0. Reproduced end to end on main through the HTTP action route, real retention and the HTTP stream route on all three stores. The filesystem store already swept these sidecars and passed as the control. RequestStore.delete now removes events and runOnce results with the record. SQLite and Postgres drop a not-yet-written event batch first and Postgres waits out an in-flight write, so a queued write cannot restore rows after the delete. Tests: a store-level delete block in createRequestStoreConformanceTests (memory, filesystem, SQLite, Postgres), and a new createRequestRetentionConformanceTests that drives the real path, run for memory, SQLite and Postgres (PGlite). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
CI note: the "Process guards" check is red on every open PR right now because Generated by Claude Code |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Second-look + architecture review (read-only, no edits made)Verdict: the approach is right. Deleting at the source is the leanest fix. It matches the filesystem store, which was already correct, and needs no schema change. The findings below are about hardening. None of them argue for a different shape. 1. Postgres
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0f10e2cdc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Code Snob — too small to rewrite. No POC.
I read packages/engine/src/stores/memory/request-store.ts, packages/store-sqlite/src/request-store.ts, packages/store-postgres/src/request-store.ts, packages/engine/src/stores/filesystem/request-store.ts (delete / deleteSidecars), packages/engine/src/stores/testing/request-store-conformance.ts, packages/engine/src/stores/testing/request-retention-conformance.ts, the three request-retention-conformance.test.ts files, packages/engine/src/testing.ts, the retention section of packages/engine/README.md, .changeset/request-delete-removes-events.md, and the sqlite schema note that request_events / request_runonce have no foreign key and are pruned on the request-lifecycle path (packages/store-sqlite/src/schema.ts).
A rewrite would not shrink this contract. Each store already deletes items in its own mechanism (a map, a prepared statement, a query). Events and runOnce are those same two operations beside it. One helper cannot issue all three. SQLite drops a queued map; Postgres also awaits an in-flight write. Each is a single call site. The store suite adds three claims to the existing conformance function. The HTTP harness is the recorded real-path repro: two named failures (rows left behind, and a replay of a shorter second run) plus three thin package entries. pgliteExecutor matches the other Postgres tests. The README paragraph is the adapter contract for delete. The repeated id-reuse comments sit on the non-obvious deletes; removing them drops explanation, not behavior.
Head e0f10e2cdca4b2f9a8ac84748cc7f41fe7008d36.
Sent by Cursor Automation: Code Snob
There was a problem hiding this comment.
Summary
The bug fix itself is the right shape: extend RequestStore.delete in memory, SQLite, and Postgres so stream events and runOnce results go away with the record, mirroring filesystem deleteSidecars and the existing request_items delete + pending-queue cancellation pattern. Retention already calls stores.request.delete(id) only, so this seam is correct; FK ON DELETE CASCADE would not replace the in-process pendingNewEvents / pendingEventWrites guards and would fight the schema’s intentional lack of FKs on these tables.
I would not chase a different production architecture here. The diff stays localized and consistent across adapters.
Where complexity crept in (tests / docs)
Most of the line count is conformance, not behavior:
-
Retention HTTP suite vs store delete tests — The new
createRequestStoreConformanceTestsdelete block (including the queued-persistEventsrace) is the direct guard for FIX-1647.createRequestRetentionConformanceTestsre-proves the same invariant through retention → HTTP stream replay. That is valuable as a regression story (especially Alicecount: 5vs Bobcount: 1so upsert-by-sequence cannot hide leftovers), but it is optional if you want a smaller PR: store-level delete conformance + a one-line README contract might be enough, since retention is a thin wrapper arounddelete. I would keep the E2E suite if CI time is acceptable; I would drop it only if you are optimizing for minimal surface area and trust the store contract. -
Duplicate scenario runs — The two
itblocks each callrunReusedIdScenarioend-to-end (~2× HTTP work per backend). Merging into one test with both assertions is an easy win (see inline). -
Repeated rationale comments — The caller-supplied-id / replay paragraph appears in memory, SQLite, Postgres, README, and conformance headers. The queue/microtask notes in SQL adapters are worth keeping; the rest could lean on README + conformance once.
-
SQLite test harness —
session-stream-conformance.test.tsalready usesafterAll+ tracked registries instead of a typedcleanupcast; aligning would be a small consistency cleanup (optional).
Performance / reuse
No hot-path concerns: delete remains eviction-scale. Postgres adds two parallel DELETEs in the existing Promise.all; SQLite adds two prepared statements. No third-party shortcut worth introducing.
Optional follow-up (not blocking)
Filesystem delete sweeps sidecars under lock but does not call pendingNewEvents.delete(id) like SQL adapters now do; conformance passes today, but aligning filesystem with the SQL race pattern would make cross-adapter semantics uniform by construction.
Verdict: Approve. Production change is minimal and matches established patterns; test/doc nits above are polish, not blockers.
Sent by Cursor Automation: Review For Best Approach
…n and disposes its router (FIX-1647) Postgres delete ran the child deletes and the record delete in parallel as independent autocommit queries. A failed child delete could leave events with no record, and an id claimed in the gap could have its new rows removed by the leftover cleanup. Children now go first and the record last. SQLite already deleted the record last. The retention harness returned while the reused run's tail (retention, leaving the active registry) was still writing, and never disposed its router, whose stale sweeper outlived the case. That was the filesystem teardown ENOTEMPTY. The harness now waits for the run to leave the active registry, disables the sweeper and disposes the router, and filesystem is back in it. The two cases are one scenario run; SQLite closes registries after the file. The stream route's record-less replay tests are reframed: a deleted request no longer leaves events, and a new case pins that its id 404s. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
Replies to the optionals in Cursor review 5358803060 that have no inline thread:
Generated by Claude Code |
|
@codex review The head is now c85197a:
Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c85197aea7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ete waits out event appends (FIX-1647) Postgres: Promise.all rejected on the first failed child DELETE while its siblings were still running, so a retry could free the id and a straggler then delete the next owner's rows. The child deletes now use allSettled and the first failure is thrown only once all have settled; the record is still deleted last. Filesystem: an event append takes its batch as soon as it starts, and delete swept the sidecars without waiting for it, so an in-flight append could recreate the event log under the freed id. delete now drops the unwritten batch and drains the id's event queue before the sweep. A filesystem-specific test holds the append until the sweep has listed the directory, which makes the old ordering fail every time. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 61f18b8093
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…FIX-1647) runAction marks a request completed before it persists the terminal event and runs onFinished, and leaves the active registry last. Under the default "allow" concurrency a sibling in the same session could complete in that window and its retention pass would delete the request, so the terminal event landed under the freed id for the id's next owner to replay. Retention now skips any request still in the active registry (one listAll read); the next pass evicts it. This sits above the stores, so it covers every adapter with no schema change. Docs: streaming.md no longer says events may outlive their request; RequestStore.delete documents the full contract (every child, record last, in-flight writes settled); README notes the new skip. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 904ab8f503
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…cting it (FIX-1647) Three review findings had one root: retention could delete a request while work tied to its id was still live. The active-registry skip only sees runs in a shared registry, and a live-tail stream can keep polling a deleted request for up to the liveness timeout, long enough for a new owner to take the id. Retention now spares a request until twice the live-tail liveness timeout has passed since it finished: one timeout for any stream still tailing it to end, one more to bound the run's own tail. The window is derived from LIVE_TAIL_LIVENESS_MS, now shared by the stream route and retention. The registry skip stays as a cheap extra check. A spared request is evicted when the session's next request completes; retention stays lazy, and that is documented as the bound. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Because request IDs are caller-supplied and encodeSegment does not escape periods, deleting request foo also matches a distinct request's sidecar such as foo.runonce.bar.runonce.step.json via the foo.runonce. prefix. Retention or explicit deletion of foo can therefore erase foo.runonce.bar's idempotency result, allowing its step and associated side effects to execute again; use a layout or encoding with an unambiguous request-ID boundary.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…y (FIX-1647) Deleting request "foo" matched "foo.runonce.bar.runonce.step.json" through the "foo.runonce." prefix and erased another request's idempotency result. Per-key files are now named <id>@<key>.runonce: encodeSegment always escapes @, so the first @ ends the id. Files in the older layout are still read, and removed on delete only when no longer id could own them and they are not another request's record. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
…zedAtMs (FIX-1647) runAction marks a record completed and applies retention before it awaits onFinished, which has no time limit, so no grace window bounds the run's last writes. The run now stamps finalizedAtMs as its last write, through a conditional setFieldsIfStatus that never recreates a deleted record, and retention evicts only stamped records past the grace window. New records start at null; records from before the field keep the grace-window rule. The success path heartbeats through onFinished, and the stale-request sweep stamps a terminal record whose run died before stamping it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
…into fix/fix-1647 # Conflicts: # packages/engine/src/stores/testing/request-store-conformance.ts
…(FIX-1647) After merging main, setFieldsIfStatus takes expectedCreatedAt. The run's stamp and the stale sweep's stamp now pass the createdAt of the record they read, so neither can mark another request that took the id as finished. The two-users-one-tenant retention cases shorten LIVE_TAIL_LIVENESS_MS through a harness helper so their eviction waits fit the window; what they assert is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
Re review 5359398722, the P1 on the filesystem runOnce sidecar prefix ( The bug. Deleting The fix.
Files already on disk in the old layout (BP-030).
Tests, red first.
Not fixed here: exact-name collisions.
Generated by Claude Code |
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 13d6e6a4e7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 71a2e331bc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ves the record last (FIX-1647) `<id>.runonce.json` and `<id>.events.json` are also the record files of requests `<id>.runonce` and `<id>.events`. Deleting `<id>` now leaves such a file when it reads back as that record, and the single-map runOnce fallback never serves it as `<id>`'s results. The shared filesystem record store's delete now sweeps sidecars before removing the record, and the sweep settles every removal before reporting a failure, so a failed cleanup leaves the id taken and the delete retryable. Docs: the custom-store live-tail recipe calls isStillAuthorized after every non-empty read; the side-chain page describes onFinished/onErrored work as drained before the run finishes but after the final event. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a2715190cf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…(FIX-1647) The stale sweep can mark a slow but live run interrupted, and /continue then starts a second run under the same id and incarnation. Either run could stamp finalizedAtMs, or drop the shared active-registry entry, while the other was still in onFinished or writing, so retention could free the id under a live writer. runActionInternal now counts each run as a live attempt of its request id (per request store, in this process). A run stamps and deregisters only when it is the last live attempt to end; the suspend path and the continuation's safety-net deregister follow the same rule. An overlapping run in another process is judged by its heartbeat, as the sweep already does. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 58b6c4eaae
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| * process (tests, multiple engines) never hold each other's requests open. | ||
| */ | ||
|
|
||
| const liveAttempts = new WeakMap<object, Map<string, number>>(); |
There was a problem hiding this comment.
Fence finalization across processes
In a shared-store multi-process deployment, a stale sweep can start /continue on process B while the original attempt remains alive on process A. Because this WeakMap sees attempts only in one process, each process can consider its own run the last attempt; once both have written a terminal status, the first one to finish can stamp finalizedAtMs and deregister the shared active entry while the other remains in onFinished. Retention can then delete and reuse the ID after the grace period while the remote attempt still writes. This re-raises the prior per-attempt-fence issue with fresh evidence from its process-local implementation; finalization needs a durable cross-process attempt fence or equivalent proof that no remote attempt remains.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed, this is real. It is deferred to FIX-1665 and not fixed in this PR.
Fencing runs across processes needs a durable per-attempt id on the request record and on the active-registry entry. That is a schema change in all four adapters. For now, a run in another process whose heartbeat went stale is treated as dead. That is the same rule the stale sweep already applies. The id-reuse class predates this PR, and retention is opt-in.
FIX-1665 also takes the abort-controller map, which is keyed by request id alone. I am leaving this thread open for it.
Generated by Claude Code
| if (exactJson.has(name)) { | ||
| if (!(await this.isRecordFile(name))) await rm(filePath, { force: true }); |
There was a problem hiding this comment.
Serialize suffix-collision checks with request creation
When deleting request foo, isRecordFile("foo.events.json") can read the old event sidecar as a non-record, after which a concurrent caller creates request foo.events under its distinct per-ID lock and atomically replaces that same path before rm executes; the deletion then removes the new live request's record. This is fresh evidence beyond the prior suffix-collision finding because the added content check has a TOCTOU window. Namespace sidecars so paths cannot collide, or coordinate the check and removal with creation of the colliding ID.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed, this is real. It is deferred to FIX-1665 and not fixed in this PR.
The record check and the delete are two separate filesystem steps, so a record written between them can still be deleted. The real fix is to namespace the filesystem sidecars so that a request id ending in .events or .runonce can never share a name with another request's sidecar. That needs a layout rename with a migration, and it goes together with the per-attempt id in FIX-1665.
The name collision itself predates this PR, and retention is opt-in. I am leaving this thread open for it.
Generated by Claude Code
…owing (FIX-1647) A run that ends while another run of the same request is still live leaves the stamp to it. If that last run then ended by throwing, it never reached its own finalization, so the finished record stayed unstamped and protected from retention forever. The declining run now hands over its terminal record's incarnation, and runActionInternal's exit stamps it (and deregisters) when the last attempt ends by throwing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1df50872dd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…am to the guard-checked incarnation (FIX-1647) A run that ended by throwing after writing a terminal record (a fresh setup failure writes `failed`) never reached its finalization, so the record stayed unstamped and retention kept it forever. The attempt now tracks its own terminal incarnation, and runActionInternal's exit stamps it (or one an earlier overlapping attempt left) and deregisters when it is the last live attempt. A dispatch that terminates a request no run will ever pick up stamps the record as it writes it. The request stream re-read the record after the route guard checked it, so a delete and reuse in between bound the stream to the new owner. The guard now returns the record it checked, and the stream answers as for an unknown request when the incarnation it reads differs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
…persistence docs (FIX-1647) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 35435a0059
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| items, | ||
| // The timer runs on through `onFinished` until the stamp, so a stale | ||
| // entry from here means the run died in its tail. Not with heartbeats off. | ||
| heartbeatsUntilFinalized: heartbeatIntervalMs > 0 |
There was a problem hiding this comment.
Gate finalization recovery on a safe heartbeat cadence
Fresh evidence beyond the earlier zero-heartbeat fix is that any positive interval sets this flag, although stale-request-sweeper.ts:13-14 requires staleThresholdMs >= 2 * heartbeatIntervalMs and the sweeper cannot inspect per-flow intervals. With a 120-second flow heartbeat and the default 60-second stale threshold, a request in a long onFinished is treated as dead and stamped before its next heartbeat; after the retention grace, its caller-supplied ID can be deleted and reused while the hook is still writing. Set this flag only when the effective stale threshold safely exceeds this flow's heartbeat cadence, or persist enough cadence information for recovery to verify it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed. This is deferred to FIX-1665 and not fixed here.
The run can't know the threshold it would need to check. The sweep that stamps a record may run in any process, with that host's own staleSweepThresholdMs. The run's runtimeConfig only sometimes carries its local host's threshold, and a check against that proves nothing about the sweeper's. Getting this right needs the cadence and the threshold compared where the sweep runs. One way is to persist the run's heartbeat interval on the record or registry entry and have the sweep require its threshold to be at least twice that interval. That needs new persisted fields and plumbing across hosts, which goes with the per-attempt work in FIX-1665.
On main, a cadence misconfigured against the stale threshold already misfires the sweep: live runs get marked interrupted. So this PR adds another consequence to an existing misconfiguration rather than creating a new failure mode. I'm leaving this thread open.
Generated by Claude Code
…e error; narrow side-chain drain docs (FIX-1647) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
|
@codex review Generated by Claude Code |
|
Codex Review: Something went wrong. Try again later by commenting “@codex review”. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8fd1fdcf91
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…spended run skips it (FIX-1647) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
setFieldsIfStatus takes main's single (…, updatedAt, expectedIncarnation) fence; FIX-1647's callers (finalization stamp, stale-sweep stamp, conformance cases) move onto it and the createdAt placeholder goes. One incarnation compare per store. The abort poll keeps both guards: closed after terminal status (FIX-1647) and silent until the incarnation settles (FIX-1654). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
Every final-status write now goes through main's settleRequestRecord, which writes the action result with the status. FIX-1647's terminal writes keep their fields (heartbeatsUntilFinalized) and take the written record's incarnation from it for the finalization stamp; the settle and patch helpers return that incarnation. A fresh setup failure's record and an unenqueued request's terminal write carry both the result and the stamp. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G


The problem
When a session's retention policy removes an old request, the request's record goes but its stream events stay in the store. Callers can pick their own request ids, so once an id is freed another request can take it, and that request can belong to a different user in the same tenant. Replaying that request's stream from the start then sends the previous owner's output (messages, tool calls, results) to the new owner. Until now this came only from reading the code. It now reproduces on
mainwith the memory, SQLite and Postgres stores.Before the id is reused, the freed id's leftover events can also be replayed by anyone who knows the id: the stream route's "no record" branch replays whatever events it finds and has no owner to check.
What this does
Deleting a request now deletes everything the store keeps under that id: the record, its items, its stream events and its runOnce (cached step result) records. This applies to retention and to every other delete. Retention also frees an id only after the request's run has recorded that it finished (
finalizedAtMs, written afteronFinished), and a grace window has passed since (twice the live-tail liveness timeout, 60s by default), so the run's last writes, and any stream still reading it, are done first. The filesystem store already deleted these files, but an event write that was already in progress could recreate them afterwards. It now waits for that write first. Memory, SQLite and Postgres now delete events and runOnce records too. On the filesystem store, deleting one request could also remove another's runOnce results when one id started with the other; per-key files now have an unambiguous boundary.What's asked of you
Nothing here is yours to decide. This is a bug on the direct route, so this PR is the only gate. One scope call is flagged below for the code reviewer. Three review findings are deferred to follow-up FIX-1665, and their threads are left open (see "Pre-existing, closure QA, and follow-up" under Engineering calls).
Parts worth reviewing closely
packages/store-postgres/src/request-store.ts,packages/store-sqlite/src/request-store.ts): before the DELETE,deletedrops any event batch that hasn't been written yet. Postgres also waits for a write that is already running, so a queued write can't bring the rows back afterwards. The conformance case "delete is not undone by events persisted just before it" covers this.Promise.allSettled, so a failure is reported only after every child delete has finished. A retry therefore can't free the id while an earlier attempt's straggler is still running. If a child delete fails, the record stays, so the id can't be claimed and the delete can be retried. Two tests cover this: one forces the event delete to fail and checks the record survives; the other holds the item delete and checks the failure waits for it. SQLite already deleted the record last.deletewaits for event writes (packages/engine/src/stores/filesystem/request-store.ts). An event write takes its batch as soon as it starts, sodeletenow drops the unwritten batch and drains the id's event queue before sweeping the sidecar files.filesystem-request-delete-race.test.tsholds the append until the sweep has listed the directory, which makes the old ordering fail every time.deletealso forgets the id's unreported event write error (same file, Codex P2 on 35435a0). The store holds an event write failure until the id's next flush reports it. Before, that entry outlived the delete, so the id's next owner was handed the previous owner's error by its first flush.packages/engine/src/execution/retention.ts,runAction.ts).runActionmarks a recordcompletedbefore its terminal event andonFinished, andonFinishedhas no time limit. So no grace window alone bounds the run's last writes, and a process-local registry cannot see a run in another process. Request records now carryfinalizedAtMs:null.setFieldsIfStatus. That write comes after:onFinished;.sideChain()work the hooks queued, which is drained;Promise.allSettled). If either flush fails, the run does not stamp, the same fallback as a failed stamp;setFieldsIfStatusgains an optionalexpectedIncarnationin all four adapters, with conformance cases.legacy_<createdAt>.suspended, whose run resumes;incompleteis included.2 × LIVE_TAIL_LIVENESS_MS) has passed since the stamp. Anullrecord is never evicted, however old.finalizedAtMsfield (written by a version that never stamps) may still be finishing on an old instance. It is kept until completion + max(stale-request threshold,maxAge) + grace, never on the grace window alone (BP-030).runActionstamp at their finalization points.failed, is stamped on exit fromrunActionInternal.terminateUnenqueuedRequest) writes the stamp together with the status.interrupted, and a replay setup failure's revert tosuspended/interrupted.heartbeatsUntilFinalized. It istrueonly on that path with heartbeats on; the sweep stamps only such records.request.heartbeatIntervalMs: 0, or on a failure path (which stops beating beforeonFinished), a live run looks stale. Its record is stamped only by its own run, and is kept, never evicted, if that run is gone. The same holds when the sweeper is disabled.packages/engine/src/execution/request-attempts.ts,runAction.ts, Codex P1 on a271519). Two runs of one request can overlap on purpose: the sweep marks a slow but live runinterrupted, and/continuethen starts a second run with the same id and incarnation. Each run is now counted as a live attempt of its request id, per request store and in this process. A run stamps and removes the shared registry entry only if it is the last attempt to end. The suspend path and the continuation's safety-net deregister follow the same rule.runActionInternalstill stamps under that incarnation and deregisters, so the record is not kept forever (Codex P2 on 58b6c4e).maxItems(Codex P2 on 2123d5e). A request that is still running or not yet evictable was left out of the budget, so the session could grow past the budget with nothing evicted. Its items now count, and older evictable history is evicted to make room; only its own deletion waits.server-and-client.md, the engine README and the retention row inapps/docsconfiguration.packages/engine/src/routes/stream-routes.ts, the stores; Codex P1 on 2123d5e). A time bound was not enough: a live tail can be delayed before its first read, or resumed into a gap read, after the id has changed hands.SubscribeToEventsOptionsgainsisStillAuthorized. The route passes a fence that re-reads the record and compares its incarnation with the one it checked.pollEvents: the filesystem store, and Postgres without LISTEN;deletealso ends the request's live streams.authorizeManagementRoutereturns the checked request record (RouteAuthResult.request, likesession). The stream answers as for an unknown request when its own read finds a different incarnation.isBatchStillAuthorizedis exported for this.packages/engine/src/stores/filesystem/request-store.ts, Codex P1 on 811f4a2). Deleting"foo"matchedfoo.runonce.bar.runonce.step.jsonthrough thefoo.runonce.prefix and erased another request's idempotency result, so its step could run again.<enc(id)>@<enc(key)>.runonce.encodeSegmentalways escapes@, so the first@ends the id..runonce.appears more than once) is renamed to.quarantined, out of every read path, so the id's next owner cannot read a previous owner's result. The cost is limited to ids containing.runonce.and pre-upgrade results: the other candidate owner may run that step again. A file that reads back as another request's record is left alone..eventsor.runonceshares its record file with another id's event log or single-map file (foo.runonce.jsonis bothfoo's single-map runOnce file and requestfoo.runonce's record). Deletingfooleaves such a file when it reads back as that record, and the single-map fallback never serves it asfoo's results (Codex P1 on 71a2e33). The same check coversfoo.events.json. The check and the delete are separate filesystem steps; closing that window moves to FIX-1665.packages/engine/src/stores/filesystem/shared.ts, Codex P2 on 71a2e33). The sidecar sweep runs first under the per-id lock, settles every removal before reporting a failure, and the record is removed only once it has succeeded. A failed cleanup leaves the id taken and the delete retryable, as in Postgres.packages/engine/src/routes/stream-routes.ts, Codex P1 on 13d6e6a). Between the ownership check and the event read, retention can delete the request and another caller can take the id. The route now re-reads the record aftergetEvents; if its incarnation changed, it answers as for an unknown request instead of replaying the new owner's events. The record-less branch likewise answers 404 if a record appears during its read.docs/architecture/streaming.md("Storage independence") no longer says events may outlive their request. Events may be pruned earlier, butRequestStore.deletemust remove every child under the id before the record. The exportedRequestStore.deleteinpackages/engine/src/stores/types.tshas a doc comment with the same contract: every child, record last, in-flight writes settled.onFinished/onErroredis drained after the final event, before the run counts as finished and before the id can be freed, but the stream may already have closed. This is stated the same way insequencer-side-chains.mdand in bothsequencer-dsl.mdlifecycle lines.ctx.suspend()ends assuspendedand returns. It fires noonCompleted/onErrored/onFinished, drains no terminal-hook work and writes nofinalizedAtMs, because the id stays resumable. Retention evicts only completed requests, so it keeps a suspended one. The sequence is now scoped to a request that finishes inexecution-and-errors.md(a suspended branch at step 8, the guarantees, and a suspended bullet),flows-and-actions.md,sequencer-dsl.mdandserver-and-client.md.sequencer-side-chains.mdalready scoped its guarantee to a finishing request.stream-routes-resume.test.ts: updated on purpose. The two "GC'd-record replay" tests assumed a deleted request leaves its events behind, which is no longer true. They are renamed "record-less replay" and still cover the route branch for events stored without a record. A new case pins that a deleted request's id now gets a 404. Apart from the 404 when a record appears during the read (round 9), the branch is unchanged: who may read record-less events is FIX-1046 (fix(engine): another user's request reads as unused; a session's reads keep to its own flow (FIX-1046) #2389).Linear FIX-1647 · epic FIX-1635 (Hard gates) · follow-up FIX-1665 · GitHub #2388 · merges to
main, not stackedHow to review this
Engineering calls
createRequestRetentionConformanceTests(new, in@flow-state-dev/engine/testing) drives the whole path. Alice POSTs a 5-message run underreq_retention_reused, and a second run in her session triggers retention (maxItems: 1), which evicts it. Bob, in another session, POSTs a 1-message run under the same id, then GETs/requests/:id/streamwith no cursor. The earlier run is deliberately longer than the later one, because SQL stores upsert by(request_id, sequence_number)and runs of equal length would overwrite every leftover row and hide the leak. It is one scenario run with three assertions: nothing is left under the id after retention, Bob's replay contains his run, and it does not contain Alice's. Memory, filesystem, SQLite and Postgres (PGlite) each run it. (The repro below was taken on the first version, which split the assertions across two cases.)createRequestStoreConformanceTests, so memory, filesystem, SQLite and Postgres all run them: delete removes events and runOnce results; a queued write can't undo the delete; delete leaves other requests' events alone.runActionmarks the recordcompletedbefore its tail runs (retention,onFinished, leaving the active registry). The first version waited only forcompleted, so cleanup raced the reused run's last writes. On filesystem that showed up asENOTEMPTYfromrm -rf. The harness now waits for the request to leave the active registry, disables the stale sweeper, and disposes the router before store cleanup. Filesystem runs the harness again (memory and filesystem in engine, plus SQLite and Postgres).RequestStorecontract changes.setFieldsIfStatusgainsexpectedIncarnation, and records gainfinalizedAtMsandheartbeatsUntilFinalized, which a custom store must persist. Patch for store-sqlite and store-postgres: their exported API is unchanged, and they only implement the contract. Users of a published store see different behaviour on delete, sessions can briefly exceed their retention limits during the grace window, and adapter authors get a new harness.onFinishedoutlasts the rollout bound (max(stale threshold,maxAge) + grace) can still have its record evicted while it writes. Nothing on its record can say otherwise.sequencer_checkpoints, which are already removed at terminal completion and by the durability sweeper; FIX-1619's read-window splice.onFinished. The running process cannot know the effective threshold, because the sweep can run on any host with its ownstaleSweepThresholdMs. Closing it needs the cadence persisted on the record and plumbing across hosts. Onmain, a misconfigured cadence already misfires the sweep (it marks such a live runinterrupted)./abortcannot reach that run.main70f777d, not changed here. These all come from the filesystem store naming sidecars<id>.<suffix>.json, so a request whose id ends in.eventsor.runonceshares a file name with another request's sidecar. The sidecar namespacing in FIX-1665 fixes them too:getEvents("foo")parsesfoo.events.jsonline by line. If that file is requestfoo.events's record, a replay offoowith no cursor yields the record as an event.persistEvents("foo")appends to the same file, corrupting that record.listskips*.events.jsonand*.runonce.json, so requests with those ids are never listed.main, deletingfooalready removed bothfoo.events.jsonandfoo.runonce.jsonwhatever they held. This PR now leaves them when they are another request's record (above).Everything else
Repro on
origin/main(before the fix), viacreateRequestRetentionConformanceTests:Store-level delete cases with the fix reverted: memory, SQLite and Postgres each fail "removes events and runOnce" and "not undone by events persisted just before it". Filesystem passes both.
Red states for the later fixes:
Postgres, record deleted in parallel with the children: the delete-order test fails (
expected undefined to be defined).Postgres,
Promise.allinstead ofallSettled: the settle test fails (expected true to be false).Filesystem,
deletenot waiting for the event queue: the race test fails in 3 of 3 runs (expected [ {…} ] to deeply equal []).Retention without the skip for still-finishing requests: the retention test fails (
expected [ 'req_finishing' ] to deeply equal []).Retention without the grace window: the six new grace-window tests in
retention.test.tsfail (resolved policy lacksterminalGraceMs; a request finished 59,999 ms ago is evicted).Retention on time alone (811f4a2): the new cross-process case in
retention-integration.test.ts, where A'sonFinishedis held past the window and B has its own registry, fails (expected undefined to be defined). The two newretention.test.tscases fail: anullrecord a year old is evicted; a stamped record is evicted inside the window.request-finalization.test.tson 811f4a2: three cases fail:nullduringonFinished, then set;onFinished;"Does not stamp another request that took the id" fails with the
createdAtfence removed.Two guard cases pass on both heads: no recreate after delete, and interrupted mid-flight stays unstamped.
Filesystem prefix collision: the new conformance case fails on filesystem (
expected { found: false } to deeply equal { Object (found, value) }). The two older-layout ownership cases also fail on the old code.Integration: without the shortened liveness, both eviction waits time out (the CI failure on 811f4a2).
Round 6 (Codex 5359669941). The new tests were run against the 13d6e6a source; 13 failed, each for its own reason:
onFinished.sideChain(): the side chain never saw an unstamped record.incomplete: stayed unstamped.resolveRetentionPolicyshape and bound tests.foo/bar.runonce.step: the next owner read the old result.Round 7 (Codex 5359925535): "never hands a result shaped like a record to the id's next owner" returned
found: trueon c183148.Round 8 (Codex 5360083826), on 48a8f4b:
onFinishedstamped the live run.heartbeatsUntilFinalized: false.Round 9 (Codex 5360207540), on 2123d5e:
[1,2,3]/[2,3]) on memory, filesystem and SQLite.expected undefined to be defined), and so does the record-less-read test (Bob's events are replayed).maxItems10 deletes nothing ([]).Round 10 (Codex 5360364765), on 71a2e33:
fooremovesfoo.runonce's andfoo.events's records (expected undefined to be 'foo.runonce').getRunOnceResult("foo", "status")returnsfoo.runonce's status.expected undefined to be 'req_stuck').Round 11 (Codex 5360479283), on a271519: in both orders (continuation first, original first), the run that ends first stamps the record while the other is still in
onFinished(expected { finalizedAtMs: 1790732376610, … } to deeply equal { finalizedAtMs: null, … }).Round 12 (Codex 5360608432), on 58b6c4e: when the last overlapping run ends by throwing, the finished record is left unstamped (
expected 'object' to be 'number').Post-cap review (Codex 5360726240), on 1df5087:
failedrecord unstamped (expected 'object' to be 'number').terminateUnenqueuedRequeststamp removed.Final cleanup (Codex 5360879994), on 35435a0: the new
runonce-per-key.test.tscase fails with thelastEventErrorclear removed: the id's next owner's first flush throws the deleted request's event write error.After merging
main(merge commits 8c4619e and f0567bd; head f0567bd):setFieldsIfStatuskeeps main's single(…, updatedAt, expectedIncarnation)fence. This PR's finalization stamp, the stale-sweep stamp and its conformance cases now call it, and thecreatedAtplaceholder argument is gone. Each store makes one incarnation compare. The abort poll keeps both guards: it is silent until the run's incarnation settles (FIX-1654) and closed once the run reaches terminal status (this PR). Both conformance blocks are kept.settleRequestRecord, which writes the action result with the status. This PR's terminal writes keepheartbeatsUntilFinalizedand take the written record's incarnation from it for the stamp. A fresh setup failure's record and an unenqueued request's terminal write carry both the result and the stamp.packages/engine: 239 files, 3212 passed, 6 skipped. This includesabort.test.tsandrequest-finalization*.test.tsin full.packages/store-sqlite: 339 passed.packages/store-postgres: 319 passed.packages/integration-tests: 143 passed.pnpm typecheck(whole repo): clean.After the fix, on head 8fd1fdc (eab30a4 after it changes docs only):
packages/engine: 3120 passed, 1 failed, the BR-23 timeout described below.pnpm typecheck: clean.On head 35435a0:
packages/engine: 3119 passed, 6 skipped, 1 failed.main70f777d's full suite here.mainand on this branch, and the whole file passes on both (48 tests on this head). This PR does not touch it.packages/store-sqlite: 331 passed.packages/store-postgres: 311 passed on a rerun.packages/integration-tests: 139 passed on a rerun.packages/memory: 522 passed.pnpm typecheck: clean.On head c183148: engine 3099 passed, 6 skipped; the other packages as above.
The
@flow-state-dev/memory#testCI failure on 13d6e6a is pre-existing and not caused by this PR. The failing test iscapture-evaluator.test.ts› "a delayed skip racing an evaluator failure never marks the failed turn read". History items sort bytsand then by per-requestitemIndex. When two turns share a millisecond, turn C's user message sorts ahead of turn B's later items, and B's index-based skip watermark then covers C. FreezingDate.now()makes the test fail the same way on this branch and onmain70f777d. Unfrozen, it passes on both (25/25 alone, 6 full-suite runs), so the CI failure was a timing tie. It is not fixed here.Earlier, on head 13d6e6a (includes a merge of
main):packages/engine: 234/234 files, 3086 passed, 6 skipped.packages/store-sqlite: 25 files, 327 tests passed.packages/store-postgres: 19 files, 307 tests passed.packages/integration-tests: 38 files, 139 tests passed.typecheck: engine, store-sqlite, store-postgres and integration-tests all clean.Files
packages/{engine/src/stores/memory,store-sqlite/src,store-postgres/src}/request-store.ts:deleteremoves events and runOnce results; Postgres settles every child delete and deletes the record last.packages/engine/src/stores/filesystem/request-store.ts:deletewaits for event writes before the sweep, and forgets the id's unreported event write error.packages/engine/test/filesystem-request-delete-race.test.ts: deterministic race test for the filesystem store.packages/engine/src/stores/testing/request-store-conformance.ts: new "delete conformance" block.packages/engine/src/stores/testing/request-retention-conformance.ts: new real-path harness, exported fromtesting.ts.packages/*/test/request-retention-conformance.test.ts: runs the harness for memory, filesystem, SQLite and Postgres, plus the two Postgres delete-order tests.packages/engine/test/stream-routes-resume.test.ts: record-less replay tests reframed, and the deleted-id 404 case added.packages/engine/src/execution/retention.tsandpackages/engine/test/retention.test.ts: the finalization fence, the grace window and the registry fast path, with deterministic tests.packages/engine/src/execution/runAction.ts: the finalfinalizedAtMsstamp in both exits, the heartbeat kept throughonFinished, the stamp reset on replay, and the stamp on a throwing exit.packages/engine/src/execution/request-recovery.ts: the stale sweep stamps a terminal record whose run died.packages/engine/src/transports/host/createInboundTransportHost.tsandtest/transports/host.test.ts: an unenqueued request's terminal record is stamped as written.packages/engine/src/stores/types.tsandcontext/initial-request-record.ts:finalizedAtMs,nullon new records.packages/engine/test/request-finalization.test.ts(new): the stamp's order and fence, the heartbeat throughonFinished, and the sweep.packages/engine/src/stores/filesystem/{request-store,shared}.tsandtest/stores/filesystem/runonce-per-key.test.ts: the@layout, the older-layout read and delete, the record check on the exact sidecar names, delete removing the record last, thelastEventErrorclear, and their tests.apps/docs/docs/persistence/overview.md: the custom-store live-tail recipe callsisStillAuthorized.apps/docs/docs/advanced/sequencer-side-chains.md: the drain guarantee covers work queued before the terminal hooks;onFinished/onErroredbackground work is drained after the final event, before the run finishes.packages/integration-tests/src/two-users-one-tenant/{harness,sibling-flow.test,run-workspace.test}.ts:shortenRetentionGrace()for the two eviction waits. The assertions are unchanged.packages/engine/src/streaming/live-tail-liveness.ts(new) andpackages/engine/src/routes/stream-routes.ts: the liveness default and env read, shared by the route and retention.packages/engine/test/retention-integration.test.tsand the retention harness: shortenLIVE_TAIL_LIVENESS_MSin the test and wait out the window between runs.packages/engine/src/stores/types.ts: doc comment onRequestStore.delete.docs/architecture/streaming.md: the delete contract under "Storage independence".docs/architecture/server-and-client.md: the grace window, the lazy bound, the rollout rule, the heartbeat condition, the overlapping-run rule and suspended requests under retention.docs/architecture/execution-and-errors.mdandflows-and-actions.md: the request lifecycle in its real order (terminal persist and emit,onFinished, side-chain drain, stamp), scoped to a request that finishes, with what a suspended run does instead.docs/architecture/sequencer-dsl.md: both side-chain lifecycle lines narrowed the same way, and the suspended case.packages/engine/README.md: adapter note, the grace window and the lazy bound under retention.apps/docs/docs/configuration/flow.md: the retention row gives the grace window as twice the configuredLIVE_TAIL_LIVENESS_MS(one minute by default).packages/engine/src/routes/{route-auth,http-handlers}.tsandpackages/engine/test/stream-routes-reused-id.test.ts(new): replay of an id taken again during the event read, the live-tail fence, the record-less read, and the guard-checked incarnation.packages/engine/src/stores/{types,subscribe-helpers}.ts,memory/request-store.ts,packages/store-sqlite/src/live-tail.ts,packages/store-postgres/src/request-store.ts: theisStillAuthorizedfence; conformance cases under "incarnation fence".packages/engine/src/execution/request-attempts.ts(new),runAction.ts,request-continuation.tsandtest/request-finalization-overlap.test.ts(new): only the last overlapping run finalizes and deregisters, including when it ends by throwing..changeset/request-delete-removes-events.md🤖 Generated with Claude Code
https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
Generated by Claude Code