Skip to content

fix(stores): delete a request's stream events with its record (FIX-1647) - #2393

Merged
jhoffner merged 22 commits into
mainfrom
fix/fix-1647
Sep 30, 2026
Merged

jhoffner merged 22 commits into
mainfrom
fix/fix-1647

Conversation

@jhoffner

@jhoffner jhoffner commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

The problem

When a session's retention policy removes an old request, the request's record goes but its stream events stay in the store. Callers can pick their own request ids, so once an id is freed another request can take it, and that request can belong to a different user in the same tenant. Replaying that request's stream from the start then sends the previous owner's output (messages, tool calls, results) to the new owner. Until now this came only from reading the code. It now reproduces on main with the memory, SQLite and Postgres stores.

Before the id is reused, the freed id's leftover events can also be replayed by anyone who knows the id: the stream route's "no record" branch replays whatever events it finds and has no owner to check.

What this does

Deleting a request now deletes everything the store keeps under that id: the record, its items, its stream events and its runOnce (cached step result) records. This applies to retention and to every other delete. Retention also frees an id only after the request's run has recorded that it finished (finalizedAtMs, written after onFinished), and a grace window has passed since (twice the live-tail liveness timeout, 60s by default), so the run's last writes, and any stream still reading it, are done first. The filesystem store already deleted these files, but an event write that was already in progress could recreate them afterwards. It now waits for that write first. Memory, SQLite and Postgres now delete events and runOnce records too. On the filesystem store, deleting one request could also remove another's runOnce results when one id started with the other; per-key files now have an unambiguous boundary.

What's asked of you

Nothing here is yours to decide. This is a bug on the direct route, so this PR is the only gate. One scope call is flagged below for the code reviewer. Three review findings are deferred to follow-up FIX-1665, and their threads are left open (see "Pre-existing, closure QA, and follow-up" under Engineering calls).

Parts worth reviewing closely

  • runOnce results are deleted too. The issue names events. runOnce results sit under the same id in the same stores, and a reused id would return the previous run's cached step result to the new owner's block. That is the same leak, fixed on the same line, and the filesystem store already deletes them. If you want this PR limited to events, say so and I'll split it out.
  • The in-flight window (packages/store-postgres/src/request-store.ts, packages/store-sqlite/src/request-store.ts): before the DELETE, delete drops any event batch that hasn't been written yet. Postgres also waits for a write that is already running, so a queued write can't bring the rows back afterwards. The conformance case "delete is not undone by events persisted just before it" covers this.
  • Delete order in Postgres. The executor has no transaction, so the order does the work: child rows go first and the record goes last. The child deletes run under Promise.allSettled, so a failure is reported only after every child delete has finished. A retry therefore can't free the id while an earlier attempt's straggler is still running. If a child delete fails, the record stays, so the id can't be claimed and the delete can be retried. Two tests cover this: one forces the event delete to fail and checks the record survives; the other holds the item delete and checks the failure waits for it. SQLite already deleted the record last.
  • Filesystem delete waits for event writes (packages/engine/src/stores/filesystem/request-store.ts). An event write takes its batch as soon as it starts, so delete now drops the unwritten batch and drains the id's event queue before sweeping the sidecar files. filesystem-request-delete-race.test.ts holds the append until the sweep has listed the directory, which makes the old ordering fail every time.
  • Filesystem delete also forgets the id's unreported event write error (same file, Codex P2 on 35435a0). The store holds an event write failure until the id's next flush reports it. Before, that entry outlived the delete, so the id's next owner was handed the previous owner's error by its first flush.
  • Retention evicts a request only after its run has stamped that it finished (packages/engine/src/execution/retention.ts, runAction.ts). runAction marks a record completed before its terminal event and onFinished, and onFinished has no time limit. So no grace window alone bounds the run's last writes, and a process-local registry cannot see a run in another process. Request records now carry finalizedAtMs:
    • New records start at null.
    • The run stamps it as its last write, through a conditional setFieldsIfStatus. That write comes after:
      • onFinished;
      • any .sideChain() work the hooks queued, which is drained;
      • pending item and event writes, which are settled independently (Promise.allSettled). If either flush fails, the run does not stamp, the same fallback as a failed stamp;
      • and just before the run leaves the registry.
    • The stamp is fenced on the incarnation of the record its terminal patch wrote. It never recreates a deleted record and never marks another request that took the id, even one created in the same millisecond.
      • For this, setFieldsIfStatus gains an optional expectedIncarnation in all four adapters, with conformance cases.
      • A legacy record resolves to legacy_<createdAt>.
    • Finalizable statuses are every terminal status except suspended, whose run resumes; incomplete is included.
    • If the store refuses the stamp 3 times, or a late flush failed, the run stays registered with its heartbeat stopped, so the stale-request sweep stamps it later.
    • Retention evicts only stamped records, once the grace window (2 × LIVE_TAIL_LIVENESS_MS) has passed since the stamp. A null record is never evicted, however old.
    • Rollout safety. During a rolling deploy, a record with no finalizedAtMs field (written by a version that never stamps) may still be finishing on an old instance. It is kept until completion + max(stale-request threshold, maxAge) + grace, never on the grace window alone (BP-030).
    • The registry check stays as a fast path.
    • Every terminal write reaches a stamp (Codex P2 on 1df5087):
      • The success and failure patches in runAction stamp at their finalization points.
      • A run that throws after a terminal write, including a fresh setup failure that writes failed, is stamped on exit from runActionInternal.
      • A dispatch that terminates a request no run will ever pick up (terminateUnenqueuedRequest) writes the stamp together with the status.
      • Two writes are left unstamped on purpose, because the request can still continue: the sweep's interrupted, and a replay setup failure's revert to suspended/interrupted.
  • A run that dies before stamping is stamped by the stale-request sweep, which is how the engine already declares a run dead, but only when a stale entry really means death.
    • The success path heartbeats until it stamps. The abort poll closes where the timer used to stop, so cancellation is unchanged.
    • The terminal write records heartbeatsUntilFinalized. It is true only on that path with heartbeats on; the sweep stamps only such records.
    • With request.heartbeatIntervalMs: 0, or on a failure path (which stops beating before onFinished), a live run looks stale. Its record is stamped only by its own run, and is kept, never evicted, if that run is gone. The same holds when the sweeper is disabled.
  • Only the last overlapping run of a request finalizes it (packages/engine/src/execution/request-attempts.ts, runAction.ts, Codex P1 on a271519). Two runs of one request can overlap on purpose: the sweep marks a slow but live run interrupted, and /continue then starts a second run with the same id and incarnation. Each run is now counted as a live attempt of its request id, per request store and in this process. A run stamps and removes the shared registry entry only if it is the last attempt to end. The suspend path and the continuation's safety-net deregister follow the same rule.
    • A run that ends while another is live passes its terminal record's incarnation to the shared attempt state. If the last run then ends by throwing, the exit of runActionInternal still stamps under that incarnation and deregisters, so the record is not kept forever (Codex P2 on 58b6c4e).
    • Overlap across processes is still judged by the heartbeat. A per-attempt id on the record and on the registry entry would close that, and it moves to FIX-1665.
  • Kept requests still count toward maxItems (Codex P2 on 2123d5e). A request that is still running or not yet evictable was left out of the budget, so the session could grow past the budget with nothing evicted. Its items now count, and older evictable history is evicted to make room; only its own deletion waits.
  • Spared requests are evicted lazily, by the session's next completed request (Codex P2, earlier round). Retention has always been lazy, and a timer would be new per-session machinery. The bound is written in server-and-client.md, the engine README and the retention row in apps/docs configuration.
  • Live streams are fenced on the incarnation the route authorized (packages/engine/src/routes/stream-routes.ts, the stores; Codex P1 on 2123d5e). A time bound was not enough: a live tail can be delayed before its first read, or resumed into a gap read, after the id has changed hands.
    • SubscribeToEventsOptions gains isStillAuthorized. The route passes a fence that re-reads the record and compares its incarnation with the one it checked.
    • Each store calls the fence after every read that returned events and before yielding them. When it fails, the stream yields none of that batch and ends as for an unknown request. Where it is called:
      • pollEvents: the filesystem store, and Postgres without LISTEN;
      • memory: catch-up and gap reads;
      • the SQLite live tail: initial read and buffer drain;
      • Postgres LISTEN: initial, gap and next reads.
    • The memory store's delete also ends the request's live streams.
    • The authorized incarnation is the one the route guard checked (Codex P1 on 1df5087). authorizeManagementRoute returns the checked request record (RouteAuthResult.request, like session). The stream answers as for an unknown request when its own read finds a different incarnation.
    • Cost: one record read per non-empty batch.
    • The Postgres LISTEN path has no test harness in the repo, since PGlite covers only polling. It gets the same fence, but nothing tests it there.
    • A custom store must honor the fence. isBatchStillAuthorized is exported for this.
  • Filesystem runOnce files have an unambiguous id boundary (packages/engine/src/stores/filesystem/request-store.ts, Codex P1 on 811f4a2). Deleting "foo" matched foo.runonce.bar.runonce.step.json through the foo.runonce. prefix and erased another request's idempotency result, so its step could run again.
    • Per-key files are now <enc(id)>@<enc(key)>.runonce. encodeSegment always escapes @, so the first @ ends the id.
    • Files in the older layout are still read, except one that reads back as the record whose id is its whole name. A stored result is arbitrary JSON, so that file cannot be told apart from another request's record. It is never served as a result, and delete leaves it in place.
    • On delete, an older file that only this id could have written is removed. One whose name another id could also have written (.runonce. appears more than once) is renamed to .quarantined, out of every read path, so the id's next owner cannot read a previous owner's result. The cost is limited to ids containing .runonce. and pre-upgrade results: the other candidate owner may run that step again. A file that reads back as another request's record is left alone.
    • A caller-chosen id ending in .events or .runonce shares its record file with another id's event log or single-map file (foo.runonce.json is both foo's single-map runOnce file and request foo.runonce's record). Deleting foo leaves such a file when it reads back as that record, and the single-map fallback never serves it as foo's results (Codex P1 on 71a2e33). The same check covers foo.events.json. The check and the delete are separate filesystem steps; closing that window moves to FIX-1665.
  • Filesystem delete removes the record last (packages/engine/src/stores/filesystem/shared.ts, Codex P2 on 71a2e33). The sidecar sweep runs first under the per-id lock, settles every removal before reporting a failure, and the record is removed only once it has succeeded. A failed cleanup leaves the id taken and the delete retryable, as in Postgres.
  • Terminal replay re-checks the record after reading events (packages/engine/src/routes/stream-routes.ts, Codex P1 on 13d6e6a). Between the ownership check and the event read, retention can delete the request and another caller can take the id. The route now re-reads the record after getEvents; if its incarnation changed, it answers as for an unknown request instead of replaying the new owner's events. The record-less branch likewise answers 404 if a record appears during its read.
  • The delete contract is now written down. docs/architecture/streaming.md ("Storage independence") no longer says events may outlive their request. Events may be pruned earlier, but RequestStore.delete must remove every child under the id before the record. The exported RequestStore.delete in packages/engine/src/stores/types.ts has a doc comment with the same contract: every child, record last, in-flight writes settled.
  • The finish sequence is documented as it runs, and only for a request that finishes (Codex P2s on 35435a0 and 8fd1fdc; docs only).
    • Side-chain drain: work queued before the terminal hooks is drained before terminal status, and the stream stays open for it. Work queued from onFinished/onErrored is drained after the final event, before the run counts as finished and before the id can be freed, but the stream may already have closed. This is stated the same way in sequencer-side-chains.md and in both sequencer-dsl.md lifecycle lines.
    • Suspension: a run that calls ctx.suspend() ends as suspended and returns. It fires no onCompleted/onErrored/onFinished, drains no terminal-hook work and writes no finalizedAtMs, because the id stays resumable. Retention evicts only completed requests, so it keeps a suspended one. The sequence is now scoped to a request that finishes in execution-and-errors.md (a suspended branch at step 8, the guarantees, and a suspended bullet), flows-and-actions.md, sequencer-dsl.md and server-and-client.md. sequencer-side-chains.md already scoped its guarantee to a finishing request.
  • stream-routes-resume.test.ts: updated on purpose. The two "GC'd-record replay" tests assumed a deleted request leaves its events behind, which is no longer true. They are renamed "record-less replay" and still cover the route branch for events stored without a record. A new case pins that a deleted request's id now gets a 404. Apart from the 404 when a record appears during the read (round 9), the branch is unchanged: who may read record-less events is FIX-1046 (fix(engine): another user's request reads as unused; a session's reads keep to its own flow (FIX-1046) #2389).
  • Why deletion and not re-keying. Retention now deletes only requests whose runs have stamped that they finished, more than the grace window ago (see above), so their writes are done and their streams have closed. The one caller that deletes a live request (a dropped delivery) is covered by the in-flight case above. Keying events by a per-run token would need a schema change and a migration to close a window this change already closes. A reuse while the old run is still live is the FIX-1018 reparenting path, which is out of scope here.

Linear FIX-1647 · epic FIX-1635 (Hard gates) · follow-up FIX-1665 · GitHub #2388 · merges to main, not stacked

How to review this

How to review this. This is a bug fix on the direct route: there is no spec and no
spec PR, because the hard part was the diagnosis and it happened in code. This PR is the
only gate
, so unlike a spec-backed change, the approach is in scope: nothing here was
signed off upstream.

Most valuable here: the diagnosis chain. Does the repro actually reproduce the reported
failure, is the named cause the real one, does the fix address that cause rather than the
symptom, and does the regression test fail without the fix? Then the second path (the legacy
shape, the null boundary, the concurrent case, the cancel path).

In scope to challenge: where the guard was placed, any behaviour change a user could
notice, and whether this should have been a spec'd feature instead of a fix.

Engineering calls
  • Real-path repro as a shared harness. createRequestRetentionConformanceTests (new, in @flow-state-dev/engine/testing) drives the whole path. Alice POSTs a 5-message run under req_retention_reused, and a second run in her session triggers retention (maxItems: 1), which evicts it. Bob, in another session, POSTs a 1-message run under the same id, then GETs /requests/:id/stream with no cursor. The earlier run is deliberately longer than the later one, because SQL stores upsert by (request_id, sequence_number) and runs of equal length would overwrite every leftover row and hide the leak. It is one scenario run with three assertions: nothing is left under the id after retention, Bob's replay contains his run, and it does not contain Alice's. Memory, filesystem, SQLite and Postgres (PGlite) each run it. (The repro below was taken on the first version, which split the assertions across two cases.)
  • Store-level cases are added to the existing createRequestStoreConformanceTests, so memory, filesystem, SQLite and Postgres all run them: delete removes events and runOnce results; a queued write can't undo the delete; delete leaves other requests' events alone.
  • The harness waits for the whole run. runAction marks the record completed before its tail runs (retention, onFinished, leaving the active registry). The first version waited only for completed, so cleanup raced the reused run's last writes. On filesystem that showed up as ENOTEMPTY from rm -rf. The harness now waits for the request to leave the active registry, disables the stale sweeper, and disposes the router before store cleanup. Filesystem runs the harness again (memory and filesystem in engine, plus SQLite and Postgres).
  • Changeset: minor for engine, because the RequestStore contract changes. setFieldsIfStatus gains expectedIncarnation, and records gain finalizedAtMs and heartbeatsUntilFinalized, which a custom store must persist. Patch for store-sqlite and store-postgres: their exported API is unchanged, and they only implement the contract. Users of a published store see different behaviour on delete, sessions can briefly exceed their retention limits during the grace window, and adapter authors get a new harness.
  • Rollout residual, not fixable from data: an old-version run with heartbeats off whose onFinished outlasts the rollout bound (max(stale threshold, maxAge) + grace) can still have its record evicted while it writes. Nothing on its record can say otherwise.
  • Not touched: sequencer_checkpoints, which are already removed at terminal completion and by the durability sweeper; FIX-1619's read-window splice.
  • Pre-existing, closure QA, and follow-up FIX-1665. The id-reuse class predates this PR, and retention is opt-in.
    • Deferred to FIX-1665 (threads left open):
      • P1 4140101770 (Codex round 12): overlapping runs across processes. Closing it needs a durable per-attempt id on the request record and the active-registry entry in all four adapters. Until then, a run in another process whose heartbeat went stale is treated as dead, as the sweep already does.
      • P2 4140101779 (Codex round 12): the filesystem record check and sidecar delete are two steps, so a record written between them can still be deleted. Closing it needs namespaced filesystem sidecars (a layout rename with a migration).
      • P1 4140317041 (Codex on 35435a0): heartbeat cadence against the stale threshold. A heartbeat interval at or above the sweeper's stale threshold lets the sweep stamp a live run during onFinished. The running process cannot know the effective threshold, because the sweep can run on any host with its own staleSweepThresholdMs. Closing it needs the cadence persisted on the record and plumbing across hosts. On main, a misconfigured cadence already misfires the sweep (it marks such a live run interrupted).
      • Not a review thread: the abort-controller map is keyed by request id alone. When the first of two overlapping runs ends, it drops the controller the other run registered, so /abort cannot reach that run.
    • On main 70f777d, not changed here. These all come from the filesystem store naming sidecars <id>.<suffix>.json, so a request whose id ends in .events or .runonce shares a file name with another request's sidecar. The sidecar namespacing in FIX-1665 fixes them too:
      • getEvents("foo") parses foo.events.json line by line. If that file is request foo.events's record, a replay of foo with no cursor yields the record as an event.
      • persistEvents("foo") appends to the same file, corrupting that record.
      • list skips *.events.json and *.runonce.json, so requests with those ids are never listed.
      • On main, deleting foo already removed both foo.events.json and foo.runonce.json whatever they held. This PR now leaves them when they are another request's record (above).
Everything else

Repro on origin/main (before the fix), via createRequestRetentionConformanceTests:

× memory stores   > retention removes a request's stream events with its record  → expected 16 to be +0
× memory stores   > a replay of a reused id carries only the new owner's run      → expected 'id: req_retention_reused:1…' not to contain 'alice-secret'
× SQLite stores   > (same two)                                                    → expected 16 to be +0 / … not to contain 'alice-secret'
× Postgres stores > (same two)                                                    → expected 16 to be +0 / … not to contain 'alice-secret'
✓ filesystem stores > (same two)                                                  control, already correct

Store-level delete cases with the fix reverted: memory, SQLite and Postgres each fail "removes events and runOnce" and "not undone by events persisted just before it". Filesystem passes both.

Red states for the later fixes:

  • Postgres, record deleted in parallel with the children: the delete-order test fails (expected undefined to be defined).

  • Postgres, Promise.all instead of allSettled: the settle test fails (expected true to be false).

  • Filesystem, delete not waiting for the event queue: the race test fails in 3 of 3 runs (expected [ {…} ] to deeply equal []).

  • Retention without the skip for still-finishing requests: the retention test fails (expected [ 'req_finishing' ] to deeply equal []).

  • Retention without the grace window: the six new grace-window tests in retention.test.ts fail (resolved policy lacks terminalGraceMs; a request finished 59,999 ms ago is evicted).

  • Retention on time alone (811f4a2): the new cross-process case in retention-integration.test.ts, where A's onFinished is held past the window and B has its own registry, fails (expected undefined to be defined). The two new retention.test.ts cases fail: a null record a year old is evicted; a stamped record is evicted inside the window.

  • request-finalization.test.ts on 811f4a2: three cases fail:

    • the stamp is null during onFinished, then set;
    • the heartbeat advances during a slow onFinished;
    • the sweep stamps a record whose run died.
  • "Does not stamp another request that took the id" fails with the createdAt fence removed.

  • Two guard cases pass on both heads: no recreate after delete, and interrupted mid-flight stays unstamped.

  • Filesystem prefix collision: the new conformance case fails on filesystem (expected { found: false } to deeply equal { Object (found, value) }). The two older-layout ownership cases also fail on the old code.

  • Integration: without the shortened liveness, both eviction waits time out (the CI failure on 811f4a2).

  • Round 6 (Codex 5359669941). The new tests were run against the 13d6e6a source; 13 failed, each for its own reason:

    • onFinished .sideChain(): the side chain never saw an unstamped record.
    • incomplete: stayed unstamped.
    • Same-millisecond newcomer: stamped.
    • Stamp retry: 1 attempt, record unstamped.
    • Stamp down: registry entry gone, so nothing would ever stamp it.
    • Legacy record: evicted just past the grace window.
    • Five resolveRetentionPolicy shape and bound tests.
    • Reused-id replay: Bob's events reached Alice.
    • foo / bar.runonce.step: the next owner read the old result.
  • Round 7 (Codex 5359925535): "never hands a result shaped like a record to the id's next owner" returned found: true on c183148.

  • Round 8 (Codex 5360083826), on 48a8f4b:

    • With heartbeats off, a sweep during onFinished stamped the live run.
    • A failed late item flush still let the run stamp.
    • The sweep stamped a record with heartbeatsUntilFinalized: false.
  • Round 9 (Codex 5360207540), on 2123d5e:

    • Two new fence conformance cases (the id changes hands before the first read; after a later read) yield the new owner's events ([1,2,3] / [2,3]) on memory, filesystem and SQLite.
    • The route test for the live-tail fence fails (expected undefined to be defined), and so does the record-less-read test (Bob's events are replayed).
    • Retention with old 5 / protected 8 / current 3 and maxItems 10 deletes nothing ([]).
  • Round 10 (Codex 5360364765), on 71a2e33:

    • Deleting foo removes foo.runonce's and foo.events's records (expected undefined to be 'foo.runonce').
    • getRunOnceResult("foo", "status") returns foo.runonce's status.
    • A sidecar that cannot be removed leaves the record deleted (expected undefined to be 'req_stuck').
  • Round 11 (Codex 5360479283), on a271519: in both orders (continuation first, original first), the run that ends first stamps the record while the other is still in onFinished (expected { finalizedAtMs: 1790732376610, … } to deeply equal { finalizedAtMs: null, … }).

  • Round 12 (Codex 5360608432), on 58b6c4e: when the last overlapping run ends by throwing, the finished record is left unstamped (expected 'object' to be 'number').

  • Post-cap review (Codex 5360726240), on 1df5087:

    • A single run that fails during setup leaves its failed record unstamped (expected 'object' to be 'number').
    • The host's enqueue-failure test fails the same way with the terminateUnenqueuedRequest stamp removed.
    • With the handler fence disabled, a stream whose guard checked Alice's record serves Bob's events under the same id.
  • Final cleanup (Codex 5360879994), on 35435a0: the new runonce-per-key.test.ts case fails with the lastEventError clear removed: the id's next owner's first flush throws the deleted request's event write error.

After merging main (merge commits 8c4619e and f0567bd; head f0567bd):

  • FIX-1654's incarnation fence (fix(engine): fence cancels on the request's incarnation (FIX-1654) #2430). setFieldsIfStatus keeps main's single (…, updatedAt, expectedIncarnation) fence. This PR's finalization stamp, the stale-sweep stamp and its conformance cases now call it, and the createdAt placeholder argument is gone. Each store makes one incarnation compare. The abort poll keeps both guards: it is silent until the run's incarnation settles (FIX-1654) and closed once the run reaches terminal status (this PR). Both conformance blocks are kept.
  • FIX-1661's settle helper (FIX-1661: the engine records each request's action result; the DevTool row reads it #2429). Every final-status write goes through settleRequestRecord, which writes the action result with the status. This PR's terminal writes keep heartbeatsUntilFinalized and take the written record's incarnation from it for the stamp. A fresh setup failure's record and an unenqueued request's terminal write carry both the result and the stamp.
  • Validation on f0567bd:
    • packages/engine: 239 files, 3212 passed, 6 skipped. This includes abort.test.ts and request-finalization*.test.ts in full.
    • packages/store-sqlite: 339 passed.
    • packages/store-postgres: 319 passed.
    • packages/integration-tests: 143 passed.
    • pnpm typecheck (whole repo): clean.
    • CI: green (run 36765276073).
  • The first merge alone (8c4619e) also passed engine, SQLite and integration; Postgres timed out one load-sensitive cost test, which passed alone.

After the fix, on head 8fd1fdc (eab30a4 after it changes docs only):

  • packages/engine: 3120 passed, 1 failed, the BR-23 timeout described below.
  • pnpm typecheck: clean.

On head 35435a0:

  • packages/engine: 3119 passed, 6 skipped, 1 failed.
    • The failure is the filesystem "keeps an unfinished run older than a page of runs listed (BR-23)", which times out at 10s under full-suite load in this container.
    • It fails the same way in main 70f777d's full suite here.
    • Alone it takes about 6.5s on main and on this branch, and the whole file passes on both (48 tests on this head). This PR does not touch it.
  • packages/store-sqlite: 331 passed.
  • packages/store-postgres: 311 passed on a rerun.
    • The first full run timed out one test at 60s: "Postgres cost of the session stream's reads > a caller bound to no tenant…". That test does not touch this change.
  • packages/integration-tests: 139 passed on a rerun.
    • The first full run failed one timing assertion: "task-board … claimed promptly, not at the poll timeout", at 432ms.
  • packages/memory: 522 passed.
  • pnpm typecheck: clean.

On head c183148: engine 3099 passed, 6 skipped; the other packages as above.

The @flow-state-dev/memory#test CI failure on 13d6e6a is pre-existing and not caused by this PR. The failing test is capture-evaluator.test.ts › "a delayed skip racing an evaluator failure never marks the failed turn read". History items sort by ts and then by per-request itemIndex. When two turns share a millisecond, turn C's user message sorts ahead of turn B's later items, and B's index-based skip watermark then covers C. Freezing Date.now() makes the test fail the same way on this branch and on main 70f777d. Unfrozen, it passes on both (25/25 alone, 6 full-suite runs), so the CI failure was a timing tie. It is not fixed here.

Earlier, on head 13d6e6a (includes a merge of main):

  • packages/engine: 234/234 files, 3086 passed, 6 skipped.
    • One earlier full run instead had 1 failure: the filesystem "keeps an unfinished run older than a page of runs listed (BR-23)" timed out at 10s.
    • That test seeds 106 records directly and runs no code this PR touches. It passes in 2.3s alone and 3 out of 3 as a file, and the next full run was green.
  • packages/store-sqlite: 25 files, 327 tests passed.
  • packages/store-postgres: 19 files, 307 tests passed.
  • packages/integration-tests: 38 files, 139 tests passed.
  • typecheck: engine, store-sqlite, store-postgres and integration-tests all clean.

Files

  • packages/{engine/src/stores/memory,store-sqlite/src,store-postgres/src}/request-store.ts: delete removes events and runOnce results; Postgres settles every child delete and deletes the record last.
  • packages/engine/src/stores/filesystem/request-store.ts: delete waits for event writes before the sweep, and forgets the id's unreported event write error.
  • packages/engine/test/filesystem-request-delete-race.test.ts: deterministic race test for the filesystem store.
  • packages/engine/src/stores/testing/request-store-conformance.ts: new "delete conformance" block.
  • packages/engine/src/stores/testing/request-retention-conformance.ts: new real-path harness, exported from testing.ts.
  • packages/*/test/request-retention-conformance.test.ts: runs the harness for memory, filesystem, SQLite and Postgres, plus the two Postgres delete-order tests.
  • packages/engine/test/stream-routes-resume.test.ts: record-less replay tests reframed, and the deleted-id 404 case added.
  • packages/engine/src/execution/retention.ts and packages/engine/test/retention.test.ts: the finalization fence, the grace window and the registry fast path, with deterministic tests.
  • packages/engine/src/execution/runAction.ts: the final finalizedAtMs stamp in both exits, the heartbeat kept through onFinished, the stamp reset on replay, and the stamp on a throwing exit.
  • packages/engine/src/execution/request-recovery.ts: the stale sweep stamps a terminal record whose run died.
  • packages/engine/src/transports/host/createInboundTransportHost.ts and test/transports/host.test.ts: an unenqueued request's terminal record is stamped as written.
  • packages/engine/src/stores/types.ts and context/initial-request-record.ts: finalizedAtMs, null on new records.
  • packages/engine/test/request-finalization.test.ts (new): the stamp's order and fence, the heartbeat through onFinished, and the sweep.
  • packages/engine/src/stores/filesystem/{request-store,shared}.ts and test/stores/filesystem/runonce-per-key.test.ts: the @ layout, the older-layout read and delete, the record check on the exact sidecar names, delete removing the record last, the lastEventError clear, and their tests.
  • apps/docs/docs/persistence/overview.md: the custom-store live-tail recipe calls isStillAuthorized.
  • apps/docs/docs/advanced/sequencer-side-chains.md: the drain guarantee covers work queued before the terminal hooks; onFinished/onErrored background work is drained after the final event, before the run finishes.
  • packages/integration-tests/src/two-users-one-tenant/{harness,sibling-flow.test,run-workspace.test}.ts: shortenRetentionGrace() for the two eviction waits. The assertions are unchanged.
  • packages/engine/src/streaming/live-tail-liveness.ts (new) and packages/engine/src/routes/stream-routes.ts: the liveness default and env read, shared by the route and retention.
  • packages/engine/test/retention-integration.test.ts and the retention harness: shorten LIVE_TAIL_LIVENESS_MS in the test and wait out the window between runs.
  • packages/engine/src/stores/types.ts: doc comment on RequestStore.delete.
  • docs/architecture/streaming.md: the delete contract under "Storage independence".
  • docs/architecture/server-and-client.md: the grace window, the lazy bound, the rollout rule, the heartbeat condition, the overlapping-run rule and suspended requests under retention.
  • docs/architecture/execution-and-errors.md and flows-and-actions.md: the request lifecycle in its real order (terminal persist and emit, onFinished, side-chain drain, stamp), scoped to a request that finishes, with what a suspended run does instead.
  • docs/architecture/sequencer-dsl.md: both side-chain lifecycle lines narrowed the same way, and the suspended case.
  • packages/engine/README.md: adapter note, the grace window and the lazy bound under retention.
  • apps/docs/docs/configuration/flow.md: the retention row gives the grace window as twice the configured LIVE_TAIL_LIVENESS_MS (one minute by default).
  • packages/engine/src/routes/{route-auth,http-handlers}.ts and packages/engine/test/stream-routes-reused-id.test.ts (new): replay of an id taken again during the event read, the live-tail fence, the record-less read, and the guard-checked incarnation.
  • packages/engine/src/stores/{types,subscribe-helpers}.ts, memory/request-store.ts, packages/store-sqlite/src/live-tail.ts, packages/store-postgres/src/request-store.ts: the isStillAuthorized fence; conformance cases under "incarnation fence".
  • packages/engine/src/execution/request-attempts.ts (new), runAction.ts, request-continuation.ts and test/request-finalization-overlap.test.ts (new): only the last overlapping run finalizes and deregisters, including when it ends by throwing.
  • .changeset/request-delete-removes-events.md

🤖 Generated with Claude Code

https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G


Generated by Claude Code

…ecord (FIX-1647)

Session retention deleted a request's record but left its stream events
(and runOnce results) in the memory, SQLite and Postgres stores. Request
ids are caller-supplied, so a later request, from another user, could
take the freed id and replay the previous run's events from sequence 0.

Reproduced end to end on main through the HTTP action route, real
retention and the HTTP stream route on all three stores. The filesystem
store already swept these sidecars and passed as the control.

RequestStore.delete now removes events and runOnce results with the
record. SQLite and Postgres drop a not-yet-written event batch first and
Postgres waits out an in-flight write, so a queued write cannot restore
rows after the delete.

Tests: a store-level delete block in createRequestStoreConformanceTests
(memory, filesystem, SQLite, Postgres), and a new
createRequestRetentionConformanceTests that drives the real path, run
for memory, SQLite and Postgres (PGlite).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G

Copy link
Copy Markdown
Collaborator Author

CI note: the "Process guards" check is red on every open PR right now because spec/_projects is on main. This PR doesn't touch it, and that failure is unrelated to this change. Every other check on this PR is meant to pass on its own.


Generated by Claude Code

@vercel

vercel Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
flow-state-dev-docs Ignored Ignored Sep 30, 2026 7:23pm UTC
flow-state-dev-kitchen-sink Ignored Ignored Sep 30, 2026 7:23pm UTC
flow-state-dev-ui Ignored Ignored Preview Sep 30, 2026 7:23pm UTC

Request Review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-30T02:59:28.182042Z 8fd1fdc Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@linear-code

linear-code Bot commented Sep 29, 2026

Copy link
Copy Markdown

FIX-1647

Copy link
Copy Markdown
Collaborator Author

Second-look + architecture review (read-only, no edits made)

Verdict: the approach is right. Deleting at the source is the leanest fix. It matches the filesystem store, which was already correct, and needs no schema change. The findings below are about hardening. None of them argue for a different shape.

1. Postgres delete is not atomic, and the order can strand orphans (should fix)

packages/store-postgres/src/request-store.ts (the Promise.all in delete) runs request_items, request_events, request_runonce and base.delete(id) concurrently, with no transaction. If the record delete succeeds and an events or runOnce delete fails (connection drop, statement timeout), the record is gone but its events stay. Retention discovers requests through their records, so it never retries that id. That recreates the leak this PR fixes, and the stream route's record-less replay branch would serve it.

Suggested fix:

  • Delete the children first and the record last, sequentially. A failure then leaves the record in place, and the next retention sweep retries.
  • Or wrap all four deletes in one transaction if the executor supports it.

The SQLite store deletes children first, synchronously, and has no problem. Only Postgres needs the reorder.

2. Ids already orphaned before this fix stay leaked (BP-030 legacy shape)

The fix only stops new orphans. Existing SQLite and Postgres deployments already have events and runOnce rows under ids whose records retention has removed. Those rows can still be replayed to the next request that takes the id, and by anyone who knows the id through the stream route's no-record branch. The PR body says that branch "no longer finds retention leftovers", which holds only for data written after this change. Options:

  • Say explicitly in the changeset or PR that pre-existing orphans are not cleaned up.
  • Add a one-off orphan sweep, e.g. DELETE FROM request_events WHERE request_id NOT IN (SELECT id FROM requests), run next to the durability sweeper.

I'd take the first at minimum. If the sweep is out of scope, file it as a follow-up.

3. Second-path checklist (BP-035)

  • Reuse of a live id: correctly left to FIX-1018.
  • Concurrent persistEvents during delete: the Postgres in-flight wait is sound. pendingNewEvents.delete plus awaiting pendingEventWrites covers a batch that was queued or already running. Note that the guard only holds if no new persistEvents arrives for the same id during the await. That only happens for a live request, which is the runAction.ts:211 dropped-delivery path, so it is acceptable. Worth one line in the code comment.
  • Multi-tenant: the delete is keyed on request_id alone. That is consistent with the store's existing keying, and no cross-tenant delete path is introduced.

4. Test-harness duplication (minor, optional)

request-retention-conformance.ts adds a local waitFor. engine/test/served-setup-failure.test.ts has a similar waitForServedStatus. If the testing utilities have a shared polling helper, use it. If not, a local helper is fine, since this file ships from @flow-state-dev/engine/testing and can't depend on test-only files.

Test-gap nit: the e2e harness proves events are removed. The store-level case covers runOnce removal, but nothing exercises the runOnce leak through the real path. That is acceptable given the shared line of code.

5. Architecture note (no action in this PR)

Three stores now each hand-roll "what belongs to a request id" in their delete. The filesystem store had it right and the other three drifted, and the conformance cases now pin it. The durable fix is a store contract, not another comment: the RequestStore.delete interface doc should state "removes the record, items, events and runOnce results", so a fourth adapter author sees it at the type. The README note covers docs, but the interface JSDoc in stores/types is where an adapter author reads first. Adding one line there is cheap and in scope.

Not flagged

  • Scope call on runOnce: keep it in this PR. It is the same leak on the same line.
  • Changeset level and content are fine.
  • Filesystem exclusion from the e2e harness is reasonable, given the store-level coverage.

Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e0f10e2cdc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Snob — too small to rewrite. No POC.

I read packages/engine/src/stores/memory/request-store.ts, packages/store-sqlite/src/request-store.ts, packages/store-postgres/src/request-store.ts, packages/engine/src/stores/filesystem/request-store.ts (delete / deleteSidecars), packages/engine/src/stores/testing/request-store-conformance.ts, packages/engine/src/stores/testing/request-retention-conformance.ts, the three request-retention-conformance.test.ts files, packages/engine/src/testing.ts, the retention section of packages/engine/README.md, .changeset/request-delete-removes-events.md, and the sqlite schema note that request_events / request_runonce have no foreign key and are pruned on the request-lifecycle path (packages/store-sqlite/src/schema.ts).

A rewrite would not shrink this contract. Each store already deletes items in its own mechanism (a map, a prepared statement, a query). Events and runOnce are those same two operations beside it. One helper cannot issue all three. SQLite drops a queued map; Postgres also awaits an in-flight write. Each is a single call site. The store suite adds three claims to the existing conformance function. The HTTP harness is the recorded real-path repro: two named failures (rows left behind, and a replay of a shorter second run) plus three thin package entries. pgliteExecutor matches the other Postgres tests. The README paragraph is the adapter contract for delete. The repeated id-reuse comments sit on the non-obvious deletes; removing them drops explanation, not behavior.

Head e0f10e2cdca4b2f9a8ac84748cc7f41fe7008d36.

Open in Web View Automation 

Sent by Cursor Automation: Code Snob

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The bug fix itself is the right shape: extend RequestStore.delete in memory, SQLite, and Postgres so stream events and runOnce results go away with the record, mirroring filesystem deleteSidecars and the existing request_items delete + pending-queue cancellation pattern. Retention already calls stores.request.delete(id) only, so this seam is correct; FK ON DELETE CASCADE would not replace the in-process pendingNewEvents / pendingEventWrites guards and would fight the schema’s intentional lack of FKs on these tables.

I would not chase a different production architecture here. The diff stays localized and consistent across adapters.

Where complexity crept in (tests / docs)

Most of the line count is conformance, not behavior:

  1. Retention HTTP suite vs store delete tests — The new createRequestStoreConformanceTests delete block (including the queued-persistEvents race) is the direct guard for FIX-1647. createRequestRetentionConformanceTests re-proves the same invariant through retention → HTTP stream replay. That is valuable as a regression story (especially Alice count: 5 vs Bob count: 1 so upsert-by-sequence cannot hide leftovers), but it is optional if you want a smaller PR: store-level delete conformance + a one-line README contract might be enough, since retention is a thin wrapper around delete. I would keep the E2E suite if CI time is acceptable; I would drop it only if you are optimizing for minimal surface area and trust the store contract.

  2. Duplicate scenario runs — The two it blocks each call runReusedIdScenario end-to-end (~2× HTTP work per backend). Merging into one test with both assertions is an easy win (see inline).

  3. Repeated rationale comments — The caller-supplied-id / replay paragraph appears in memory, SQLite, Postgres, README, and conformance headers. The queue/microtask notes in SQL adapters are worth keeping; the rest could lean on README + conformance once.

  4. SQLite test harness — session-stream-conformance.test.ts already uses afterAll + tracked registries instead of a typed cleanup cast; aligning would be a small consistency cleanup (optional).

Performance / reuse

No hot-path concerns: delete remains eviction-scale. Postgres adds two parallel DELETEs in the existing Promise.all; SQLite adds two prepared statements. No third-party shortcut worth introducing.

Optional follow-up (not blocking)

Filesystem delete sweeps sidecars under lock but does not call pendingNewEvents.delete(id) like SQL adapters now do; conformance passes today, but aligning filesystem with the SQL race pattern would make cross-adapter semantics uniform by construction.

Verdict: Approve. Production change is minimal and matches established patterns; test/doc nits above are polish, not blockers.

Open in Web View Automation 

Sent by Cursor Automation: Review For Best Approach

…n and disposes its router (FIX-1647)

Postgres delete ran the child deletes and the record delete in parallel
as independent autocommit queries. A failed child delete could leave
events with no record, and an id claimed in the gap could have its new
rows removed by the leftover cleanup. Children now go first and the
record last. SQLite already deleted the record last.

The retention harness returned while the reused run's tail (retention,
leaving the active registry) was still writing, and never disposed its
router, whose stale sweeper outlived the case. That was the filesystem
teardown ENOTEMPTY. The harness now waits for the run to leave the
active registry, disables the sweeper and disposes the router, and
filesystem is back in it. The two cases are one scenario run; SQLite
closes registries after the file.

The stream route's record-less replay tests are reframed: a deleted
request no longer leaves events, and a new case pins that its id 404s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G

Copy link
Copy Markdown
Collaborator Author

Replies to the optionals in Cursor review 5358803060 that have no inline thread:

  • Dropping the E2E retention suite: declined. It is the real-path repro: action POST, then retention, then id reuse, then HTTP replay. The store-level cases can't show that retention is the path that frees the id. It now costs one scenario run per backend.
  • Filesystem delete adopting the SQL pendingNewEvents.delete(id) pattern: declined for this PR, fine as a follow-up. It isn't a one-line match: the filesystem store queues event appends on a per-request write queue and sweeps sidecars under the per-id lock, so the equivalent change would need eventWriteQueued touched as well. It also isn't needed for correctness here: the new conformance case "delete is not undone by events persisted just before it" already passes on filesystem.

Generated by Claude Code

Copy link
Copy Markdown
Collaborator Author

@codex review

The head is now c85197a:

  • Postgres now deletes the request record only after its events and runOnce results are gone.
  • The retention test harness disposes its router and waits for runs to finish before cleanup.
  • The filesystem store is back in the retention harness.

Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c85197aea7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ete waits out event appends (FIX-1647)

Postgres: Promise.all rejected on the first failed child DELETE while
its siblings were still running, so a retry could free the id and a
straggler then delete the next owner's rows. The child deletes now use
allSettled and the first failure is thrown only once all have settled;
the record is still deleted last.

Filesystem: an event append takes its batch as soon as it starts, and
delete swept the sidecars without waiting for it, so an in-flight
append could recreate the event log under the freed id. delete now
drops the unwritten batch and drains the id's event queue before the
sweep. A filesystem-specific test holds the append until the sweep has
listed the directory, which makes the old ordering fail every time.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G

Copy link
Copy Markdown
Collaborator Author

@codex review


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 61f18b8093

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/engine/README.md Resolved
…FIX-1647)

runAction marks a request completed before it persists the terminal
event and runs onFinished, and leaves the active registry last. Under
the default "allow" concurrency a sibling in the same session could
complete in that window and its retention pass would delete the
request, so the terminal event landed under the freed id for the id's
next owner to replay. Retention now skips any request still in the
active registry (one listAll read); the next pass evicts it. This sits
above the stores, so it covers every adapter with no schema change.

Docs: streaming.md no longer says events may outlive their request;
RequestStore.delete documents the full contract (every child, record
last, in-flight writes settled); README notes the new skip.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G

Copy link
Copy Markdown
Collaborator Author

@codex review


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 904ab8f503

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…cting it (FIX-1647)

Three review findings had one root: retention could delete a request
while work tied to its id was still live. The active-registry skip only
sees runs in a shared registry, and a live-tail stream can keep polling
a deleted request for up to the liveness timeout, long enough for a new
owner to take the id.

Retention now spares a request until twice the live-tail liveness
timeout has passed since it finished: one timeout for any stream still
tailing it to end, one more to bound the run's own tail. The window is
derived from LIVE_TAIL_LIVENESS_MS, now shared by the stream route and
retention. The registry skip stays as a cheap extra check. A spared
request is evicted when the session's next request completes; retention
stays lazy, and that is documented as the bound.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G

Copy link
Copy Markdown
Collaborator Author

@codex review


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

runOncePrefixes.some((prefix) => name.startsWith(prefix));

P1 Badge Match runOnce sidecars by an unambiguous request ID

Because request IDs are caller-supplied and encodeSegment does not escape periods, deleting request foo also matches a distinct request's sidecar such as foo.runonce.bar.runonce.step.json via the foo.runonce. prefix. Retention or explicit deletion of foo can therefore erase foo.runonce.bar's idempotency result, allowing its step and associated side effects to execute again; use a layout or encoding with an unambiguous request-ID boundary.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…y (FIX-1647)

Deleting request "foo" matched "foo.runonce.bar.runonce.step.json" through
the "foo.runonce." prefix and erased another request's idempotency result.
Per-key files are now named <id>@<key>.runonce: encodeSegment always escapes
@, so the first @ ends the id. Files in the older layout are still read, and
removed on delete only when no longer id could own them and they are not
another request's record.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
…zedAtMs (FIX-1647)

runAction marks a record completed and applies retention before it awaits
onFinished, which has no time limit, so no grace window bounds the run's
last writes. The run now stamps finalizedAtMs as its last write, through a
conditional setFieldsIfStatus that never recreates a deleted record, and
retention evicts only stamped records past the grace window. New records
start at null; records from before the field keep the grace-window rule.
The success path heartbeats through onFinished, and the stale-request sweep
stamps a terminal record whose run died before stamping it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
…into fix/fix-1647

# Conflicts:
#	packages/engine/src/stores/testing/request-store-conformance.ts
…(FIX-1647)

After merging main, setFieldsIfStatus takes expectedCreatedAt. The run's
stamp and the stale sweep's stamp now pass the createdAt of the record they
read, so neither can mark another request that took the id as finished.

The two-users-one-tenant retention cases shorten LIVE_TAIL_LIVENESS_MS
through a harness helper so their eviction waits fit the window; what they
assert is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G

Copy link
Copy Markdown
Collaborator Author

Re review 5359398722, the P1 on the filesystem runOnce sidecar prefix (filesystem/request-store.ts:529, no inline thread): confirmed and fixed in d70c31f.

The bug. Deleting "foo" matched foo.runonce.bar.runonce.step.json through the foo.runonce. prefix, which erased request foo.runonce.bar's idempotency result. The old name was ambiguous at write time too, because . is not escaped.

The fix.

  • Per-key files are now named <enc(id)>@<enc(key)>.runonce.
  • encodeSegment always escapes @, so the first @ ends the id, and the delete sweep's <enc(id)>@ prefix matches that id's files and no other's.
  • The new names don't end in .json, so they can't collide with record files either.

Files already on disk in the old layout (BP-030).

  • Reads fall back to the old per-key name, and then to the single-map file.
  • On delete, an old per-key file is removed only when nothing else could own it:
    • it is kept if the part after the prefix still contains .runonce., because a longer id could have written it;
    • it is kept if it reads back as the record of the request whose id is the whole file name.

Tests, red first.

  • New conformance case, run by all four stores, "delete leaves the runOnce results of an id that merely starts with it". It uses exactly foo and foo.runonce.bar. It failed on filesystem (expected { found: false } to deeply equal { Object (found, value) }); memory, SQLite and Postgres already passed.
  • Old-layout cases in test/stores/filesystem/runonce-per-key.test.ts:
    • "reads a result stored in the older per-key layout" passes on both heads, as a compatibility guard.
    • "delete removes the request's older per-key files" passes on both heads, as a compatibility guard.
    • "delete leaves an older per-key file a longer id could have written" failed on the old code.
    • "delete leaves another request's record that looks like an older per-key file" failed on the old code.

Not fixed here: exact-name collisions.

  • For a caller-chosen id ending in .events or .runonce, the record file is the same file as another id's event log or single-map file. For example, the record of foo.events is the event log of foo.
  • This is on main too; shared.ts already documents it as a known limit of isSidecarFile.
  • The only fix is to rename the event log or the record file, with a migration, so I've left it out of this PR.

Generated by Claude Code

Copy link
Copy Markdown
Collaborator Author

@codex review


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 13d6e6a4e7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copy link
Copy Markdown
Collaborator Author

@codex review


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 71a2e331bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/engine/src/stores/types.ts Resolved
…ves the record last (FIX-1647)

`<id>.runonce.json` and `<id>.events.json` are also the record files of
requests `<id>.runonce` and `<id>.events`. Deleting `<id>` now leaves such
a file when it reads back as that record, and the single-map runOnce
fallback never serves it as `<id>`'s results.

The shared filesystem record store's delete now sweeps sidecars before
removing the record, and the sweep settles every removal before
reporting a failure, so a failed cleanup leaves the id taken and the
delete retryable.

Docs: the custom-store live-tail recipe calls isStillAuthorized after
every non-empty read; the side-chain page describes onFinished/onErrored
work as drained before the run finishes but after the final event.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G

Copy link
Copy Markdown
Collaborator Author

@codex review


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a2715190cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…(FIX-1647)

The stale sweep can mark a slow but live run interrupted, and /continue
then starts a second run under the same id and incarnation. Either run
could stamp finalizedAtMs, or drop the shared active-registry entry,
while the other was still in onFinished or writing, so retention could
free the id under a live writer.

runActionInternal now counts each run as a live attempt of its request
id (per request store, in this process). A run stamps and deregisters
only when it is the last live attempt to end; the suspend path and the
continuation's safety-net deregister follow the same rule. An
overlapping run in another process is judged by its heartbeat, as the
sweep already does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G

Copy link
Copy Markdown
Collaborator Author

@codex review


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 58b6c4eaae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

* process (tests, multiple engines) never hold each other's requests open.
*/

const liveAttempts = new WeakMap<object, Map<string, number>>();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Fence finalization across processes

In a shared-store multi-process deployment, a stale sweep can start /continue on process B while the original attempt remains alive on process A. Because this WeakMap sees attempts only in one process, each process can consider its own run the last attempt; once both have written a terminal status, the first one to finish can stamp finalizedAtMs and deregister the shared active entry while the other remains in onFinished. Retention can then delete and reuse the ID after the grace period while the remote attempt still writes. This re-raises the prior per-attempt-fence issue with fresh evidence from its process-local implementation; finalization needs a durable cross-process attempt fence or equivalent proof that no remote attempt remains.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, this is real. It is deferred to FIX-1665 and not fixed in this PR.

Fencing runs across processes needs a durable per-attempt id on the request record and on the active-registry entry. That is a schema change in all four adapters. For now, a run in another process whose heartbeat went stale is treated as dead. That is the same rule the stale sweep already applies. The id-reuse class predates this PR, and retention is opt-in.

FIX-1665 also takes the abort-controller map, which is keyed by request id alone. I am leaving this thread open for it.


Generated by Claude Code

Comment on lines +618 to +619
if (exactJson.has(name)) {
if (!(await this.isRecordFile(name))) await rm(filePath, { force: true });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Serialize suffix-collision checks with request creation

When deleting request foo, isRecordFile("foo.events.json") can read the old event sidecar as a non-record, after which a concurrent caller creates request foo.events under its distinct per-ID lock and atomically replaces that same path before rm executes; the deletion then removes the new live request's record. This is fresh evidence beyond the prior suffix-collision finding because the added content check has a TOCTOU window. Namespace sidecars so paths cannot collide, or coordinate the check and removal with creation of the colliding ID.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, this is real. It is deferred to FIX-1665 and not fixed in this PR.

The record check and the delete are two separate filesystem steps, so a record written between them can still be deleted. The real fix is to namespace the filesystem sidecars so that a request id ending in .events or .runonce can never share a name with another request's sidecar. That needs a layout rename with a migration, and it goes together with the per-attempt id in FIX-1665.

The name collision itself predates this PR, and retention is opt-in. I am leaving this thread open for it.


Generated by Claude Code

…owing (FIX-1647)

A run that ends while another run of the same request is still live
leaves the stamp to it. If that last run then ended by throwing, it
never reached its own finalization, so the finished record stayed
unstamped and protected from retention forever. The declining run now
hands over its terminal record's incarnation, and runActionInternal's
exit stamps it (and deregisters) when the last attempt ends by throwing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G

Copy link
Copy Markdown
Collaborator Author

@codex review


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1df50872dd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…am to the guard-checked incarnation (FIX-1647)

A run that ended by throwing after writing a terminal record (a fresh
setup failure writes `failed`) never reached its finalization, so the
record stayed unstamped and retention kept it forever. The attempt now
tracks its own terminal incarnation, and runActionInternal's exit stamps
it (or one an earlier overlapping attempt left) and deregisters when it
is the last live attempt. A dispatch that terminates a request no run
will ever pick up stamps the record as it writes it.

The request stream re-read the record after the route guard checked it,
so a delete and reuse in between bound the stream to the new owner. The
guard now returns the record it checked, and the stream answers as for
an unknown request when the incarnation it reads differs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G

Copy link
Copy Markdown
Collaborator Author

@codex review


Generated by Claude Code

…persistence docs (FIX-1647)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 35435a0059

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

items,
// The timer runs on through `onFinished` until the stamp, so a stale
// entry from here means the run died in its tail. Not with heartbeats off.
heartbeatsUntilFinalized: heartbeatIntervalMs > 0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Gate finalization recovery on a safe heartbeat cadence

Fresh evidence beyond the earlier zero-heartbeat fix is that any positive interval sets this flag, although stale-request-sweeper.ts:13-14 requires staleThresholdMs >= 2 * heartbeatIntervalMs and the sweeper cannot inspect per-flow intervals. With a 120-second flow heartbeat and the default 60-second stale threshold, a request in a long onFinished is treated as dead and stamped before its next heartbeat; after the retention grace, its caller-supplied ID can be deleted and reused while the hook is still writing. Set this flag only when the effective stale threshold safely exceeds this flow's heartbeat cadence, or persist enough cadence information for recovery to verify it.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. This is deferred to FIX-1665 and not fixed here.

The run can't know the threshold it would need to check. The sweep that stamps a record may run in any process, with that host's own staleSweepThresholdMs. The run's runtimeConfig only sometimes carries its local host's threshold, and a check against that proves nothing about the sweeper's. Getting this right needs the cadence and the threshold compared where the sweep runs. One way is to persist the run's heartbeat interval on the record or registry entry and have the sweep require its threshold to be at least twice that interval. That needs new persisted fields and plumbing across hosts, which goes with the per-attempt work in FIX-1665.

On main, a cadence misconfigured against the stale threshold already misfires the sweep: live runs get marked interrupted. So this PR adds another consequence to an existing misconfiguration rather than creating a new failure mode. I'm leaving this thread open.


Generated by Claude Code

…e error; narrow side-chain drain docs (FIX-1647)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G

Copy link
Copy Markdown
Collaborator Author

@codex review


Generated by Claude Code

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Unknown error
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copy link
Copy Markdown
Collaborator Author

@codex review


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8fd1fdcf91

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…spended run skips it (FIX-1647)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
setFieldsIfStatus takes main's single (…, updatedAt, expectedIncarnation)
fence; FIX-1647's callers (finalization stamp, stale-sweep stamp,
conformance cases) move onto it and the createdAt placeholder goes. One
incarnation compare per store. The abort poll keeps both guards: closed
after terminal status (FIX-1647) and silent until the incarnation settles
(FIX-1654).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
Every final-status write now goes through main's settleRequestRecord,
which writes the action result with the status. FIX-1647's terminal
writes keep their fields (heartbeatsUntilFinalized) and take the written
record's incarnation from it for the finalization stamp; the settle and
patch helpers return that incarnation. A fresh setup failure's record
and an unenqueued request's terminal write carry both the result and
the stamp.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants