Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
e0f10e2
fix(stores): delete a request's events and runOnce results with its r…
claude Sep 29, 2026
c85197a
fix(stores): delete a request's record last; harness waits for the ru…
claude Sep 29, 2026
61f18b8
fix(stores): settle every child delete before failing; filesystem del…
claude Sep 29, 2026
904ab8f
fix(retention): do not evict a request whose run is still finishing (…
claude Sep 29, 2026
811f4a2
fix(retention): keep a finished request for a grace window before evi…
claude Sep 29, 2026
d70c31f
fix(filesystem): give per-key runOnce files an unambiguous id boundar…
claude Sep 29, 2026
f82387a
fix(retention): evict a request only after its run has stamped finali…
claude Sep 29, 2026
0fc2df2
Merge branch 'main' of https://github.com/fixpoint-labs/flow-state-de…
claude Sep 29, 2026
13d6e6a
fix(retention): fence the finalization stamp to the run's own record …
claude Sep 29, 2026
c183148
fix(retention): fence the stamp on incarnation and close the rollout …
claude Sep 29, 2026
48a8f4b
fix(filesystem): never serve a record-shaped legacy runOnce result; f…
claude Sep 30, 2026
2123d5e
fix(retention): sweep stamps only runs that heartbeat through their t…
claude Sep 30, 2026
71a2e33
fix(engine): fence live streams on the request incarnation; count kep…
claude Sep 30, 2026
a271519
fix(engine): filesystem delete keeps other requests' records and remo…
claude Sep 30, 2026
58b6c4e
fix(engine): only the last overlapping run of a request finalizes it …
claude Sep 30, 2026
1df5087
fix(engine): the last overlapping run stamps even when it ends by thr…
claude Sep 30, 2026
35435a0
fix(engine): stamp every terminal record a run leaves; fence the stre…
claude Sep 30, 2026
68e8f2f
docs: condense FIX-1647 changeset to one sentence; move details into …
claude Sep 30, 2026
8fd1fdc
fix(engine): filesystem delete forgets the id's unreported event writ…
claude Sep 30, 2026
eab30a4
docs(engine): scope the finish sequence to requests that finish; a su…
claude Sep 30, 2026
8c4619e
Merge origin/main into fix/fix-1647 (FIX-1654's incarnation fence)
claude Sep 30, 2026
f0567bd
Merge origin/main into fix/fix-1647 (FIX-1661's settle helper)
claude Sep 30, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(engine): fence live streams on the request incarnation; count kep…
…t requests toward maxItems (FIX-1647)

A live tail reads events after awaits for as long as a request runs, so
the id can be deleted and taken by another owner between reads.
SubscribeToEventsOptions gains isStillAuthorized; the stream route
passes a fence comparing the record's incarnation with the one it
authorized, and every store (memory, filesystem via pollEvents, SQLite
live tail, Postgres LISTEN) calls it after each non-empty read and
before yielding, ending the stream when it fails. The record-less
replay branch now also answers 404 if a record appears during its
read. The in-memory store's delete ends the request's live streams.

Retention: a request kept because it is still running or not yet
finalized now counts toward maxItems, so older evictable history is
evicted to make room instead of the session growing past the budget.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNTQkGXCUryuVD2TEBLi6G
  • Loading branch information
claude committed Sep 30, 2026
commit 71a2e331bc368e8f98bf033372b964389cb455be
2 changes: 1 addition & 1 deletion .changeset/request-delete-removes-events.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@
"@flow-state-dev/store-postgres": patch
---

Deleting a request, including through session retention, now also deletes its stream events and runOnce results, so a later request that reuses the id cannot replay the earlier run; retention also keeps a request until its run has finished, `onFinished` included, and for twice the live-tail liveness timeout (60s by default) after, so a session can briefly exceed its limits. Request records gain `finalizedAtMs`, which the run sets as its last write, and `heartbeatsUntilFinalized`; a custom `RequestStore` must persist both. With heartbeats off, a request whose run dies before setting `finalizedAtMs` is kept rather than evicted. `RequestStore.setFieldsIfStatus` gains an optional sixth argument, `expectedIncarnation`, which a custom store must honor: a record whose incarnation differs is reported as absent and left unwritten. Records from versions that never set `finalizedAtMs` are kept for the stale-request threshold or `maxAge`, whichever is larger, so a rolling deploy is safe. The filesystem store names per-key runOnce files `<id>@<key>.runonce`, so deleting one request can no longer remove another's results; files in the older layout are still read (except one that reads back as a request record, which may be one), and on delete one whose name another id could also have written is renamed aside (`.quarantined`) so no later request reads it (FIX-1647).
Deleting a request, including through session retention, now also deletes its stream events and runOnce results, so a later request that reuses the id cannot replay the earlier run; retention also keeps a request until its run has finished, `onFinished` included, and for twice the live-tail liveness timeout (60s by default) after, so a session can briefly exceed its limits (a request it keeps still counts toward `maxItems`, so older history is evicted in its place). Request records gain `finalizedAtMs`, which the run sets as its last write, and `heartbeatsUntilFinalized`; a custom `RequestStore` must persist both. With heartbeats off, a request whose run dies before setting `finalizedAtMs` is kept rather than evicted. `RequestStore.setFieldsIfStatus` gains an optional sixth argument, `expectedIncarnation`, which a custom store must honor: a record whose incarnation differs is reported as absent and left unwritten. Records from versions that never set `finalizedAtMs` are kept for the stale-request threshold or `maxAge`, whichever is larger, so a rolling deploy is safe. The filesystem store names per-key runOnce files `<id>@<key>.runonce`, so deleting one request can no longer remove another's results; files in the older layout are still read (except one that reads back as a request record, which may be one), and on delete one whose name another id could also have written is renamed aside (`.quarantined`) so no later request reads it. `SubscribeToEventsOptions` gains `isStillAuthorized`, which the request stream route passes to fence a live tail on the request's incarnation; a custom store must call it after every read that returned events and before yielding them, ending the stream when it fails (`isBatchStillAuthorized` is exported for this). The in-memory store's `delete` now ends the request's live streams (FIX-1647).
2 changes: 1 addition & 1 deletion docs/architecture/server-and-client.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ defineFlow({
- Eviction is lazy (runs after each completed request). No background process.
- Operates at **request granularity** — entire old requests are removed, not individual items.
- The current request is never evicted. Failed requests are not eviction candidates.
- A request is not evicted until its run has finished, `onFinished` included, and twice the live-tail liveness timeout (`LIVE_TAIL_LIVENESS_MS`, 30s by default, so 60s) has passed since. Its record turns terminal before its run finishes writing, and a live stream may still be following it; evicting it earlier frees a caller-supplied id while either is active. The run marks its record `finalizedAtMs` as its last write, after `.sideChain()` work its hooks queued has settled, and retention waits for that mark rather than for a length of time, because `onFinished` has no time limit and another process's run is invisible to this one. The mark is fenced on the record's `incarnation`, so it never lands on a request that took the id since. A run that dies before marking it, or whose late writes failed to flush, or whose mark the store refused three times, is marked by the stale-request sweep, but only when the run was heartbeating through its tail (a completed run with heartbeats on): with heartbeats off (`request.heartbeatIntervalMs: 0`) a stale entry proves nothing, so such a record is marked only by its own run, and is kept for good if that run is gone. With the sweep disabled, such a request is kept too. **Rollout safety:** a record with no `finalizedAtMs` field at all was written by a version that never marks, and during a rolling deploy its run may still be finishing on an old instance. It is kept until the stale-request threshold (60s, or the host's `staleSweepThresholdMs` if longer) or `maxAge`, whichever is larger, plus the window, has passed since it completed. A request spared this way is evicted when the session's next request completes, so a session can sit above its limits until then.
- A request is not evicted until its run has finished, `onFinished` included, and twice the live-tail liveness timeout (`LIVE_TAIL_LIVENESS_MS`, 30s by default, so 60s) has passed since. Its record turns terminal before its run finishes writing, and a live stream may still be following it; evicting it earlier frees a caller-supplied id while either is active. The run marks its record `finalizedAtMs` as its last write, after `.sideChain()` work its hooks queued has settled, and retention waits for that mark rather than for a length of time, because `onFinished` has no time limit and another process's run is invisible to this one. The mark is fenced on the record's `incarnation`, so it never lands on a request that took the id since. A run that dies before marking it, or whose late writes failed to flush, or whose mark the store refused three times, is marked by the stale-request sweep, but only when the run was heartbeating through its tail (a completed run with heartbeats on): with heartbeats off (`request.heartbeatIntervalMs: 0`) a stale entry proves nothing, so such a record is marked only by its own run, and is kept for good if that run is gone. With the sweep disabled, such a request is kept too. **Rollout safety:** a record with no `finalizedAtMs` field at all was written by a version that never marks, and during a rolling deploy its run may still be finishing on an old instance. It is kept until the stale-request threshold (60s, or the host's `staleSweepThresholdMs` if longer) or `maxAge`, whichever is larger, plus the window, has passed since it completed. A request spared this way is evicted when the session's next request completes, so a session can sit above its limits until then. Its items still count toward `maxItems` while it waits, so older history is evicted to make room for them.
- For items that should never be stored, use `transient: true` on block definitions instead.

## Client Setup
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture/streaming.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ Because events are operationally independent from items, they can be:
- Pruned earlier than the items they describe (e.g., capped collection, age-based pruning)
- Disabled entirely in production without affecting app behavior

Their lifetime is bounded by the request's, though. `RequestStore.delete(id)` must remove every child stored under that id, events and runOnce results included, before it removes the record. Request ids can be caller-supplied, so session retention frees an id that a later request, possibly another user's, may take. Any events left under it would replay into that request's stream. For the same reason, retention does not evict a request until its run has recorded that it finished (`finalizedAtMs`, written after `onFinished`) and twice the live-tail liveness timeout has passed since. The run's terminal event and `onFinished` come after its record turns `completed`, and a live-tail stream following the request would otherwise keep polling a freed id for up to the liveness timeout. It also skips a request still in the active registry. A terminal replay reads the record, checks the caller owns it, then reads the events; if the record under the id is no longer the same incarnation after that read, the route answers as for an unknown request rather than replay events that may be the next owner's.
Their lifetime is bounded by the request's, though. `RequestStore.delete(id)` must remove every child stored under that id, events and runOnce results included, before it removes the record. Request ids can be caller-supplied, so session retention frees an id that a later request, possibly another user's, may take. Any events left under it would replay into that request's stream. For the same reason, retention does not evict a request until its run has recorded that it finished (`finalizedAtMs`, written after `onFinished`) and twice the live-tail liveness timeout has passed since. The run's terminal event and `onFinished` come after its record turns `completed`, and a live-tail stream following the request would otherwise keep polling a freed id for up to the liveness timeout. It also skips a request still in the active registry. A terminal replay reads the record, checks the caller owns it, then reads the events; if the record under the id is no longer the same incarnation after that read, the route answers as for an unknown request rather than replay events that may be the next owner's. A live tail reads for as long as the request runs, so the route passes `subscribeToEvents` an `isStillAuthorized` fence that compares the record's incarnation with the one it authorized. Every store calls it after each read that returned events and before yielding any of them; when it fails, the stream yields none of that batch and ends as for an unknown request. A custom `RequestStore` must honor it (`pollEvents` and `isBatchStillAuthorized` already do). The fence costs one record read per non-empty batch. The in-memory store also ends a request's live streams when the request is deleted.

This separation means observability-only item types (like `state_snapshot`) should use `transient: true` — they flow through the event stream for live and replay consumption without bloating the persisted item record.

Expand Down
30 changes: 20 additions & 10 deletions packages/engine/src/execution/retention.ts
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,10 @@ export async function applyRetentionPolicy(
// check. Skipped requests are evicted by a later pass, which runs when the
// session's next request completes: retention is lazy, and a session nothing
// is written to is not growing.
//
// Only the deletion of a protected request waits. Its items are still
// history the session holds, so they count toward `maxItems` below, and
// evictable history makes room for them.
const graceCutoff = now - (policy.terminalGraceMs ?? 0);
const legacyCutoff = now - (policy.legacyGraceMs ?? policy.terminalGraceMs ?? 0);
const stillRunning = new Set(
Expand All @@ -131,11 +135,12 @@ export async function applyRetentionPolicy(
};

// Exclude current request, sort oldest-first by completion time
const sorted = requests
.filter((r) => {
if (r.id === currentRequestId || stillRunning.has(r.id)) return false;
return isEvictable(r);
})
const history = requests.filter((r) => r.id !== currentRequestId);
const isProtected = (r: RequestRecord): boolean =>
stillRunning.has(r.id) || !isEvictable(r);
const protectedIds = history.filter(isProtected).map((r) => r.id);
const sorted = history
.filter((r) => !isProtected(r))
.sort(
(a, b) =>
(a.completedAtMs ?? a.startedAtMs) - (b.completedAtMs ?? b.startedAtMs)
Expand Down Expand Up @@ -166,10 +171,14 @@ export async function applyRetentionPolicy(
if (policy.maxItems !== undefined) {
const maxItems = policy.maxItems;
// Count in bounded batches so attacker-grown histories cannot enqueue an
// unbounded burst of database work. The current request is always kept;
// count it too.
// unbounded burst of database work. The current request and protected
// requests are always kept; count them too.
const newestFirst = [...remaining].reverse();
const requestIds = [currentRequestId, ...newestFirst.map((req) => req.id)];
const requestIds = [
currentRequestId,
...protectedIds,
...newestFirst.map((req) => req.id)
];
const counts: number[] = [];
for (let i = 0; i < requestIds.length; i += RETENTION_COUNT_BATCH_SIZE) {
counts.push(
Expand All @@ -180,10 +189,11 @@ export async function applyRetentionPolicy(
))
);
}
const [currentCount = 0, ...historyCounts] = counts;
const keptCounts = counts.slice(0, 1 + protectedIds.length);
const historyCounts = counts.slice(1 + protectedIds.length);

// Walk newest-first, accumulating items until budget is exceeded
let totalItems = currentCount;
let totalItems = keptCounts.reduce((sum, n) => sum + n, 0);
const keep = new Set<string>();
newestFirst.forEach((req, i) => {
const reqItemCount = historyCounts[i] ?? 0;
Expand Down
1 change: 1 addition & 0 deletions packages/engine/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@ export {
synthesizeRequestInterrupted,
pollEvents,
abortableSleep,
isBatchStillAuthorized,
type ReadEventsFn
} from "./stores/subscribe-helpers";
export {
Expand Down
38 changes: 34 additions & 4 deletions packages/engine/src/routes/stream-routes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,22 @@ import {
import type { ParsedFlowRoute } from "./parseFlowRoute";
import { resolveLiveTailLivenessMs } from "../streaming/live-tail-liveness";

/**
* Whether the record under `requestId` is still the request whose incarnation
* the route authorized. A request id is the caller's to choose, so once the
* route has checked who owns the record, any read it makes after an await may
* find the request deleted and the id taken by another owner. Incarnations
* are never reused, so a match proves the id has not changed hands since.
*/
async function stillHoldsIncarnation(
stores: StoreRegistry,
requestId: string,
incarnation: string
): Promise<boolean> {
const current = await stores.request.get(requestId);
return current !== undefined && resolveRequestIncarnation(current) === incarnation;
}

type StreamRouteContext = {
registry: FlowRegistry;
stores: StoreRegistry;
Expand Down Expand Up @@ -134,6 +150,7 @@ export async function handleRequestStream(
});

const fromSequence = cursor.sequenceNumber ?? 0;
const authorizedIncarnation = resolveRequestIncarnation(requestRecord);
Comment thread
jhoffner marked this conversation as resolved.
Resolved
const shouldForward = includeTrace ? undefined : createClientEventFilter();
const handle = createSSEStream({
pingIntervalMs: sseHeartbeatMs,
Expand All @@ -148,7 +165,12 @@ export async function handleRequestStream(
livenessTimeoutMs: resolveLiveTailLivenessMs(),
// While a continuation lease is held, follow through `request.suspended`
// (the run-1 suspension being continued past) instead of ending there.
followThroughSuspend: leaseHeld
followThroughSuspend: leaseHeld,
// Every batch the store reads after an await is yielded only while the
// id still holds the request authorized above; otherwise the stream
// ends as for a request that is gone.
isStillAuthorized: () =>
stillHoldsIncarnation(ctx.stores, route.requestId, authorizedIncarnation)
}
);

Expand Down Expand Up @@ -185,6 +207,12 @@ export async function handleRequestStream(
if (events.length === 0) {
return unknownRequestStreamResponse(request, route, ctx.registry);
}
// No record was found before this read. If one exists now, a request
// took the id meanwhile and these may be its events, which nothing here
// has authorized; answer as for an unknown request.
if ((await ctx.stores.request.get(route.requestId)) !== undefined) {
return unknownRequestStreamResponse(request, route, ctx.registry);
}
let replay = replayRequestEvents({
requestId: route.requestId,
events,
Expand Down Expand Up @@ -232,10 +260,12 @@ export async function handleRequestStream(
// take the id, so these may be the new request's events. Replay only if
// the record under the id is still the one that was checked; otherwise
// answer as for a request that is gone.
const recheck = await ctx.stores.request.get(route.requestId);
if (
recheck === undefined ||
resolveRequestIncarnation(recheck) !== resolveRequestIncarnation(requestRecord)
!(await stillHoldsIncarnation(
ctx.stores,
route.requestId,
resolveRequestIncarnation(requestRecord)
))
) {
return unknownRequestStreamResponse(request, route, ctx.registry);
}
Expand Down
Loading
Loading