Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
e0f10e2
fix(stores): delete a request's events and runOnce results with its r…
claude Sep 29, 2026
c85197a
fix(stores): delete a request's record last; harness waits for the ru…
claude Sep 29, 2026
61f18b8
fix(stores): settle every child delete before failing; filesystem del…
claude Sep 29, 2026
904ab8f
fix(retention): do not evict a request whose run is still finishing (…
claude Sep 29, 2026
811f4a2
fix(retention): keep a finished request for a grace window before evi…
claude Sep 29, 2026
d70c31f
fix(filesystem): give per-key runOnce files an unambiguous id boundar…
claude Sep 29, 2026
f82387a
fix(retention): evict a request only after its run has stamped finali…
claude Sep 29, 2026
0fc2df2
Merge branch 'main' of https://github.com/fixpoint-labs/flow-state-de…
claude Sep 29, 2026
13d6e6a
fix(retention): fence the finalization stamp to the run's own record …
claude Sep 29, 2026
c183148
fix(retention): fence the stamp on incarnation and close the rollout …
claude Sep 29, 2026
48a8f4b
fix(filesystem): never serve a record-shaped legacy runOnce result; f…
claude Sep 30, 2026
2123d5e
fix(retention): sweep stamps only runs that heartbeat through their t…
claude Sep 30, 2026
71a2e33
fix(engine): fence live streams on the request incarnation; count kep…
claude Sep 30, 2026
a271519
fix(engine): filesystem delete keeps other requests' records and remo…
claude Sep 30, 2026
58b6c4e
fix(engine): only the last overlapping run of a request finalizes it …
claude Sep 30, 2026
1df5087
fix(engine): the last overlapping run stamps even when it ends by thr…
claude Sep 30, 2026
35435a0
fix(engine): stamp every terminal record a run leaves; fence the stre…
claude Sep 30, 2026
68e8f2f
docs: condense FIX-1647 changeset to one sentence; move details into …
claude Sep 30, 2026
8fd1fdc
fix(engine): filesystem delete forgets the id's unreported event writ…
claude Sep 30, 2026
eab30a4
docs(engine): scope the finish sequence to requests that finish; a su…
claude Sep 30, 2026
8c4619e
Merge origin/main into fix/fix-1647 (FIX-1654's incarnation fence)
claude Sep 30, 2026
f0567bd
Merge origin/main into fix/fix-1647 (FIX-1661's settle helper)
claude Sep 30, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/request-delete-removes-events.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"@flow-state-dev/engine": minor
"@flow-state-dev/store-sqlite": patch
"@flow-state-dev/store-postgres": patch
---

Deleting a request (including through session retention) now also removes its events and runOnce results once its run has finished, and custom `RequestStore`s must persist `finalizedAtMs`/`heartbeatsUntilFinalized`, honor `setFieldsIfStatus`'s new `expectedIncarnation` argument, and call `isStillAuthorized` in live tails (FIX-1647).
8 changes: 4 additions & 4 deletions apps/docs/docs/advanced/sequencer-side-chains.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,19 +67,19 @@ If you need to know whether background work succeeded, use `.waitForSideChain()`

## Side chains are request-scoped

Background work outlives the sequencer that dispatched it. `.sideChain()`, `.sideChainIf()`, and `.forEachSideChain()` queue tasks on a single per-request pool. Inner sequencers do not block their parent on their own background work, and sibling sequencers run their `.sideChain()` tasks concurrently. The request executor drains the pool before the SSE stream closes — your stream stays open until every queued task settles, regardless of which sequencer queued it.
Background work outlives the sequencer that dispatched it. `.sideChain()`, `.sideChainIf()`, and `.forEachSideChain()` queue tasks on a single per-request pool. Inner sequencers do not block their parent on their own background work, and sibling sequencers run their `.sideChain()` tasks concurrently. The request executor drains the pool before the request's final event — your stream stays open until every task queued by then settles, regardless of which sequencer queued it. Work queued later, from `onFinished` or `onErrored`, is covered below.

The drain runs until the pool is empty, so a background task that queues more background work is waited on too, however deep it nests.

### The guarantee holds however the request ends

The request is not reported finished until its background work has settled — whether it succeeded or not. If the action throws, if the user hits stop, or if the client disconnects mid-stream, the request still waits for its queued tasks before the terminal record is written and the stream closes.
The request is not reported finished until the background work queued before its terminal hooks has settled — whether it succeeded or not. If the action throws, if the user hits stop, or if the client disconnects mid-stream, the request still waits for those tasks before the terminal record is written and the stream closes.

That matters most on ephemeral hosts (Vercel, Next.js `after()`, Lambda), where the platform is free to freeze the container the moment the request returns. A task still running at that point can be cut off with a write half-applied. Because the request holds itself open until the pool settles, a `.sideChain()` task that captures memory or writes a summary gets to finish even on a turn the user cancelled.

Two gaps are worth knowing about.
Two timing details are worth knowing about.

`onFinished` and `onErrored` run *after* the terminal record is written, so background work they queue is not covered — it lands in a pool nothing is waiting on. If one of those hooks needs to do durable work, do it inline in the hook rather than dispatching it with `.sideChain()`. `onCompleted` is different: it runs before the request is finalized, so work it queues is drained like any other.
`onFinished` and `onErrored` run *after* the request's final event has gone out on the stream. Background work they queue is still waited on: the run drains it before it counts as finished, and the request's id is not freed for reuse until it settles. But the client has already seen the request end by then, and its stream may have closed. If the client needs to see a task's effect when the request ends, queue it from `onCompleted` instead, which runs before the final event.
Comment thread
jhoffner marked this conversation as resolved.
Resolved

**Suspension is not an ending, and does not wait.** A request that suspends at a gate closes its stream with background work still in flight, by design — see [Suspension does not wait for background work](#suspension-does-not-wait-for-background-work) below. The guarantee above is about a request *finishing*; a suspended request has not finished, it is paused.

Expand Down
2 changes: 1 addition & 1 deletion apps/docs/docs/configuration/flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ The host verifies credentials. The framework applies `defaultUserId` and `requir
| `stateSchema` | Zod object | — | Session state shape. Use `.nullable().default(null)` for fields that start empty. |
| `client` | `{ expose?, derived? }` | private | What crosses to the browser under `clientData.session`. `expose` copies named fields verbatim. `derived` computes named projections from `{ state, resources }`. Names must not collide. |
| `metadata` | Zod schema | — | Declares the session metadata shape (`title`, tags, and so on) for typing. Not enforced at runtime today: neither the session-metadata route nor `ctx.session.setMetadata()` parses against it, so a value outside the schema is persisted unchanged. |
| `retention` | `{ maxItems?, maxAge? }` | unbounded | Bounds the persisted item log. `maxAge` is milliseconds or a duration string (`"7d"`). Oldest completed requests evict first. |
| `retention` | `{ maxItems?, maxAge? }` | unbounded | Bounds the persisted item log. `maxAge` is milliseconds or a duration string (`"7d"`). Oldest completed requests evict first. A request stays until its run has finished, `onFinished` and any background work it started included, and for twice the configured `LIVE_TAIL_LIVENESS_MS` after (one minute by default), so a busy session can briefly exceed its limits; a request kept this way still counts toward `maxItems`, so older history is evicted in its place. Deleting a request also removes its stream events and cached step results. See [Deleting requests](/docs/persistence/overview#deleting-requests) for the edge cases and what a custom store must support. |
| `historyWindow` | `{ turns: number }` | `50` | Caps cross-turn history loaded per request. `0` or a negative number disables it. Per-call `history({ limit })` can only shrink this window. |
| `cas` | `CASOptions` | — | Optimistic-concurrency options for this scope. |

Expand Down
34 changes: 33 additions & 1 deletion apps/docs/docs/persistence/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ const flowstate = createFlowState({

The directory layout has two halves. **Scope records** sit at the root, one directory per store: `sessions/`, `users/`, `requests/`, and `projects/`. That last one holds org records — the directory name predates the `project` → `org` scope rename and is kept as-is so existing data keeps resolving. **Resource content and state** sit under `content/` and `state/`, and there each scope gets a subdirectory named for it: `session/`, `user/`, `org/`. So a resource key like `notes/meeting` on session `s1` lands at `content/session/s1/notes/meeting.md`.

Each request's cached step results (runOnce) are stored beside its record as one file per key, named `<requestId>@<key>.runonce`. Files written by older versions, `<requestId>.runonce.<key>.json`, are still read. When a request is deleted, an older-layout file whose name another request id could also have produced is renamed with a `.quarantined` suffix instead of being removed, so no later request reads it; you can delete those files once no run from before the upgrade is still in flight.

### Postgres

For production and multi-instance fleets. Stores state in PostgreSQL with `LISTEN/NOTIFY` for cross-process live tail.
Expand Down Expand Up @@ -315,6 +317,17 @@ A custom store also has to honor session parentage. A session can belong to anot

Omitting `parentage` narrows to `"top-level"`, which is the reverse of how an omitted `tenantId` behaves in the same object. A store that ignores the field hands back child sessions to a caller that asked for top-level ones, and nothing in the type system catches it — the field is optional. Treat a parent id of `null` the same as an absent one, and conjoin `parentage` with the other filters rather than letting it widen past them.

### Deleting requests

Request ids can be chosen by the caller, so an id freed by a delete (session retention included) can be taken by another request, possibly another user's. `RequestStore.delete(id)` therefore removes everything stored under the id, its items, stream events and runOnce results, before the record itself. Remove the record last: if a child delete fails, the record keeps the id taken and the delete can be retried. Writes still queued for the id must not land after the delete.

Retention only deletes a request once its run has marked the record finished, so a custom store also has to:

- persist the record's `finalizedAtMs` and `heartbeatsUntilFinalized` fields;
- honor `setFieldsIfStatus`'s `expectedIncarnation` argument. When the stored record's incarnation differs, report it as absent and leave it unwritten.

Two cases keep a finished request longer than the retention window. A run that dies before marking its record is released by the stale-request sweeper, but only when the run was heartbeating; with heartbeats off (`request.heartbeatIntervalMs: 0`) or the sweeper disabled, its request is kept. And a record written by an engine version that never marked records is kept for the stale-request threshold or `maxAge`, whichever is larger, so a rolling deploy cannot free the id of a run still finishing on an old instance.

### Live tail

`RequestStore` exposes `subscribeToEvents(requestId, options)` so the SSE wire can serve a request started on any instance. Stores choose their own delivery mechanism:
Expand All @@ -323,6 +336,25 @@ Omitting `parentage` narrows to `"top-level"`, which is the reverse of how an om
- SQLite runs one shared poll loop per request, fanned out to every subscriber and woken in-process by the write path; the filesystem store polls `getEvents(requestId, fromSequence)`.
- Postgres uses `LISTEN/NOTIFY` on a dedicated client. See `@flow-state-dev/store-postgres` for details.

Request ids can be chosen by the caller, so while a stream is open its request can be deleted and a different request, possibly another user's, can take the same id. `options.isStillAuthorized` guards against that. Whatever mechanism your store uses, call it after every read that returned events and before yielding any of them. If it resolves `false` or throws, yield none of that batch and end the iterator. `isBatchStillAuthorized(options)` from `@flow-state-dev/engine` does the call and treats a throw as `false`, and `pollEvents` already applies it for you.

```ts
async function* subscribeToEvents(requestId, options) {
let lastSeen = options.fromSequence;
while (!options.signal?.aborted) {
const batch = await readEventsAfter(requestId, lastSeen);
if (batch.length > 0) {
if (!(await isBatchStillAuthorized(options))) return;
for (const event of batch) {
yield event;
lastSeen = event.sequence_number;
}
}
await waitForMoreEvents(requestId, options.signal);
}
}
```

### Incremental items storage

Backed adapters store items incrementally rather than inside the request record's JSONB column. The SQLite and Postgres adapters write one row per item into a dedicated child table (`request_items`); the filesystem store appends items and events to an append-only log instead. `RequestStore.persistItems` and `get` keep the same shape either way, so a flow author sees no difference. An operator does — see [Persistence cost model](/docs/server/setup#persistence-cost-model) for how each backend stores its data.
Expand All @@ -333,7 +365,7 @@ Backed adapters store items incrementally rather than inside the request record'

See the [`@flow-state-dev/store-postgres` README](https://github.com/fixpoint-labs/flow-state-dev/blob/main/packages/store-postgres/README.md#items-storage) for the schema, the optional storage-reclamation steps (`pg_repack`), and the rollback constraints.

`getEvents` accepts an optional `fromSequence` for cursor reads — omitting it returns the full log (used by completed-request replay). A custom store that doesn't need cross-process tail can implement `subscribeToEvents` as an iterator that yields the catch-up via `getEvents` and then ends; clients fall back to bulk replay for completed requests.
`getEvents` accepts an optional `fromSequence` for cursor reads — omitting it returns the full log (used by completed-request replay). A custom store that doesn't need cross-process tail can implement `subscribeToEvents` as an iterator that yields the catch-up via `getEvents` and then ends; clients fall back to bulk replay for completed requests. The catch-up read still calls `isStillAuthorized` before yielding, like any other read.

The exact interface may evolve. Check the `@flow-state-dev/engine` package source for the current contract.

Expand Down
17 changes: 13 additions & 4 deletions docs/architecture/execution-and-errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -290,22 +290,31 @@ The full request execution sequence:
5. Create request scope
6. Emit user message item (if userMessage defined)
7. Fire request.onStarted
8. Execute action root block
8. Execute action root block, then wait for its .sideChain() work
├─ Suspended (ctx.suspend()): persist status suspended, emit it, return.
│ Steps below do not run; the request resumes under the same id.
├─ Success path:
│ ├─ Fire action.onCompleted
│ ├─ Fire request.onCompleted
│ ├─ Wait for .sideChain() work those hooks queued
│ ├─ Persist the terminal record, emit the terminal stream status
│ ├─ Apply session retention (completed requests only)
│ └─ Fire request.onFinished
└─ Error path:
├─ Persist the terminal record, emit the terminal stream status
├─ Fire action.onErrored
├─ Fire request.onErrored
└─ Fire request.onFinished
9. Persist state, emit terminal stream status
9. Wait for .sideChain() work the terminal hooks queued, then stamp the
record's finalizedAtMs (its last write) and leave the active registry
Comment thread
jhoffner marked this conversation as resolved.
Resolved
```

**Guarantees:**
**Guarantees** (for a request that finishes; see the suspended note below):
- `onCompleted` fires only on terminal success
- `onErrored` fires only on terminal failure
- `onFinished` fires always
- `onFinished` fires on every finished request, after the terminal record is persisted and the terminal stream status is emitted
- `finalizedAtMs` is written only after `onFinished` and any `.sideChain()` work it queued have settled; session retention frees a request's id only after that
- A run that calls `ctx.suspend()` ends as `suspended` instead: it fires no `onCompleted`, `onErrored` or `onFinished`, drains no terminal-hook work and writes no `finalizedAtMs`, because the id stays resumable. Session retention evicts only completed requests, so it keeps a suspended one.
- `onStepErrored` fires for non-terminal step/side-chain failures (visibility hook)
- A setup failure after the request has been accepted (`onRegistered`) settles a **fresh** request as `failed` with an `error` item. The HTTP 202 path awaits acceptance, not `finished`; leaving the row `in_progress` (or never writing one) is a silent hang. The failed record is written before `request.failed` is published, and the write leaves a row this run does not own untouched. Replay continuations still stay `suspended` / `interrupted` so they remain re-attemptable. Request observers (`onErrored` / `onFinished`) do not run on this path — there is no execution context yet.

Expand Down
6 changes: 4 additions & 2 deletions docs/architecture/flows-and-actions.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,9 +167,11 @@ When an action is invoked, the framework executes this sequence:
7. Emit user message item (if `userMessage` defined)
8. Fire `request.onStarted`
9. Execute action root block via `block.run(input, ctx)`
10. Fire action + request completion/error hooks
10. On success, fire action + request `onCompleted`, then persist the terminal record and emit the terminal stream status. On failure, persist and emit first, then fire action + request `onErrored`
11. Fire `request.onFinished`
12. Persist state and emit terminal stream status
12. Wait for `.sideChain()` work the hooks queued, then stamp the record's `finalizedAtMs` as its last write (see [Streaming](./streaming.md))

Steps 10–12 apply to a request that finishes. A run that calls `ctx.suspend()` ends as `suspended` instead: it fires no `onCompleted`, `onErrored` or `onFinished`, drains no terminal-hook work and writes no `finalizedAtMs`, because the id stays resumable. Session retention evicts only completed requests, so it keeps a suspended one.

Retry, continue and resume re-enter the instance recorded as the request's owner, and interrupted-request recovery groups entries by owner; the `:flowKind` path segment on those routes must equal it.

Expand Down
4 changes: 2 additions & 2 deletions docs/architecture/sequencer-dsl.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,7 @@ pipeline.forEachSideChain((item, index, ctx) => {
|--------|---------|--------|
| `concurrency` | 16 | Maximum number of iterations running simultaneously |

**Lifecycle:** The whole batch is queued on the per-request work pool (same as `.sideChain()`). The request executor drains it before terminal status; inner sequencers do not block. Parent flow cancellation cancels in-flight iterations via the abort signal. See FIX-554.
**Lifecycle:** The whole batch is queued on the per-request work pool (same as `.sideChain()`). Work queued before the terminal hooks is drained before terminal status; inner sequencers do not block. Work queued from `onFinished`/`onErrored` is drained after the final event, before the request's id can be freed. Parent flow cancellation cancels in-flight iterations via the abort signal. See FIX-554.

### `doUntil(condition, block)` — Loop Until True

Expand Down Expand Up @@ -285,7 +285,7 @@ pipeline.sideChain(

**Key:** Work failures do NOT abort the main chain. They are logged and surface on the DevTool's trace channel.

**Lifetime — request-scoped pool (FIX-554):** Background work is queued on a single per-request pool, not the sequencer that dispatched it. Inner sequencers do not block their parent on their own background work. The request executor drains the pool to quiescence before terminal status, on every terminal path including `failed` / `aborted` / `interrupted` (FIX-1001); the SSE stream stays open until the drain completes. As tasks settle, the executor emits a `StatusItem` with `blocked: false` and `sideChainTasks: N` — clients use `blocked` to know it's safe to accept new user input (see `isFinishing` on `SessionView` / `UseRequestStreamResult`). When you need a downstream step to read state mutated by a queued task, use `.waitForSideChain()` as an explicit barrier in the dispatching sequencer.
**Lifetime — request-scoped pool (FIX-554):** Background work is queued on a single per-request pool, not the sequencer that dispatched it. Inner sequencers do not block their parent on their own background work. The request executor drains work queued before the terminal hooks to quiescence before terminal status, on every terminal path including `failed` / `aborted` / `interrupted` (FIX-1001); the SSE stream stays open until that drain completes. Work queued from `onFinished`/`onErrored` runs after the final event: the executor drains it before the run counts as finished and before the request's id can be freed, but the stream may already have closed. A request that suspends (`ctx.suspend()`) reaches none of this: it runs no terminal hooks and is not finalized, since it resumes under the same id. As tasks settle, the executor emits a `StatusItem` with `blocked: false` and `sideChainTasks: N` — clients use `blocked` to know it's safe to accept new user input (see `isFinishing` on `SessionView` / `UseRequestStreamResult`). When you need a downstream step to read state mutated by a queued task, use `.waitForSideChain()` as an explicit barrier in the dispatching sequencer.

### `sideChainIf(condition, block)` — Conditional Background Work

Expand Down
Loading
Loading