From d3eaa00491153d2672eb3c56f47e64ca4b417cf4 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 3 Nov 2026 02:57:35 +0000 Subject: [PATCH 1/5] FIX-2427: the Postgres bus, so a change on one process reaches views on another postgresStores backs the bus slot. A notice reaches the writing process's views at once and goes out as pg_notify('fsd_bus'); one LISTEN connection per process with an open view, from the tail pool, fans notices out. The bus says it crosses processes only while it hears its own ping (every 2 s, stale after 3 s), so behind a pooler or after a drop views keep the timed read; it reconnects without giving up, resyncs, and warns once when the first ping never comes back. Request tails share the new listen helper. Engine cleanups promised on #4594: one bus per wrapped store (a second is refused), one deferral hop and one microtask per publish, and notice-assembly failures debug-logged instead of swallowed. Goal check goals/session-stream/wakes-across-processes/ on two forked processes over a real Postgres, with goals/lib/postgres.mts and peers.mts shared with the closure goal (FIX-2368). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn --- .changeset/postgres-bus.md | 6 + apps/docs/docs/persistence/overview.md | 6 +- docs/mechanisms/mechanisms.yaml | 1 + goals/README.md | 2 + goals/lib/peers.mts | 134 +++++++ goals/lib/postgres.mts | 160 ++++++++ goals/package.json | 2 + .../wakes-across-processes/goal.md | 27 ++ .../wakes-across-processes/peer.mts | 217 ++++++++++ .../wakes-across-processes/run.mts | 247 ++++++++++++ packages/engine/README.md | 2 +- packages/engine/src/bus/in-memory-bus.ts | 23 +- packages/engine/src/bus/publish-writes.ts | 136 ++++--- packages/engine/test/bus/change-bus.test.ts | 49 ++- packages/store-postgres/README.md | 35 ++ packages/store-postgres/src/bus.ts | 369 ++++++++++++++++++ packages/store-postgres/src/index.ts | 36 +- packages/store-postgres/src/listen.ts | 88 +++++ packages/store-postgres/src/request-store.ts | 64 ++- .../test/fake-notify-network.ts | 104 +++++ .../store-postgres/test/listen-tail.test.ts | 52 +++ .../store-postgres/test/postgres-bus.test.ts | 289 ++++++++++++++ pnpm-lock.yaml | 6 + 23 files changed, 1942 insertions(+), 113 deletions(-) create mode 100644 .changeset/postgres-bus.md create mode 100644 goals/lib/peers.mts create mode 100644 goals/lib/postgres.mts create mode 100644 goals/session-stream/wakes-across-processes/goal.md create mode 100644 goals/session-stream/wakes-across-processes/peer.mts create mode 100644 goals/session-stream/wakes-across-processes/run.mts create mode 100644 packages/store-postgres/src/bus.ts create mode 100644 packages/store-postgres/src/listen.ts create mode 100644 packages/store-postgres/test/fake-notify-network.ts create mode 100644 packages/store-postgres/test/listen-tail.test.ts create mode 100644 packages/store-postgres/test/postgres-bus.test.ts diff --git a/.changeset/postgres-bus.md b/.changeset/postgres-bus.md new file mode 100644 index 0000000000..4835eae504 --- /dev/null +++ b/.changeset/postgres-bus.md @@ -0,0 +1,6 @@ +--- +"@flow-state-dev/store-postgres": minor +"@flow-state-dev/engine": patch +--- + +`postgresStores` now backs the `bus` slot, so on a host with more than one process over one Postgres a session view open on one process sees a change saved on another as soon as it is saved, and an idle view costs the database nothing. Each process with an open view holds one listening connection from `liveTailPool` and pings itself through it every 2 s; it reports `crossesProcesses: true` only while the ping comes back, so behind a transaction pooler (or after a silent drop) views keep the once-a-second read, and the process logs one warning naming the fix (a `liveTailPool` on a direct connection string). `createPostgresStores` returns the bus as `registry.bus`, and `close()` hands its connection back before ending the pools. Without a tail pool (`liveTailPool: null`, `{ pool }` without one, `{ executor }`) the bus stays in process. In `@flow-state-dev/engine`, `withChangeBus` now refuses stores that already publish to a different bus instead of publishing to both, a write's notice reaches subscribers one microtask sooner, and a notice that can't be assembled is debug-logged (FIX-2427). diff --git a/apps/docs/docs/persistence/overview.md b/apps/docs/docs/persistence/overview.md index 1b25feaef0..1d7ebb48f6 100644 --- a/apps/docs/docs/persistence/overview.md +++ b/apps/docs/docs/persistence/overview.md @@ -15,9 +15,9 @@ The `bus` slot carries change notices. Every request and session write sends one | Bus | What an open session view reads | |---|---| | Crosses processes | Only after a change, at most once a second. A view of a quiet session costs the store nothing | -| In memory (the default today) | After a change, and also about once a second, so a reply written by another process still arrives | +| In memory | After a change, and also about once a second, so a reply written by another process still arrives | -None of the adapters below backs `bus` yet, so every profile runs the in-memory bus for now. Views keep the once-a-second read, as they always have. +The Postgres adapter backs `bus` (see [Postgres](#postgres)). The in-memory, file and SQLite adapters don't, so their profiles run the in-memory bus, and views keep the once-a-second read. | Adapter | Persistence | When to use | |---------|------------|-------------| @@ -76,6 +76,8 @@ const flowstate = createFlowState({ On Vercel, use `vercelPostgresStores()` from `@flow-state-dev/vercel/store` instead — it bakes in the pool tuning and Neon client swap. +With more than one process on one database (a web process and BullMQ workers, or several web processes), a session view open on one process sees a change saved on another as soon as it is saved, and a view of a quiet session costs the database nothing. You don't configure anything for this: the Postgres adapter backs the `bus` slot, and each process with an open view holds one listening connection and pings itself through it every 2 seconds. When the ping stops coming back, because a transaction pooler doesn't deliver `LISTEN` notifications or the connection dropped, views on that process go back to the once-a-second read until it recovers, and nothing is missed. `vercelPostgresStores()` doesn't back `bus`, so views there keep the once-a-second read. The [package README](https://github.com/fixpoint-labs/flow-state-dev/tree/main/packages/store-postgres#cross-process-notices) covers the connection, the pooler warning and `liveTailPool: null`. + Postgres provides the concurrency safety that compare-and-swap relies on. Compare-and-swap means a write carries the version it expects to find, and the store applies it only if that version is still current — so a write built on a stale read is refused instead of silently overwriting someone else's. Not every scope-state write carries a version. An increment, an append, or a write to one key of a record field is handed to the store as the operation itself, and the store applies it to the value it currently holds. On the built-in stores, concurrent increments and appends both land; a custom store that doesn't implement those operations takes a version-checked whole-record write instead, and one of the two is refused. Concurrent writes to one field don't combine — the second one replaces the first, and neither call reports a conflict. [State Operations](../fundamentals/state-operations.md#cas-semantics) lists which calls go which way. diff --git a/docs/mechanisms/mechanisms.yaml b/docs/mechanisms/mechanisms.yaml index 6f591db19c..03c4aa10be 100644 --- a/docs/mechanisms/mechanisms.yaml +++ b/docs/mechanisms/mechanisms.yaml @@ -299,6 +299,7 @@ mechanisms: package: "@flow-state-dev/engine" anchor: createInMemoryBus file: packages/engine/src/bus/in-memory-bus.ts + adapters: ["@flow-state-dev/store-postgres"] builtOn: [store] overlaps: - { id: streaming, relation: distinct-jobs, note: "streaming sends one request's item log and ends with it; a bus topic is a scope's and outlives every request." } diff --git a/goals/README.md b/goals/README.md index 8f44b4fe1f..da9baef44b 100644 --- a/goals/README.md +++ b/goals/README.md @@ -114,6 +114,8 @@ Every goal repeats the same scaffolding around the part that is actually its own | `server` | `refuseIfAnswering`, `stopProcessGroup`, `onClosed` — for a goal that spawns its own server: refuse when anything already answers on the port, and stop the whole detached process group so no survivor is left for the next run to grade; `onClosed(child, done)` waits for a piped child's output to be read to the end (`close`, not `exit`), but at most 2s past `exit`, so a grandchild still holding the pipe can't hang a stop | | `module-patch` | `modulePatchEnv`, `describePatches`, `assertPatched` — a named control as scratch patches to source modules, applied as the process under test loads them (`module-patch.mjs` through `NODE_OPTIONS`), never written to the checkout; the run prints the patch and fails when it never reached the code it served | | `http-actions` | `actingAs(router, userId)` (`call`, `act`, `open`) — drive an in-process host's router as a verified caller and read an action's answer back from the session's ended request, as an app does; `untilIdle(stores)` — wait until no request is running. The host's `resolvePrincipal` reads the caller from `x-user` | +| `postgres` | `startPostgres({ slug, settings? })` — a real Postgres for a goal that grades what crosses between processes: `FSD_DB_URL` (with a scratch database on it when the role may make one), or a throwaway cluster from `initdb` and `pg_ctl` on a free port, run as the `postgres` user under root; `query` runs setup and `pg_terminate_backend`, `stop` removes it. PGlite can't stand in: it can't `LISTEN` across processes. **Not re-exported from `index.mts`** (it loads `pg`); import `../../lib/postgres.mts` directly | +| `peers` | `forkPeer(module, name, env)` and, in the forked module, `servePeer(commands)` — real processes (`child_process.fork`, under the same loader and control patches) driven over IPC: `call` a named command, hear what the peer `push`es. One process with two registries is not two processes. **Not re-exported from `index.mts`**; import `../../lib/peers.mts` directly | | `playwright` | `launchChromium`, `preinstalledChromium` — Chromium for a goal that reads the shipped DevTool. **Not re-exported from `index.mts`**, so a goal that opens no browser does not load Playwright; import `../../lib/playwright.mts` directly | | `shift-manager` | `buildShiftManagerCopy` (a patched scratch copy, built with Vite), `startShiftManager` (its command over a Lab, with `NODE_ENV` as it was before any in-process build; optional `root`, `tsx` and `timeoutMs` to serve another checkout, including an older one that has only `bin/start.mts`), `labApi` (the Lab's routes), `pendingSeatAsks` (the person's pending asks on the seats' sessions, picked as Shift Manager picks them), `workerOf` (the worker a listed session names) and `pickShift` (a theme picked with the sidebar's theme mark) — for the Shift Manager goals; and, for a goal that drives Shift Manager as a person in Playwright, `buildShiftManagerPages`, `labRoutes` (`labApi` read as one verified user) and `personPage` (a browser context as a given person). **Not re-exported from `index.mts`**; import `../../lib/shift-manager.mts` directly | diff --git a/goals/lib/peers.mts b/goals/lib/peers.mts new file mode 100644 index 0000000000..9240cf495a --- /dev/null +++ b/goals/lib/peers.mts @@ -0,0 +1,134 @@ +/** + * Real processes for a goal that grades what crosses between them (FIX-2427, + * shared with the epic closure FIX-2368). + * + * One process with two registries is not two processes: it shares module + * state, timers and an event loop, and a bus that only reaches its own + * process passes there. So the goal forks each peer (`child_process.fork`, + * under the same `tsx` loader) and drives it over IPC: the runner calls a + * named command and awaits its answer, and the peer may push events back + * unasked (an SSE frame its view received, stamped where it was received). + * + * The runner side is {@link forkPeer}; the peer side, in the forked module, + * is {@link servePeer}. Not re-exported from `index.mts`; import directly: + * + * import { forkPeer } from "../../lib/peers.mts"; // the runner + * import { servePeer } from "../../lib/peers.mts"; // the peer module + */ +import { fork, type ChildProcess } from "node:child_process"; + +type Call = { id: number; command: string; args: unknown }; +type Answer = { id: number; ok: true; value: unknown } | { id: number; ok: false; error: string }; +type Pushed = { event: string; data: unknown }; + +/** A forked peer, as the runner holds it. */ +export type Peer = { + readonly name: string; + /** Run `command` in the peer and await what it returns. Rejects with the peer's error. */ + call(command: string, args?: unknown): Promise; + /** Hear every `event` the peer pushes. */ + on(event: string, listener: (data: unknown) => void): void; + /** Ask the peer to stop (its `stop` command, when it has one), then end the process. */ + stop(): Promise; +}; + +/** + * Fork `modulePath` as a peer named `name`, with `env` added to this + * process's environment, and wait until it says it is ready. + * + * @throws When the peer exits or fails before it is ready. + */ +export async function forkPeer(modulePath: string, name: string, env: Record = {}): Promise { + const child: ChildProcess = fork(modulePath, [], { + env: { ...process.env, ...env, GOAL_PEER_NAME: name }, + stdio: ["ignore", "inherit", "inherit", "ipc"] + }); + const waiting = new Map(); + const listeners = new Map void>>(); + let next = 0; + let exited = false; + + const ready = new Promise((resolve, reject) => { + child.on("message", (message: Answer | Pushed | { ready: true }) => { + if ("ready" in message) return resolve(); + if ("event" in message) { + for (const listener of listeners.get(message.event) ?? []) listener(message.data); + return; + } + const pending = waiting.get(message.id); + waiting.delete(message.id); + if (message.ok) pending?.resolve(message.value); + else pending?.reject(new Error(`${name}: ${message.error}`)); + }); + child.on("exit", (code, signal) => { + exited = true; + const error = new Error(`${name} exited (${code ?? signal})`); + reject(error); + for (const pending of waiting.values()) pending.reject(error); + waiting.clear(); + }); + }); + await ready; + + return { + name, + call(command: string, args?: unknown): Promise { + if (exited) return Promise.reject(new Error(`${name} has exited`)); + const id = (next += 1); + return new Promise((resolve, reject) => { + waiting.set(id, { resolve: resolve as (value: unknown) => void, reject }); + child.send({ id, command, args } satisfies Call); + }); + }, + on(event, listener) { + listeners.set(event, [...(listeners.get(event) ?? []), listener]); + }, + async stop() { + if (exited) return; + const gone = new Promise((resolve) => child.once("exit", () => resolve())); + try { + await Promise.race([ + this.call("stop"), + new Promise((resolve) => setTimeout(resolve, 10_000)) + ]); + } catch { + // A peer with no `stop`, or one already failing, is ended below. + } + if (!exited) child.kill("SIGTERM"); + await gone; + } + }; +} + +/** + * Serve `commands` to the runner that forked this process, then say ready. + * A `stop` command, when given, runs before the process exits on the + * runner's `stop()`. + * + * @returns `push`, to send the runner an event unasked. + */ +export function servePeer(commands: Record unknown>): { push(event: string, data: unknown): void } { + if (process.send === undefined) throw new Error("servePeer: this module runs only as a forked peer (forkPeer)"); + const send = process.send.bind(process); + process.on("message", (message: Call) => { + const handler = commands[message.command]; + void (async () => { + try { + if (handler === undefined) throw new Error(`no command "${message.command}"`); + const value = await handler(message.args); + send({ id: message.id, ok: true, value } satisfies Answer); + if (message.command === "stop") setTimeout(() => process.exit(0), 10); + } catch (error) { + send({ id: message.id, ok: false, error: error instanceof Error ? (error.stack ?? error.message) : String(error) } satisfies Answer); + } + })(); + }); + // The runner owns this process: it lives while the IPC channel does. + process.on("disconnect", () => process.exit(0)); + send({ ready: true }); + return { + push(event, data) { + send({ event, data } satisfies Pushed); + } + }; +} diff --git a/goals/lib/postgres.mts b/goals/lib/postgres.mts new file mode 100644 index 0000000000..aaf788c80c --- /dev/null +++ b/goals/lib/postgres.mts @@ -0,0 +1,160 @@ +/** + * A real Postgres for a goal that runs more than one process over one + * database (FIX-2427, shared with the epic closure FIX-2368). + * + * PGlite can't stand in: it lives inside one process, and `LISTEN` / + * `NOTIFY` between processes is what these goals grade. So either the + * caller's `FSD_DB_URL` (a scratch database is made on it when the role may, + * dropped on stop), or a throwaway cluster this starts with `initdb` and + * `pg_ctl` on a free port and removes on stop. + * + * **Not re-exported from `index.mts`**: it loads `pg`. Import it directly: + * + * import { startPostgres } from "../../lib/postgres.mts"; + */ +import { spawnSync } from "node:child_process"; +import { chownSync, existsSync, mkdtempSync, readdirSync, rmSync } from "node:fs"; +import { createServer } from "node:net"; +import { tmpdir, userInfo } from "node:os"; +import { join } from "node:path"; +import pg from "pg"; + +export type RunningPostgres = { + /** A connection string for the database the goal uses. */ + url: string; + /** `"initdb"` for a cluster this started, `"FSD_DB_URL"` for the caller's. */ + source: "initdb" | "FSD_DB_URL"; + /** Run one statement as the database's owner (setup, `pg_terminate_backend`). */ + query = Record>(text: string, values?: unknown[]): Promise; + /** Stop the cluster or drop the scratch database. Idempotent. */ + stop(): Promise; +}; + +export type StartPostgresOptions = { + /** Names the scratch directory or database. */ + slug: string; + /** + * Server settings for a cluster this starts (`-c name=value`), e.g. + * `{ log_statement: "all" }`. Ignored on `FSD_DB_URL`, whose server is the + * caller's: a goal that needs one must say it can't tell without it. + */ + settings?: Record; +}; + +/** + * Start (or attach to) a Postgres for one goal run. + * + * @throws When neither `FSD_DB_URL` is set nor `initdb` and `pg_ctl` can be found. + */ +export async function startPostgres(options: StartPostgresOptions): Promise { + const given = process.env.FSD_DB_URL; + if (given !== undefined && given !== "") return attach(given, options.slug); + return startCluster(options); +} + +/** The caller's server: a scratch database on it when the role may make one, else the database as given. */ +async function attach(given: string, slug: string): Promise { + const name = `goal_${slug.replace(/[^a-z0-9]+/gi, "_")}_${Date.now()}`.toLowerCase(); + const admin = new pg.Pool({ connectionString: given, max: 1 }); + let url = given; + let created = false; + try { + await admin.query(`CREATE DATABASE ${name}`); + const parsed = new URL(given); + parsed.pathname = `/${name}`; + url = parsed.toString(); + created = true; + } catch { + // No CREATEDB: the goal shares the database as given, and keys its rows freshly. + } + const pool = new pg.Pool({ connectionString: url, max: 2 }); + let stopped = false; + return { + url, + source: "FSD_DB_URL", + query: async (text, values) => (await pool.query(text, values)).rows, + async stop() { + if (stopped) return; + stopped = true; + await pool.end().catch(() => {}); + if (created) await admin.query(`DROP DATABASE IF EXISTS ${name} WITH (FORCE)`).catch(() => {}); + await admin.end().catch(() => {}); + } + }; +} + +/** Where `initdb` and `pg_ctl` live: `PATH`, or the newest Debian-style install. */ +function binDir(): string { + const onPath = spawnSync("pg_config", ["--bindir"], { encoding: "utf8" }); + if (onPath.status === 0 && existsSync(join(onPath.stdout.trim(), "initdb"))) return onPath.stdout.trim(); + const root = "/usr/lib/postgresql"; + if (existsSync(root)) { + const versions = readdirSync(root) + .filter((v) => existsSync(join(root, v, "bin", "initdb"))) + .sort((a, b) => Number(b) - Number(a)); + if (versions[0] !== undefined) return join(root, versions[0], "bin"); + } + const which = spawnSync("sh", ["-c", "command -v initdb"], { encoding: "utf8" }); + if (which.status === 0) return join(which.stdout.trim(), ".."); + throw new Error("no Postgres to start: set FSD_DB_URL, or install Postgres so `initdb` and `pg_ctl` are found"); +} + +function freePort(): Promise { + return new Promise((resolve, reject) => { + const server = createServer(); + server.unref(); + server.on("error", reject); + server.listen(0, "127.0.0.1", () => { + const address = server.address(); + server.close(() => (typeof address === "object" && address !== null ? resolve(address.port) : reject(new Error("no port")))); + }); + }); +} + +/** + * `initdb` refuses to run as root, so a root caller (a CI container) runs the + * cluster as the `postgres` user, which owns the scratch directory. + */ +function runAs(): { uid: number; gid: number } | undefined { + if (userInfo().uid !== 0) return undefined; + const id = spawnSync("id", ["-u", "postgres"], { encoding: "utf8" }); + const gid = spawnSync("id", ["-g", "postgres"], { encoding: "utf8" }); + if (id.status !== 0 || gid.status !== 0) throw new Error("running as root with no `postgres` user to run initdb as; set FSD_DB_URL"); + return { uid: Number(id.stdout.trim()), gid: Number(gid.stdout.trim()) }; +} + +async function startCluster(options: StartPostgresOptions): Promise { + const bin = binDir(); + const user = runAs(); + const dir = mkdtempSync(join(tmpdir(), `fsd-goal-pg-${options.slug}-`)); + if (user !== undefined) chownSync(dir, user.uid, user.gid); + const data = join(dir, "data"); + const run = (cmd: string, args: string[]): void => { + const result = spawnSync(join(bin, cmd), args, { encoding: "utf8", cwd: dir, ...(user ?? {}) }); + if (result.status !== 0) throw new Error(`${cmd} failed (${result.status}): ${result.stderr || result.stdout}`); + }; + + run("initdb", ["-D", data, "-A", "trust", "-U", "postgres", "-E", "UTF8", "--no-instructions"]); + const port = await freePort(); + const settings = { listen_addresses: "127.0.0.1", port: String(port), unix_socket_directories: dir, fsync: "off", ...options.settings }; + const serverArgs = Object.entries(settings) + .map(([name, value]) => `-c ${name}=${value}`) + .join(" "); + run("pg_ctl", ["-D", data, "-l", join(dir, "server.log"), "-o", serverArgs, "-w", "-t", "30", "start"]); + + const url = `postgres://postgres@127.0.0.1:${port}/postgres`; + const pool = new pg.Pool({ connectionString: url, max: 2 }); + let stopped = false; + return { + url, + source: "initdb", + query: async (text, values) => (await pool.query(text, values)).rows, + async stop() { + if (stopped) return; + stopped = true; + await pool.end().catch(() => {}); + spawnSync(join(bin, "pg_ctl"), ["-D", data, "-m", "immediate", "-w", "stop"], { cwd: dir, ...(user ?? {}) }); + rmSync(dir, { recursive: true, force: true }); + } + }; +} diff --git a/goals/package.json b/goals/package.json index fe3de265dc..1bdb431b38 100644 --- a/goals/package.json +++ b/goals/package.json @@ -36,10 +36,12 @@ "@flow-state-dev/workspace": "workspace:*", "@openai/codex-sdk": "0.164.0", "bullmq": "^5.0.0", + "pg": "^8.13.0", "zod": "^3.25.76" }, "devDependencies": { "@types/node": "^22.10.5", + "@types/pg": "^8.11.0", "esbuild": "0.27.3", "playwright": "^1.49.0", "tsx": "^4.19.4", diff --git a/goals/session-stream/wakes-across-processes/goal.md b/goals/session-stream/wakes-across-processes/goal.md new file mode 100644 index 0000000000..7bfb0e526f --- /dev/null +++ b/goals/session-stream/wakes-across-processes/goal.md @@ -0,0 +1,27 @@ +# session-stream › it wakes across processes + +**Issue:** FIX-2427 +**Outcome:** On two or more processes over one Postgres, a change saved on any process reaches every open view of it on the others within a second, and a view where nothing changes costs the database nothing; whenever Postgres can't carry the notices, views keep the one-second read and miss nothing. +**Input:** Two processes (`child_process.fork`, through `goals/lib/peers.mts`), each `createFlowState` with `postgresStores` at its defaults, over one real Postgres that `goals/lib/postgres.mts` starts (`initdb`, `pg_ctl`) or `FSD_DB_URL`. Process 1 answers in a session with a handler that writes only items and a status (a plain reply), and starts a background run through a dispatcher. Process 2 holds two open views of the session (`GET /sessions/:id/stream`). Every statement process 2 sends, on any of its connections, passes one counting `pg.Client` (`poolOptions.Client`, which the tail pool inherits); the bus's own ping and `LISTEN` are counted apart. Held-out: any reply text and any run key must pass. +**Signal:** +- `1:reply` — process 2's view receives process 1's reply less than 1000 ms after process 1 saved it (polled off process 1's store every 5 ms). +- `1:run` — process 2's view names process 1's background run in `session.runs` less than 1000 ms after the run's start is saved; started after a quiet interval, since a view reads at most once an interval. +- `2:idle` — over a 10 s idle window with two views open, process 2 makes 0 statements for its views; its bus pings at most 6 times (one per 2 s, plus one). +- `3:drop` — with process 2's view idle and waiting, its listening backend is terminated (`pg_terminate_backend`); one reply from process 1, with no write after it, reaches the view within 10 s. `3:back` — once process 2's bus says it crosses again, the next reply arrives within 1000 ms. +- `4:no-tail` — with `liveTailPool: null` on both, process 2's bus says it does not cross, and the reply arrives within 2000 ms, on the timed read. + +**Anti-game:** Not `pg_notify` calls and not a mocked pool: both pass while process 2 stays empty. Every assertion reads what process 2's stream delivered or what process 2 asked the database. One process with two registries is not two processes, and PGlite can't `LISTEN` across them. `2:idle` is floored on everything process 2 sent in the window, so a count that never reached its connections is CAN'T TELL, not a pass. +**Model:** n/a +**Run:** `pnpm tsx goals/session-stream/wakes-across-processes/run.mts` (root needs a `postgres` user to run `initdb` as; or set `FSD_DB_URL`) +**Controls:** +- `GOAL_CONTROL=no-listen` — the bus never `LISTEN`s but says it crosses (a scratch patch to `packages/store-postgres/src/bus.ts`). Must FAIL `1:reply`. It fails `3` and `4` too: the patch says "crosses" with no tail pool as well. +- `GOAL_CONTROL=trust-listen` — the bus trusts a `LISTEN` that succeeded: no ping, and a lost connection changes nothing. Must FAIL `3:drop`. +- `GOAL_CONTROL=no-bus` — today's `main`: the Postgres adapter declares no `bus`, so each process gets an in-memory one (a scratch patch to `packages/store-postgres/src/index.ts`). Must FAIL `2:idle`: process 2's two views read the store 4 times a second each. + +## Verdict log +| Date | Commit | Model | Verdict | Notes | +|------|--------|-------|---------|-------| +| 2026-11-03 | fix/FIX-2427-postgres-bus (pre-commit) | n/a | PASS ×3 runs | initdb, PG 16. 2: 0 view statements, 5 pings in 10 s idle, two views · 1: reply 105–117 ms, run 112–165 ms after it started · 3: 1 listener terminated, reply 431–468 ms after saved; back, next reply 111–117 ms · 4: reply 481–487 ms on the timed read | +| 2026-11-03 | fix/FIX-2427-postgres-bus (pre-commit) | n/a | FAIL (expected) | `GOAL_CONTROL=no-listen`: `1:reply` and `1:run` never arrived within 3 s (also `3`, `4`, since the patch says "crosses" everywhere) | +| 2026-11-03 | fix/FIX-2427-postgres-bus (pre-commit) | n/a | FAIL (expected) | `GOAL_CONTROL=trust-listen`: `3:drop` never arrived within 10 s, nor `3:back` | +| 2026-11-03 | fix/FIX-2427-postgres-bus (pre-commit) | n/a | FAIL (expected) | `GOAL_CONTROL=no-bus` (today's `main`): `2:idle` 80 statements in 10 s for two views | diff --git a/goals/session-stream/wakes-across-processes/peer.mts b/goals/session-stream/wakes-across-processes/peer.mts new file mode 100644 index 0000000000..8d45c4aa23 --- /dev/null +++ b/goals/session-stream/wakes-across-processes/peer.mts @@ -0,0 +1,217 @@ +/** + * One process of the goal (FIX-2427): `createFlowState` over `postgresStores` + * on the shared database, as an app's process holds it, driven by the runner + * over IPC (`goals/lib/peers.mts`). + * + * Every statement this process sends, on any connection (the stores' pool and + * the tail pool alike), passes through one `pg.Client` subclass, so the + * runner can ask what this process asked the database while counting was on. + * The bus's own traffic (its ping, its `LISTEN`) is counted apart. + */ +import { z } from "zod"; +import pg from "pg"; +import { DEFAULT_ORG_ID, defineFlow, dispatcher, handler } from "@flow-state-dev/core"; +import type { SessionStreamEvent } from "@flow-state-dev/core/items"; +import { createFlowState, type FlowApiRouter, type StoreRegistry } from "@flow-state-dev/engine"; +import { postgresStores } from "@flow-state-dev/store-postgres"; +import { servePeer } from "../../lib/peers.mts"; + +const NAME = process.env.GOAL_PEER_NAME ?? "peer"; +const URL_ = process.env.GOAL_DB_URL!; +const TAIL = process.env.GOAL_TAIL !== "off"; + +// --------------------------------------------------------------------------- +// What this process asks the database +// --------------------------------------------------------------------------- + +const counted = { on: false, statements: 0, pings: 0, bus: 0, texts: [] as string[] }; + +/** Every connection this process opens is one of these. */ +class CountingClient extends pg.Client { + // pg's overloads; only the text and values are read. + override query(...args: any[]): any { + if (counted.on) { + const config = args[0] as string | { text?: string; values?: unknown[] }; + const text = typeof config === "string" ? config : (config?.text ?? ""); + const values = (Array.isArray(args[1]) ? args[1] : typeof config === "object" ? config?.values : undefined) ?? []; + if (/^\s*(UN)?LISTEN fsd_bus\b/i.test(text)) counted.bus += 1; + else if (/pg_notify/i.test(text) && String(values[1] ?? "").includes('"ping"')) counted.pings += 1; + else { + counted.statements += 1; + if (counted.texts.length < 20) counted.texts.push(text.replace(/\s+/g, " ").slice(0, 120)); + } + } + return (super.query as (...a: any[]) => any)(...args); + } +} + +// --------------------------------------------------------------------------- +// The app +// --------------------------------------------------------------------------- + +let releaseRun: () => void = () => {}; + +function chatFlow() { + const say = handler({ + name: "chat-say", + execute: async (input, ctx) => { + ctx.emit.message((input as { text: string }).text); + return {}; + } + }); + const work = handler({ + name: "chat-work", + execute: async () => { + await new Promise((resolve) => (releaseRun = resolve)); + return {}; + } + }); + return defineFlow({ + kind: "chat", + actions: { + say: { block: say }, + spawn: { + block: dispatcher({ + name: "chat-spawn", + action: "work", + inputSchema: z.object({ key: z.string() }), + session: { key: (input: { key: string }) => input.key }, + payload: () => ({}) + }) + } + }, + internal: { actions: { work: { block: work } } } + })(); +} + +const flowState = createFlowState({ + flows: { chat: chatFlow() }, + stores: { + default: { + primary: postgresStores({ + connectionString: URL_, + poolOptions: { Client: CountingClient as unknown as typeof pg.Client, application_name: `goal-${NAME}` }, + ...(TAIL ? {} : { liveTailPool: null }) + }) + } + }, + staleSweepIntervalMs: 0, + detectInterruptedOnStartup: false +}); +const router: FlowApiRouter = await flowState.getRouter(); +const stores: StoreRegistry = (await flowState.getRuntime()).stores; + +const delay = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +// --------------------------------------------------------------------------- +// Commands +// --------------------------------------------------------------------------- + +const views = new Map(); +let nextView = 0; + +const peer = servePeer({ + async act({ session, action, input }: { session: string; action: string; input: Record }) { + const res = await router.POST( + new Request(`http://localhost/api/flows/chat/${session}/actions/${action}`, { + method: "POST", + body: JSON.stringify({ userId: "alice", orgId: DEFAULT_ORG_ID, input }) + }), + { params: { path: ["chat", session, "actions", action] } } + ); + if (res.status !== 202) throw new Error(`POST ${action} answered ${res.status}: ${await res.text()}`); + return ((await res.json()) as { request: { id: string } }).request.id; + }, + + /** When `requestId` is saved finished, read off this process's store every 5 ms. */ + async savedAt({ requestId }: { requestId: string }) { + const deadline = Date.now() + 5_000; + for (;;) { + const record = await stores.request.get(requestId); + if (record !== undefined && record.status !== "in_progress") return Date.now(); + if (Date.now() > deadline) throw new Error(`request ${requestId} never finished`); + await delay(5); + } + }, + + /** When a run under `session` first has a running request, polled every 5 ms. */ + async runStartedAt({ session }: { session: string }) { + const deadline = Date.now() + 5_000; + for (;;) { + const children = await stores.session.list({ parentage: { parentOf: session } }); + if (children.length > 0) { + const running = await stores.request.list({ + sessionId: children.map((child) => child.id), + status: ["in_progress", "suspended"], + orderBy: "none" + }); + if (running.length > 0) return Date.now(); + } + if (Date.now() > deadline) throw new Error("the background run never started"); + await delay(5); + } + }, + + releaseRun() { + releaseRun(); + }, + + /** Open a view of `session`; each event it receives is pushed as `frame`, stamped when received. */ + async openView({ session }: { session: string }) { + const view = (nextView += 1); + const controller = new AbortController(); + const res = await router.GET( + new Request(`http://localhost/api/flows/sessions/${session}/stream`, { signal: controller.signal }), + { params: { path: ["sessions", session, "stream"] } } + ); + if (res.status !== 200 || res.body === null) throw new Error(`the stream answered ${res.status}`); + views.set(view, controller); + const reader = res.body.getReader(); + const decoder = new TextDecoder(); + void (async () => { + let buffer = ""; + try { + for (;;) { + const { value, done } = await reader.read(); + if (done) return; + buffer += decoder.decode(value, { stream: true }); + for (let at = buffer.indexOf("\n\n"); at >= 0; at = buffer.indexOf("\n\n")) { + const data = buffer + .slice(0, at) + .split("\n") + .filter((line) => line.startsWith("data: ")) + .map((line) => line.slice(6)) + .join("\n"); + buffer = buffer.slice(at + 2); + if (data !== "") peer.push("frame", { view, at: Date.now(), event: JSON.parse(data) as SessionStreamEvent }); + } + } + } catch { + // Closed. + } + })(); + return view; + }, + + closeView({ view }: { view: number }) { + views.get(view)?.abort(); + views.delete(view); + }, + + /** Start counting from zero, or stop and return what was counted. */ + count({ on }: { on: boolean }) { + if (on) Object.assign(counted, { on: true, statements: 0, pings: 0, bus: 0, texts: [] }); + else counted.on = false; + return { statements: counted.statements, pings: counted.pings, bus: counted.bus, texts: counted.texts }; + }, + + crosses() { + return stores.bus?.crossesProcesses === true; + }, + + async stop() { + for (const controller of views.values()) controller.abort(); + releaseRun(); + await flowState.dispose(); + } +}); diff --git a/goals/session-stream/wakes-across-processes/run.mts b/goals/session-stream/wakes-across-processes/run.mts new file mode 100644 index 0000000000..3cf8ab51a2 --- /dev/null +++ b/goals/session-stream/wakes-across-processes/run.mts @@ -0,0 +1,247 @@ +/** + * session-stream › it wakes across processes (FIX-2427). + * + * Two real processes (`child_process.fork`), each `createFlowState` over + * `postgresStores`, over one real Postgres (`initdb` + `pg_ctl`, or + * `FSD_DB_URL`). Process 1 answers in a session; process 2 holds two open + * views of it. What is graded is what process 2's views receive and what + * process 2 asks the database, never whether `pg_notify` ran. + * + * 1 process 1's plain reply (items and a status), and a run it starts, + * reach process 2's view within a second of being saved; + * 2 over a 10 s idle window process 2 makes no statement for its views; + * the bus's pings are counted apart, at most one per 2 s for the process; + * 3 with process 2's listening backend terminated (`pg_terminate_backend`) + * and its view waiting, one reply with no write after it still reaches + * the view within 10 s; once the listener is back, the next within 1 s; + * 4 with `liveTailPool: null` on both, the reply still arrives within 2 s, + * on the timed read. + * + * Controls (each runs this file again with a patch applied as the peers load): + * GOAL_CONTROL=no-listen the bus never LISTENs but says it crosses: 1 must FAIL. + * GOAL_CONTROL=trust-listen the bus trusts LISTEN, no ping: 3 must FAIL. + * GOAL_CONTROL=no-bus today's `main`: the adapter declares no bus: 2 must FAIL. + */ +import { spawnSync } from "node:child_process"; +import { rmSync } from "node:fs"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import type { SessionStreamEvent } from "@flow-state-dev/core/items"; +import { + REPO_ROOT, + RUN_STAMP, + assertPatched, + describePatches, + goalTmpDir, + modulePatchEnv, + runGoal, + type ModulePatch +} from "../../lib/index.mts"; +import { startPostgres, type RunningPostgres } from "../../lib/postgres.mts"; +import { forkPeer, type Peer } from "../../lib/peers.mts"; + +const CONTROL = process.env.GOAL_CONTROL ?? ""; +const SCRATCH = process.env.GOAL_SCRATCH ?? goalTmpDir("session-stream-across"); +const PEER = fileURLToPath(new URL("./peer.mts", import.meta.url)); +const IDLE_WINDOW_MS = 10_000; +const PING_INTERVAL_MS = 2_000; +const BUS = "packages/store-postgres/src/bus.ts"; + +const PATCHES: Record = { + // The listener never opens, and the bus says it crosses anyway. + "no-listen": [ + { module: BUS, from: String.raw`#start\(\)(?:: void)? \{`, to: "#start() { return;" }, + { module: BUS, from: String.raw`get crossesProcesses\(\)(?:: boolean)? \{`, to: "get crossesProcesses() { return true;" } + ], + // A LISTEN that succeeds is trusted: no ping, and a lost connection changes nothing. + "trust-listen": [ + { module: BUS, from: String.raw`this\.#startPinging\(\);`, to: "this.#setCrossing(true);" }, + { module: BUS, from: String.raw`#lost\(connection(?:: ListenConnection \| undefined)?\)(?:: void)? \{`, to: "#lost(connection) { return;" } + ], + // Today's main: the Postgres adapter backs no bus, so each process gets an in-memory one. + "no-bus": [ + { + module: "packages/store-postgres/src/index.ts", + from: String.raw`capabilities: \["primary", "bus"\]`, + to: 'capabilities: ["primary"]' + } + ] +}; + +if (CONTROL !== "" && PATCHES[CONTROL] === undefined) { + console.error(`unknown GOAL_CONTROL "${CONTROL}"; known: ${Object.keys(PATCHES).join(", ")}`); + process.exit(2); +} +if (PATCHES[CONTROL] !== undefined && process.env.GOAL_MODULE_PATCH === undefined) { + const mark = join(SCRATCH, `patch-mark-${RUN_STAMP}`); + rmSync(mark, { force: true }); + console.error(`control ${CONTROL}:\n${describePatches(PATCHES[CONTROL]!, REPO_ROOT)}`); + const child = spawnSync(process.execPath, [...process.execArgv, ...process.argv.slice(1)], { + stdio: "inherit", + env: { ...process.env, GOAL_SCRATCH: SCRATCH, ...modulePatchEnv(PATCHES[CONTROL]!, REPO_ROOT, mark) } + }); + assertPatched(PATCHES[CONTROL]!, REPO_ROOT, mark); + process.exit(child.status ?? 1); +} + +// --------------------------------------------------------------------------- +// The two processes +// --------------------------------------------------------------------------- + +type Frame = { view: number; at: number; event: SessionStreamEvent }; +type Counted = { statements: number; pings: number; bus: number; texts: string[] }; +type Pair = { p1: Peer; p2: Peer; frames: Frame[]; stop(): Promise }; + +async function pair(db: RunningPostgres, tail: boolean): Promise { + // NODE_ENV=test only quiets the engine's per-block console trace. + const env = { GOAL_DB_URL: db.url, GOAL_TAIL: tail ? "on" : "off", NODE_ENV: "test" }; + const p1 = await forkPeer(PEER, "p1", env); + const p2 = await forkPeer(PEER, "p2", env); + const frames: Frame[] = []; + p2.on("frame", (data) => frames.push(data as Frame)); + return { p1, p2, frames, stop: async () => void (await Promise.all([p1.stop(), p2.stop()])) }; +} + +const delay = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +function textOf(event: SessionStreamEvent): string | undefined { + if (event.type !== "session.item" || event.item.type !== "message") return undefined; + return (event.item.content as Array<{ text?: string }>)[0]?.text; +} + +/** When process 2's `view` first received an event `match` accepts, waiting up to `withinMs`. */ +async function receivedAt(frames: Frame[], view: number, match: (e: SessionStreamEvent) => boolean, withinMs: number): Promise { + const deadline = Date.now() + withinMs; + for (;;) { + const hit = frames.find((frame) => frame.view === view && match(frame.event)); + if (hit !== undefined) return hit.at; + if (Date.now() > deadline) return undefined; + await delay(5); + } +} + +/** Process 1 says `text` in `session`; the moment it is saved. */ +async function reply(p1: Peer, session: string, text: string): Promise { + const requestId = await p1.call("act", { session, action: "say", input: { text } }); + return p1.call("savedAt", { requestId }); +} + +async function until(check: () => Promise, withinMs: number): Promise { + const deadline = Date.now() + withinMs; + while (Date.now() < deadline) { + if (await check()) return true; + await delay(50); + } + return false; +} + +// --------------------------------------------------------------------------- +// The signals +// --------------------------------------------------------------------------- + +await runGoal(async (failures, tell) => { + const evidence: string[] = []; + const db = await startPostgres({ slug: "wakes-across" }); + evidence.push(`postgres: ${db.source}`); + const session = `s-${RUN_STAMP}`; + + try { + // 1, 2 and 3: the Postgres bus, as an app gets it. + { + const { p1, p2, frames, stop } = await pair(db, true); + try { + await reply(p1, session, "opening"); + const view = await p2.call("openView", { session }); + await p2.call("openView", { session }); + const crossed = await until(() => p2.call("crosses"), 5_000); + evidence.push(`p2 ${crossed ? "crosses" : "never said it crosses"}`); + await delay(1_500); + + // 2 · an idle view costs the database nothing. + await p2.call("count", { on: true }); + await delay(IDLE_WINDOW_MS); + const idle = await p2.call("count", { on: false }); + // Floored on everything p2 sent in the window, pings included: a count + // that saw none of p2's connections reads nothing, not a pass. + const seen = idle.statements + idle.pings + idle.bus; + tell.over("2:idle · statements p2 sent in the window, its bus's included", seen > 0 ? [seen] : []); + if (idle.statements !== 0) failures.push(`2:idle: p2 made ${idle.statements} statements for its views in ${IDLE_WINDOW_MS / 1_000} s (${idle.texts.slice(0, 4).join(" | ")})`); + const pingBound = Math.ceil(IDLE_WINDOW_MS / PING_INTERVAL_MS) + 1; + if (idle.pings > pingBound) failures.push(`2:idle: p2 pinged ${idle.pings} times in ${IDLE_WINDOW_MS / 1_000} s with two views open (bound ${pingBound})`); + evidence.push(`2: ${idle.statements} view statements, ${idle.pings} pings in ${IDLE_WINDOW_MS / 1_000} s idle, two views`); + + // 1 · a plain reply from process 1 reaches process 2's view. + const saved = await reply(p1, session, "the reply, from p1"); + const got = await receivedAt(frames, view, (e) => textOf(e) === "the reply, from p1", 3_000); + const replies = tell.over("1:reply · p2's copies of p1's reply", got === undefined ? [] : [got]); + if (replies.length === 0) failures.push("1:reply: p2's view never received p1's reply within 3 s"); + else if (replies[0]! - saved >= 1_000) failures.push(`1:reply: p2's view received p1's reply ${replies[0]! - saved} ms after it was saved (bound 1000)`); + else evidence.push(`1: reply ${replies[0]! - saved} ms after saved`); + + // 1 · a run process 1 starts reaches process 2's view. After a quiet + // interval, since a view reads at most once an interval. + await delay(1_500); + await p1.call("act", { session, action: "spawn", input: { key: `bg-${RUN_STAMP}` } }); + const started = await p1.call("runStartedAt", { session }); + const runAt = await receivedAt(frames, view, (e) => e.type === "session.runs" && e.runs.length > 0, 3_000); + const runs = tell.over("1:run · p2's notices naming p1's run", runAt === undefined ? [] : [runAt]); + if (runs.length === 0) failures.push("1:run: p2's view never named p1's run within 3 s"); + else if (runs[0]! - started >= 1_000) failures.push(`1:run: p2's view named p1's run ${runs[0]! - started} ms after it started (bound 1000)`); + else evidence.push(`1: run ${runs[0]! - started} ms after it started`); + await p1.call("releaseRun"); + + // 3 · the listener is killed while the view waits: one reply, nothing after it. + await delay(1_500); + const killed = await db.query<{ pid: number }>( + "SELECT pid FROM pg_stat_activity WHERE application_name = 'goal-p2' AND query ILIKE 'LISTEN fsd_bus%'" + ); + await db.query("SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE application_name = 'goal-p2' AND query ILIKE 'LISTEN fsd_bus%'"); + const dropSaved = await reply(p1, session, "the reply, during the drop"); + const dropGot = await receivedAt(frames, view, (e) => textOf(e) === "the reply, during the drop", 10_000); + tell.over("3:drop · p2's listening backends terminated", killed); + tell.over("3:drop · p2's copies of the reply", dropGot === undefined ? [] : [dropGot]); + if (killed.length === 0) failures.push("3:drop: p2 held no listening backend to terminate"); + else if (dropGot === undefined) failures.push("3:drop: with p2's listener terminated, p1's reply never reached p2's view within 10 s"); + else evidence.push(`3: ${killed.length} listener terminated, reply ${dropGot - dropSaved} ms after saved`); + + const back = await until(() => p2.call("crosses"), 10_000); + if (!back) failures.push("3:back: p2's bus never said it crosses again within 10 s of the drop"); + else { + await delay(1_500); + const afterSaved = await reply(p1, session, "the reply, after the listener is back"); + const afterGot = await receivedAt(frames, view, (e) => textOf(e) === "the reply, after the listener is back", 3_000); + if (afterGot === undefined) failures.push("3:back: once the listener was back, p1's next reply never reached p2's view within 3 s"); + else if (afterGot - afterSaved >= 1_000) failures.push(`3:back: the next reply arrived ${afterGot - afterSaved} ms after saved (bound 1000)`); + else evidence.push(`3: back, next reply ${afterGot - afterSaved} ms after saved`); + } + } finally { + await stop(); + } + } + + // 4 · no tail pool: the timed read still reaches process 2. + { + const { p1, p2, frames, stop } = await pair(db, false); + try { + const quiet = `${session}-no-tail`; + await reply(p1, quiet, "opening"); + const view = await p2.call("openView", { session: quiet }); + await delay(1_500); + const crosses = await p2.call("crosses"); + if (crosses) failures.push("4:no-tail: with liveTailPool: null, p2's bus says it crosses processes"); + const saved = await reply(p1, quiet, "the reply, with no tail pool"); + const got = await receivedAt(frames, view, (e) => textOf(e) === "the reply, with no tail pool", 4_000); + const replies = tell.over("4:no-tail · p2's copies of p1's reply", got === undefined ? [] : [got]); + if (replies.length === 0) failures.push("4:no-tail: p2's view never received p1's reply within 4 s"); + else if (replies[0]! - saved >= 2_000) failures.push(`4:no-tail: reply arrived ${replies[0]! - saved} ms after it was saved (bound 2000)`); + else evidence.push(`4: reply ${replies[0]! - saved} ms after saved, on the timed read`); + } finally { + await stop(); + } + } + } finally { + await db.stop(); + } + + return { failures, evidence: evidence.join(" · ") }; +}); diff --git a/packages/engine/README.md b/packages/engine/README.md index 48d8058199..c09b8bd0c8 100644 --- a/packages/engine/README.md +++ b/packages/engine/README.md @@ -81,7 +81,7 @@ Shutdown mostly does not write a terminal status on background work's behalf. It `stores` is a map of named profiles. A profile maps capability slots (typed containers for a category of storage) to adapters. The required slot is `primary` — the catch-all state store for sessions, requests, users, orgs, active requests, checkpoints, content, and traces. The `blobs`, `queue`, and `scheduler` slots are declared but forward-compatible; no backing store ships for them yet. -The `bus` slot carries change notices: every request and session write publishes one on its session's topic, and the session stream wakes on it (see [Change bus](#change-bus)). Leave it out and the primary backs it when the primary's adapter declares `bus`; otherwise the bus is in memory and reaches this process only. An adapter named in the `bus` slot that doesn't declare it fails at `ready()`. +The `bus` slot carries change notices: every request and session write publishes one on its session's topic, and the session stream wakes on it (see [Change bus](#change-bus)). Leave it out and the primary backs it when the primary's adapter declares `bus`, as `postgresStores` does (its bus crosses processes over `LISTEN`/`NOTIFY`); otherwise the bus is in memory and reaches this process only. An adapter named in the `bus` slot that doesn't declare it fails at `ready()`. ```ts import { createFlowState, inMemoryStores } from "@flow-state-dev/engine"; diff --git a/packages/engine/src/bus/in-memory-bus.ts b/packages/engine/src/bus/in-memory-bus.ts index f0f0fe89fe..e749b0970c 100644 --- a/packages/engine/src/bus/in-memory-bus.ts +++ b/packages/engine/src/bus/in-memory-bus.ts @@ -26,8 +26,9 @@ function deliver(listener: ChangeListener, notice: ChangeNotice): void { } /** - * Create a bus that delivers each notice to this process's subscribers, in a - * microtask after `publish`. `crossesProcesses` is always `false`. + * Create a bus that delivers each notice to this process's subscribers, in + * one microtask after `publish` (one per publish, however many subscribe). + * `crossesProcesses` is always `false`. */ export function createInMemoryBus(): ChangeBus { const byScope = new Map>(); @@ -38,14 +39,16 @@ export function createInMemoryBus(): ChangeBus { publish(notice) { const subscribers = byScope.get(scopeKey(notice.topic)); if (subscribers === undefined) return Promise.resolve(); - for (const subscriber of [...subscribers]) { - // Unnarrowed subscribers hear every notice under the scope; a narrowed - // one hears only its own resource. - if (subscriber.resource !== undefined && subscriber.resource !== notice.topic.resource) continue; - queueMicrotask(() => { - if (subscribers.has(subscriber)) deliver(subscriber.listener, notice); - }); - } + // Unnarrowed subscribers hear every notice under the scope; a narrowed + // one hears only its own resource. + const hearing = [...subscribers].filter( + (subscriber) => subscriber.resource === undefined || subscriber.resource === notice.topic.resource + ); + if (hearing.length === 0) return Promise.resolve(); + queueMicrotask(() => { + // One that unsubscribed since the publish no longer hears it. + for (const subscriber of hearing) if (subscribers.has(subscriber)) deliver(subscriber.listener, notice); + }); return Promise.resolve(); }, diff --git a/packages/engine/src/bus/publish-writes.ts b/packages/engine/src/bus/publish-writes.ts index 5eac8d7d97..fb189fe916 100644 --- a/packages/engine/src/bus/publish-writes.ts +++ b/packages/engine/src/bus/publish-writes.ts @@ -14,9 +14,10 @@ * A store is wrapped once, and every entry that wraps it gets that same * wrapper back. The engine keys per-store bookkeeping by store identity (live * request attempts, interrupted endings), so two wrappers of one store, one - * per entry, would split it in two. Wrapping a wrapper returns it; wrapping a - * store for a second bus adds that bus to the ones its wrapper publishes to. - * A write made on the raw store, past every entry, publishes nothing. + * per entry, would split it in two. Wrapping a wrapper returns it. A wrapped + * store publishes to one bus: an entry that brings a different bus for it is + * refused, since the views on one of the two would miss its writes. A write + * made on the raw store, past every entry, publishes nothing. * * ## What publishes what * @@ -29,9 +30,12 @@ * Not `persistEvents`, which feeds the request's own stream. A write that did * not land (a version conflict, a status predicate that failed) publishes * nothing. A failed publish never fails the write. Notices for one topic and - * kind raised in the same microtask turn go out as one. + * kind raised in the same microtask turn go out as one, at the end of that + * turn: the one deferral between a write and the bus. */ import { createInMemoryBus } from "./in-memory-bus"; +import { DEFAULT_RUNTIME_LOGGER } from "../execution/logging"; +import { FlowStateConfigError } from "../errors/flow-error"; import type { BusTopic, ChangeBus, ChangeNotice } from "./types"; import type { ChangeKind } from "@flow-state-dev/core"; import type { RequestRecord, RequestStore, SessionRecord, SessionStore, StoreRegistry } from "../stores/types"; @@ -40,8 +44,8 @@ import { resolveSessionStorageKey, toBareSessionId } from "../stores/scope-keys" type Wrapping = { /** Raw store → its one wrapper. */ wrappers: WeakMap; - /** Wrapper → the buses it publishes to. */ - buses: WeakMap; + /** Wrapper → the bus it publishes to. */ + buses: WeakMap; }; /** @@ -70,8 +74,8 @@ export function storeIdentity(store: object): object { return RAW.get(store) ?? store; } -/** The buses a store's wrapper publishes to: the store's own, or its wrapper's. */ -function busesOf(store: object): ChangeBus[] | undefined { +/** The bus a store's wrapper publishes to: the store's own, or its wrapper's. */ +function busOf(store: object): ChangeBus | undefined { const wrapper = WRAPPING.wrappers.get(store) ?? store; return WRAPPING.buses.get(wrapper); } @@ -82,15 +86,16 @@ function busesOf(store: object): ChangeBus[] | undefined { * publish to; else a new in-memory bus. Idempotent: a registry this returned * comes back unchanged. * + * @throws FlowStateConfigError when the registry's `bus` differs from the one + * its stores already publish to. * @returns The registry with `bus` set and its `request` and `session` stores * wrapped: `stores` itself when nothing changed, otherwise a copy. */ export function withChangeBus(stores: T): T & { bus: ChangeBus } { - const bus = - stores.bus ?? busesOf(stores.request)?.[0] ?? busesOf(stores.session)?.[0] ?? createInMemoryBus(); + const bus = stores.bus ?? busOf(stores.request) ?? busOf(stores.session) ?? createInMemoryBus(); const identities = new Identities(stores.session, stores.request); - const session = wrap(stores.session, bus, (buses) => sessionWrites(stores.session, buses, identities)); - const request = wrap(stores.request, bus, (buses) => requestWrites(stores.request, buses, identities)); + const session = wrap(stores.session, bus, () => sessionWrites(stores.session, bus, identities)); + const request = wrap(stores.request, bus, () => requestWrites(stores.request, bus, identities)); if (stores.bus === bus && session === stores.session && request === stores.request) { return stores as T & { bus: ChangeBus }; } @@ -111,30 +116,44 @@ function noticeKey(notice: ChangeNotice): string { return `${scope}\u0000${tenantId}\u0000${id}\u0000${resource ?? ""}\u0000${notice.kind}`; } -/** Queue `notices` for every bus in `buses`. Never throws. */ -function publish(buses: readonly ChangeBus[], notices: readonly ChangeNotice[]): void { - for (const bus of buses) { - let queued = pending.get(bus); - const first = queued === undefined; - if (queued === undefined) pending.set(bus, (queued = new Map())); - for (const notice of notices) queued.set(noticeKey(notice), notice); - if (!first) continue; - queueMicrotask(() => { - const batch = pending.get(bus); - pending.delete(bus); - for (const notice of batch?.values() ?? []) { - // Best effort (epic ER-6): a notice that fails to go out costs a view - // its wake, never the write its change. - void Promise.resolve() - .then(() => bus.publish(notice)) - .catch((error: unknown) => { - if (warnedPublishFailure) return; - warnedPublishFailure = true; - console.warn("[flow-state] a change notice failed to publish; later failures are not logged.", error); - }); +/** Queue `notices` for `bus`, to go out at the end of this microtask turn. Never throws. */ +function publish(bus: ChangeBus, notices: readonly ChangeNotice[]): void { + let queued = pending.get(bus); + const first = queued === undefined; + if (queued === undefined) pending.set(bus, (queued = new Map())); + for (const notice of notices) queued.set(noticeKey(notice), notice); + if (!first) return; + queueMicrotask(() => { + const batch = pending.get(bus); + pending.delete(bus); + for (const notice of batch?.values() ?? []) { + // Best effort (epic ER-6): a notice that fails to go out costs a view + // its wake, never the write its change. + try { + bus.publish(notice).catch(warnPublishFailure); + } catch (error) { + warnPublishFailure(error); } - }); - } + } + }); +} + +function warnPublishFailure(error: unknown): void { + if (warnedPublishFailure) return; + warnedPublishFailure = true; + console.warn("[flow-state] a change notice failed to publish; later failures are not logged.", error); +} + +/** + * A notice that could not be put together (the read naming a write's session + * failed) costs a view its wake: not worth a warning, as a failed publish is, + * but not silent either. + */ +function debugUnassembled(error: unknown, write: { store: "request" | "session"; id: string }): void { + DEFAULT_RUNTIME_LOGGER.debug?.("[flow-state] a change notice could not be assembled; no view hears this write.", { + ...write, + error + }); } /** A session's topic under its tenant. */ @@ -202,7 +221,10 @@ class Identities { async session(storageKey: string): Promise { const cached = cacheOf(sessionIdentities, this.sessions).get(storageKey); if (cached !== undefined) return cached; - const record = await this.sessions.get(storageKey).catch(() => undefined); + const record = await this.sessions.get(storageKey).catch((error: unknown) => { + debugUnassembled(error, { store: "session", id: storageKey }); + return undefined; + }); return record === undefined ? undefined : this.rememberSession(storageKey, record); } @@ -219,7 +241,10 @@ class Identities { async request(id: string): Promise { const cached = cacheOf(requestIdentities, this.requests).get(id); if (cached !== undefined) return cached; - const record = await this.requests.get(id).catch(() => undefined); + const record = await this.requests.get(id).catch((error: unknown) => { + debugUnassembled(error, { store: "request", id }); + return undefined; + }); return record === undefined ? undefined : this.rememberRequest(id, record); } @@ -258,19 +283,22 @@ type AnyMethod = (...args: any[]) => any; type Writes = Record; /** - * `store`'s one wrapper, publishing to `bus` among others: made on the first - * call from `writes`, the methods it replaces; every other property reads - * through to the store. + * `store`'s one wrapper, publishing to `bus`: made on the first call from + * `writes`, the methods it replaces; every other property reads through to + * the store. */ -function wrap(store: S, bus: ChangeBus, writes: (buses: ChangeBus[]) => Writes): S { +function wrap(store: S, bus: ChangeBus, writes: () => Writes): S { const known = WRAPPING.buses.has(store) ? store : (WRAPPING.wrappers.get(store) as S | undefined); if (known !== undefined) { - const buses = WRAPPING.buses.get(known)!; - if (!buses.includes(bus)) buses.push(bus); + if (WRAPPING.buses.get(known) !== bus) { + throw new FlowStateConfigError( + "These stores already publish their changes to another bus. A store publishes to one bus: " + + "resolve the stores once with the bus you want, and hand that registry to every entry." + ); + } return known; } - const buses = [bus]; - const replaced = writes(buses); + const replaced = writes(); const isReplaced = (key: string | symbol): key is string => typeof key === "string" && Object.prototype.hasOwnProperty.call(replaced, key); // A method assigned on the wrapper (an instrumented call, a test's race) @@ -292,7 +320,7 @@ function wrap(store: S, bus: ChangeBus, writes: (buses: Change } }); WRAPPING.wrappers.set(store, wrapper); - WRAPPING.buses.set(wrapper, buses); + WRAPPING.buses.set(wrapper, bus); RAW.set(wrapper, store); return wrapper; } @@ -314,38 +342,38 @@ function present(store: object, writes: Record const DELTA_METHODS = ["patchField", "incField", "pushToArray", "deleteField"] as const; -function sessionWrites(store: SessionStore, buses: ChangeBus[], identities: Identities): Writes { +function sessionWrites(store: SessionStore, bus: ChangeBus, identities: Identities): Writes { const delta = (original: AnyMethod) => async (id: string, ...rest: unknown[]) => { const result = await original(id, ...rest); if (result?.ok !== true) return result; const identity = result.record !== undefined ? identities.rememberSession(id, result.record) : await identities.session(id); - if (identity !== undefined) publish(buses, identities.sessionNotices(identity, "run")); + if (identity !== undefined) publish(bus, identities.sessionNotices(identity, "run")); return result; }; return present(store, { set: (original) => async (id: string, value: SessionRecord, expected: unknown) => { const result = await original(id, value, expected); - if (result?.ok === true) publish(buses, identities.sessionNotices(identities.rememberSession(id, value), "run")); + if (result?.ok === true) publish(bus, identities.sessionNotices(identities.rememberSession(id, value), "run")); return result; }, delete: (original) => async (id: string) => { const identity = await identities.session(id); await original(id); identities.forgetSession(id); - if (identity !== undefined) publish(buses, identities.sessionNotices(identity, "run")); + if (identity !== undefined) publish(bus, identities.sessionNotices(identity, "run")); }, ...Object.fromEntries(DELTA_METHODS.map((name) => [name, delta])) }); } -function requestWrites(store: RequestStore, buses: ChangeBus[], identities: Identities): Writes { +function requestWrites(store: RequestStore, bus: ChangeBus, identities: Identities): Writes { const after = (identity: RequestIdentity | undefined, kind: ChangeKind): void => { void identities .requestNotices(identity, kind) - .then((notices) => publish(buses, notices)) - .catch(() => {}); + .then((notices) => publish(bus, notices)) + .catch((error: unknown) => debugUnassembled(error, { store: "session", id: identity?.sessionId ?? "" })); }; const delta = (original: AnyMethod) => async (id: string, ...rest: unknown[]) => { const result = await original(id, ...rest); @@ -409,7 +437,7 @@ function requestWrites(store: RequestStore, buses: ChangeBus[], identities: Iden void landed.then(async (ok) => { if (background.get(requestId) === landed) background.delete(requestId); if (ok) after(await identities.request(requestId), "items"); - }).catch(() => {}); + }).catch((error: unknown) => debugUnassembled(error, { store: "request", id: requestId })); }, // A failed write rejects the flush a caller waits on, as the store's own // does, though the wrapper's background flush heard it first. diff --git a/packages/engine/test/bus/change-bus.test.ts b/packages/engine/test/bus/change-bus.test.ts index 7d43240776..dfc6d9be7e 100644 --- a/packages/engine/test/bus/change-bus.test.ts +++ b/packages/engine/test/bus/change-bus.test.ts @@ -3,7 +3,7 @@ * every request and session write publish on its session's topic once the * write can be read. */ -import { describe, expect, it } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import { DEFAULT_ORG_ID } from "@flow-state-dev/core"; import { createInMemoryBus, @@ -275,6 +275,46 @@ describe("withChangeBus · every request and session write publishes its session await tick(); }); + it("defers a write's notice once between write and bus, and the bus once per publish", async () => { + const { stores, bus } = wired(); + const heard = hear(bus, sessionTopic("s1")); + hear(bus, sessionTopic("s1")); + hear(bus, sessionTopic("s1")); + await stores.session.set("s1", session("s1"), "any"); + // The fold's microtask runs as the write resolves; then the in-memory + // bus's one, for all three subscribers. + let turns = 0; + while (heard.length === 0 && turns < 10) { + await Promise.resolve(); + turns += 1; + } + expect(turns).toBe(1); + expect(heard.map((n) => n.kind)).toEqual(["run"]); + }); + + it("debug-logs a notice it could not name, rather than losing it unheard", async () => { + const raw = createInMemoryStores(); + const stores = withChangeBus({ ...raw, bus: createInMemoryBus() }); + await raw.request.set("r1", request("r1", "s1"), "any"); + // The write names only an id; reading whose session it is fails. + raw.request.get = () => Promise.reject(new Error("read down")); + const debug = vi.spyOn(console, "debug").mockImplementation(() => {}); + const env = process.env.NODE_ENV; + process.env.NODE_ENV = "development"; + try { + await stores.request.setFieldsIfStatus("r1", { abortRequested: true }, ["in_progress"], Date.now()); + await tick(); + await tick(); + expect(debug).toHaveBeenCalledWith( + expect.stringContaining("change notice"), + expect.objectContaining({ error: expect.any(Error) }) + ); + } finally { + process.env.NODE_ENV = env; + debug.mockRestore(); + } + }); + it("folds one topic's notices of a kind raised in one turn into one", async () => { const { stores, bus } = wired(); await stores.request.set("r1", request("r1", "s1"), "any"); @@ -322,6 +362,13 @@ describe("withChangeBus · applied at every entry, once", () => { expect(heard.map((n) => n.kind)).toEqual(["request"]); }); + it("refuses a second bus for stores that already publish to one", () => { + const raw = createInMemoryStores(); + withChangeBus({ ...raw, bus: createInMemoryBus() }); + // One wrapped store publishes to one bus: a second would split who hears it. + expect(() => withChangeBus({ ...raw, bus: createInMemoryBus() })).toThrow(/already publish/); + }); + it("makes a raw registry's writes reach the host's bus when they share stores", async () => { const bus = createInMemoryBus(); const raw = createInMemoryStores(); diff --git a/packages/store-postgres/README.md b/packages/store-postgres/README.md index 964b122cc2..de75b7aec0 100644 --- a/packages/store-postgres/README.md +++ b/packages/store-postgres/README.md @@ -368,6 +368,41 @@ The liveness timeout (how long subscribers wait before yielding a synthetic `req - A dropped originating instance surfaces as a clean stream end with a synthetic `request.interrupted` event after the liveness timeout (default 30s). Clients can reconnect with `Last-Event-ID` to resume. - LISTEN/NOTIFY's scaling ceiling is per-NOTIFY-volume — see [Recall.ai's writeup](https://www.recall.ai/blog/postgres-listen-notify-does-not-scale). Below tens of thousands of simultaneous in-flight requests the pattern is fine; above that, look at Redis pub/sub or NATS. +## Cross-process notices + +The adapter backs the engine's `bus` slot: a change saved on one process reaches session views open on another as soon as it is saved, instead of at their next once-a-second read, and a view of a quiet session costs the database nothing. `postgresStores` declares the slot, so a profile on Postgres gets this with no extra config. `createPostgresStores` returns it as `registry.bus`. + +### How it works + +Every request and session write publishes a notice (a topic and a kind, never rows). The writing process's own views hear it at once, in process. It also goes out as `pg_notify('fsd_bus', …)` on the stores' pool, outside any transaction; the request tail's `flow_events` channel is not used. A notice too large for `NOTIFY` (8,000 bytes) is sent for its whole scope instead of one resource, which wakes more views than needed and never fewer. The first time that happens, a warning is logged. + +A process listens only while it has a view open. Its first subscriber checks one connection out of `liveTailPool`, issues `LISTEN fsd_bus`, and fans every notice out to that process's views; the last subscriber to leave hands it back. A process with no open view, such as a BullMQ worker or a migration script, publishes and holds no listening connection. With the default tail pool (`max: 10`), that leaves nine connections for request tails. + +### When it says it crosses processes + +A `LISTEN` that succeeds proves nothing on its own. A transaction-mode pooler (PgBouncer, or Supabase's or Neon's pooled endpoint) accepts it and never delivers a notification, and a connection a load balancer drops can go quiet without an error. So the bus pings itself through Postgres every 2 seconds and reports `crossesProcesses: true` only while it hears its own ping come back: + +- Until the first ping is heard, a new view uses the once-a-second read, and switches within one ping. +- If no ping comes back for 3 seconds, or the connection fails, the bus reports `false` and waiting views go back to the once-a-second read, within about 5 seconds of a silent drop. It reconnects with a capped backoff, never giving up, sends each view one `resync` notice so it reads once, and reports `true` again on the next ping it hears. +- If the first ping never comes back, the process logs one warning naming the likely cause. Pass `liveTailPool` a `pg.Pool` on a direct (session-mode) connection string. There is no other option for it: the auto-created tail pool uses the same connection string as the main pool. + +```ts +import pg from "pg"; +import { postgresStores } from "@flow-state-dev/store-postgres"; + +// Queries through the pooler; the listening connection goes direct. +const stores = postgresStores({ + connectionString: process.env.DATABASE_URL, // e.g. a transaction pooler + liveTailPool: new pg.Pool({ connectionString: process.env.DIRECT_DATABASE_URL, max: 10 }) +}); +``` + +### With no tail pool + +`liveTailPool: null`, the `{ pool }` shape without a `liveTailPool`, and the `{ executor }` shape (PGlite) never listen, so the bus reports `crossesProcesses: false` and views keep the once-a-second read. The `{ pool }` shape still sends its notices, so views on processes that do listen hear its writes. `vercelPostgresStores` from `@flow-state-dev/vercel/store` passes `liveTailPool: null` and doesn't back the `bus` slot. + +`close()` stops the ping and hands the listening connection back before it ends the pools. + ## Testing with PGlite For zero-infrastructure testing, use `@electric-sql/pglite` (embedded PostgreSQL via WASM): diff --git a/packages/store-postgres/src/bus.ts b/packages/store-postgres/src/bus.ts new file mode 100644 index 0000000000..ea9af64266 --- /dev/null +++ b/packages/store-postgres/src/bus.ts @@ -0,0 +1,369 @@ +/** + * The Postgres change bus (FIX-2427): notices cross processes on `LISTEN` / + * `NOTIFY`, through the Postgres the stores already use. + * + * - **Publish.** A notice reaches this process's own subscribers at once, in + * process, then goes out as `pg_notify` on the stores' pool, outside any + * transaction. Its echo, which comes back over `LISTEN`, is dropped by the + * bus's id, so a view in the publishing process wakes once, and still wakes + * while the listener is down. + * - **Listen.** One connection per process, checked out of the tail pool by + * the first subscriber and handed back when the last one leaves. A process + * with no open view (a BullMQ worker, `fsdev run`) publishes and holds no + * listener, ping or warning. + * - **Whether it crosses** (D1). A `LISTEN` that succeeds proves nothing: a + * transaction-mode pooler accepts it and delivers nothing, and a connection + * a load balancer drops goes quiet without an error. So the bus pings itself + * every 2 s the way a notice travels, and says it crosses processes only + * while it hears its own ping come back. Unheard for 3 s, or the connection + * lost, it says it does not, and the session stream goes back to its + * once-a-second read; it reconnects, sends every subscriber a `resync`, and + * says it crosses again on the next ping it hears. It never gives up. + * - **No tail pool, no crossing.** Without one (`{ executor }`, `{ pool }` + * without `liveTailPool`, or `liveTailPool: null`) the bus never listens + * and never says it crosses. + * + * One channel, filtered in process: the payload is the notice's topic and + * kind and the bus's id, nothing else. A payload over `NOTIFY`'s 8000 bytes + * drops its `resource` narrowing and goes out as the bare scope: a wider + * wake, never a lost one. + */ +import { randomUUID } from "node:crypto"; +import { createInMemoryBus, type BusTopic, type ChangeBus, type ChangeListener, type ChangeNotice } from "@flow-state-dev/engine"; +import type { Pool } from "pg"; +import type { QueryExecutor } from "./types"; +import { openListen, reconnectDelay, type ListenConnection } from "./listen"; + +/** The bus's own channel, apart from request tails' `flow_events`. */ +export const BUS_CHANNEL = "fsd_bus"; + +/** `NOTIFY` refuses a payload of 8000 bytes or more. */ +const MAX_PAYLOAD_BYTES = 7_999; + +const DEFAULT_TIMINGS: PostgresBusTimings = { + pingIntervalMs: 2_000, + staleAfterMs: 3_000, + reconnectMinMs: 100, + reconnectMaxMs: 30_000 +}; + +/** How often the bus pings, and how long unheard before it stops crossing. */ +export type PostgresBusTimings = { + /** A ping goes out this often while the listener is open. */ + pingIntervalMs: number; + /** Unheard for longer than this, the bus says it does not cross, and reconnects. */ + staleAfterMs: number; + /** First wait before a reconnect, doubling per failed attempt. */ + reconnectMinMs: number; + /** The longest wait between reconnects. */ + reconnectMaxMs: number; +}; + +/** The Postgres bus, with the `close` its registry calls before ending the pools. */ +export type PostgresBus = ChangeBus & { + /** Stop the ping and hand the listening connection back. Resolves once it is back. */ + close(): Promise; +}; + +export type PostgresBusOptions = { + /** The stores' own executor: `pg_notify` runs here. */ + executor: QueryExecutor; + /** The pool the listening connection comes from; `null` keeps the bus in process. */ + liveTailPool: Pool | null; + /** + * Whether notices go out over Postgres at all. `false` for an executor that + * is not a shared Postgres server (PGlite), whose process no other reaches. + */ + notify: boolean; + /** Test-only override of the ping and stale timings. */ + timings?: Partial; +}; + +let warnedNeverHeard = false; +let warnedOversized = false; +let warnedListenerFailure = false; + +/** Create the Postgres bus. */ +export function createPostgresBus(options: PostgresBusOptions): PostgresBus { + return new PgBus(options); +} + +type Subscription = { topic: BusTopic; listener: ChangeListener }; + +class PgBus implements PostgresBus { + readonly #id = randomUUID(); + readonly #executor: QueryExecutor; + readonly #pool: Pool | null; + readonly #notify: boolean; + readonly #timings: PostgresBusTimings; + /** This process's fan-out. */ + readonly #local = createInMemoryBus(); + readonly #subscriptions = new Set(); + readonly #crossingListeners = new Set<(crosses: boolean) => void>(); + + #crosses = false; + #closed = false; + /** Bumped on every start and stop, so a connect that finishes late knows it is stale. */ + #generation = 0; + #connection: ListenConnection | undefined; + /** Whether this run of the listener has held a connection before: the next one resyncs. */ + #reconnecting = false; + #attempt = 0; + #openedAt = 0; + #lastHeardAt = 0; + #everHeard = false; + #pingSeq = 0; + #ticker: ReturnType | undefined; + /** The connect in flight, if one is: `close` waits for it to hand its connection back. */ + #connecting: Promise | undefined; + #retry: { timer: ReturnType; resolve(): void } | undefined; + + constructor(options: PostgresBusOptions) { + this.#executor = options.executor; + this.#pool = options.liveTailPool; + this.#notify = options.notify; + this.#timings = { ...DEFAULT_TIMINGS, ...options.timings }; + } + + get crossesProcesses(): boolean { + return this.#crosses; + } + + onCrossingChange(listener: (crosses: boolean) => void): () => void { + this.#crossingListeners.add(listener); + return () => { + this.#crossingListeners.delete(listener); + }; + } + + async publish(notice: ChangeNotice): Promise { + // This process's views hear it at once, whatever the listener's state. + await this.#local.publish(notice); + if (!this.#notify || this.#closed) return; + await this.#executor.query("SELECT pg_notify($1, $2)", [BUS_CHANNEL, payloadOf(this.#id, notice)]); + } + + subscribe(topic: BusTopic, listener: ChangeListener): () => void { + const subscription: Subscription = { topic, listener }; + const stop = this.#local.subscribe(topic, listener); + this.#subscriptions.add(subscription); + if (this.#subscriptions.size === 1) this.#start(); + return () => { + stop(); + if (!this.#subscriptions.delete(subscription)) return; + if (this.#subscriptions.size === 0) void this.#stop(); + }; + } + + async close(): Promise { + this.#closed = true; + await this.#stop(); + } + + // ------------------------------------------------------------------------- + // The listener + // ------------------------------------------------------------------------- + + #start(): void { + if (this.#pool === null || this.#closed) return; + this.#generation += 1; + this.#reconnecting = false; + this.#attempt = 0; + this.#connecting = this.#connect(this.#generation); + } + + async #stop(): Promise { + this.#generation += 1; + this.#stopTicker(); + this.#retry?.resolve(); + this.#setCrossing(false); + const connection = this.#connection; + this.#connection = undefined; + await connection?.release(); + await this.#connecting; + } + + /** Open the listening connection, retrying with backoff until it opens or the run ends. */ + async #connect(generation: number): Promise { + const pool = this.#pool!; + while (generation === this.#generation) { + let connection: ListenConnection | undefined; + try { + connection = await openListen(pool, BUS_CHANNEL, { + notification: (payload) => this.#heard(payload), + lost: () => this.#lost(connection) + }); + } catch { + connection = undefined; + } + if (generation !== this.#generation) { + await connection?.release(); + return; + } + if (connection !== undefined && !connection.lost) { + this.#opened(connection); + return; + } + // Failed to open, or lost before it could be used. + await connection?.release(true); + this.#attempt += 1; + await this.#wait(reconnectDelay(this.#attempt, this.#timings.reconnectMinMs, this.#timings.reconnectMaxMs)); + } + } + + #opened(connection: ListenConnection): void { + this.#connection = connection; + this.#openedAt = Date.now(); + // A reconnect may have missed notices: every subscriber reads once. + if (this.#reconnecting) this.#resync(); + this.#reconnecting = true; + this.#startPinging(); + } + + /** The connection failed, or went quiet past the stale bound: stop crossing and reconnect. */ + #lost(connection: ListenConnection | undefined): void { + if (connection === undefined || this.#connection !== connection) return; + this.#connection = undefined; + this.#stopTicker(); + this.#setCrossing(false); + void connection.release(true); + const generation = this.#generation; + this.#attempt += 1; + this.#connecting = this.#wait( + reconnectDelay(this.#attempt, this.#timings.reconnectMinMs, this.#timings.reconnectMaxMs) + ).then(() => this.#connect(generation)); + } + + #startPinging(): void { + this.#stopTicker(); + this.#ping(); + this.#ticker = setInterval(() => this.#tick(), this.#timings.pingIntervalMs); + this.#ticker.unref?.(); + } + + #stopTicker(): void { + if (this.#ticker !== undefined) clearInterval(this.#ticker); + this.#ticker = undefined; + } + + /** Check the last ping heard, then send the next. */ + #tick(): void { + const connection = this.#connection; + if (connection === undefined) return; + const since = Math.max(this.#lastHeardAt, this.#openedAt); + if (Date.now() - since > this.#timings.staleAfterMs) { + if (!this.#everHeard) warnNeverHeard(); + this.#lost(connection); + return; + } + this.#ping(); + } + + #ping(): void { + this.#pingSeq += 1; + const payload = JSON.stringify({ p: this.#id, ping: this.#pingSeq }); + // A ping that fails to go out is a ping unheard: the stale rule answers it. + this.#executor.query("SELECT pg_notify($1, $2)", [BUS_CHANNEL, payload]).catch(() => {}); + } + + /** A payload on the bus channel: our own ping, our own echo, or another process's notice. */ + #heard(payload: string): void { + let message: unknown; + try { + message = JSON.parse(payload); + } catch { + return; + } + if (typeof message !== "object" || message === null) return; + const { p, ping } = message as { p?: unknown; ping?: unknown }; + if (p === this.#id) { + if (ping === undefined) return; // our own notice, already delivered in process + this.#lastHeardAt = Date.now(); + this.#everHeard = true; + this.#attempt = 0; + this.#setCrossing(true); + return; + } + if (ping !== undefined) return; // another process's ping + const notice = noticeOf(message); + if (notice !== undefined) void this.#local.publish(notice); + } + + #resync(): void { + for (const subscription of [...this.#subscriptions]) { + if (!this.#subscriptions.has(subscription)) continue; + const { topic, listener } = subscription; + try { + listener({ topic, kind: "resync" }); + } catch (error) { + if (warnedListenerFailure) continue; + warnedListenerFailure = true; + console.warn("[flow-state] a change-bus listener threw on resync; later failures are not logged.", error); + } + } + } + + #setCrossing(crosses: boolean): void { + if (this.#crosses === crosses) return; + this.#crosses = crosses; + for (const listener of [...this.#crossingListeners]) { + try { + listener(crosses); + } catch { + // A listener's failure is its own; the others still hear. + } + } + } + + /** Sleep `ms`, unless the run stops first. Never holds the process open. */ + #wait(ms: number): Promise { + return new Promise((resolve) => { + const done = (): void => { + clearTimeout(timer); + if (this.#retry?.timer === timer) this.#retry = undefined; + resolve(); + }; + const timer = setTimeout(done, ms); + timer.unref?.(); + this.#retry = { timer, resolve: done }; + }); + } +} + +/** The `NOTIFY` payload for `notice`: its topic and kind, and the bus's id. */ +function payloadOf(id: string, notice: ChangeNotice): string { + const { topic, kind } = notice; + const payload = JSON.stringify({ p: id, topic, kind }); + if (Buffer.byteLength(payload) <= MAX_PAYLOAD_BYTES || topic.resource === undefined) return payload; + if (!warnedOversized) { + warnedOversized = true; + console.warn( + "[flow-state] a change notice was too large for Postgres NOTIFY; it went out as its whole scope, waking every view of it. Later ones are not logged." + ); + } + const { resource: _resource, ...scope } = topic; + return JSON.stringify({ p: id, topic: scope, kind }); +} + +/** A notice read off the wire, or `undefined` when it isn't one. */ +function noticeOf(message: object): ChangeNotice | undefined { + const { topic, kind } = message as { topic?: Partial; kind?: unknown }; + if (topic === undefined || typeof kind !== "string") return undefined; + const { scope, id, tenantId, resource } = topic; + if (typeof scope !== "string" || typeof id !== "string" || typeof tenantId !== "string") return undefined; + if (resource !== undefined && typeof resource !== "string") return undefined; + return { + topic: { scope, id, tenantId, ...(resource !== undefined ? { resource } : {}) }, + kind: kind as ChangeNotice["kind"] + }; +} + +function warnNeverHeard(): void { + if (warnedNeverHeard) return; + warnedNeverHeard = true; + console.warn( + "[flow-state] the Postgres change bus never heard its own ping come back, so views on this process keep reading the store once a second. " + + "The likely cause is a transaction-mode pooler (PgBouncer, or Supabase's or Neon's pooled endpoint), which accepts LISTEN and delivers nothing. " + + "To fix it, pass `liveTailPool` a pg.Pool on a direct (session-mode) connection string." + ); +} diff --git a/packages/store-postgres/src/index.ts b/packages/store-postgres/src/index.ts index 6f4a88de56..acea9ff8e5 100644 --- a/packages/store-postgres/src/index.ts +++ b/packages/store-postgres/src/index.ts @@ -21,6 +21,7 @@ import { createPostgresSuspensionStore } from "./suspension-store"; import { createPostgresLeaseStore } from "./lease-store"; import { createPostgresWakeStore } from "./wake-store"; import { createPgPoolTx } from "./tx"; +import { createPostgresBus, type PostgresBus } from "./bus"; const DEFAULT_LIVE_TAIL_POOL_MAX = 10; @@ -33,7 +34,12 @@ function resolveLiveTailPoolMax(): number { } export type PostgresStoreRegistry = StoreRegistry & { - /** Drain the connection pool and disconnect */ + /** + * The Postgres change bus (FIX-2427). It crosses processes while it hears + * its own ping over the tail pool; without a tail pool it stays in process. + */ + bus: PostgresBus; + /** Hand back the bus's listening connection, then drain the pools and disconnect. */ close(): Promise; }; @@ -194,6 +200,8 @@ export async function createPostgresStores( liveTailPool = null; } + // Every write's notice goes out over this server; the listener needs the tail pool. + const bus = createPostgresBus({ executor, liveTailPool: liveTailPool ?? null, notify: true }); return { session: createPostgresSessionStore(executor), request: createPostgresRequestStore(executor, { liveTailPool }), @@ -209,7 +217,10 @@ export async function createPostgresStores( suspensions: createPostgresSuspensionStore(executor), leases: createPostgresLeaseStore(executor), wakes: createPostgresWakeStore(executor), + bus, async close() { + // The listener goes back to the tail pool before the pools end. + await bus.close(); await closePool(); } }; @@ -219,6 +230,15 @@ export async function createPostgresStores( await initializeSchema(executor); } + // `{ pool }` reaches a server every process shares, so its notices go out + // there even with no tail pool to hear them on. An `{ executor }` (PGlite) + // is this process's alone, unless a tail pool says otherwise. + const bus = createPostgresBus({ + executor, + liveTailPool: liveTailPool ?? null, + notify: !("executor" in options) || (liveTailPool ?? null) !== null + }); + return { session: createPostgresSessionStore(executor), request: createPostgresRequestStore(executor, { liveTailPool: liveTailPool ?? null }), @@ -234,21 +254,26 @@ export async function createPostgresStores( suspensions: createPostgresSuspensionStore(executor), leases: createPostgresLeaseStore(executor), wakes: createPostgresWakeStore(executor), + bus, async close() { + // The listener goes back to the tail pool before the pools end. + await bus.close(); await closePool(); } }; } /** - * Postgres store adapter for `createFlowState`. Backs the `primary` - * capability slot. Wraps `createPostgresStores`, memoizing the registry so - * lazy resolution opens the pool once, and disposes it via `close()`. + * Postgres store adapter for `createFlowState`. Backs the `primary` and `bus` + * capability slots: with no `bus` named in the profile, the Postgres bus + * carries change notices between processes. Wraps `createPostgresStores`, + * memoizing the registry so lazy resolution opens the pool once, and + * disposes it via `close()`. */ export function postgresStores(options: PostgresStoreOptions): StoreAdapter { let registry: PostgresStoreRegistry | undefined; return { - capabilities: ["primary"], + capabilities: ["primary", "bus"], async resolve() { registry ??= await createPostgresStores(options); return registry; @@ -277,3 +302,4 @@ export { initializeSchema } from "./schema"; export { createPostgresScheduleIndex } from "./schedule-index"; export { createPgPoolTx } from "./tx"; export type { PostgresStoreOptions, PoolConfig, QueryExecutor, TxClient } from "./types"; +export type { PostgresBus } from "./bus"; diff --git a/packages/store-postgres/src/listen.ts b/packages/store-postgres/src/listen.ts new file mode 100644 index 0000000000..921a7a12fe --- /dev/null +++ b/packages/store-postgres/src/listen.ts @@ -0,0 +1,88 @@ +/** + * One `LISTEN` connection, checked out of the tail pool: the part request + * tails (`LISTEN flow_events`) and the change bus (`LISTEN fsd_bus`) share. + * + * Each caller keeps its own loop and its own end. A request tail gives up + * after its reconnect budget; the bus never does, since a bus that stops is a + * process whose views stop. What they share is how a connection is opened, + * how its loss is heard, how it goes back to the pool, and the capped backoff + * between attempts. + */ +import type { Pool, PoolClient } from "pg"; + +/** A held `LISTEN` connection. */ +export type ListenConnection = { + /** Whether the connection failed (an error, or the server ended it). */ + readonly lost: boolean; + /** + * Hand the connection back: `UNLISTEN` and back to the pool when healthy, + * discarded when lost or `broken`. Safe to call more than once. + */ + release(broken?: boolean): Promise; +}; + +/** + * Check a connection out of `pool` and `LISTEN` on `channel`. + * + * @param handlers.notification Called with each payload on `channel`. + * @param handlers.lost Called once, when the connection fails after `LISTEN` succeeded. + * @throws When the checkout or the `LISTEN` fails; the connection is discarded first. + */ +export async function openListen( + pool: Pool, + channel: string, + handlers: { notification(payload: string): void; lost(): void } +): Promise { + const client: PoolClient = await pool.connect(); + let lost = false; + let listening = false; + let released = false; + + const onNotification = (msg: { channel: string; payload?: string }): void => { + if (msg.channel !== channel || msg.payload === undefined) return; + handlers.notification(msg.payload); + }; + const onLost = (): void => { + if (lost) return; + lost = true; + if (listening && !released) handlers.lost(); + }; + + const connection: ListenConnection = { + get lost() { + return lost; + }, + async release(broken = false) { + if (released) return; + released = true; + client.removeListener("notification", onNotification); + try { + if (!lost && !broken) await client.query(`UNLISTEN ${channel}`).catch(() => {}); + } finally { + client.removeListener("error", onLost); + client.removeListener("end", onLost); + // `true` makes `pg` discard a broken connection instead of handing a + // half-dead client to the next checkout. + client.release(lost || broken ? true : undefined); + } + } + }; + + client.on("notification", onNotification); + client.on("error", onLost); + client.on("end", onLost); + try { + await client.query(`LISTEN ${channel}`); + } catch (error) { + lost = true; + await connection.release(true); + throw error; + } + listening = true; + return connection; +} + +/** The wait before reconnect attempt `attempt` (1-based): doubling from `minMs`, capped at `maxMs`. */ +export function reconnectDelay(attempt: number, minMs: number, maxMs: number): number { + return Math.min(minMs * 2 ** Math.max(0, attempt - 1), maxMs); +} diff --git a/packages/store-postgres/src/request-store.ts b/packages/store-postgres/src/request-store.ts index 42bc8d33f5..f8b86a6599 100644 --- a/packages/store-postgres/src/request-store.ts +++ b/packages/store-postgres/src/request-store.ts @@ -38,8 +38,9 @@ import { type SetResult, type SubscribeToEventsOptions } from "@flow-state-dev/engine"; -import type { Pool, PoolClient } from "pg"; +import type { Pool } from "pg"; import type { QueryExecutor } from "./types"; +import { openListen, reconnectDelay, type ListenConnection } from "./listen"; import { createPgRecordStore, nullSafeEqualsClause, type PgCompanionWrite } from "./pg-store"; const DEFAULT_LIVENESS_TIMEOUT_MS = 30_000; @@ -750,7 +751,8 @@ export function createPostgresRequestStore( /** * `LISTEN flow_events` subscription on a dedicated checkout from - * `liveTailPool`. Implements the Notifier Pattern: NOTIFYs are signals + * `liveTailPool`, opened through the listen helper the change bus shares. + * Implements the Notifier Pattern: NOTIFYs are signals * only; the dirty-bit promise is resolved on each notification, and the * drain loop reads `getEvents(id, lastSeen)` once per cycle so N * notifications collapse into one query. @@ -779,35 +781,31 @@ async function* subscribeViaListen( let lastTickAt = Date.now(); while (!options.signal?.aborted) { - let client: PoolClient | undefined; + let connection: ListenConnection | undefined; let dirty = false; let dirtyResolve: (() => void) | undefined; let connectionFailed = false; - const onNotification = (msg: { channel: string; payload?: string }): void => { - if (msg.channel !== NOTIFY_CHANNEL) return; - if (msg.payload === undefined) return; - const colon = msg.payload.indexOf(":"); - const id = colon === -1 ? msg.payload : msg.payload.slice(0, colon); - if (id !== requestId) return; - dirty = true; - const resolve = dirtyResolve; - dirtyResolve = undefined; - resolve?.(); - }; - - const onClientError = (): void => { - connectionFailed = true; + const wake = (): void => { const resolve = dirtyResolve; dirtyResolve = undefined; resolve?.(); }; try { - client = await pool.connect(); - client.on("notification", onNotification); - client.on("error", onClientError); - await client.query(`LISTEN ${NOTIFY_CHANNEL}`); + connection = await openListen(pool, NOTIFY_CHANNEL, { + notification(payload) { + const colon = payload.indexOf(":"); + const id = colon === -1 ? payload : payload.slice(0, colon); + if (id !== requestId) return; + dirty = true; + wake(); + }, + lost() { + connectionFailed = true; + wake(); + } + }); attempt = 0; // Drain anything persisted between the catch-up SELECT and LISTEN setup. @@ -867,21 +865,8 @@ async function* subscribeViaListen( } catch { connectionFailed = true; } finally { - if (client !== undefined) { - try { - client.removeListener("notification", onNotification); - client.removeListener("error", onClientError); - if (!connectionFailed) { - await client.query(`UNLISTEN ${NOTIFY_CHANNEL}`).catch(() => {}); - } - } catch { - // Best-effort cleanup. The release call below decides whether the - // client returns to the pool clean or is discarded. - } - // Pass `true` for a broken connection so `pg` discards it instead of - // returning a half-dead client to the pool. - client.release(connectionFailed ? true : undefined); - } + // A broken connection is discarded rather than returned to the pool. + await connection?.release(connectionFailed); } if (options.signal?.aborted) return; @@ -894,10 +879,9 @@ async function* subscribeViaListen( `Postgres LISTEN connection unrecoverable after ${RECONNECT_BUDGET} attempts` ); } - const backoff = Math.min( - RECONNECT_BACKOFF_MIN_MS * 2 ** (attempt - 1), - RECONNECT_BACKOFF_MAX_MS + await abortableSleep( + reconnectDelay(attempt, RECONNECT_BACKOFF_MIN_MS, RECONNECT_BACKOFF_MAX_MS), + options.signal ); - await abortableSleep(backoff, options.signal); } } diff --git a/packages/store-postgres/test/fake-notify-network.ts b/packages/store-postgres/test/fake-notify-network.ts new file mode 100644 index 0000000000..8b022467ab --- /dev/null +++ b/packages/store-postgres/test/fake-notify-network.ts @@ -0,0 +1,104 @@ +/** + * A stand-in for a Postgres server's `LISTEN` / `NOTIFY`, for the bus's and + * the listen helper's unit tests: a pool whose clients `LISTEN`, an executor + * whose `pg_notify` reaches them, and the ways a real path fails (a pooler + * that delivers nothing, a connection killed, one that goes quiet). + */ +import { EventEmitter } from "node:events"; +import type { Pool } from "pg"; +import type { QueryExecutor } from "../src/types"; + +export class FakeClient extends EventEmitter { + readonly listening = new Set(); + /** Set on release: whether it was handed back broken. */ + released: { broken: boolean } | undefined; + /** A connection that stopped hearing, with no error. */ + silent = false; + readonly statements: string[] = []; + + constructor(private readonly network: FakeNotifyNetwork) { + super(); + } + + async query(text: string): Promise<{ rows: never[]; rowCount: number }> { + this.statements.push(text); + if (this.network.failListen) throw new Error("LISTEN refused"); + const [verb, channel] = text.split(/\s+/); + if (verb === "LISTEN") this.listening.add(channel!); + if (verb === "UNLISTEN") this.listening.delete(channel!); + return { rows: [], rowCount: 0 }; + } + + release(broken?: boolean): void { + if (this.released !== undefined) throw new Error("released twice"); + this.released = { broken: broken === true }; + this.network.open.delete(this); + } + + /** The server ends the backend, as `pg_terminate_backend` does. */ + kill(): void { + this.emit("error", new Error("terminating connection due to administrator command")); + this.emit("end"); + } +} + +export class FakeNotifyNetwork { + /** Clients checked out and not yet released. */ + readonly open = new Set(); + /** Every client ever checked out, in order. */ + readonly clients: FakeClient[] = []; + /** Every `pg_notify` the executor ran: channel and payload. */ + readonly notified: Array<{ channel: string; payload: string }> = []; + /** `false` is a transaction pooler: `LISTEN` succeeds, nothing is delivered. */ + deliver = true; + /** Checkouts still to fail. */ + failConnects = 0; + failListen = false; + + readonly executor: QueryExecutor = { + query: async (text: string, values?: unknown[]) => { + if (text.includes("pg_notify")) this.notify(String(values?.[0]), String(values?.[1])); + return { rows: [], rowCount: 0 }; + } + }; + + readonly pool = { + connect: async () => { + if (this.failConnects > 0) { + this.failConnects -= 1; + throw new Error("connect refused"); + } + const client = new FakeClient(this); + this.open.add(client); + this.clients.push(client); + return client; + } + } as unknown as Pool; + + notify(channel: string, payload: string): void { + this.notified.push({ channel, payload }); + if (!this.deliver) return; + setTimeout(() => { + for (const client of [...this.open]) { + if (client.silent || !client.listening.has(channel)) continue; + client.emit("notification", { channel, payload }); + } + }, 1); + } + + /** The clients holding `LISTEN` on `channel` now. */ + listeners(channel: string): FakeClient[] { + return [...this.open].filter((client) => client.listening.has(channel)); + } +} + +/** Wait until `check` holds, up to `ms`. */ +export async function until(check: () => boolean, ms = 1_000): Promise { + const deadline = Date.now() + ms; + while (!check()) { + if (Date.now() > deadline) throw new Error("timed out waiting"); + await new Promise((resolve) => setTimeout(resolve, 2)); + } +} + +export const delay = (ms: number): Promise => new Promise((resolve) => setTimeout(resolve, ms)); diff --git a/packages/store-postgres/test/listen-tail.test.ts b/packages/store-postgres/test/listen-tail.test.ts new file mode 100644 index 0000000000..242ac8deae --- /dev/null +++ b/packages/store-postgres/test/listen-tail.test.ts @@ -0,0 +1,52 @@ +/** + * Request tails on the listen helper they share with the change bus + * (FIX-2427): a tail still reconnects when its `LISTEN flow_events` + * connection is killed, and still gives up after its budget, which the bus + * never does. + */ +import { describe, expect, it } from "vitest"; +import { StoreSubscriptionError } from "@flow-state-dev/engine"; +import { createPostgresRequestStore, initializeSchema, type QueryExecutor } from "../src"; +import { FakeNotifyNetwork, until } from "./fake-notify-network"; +import { freshPglite } from "./shared-pglite"; + +async function executor(): Promise { + const pglite = await freshPglite(); + const exec: QueryExecutor = { + async query(text: string, values?: unknown[]) { + const result = await pglite.query(text, values); + return { rows: result.rows as Record[], rowCount: result.affectedRows ?? 0 }; + } + }; + await initializeSchema(exec); + return exec; +} + +describe("a request tail on the shared listen helper", () => { + it("reconnects when its listening connection is killed", async () => { + const network = new FakeNotifyNetwork(); + const store = createPostgresRequestStore(await executor(), { liveTailPool: network.pool }); + const controller = new AbortController(); + const tail = store.subscribeToEvents("r1", { fromSequence: 0, signal: controller.signal }); + const pending = tail.next(); + await until(() => network.listeners("flow_events").length === 1); + const first = network.listeners("flow_events")[0]!; + first.kill(); + await until(() => network.listeners("flow_events").length === 1 && network.clients.length === 2); + expect(first.released).toEqual({ broken: true }); + controller.abort(); + await pending; + await until(() => network.open.size === 0); + // Handed back healthy on the way out, so the pool keeps it. + expect(network.clients[1]!.released).toEqual({ broken: false }); + }); + + it("gives up after its reconnect budget", async () => { + const network = new FakeNotifyNetwork(); + network.failConnects = 100; + const store = createPostgresRequestStore(await executor(), { liveTailPool: network.pool }); + const tail = store.subscribeToEvents("r1", { fromSequence: 0 }); + await expect(tail.next()).rejects.toBeInstanceOf(StoreSubscriptionError); + expect(network.failConnects).toBe(100 - 6); + }); +}); diff --git a/packages/store-postgres/test/postgres-bus.test.ts b/packages/store-postgres/test/postgres-bus.test.ts new file mode 100644 index 0000000000..64f8f537b8 --- /dev/null +++ b/packages/store-postgres/test/postgres-bus.test.ts @@ -0,0 +1,289 @@ +/** + * The Postgres change bus (FIX-2427): when it says it crosses processes, what + * reaches a subscriber, and what it holds open. Against a fake `LISTEN` / + * `NOTIFY` network; the real two-process path is the goal check's + * (`goals/session-stream/wakes-across-processes/`). + * + * Why these matter: saying "crosses" turns the session stream's once-a-second + * read off. A wrong "yes" is a view that stops updating with no error, so + * every way the path can go quiet must turn the answer back to "no". + */ +import { afterEach, describe, expect, it, vi } from "vitest"; +import type { BusTopic, ChangeNotice } from "@flow-state-dev/engine"; +import { BUS_CHANNEL, createPostgresBus, type PostgresBus } from "../src/bus"; +import { FakeNotifyNetwork, delay, until } from "./fake-notify-network"; +import type { PGlite } from "@electric-sql/pglite"; +import { createPostgresStores, postgresStores, type QueryExecutor } from "../src"; +import { freshPglite } from "./shared-pglite"; + +const TIMINGS = { pingIntervalMs: 20, staleAfterMs: 50, reconnectMinMs: 5, reconnectMaxMs: 20 }; + +const s1: BusTopic = { scope: "session", id: "s1", tenantId: "" }; + +const open: PostgresBus[] = []; +afterEach(async () => { + for (const bus of open.splice(0)) await bus.close(); + vi.restoreAllMocks(); +}); + +function busOn(network: FakeNotifyNetwork, options: { tailPool?: boolean; notify?: boolean } = {}): PostgresBus { + const bus = createPostgresBus({ + executor: network.executor, + liveTailPool: options.tailPool === false ? null : network.pool, + notify: options.notify ?? true, + timings: TIMINGS + }); + open.push(bus); + return bus; +} + +function hear(bus: PostgresBus, topic: BusTopic): ChangeNotice[] { + const heard: ChangeNotice[] = []; + bus.subscribe(topic, (notice) => heard.push(notice)); + return heard; +} + +function crossings(bus: PostgresBus): boolean[] { + const seen: boolean[] = []; + bus.onCrossingChange((crosses) => seen.push(crosses)); + return seen; +} + +describe("the Postgres bus · when it says it crosses processes", () => { + it("says so only once its own ping comes back over LISTEN, and says it as an event", async () => { + const network = new FakeNotifyNetwork(); + const bus = busOn(network); + const seen = crossings(bus); + expect(bus.crossesProcesses).toBe(false); + hear(bus, s1); + await until(() => bus.crossesProcesses); + expect(seen).toEqual([true]); + }); + + it("never says so behind a pooler that accepts LISTEN and delivers nothing, and warns once with the fix", async () => { + const network = new FakeNotifyNetwork(); + network.deliver = false; + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + const bus = busOn(network); + const seen = crossings(bus); + hear(bus, s1); + await delay(TIMINGS.staleAfterMs * 4); + expect(bus.crossesProcesses).toBe(false); + expect(seen).toEqual([]); + const lines = warn.mock.calls.map((call) => String(call[0])).filter((line) => line.includes("ping")); + expect(lines).toHaveLength(1); + expect(lines[0]).toMatch(/pooler/); + expect(lines[0]).toMatch(/liveTailPool/); + }); + + it("stops saying so at once when the listening connection is killed, then reconnects, resyncs and says so again", async () => { + const network = new FakeNotifyNetwork(); + const bus = busOn(network); + const seen = crossings(bus); + const heard = hear(bus, s1); + await until(() => bus.crossesProcesses); + + const [first] = network.listeners(BUS_CHANNEL); + first!.kill(); + expect(bus.crossesProcesses).toBe(false); + expect(first!.released).toEqual({ broken: true }); + + // Back on a new connection: every subscriber reads once, since notices + // sent while it was down are gone. + await until(() => bus.crossesProcesses); + expect(network.listeners(BUS_CHANNEL)).toHaveLength(1); + expect(heard.map((n) => n.kind)).toEqual(["resync"]); + expect(heard[0]!.topic).toEqual(s1); + expect(seen).toEqual([true, false, true]); + }); + + it("stops saying so when the connection goes quiet without an error, within the stale bound plus a ping", async () => { + const network = new FakeNotifyNetwork(); + const bus = busOn(network); + hear(bus, s1); + await until(() => bus.crossesProcesses); + const quietAt = Date.now(); + network.listeners(BUS_CHANNEL)[0]!.silent = true; + await until(() => !bus.crossesProcesses); + expect(Date.now() - quietAt).toBeLessThanOrEqual(TIMINGS.staleAfterMs + TIMINGS.pingIntervalMs + 20); + // And it does not stay there: a fresh connection hears its ping again. + await until(() => bus.crossesProcesses); + }); + + it("never gives up reconnecting", async () => { + const network = new FakeNotifyNetwork(); + network.failConnects = 8; + const bus = busOn(network); + hear(bus, s1); + await until(() => bus.crossesProcesses, 2_000); + expect(network.failConnects).toBe(0); + }); + + it("never says so without a tail pool, and never opens a connection", async () => { + const network = new FakeNotifyNetwork(); + const bus = busOn(network, { tailPool: false }); + const heard = hear(bus, s1); + await bus.publish({ topic: s1, kind: "items" }); + await delay(TIMINGS.staleAfterMs * 2); + expect(bus.crossesProcesses).toBe(false); + expect(network.clients).toEqual([]); + // Its own views still hear it, and other processes still get it. + expect(heard.map((n) => n.kind)).toEqual(["items"]); + expect(network.notified.filter((n) => !n.payload.includes("ping"))).toHaveLength(1); + }); +}); + +describe("the Postgres bus · what a subscriber hears", () => { + it("hears another process's notice, and its own once, in process", async () => { + const network = new FakeNotifyNetwork(); + const here = busOn(network); + const there = busOn(network); + const heardHere = hear(here, s1); + const heardThere = hear(there, s1); + await until(() => here.crossesProcesses && there.crossesProcesses); + + await here.publish({ topic: s1, kind: "items" }); + await until(() => heardThere.length === 1); + await delay(10); + // Its echo came back over LISTEN and was dropped: one wake, not two. + expect(heardHere.map((n) => n.kind)).toEqual(["items"]); + expect(heardThere).toEqual([{ topic: s1, kind: "items" }]); + }); + + it("hears its own process's notice while the listener is down", async () => { + const network = new FakeNotifyNetwork(); + network.failConnects = 1_000; + const bus = busOn(network); + const heard = hear(bus, s1); + await bus.publish({ topic: s1, kind: "request" }); + await delay(1); + expect(heard.map((n) => n.kind)).toEqual(["request"]); + }); + + it("keeps tenants apart: one tenant's notice never reaches another's subscriber", async () => { + const network = new FakeNotifyNetwork(); + const here = busOn(network); + const there = busOn(network); + const t1 = hear(there, { ...s1, tenantId: "t1" }); + const t2 = hear(there, { ...s1, tenantId: "t2" }); + hear(here, s1); + await until(() => there.crossesProcesses && here.crossesProcesses); + await here.publish({ topic: { ...s1, tenantId: "t2" }, kind: "run" }); + await until(() => t2.length === 1); + await delay(10); + expect(t1).toEqual([]); + }); + + it("sends a notice too large for NOTIFY as its whole scope, never dropping it", async () => { + const network = new FakeNotifyNetwork(); + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + const here = busOn(network); + const there = busOn(network); + const heard = hear(there, s1); + hear(here, s1); + await until(() => there.crossesProcesses && here.crossesProcesses); + await here.publish({ topic: { ...s1, resource: "r".repeat(9_000) }, kind: "run" }); + await until(() => heard.length === 1); + expect(heard[0]).toEqual({ topic: s1, kind: "run" }); + expect(warn.mock.calls.some((call) => String(call[0]).includes("too large"))).toBe(true); + }); +}); + +describe("the Postgres bus · what it holds open", () => { + it("opens no connection until a subscriber does, and hands it back when the last one leaves", async () => { + const network = new FakeNotifyNetwork(); + const bus = busOn(network); + await bus.publish({ topic: s1, kind: "items" }); + await delay(TIMINGS.pingIntervalMs * 2); + // A process that only publishes (a worker) holds nothing and pings nothing. + expect(network.clients).toEqual([]); + expect(network.notified.filter((n) => n.payload.includes("ping"))).toEqual([]); + + const stopA = bus.subscribe(s1, () => {}); + const stopB = bus.subscribe({ ...s1, id: "s2" }, () => {}); + await until(() => bus.crossesProcesses); + // One connection for the process, however many views. + expect(network.clients).toHaveLength(1); + stopA(); + expect(network.open.size).toBe(1); + stopB(); + await until(() => network.open.size === 0); + expect(network.clients[0]!.released).toEqual({ broken: false }); + expect(network.clients[0]!.statements).toContain(`UNLISTEN ${BUS_CHANNEL}`); + expect(bus.crossesProcesses).toBe(false); + }); + + it("pings once per interval per process, not per view", async () => { + const network = new FakeNotifyNetwork(); + const bus = busOn(network); + hear(bus, s1); + hear(bus, { ...s1, id: "s2" }); + await until(() => bus.crossesProcesses); + const before = network.notified.length; + await delay(TIMINGS.pingIntervalMs * 10); + const pings = network.notified.slice(before).filter((n) => n.payload.includes("ping")).length; + expect(pings).toBeGreaterThanOrEqual(7); + expect(pings).toBeLessThanOrEqual(11); + }); + + it("closes with the connection handed back and the ping stopped, before it resolves", async () => { + const network = new FakeNotifyNetwork(); + const bus = busOn(network); + hear(bus, s1); + await until(() => bus.crossesProcesses); + await bus.close(); + expect(network.open.size).toBe(0); + const after = network.notified.length; + await bus.publish({ topic: s1, kind: "items" }); + await delay(TIMINGS.pingIntervalMs * 3); + expect(network.notified.length).toBe(after); + expect(bus.crossesProcesses).toBe(false); + }); + + it("closes cleanly while a connect is still being retried", async () => { + const network = new FakeNotifyNetwork(); + network.failConnects = 1_000; + const bus = busOn(network); + hear(bus, s1); + await delay(15); + await bus.close(); + network.failConnects = 0; + await delay(TIMINGS.reconnectMaxMs * 3); + expect(network.open.size).toBe(0); + }); +}); + +describe("the Postgres adapter · backs the bus slot", () => { + it("declares `bus`, so a profile on Postgres gets the Postgres bus with no config", async () => { + const pglite = await freshPglite(); + const adapter = postgresStores({ executor: pgliteExecutor(pglite) }); + expect(adapter.capabilities).toContain("bus"); + const registry = await adapter.resolve(["bus"]); + expect(registry.bus).toBeDefined(); + await adapter.dispose?.(); + }); + + it("keeps an executor's bus (PGlite) in process: it never says it crosses and sends nothing", async () => { + const pglite = await freshPglite(); + const statements: string[] = []; + const registry = await createPostgresStores({ executor: pgliteExecutor(pglite, statements) }); + const heard = hear(registry.bus, s1); + statements.length = 0; + await registry.bus.publish({ topic: s1, kind: "items" }); + await delay(5); + expect(heard.map((n) => n.kind)).toEqual(["items"]); + expect(registry.bus.crossesProcesses).toBe(false); + expect(statements.filter((text) => text.includes("pg_notify"))).toEqual([]); + await registry.close(); + }); +}); + +function pgliteExecutor(pglite: PGlite, statements?: string[]): QueryExecutor { + return { + async query(text: string, values?: unknown[]) { + statements?.push(text); + const result = await pglite.query(text, values); + return { rows: result.rows as Record[], rowCount: result.affectedRows ?? 0 }; + } + }; +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 3eb804e000..c51b93995b 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -734,6 +734,9 @@ importers: bullmq: specifier: ^5.0.0 version: 5.78.0 + pg: + specifier: ^8.13.0 + version: 8.20.0 zod: specifier: ^3.25.76 version: 3.25.76 @@ -741,6 +744,9 @@ importers: '@types/node': specifier: ^22.10.5 version: 22.19.13 + '@types/pg': + specifier: ^8.11.0 + version: 8.20.0 esbuild: specifier: 0.27.3 version: 0.27.3 From fc0613c8062efe07622d0d96866ebb96e6b18457 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 3 Nov 2026 03:01:24 +0000 Subject: [PATCH 2/5] FIX-2427: un-export the test-only FakeClient Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn --- packages/store-postgres/test/fake-notify-network.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/store-postgres/test/fake-notify-network.ts b/packages/store-postgres/test/fake-notify-network.ts index 8b022467ab..c945c41bb2 100644 --- a/packages/store-postgres/test/fake-notify-network.ts +++ b/packages/store-postgres/test/fake-notify-network.ts @@ -8,7 +8,7 @@ import { EventEmitter } from "node:events"; import type { Pool } from "pg"; import type { QueryExecutor } from "../src/types"; -export class FakeClient extends EventEmitter { +class FakeClient extends EventEmitter { readonly listening = new Set(); /** Set on release: whether it was handed back broken. */ released: { broken: boolean } | undefined; From 19da0cfb0b85d236b5654103e3f69969648ccb90 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 3 Nov 2026 03:02:34 +0000 Subject: [PATCH 3/5] FIX-2427: docs-editor fixes to the Postgres bus docs Gloss crossesProcesses and link the engine's Change bus section; state the notice's real guarantee (sent only once the write lands); give the silent-drop bound as computed; make the serverless advice agree with the liveTailPool fix. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn --- apps/docs/docs/persistence/overview.md | 6 +++--- packages/store-postgres/README.md | 10 +++++----- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/apps/docs/docs/persistence/overview.md b/apps/docs/docs/persistence/overview.md index 1d7ebb48f6..51140f3d64 100644 --- a/apps/docs/docs/persistence/overview.md +++ b/apps/docs/docs/persistence/overview.md @@ -17,7 +17,7 @@ The `bus` slot carries change notices. Every request and session write sends one | Crosses processes | Only after a change, at most once a second. A view of a quiet session costs the store nothing | | In memory | After a change, and also about once a second, so a reply written by another process still arrives | -The Postgres adapter backs `bus` (see [Postgres](#postgres)). The in-memory, file and SQLite adapters don't, so their profiles run the in-memory bus, and views keep the once-a-second read. +The Postgres adapter backs `bus` (see [Postgres](#postgres)). The in-memory, file and SQLite adapters don't, so their profiles run the in-memory bus, and views use the once-a-second read. | Adapter | Persistence | When to use | |---------|------------|-------------| @@ -76,7 +76,7 @@ const flowstate = createFlowState({ On Vercel, use `vercelPostgresStores()` from `@flow-state-dev/vercel/store` instead — it bakes in the pool tuning and Neon client swap. -With more than one process on one database (a web process and BullMQ workers, or several web processes), a session view open on one process sees a change saved on another as soon as it is saved, and a view of a quiet session costs the database nothing. You don't configure anything for this: the Postgres adapter backs the `bus` slot, and each process with an open view holds one listening connection and pings itself through it every 2 seconds. When the ping stops coming back, because a transaction pooler doesn't deliver `LISTEN` notifications or the connection dropped, views on that process go back to the once-a-second read until it recovers, and nothing is missed. `vercelPostgresStores()` doesn't back `bus`, so views there keep the once-a-second read. The [package README](https://github.com/fixpoint-labs/flow-state-dev/tree/main/packages/store-postgres#cross-process-notices) covers the connection, the pooler warning and `liveTailPool: null`. +With more than one process on one database (a web process and BullMQ workers, or several web processes), a session view open on one process sees a change saved on another as soon as it is saved, and a view of a quiet session costs the database nothing. You don't configure anything for this: the Postgres adapter backs the `bus` slot, and each process with an open view holds one listening connection and pings itself through it every 2 seconds. When the ping stops coming back, because a transaction pooler doesn't deliver `LISTEN` notifications or the connection dropped, views on that process go back to the once-a-second read until it recovers, and nothing is missed. `vercelPostgresStores()` doesn't back `bus`, so views there use the once-a-second read. The [package README](https://github.com/fixpoint-labs/flow-state-dev/tree/main/packages/store-postgres#cross-process-notices) covers the connection, the pooler warning and `liveTailPool: null`. Postgres provides the concurrency safety that compare-and-swap relies on. Compare-and-swap means a write carries the version it expects to find, and the store applies it only if that version is still current — so a write built on a stale read is refused instead of silently overwriting someone else's. @@ -412,7 +412,7 @@ Pick by where you run — there's no single "recommended stack": - **Local dev** — in-memory (default), or the filesystem store when you want data to survive restarts. Zero setup. - **Single server** (Railway, Fly, a VPS) — SQLite or Postgres, for durable concurrency-safe writes. The filesystem store is fine for very low concurrency, but its per-request event log doesn't hold up under real load — reach for SQLite instead. - **Multiple servers / cloud** — Postgres. You need a shared store with real concurrency semantics and cross-instance live tail (Postgres uses `LISTEN/NOTIFY`). `LISTEN/NOTIFY` is a wake-up signal with a real fan-out ceiling; at high event volume, put streaming on Redis or NATS behind the same `subscribeToEvents` interface. -- **Serverless / edge** — Postgres over an HTTP driver (e.g. a Neon/Supabase pooler). `LISTEN/NOTIFY` doesn't survive transaction poolers, so use an HTTP-based service (e.g. Upstash) for live updates. +- **Serverless / edge** — Postgres through a pooled endpoint (e.g. Neon or Supabase), with `vercelPostgresStores()` on Vercel. A transaction pooler doesn't deliver `LISTEN` notifications, so session views there use the once-a-second read. On a long-lived host behind a pooler, pass `liveTailPool` a direct connection string to get changes across processes as they are saved; see [Postgres](#postgres). - **Special requirements** — implement a custom store against the `StoreAdapter` interface. ### Resource vs. app-owned tables diff --git a/packages/store-postgres/README.md b/packages/store-postgres/README.md index de75b7aec0..5f8141e56c 100644 --- a/packages/store-postgres/README.md +++ b/packages/store-postgres/README.md @@ -374,16 +374,16 @@ The adapter backs the engine's `bus` slot: a change saved on one process reaches ### How it works -Every request and session write publishes a notice (a topic and a kind, never rows). The writing process's own views hear it at once, in process. It also goes out as `pg_notify('fsd_bus', …)` on the stores' pool, outside any transaction; the request tail's `flow_events` channel is not used. A notice too large for `NOTIFY` (8,000 bytes) is sent for its whole scope instead of one resource, which wakes more views than needed and never fewer. The first time that happens, a warning is logged. +Every request and session write publishes a notice (a topic and a kind, never rows). The writing process's own views hear it at once, in process. It also goes out as `pg_notify('fsd_bus', …)` on the stores' pool, and only once the write has landed: a write that fails, or is refused by a version or status check, sends no notice. A notice is best effort: if it fails to go out, the write still succeeds, and the view picks the change up at its next read. A notice too large for `NOTIFY` (8,000 bytes) is sent for its whole scope instead of one resource, which wakes more views than needed and never fewer. The first time that happens, a warning is logged. A process listens only while it has a view open. Its first subscriber checks one connection out of `liveTailPool`, issues `LISTEN fsd_bus`, and fans every notice out to that process's views; the last subscriber to leave hands it back. A process with no open view, such as a BullMQ worker or a migration script, publishes and holds no listening connection. With the default tail pool (`max: 10`), that leaves nine connections for request tails. -### When it says it crosses processes +### Detecting a pooler or a dropped connection -A `LISTEN` that succeeds proves nothing on its own. A transaction-mode pooler (PgBouncer, or Supabase's or Neon's pooled endpoint) accepts it and never delivers a notification, and a connection a load balancer drops can go quiet without an error. So the bus pings itself through Postgres every 2 seconds and reports `crossesProcesses: true` only while it hears its own ping come back: +The bus's `crossesProcesses` flag says whether a notice sent by another process reaches this one right now; while it is `true`, session views stop their once-a-second read (see the engine's [Change bus](../engine/README.md#change-bus)). A transaction-mode pooler (PgBouncer, or Supabase's or Neon's pooled endpoint) accepts it and never delivers a notification, and a connection a load balancer drops can go quiet without an error. So the bus pings itself through Postgres every 2 seconds and reports `crossesProcesses: true` only while it hears its own ping come back: - Until the first ping is heard, a new view uses the once-a-second read, and switches within one ping. -- If no ping comes back for 3 seconds, or the connection fails, the bus reports `false` and waiting views go back to the once-a-second read, within about 5 seconds of a silent drop. It reconnects with a capped backoff, never giving up, sends each view one `resync` notice so it reads once, and reports `true` again on the next ping it hears. +- If no ping comes back for 3 seconds, or the connection fails, the bus reports `false` and waiting views go back to the once-a-second read. After a silent drop, the bus reports `false` within 4 seconds of the last ping it heard, and a waiting view reads within a second after that, so about 5 seconds in all. It reconnects with a capped backoff, never giving up, sends each view one `resync` notice so it reads once, and reports `true` again on the next ping it hears. - If the first ping never comes back, the process logs one warning naming the likely cause. Pass `liveTailPool` a `pg.Pool` on a direct (session-mode) connection string. There is no other option for it: the auto-created tail pool uses the same connection string as the main pool. ```ts @@ -399,7 +399,7 @@ const stores = postgresStores({ ### With no tail pool -`liveTailPool: null`, the `{ pool }` shape without a `liveTailPool`, and the `{ executor }` shape (PGlite) never listen, so the bus reports `crossesProcesses: false` and views keep the once-a-second read. The `{ pool }` shape still sends its notices, so views on processes that do listen hear its writes. `vercelPostgresStores` from `@flow-state-dev/vercel/store` passes `liveTailPool: null` and doesn't back the `bus` slot. +`liveTailPool: null`, the `{ pool }` shape without a `liveTailPool`, and the `{ executor }` shape (PGlite) never listen, so the bus reports `crossesProcesses: false` and views use the once-a-second read. The `{ pool }` shape still sends its notices, so views on processes that do listen hear its writes. `vercelPostgresStores` from `@flow-state-dev/vercel/store` passes `liveTailPool: null` and doesn't back the `bus` slot. `close()` stops the ping and hands the listening connection back before it ends the pools. From c7e742873f101ec80b10ba4f40c457ab6af96c40 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 3 Nov 2026 03:16:33 +0000 Subject: [PATCH 4/5] FIX-2427: resync on the first connection; refuse a listener lost as it opens Codex on #4598. A notice sent after a view's first read and before the process's first LISTEN took effect was heard by nobody, and the first ping then said "crosses" and stopped the view's timed read: the view stayed stale. The bus now sends every subscriber a resync on each connection, the first included, before a ping can say it crosses. openListen could hand back a connection that failed between LISTEN answering and the helper resuming, with no loss callback; a request tail then waited out its liveness timeout instead of reconnecting. The helper now refuses it, as a failed LISTEN is refused, so every caller reconnects. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn --- .../wakes-across-processes/goal.md | 1 + packages/store-postgres/README.md | 2 +- packages/store-postgres/src/bus.ts | 15 +- packages/store-postgres/src/listen.ts | 10 +- .../test/fake-notify-network.ts | 14 +- .../store-postgres/test/listen-tail.test.ts | 29 ++++ .../test/postgres-bus-stream.test.ts | 158 ++++++++++++++++++ .../store-postgres/test/postgres-bus.test.ts | 32 +++- 8 files changed, 247 insertions(+), 14 deletions(-) create mode 100644 packages/store-postgres/test/postgres-bus-stream.test.ts diff --git a/goals/session-stream/wakes-across-processes/goal.md b/goals/session-stream/wakes-across-processes/goal.md index 7bfb0e526f..efbbd534a6 100644 --- a/goals/session-stream/wakes-across-processes/goal.md +++ b/goals/session-stream/wakes-across-processes/goal.md @@ -25,3 +25,4 @@ | 2026-11-03 | fix/FIX-2427-postgres-bus (pre-commit) | n/a | FAIL (expected) | `GOAL_CONTROL=no-listen`: `1:reply` and `1:run` never arrived within 3 s (also `3`, `4`, since the patch says "crosses" everywhere) | | 2026-11-03 | fix/FIX-2427-postgres-bus (pre-commit) | n/a | FAIL (expected) | `GOAL_CONTROL=trust-listen`: `3:drop` never arrived within 10 s, nor `3:back` | | 2026-11-03 | fix/FIX-2427-postgres-bus (pre-commit) | n/a | FAIL (expected) | `GOAL_CONTROL=no-bus` (today's `main`): `2:idle` 80 statements in 10 s for two views | +| 2026-11-03 | fix/FIX-2427-postgres-bus (after 19da0cf, resync on the first connection) | n/a | PASS · controls FAIL (expected) | 2: 0 view statements, 5 pings · 1: reply 110 ms, run 123 ms · 3: reply 453 ms after the drop; back, next 111 ms · 4: 479 ms. `no-listen` FAILs 1, `trust-listen` FAILs 3, `no-bus` FAILs 2 (80 statements) | diff --git a/packages/store-postgres/README.md b/packages/store-postgres/README.md index 5f8141e56c..624fccad27 100644 --- a/packages/store-postgres/README.md +++ b/packages/store-postgres/README.md @@ -382,7 +382,7 @@ A process listens only while it has a view open. Its first subscriber checks one The bus's `crossesProcesses` flag says whether a notice sent by another process reaches this one right now; while it is `true`, session views stop their once-a-second read (see the engine's [Change bus](../engine/README.md#change-bus)). A transaction-mode pooler (PgBouncer, or Supabase's or Neon's pooled endpoint) accepts it and never delivers a notification, and a connection a load balancer drops can go quiet without an error. So the bus pings itself through Postgres every 2 seconds and reports `crossesProcesses: true` only while it hears its own ping come back: -- Until the first ping is heard, a new view uses the once-a-second read, and switches within one ping. +- Until the first ping is heard, a new view uses the once-a-second read, and switches within one ping. When the listening connection opens, each view gets one `resync` notice and reads once, so a change saved after its first read and before the process was listening still arrives. - If no ping comes back for 3 seconds, or the connection fails, the bus reports `false` and waiting views go back to the once-a-second read. After a silent drop, the bus reports `false` within 4 seconds of the last ping it heard, and a waiting view reads within a second after that, so about 5 seconds in all. It reconnects with a capped backoff, never giving up, sends each view one `resync` notice so it reads once, and reports `true` again on the next ping it hears. - If the first ping never comes back, the process logs one warning naming the likely cause. Pass `liveTailPool` a `pg.Pool` on a direct (session-mode) connection string. There is no other option for it: the auto-created tail pool uses the same connection string as the main pool. diff --git a/packages/store-postgres/src/bus.ts b/packages/store-postgres/src/bus.ts index ea9af64266..a72d203123 100644 --- a/packages/store-postgres/src/bus.ts +++ b/packages/store-postgres/src/bus.ts @@ -18,7 +18,9 @@ * while it hears its own ping come back. Unheard for 3 s, or the connection * lost, it says it does not, and the session stream goes back to its * once-a-second read; it reconnects, sends every subscriber a `resync`, and - * says it crosses again on the next ping it hears. It never gives up. + * says it crosses again on the next ping it hears. It never gives up. The + * first connection sends the `resync` too: a notice sent between a view's + * first read and the `LISTEN` taking effect was heard by nobody. * - **No tail pool, no crossing.** Without one (`{ executor }`, `{ pool }` * without `liveTailPool`, or `liveTailPool: null`) the bus never listens * and never says it crosses. @@ -106,8 +108,6 @@ class PgBus implements PostgresBus { /** Bumped on every start and stop, so a connect that finishes late knows it is stale. */ #generation = 0; #connection: ListenConnection | undefined; - /** Whether this run of the listener has held a connection before: the next one resyncs. */ - #reconnecting = false; #attempt = 0; #openedAt = 0; #lastHeardAt = 0; @@ -167,7 +167,6 @@ class PgBus implements PostgresBus { #start(): void { if (this.#pool === null || this.#closed) return; this.#generation += 1; - this.#reconnecting = false; this.#attempt = 0; this.#connecting = this.#connect(this.#generation); } @@ -214,9 +213,11 @@ class PgBus implements PostgresBus { #opened(connection: ListenConnection): void { this.#connection = connection; this.#openedAt = Date.now(); - // A reconnect may have missed notices: every subscriber reads once. - if (this.#reconnecting) this.#resync(); - this.#reconnecting = true; + // Every connection, the first included, may have missed notices: one sent + // after a view's last read and before this LISTEN took effect is gone. So + // every subscriber reads once, and before the first ping heard can say + // "crosses" and stop the views' timed read. + this.#resync(); this.#startPinging(); } diff --git a/packages/store-postgres/src/listen.ts b/packages/store-postgres/src/listen.ts index 921a7a12fe..0c8fe511ca 100644 --- a/packages/store-postgres/src/listen.ts +++ b/packages/store-postgres/src/listen.ts @@ -26,7 +26,8 @@ export type ListenConnection = { * * @param handlers.notification Called with each payload on `channel`. * @param handlers.lost Called once, when the connection fails after `LISTEN` succeeded. - * @throws When the checkout or the `LISTEN` fails; the connection is discarded first. + * @throws When the checkout or the `LISTEN` fails, or the connection fails + * before it is handed back; the connection is discarded first. */ export async function openListen( pool: Pool, @@ -78,6 +79,13 @@ export async function openListen( await connection.release(true); throw error; } + // Failed after LISTEN answered and before this resumed: `lost` was not + // called (nobody held it yet), so a caller handed it would wait on a dead + // connection. Refuse it, as a LISTEN that failed is refused. + if (lost) { + await connection.release(true); + throw new Error(`the LISTEN ${channel} connection failed as it opened`); + } listening = true; return connection; } diff --git a/packages/store-postgres/test/fake-notify-network.ts b/packages/store-postgres/test/fake-notify-network.ts index c945c41bb2..1e0aedcafd 100644 --- a/packages/store-postgres/test/fake-notify-network.ts +++ b/packages/store-postgres/test/fake-notify-network.ts @@ -24,7 +24,15 @@ class FakeClient extends EventEmitter { this.statements.push(text); if (this.network.failListen) throw new Error("LISTEN refused"); const [verb, channel] = text.split(/\s+/); - if (verb === "LISTEN") this.listening.add(channel!); + if (verb === "LISTEN") { + await this.network.listenGate; + this.listening.add(channel!); + if (this.network.errorAfterListen > 0) { + this.network.errorAfterListen -= 1; + // The connection fails after LISTEN succeeded and before its caller resumes. + queueMicrotask(() => this.kill()); + } + } if (verb === "UNLISTEN") this.listening.delete(channel!); return { rows: [], rowCount: 0 }; } @@ -54,6 +62,10 @@ export class FakeNotifyNetwork { /** Checkouts still to fail. */ failConnects = 0; failListen = false; + /** LISTENs still to fail in the microtask after they succeed. */ + errorAfterListen = 0; + /** Every LISTEN waits on this before it takes effect. */ + listenGate: Promise = Promise.resolve(); readonly executor: QueryExecutor = { query: async (text: string, values?: unknown[]) => { diff --git a/packages/store-postgres/test/listen-tail.test.ts b/packages/store-postgres/test/listen-tail.test.ts index 242ac8deae..862cc57609 100644 --- a/packages/store-postgres/test/listen-tail.test.ts +++ b/packages/store-postgres/test/listen-tail.test.ts @@ -8,6 +8,7 @@ import { describe, expect, it } from "vitest"; import { StoreSubscriptionError } from "@flow-state-dev/engine"; import { createPostgresRequestStore, initializeSchema, type QueryExecutor } from "../src"; import { FakeNotifyNetwork, until } from "./fake-notify-network"; +import { openListen } from "../src/listen"; import { freshPglite } from "./shared-pglite"; async function executor(): Promise { @@ -22,6 +23,20 @@ async function executor(): Promise { return exec; } +describe("the listen helper", () => { + it("refuses a connection that failed before it could be handed over, rather than returning it dead", async () => { + const network = new FakeNotifyNetwork(); + network.errorAfterListen = 1; + let lostHeard = 0; + await expect( + openListen(network.pool, "flow_events", { notification: () => {}, lost: () => (lostHeard += 1) }) + ).rejects.toThrow(); + expect(network.clients[0]!.released).toEqual({ broken: true }); + // The caller never held it, so it isn't told it lost it. + expect(lostHeard).toBe(0); + }); +}); + describe("a request tail on the shared listen helper", () => { it("reconnects when its listening connection is killed", async () => { const network = new FakeNotifyNetwork(); @@ -41,6 +56,20 @@ describe("a request tail on the shared listen helper", () => { expect(network.clients[1]!.released).toEqual({ broken: false }); }); + it("reconnects when its connection fails just after LISTEN, before it resumes", async () => { + const network = new FakeNotifyNetwork(); + network.errorAfterListen = 1; + const store = createPostgresRequestStore(await executor(), { liveTailPool: network.pool }); + const controller = new AbortController(); + const tail = store.subscribeToEvents("r1", { fromSequence: 0, signal: controller.signal, livenessTimeoutMs: 60_000 }); + const pending = tail.next(); + // Not left waiting out its liveness timeout on a dead connection. + await until(() => network.clients.length === 2 && network.listeners("flow_events").length === 1, 2_000); + expect(network.clients[0]!.released).toEqual({ broken: true }); + controller.abort(); + await pending; + }); + it("gives up after its reconnect budget", async () => { const network = new FakeNotifyNetwork(); network.failConnects = 100; diff --git a/packages/store-postgres/test/postgres-bus-stream.test.ts b/packages/store-postgres/test/postgres-bus-stream.test.ts new file mode 100644 index 0000000000..a73405b667 --- /dev/null +++ b/packages/store-postgres/test/postgres-bus-stream.test.ts @@ -0,0 +1,158 @@ +/** + * The Postgres bus under the session stream (FIX-2427): a write that lands + * after a view's first read but before the view's process is listening must + * still reach the view. Until the bus hears its first ping it says it does + * not cross, so the view is on its timed read; the moment it says it does, + * the timer stops, and a change made in that gap would never be read unless + * the first connection tells every subscriber to read once. + * + * Two routers over one PGlite stand in for two processes; the fake network + * carries `NOTIFY`, and holds process 2's `LISTEN` until the write has landed. + */ +import { afterEach, describe, expect, it } from "vitest"; +import { DEFAULT_ORG_ID, defineFlow, handler } from "@flow-state-dev/core"; +import type { SessionStreamEvent } from "@flow-state-dev/core/items"; +import { createFlowApiRouter, createFlowRegistry, disposeFlowApiRouter, type StoreRegistry } from "@flow-state-dev/engine"; +import { PGlite } from "@electric-sql/pglite"; +import { createPostgresStores, type QueryExecutor } from "../src"; +import { createPostgresBus, type PostgresBus } from "../src/bus"; +import { FakeNotifyNetwork, delay, until } from "./fake-notify-network"; + +const TIMINGS = { pingIntervalMs: 20, staleAfterMs: 50, reconnectMinMs: 5, reconnectMaxMs: 20 }; + +function chatFlow() { + const say = handler({ + name: "chat-say", + execute: async (input, ctx) => { + ctx.emit.message((input as { text: string }).text); + return {}; + } + }); + return defineFlow({ kind: "chat", actions: { say: { block: say } } })(); +} + +function executorOver(pglite: PGlite): QueryExecutor { + return { + async query(text: string, values?: unknown[]) { + const result = await pglite.query(text, values); + return { rows: result.rows as Record[], rowCount: result.affectedRows ?? 0 }; + } + }; +} + +type Server = { router: ReturnType; stores: StoreRegistry; bus: PostgresBus }; + +async function server(pglite: PGlite, network: FakeNotifyNetwork, listens: boolean): Promise { + const stores = await createPostgresStores({ executor: executorOver(pglite) }); + const bus = createPostgresBus({ + executor: network.executor, + liveTailPool: listens ? network.pool : null, + notify: true, + timings: TIMINGS + }); + const registry = createFlowRegistry(); + registry.register(chatFlow()); + const withBus = { ...stores, bus }; + return { router: createFlowApiRouter({ registry, stores: withBus, staleSweepIntervalMs: 0 }), stores: withBus, bus }; +} + +async function say(on: Server, text: string): Promise { + const res = await on.router.POST( + new Request("http://localhost/api/flows/chat/s1/actions/say", { + method: "POST", + body: JSON.stringify({ userId: "alice", orgId: DEFAULT_ORG_ID, input: { text } }) + }), + { params: { path: ["chat", "s1", "actions", "say"] } } + ); + expect(res.status).toBe(202); + const id = ((await res.json()) as { request: { id: string } }).request.id; + const deadline = Date.now() + 3_000; + for (;;) { + const record = await on.stores.request.get(id); + if (record !== undefined && record.status !== "in_progress") return; + if (Date.now() > deadline) throw new Error("the reply never finished"); + await delay(5); + } +} + +function textOf(event: SessionStreamEvent): string | undefined { + if (event.type !== "session.item" || event.item.type !== "message") return undefined; + return (event.item.content as Array<{ text?: string }>)[0]?.text; +} + +async function openView(on: Server): Promise<{ events: SessionStreamEvent[]; close(): Promise }> { + const controller = new AbortController(); + const res = await on.router.GET(new Request("http://localhost/api/flows/sessions/s1/stream", { signal: controller.signal }), { + params: { path: ["sessions", "s1", "stream"] } + }); + expect(res.status).toBe(200); + const events: SessionStreamEvent[] = []; + const reader = res.body!.getReader(); + const decoder = new TextDecoder(); + const reading = (async () => { + let buffer = ""; + try { + for (;;) { + const { value, done } = await reader.read(); + if (done) return; + buffer += decoder.decode(value, { stream: true }); + for (let at = buffer.indexOf("\n\n"); at >= 0; at = buffer.indexOf("\n\n")) { + const data = buffer + .slice(0, at) + .split("\n") + .filter((line) => line.startsWith("data: ")) + .map((line) => line.slice(6)) + .join("\n"); + buffer = buffer.slice(at + 2); + if (data !== "") events.push(JSON.parse(data) as SessionStreamEvent); + } + } + } catch { + // Closed by the test. + } + })(); + return { + events, + async close() { + controller.abort(); + await reading; + } + }; +} + +const cleanup: Array<() => Promise> = []; +afterEach(async () => { + for (const step of cleanup.splice(0).reverse()) await step(); +}); + +describe("the Postgres bus under the session stream", () => { + it("reaches a view with a write made after its first read and before its process listens", async () => { + const pglite = new PGlite(); + const network = new FakeNotifyNetwork(); + let listen!: () => void; + network.listenGate = new Promise((resolve) => (listen = resolve)); + const p1 = await server(pglite, network, false); + const p2 = await server(pglite, network, true); + cleanup.push(async () => { + listen(); + await Promise.all([disposeFlowApiRouter(p1.router), disposeFlowApiRouter(p2.router)]); + await Promise.all([p1.bus.close(), p2.bus.close()]); + await pglite.close(); + }); + + await say(p1, "opening"); + const view = await openView(p2); + cleanup.push(() => view.close()); + // The view's first read is done; its process is not listening yet. + await until(() => view.events.some((e) => e.type === "session.runs")); + expect(network.listeners("fsd_bus")).toEqual([]); + + // Its notice goes out while nobody on process 2 listens, and is gone. + await say(p1, "the reply, in the gap"); + listen(); + await until(() => p2.bus.crossesProcesses); + + // No later write: only the first connection's resync can make the view read. + await until(() => view.events.some((e) => textOf(e) === "the reply, in the gap"), 3_000); + }); +}); diff --git a/packages/store-postgres/test/postgres-bus.test.ts b/packages/store-postgres/test/postgres-bus.test.ts index 64f8f537b8..2457a8f43d 100644 --- a/packages/store-postgres/test/postgres-bus.test.ts +++ b/packages/store-postgres/test/postgres-bus.test.ts @@ -37,12 +37,22 @@ function busOn(network: FakeNotifyNetwork, options: { tailPool?: boolean; notify return bus; } -function hear(bus: PostgresBus, topic: BusTopic): ChangeNotice[] { +/** Every notice `bus` delivers on `topic`, a connection's `resync` included. */ +function hearAll(bus: PostgresBus, topic: BusTopic): ChangeNotice[] { const heard: ChangeNotice[] = []; bus.subscribe(topic, (notice) => heard.push(notice)); return heard; } +/** The notices `bus` delivers on `topic` that some write sent: every `resync` left out. */ +function hear(bus: PostgresBus, topic: BusTopic): ChangeNotice[] { + const heard: ChangeNotice[] = []; + bus.subscribe(topic, (notice) => { + if (notice.kind !== "resync") heard.push(notice); + }); + return heard; +} + function crossings(bus: PostgresBus): boolean[] { const seen: boolean[] = []; bus.onCrossingChange((crosses) => seen.push(crosses)); @@ -50,6 +60,19 @@ function crossings(bus: PostgresBus): boolean[] { } describe("the Postgres bus · when it says it crosses processes", () => { + it("tells every subscriber to read once on its first connection, before it says it crosses", async () => { + const network = new FakeNotifyNetwork(); + const bus = busOn(network); + const order: string[] = []; + bus.onCrossingChange((crosses) => order.push(`crosses:${crosses}`)); + bus.subscribe(s1, (notice) => order.push(`s1:${notice.kind}`)); + bus.subscribe({ ...s1, id: "s2" }, (notice) => order.push(`s2:${notice.kind}`)); + await until(() => bus.crossesProcesses); + // A notice sent before the LISTEN took effect reached nobody here: each + // view reads once, while its timed read is still on. + expect(order).toEqual(["s1:resync", "s2:resync", "crosses:true"]); + }); + it("says so only once its own ping comes back over LISTEN, and says it as an event", async () => { const network = new FakeNotifyNetwork(); const bus = busOn(network); @@ -80,8 +103,9 @@ describe("the Postgres bus · when it says it crosses processes", () => { const network = new FakeNotifyNetwork(); const bus = busOn(network); const seen = crossings(bus); - const heard = hear(bus, s1); + const heard = hearAll(bus, s1); await until(() => bus.crossesProcesses); + expect(heard.map((n) => n.kind)).toEqual(["resync"]); const [first] = network.listeners(BUS_CHANNEL); first!.kill(); @@ -92,8 +116,8 @@ describe("the Postgres bus · when it says it crosses processes", () => { // sent while it was down are gone. await until(() => bus.crossesProcesses); expect(network.listeners(BUS_CHANNEL)).toHaveLength(1); - expect(heard.map((n) => n.kind)).toEqual(["resync"]); - expect(heard[0]!.topic).toEqual(s1); + expect(heard.map((n) => n.kind)).toEqual(["resync", "resync"]); + expect(heard[1]!.topic).toEqual(s1); expect(seen).toEqual([true, false, true]); }); From d9c31ceadea777d1ed68016bf044e789b9b35c40 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Tue, 3 Nov 2026 11:50:10 +0000 Subject: [PATCH 5/5] POC: one registry return in createPostgresStores The connection-string branch returned its own copy of the store list, including the bus. It now joins the return the other shapes already use. Schema init stays on the dedicated client. Co-authored-by: Jake Hoffner --- packages/store-postgres/src/index.ts | 38 +++++++--------------------- 1 file changed, 9 insertions(+), 29 deletions(-) diff --git a/packages/store-postgres/src/index.ts b/packages/store-postgres/src/index.ts index acea9ff8e5..79593dd6d0 100644 --- a/packages/store-postgres/src/index.ts +++ b/packages/store-postgres/src/index.ts @@ -63,6 +63,9 @@ export async function createPostgresStores( options: PostgresStoreOptions ): Promise { const skipSchemaInit = options.skipSchemaInit === true; + // A connection string initializes on a dedicated client, then uses the same + // registry as the other shapes. This keeps that path from initializing twice. + let schemaInitialized = skipSchemaInit; let executor: QueryExecutor; let closePool: () => Promise; // `null` = explicitly disabled (caller passed `liveTailPool: null`). @@ -177,6 +180,7 @@ export async function createPostgresStores( // run on the same pg connection. Use a dedicated client, not the pool. if (!skipSchemaInit) { await initializeSchemaWithDedicatedClient(pool); + schemaInitialized = true; } if (liveTailPool === undefined && poolConfig.connectionString) { @@ -199,40 +203,16 @@ export async function createPostgresStores( // (caller used `host` etc. without a URL). Fall back to polling. liveTailPool = null; } - - // Every write's notice goes out over this server; the listener needs the tail pool. - const bus = createPostgresBus({ executor, liveTailPool: liveTailPool ?? null, notify: true }); - return { - session: createPostgresSessionStore(executor), - request: createPostgresRequestStore(executor, { liveTailPool }), - user: createPostgresUserStore(executor), - org: createPostgresOrgStore(executor), - activeRequests: createPostgresActiveRequestRegistry(executor, { - sharedAcrossProcesses: registrySharedAcrossProcesses - }), - content: createPostgresContentStore(executor), - resourceState: createPostgresResourceStateStore(executor), - checkpoints: createPostgresCheckpointStore(executor), - traces: createInMemoryTraceStore(), - suspensions: createPostgresSuspensionStore(executor), - leases: createPostgresLeaseStore(executor), - wakes: createPostgresWakeStore(executor), - bus, - async close() { - // The listener goes back to the tail pool before the pools end. - await bus.close(); - await closePool(); - } - }; } - if (!skipSchemaInit) { + if (!schemaInitialized) { await initializeSchema(executor); } - // `{ pool }` reaches a server every process shares, so its notices go out - // there even with no tail pool to hear them on. An `{ executor }` (PGlite) - // is this process's alone, unless a tail pool says otherwise. + // `{ pool }` and a connection string reach a server every process shares, so + // their notices go out even with no tail pool to hear them on. An + // `{ executor }` (PGlite) is this process's alone, unless a tail pool says + // otherwise. const bus = createPostgresBus({ executor, liveTailPool: liveTailPool ?? null,