Skip to content

FIX-2427: the Postgres bus, so a change on one process reaches views on another - #4598

Merged
jhoffner merged 7 commits into
mainfrom
fix/FIX-2427-postgres-bus
Nov 3, 2026
Merged

jhoffner merged 7 commits into
mainfrom
fix/FIX-2427-postgres-bus

Conversation

@jhoffner

@jhoffner jhoffner commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

The problem

A team running a web process and BullMQ workers over one Postgres (or several web processes) sees a reply written on one process reach a view open on another only at that view's next once-a-second read. Every open session view also costs the database about 4 statements a second while nothing happens. The bus from FIX-2426 can stop that, but only a bus that actually crosses processes may turn the timed read off, and until now none did.

What this does

postgresStores now backs the bus slot, so a Postgres profile gets it with no config. A write's notice reaches the writing process's own views at once and goes out as pg_notify on the stores' pool. Each process with an open view holds one LISTEN connection from the tail pool and fans notices out to its views, which read once. A process with no open view (a worker, a script) only publishes.

The bus reports "crosses processes" only while it hears its own ping, sent every 2 s the way a notice travels. If the ping goes unheard for 3 s, or the connection fails, it reports "does not cross", and views go back to the once-a-second read. This happens behind a transaction pooler, after a silent drop, or after a killed backend. The bus then reconnects, tells each view to read once, and reports "crosses" again on the next ping. When the first ping never comes back, it logs one warning naming the fix. Without a tail pool (liveTailPool: null, { pool } alone, PGlite), it never listens and never reports "crosses".

sequenceDiagram
  participant P1 as process 1
  participant PG as Postgres
  participant L as process 2 listener
  participant V as process 2 views
  P1->>PG: write, then pg_notify('fsd_bus', topic + kind)
  PG-->>L: the notice, on the one LISTEN connection
  L->>V: wake every view of that topic
  V->>PG: one read
  L->>PG: ping every 2 s
  PG-->>L: ping heard, so it still crosses
Loading

The goal check passes on two forked processes over a real Postgres. A reply crossed in about 110 ms, and an idle view made 0 statements in 10 s (5 pings). After pg_terminate_backend on the listener, the reply arrived in about 450 ms. Each of the three controls fails the signal it targets.

What's asked of you

Nothing new: this implements the approved direction (FIX-2427 D1). One engine behaviour changed to keep a cleanup promised on #4594, and you may want to know about it. Stores already wrapped for one bus now throw if a later entry brings a different bus. Before, they published to both. Nothing in the repo did that; the test suites pass. If something outside the repo does, it gets a clear config error at startup instead of a view that silently misses half its writes.

Parts worth reviewing closely

packages/store-postgres/src/bus.ts: the crossing state machine

#connect / #opened / #lost / #tick / #stop. The question is whether any path leaves crossesProcesses true while no notice can arrive. Cases to check: a lost event racing openListen's return, an unsubscribe-then-subscribe while a connect is in flight, and close() during a retry wait. #generation is what makes each late connect hand its connection back.

The { pool } shape publishes with no tail pool

notify is true for { pool } even without liveTailPool, so a worker built that way still reaches web processes that listen. The spec says such a process "gets the bus in process with crossesProcesses false". That covers hearing. Publishing is the reading I took, because the alternative is a worker whose writes a crossing web view never hears (a "not done if" in the spec). The cost is one pg_notify per write on single-process { pool } hosts.

packages/store-postgres/src/listen.ts and the request-tail refactor

subscribeViaListen now checks out, LISTENs and releases through openListen, and backs off through reconnectDelay. Each loop keeps its own end: a tail gives up after 5 attempts, the bus never does. The helper also treats the client's end event as a loss. The old tail code heard only error. test/listen-tail.test.ts pins both ends.

packages/engine/src/bus/publish-writes.ts: one bus per wrapped store

The refusal above, plus one deferral hop and debug logging. See the cleanup ledger below.

Spec: specs/issues/FIX-2427/ · Linear FIX-2427 · epic FIX-2367 · builds on #4594 (FIX-2426, merged) · draft; implementation merge is the owner's

How to review this — spec-backed implementation: review the code against the approved direction

How to review this. This implements an approved spec — the approach and the numbered decisions in its DECISIONS.md are already signed off by a human, so please review the code against that direction, not the direction itself. The canonical set is retained at specs/issues/FIX-2427/ on main; Linear links to it and the original merged spec PR, which keeps review history. Follow-up amendment PRs explain later direction changes. Current architecture docs remain the authority for shipped behavior (BP-037).

Most valuable here: correctness on the second path (the legacy shape, the null boundary, the concurrent case, the cancel path), anything deriving an auth or routing decision from caller-controllable input, and a behaviour the tests assert around rather than on.

Already settled upstream: the approach and every decision in the spec's DECISIONS.md (human-approved — if one is wrong, say so as a spec finding and it gets folded back; don't re-argue it inline), and the conventions in docs/contributing/best-practices.md.

Engineering calls — decided along the way, no product sign-off needed
  • The listener closes when the last view leaves, not only on close(). The spec's "not done if a process with no open view holds a listening connection" reads that way. If wrong: a host whose views churn opens and closes one connection per churn; a linger timer would fix it.
  • A known connection loss reports "does not cross" at once, without waiting for the 3 s stale rule. The rule still catches the silent drop. If wrong: none, since this only shortens the stale window.
  • On stale, the bus recycles the connection (discards it and reconnects), with backoff from 100 ms doubling to 30 s, reset by a heard ping. Without the recycle, a silently dropped connection would never recover. Behind a pooler this means one reconnect per 30 s. If wrong: connection churn against a pooler; the cap is one constant.
  • The bus id is per bus instance (randomUUID), not the OS pid. Two Postgres registries in one process (tests, the FIX-2426 goal's shape) then hear each other over LISTEN as two processes would. If wrong: none in a one-registry process.
  • resync goes on every connection, the first included, before a ping can say "crosses". A notice sent between a view's first read and the first LISTEN taking effect reaches nobody, so the view must read once (Codex, r4450515613). The stream's onCrossingChange is left alone: the contract puts that read on the bus's resync.
  • openListen refuses a connection lost as it opens (Codex, r4450515621): one that fails between LISTEN answering and the helper resuming is discarded and thrown, like a failed LISTEN, so callers reconnect instead of waiting on it.
  • The ping goes on the stores' pool (pg_notify), the same path as a notice, not on the listening connection. It proves the publish path and the listen path together.
  • The goal counts process 2's statements client-side, through a pg.Client subclass passed as poolOptions.Client, which the auto-created tail pool inherits. That covers every connection process 2 opens and works on FSD_DB_URL. It is not log_statement, which needs the server's config.
  • goals gains a direct pg dependency for goals/lib/postgres.mts (FSD_DB_URL scratch database, pg_terminate_backend).
  • The docs prose was written in this session, not by the docs-writer / docs-editor agents CLAUDE.md asks for: this dispatch has no sub-agent tool. It was checked against the outsider rule by hand. Worth a polish-docs pass at epic wrap.
Everything else — the cleanups promised on #4594, deviations, verification, files

Cleanups promised on #4594

# Promise Done? Notes
1 Drop the buses[] multi-bus in publish-writes.ts Done A wrapped store publishes to one bus. A second, different bus for the same stores is a FlowStateConfigError (test: refuses a second bus for stores that already publish to one)
2 One deferral hop; one microtask per publish in in-memory-bus.ts Done The fold's microtask is the one hop between write and bus; it calls bus.publish directly (the Promise.resolve().then hop is gone). The in-memory bus queues one microtask per publish for all subscribers. A write's notice now reaches a subscriber 1 microtask after the write resolves, down from 2 (test: defers a write's notice once…, red on main with expected 2 to be 1). The bus's own microtask stays: a direct publish caller must never run listeners re-entrantly
3 Debug-log notice-assembly failures Done A failed read naming a write's session, and any rejection while assembling, goes to the runtime logger's debug (silent under NODE_ENV=test). A failed publish still warns once, as before
4 Share one crossing-bus test double between session-stream-conformance.ts and goals/lib/crossing-bus.mts Not done The Cursor review holds. The conformance suite ships inside the published @flow-state-dev/engine/testing, which can't import from goals/. The other direction needs the double exported from a package, which FIX-2426's decisions forbid ("no package exports it": no app option may make the in-memory bus cross). Two 20-line copies is the price of that rule
5a "Items landed" as a store-contract callback Weighed, not adopted It would change the contract of all five request-store adapters. The wrapper times items off flushItems, which resolves after Postgres commits the batch's upsert. The goal shows a cross-process reply in about 110 ms, of which 100 ms is the stream's settle. Nothing measured would get better
5b The Proxy vs a hand-written wrapper Weighed, not adopted Dropping buses[] removed the part that was hardest to follow. A hand-written wrapper must list every RequestStore/SessionStore method and track each contract addition. Today, an adapter's extra methods and a test's spies pass through untouched, and the identity and method-swap tests pin that

Deviations from the spec

  • Engine changes. The spec says "No engine change here". The engine diff here is only the cleanups above, which were promised on FIX-2426: the change bus, and the session stream wakes on it #4594 and dispatched with this issue. The bus contract, the stream and resolveStores are untouched.
  • { pool } without a tail pool still publishes (see Parts worth reviewing closely).
  • The listen helper is shared, but not the reconnect loop. The spec asked for one listen-with-reconnect helper, "or the PR says why". Checkout, LISTEN, loss detection, release and the capped backoff are one helper (listen.ts). Each caller keeps its own loop. The tail is an async generator that yields a gap read between LISTEN and its wait loop, and gives up after its budget. The bus is callback-driven and never gives up. One loop serving both would need a queue adapter around the generator, for no shared behaviour beyond what the helper already holds.
  • Timings as specified: ping every 2 s, stale after 3 s.

Verification

  • Goal check goals/session-stream/wakes-across-processes/ (initdb cluster, PG 16), 3 runs: PASS. 2: 0 view statements, 5 pings in 10 s idle, two views · 1: reply 105–117 ms, run 112–165 ms after it started · 3: 1 listener terminated, reply 431–468 ms after saved; back, next reply 111–117 ms · 4: reply 481–487 ms on the timed read.
  • Controls, each FAIL as required: no-listen FAILs 1:reply and 1:run (never arrived in 3 s). trust-listen FAILs 3:drop (never arrived in 10 s). no-bus (today's main: the adapter declares no bus) FAILs 2:idle with 80 statements in 10 s for two views, the 4/s per view FIX-2426 measured.
  • Unit red states: postgres-bus.test.ts fails 3 cases with the bus mutated to trust LISTEN (pooler, silent drop, ping count), and 1 with the self-echo drop removed. change-bus.test.ts: the hop test, the refusal test and the debug-log test were red before the cleanups.
  • pnpm typecheck (whole repo, including the boundary validators and goals): pass. eslint on the touched files: clean.
  • store-postgres suite: 25 files, 411 tests pass. The session-stream conformance suite passes on in-memory (66 + 2 skipped), SQLite (34) and Postgres/PGlite (34).
  • CI's test shard 2, run as CI runs it (turbo run test --concurrency=1 over store-postgres, workforce, engine, devtool, orchestration, store-sqlite, after packages:build): engine 323 files / 4300 tests pass, devtool, orchestration, store-sqlite and store-postgres pass. workforce failed 1 of 1738 (conversation-board-wait › a board run whose resume throws still delivers another row's owed notice). A rerun of the full workforce suite failed a different single test on a 5 s timeout (refuses a post's round at 50 open), with the load average near 9 from other agents on this 4-core box. Once the load fell to about 4, the suite passed 2/2 on this branch, and 1/1 with main's bus files swapped in. I read both failures as load timeouts, not this change. If CI's workforce shard flakes on either test, that is where to look.

Files

  • packages/store-postgres/src/bus.ts (new): the bus.
  • packages/store-postgres/src/listen.ts (new): the shared LISTEN helper.
  • packages/store-postgres/src/index.ts: registry.bus, close() order, capabilities: ["primary", "bus"].
  • packages/store-postgres/src/request-store.ts: tails on the helper.
  • packages/store-postgres/test/{postgres-bus,listen-tail}.test.ts, fake-notify-network.ts (new).
  • packages/engine/src/bus/{publish-writes,in-memory-bus}.ts, test/bus/change-bus.test.ts: cleanups 1–3.
  • goals/lib/postgres.mts, goals/lib/peers.mts (new): the two-process Postgres bootstrap FIX-2368's closure goal reuses.
  • goals/session-stream/wakes-across-processes/ (new): the goal, its peer and its controls.
  • Docs: packages/store-postgres/README.md (Cross-process notices), apps/docs/docs/persistence/overview.md (the Postgres paragraph, the bus lines), packages/engine/README.md (the bus line), goals/README.md (the two lib rows), docs/mechanisms/mechanisms.yaml (store-postgres as the change bus's adapter).
  • .changeset/postgres-bus.md: store-postgres minor, engine patch (FIX-2427).

https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn


Generated by Claude Code

…on another

postgresStores backs the bus slot. A notice reaches the writing process's
views at once and goes out as pg_notify('fsd_bus'); one LISTEN connection per
process with an open view, from the tail pool, fans notices out. The bus says
it crosses processes only while it hears its own ping (every 2 s, stale after
3 s), so behind a pooler or after a drop views keep the timed read; it
reconnects without giving up, resyncs, and warns once when the first ping
never comes back. Request tails share the new listen helper.

Engine cleanups promised on #4594: one bus per wrapped store (a second is
refused), one deferral hop and one microtask per publish, and notice-assembly
failures debug-logged instead of swallowed.

Goal check goals/session-stream/wakes-across-processes/ on two forked
processes over a real Postgres, with goals/lib/postgres.mts and peers.mts
shared with the closure goal (FIX-2368).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn
@vercel

vercel Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

4 Skipped Deployments
Project Deployment Actions Updated
flow-state-dev-docs Ignored Ignored Nov 3, 2026 12:03pm UTC
flow-state-dev-kitchen-sink Ignored Ignored Nov 3, 2026 12:03pm UTC
flow-state-dev-ui Ignored Ignored Preview Nov 3, 2026 12:03pm UTC
flow-state-dev-build-site Skipped Skipped Nov 3, 2026 12:03pm UTC

Request Review

@linear-code

linear-code Bot commented Nov 3, 2026

Copy link
Copy Markdown

FIX-2427

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn
Gloss crossesProcesses and link the engine's Change bus section; state the
notice's real guarantee (sent only once the write lands); give the silent-drop
bound as computed; make the serverless advice agree with the liveTailPool fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn

jhoffner commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Where I'm least sure:

  1. Liveness (bus.ts, listen.ts). Can crossesProcesses read true while notifications are not actually being delivered? Cases to check: a transaction pooler, a listener connection killed by pg_terminate_backend, and a reconnect that races a ping. Does the self-echo filter ever drop another process's notice?
  2. Listener connection lifecycle. Is the listener released on shutdown, and does the reconnect loop back off without leaking clients? Does a { pool } host without a tail pool still publish but never claim to cross?
  3. The engine cleanups (publish-writes.ts, in-memory-bus.ts). Check that dropping the multi-bus and collapsing to one deferral hop can't reorder a notice ahead of its write, or deliver it twice.

Generated by Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T12:05:26.619186Z 073cbda New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 19da0cfb0b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/store-postgres/src/bus.ts Outdated
Comment on lines +217 to +219
// A reconnect may have missed notices: every subscriber reads once.
if (this.#reconnecting) this.#resync();
this.#reconnecting = true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Resync subscribers when the initial LISTEN becomes ready

If another process writes after a session stream's initial read but before its asynchronous LISTEN completes, that notice is lost. The first connection skips resync, and its ping then enables crossesProcesses, disabling the stream's polling without scheduling another read: StreamWake.onCrossingChange only wakes the wait, without marking a pending notice. The view remains stale until another notice or stream reconnection. Send a resync on initial connection establishment too, before enabling crossing.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c7e7428. You were right: until the first ping is heard the view is on its timed read, and a notice sent in the gap between the view's first read and the LISTEN taking effect reached nobody. The ping then said "crosses" and the timer stopped.

What changed. #opened now sends every subscriber a resync on every connection, the first included (bus.ts #opened, and the header says why). That resync is delivered when the connection opens, before any ping can be heard, so the view marks a pending read while its timed read is still on. If a timed read happens to run between the resync and "crosses", that read began after LISTEN, so it already covers the gap.

Tests, red first:

  • test/postgres-bus-stream.test.ts: two routers over one PGlite. Process 2's LISTEN is held until a reply from process 1 has landed after the view's first read, then released, and no write follows. It asserts the view receives the reply. Without the fix it fails at that assertion. The full test is green with the fix.
  • test/postgres-bus.test.ts: tells every subscriber to read once on its first connection, before it says it crosses pins the order resync, resync, crosses:true. The kill case now expects two resyncs (first open, then the reconnect).

On StreamWake.onCrossingChange. I checked whether it should mark a pending read when crossing turns true, and left it alone. The bus contract puts "you may have missed some, read once" on the bus as a resync. With the first connection resyncing, the Postgres bus can't say "crosses" before every subscriber has been told to read. Marking a read in the stream as well would cost each view one extra read per transition, for a bus that already does this. It would also be a change to FIX-2426's stream, which this issue leaves alone. If a future bus can say "crosses" without that resync, the stream-side mark is the right guard to add then.

Also run on this head: the store-postgres suite (415/415), session-stream conformance on in-memory, SQLite and Postgres, and CI shard 2 (18/18 tasks). The two-process goal check passes, and each of its controls fails the signal it targets.


🤖 Generated with Claude Code

https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn


Generated by Claude Code

Comment on lines +808 to 809
});
attempt = 0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Check for listener loss before entering the request-tail loop

If the client errors after the LISTEN query resolves but before openListen resumes, the helper returns a connection with lost === true without invoking its loss callback, because listening was still false. Unlike the bus, this caller never checks connection.lost, so connectionFailed remains false and the request tail waits out its liveness timeout, then emits request.interrupted instead of reconnecting. This is reproducible with a query that queues an error microtask before returning a resolved promise. Check the returned connection's lost state before proceeding.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c7e7428, in the helper so every caller is covered. If the client fails after LISTEN answers and before openListen resumes, the loss callback is suppressed, because nobody held the connection yet. openListen now checks lost before handing the connection over. When it is set, the helper discards the connection and throws, the same way it treats a LISTEN that failed. The request tail's catch then counts a reconnect attempt and backs off, rather than waiting out its liveness timeout and emitting request.interrupted. The bus's #connect retries the same way.

Tests, red first. The fake network can now fail a connection in a microtask queued just before LISTEN resolves, which is the ordering you described.

  • test/listen-tail.test.ts › the listen helper refuses a connection that failed before it could be handed over: openListen rejects, discards the client as broken, and never calls lost. It was red, resolving { lost: true }.
  • test/listen-tail.test.ts › a request tail reconnects when its connection fails just after LISTEN, before it resumes: with a 60 s liveness timeout, a second client is LISTENing within 2 s. It was red, still waiting at 2 s.

🤖 Generated with Claude Code

https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn


Generated by Claude Code

…t opens

Codex on #4598. A notice sent after a view's first read and before the
process's first LISTEN took effect was heard by nobody, and the first ping then
said "crosses" and stopped the view's timed read: the view stayed stale. The
bus now sends every subscriber a resync on each connection, the first
included, before a ping can say it crosses.

openListen could hand back a connection that failed between LISTEN answering
and the helper resuming, with no loss callback; a request tail then waited out
its liveness timeout instead of reconnecting. The helper now refuses it, as a
failed LISTEN is refused, so every caller reconnects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn

jhoffner commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Kitchen-sink E2E (Playwright) failed at 19da0cf. The failure comes from main, not this PR: the same three specs fail at the same lines (background-work.spec.ts:17, talk-from-page.spec.ts:403, workforce-shell.spec.ts:82), and the other 45 pass. The cause is FIX-2506, and the fix is in #4596, which is green, Kitchen-sink E2E included. I won't port it into this PR. Once #4596 merges, I'll merge main into this branch so CI re-runs.


Generated by Claude Code

@jhoffner
jhoffner marked this pull request as ready for review November 3, 2026 11:42

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplify / architecture review

Verdict: The Postgres-only bus shape looks like the right trade for FIX-2427. Composing createInMemoryBus() (same-process fan-out + filtering) with pg_notify / LISTEN fsd_bus, plus self-ping to drive crossesProcesses, matches a real constraint: a successful LISTEN does not prove NOTIFY works through transaction poolers. The tests encode that failure mode; turning crossing on from "listener open" alone would disable the 1 Hz safety read while views stop updating.

Alternatives considered (and why I would not swap the whole approach):

Alternative Cleaner how? Why it does not replace this PR
NOTIFY-only, no ping Drop ~80 lines of ping/stale/reconnect Either keep crossesProcesses false (always poll) or risk silent pooler failure — the product contract wants auto crossing on direct connections with poll fallback.
npm pg-listen (or similar) Less hand-rolled LISTEN lifecycle Design already consolidated on listen.ts shared with request tails; adding a dependency does not remove ping/stale or resync policy.
Redis / NATS bus Battle-tested pub/sub Correct at scale, but a different adapter; this PR is deliberately "no new runtime" on the default Postgres path.
Multiplex flow_events + fsd_bus on one tail checkout One fewer connection per process Good follow-up optimization; not evidence the current split is wrong.

What reads as necessary complexity vs. optional trim:

  • Necessary: listen.ts extraction + infinite reconnect on the bus (vs tail budget 5), echo filtering by bus id, resync on every connect, oversized NOTIFY widening, withChangeBus single-bus enforcement and microtask coalescing — all trace to locked stream behavior.
  • Worth a follow-up (not blocking): duplicate #subscriptions set mirroring in-memory subscribers (needed because #local does not expose listeners for #resync); module-global warn-once flags (parallel tests); empty catch on LISTEN connect failures (operational blind spot); weak kind parsing off the wire; debugUnassembled metadata using store: "session" on a request-path failure.
  • Goal harness (~600 LOC): goals/lib/postgres.mts + peers.mts is a lot in one PR, but there was no shared real-Postgres bootstrap before; the cross-process goal is the right place to prove LISTEN/NOTIFY end-to-end. Minor goal-local reuse wins: sleep from goals/lib/time.mts, optional intentFreeEnv in forkPeer before FIX-2368 peers load generators.

Performance tradeoff to keep in mind: every process with an open view pings every 2s on fsd_bus, and Postgres fan-out delivers to all listeners — intentional liveness tax. Documented well in the file header; worth monitoring on large multi-process fleets.

Approving: direction and implementation hang together; inline notes are nits and follow-ups, not a different design.

Open in Web View Automation 

Sent by Cursor Automation: Review For Best Approach

subscribe(topic: BusTopic, listener: ChangeListener): () => void {
const subscription: Subscription = { topic, listener };
const stop = this.#local.subscribe(topic, listener);
this.#subscriptions.add(subscription);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate subscription registry. #subscriptions mirrors #local.subscribe because the in-memory bus does not expose its listeners, yet #resync must fan out to the same set. It works, but every subscribe/unsubscribe must keep two structures in sync — the main structural smell in this file.

Possible simplification (follow-up, not blocking): a tiny hook on createInMemoryBus (e.g. optional onSubscribe / iterate subscribers) so Postgres resync does not maintain a parallel Set. Only worth it if another adapter needs the same thing.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree this is a follow-up and doesn't block the merge. As you say, a hook on createInMemoryBus only pays off once a second adapter needs one, so the parallel set stays until then.


Generated by Claude Code

Comment thread packages/store-postgres/src/bus.ts Outdated
notification: (payload) => this.#heard(payload),
lost: () => this.#lost(connection)
});
} catch {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Empty catch on openListen failure swallows the root cause; reconnect loop keeps trying but operators only see backoff, not why LISTEN failed (auth, pool exhausted, etc.).

Suggestion: one debug log via DEFAULT_RUNTIME_LOGGER or a warn-once flag (same pattern as warnNeverHeard) — keeps behavior, improves operability.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A fair point, but it's marked optional, so it won't trigger a push on its own. If the PR gets another code push, a one-time warning that carries the first LISTEN error, like warnNeverHeard, will go in with it. I've written that change and its tests pass.


Generated by Claude Code

if (resource !== undefined && typeof resource !== "string") return undefined;
return {
topic: { scope, id, tenantId, ...(resource !== undefined ? { resource } : {}) },
kind: kind as ChangeNotice["kind"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kind as ChangeNotice["kind"] accepts any string from NOTIFY. Malformed or hostile payloads could wake views with nonsense kinds.

Suggestion: validate against the known ChangeKind set from core before publishing (cheap guard; unclear if this is worth strictness in practice).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this one. Sending a NOTIFY on fsd_bus already requires a database connection that can write the store. And a notice with an unknown kind only wakes a view, which then re-reads data it was already allowed to read. A bogus kind can't do more than that. A check against the allowed kinds would also need updating every time a kind is added, which isn't worth it for a wake.


Generated by Claude Code

.then((notices) => publish(buses, notices))
.catch(() => {});
.then((notices) => publish(bus, notices))
.catch((error: unknown) => debugUnassembled(error, { store: "session", id: identity?.sessionId ?? "" }));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a copy/paste slip: debugUnassembled is called with store: "session" on the requestNotices failure path. Metadata-only, but it will mislead anyone grepping debug logs for request assembly failures.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one stays as it is, and it isn't a slip. In debugUnassembled, store names the store whose read failed. On the request path, the only read in requestNotices that can fail is the session lookup for the parent (identities.session), and that lookup already catches its own error and logs store: "session" with the session's storage key. I checked by writing the "request" version with a test that makes that read fail: the debug line still came from the session lookup, with id: "s1". So the .catch here only guards against an unexpected throw, and "session" is the store involved even then.


Generated by Claude Code

const bus = createPostgresBus({
executor,
liveTailPool: liveTailPool ?? null,
notify: !("executor" in options) || (liveTailPool ?? null) !== null

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The notify: !("executor" in options) || (liveTailPool ?? null) !== null matrix is correct but dense on first read.

Optional: extract postgresBusNotifyEnabled(options) with a one-line comment on the three cases (pool-only, executor+tail, PGlite) — reduces cognitive load without changing behavior.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marked optional, so I'm leaving it for now. The expression is one line, and the comment above it already lists the three cases. If this area is touched again, the helper can come with that change.


Generated by Claude Code

client.on("error", onLost);
client.on("end", onLost);
try {
await client.query(`LISTEN ${channel}`);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LISTEN ${channel} / UNLISTEN ${channel} are safe today (BUS_CHANNEL, flow_events constants). If openListen ever takes a dynamic channel, this becomes an injection footgun.

Optional: allowlist or validate ^[a-z_][a-z0-9_]*$ since channels are identifiers, not arbitrary strings.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this one. openListen is internal to the package, not exported, and both of its callers pass constants. Validation can be added if a caller ever needs a dynamic channel.


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1f396b91a2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/store-postgres/src/bus.ts Outdated
Comment on lines +345 to +346
const { resource: _resource, ...scope } = topic;
return JSON.stringify({ p: id, topic: scope, kind });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wake narrowed subscribers when widening an oversized notice

When a notice exceeds the NOTIFY payload limit because of topic.resource, removing that field causes remote subscribers narrowed to that resource to miss the change. Remote delivery uses createInMemoryBus, whose filter only delivers to a narrowed subscriber when its resource exactly matches the notice's resource; an omitted resource is not a wildcard. This reproduces with a 9,000-character resource: the remote bus remains crossing but delivers no notice to the matching subscriber. Represent the widened wake explicitly and resync matching scope subscriptions, including resource-narrowed ones.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 073cbda. The in-memory fan-out gives a narrowed subscriber only an exact resource match (in-memory-bus.ts), and a widened notice names no resource. So remote views narrowed to that resource heard nothing, while the bus still said it crossed.

What changed. A widened payload now carries w: true. On receipt, #wakeScope wakes every local subscriber of that scope, id and tenant, narrowed or not, each on its own subscribed topic, in one microtask. Notices that aren't widened go through the in-memory bus exactly as before; its matching is unchanged.

Test, red first. In test/postgres-bus.test.ts › a notice too large for NOTIFY still reaches the other process's subscribers narrowed to its resource, a 9,000-character resource is published from one bus. The other bus's subscribers are checked:

  • the subscriber narrowed to that resource hears it, on its own topic;
  • one narrowed to another resource under the same scope hears it too (a wider wake);
  • one on another session hears nothing.

Before the fix it timed out with nothing delivered.

Also in this push: the first failure to open the listening connection now logs one warning carrying the error (warnListenFailed, in its own test file since the flag is per process). On this head, store-postgres (417/417), the engine bus tests, the repo-wide pnpm typecheck and the two-process goal check all pass.


🤖 Generated with Claude Code

https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Snob: the registry is built twice.

createPostgresStores returned the same store list from the connection-string branch and again for { pool } / { executor }. This PR copied the bus and the close() order into both. One return covers every shape. The connection-string path still initializes on a dedicated client; schemaInitialized stops the shared path from doing it again. Notify is unchanged: { pool } and a connection string still publish with no tail pool, an executor still does not.

POC, do not merge: #4605

Fold that subtraction into this branch, then close the POC unmerged after this lands.

I left the bus state machine, the split listen loops, and the goal helpers. Those are the contract, or this PR already recorded why they stay (the #4594 cleanups it weighed and did not take, and the separate listen loops).

View PR

Open in Web View Automation 

Sent by Cursor Automation: Code Snob

…oder-label test (FIX-2510)

FIX-2391 changed start() to take a readiness array and removed ACCOUNT;
FIX-2487 then added a test still using the old shape. Both merged, so
typecheck was red on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn
(cherry picked from commit 27c8abd)

jhoffner commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

The red Typecheck check comes from main, not from this PR. It fails at packages/shift-manager/test/harness-start.test.ts:176 (TS2304, ACCOUNT is not defined) because two shift-manager PRs collided on main. The fix is #4606 (FIX-2510), and I've copied it into this branch as e61160a9b so CI can go green here. Once #4606 merges, that commit becomes a no-op.


Generated by Claude Code

…STEN failure; one registry build

Codex on #4598 (r4454225127): a notice too large for NOTIFY went out without
its resource, and the receiving in-process fan-out gives a narrowed subscriber
only an exact resource match, so remote views narrowed to that resource never
heard it. A widened payload now carries `w: true`, and the receiver wakes every
subscriber of the scope, narrowed or not, each on its own topic.

The first failure to open the listening connection now warns once, with the
error. createPostgresStores builds its registry once for every shape (Cursor's
review, PoC #4605): the connection-string path still initializes on its
dedicated client and publishes as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn

jhoffner commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Taken in 073cbda: createPostgresStores now builds its registry once for every shape. #4605's index.ts comes over as-is. It deletes the connection-string branch's copy of the registry, and its bus and close() wiring (−29 lines). A 3-line schemaInitialized flag keeps that path from initializing twice.

I checked that behaviour is unchanged:

  • Initialization: the connection-string path still initializes on its dedicated client, and skipSchemaInit still skips both kinds.
  • Who publishes: !("executor" in options) || tailPool. A connection string or { pool } publishes; an { executor } publishes only with a tail pool.
  • The request store's tail pool: the request store gets the same tail pool (liveTailPool ?? null, where that branch had already resolved it to a pool or null).

The store-postgres suite passes with no test changed by the refactor (417/417), and the two-process goal check passes. #4605 stays open for you to close.


🤖 Generated with Claude Code

https://claude.ai/code/session_011UNkLEJU1iqxFYoPadTRfn


Generated by Claude Code

@jhoffner
jhoffner added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit e3a6279 Nov 3, 2026
20 checks passed
@jhoffner
jhoffner deleted the fix/FIX-2427-postgres-bus branch November 3, 2026 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants