Skip to content

About

Real-time sync for Cloudflare Durable Objects using TanStack DB — optimistic mutations, live queries, and single-ordered-stream write confirmation.

Topics

Resources

Stars

15 stars

Watchers

0 watching

Forks

Repository files navigation

tanstack-durable-object-sync

Real-time sync for your Cloudflare Durable Object using TanStack DB. Optimistic writes, live queries, and efficient reconnect catch-up. Your DO is the global source of truth, with zero extra infrastructure dependencies.

Your data already lives in a Durable Object, which is a strongly-consistent little SQLite database running close to your users. This library adds instantaneous sync to browsers in real time: subscribe to live queries, mutate it optimistically, and real-time changes sync via DO to clients. Simple, secure, and with low overhead.

It's built on TanStack DB — the reactive-store sibling of TanStack Query. If you've used useQuery, this will feel familiar: useLiveQuery for reads, collection.insert() / .update() / .delete() for writes. TanStack DB gives the client its reactive layer; this library handles the sync to your DO. It's as at home streaming an LLM's output token-by-token as it is doing ordinary CRUD.

It's a deliberately plain topology, each part doing what it does best. One authoritative writer keeps the change log totally ordered and contiguous, so a single cursor drives live deltas, reconnect catch-up, and write confirmation alike — no second ack channel, no CRDT to merge, no Postgres to mirror. The Durable Object holds authoritative state and assigns order; TanStack DB gives the client its reactive layer (live queries, IVM, optimistic rollback); this library carries the diffs between. You stop trading one good thing for another — optimistic CRUD and a single source of truth, a simple transport and a fully reactive client, at once.


Why this exists

If you reach for sync on Cloudflare today, the good options each ask you to give something up. CRDT engines — Cloudflare's own PartyKit — are superb for collaborative editing, but they're Yjs-shaped (merge semantics, document baggage) with a thin authorization story. Zero is excellent for authz-filtered CRUD but wants a Postgres to mirror into and doesn't run on a DO. LiveStore does sync against DO SQLite, but it's event-log-based: events are what's stored at rest, so a token-by-token LLM stream grows the log without bound, and there's no plain SQL or ORM underneath. So this library takes the third path — the DO is the source of truth, its own SQLite is what's stored, and the entire client-side reactive layer (live queries, incremental view maintenance, optimistic rollback) comes from TanStack DB for free.

This library
Source of truth The Durable Object's own SQLite. No Postgres, no external sync service.
Transport One WebSocket per DO. Hibernation-native.
Writes Bidirectional. Optimistic on the client; authoritative in the DO.
Confirmation A position in the one stream the client already tails — no second ack channel.
Reads Live queries via TanStack DB's client-side IVM. The DO never joins or aggregates.
Consistency Server-authoritative. Single writer per DO. No CRDTs, no multi-DO transactions.

The model in 30 seconds

  • One DO instance = one sync scope (a session, a workspace, a document — whatever you shard by). It owns a SQLite database with one or more collections.
  • Change-data-capture via triggers. Every write to a collection table fires a trigger appending to a single per-DO change log. That log is the one ordered stream — the source of truth for live deltas, reconnect catch-up, and write confirmation.
  • One cursor. The client tracks a single position (appliedSeq). A write is confirmed when that position passes the sequence the DO assigned the write — exactly Electric's awaitTxId, reduced to a >= comparison because a single writer produces a contiguous log.
  • Client-supplied keys. The client mints the primary key (ULID / UUIDv7), so the optimistic row and the confirmed row are the same row — the write applies locally under the chosen id and the server confirms that same id, with no key reconciliation or id swap on commit.
  • Bounded retention. The change log stays light: compaction keeps only the latest op per key, and changes are swept after a retention window (2 days by default, configurable). A client reconnecting from beyond that window gets a fresh snapshot instead of a delta. Bounded storage, no event-log explosion.

See ADR-0001 for the full rationale, and the build plan for the milestone sequence.


Quick start

Three files: the Durable Object that owns the data, the Worker that fronts it and stamps trusted claims, and the browser. Here's the whole stack.

1. Define your Durable Object

import { defineSync, SyncDurableObject } from "tanstack-durable-object-sync"

interface Claims { userId: string }
interface Env { /* your bindings */ }
interface Message { id: string; author: string; content: string; created_at: number }

// defineSync binds identity (Claims) and binding-env (Env) once and returns
// three co-located helpers. They flow `user`/`env` into every handler ctx.
const sync = defineSync<Claims, Env>()

// The schema VALUE is both the DO registration and the client contract. The
// collection KEY ("messages") is the DB table name. `pk` must be a real column
// of Row — the sole TEXT, client-supplied key (ADR-0007).
export const chatSchema = sync.schema({
  collections: {
    messages: sync.collection<Message>({
      pk: "id",
      // The closed mutation trio { insert?; update?; delete? } — a 4th key is a
      // type error. op.cols is typed per op: full Row on insert, Partial on
      // update, absent on delete.
      mutations: {
        insert: {
          // authorize runs BEFORE the tx (async ok); throw to deny.
          // op.cols is typed Message here — no cast.
          authorize: ({ user, op }) => {
            if (op.cols.author !== user.userId) {
              throw new Error("author mismatch")
            }
          },
          // execute runs INSIDE transactionSync — synchronous only.
          execute: ({ op, sql }) => {
            const m = op.cols // Message
            sql.exec(
              "INSERT INTO messages(id, author, content, created_at) VALUES (?, ?, ?, ?)",
              m.id, m.author, m.content, m.created_at,
            )
          },
          // afterCommit (optional): fire-and-forget AFTER the commit + receipt —
          // the home for external side effects execute can't do (delete an R2
          // object, enqueue a job). Receives `env`; owns its own idempotency.
          // afterCommit: async ({ op, env }) => { await env.BUCKET.delete(op.key) },
        },
        delete: {
          execute: ({ op, sql }) => {
            // delete carries op.key only — no op.cols.
            sql.exec("DELETE FROM messages WHERE id = ?", op.key)
          },
        },
      },
    }),
  },
  // Commands are the escape hatch for writes that aren't a single typed row op.
  // Their own SQL still flows through the CDC triggers, and they can return a
  // result. Type-only Args is curried (call the factory twice); Result is
  // inferred from the return. If you have commands you MUST declare them inline
  // here so Args/Result inference flows into the Api type.
  commands: {
    clearRoom: sync.command()(({ sql }) => {
      const before = Array.from(sql.exec("SELECT count(*) AS c FROM messages"))[0]!.c as number
      sql.exec("DELETE FROM messages")
      return { deleted: before }
    }),
  },
})

// Export the schema type as the client contract.
export type Api = typeof chatSchema

export class SessionDO extends SyncDurableObject<Env, Claims> {
  constructor(ctx: DurableObjectState, env: Env) {
    super(ctx, env)

    // You own your schema — migrate with anything (raw DDL, Drizzle, …), then
    // call registerSync to wire CDC. blockConcurrencyWhile runs it before the
    // first request.
    ctx.blockConcurrencyWhile(async () => {
      this.sql.exec(`CREATE TABLE IF NOT EXISTS messages (
        id         TEXT PRIMARY KEY,        -- client-supplied TEXT key (ULID/UUIDv7)
        author     TEXT NOT NULL,
        content    TEXT NOT NULL,
        created_at INTEGER NOT NULL
      )`)

      // registerSync takes the schema VALUE — it compiles it, validates pk
      // affinity, and wires the CDC triggers.
      this.registerSync(chatSchema)
    })
  }

  // Read the Worker-forged claims header into the per-socket attachment.
  protected parseAttachment(req: Request): Claims {
    return JSON.parse(req.headers.get("x-claims") ?? "{}") as Claims
  }
}

Important

Schema & migrations. You own the table — create it with anything (raw CREATE TABLE, Drizzle, a versioned migrator), then call registerSync to wire CDC. The pk must have TEXT affinity (TEXT, VARCHAR, CHAR, …) so it stores the client-supplied id verbatim; an INTEGER key is rejected — it aliases rowid (server-assigned) and breaks optimistic id parity. registerSync also rejects a WITHOUT ROWID table, a column named rowid (generated or not), and a column named __proto__. It throws, and the whole object fails on every wake, so rename such a column in a migration that runs before registerSync (ADR-0007). Evolve freely: the CDC triggers capture only the row key, so ALTER TABLE ADD COLUMN flows to clients with no re-wiring, and re-running registerSync on the next deploy is idempotent (ADR-0007). A table or command name over 128 bytes fails registerSync (ADR-0012).

Note

Server-side writes outside the client flow — an agent inserting a row, a webhook, a cron job, a bulk seed — go through this.runSyncedWrite(sql => …): it applies your write and broadcasts it to connected clients (ADR-0006).

2. Route the upgrade from your Worker (the trust boundary)

This Worker fronts every /sync/<sessionId> WebSocket upgrade: match the path, authenticate, then forge the claims header and hand off to the right DO.

export default {
  async fetch(req: Request, env: Env): Promise<Response> {
    // Only handle /sync/<sessionId> — the sessionId is the DO shard key.
    const match = new URL(req.url).pathname.match(/^\/sync\/(.+)$/)
    if (!match) return new Response("not found", { status: 404 })
    const sessionId = match[1]!

    // The trust boundary: authenticate here, then stamp claims the DO can trust.
    const claims = await verifyToken(req) // your auth
    if (!claims) return new Response("unauthorized", { status: 401 })
    const headers = new Headers(req.headers)
    headers.set("x-claims", JSON.stringify(claims)) // .set() overwrites any client-injected value

    const id = env.SESSION_DO.idFromName(sessionId)
    return env.SESSION_DO.get(id).fetch(new Request(req, { headers }))
  },
} satisfies ExportedHandler<Env>

3. Use it from the browser

import { createCollection } from "@tanstack/db"
import { useLiveQuery } from "@tanstack/react-db"
import { doCollectionOptions, WebSocketTransport } from "tanstack-durable-object-sync/client"
import { ulid } from "ulid"
import type { Api } from "./session-do" // TYPE-ONLY — nothing server-side is bundled

const transport = new WebSocketTransport<Api>({ url: `wss://${host}/sync/${sessionId}` })

// Api-driven: the row type is inferred from the schema Api + table name, so
// there's no runtime schema value and no explicit Row generic. `m` is Message.
const messages = createCollection(
  doCollectionOptions<Api, "messages">({ transport, table: "messages", getKey: (m) => m.id }),
)

function ChatRoom({ userId }: { userId: string }) {
  const { data } = useLiveQuery((q) => q.from({ m: messages }).orderBy(({ m }) => m.created_at, "asc"))
  const send = (content: string) =>
    // Optimistic; resolves once the server confirms on the single stream.
    messages.insert({ id: ulid(), author: userId, content, created_at: Date.now() })
  // Commands run over the transport (not the collection). `call` is a typed
  // Proxy — the name autocompletes and the result is checked against the Api.
  const clear = () => transport.call.clearRoom() // Promise<{ deleted: number }>
  return <ChatView rows={data} onSend={send} onClear={clear} />
}

One WebSocketTransport per DO is shared by every collection on that DO (multiplexed over the single socket). Pass where to doCollectionOptions (eager mode) to sync only a matching subset. Commands go through the same socket: transport.call.<name>(args) (typed sugar) or the low-level transport.sendCall("clearRoom", undefined) — both mint the txId for you and resolve with the command's result on committed.

Transport lifecycle & reconnect

The transport auto-reconnects on an unexpected drop and resubscribes every collection from its single applied cursor (a windowed catch-up, not a re-snapshot). Tune the pacing with reconnectDelay (a base-delay number for the default jittered backoff, or a full (attempt, closeCode?, closeReason?) => number | null policy — return null to stop). Application close codes (4000-4999) are terminal by default; wire onClosed(code, reason) to observe a deliberate server close (e.g. an auth rejection) — see ADR-0016.

close() disposes the transport, and connect() revives it: a later connect() clears the intentional-close latch, so a reused transport auto-reconnects and delivers onClosed again. A connect() (and any subscribe/sendMut/fetch awaiting it) never resolves disconnected — it rejects TransportClosedError if the transport was closed and stays closed (ADR-0020).

The socket opener is injectable (open, for non-browser runtimes or tests). It receives an optional AbortSignal that close() aborts while the handshake is still in flight — honour it (close the socket, reject) so disposing a transport before its socket finishes connecting never leaks a CONNECTING socket:

new WebSocketTransport<Api>({
  url,
  open: (signal) =>
    new Promise((resolve, reject) => {
      const ws = new WebSocket(url)
      signal?.addEventListener("abort", () => { ws.close(); reject(new Error("aborted")) }, { once: true })
      ws.addEventListener("open", () => resolve(ws))
      ws.addEventListener("error", () => reject(new Error("ws error")))
    }),
})

An opener that ignores the signal still works (the transport closes the socket once open() resolves), but a handshake that never resolves then leaks.

4. SSR (experimental)

Built on TanStack DB's SSR support (DbClient dehydrate()/hydrate() and the exportSyncMeta/importSyncMeta/mergeSyncMeta sync hooks, shipped in @tanstack/db 0.8.0). Why/how trade-offs live in ADR-0011.

On the worker, render through a per-request DbClient backed by snapshot reads (one per subscription, one or two per fetch) — no WebSocket from the render path:

// Route loader / server handler (per request!)
import { DbClient, collectionOptions } from "@tanstack/db"
import { doCollectionOptions, SsrSnapshotTransport } from "tanstack-durable-object-sync/client"

const stub = env.CHAT_DO.get(env.CHAT_DO.idFromName(sessionId))
// `request` is the incoming (claims-bearing) Request — the DO runs it through
// parseAttachment, the SAME auth gate as the WebSocket upgrade.
const transport = new SsrSnapshotTransport<Api>({
  read: (req) => stub.readSyncSnapshot(req, request),
})
const db = new DbClient()
const messages = db.collection(
  collectionOptions("messages", () =>
    doCollectionOptions<Api, "messages">({ transport, table: "messages", getKey: (m) => m.id }),
  ),
)
await messages.preload()
return { dbState: db.dehydrate() } // rows + our resume cursor (opaque syncMeta)

In the browser, hydrate before going live. The collection is ready immediately with the dehydrated rows (stale-while-revalidate); the first sub resumes from the dehydrated cursor, so the catch-up applies exactly what changed while the HTML was in flight — updates and deletes:

const db = new DbClient()
db.hydrate(dbState) // or, in React: <DbProvider client={db}><HydrationBoundary state={dbState}>…</HydrationBoundary></DbProvider>
const messages = db.collection(
  collectionOptions("messages", () =>
    doCollectionOptions<Api, "messages">({ transport: wsTransport, table: "messages", getKey: (m) => m.id }),
  ),
)

A render refuses writes and answers every read: mutations during SSR throw (SsrReadOnlyError), and every request an on-demand collection makes is read from the DO, as the browser's socket would answer it. That includes what @tanstack/db asks for on its own, so a render reads what the browser reads. One difference: a request the DO refuses (a predicate outside the floor, or a statement over a SQLite limit) fails the render, where the socket answers it with no rows and the browser's query does not fail (ADR-0011).

Warning

On @tanstack/db 0.9 and later, a limited on-demand query ordered by text with the default sort (this includes orderBy(id): the primary key is TEXT) makes @tanstack/db ask for every matching row after the first page. In a render those rows are read, held and dehydrated into the HTML. Top 10 of 100 000 matching rows read 100 010 rows and dehydrated 100 000. With stringSort: "lexical" (see the note on text ordering below) the same render read 11 rows and dehydrated 10, with or without an index. Lexical order does not help in two other cases. On the same versions, when the first page ends on a NULL in the order column, for any type and any sort, @tanstack/db also asks for every matching row; NULLs sort first by default in both directions, so orderBy(dueDate).limit(10) over ten or more NULL dueDate rows reads them all. And the rows that tie with the last row of the page are read whole. ADR-0011 has the details.

readSyncSnapshot is callable by any worker holding the DO binding, and its required request argument runs through parseAttachment — one auth gate for both the socket and the read path, so a tenant check in parseAttachment can't be bypassed by SSR.


Examples

Each is a runnable Worker + browser client (npm install && npm run dev), browser-verified.

  • examples/chat — eager sync of a room's messages; multi-tab live updates. The smallest end-to-end shape.
  • examples/on-demand — syncMode: 'on-demand': categorised items where each panel loads only its subset (loadSubset/ unloadSubset) and unopened categories are never synced.
  • examples/board — the at-scale stress test: 5,000 tasks on one DO with a bounded window, useLiveInfiniteQuery cursor scroll-back, and a mutable order key so voting bumps a task to the top (move-in). Its firehose makes the deferred bounded-window-under-churn limitation visible — loaded climbs past window.
  • examples/multi-do — two separate DOs (a room and an inbox) behind one Worker: one transport per DO, each typed by its own Api so transport.call.* is scoped to that DO's commands, and a cross-DO feed merged client-side (the DO never joins — ADR-0001).
  • examples/ssr — TanStack Start on Cloudflare with routerWithDbClient: a loader-preloaded page (rows in the server HTML, WebSocket resumes from the dehydrated cursor) and a Suspense-streaming page (useLiveSuspenseQuery, result streamed into the document). ADR-0011.

Tip

Using on-demand with orderBy + limit? Add a range index on the order column (collection.createIndex((r) => r.field, { indexType: BTreeIndex })) — without it the window can't page lazily: growing it re-requests the whole window from the start instead of fetching only the next page. See examples/board.

Note

Ordering a limited query by a text column? The server sorts text by bytes, because SQLite in a Durable Object has no locale collation, while @tanstack/db sorts strings by locale unless you say otherwise. The first window the server returns can then differ from the one the client would pick (by bytes Banana comes before apple). The client either loads the whole matching subset to correct the window, or keeps the wrong one, depending on its @tanstack/db version. Ask for lexical order on such a query and the two sides use the same order, so only the window is loaded (plus any rows that tie with its last row, and everything when that row's value is NULL; see the SSR section):

q.from({ m: messages }).orderBy(({ m }) => m.author, { direction: "asc", stringSort: "lexical" }).limit(20)

One rare difference remains, for characters outside the Basic Multilingual Plane. See ADR-0023.

Note

The server can only sort by, and filter on, a column of the table. When an on-demand query orders by or filters on something else, the client asks for every row the rest of its where matches and filters, sorts and cuts them itself. The query is right, on the socket and in a server render, but its limit saves nothing, and a where with nothing else in it loads the whole collection. This happens for a nested field such as m.title.length, in an orderBy or a where, on every @tanstack/db version, and on 0.8.x for an orderBy on a subquery's field, or one with clauses on two collections of a join. Two costs follow. A server render ships every one of those rows in the page. And when the whole collection was loaded this way, it stays loaded and live for as long as any other query on that collection is mounted, because that one subscription now covers them all. See ADR-0023.


Common patterns and recipes

Task-oriented guides in recipes/:

  • Commands vs mutations — when a write is a typed insert/update/delete and when it's a named command.
  • End-to-end types — share one schema type between server and client so the transport, commands, and collections are typed.
  • On-demand and windows — sync only the rows a query asks for, and grow a bounded window as the user scrolls.
  • Server-originated writes — write rows from the DO itself (webhooks, jobs, seeds) so clients still see them.
  • Cohosting — add sync to a DO that already extends a framework base (Agent, Think), and how to compose with @cloudflare/actors.

Cohosting: Syncable over a framework base

SyncDurableObject is the trivial application of a mixin, Syncable(Base), that adds the sync machinery to any Durable Object subclass. Use it when a DO already extends a framework base — the Cloudflare Agents SDK Agent, @cloudflare/think's Think — and you want that same DO to also be a sync source, instead of standing up a second DO and mirror-writing to it (ADR-0015).

import { Syncable } from "tanstack-durable-object-sync" // or ".../server/mixin"

// Curried: pin Env and your claims type, then apply over the runtime base.
class FeedAgent extends Syncable<Env, Claims>()(Agent<Env, State>) {
  constructor(ctx: DurableObjectState, env: Env) {
    super(ctx, env) // host constructor first
    // Auth hook for the sync upgrade (same contract as parseAttachment):
    this.sync.configure({ parseAttachment: (req) => readClaims(req) })
    ctx.blockConcurrencyWhile(async () => {
      migrate(ctx.storage.sql) // you create the tables…
      this.sync.registerSync(feedSchema) // …then register (ADR-0007)
    })
  }
}

The sync API lives behind one facade, this.sync (registerSync, runSyncedWrite, drainAndBroadcast, parseAttachment, configure, registry), so the only public names the mixin adds to your class are sync, readSyncSnapshot (the SSR read — public so DO RPC can reach it), and the four WebSocket/fetch handlers. tddc's sockets carry a reserved tag and a plain attachment, and it claims only the /_sync path (configurable) — everything else is delegated to your host base, so the two protocols never cross. No framework is added to tddc's dependency graph; you supply Base.

Cohosting has a few conventions — reach this.ctx.storage.sql (not this.sql), keep __pk out of your claims, opt in to two DO-global side effects, sync only your own tables — and one caveat: a @cloudflare/actors Actor won't cohost out of the box, since its Sockets helper takes over socket connections completely. You can still compose the Actors features you do use — Alarms, Storage — over a plain DurableObject base, as Actors' own examples show. The conventions, what's verified and how, and the full @cloudflare/actors story live in recipes/cohosting.md.


Known limitations

  • Large integers in DO SQLite: workerd SqlStorage cannot bind JS bigint, and INTEGER values above Number.MAX_SAFE_INTEGER round on read before syncing. For exact values, store decimal strings in TEXT columns and convert at the application edge (workerd#4195, issue #10).
  • A txId works like a password inside one Durable Object. The server remembers every mutation and command by its txId alone. A connection that presents another connection's txId receives that connection's stored reply, including a command's result. A connection that uses a txId first makes the owner's later write with the same txId answer committed without running. The built-in clients use random UUIDs, and the server never sends a txId to another connection, so an attacker would need a leaked or guessed id and a connection to the same Durable Object. Do not pass a predictable id to createTransaction or transport.sendMut, and do not show client error messages (some contain the txId) to other users. If commands return secrets, do not put users who distrust each other in one Durable Object (ADR-0012 D4). An id you supply must be at most 256 bytes: a longer txId is rejected with LIMIT_EXCEEDED (ADR-0012).
  • like patterns over 50 bytes: SQLite in a Durable Object accepts a LIKE pattern of at most 50 bytes (SQLITE_LIMIT_LIKE_PATTERN_LENGTH, set by workerd; Cloudflare's Durable Objects limits page lists it). A longer pattern is refused, so a search box that builds %term% stops matching at 49 ASCII characters, or about 17 CJK characters. Until refusals fail loud, a refused query reads as no results.
  • A column the table does not have: a where or orderBy that names a column the table does not have is refused, and the server logs one line. Until refusals fail loud, the query reads as no results. Two names that used to return rows are refused too: a name in another letter case than the declared one (BODY for body), and rowid.
  • A nested field in a query (eq(m.title.length, 3), or an orderBy on one) loads every row the rest of the where matches, and the whole collection when nothing else is left (see the note on ordering above). This is for on-demand queries. An eager collection's own where on a nested field is still refused and the collection stays empty. A frame built by hand that sends one to the server is refused.
  • One 0.8.x subquery orders by the wrong column. On @tanstack/db 0.8.x, a limited query over a subquery whose alias equals the inner alias, ordered by a field the inner select renamed to the name of another column (select(({ f }) => ({ id: f.id, name: f.id })), then orderBy(f.name)), is sent ordered by the table's own name column. The first window can be wrong, with no error. Use another alias for the outer query. 0.9 and later are not affected.

Non-goals

  • Multi-DO transactions. A transaction touches collections in one DO.
  • Server-assigned primary keys. Optimism requires id parity.
  • Per-row read authorization. Reads are gated at the WebSocket upgrade (per DO). Shard into more DOs for finer read isolation.
  • Server-side joins / IVM. The DO stores and emits; the client composes.
  • An event log. The change log is a state-convergence log, compacted to latest-op-per-key. It is not an audit trail.

Acknowledgements

The design is indebted to, and learns directly from, the open-source work of ElectricSQL and TanStack DB. This library is offered back to that community under the MIT license.

License

MIT © Tom McKenzie

About

Real-time sync for Cloudflare Durable Objects using TanStack DB — optimistic mutations, live queries, and single-ordered-stream write confirmation.

Topics

Resources

Stars

15 stars

Watchers

0 watching

Forks

Releases

Contributors

Languages