diff --git a/specs/issues/FIX-1790/BUSINESS-RULES.md b/specs/issues/FIX-1790/BUSINESS-RULES.md new file mode 100644 index 0000000000..badf9d317b --- /dev/null +++ b/specs/issues/FIX-1790/BUSINESS-RULES.md @@ -0,0 +1,56 @@ +# FIX-1790 · Business rules + +[Spec](SPEC.md) · [Decisions](DECISIONS.md) · **Rules** · [Plan](PLAN.md) · [Docs](DOCS.md) · [Evolution](EVOLUTION.md) + +The cases, written as rules. "Her cell in Acme" is the cell for Alice and the Acme org. "User +data" is user state and every user-scoped resource, shared or flow-isolated, state and content. +The *proved by* column names the check in [PLAN.md](PLAN.md#checks). + +## Where a person's data lives + +| # | When | Then | Proved by | +|---|---|---|---| +| BR-1 | Alice, in Acme, saves user data on any flow | It lands in her cell in Acme, with the flow added for a flow-isolated resource. Her next Acme run, on that flow, reads it back | Goal leg a · V2 | +| BR-2 | Alice opens the same flow in Globex | Her cell in Globex starts empty. Nothing she saved in Acme is read, through a run or any view | Goal leg b · V2 | +| BR-3 | Bob, in Acme, opens the same flow | His own cell in Acme. Nothing of Alice's is read | Goal leg b · V2 | +| BR-4 | Two flows in Acme declare the same shared user resource or user state | Both read and write Alice's one cell in Acme. Shared still means shared, inside one org. The registry still compares their schemas at startup | V2 · existing registry suite | +| BR-5 | Alice uses a hired seat in Acme | Its shared cell is her cell in Acme, the key it already had, now shared with the app's other flows in Acme. Its flow-isolated data keys by her, Acme and the seat | V1 key table · V2 | +| BR-6 | Any org-scoped or session-scoped data, with or without a tenant | Unchanged, byte for byte | V1 key table · existing suites | +| BR-7 | A user id, org id or flow id contains `:`, `\` or the text `~org` | No two (person, org, flow) tuples share a key, and no new key equals a key an older release wrote | V1 collision table · `poc/key-shape/` | + +## Where the org comes from + +| # | When | Then | Proved by | +|---|---|---|---| +| BR-8 | A run keys user data | The org is the admitted run's. For a pinned seat it equals the pin's org, which admission already checked. A header or body never chooses it | V2 | +| BR-9 | A run, a view or a helper call has no org, or a blank one | Refused, as FIX-1442 refuses it. No user data is read or written, and the cross-org cell is never the answer | V1 · V3 | +| BR-10 | Any read-side view of a session (state route, resource routes, debug snapshot, a sibling transport) | It resolves the cell the run wrote, from the session's stored org | V3, one per view family | +| BR-11 | A run dispatches a child session on another flow | The child keeps the person and the org, and reads that flow's buckets in the same cell | V2 | +| BR-12 | A dynamic schedule fires | The dispatch names the org; the resolver reads that cell and dispatches only a row naming the same org. No org in the dispatch, or a row naming another, resolves as missing, as does a user-owned seat's id naming another person | V5 | +| BR-13 | A test seeds user data through the testing harness | It lands in the cell the run will read: the harness's org, or the default org when none is given | V6 | +| BR-14 | A flow declares a user-scoped projected resource | Its hooks receive the person and the org. The app keys its own rows by both; the framework can't, and the docs say so | DOCS review | + +## Data saved before the upgrade + +| # | When | Then | Proved by | +|---|---|---|---| +| BR-15 | A person has user data saved by an older release, and the step has not run | No flow and no view reads it, in any org. Each of their cells starts empty. The old cell is untouched. Nothing refuses | Goal leg c · V4 | +| BR-16 | The operator runs the step on a cell whose possible writers' sessions all name one org: all the person's sessions for a shared cell or user record, their sessions on that flow for a flow-isolated cell. The operator also vouches that none of those sessions was deleted, or that their own records never placed the person in another org | The cell is copied into that org: state and content together, versions and deletion markers kept, a user record's id rewritten. The original stays | Goal leg c · V7 | +| BR-17 | Those sessions name two or more orgs, or none, or some may have been deleted and the operator's records can't clear every other org. A session with no flow owner on a collection flow counts as unknown until the owner attribution has run | The step stops for that cell with `migration-required`, names it in the upgrade record, and copies nothing | Goal leg c · V7 | +| BR-18 | The destination already holds a row, a record or a deletion marker for the person | The step stops for that person and names the keys. Nothing is overwritten or merged | V7 | +| BR-19 | A copied cell held a dynamic schedule | After its index row is rebuilt from the new cell and the old cell's row removed, it fires once per beat. Before the step it does not fire | V5 · V7 | +| BR-20 | A deployment's sessions only ever named one org, such as a development app on the default org | Every cell is attributable to it and copied | V7 | + +## Failure taxonomy + +Nothing here is a new runtime error. A run or a view that finds a cell empty behaves as a first +run. A missing org is refused exactly as today. The only new stop is the operator step's, per +cell or per person, named `migration-required` like the other upgrade procedures, and it leaves +the store as it found it for that cell. + +## Acceptance criteria this issue owns + +[The goal](SPEC.md#the-goal-and-how-well-know-its-met) passes on the real router and a SQLite +store, and fails under each named control. The epic's closure step "Alice in a second org sees +nothing stored before FIX-1790" then holds on what this ships +([ER-30](../../epics/FIX-1786/BUSINESS-RULES.md#the-closure)). diff --git a/specs/issues/FIX-1790/DECISIONS.md b/specs/issues/FIX-1790/DECISIONS.md new file mode 100644 index 0000000000..c88d2ff283 --- /dev/null +++ b/specs/issues/FIX-1790/DECISIONS.md @@ -0,0 +1,111 @@ +# FIX-1790 · Decisions + +[Spec](SPEC.md) · **Decisions** · [Rules](BUSINESS-RULES.md) · [Plan](PLAN.md) · [Docs](DOCS.md) · [Evolution](EVOLUTION.md) + +Two decisions are the sign-off surface. The calls above them are the epic's: user data is kept +per (user, org) for every flow, and a record stored before reads in one org at most, moved by an +operator step and never by a fallback read +([ER-3](../../epics/FIX-1786/BUSINESS-RULES.md#what-a-team-gets-and-what-it-doesnt), +[D3](../../epics/FIX-1786/DECISIONS.md#d3)). The epic left this spec one question: how a saved +record is attributed to an org. These two cards answer it and say what that costs a customer. + +## The tree + +```mermaid +flowchart TD + I["FIX-1790"] --> D1["D1 · old data is read by nothing
until the operator step copies it"] + D1 -.->|"rejected · reads the old cell on every run"| X1["refuse with migration-required at runtime"] + D1 -.->|"rejected · guesses the org at the worst moment"| X2["copy on a person's first request"] + D1 -.->|"rejected by the epic · the leak"| X3["fall back to the old cell on read"] + I --> D2["D2 · a cell moves to the one org
its writers' sessions name"] + D2 -.->|"rejected · says where they are, not where they wrote"| X4["the operator's membership records alone"] + D2 -.->|"rejected · a guess"| X5["the person's most recent org"] + D2 -.->|"rejected by the epic · the leak"| X6["a copy into every org"] +``` + +Solid edges are what you're signing. Dashed edges lost, and the label says why. + + +## D1 · Data saved before the upgrade is read by nothing until an operator copies it; until then each person starts empty in each org + +| | | +|---|---| +| **Instead of** | Refusing a person's requests with `migration-required` while they have uncopied data · copying it automatically on their first request in an org | +| **Because** | A refusal has to look in the old cell to know there is something to refuse: a read across orgs on every run, the read ER-3 closes, and a store call per request. It would also lock a person who worked in two orgs out of every flow until someone resolves them by hand. An automatic copy picks the org from the first request after the upgrade, and the framework can't tell a one-org person from one whose other org hasn't signed in yet ([FIX-1538 D2](../FIX-1538/DECISIONS.md#d2)). Starting empty is what hired seats have done since FIX-1538 | +| **Locks in** | Every deployment with saved user data has an operator step at upgrade, with writers stopped. An operator who skips it gets no error: every person's preferences, saved collections and flow-private data look empty. Nothing is lost and the step can run later, but whatever a person writes in between makes the step stop for them. The framework never moves or deletes a record | + +![D1, between the upgrade and the copy step: old data read by nothing, chosen, beside a migration-required refusal. Decides it: a person who worked in two orgs works in both instead of being refused on every flow. Price: an operator who skips the step sees no error. Locks in an upgrade step; flips if a deployment can't stop its writers](figures/d1-before-the-step.svg) + +It comes down to a person who worked in two orgs: a refusal locks them out of every flow. + +**What would change my mind:** a deployment that can't stop its writers for the step. Then a +shipped copy command earns its place, still offline per person, still never a runtime read. + + +## D2 · The step copies a saved cell to an org only when every session that could have written it names that org, and the operator vouches none was deleted + +| | | +|---|---| +| **Instead of** | The operator's membership records alone · the person's most recent org · a copy into every org the person belongs to | +| **Because** | A session is the one record in the store that says which org a run happened in, and it says so per flow. The person's shared cell and user record could have been written by any of their sessions; a flow-isolated cell only by their sessions on that flow, so a hired seat's private data keeps its one org when its person works in two. Membership says where a person is now, not where they wrote: Alice, in Acme until June and in Globex since, would have Acme's data copied into Globex. The most recent org is a guess, and every org is the leak ER-3 closes. A deleted session leaves no trace in the store, so the sessions that remain are evidence only when none of that person's is missing: if Alice's Globex sessions were deleted, the rest name Acme alone and the step would copy Globex data into Acme | +| **Locks in** | A person whose sessions name two orgs gets their shared data back in neither: it stays put, named in the upgrade record, until someone decides by hand. The step also needs the operator to vouch that none of the person's sessions was deleted, or that their own records never placed the person in another org; membership can veto a copy, never choose its org. Without that, and for a person with no sessions left, it stops the same way. A deployment that only ever ran in one org, which includes every development app on the default org, gets everything back | + +![D2, where the step copies a saved cell: the one org its writers' sessions name, chosen, beside membership records. Decides it: Alice, who moved from Acme to Globex, would have Acme's data copied into Globex by membership. Price: a person with any deleted session stops unless the operator's records clear every other org. Locks in: a two-org person's shared data returns nowhere until decided by hand](figures/d2-attribution.svg) + +It comes down to a person who changed orgs: membership copies the old org's data into the new one. + +**What would change my mind:** an app that records, per write, which org the write happened +in. That record attributes a cell more finely than sessions do, and the step should prefer it. + +## Decided, not asked + +- **One cell per (person, org) for every flow, the one hired seats already use**, + `:~org:`, plus the flow when flow-isolated ([pinned names](PLAN.md#pinned-names)). + A seat's shared data doesn't move, and it now shares a person's cell with the app's other flows + in that org, as any two flows do; the registry still refuses incompatible schemas. FIX-1538 kept + them apart only because the app's cell crossed orgs ([EVOLUTION.md](EVOLUTION.md)). +- **The org comes from the admitted run or the stored session**, never a header or body (BP-031). + A pinned seat's equals its pin's; admission checked that. +- **A missing or blank org throws.** No path builds the cross-org key (ER-13). +- **Flow-isolated keys gain the org too.** +- **A schedule's dispatch names the org**, and the row must name the same one: the id selects a + cell and grants nothing. It has to be in the id, not only on the index row: every producer, + the Vercel tick and BullMQ included, reaches the resolver through the one dispatch id, and + Alice can hold a schedule `daily` in Acme and in Globex. A two-part id names neither, and + finding it from the index would be a read across orgs. +- **The harness seeds through the engine's derivation**, with the run's org. +- **The step is a documented procedure per adapter**, not a command, and **the original cells + stay**; removing them after read-back is the operator's call. +- **No startup warning about uncopied cells**: the stores can't list scope ids. A follow-up. +- **Old-term exports left in place** for FIX-1796 and FIX-1798: `IsolationFlow.ownerPin` + (accepted, ignored, deprecated), `ScheduleResolutionContext.ownerPin`, `InstanceOwnerPin`. + +## Considered and dropped + +| Alternative | Why not | +|---|---| +| A new cell shape for unpinned flows, seats kept apart | Keys on the pin the epic is deprecating, and two cells per person per org isolate nothing | +| A deployment-wide "old data belongs to org X" setting | A permanent second read path, wrong for any deployment that ever had two orgs | +| A shipped migration command now | The stores' normal calls mint versions and drop deletion markers; the per-adapter procedures already have raw access. Revisit on D1's *change my mind* | +| Move cells instead of copying | Deletes the only record a wrong attribution could be recovered from | + +## Settled + +- **No new key equals an old one, and no two (person, org, flow) tuples share one** — + **CONFIRMED** by `poc/key-shape/`: about four million keys from ids over `a : \ ~ o` and `~org`, + all distinct; the shared key equals FIX-1538's pinned cell every time; without the `~org` + marker, 24,964 collisions. An old cell is unreachable by construction, so D1 needs no guard. +- **Every production user key comes from the one derivation; only the schedule resolver lacks an + org** — **CONFIRMED** at `fbecfe6f2` by `poc/key-sites/`: nine files, planted control failed. + +## How it got here + +- **Draft** — framed as ER-3's attribution question; one (person, org) cell for every flow, + reusing the seat's shape, with the org from the run or the stored session; old cells inert by + construction and copied by a documented operator step that attributes per writing flow; one PR. +- **Review round 1** — D2 now needs the operator to vouch that no session is missing, because a + deleted session leaves no trace and the ones left could name one org while the deleted ones + named another (Codex). The step's SQL is walked on Postgres as well as SQLite before it is + published (second look). + +**Open: none.** diff --git a/specs/issues/FIX-1790/DOCS.md b/specs/issues/FIX-1790/DOCS.md new file mode 100644 index 0000000000..a5b098d038 --- /dev/null +++ b/specs/issues/FIX-1790/DOCS.md @@ -0,0 +1,216 @@ +# FIX-1790 · Documentation draft + +[Spec](SPEC.md) · [Decisions](DECISIONS.md) · [Rules](BUSINESS-RULES.md) · [Plan](PLAN.md) · **Docs** · [Evolution](EVOLUTION.md) + +FIX-1790 owns "which org a user's record belongs to" in the epic's ownership table +([epic DOCS.md](../../epics/FIX-1786/DOCS.md#ownership)). Every page below describes user data as +one record per person per organization, which is now true for every flow, so the hired-seat +exceptions on these pages go. Voice watch-outs: describe what the framework does, not what it +used to do, except inside the upgrade section; no sentence opens with "This"; introduce "user +scope" and "flow isolation" in plain words where a page meets them first. + +## UPDATE · `apps/docs/docs/fundamentals/state-and-scopes.md` · the scope table, the figure's alt text, and "User" + +The table row: + +> | **User** | What does this person need across their conversations in one organization? | Across sessions for a user, per organization | + +The figure's sentence about user state, in the alt text and in `state-scopes.svg`: + +> User state follows one person across conversations and flows inside one organization, keyed by +> the user and the organization; with isolateUserState it is one record per flow copy instead. + +Replacing the paragraph after the `userStateSchema` example: + +> User data belongs to a person inside one organization. Alice in Acme and Alice in Globex have +> two user records, and two copies of every user-scoped resource, so nothing she saves while +> working for one organization shows up in the other. Inside one organization, every flow on the +> server shares her one record, so each flow's user state schema is compared at startup. +> Incompatible declarations throw `CrossFlowSchemaConflictError` from `FlowRegistry.register` +> before any data can be corrupted. [Flow Isolation](/docs/advanced/flow-isolation) gives each +> flow copy its own record instead, still inside the organization. + +In "Multi-tenant isolation", the sentence "so they key on `userId` / `orgId` alone" becomes: + +> so they key on the user and organization, or the organization alone, and not on the tenant. + +## UPDATE · `apps/docs/docs/advanced/flow-isolation.md` · "The default", and REMOVE "Hired seats share within one organization" + +> ## The default: shared user and org state +> +> Every flow you register on a server reads from and writes to the same `UserRecord` for a +> given person in a given organization. Same for org. Sessions and requests are different: +> those carry the flow's identity and stay separate per flow. +> +> The organization is part of the user's key. What a flow saves for Alice while she works in +> Acme is not what any flow reads for her in Globex. Inside one organization, a [hired +> seat](/docs/workforce/durable-hire) shares Alice's record with your other flows, like any +> two flows do. + +The "Hired seats share within one organization" subsection is removed; the paragraph above says +what it said, for every flow. + +## UPDATE · `apps/docs/docs/workforce/durable-hire.md` · "What a seat saves for a person" + +The figure loses its third place (the person's data "outside hired seats"), and its alt text +becomes: + +> Where a hired seat keeps what it learns about alice. Inside the acme organization: one cell for +> alice in acme, holding her user record and her shared user-scoped resources, which every flow +> in acme uses, hired seats included; and one cell per seat for flow-isolated data, such as alice +> in acme for acme.support.ada. In globex, alice has a separate cell of her own, which nothing in +> acme reads. A projected resource is stored by your own hooks, so key its rows by the +> organization too. + +> A seat keeps what it learns about a person in the same place every flow in the organization +> does: one cell for that person in that organization. With flow isolation, it keeps one cell per +> person per seat, still inside the organization. A resource's own `flowIsolation` decides which +> applies; a resource that doesn't set it follows the kind's `isolateUserState`. +> +> If you are upgrading an app that already saved user data, see [Upgrading: moving user data into +> its organization](../persistence/overview.md#upgrading-moving-user-data-into-its-organization). + +## UPDATE · `apps/docs/docs/resources/projected-collections.md` · the first example, and one paragraph after it + +The two hooks key by organization as well (`orgId: ctx.orgId` beside `userId: ctx.userId` in +both queries), and after the example: + +> A user-scoped collection belongs to a person inside one organization, the same as the user data +> the framework stores. Your hooks receive both `ctx.userId` and `ctx.orgId`. Use both in every +> query, or a person sees in one organization what they saved in another. + +## UPDATE · `apps/docs/docs/server/scheduled.md` · "Schedules on a hired seat" becomes "Schedules in a user's data" + +> ### Schedules in a user's data +> +> A schedule saved in a user-scoped collection belongs to a person inside one organization. Use +> `createResourceCollectionScheduleResolver` there. Its schedule id names the organization, the +> user and the key, `//`. It reads the row from that person's data in that +> organization, and returns `null` when the id names no organization or the row names a different +> one. On a seat only one member can reach, it also returns `null` for an id naming another user. +> +> If you write the resolver by hand, pass the organization to `resolveUserStorageKey` from +> `@flow-state-dev/engine`. It throws when the organization is missing or blank: +> +> ```ts +> resolve: async (scheduleId, ctx) => { +> const [orgId, userId, key] = scheduleId.split("/").map(decodeURIComponent); +> const scopeId = resolveUserStorageKey(userId, orgId, { id: ctx.flowKind, isolateUserState: false }); +> const row = await ctx.stores.resourceState.get("user", scopeId, `schedules/${key}`); +> if (row?.state.orgId !== orgId) return null; +> // map its `kind` to a block, return the config as above +> } +> ``` + +## UPDATE · `apps/docs/guides/scheduled-dynamic.md` · the id format, both dispatch URLs, and the impersonation guard + +Every `/` becomes `//`, and both example URLs add the segment +(`/schedules/${row.orgId}/${row.userId}/${row.key}/dispatch`, `${orgId}/${userId}/${key}`). The +guard's paragraph: + +> The default helper parses `//` from the URL and reads the resource from +> that user's data in that organization. The storage key is the guard: a request like +> `/schedules/acme/u_evil/k/dispatch` reads u_evil's data in acme, and finds nothing unless +> u_evil saved a schedule there. A row found under one organization that names another is +> refused too, so an id cannot fire a schedule into an organization its creator didn't choose. + +## CREATE · `apps/docs/docs/persistence/overview.md` · "Upgrading: moving user data into its organization", replacing "Upgrading: moving hired seats' stored data" + +> ## Upgrading: moving user data into its organization +> +> This section is for apps that saved user data on an earlier release. A new install can skip it. +> +> User data is kept per person and organization: the user record and every user-scoped resource +> live under `:~org:`, and a flow-isolated one under +> `:~org::`. An earlier release kept them under the person's id +> alone, ``, or `:` when flow-isolated, one cell across every +> organization. The server reads none of those older cells, in any organization, and does not +> move them, because one cell may hold what a person saved in two organizations. Until you copy +> it, each person starts empty in each organization. Nothing is deleted. +> +> Hired seats already kept their shared data per organization. Those cells are where they belong +> and need nothing. +> +> Copying is offline: stop the writers, schedulers included, before you start the new release, +> take a backup, and work on the copy, as in [Who owns a record](#who-owns-a-record). A person who +> writes before you copy leaves data in the destination, and the copy stops for them. +> +> 1. **List the old cells.** Every user-scope `scope_id` in `resource_state` and +> `resource_content`, and every `users.id`, that has no `:~org:` in it. A cell with no `:` +> outside an escape is a person's shared cell; one with exactly one such `:` is +> `:`. Read the raw rows, so deletion markers and content-only rows show up. +> 2. **Find the organization each cell was written in.** Run [Attributing organizations, +> once](#which-organization-a-record-belongs-to) first if any session still has no +> organization. For a shared cell and the user record, every session of that person could have +> written it. SQLite: +> +> ```sql +> SELECT user_id, COUNT(DISTINCT org_id) AS orgs, SUM(org_id IS NULL) AS unknown, +> MIN(org_id) AS org +> FROM sessions GROUP BY user_id; +> ``` +> +> For a flow-isolated cell, only that person's sessions on that flow. If a collection flow +> still has sessions with no `flow_id`, finish [Attributing owners, once](#who-owns-a-record) +> first; until you do, the person's isolated cells on that kind are a stop. After it, a +> session with no `flow_id` is an ordinary single flow's, named by its `flow_kind`: +> +> ```sql +> SELECT user_id, COALESCE(flow_id, flow_kind) AS flow, COUNT(DISTINCT org_id) AS orgs, +> SUM(org_id IS NULL) AS unknown, MIN(org_id) AS org +> FROM sessions GROUP BY user_id, COALESCE(flow_id, flow_kind); +> ``` +> +> On Postgres, write `COUNT(*) FILTER (WHERE org_id IS NULL)` for `unknown`. A deleted +> session leaves no row, so these counts only see the sessions you still have. A cell is +> attributed when `orgs` is one, `unknown` is zero, and you can vouch for the rest: no +> session of that person was ever deleted, or your own records never placed them in another +> organization. Anything else, including a person with no sessions left, is a +> `migration-required` stop for that cell: write it down and leave it. +> 3. **Check the destination.** The same raw read at `:~org:` (and the flow +> id after it, for an isolated cell). If any key is already there, stop for that person and +> write the keys down. Do not overwrite, and do not merge. +> 4. **Copy.** For each attributed cell, in one transaction, copy its rows to the new `scope_id` +> with the resource key, state, version, lifecycle and content exactly as they are. SQLite, +> for Alice in Acme: +> +> ```sql +> INSERT INTO resource_state (scope_type, scope_id, resource_key, state, version, lifecycle) +> SELECT scope_type, 'alice:~org:acme', resource_key, state, version, lifecycle +> FROM resource_state WHERE scope_type = 'user' AND scope_id = 'alice'; +> ``` +> +> Do the same for `resource_content`, and copy the `users` row with its `id`, and the `id` +> inside its data, set to the new key. Postgres takes the same statements; +> [Moving a copy's private data, per adapter](#moving-a-copys-private-data-per-adapter) covers +> the filesystem store and custom stores. +> 5. **Rebuild the schedule index** for every person whose cell held schedules: write each copied +> schedule once from its new cell, and remove the index rows whose cell is an old one, or the +> schedule fires twice. A schedule in an old cell does not fire until you do. If a cloud +> scheduler or BullMQ holds a job per schedule, re-register it: the dispatch id now names the +> organization, `//`. +> 6. **Read back, then start the writers.** Every attributed cell has its copy, every stop is +> written down, and a person's data reads through a flow in the organization it went to and +> not in another. +> +> Ids containing `:` or `\` are escaped in every key the same way, a backslash before each. The +> original cells stay. They are read by nothing, and you can remove them once the copies have +> read back. +> +> [When to stop](#when-to-stop) applies here unchanged. + +And in "Attributing organizations, once", step 3, the paragraph that begins "User-scope storage +does not move" is removed: user data now moves with its organization, by the section above. + +## UPDATE · `packages/engine/README.md` and `packages/scheduled/README.md` + +The engine's scope-key section: `resolveUserStorageKey(userId, orgId, flow)` returns +`:~org:`, or `:~org::` when the flow isolates user +state, and throws on a missing or blank `orgId`; the owner-pinned paragraph is removed. The +scheduled README's hand-written resolver matches the `scheduled.md` example above. + +## Publication ownership + +FIX-1790 publishes all of the above after V7 has walked the step on a SQLite file and on Postgres, +and VG has passed. Only statements V7 ran are published. The Workforce overview and glossary stay FIX-1796's; FIX-1788 owns the seat pages' move to +workers, and builds on this section's cell. diff --git a/specs/issues/FIX-1790/EVOLUTION.md b/specs/issues/FIX-1790/EVOLUTION.md new file mode 100644 index 0000000000..6c7b5209dd --- /dev/null +++ b/specs/issues/FIX-1790/EVOLUTION.md @@ -0,0 +1,19 @@ +# FIX-1790 · Evolution + +[Spec](SPEC.md) · [Decisions](DECISIONS.md) · [Rules](BUSINESS-RULES.md) · [Plan](PLAN.md) · [Docs](DOCS.md) · **Evolution** + +The user key and its upgrade path. Lineage across the whole epic is in the epic's +[EVOLUTION.md](../../epics/FIX-1786/EVOLUTION.md); its user-data row is the one this issue builds. + +| Prior intent and precise source | Treatment | Why / evidence | Replacement | Compatibility | +|---|---|---|---|---| +| FIX-735: a person's shared user data, and a non-isolated user scope record, key at the bare person id, one cell across every org and every flow. Source: header of `packages/engine/src/stores/scope-keys.ts`; [`state-and-scopes.md` → Cross-Flow State: Shared vs Isolated](../../../docs/architecture/state-and-scopes.md#cross-flow-state-shared-vs-isolated) | **Superseded** for user scope; retained for org scope | Epic ER-3: user data is kept per (user, org) for every flow, or private workers show in every org | S1, BR-1–BR-4 | Old cells stay on disk, read by nothing ([D1](DECISIONS.md#d1)), copied by the step ([D2](DECISIONS.md#d2)) | +| FIX-1538 D1: a hired seat keeps a person's data in its own (org, person) cell, and seats and the app's flows never see each other's. Source: [`../FIX-1538/DECISIONS.md#d1`](../FIX-1538/DECISIONS.md#d1) | **Retained** for the cell, byte for byte; **amended** for the separation | The separation existed because the app's cell crossed orgs. Once every flow keys per org, two cells per person per org isolate nothing, and BP-027's "shared means shared" applies again | DECISIONS → *Decided, not asked*, first line; BR-5 | A seat reads nothing it didn't before from its own cell, and now also the app's shared data in that org, under keys the registry already checks | +| FIX-1538 D2 and BR-14–BR-18: an operator step copies only keys a seat alone declares, for people whose seats ran in one org, never the user record, and stops on a written destination. Source: [`../FIX-1538/DECISIONS.md#d2`](../FIX-1538/DECISIONS.md#d2); [`../FIX-1538/BUSINESS-RULES.md`](../FIX-1538/BUSINESS-RULES.md) → "Upgrading a deployment that already has seats"; `apps/docs/docs/persistence/overview.md` → "Upgrading: moving hired seats' stored data" | **Amended and generalized** | Every flow now moves at once, so the reasons to strike shared keys and the user record are gone. Attribution is per writing flow, not per person: a seat's flow-isolated cell keeps its one org even when its person works in two | [D2](DECISIONS.md#d2), BR-16–BR-20; DOCS.md replaces the page section | The no-fallback rule, the stop on a written destination and the copies-not-moves rule are unchanged | +| FIX-1538 *Considered and dropped*: "Re-key user scope to (user, org) for every flow — decides the person's cross-org data, which nobody has decided". Source: [`../FIX-1538/DECISIONS.md`](../FIX-1538/DECISIONS.md) → "Considered and dropped" | **Superseded** | Decided since, by the epic ([D3](../../epics/FIX-1786/DECISIONS.md#d3), ER-3) | This issue | — | +| FIX-1323: the isolation coordinate is the instance id, with an attributable offline cutover and no kind fallback. Source: [`state-and-scopes.md` → Per-flow isolation (opt-in)](../../../docs/architecture/state-and-scopes.md#per-flow-isolation-opt-in); `apps/docs/docs/persistence/overview.md` → "Who owns a record" | **Retained**; the flow-isolated user key gains the org ahead of the instance | The same no-fallback reasoning is D1's; the per-adapter mechanics are reused by the new step | S1, pinned isolated key | Instance coordinates unchanged; org-scope isolated keys unchanged | +| BP-027: user-scoped resources default to shared, and a `false` resource keys at the bare `{userId}`, with FIX-1538's pinned exception. Source: [`best-practices/resources.md` → BP-027](../../../docs/contributing/best-practices/resources.md#bp-027-user-scoped-resources-default-to-shared-flowisolation-off-isolate-only-deliberately) | **Amended**, second bullet only | Shared means shared inside one (user, org) cell, for every flow; the pinned exception is now the rule | S7 edits the bullet | A practice, not stored data | +| FIX-1442: org is never optional. Source: `apps/docs/docs/persistence/overview.md` → "Which organization a record belongs to"; [`../../epics/FIX-1786/BUSINESS-RULES.md`](../../epics/FIX-1786/BUSINESS-RULES.md) ER-13 | **Retained** and relied on | The admitted org is always present, so the per-org cell can always be built | BR-9 | A record with no org is refused as before | + +Nothing is wholly superseded except the bare user key itself. Re-check every row against current +code before building: the rows cite approved intent, and only the code says what shipped. diff --git a/specs/issues/FIX-1790/PLAN.md b/specs/issues/FIX-1790/PLAN.md new file mode 100644 index 0000000000..0a06078598 --- /dev/null +++ b/specs/issues/FIX-1790/PLAN.md @@ -0,0 +1,162 @@ +# FIX-1790 · Plan + +[Spec](SPEC.md) · [Decisions](DECISIONS.md) · [Rules](BUSINESS-RULES.md) · **Plan** · [Docs](DOCS.md) · [Evolution](EVOLUTION.md) + +Written for the implementing agent. IDs cross-reference [BUSINESS-RULES.md](BUSINESS-RULES.md) +(BR-n), [DECISIONS.md](DECISIONS.md) (D-n) and the epic's rules (ER-n). `tdd`. One PR, because +S1's required org argument breaks every caller in S2 to S5 at compile time, so they can't merge +green apart, the goal lands with what makes it pass, and docs move with the code (ER-25). It merges +before FIX-1788 and FIX-1793 ship anything user-scoped (epic PLAN → coordination seams), and its +build waits on FIX-1787's merge-first rows (ER-23). + +## Surfaces + +| ID | Package · role | Change | Rules | +|---|---|---|---| +| S1 | `engine` · the key derivation (`stores/scope-keys.ts`) | Every user key takes the org ([pinned names](#pinned-names)), each part through the existing escaping; the pin no longer chooses it. The exported helpers take the org as a required argument and throw on one `isValidOrgId` rejects. Org and session keys untouched. The header comment becomes one account, FIX-1538's paragraph folded in | BR-1–BR-7, BR-9 | +| S2 | `engine` · execution context | Hand the admitted org to S1 for the scope record, both resource paths and the collection cell. No new read | BR-1–BR-5, BR-8, BR-11 | +| S3 | `engine` · the read side | `getPersistedData`'s user branch and the state route's scope-record read take the session's stored org. A session with no org reads no user data. `toIsolationFlow` stops forwarding the pin for keys | BR-9, BR-10 | +| S4 | `scheduled` + `vercel` + `bullmq` · every schedule dispatch | The dispatch names the org: the default schedule id parser, the dispatch route's id validator, the Vercel tick's URL and the BullMQ job's data and URL change together. The validator admits every id `isValidOrgId` admits, the default org's `__fsd_default_org__` included, and the length the org adds; today's lowercase 128-character pattern rejects both. The resolver reads the named org's cell and returns `null` unless the stored row names the same org (and the pin's, on a pinned seat). An id with no org returns `null` | BR-12, BR-19 | +| S5 | `testing` · the harness seeders | `createTestContext` and `testFlow` seed user state and resources through the engine's derivation with `orgId ?? DEFAULT_ORG_ID`, the org the run uses | BR-13 | +| S6 | `goals/user-scope/` · the goal | One model-free goal, three legs and two controls ([Checks](#checks) VG) | Goal | +| S7 | Docs | Publish [DOCS.md](DOCS.md). Internal: `docs/architecture/state-and-scopes.md` (cross-flow section; "The owner-pinned cell" becomes the per-org user cell for every flow), BP-027's second bullet, `packages/engine/README.md`'s scope-key section, `packages/scheduled/README.md`'s hand-written resolver. `minor` changesets for `engine`, `scheduled`, `vercel`, `bullmq`, `testing`, each carrying the upgrade note | ER-25 | + +**Removed:** the pin branch in the key derivation; the persistence page's hired-seats upgrade +section, which the new step subsumes; the docs passages saying seats and app flows never share a +person's data. [EVOLUTION.md](EVOLUTION.md) has what is amended. + +## Sequence + +```mermaid +flowchart TD + S1["S1 · the org in every user key"] --> S2["S2 · execution context"] + S1 --> S3["S3 · the read side"] + S1 --> S4["S4 · schedule dispatch"] + S1 --> S5["S5 · the harness"] + S2 --> S6["S6 · the goal"] + S3 --> S6 + S4 --> S6 + S6 --> S7["S7 · docs, the step walked, changesets"] +``` + +## Checks + +| ID | Runs after | Passes when | +|---|---|---| +| V1 | S1 | Key table: shared and flow-isolated keys have the pinned shapes; a pinned seat's shared key equals today's string exactly; every org and session key equals today's. A missing, blank or lone-surrogate org throws from every exported helper. A collision table carried over from `poc/key-shape/` (ids with `:`, `\`, `~org`) is all distinct and disjoint from the one- and two-part legacy forms | +| V2 | S2 | BR-1–BR-5, BR-8, BR-11 through real runs: two orgs, two people, user state, a shared and a flow-isolated resource, a two-flow pair sharing one resource, a pinned seat, a cross-flow child | +| V3 | S3 | BR-10: the state route, a user resource read and the debug snapshot each return what the run wrote, and not a value planted in another org's cell or the old cell. BR-9: a stored session with no org reads no user data | +| V4 | S2 | BR-15: values planted in a person's old one-part and two-part cells before a run are read by no org, and are still there, unchanged, after it | +| V5 | S4 | BR-12 for each dispatch producer (default parser, Vercel tick, BullMQ worker): a schedule a run saved fires for its org, including the default org and an org id with uppercase, `.` or `/`; the same row under another org's dispatch, a dispatch with no org, and a row in the old cell all resolve as missing; a user-owned seat's other-person check still holds. BR-19 | +| V6 | S5 | BR-13: a value seeded through each helper, with and without `orgId`, is what the run reads | +| V7 | S7 | The published step walked once on a SQLite file and once on Postgres (PGlite, as the adapter's own suite runs it), recorded in the PR. No SQL is published for a store it wasn't walked on; the filesystem and custom stores get the mapping in prose. Copied: a one-org person; a two-org person's flow-isolated cell whose flow ran in one org; a deletion marker with its collection; a user record, id rewritten. Stopped: that person's shared cell; a person with no sessions; a person whose remaining sessions name one org after their other org's session was deleted, with no records to vouch; a collection flow's isolated cell before owner attribution; a written destination, keys named. A schedule fires once after the index rebuild | +| VG | S6 | **Goal**: `pnpm tsx goals/user-scope/keeps-a-users-data-in-the-org-it-was-saved-in/run.mts` PASSES, model-free, real router, SQLite, per [SPEC.md](SPEC.md#the-goal-and-how-well-know-its-met) | +| VC | VG | `GOAL_CONTROL=cross-org-key` (S1 drops the org for an unpinned flow) FAILS leg b on Globex reading Alice's shared marker. `GOAL_CONTROL=fallback-read` (an empty cell reads the old one) FAILS leg c on the old marker reading in Globex. Both FAILs recorded in the goal's verdict log before the PASS | + +D1 is proved by V4 and leg c; D2 by V7 and leg c. + +## Pinned names + +| Where | Name | Why pinned | +|---|---|---| +| Shared user key | `:~org:` | Persisted, byte-identical to FIX-1538's seat cell, and named by the operator step | +| Flow-isolated user key | `:~org::` | Persisted and named by the step. The `~org` marker keeps it apart from every other form (`poc/key-shape/`) | +| Public helper | `resolveUserStorageKey(userId, orgId, flow)` | Public; the scheduled docs show it. Taking the org positionally makes an old call fail to compile and throw at runtime | +| Default dynamic schedule id | `//` | Public: the guides, cloud scheduler jobs and the operator step name it | +| The goal | `goals/user-scope/keeps-a-users-data-in-the-org-it-was-saved-in/` | Cited by path | + +How each part is escaped in the URL is yours, provided the default parser and both producers +change together and an org or user id containing `/` round-trips. + +## Guardrails + +| Rule | Because | +|---|---| +| The org comes from the admitted run or the stored session, never a header or body (BP-031). A schedule dispatch's org only selects a cell; the stored row must name the same one | A caller who could choose the org could choose whose org's data a person reads | +| Nothing reads a one-part or two-part user cell, not even to see whether it exists (D1) | An existence check is a read across orgs, and a store call on every run | +| Every user-key site passes the org into S1. Re-run `poc/key-sites/check.mjs` and update its table in the same PR (tenet 5) | Nine files touch user keys; one that keeps the old call reopens the leak silently | +| A missing org throws; it never defaults (ER-13) | A default would be the cross-org cell under a new name | +| Org and session keys stay byte-identical (BP-030) | They are not this issue's, and every deployment's org data would otherwise move | +| Test the second paths (BP-035): a pinned seat, a flow-isolated resource, a cross-flow child, an id with `:`, a session with no org, harness seeding, each schedule producer | Each is where the obvious change passes the happy path and still crosses orgs | + +## Docs + +Reconcile [DOCS.md](DOCS.md) against shipped behaviour after VG and V7 pass, then publish +through `docs-writer` and `docs-editor`. The step's SQL is the one V7 walked, on each of the two +stores. S7's internal +docs need no draft. + +## Sketch · pseudocode, illustrative + +``` +user key for (person, org, flow, resource): + require org (throws if missing) + if resource is isolated: person : ~org : org : flow address (new) + else: person : ~org : org (today's seat cell, now everyone's) +``` + +**POC:** `poc/key-shape/` showed the two forms never equal a legacy key or each other, and that +dropping the marker collides (24,964 collisions). `poc/key-sites/` counted nine files touching +user keys and one with no org in hand, the schedule resolver, which S4 fixes. Both premises held; +nothing in the design moved. + +## At implement time + +- Re-run both POCs first. A changed count means a site moved since `fbecfe6f2`. +- 35 files read or write a raw user cell by literal scope. Besides the six source files above, + they are tests and six goals' scripts (`flow-instances`, `task-board`, `harness-manager`, + `devforce-lab` twice, `index-time-facets`). Each moves through the derivation; re-run each goal + touched and append its verdict. None is a design question. +- FIX-1788 may be specced against this key in parallel. Nothing user-scoped of its ships before + this merges. +- FIX-1538's note that the registry's schema check is stricter than needed for seats is moot: + seats and app flows share a cell again, and the check is exactly right. It adds no boot + failure at upgrade: the check already compares a pinned flow's shared user declarations with + every other flow's today, pin-blind, so a deployment that boots now passes it after. +- Compare [EVOLUTION.md](EVOLUTION.md)'s predecessor rows with current code before building. + +## Notes from review + +Recorded verbatim for the implementer to weigh against real code; not folded into the design. + +- **Slash-bearing schedule keys** (Codex, [thread](https://github.com/fixpoint-labs/flow-state-dev/pull/2810#discussion_r4199567317)): + "When a schedule collection permits nested keys such as `daily/report`, this destructuring + keeps only `daily`, so the resolver reads the wrong resource and the schedule 404s. It also + cannot distinguish structural separators from `/` inside an org or user ID after the wildcard + router decodes and rejoins segments (`packages/engine/src/routes/router.ts:343-365`) [...] Use + a reversible component encoding and parse only the two structural separators while preserving + the entire key remainder." The `scheduled.md` example in DOCS.md is to match what ships. +- **No detection for the silent-empty upgrade** (second look, item 3): "Either file it now as a + sibling of this issue and gate FIX-1788 on it, or put a copy-paste smoke query in the step's + doc, so an operator can see uncopied cells before they open the app. The changeset note alone + is easy to miss." The step's first query already lists uncopied cells and can run at any time + after the upgrade; whether the upgrade note leads with it is the implementer's call. +- **A host that builds its own dispatch** (second look, item 4): "A host that builds its own + dispatch from the old two-part id still compiles, and the resolver returns `null`, so that + schedule stops firing without an error. The upgrade note needs to list this." +- **Lift the site scan into CI** (Cursor, [thread](https://github.com/fixpoint-labs/flow-state-dev/pull/2810#discussion_r4199542427)): + "Post-S1, consider lifting the scanner + allowed-set assertion into `packages/engine/test` (or a + single shared manifest) so CI enforces totality without two frozen copies (`cell-sites` + + `key-sites`) diverging over time." +- **Derivation sites vs raw literals** (Cursor, [thread](https://github.com/fixpoint-labs/flow-state-dev/pull/2810#discussion_r4199542349)): + "align this 'nine files' inventory with PLAN's '35 literal scope_id' note — one sentence on + derivation sites vs raw scope literals would save implementers maintaining two mental lists." +- **A dry-run step command** (Cursor, [thread](https://github.com/fixpoint-labs/flow-state-dev/pull/2810#discussion_r4199542402)): + "even a dry-run CLI that only runs steps 1–2 (list + attribute) would shrink operator error + without full copy logic in the framework." +- **The pin's removal owner** (second look): "give it a removal owner in the code comment, not + only in the spec. Otherwise it becomes a permanent no-op field." +- **Workforce's copy of the org-key escaping** (second look): "If the implementer exports the + escaping helper from the engine while in `scope-keys.ts`, that copy can be deleted. Otherwise, + leave it alone." (`packages/workforce/src/clear-shadowed-references.ts`) +- **Spec shape** (Cursor, review summary items 3–5): DOCS.md as "a page checklist + acceptance + bullets"; "BR tables state behavior only; link to PLAN § Checks"; V7 narrowed to "one + attributed shared cell, one two-org `migration-required`, one schedule re-index". V7 keeps its + matrix: it is the only proof of BR-16 to BR-20 before an operator runs the step on production + data. + +## Follow-ups + +- A startup warning when a store still holds one- or two-part user cells. Needs a scope-id + listing the store contracts don't have. +- A framework-shipped copy command, if D1's *what would change my mind* happens. diff --git a/specs/issues/FIX-1790/SPEC.md b/specs/issues/FIX-1790/SPEC.md new file mode 100644 index 0000000000..a8b66c2968 --- /dev/null +++ b/specs/issues/FIX-1790/SPEC.md @@ -0,0 +1,110 @@ +# FIX-1790 · A user's data is kept per org, for every flow + +**Spec** · [Decisions](DECISIONS.md) · [Rules](BUSINESS-RULES.md) · [Plan](PLAN.md) · [Docs](DOCS.md) · [Evolution](EVOLUTION.md) + +## Five people, before and after + +| Someone who… | Today | After | +|---|---|---| +| **works in Acme and Globex** | Globex shows her the preferences, notes and private flow data she saved in Acme | Each org shows only what she saved there. Globex starts empty | +| **is about to get a private roster of workers** (FIX-1788) | Her workers would be user data, so one roster would show in both orgs | Her Acme roster is not her Globex roster | +| **uses a hired seat and the app's chat flow in one org** | The seat keeps its own per-org copy, the chat flow a cross-org one, and neither sees the other | Both use her one cell in that org, as any two flows sharing user data do | +| **upgrades a deployment with saved user data** | n/a | Each person starts empty until the operator runs a documented copy step, which copies a saved cell only into the one org its sessions name. Nothing is read across orgs or deleted | +| **sends a request with no org** | Refused since FIX-1442 | Still refused, never the cross-org cell | + +Hired seats already keep a person's data per org ([FIX-1538](../FIX-1538/SPEC.md)). This does it +for every flow, so the epic can make workers and private projects user data without leaking them +between orgs ([epic ER-3](../../epics/FIX-1786/BUSINESS-RULES.md#what-a-team-gets-and-what-it-doesnt)). + +## The goal, and how we'll know it's met + +**A person who works in two orgs sees, in each, only what they saved there, in every flow; and +what they saved before the upgrade comes back in one org at most.** + +| Is it the right goal? | | +|---|---| +| **The real need** | The PRD: "what a user keeps in one org never appears in another", for every flow, nothing deleted. The epic set "data stored before still reads" to "in one org at most, after an operator step" ([ER-3](../../epics/FIX-1786/BUSINESS-RULES.md#what-a-team-gets-and-what-it-doesnt)) | +| **Smaller, and rejected** | "The key carries the org." A one-line change that leaves a view, the schedule resolver or the test harness on the old cell | +| **Bigger, and not this issue's** | Workers and private projects as user data (FIX-1788, FIX-1793). Removing owner pins (FIX-1798) | +| **Not done if** | The check ran with one person or one org · only on a hired seat, whose cell already had the org · without a record from before the upgrade · without user state or a flow-isolated resource · the copy step's SQL was published for a store it was never walked on (SQLite and Postgres both) | + +```mermaid +flowchart LR + A["one app flow · no pin · two orgs · two people"] --> L1["leg a · Alice saves in Acme"] + L1 --> L2["leg b · Alice in Globex and Bob in Acme read"] + P0["a record saved before the upgrade"] --> L3["leg c · the upgrade and the copy step"] + L2 -->|"no marker crosses"| P["PASS · goal met"] + L3 -->|"reads in one org at most"| P + C1["control · the key without the org"] -.-> L2 + L2 -.->|"under the control"| F["must FAIL · Globex reads Acme's marker"] +``` + +The check reads what each caller gets back, not the keys. The dashed path leaves the org out of +the key, and it must fail. + +| How we verify | | +|---|---| +| **Goal check** | `goals/user-scope/keeps-a-users-data-in-the-org-it-was-saved-in/` · model `n/a` · real HTTP router, SQLite · run by the implementer · verdict in the implementation PR | +| **Signal** | a: Alice's next Acme run reads back her user state, a shared and a flow-isolated resource. b: Alice in Globex and Bob in Acme read none of the three, by a run, the state route or the resource route. c: an old marker reads in neither org after the upgrade; after the published step, a one-org person's reads in that org only and a two-org person's nowhere | +| **Input** | An app flow with no pin, three markers, two orgs, three people. Other values, and ids with `:`, must pass too | +| **Anti-game** | No assertion on a key string, on a hired seat, or on a fixture that writes the new cell. The step's statements are the published page's | +| **Control that must fail** | `GOAL_CONTROL=cross-org-key`: leg b FAILS on *Globex reads Alice's shared marker*. `GOAL_CONTROL=fallback-read`: leg c FAILS on *the old marker reads in Globex*. Today's `main` fails leg b | + +## What changes + +![Today, the app's chat flow for Alice in Acme and in Globex writes one cell named alice, while her Acme seat keeps its own cell. After, every flow writes a cell for its person and org inside the Acme or Globex box; the seat and the chat flow share alice in acme. The old alice cell sits below a fence nothing crosses, copied into one org only by the operator step](figures/cells.svg) + +Red lines on the left are the leak. On the right every flow lands inside its org's box, and the +old cell is kept but read by nothing. + +**What a developer writes doesn't change**, except two public helpers that now take the org: + +```diff +- resolveUserStorageKey(userId, { id: flow.id, isolateUserState: false, ownerPin }) ++ resolveUserStorageKey(userId, orgId, { id: flow.id, isolateUserState: false }) + // a missing or blank orgId throws; it never returns the cross-org key + +- POST /api/flows/notes/schedules/alice/daily/dispatch ++ POST /api/flows/notes/schedules/acme/alice/daily/dispatch // a dynamic schedule's id names its org +``` + +## How a run finds its cell + +```mermaid +flowchart LR + P["verified principal"] -->|"admitted · user and org"| R["a run"] + S["stored session · user and org"] --> V["a read-side view"] + R --> K["the one key derivation"] + V --> K + T["schedule dispatch · names the org"] -->|"row must name the same org"| K + K -->|"person · org · and flow when isolated"| C["the cell"] + O["the old cross-org cell"] -.->|"nothing reads it"| X["kept for the operator step"] +``` + +The org comes from the run or the stored session, never a header or body. + +## What stays as it is + +- **Session and org keys**, byte for byte, and tenant handling. +- **A hired seat's shared cell**, already `:~org:`. +- **Org is never optional** (FIX-1442), and the registry's cross-flow schema check. +- **Owner pins** stay in the engine for admission, deprecated; FIX-1798 removes them. + +## Sign off + +**[The goal](#the-goal-and-how-well-know-its-met), at that size:** every flow, every view, user +state and both kinds of resource, and old records in one org at most. If wrong: FIX-1788 builds +private rosters on a key that still crosses orgs somewhere. + +1. **[D1](DECISIONS.md#d1) · Data saved before the upgrade is read by nothing until an operator + copies it; until then each person starts empty in each org.** If wrong: a deployment that + skips the step shows everyone empty preferences and collections, with no error. +2. **[D2](DECISIONS.md#d2) · The step copies a saved cell into an org only when every session that + could have written it names that org, and the operator vouches none was deleted.** If wrong: a + person who worked in two orgs, or whose sessions were deleted, gets their shared data back in + neither, until someone decides by hand. + +**Open: none.** D1 is the one to weigh. Reasoning and what lost: [DECISIONS.md](DECISIONS.md). +The cases: [BUSINESS-RULES.md](BUSINESS-RULES.md). + +Feature · `engine` + `scheduled` + `vercel` + `bullmq` + `testing` + docs + one goal · medium · 1 PR · epic [FIX-1786](../../epics/FIX-1786/SPEC.md) diff --git a/specs/issues/FIX-1790/figures/cells.svg b/specs/issues/FIX-1790/figures/cells.svg new file mode 100644 index 0000000000..b7fe6fe930 --- /dev/null +++ b/specs/issues/FIX-1790/figures/cells.svg @@ -0,0 +1,96 @@ + + + + + + + TODAY · A PERSON'S DATA FOLLOWS THEM INTO EVERY ORG + + Chat flowAlice in Acme + + Hired seatAlice in Acme + + Chat flowAlice in Globex + + + + alice:~org:acme + the seat's own cell + + + + + alice + one cell, every org · Globex reads what Acme saved + + + + AFTER · ONE CELL PER PERSON AND ORG, EVERY FLOW + + Chat flowAlice in Acme + + Hired seatAlice in Acme + + Chat flowAlice in Globex + + + ACME + + GLOBEX + + + + + alice · acme + alice:~org:acme + every flow in Acme, the seat too + flow-isolated data adds :<flow> + + + + alice · globex + alice:~org:globex + starts empty + + + no run or view + crosses this line + + + alice + the old cell · kept, read by nothing + copied by the operator step into one org + + + diff --git a/specs/issues/FIX-1790/figures/d1-before-the-step.svg b/specs/issues/FIX-1790/figures/d1-before-the-step.svg new file mode 100644 index 0000000000..cad70c9781 --- /dev/null +++ b/specs/issues/FIX-1790/figures/d1-before-the-step.svg @@ -0,0 +1,100 @@ + + + + + + + D1 · THE CALL + What does a person see between the upgrade and the operator's copy step? + + + + + CHOSEN + Old data read by nothing; start empty + + INSTEAD OF + Refuse with migration-required + + + + + + A PERSON WHO + WORKED IN TWO ORGS + ▸ DECIDES IT + + + Works in both, starting empty + their old data waits for a decision + − + Refused on every flow + until someone resolves them by hand + + + + AN OPERATOR WHO + SKIPS THE STEP + − + No error; data looks empty + people notice before the operator does + + + Loud + every affected request says why + + + + + EACH REQUEST + + + No extra read + the old cell is unreachable by construction + − + Looks in the old cell every run + a read across orgs, and a store call + + + + + + LOCKS IN + An upgrade step for every deployment + that has saved user data, run with writers stopped + + + + + FLIPS IF + A deployment can't stop its writers + then a shipped copy command, still never a runtime read + + diff --git a/specs/issues/FIX-1790/figures/d2-attribution.svg b/specs/issues/FIX-1790/figures/d2-attribution.svg new file mode 100644 index 0000000000..ae5cd73b89 --- /dev/null +++ b/specs/issues/FIX-1790/figures/d2-attribution.svg @@ -0,0 +1,102 @@ + + + + + + + D2 · THE CALL + Which org does the step copy a person's saved data into? + + + + + CHOSEN + The one org its writers' sessions name + + INSTEAD OF + The operator's membership records + + + + + + ALICE: ACME TO JUNE, + GLOBEX SINCE + ▸ DECIDES IT + + + Shared cell stops; Acme-only flow to Acme + nothing lands in an org that didn't write it + − + Acme's data copied into Globex + membership says where she is, not where she wrote + + + + SOME SESSIONS + DELETED + − + Stops, unless you vouch + your records must clear every other org + + + Still attributed + membership outlives sessions + + + + + A DEV APP ON THE + DEFAULT ORG + = + Everything comes back + one org wrote every cell + = + Everything comes back + the same + + + + + + LOCKS IN + A two-org person's shared data returns nowhere + until someone decides by hand; nothing is deleted + + + + + FLIPS IF + The app records which org each write was in + then that record attributes more finely than sessions + + diff --git a/specs/issues/FIX-1790/poc/key-shape/README.md b/specs/issues/FIX-1790/poc/key-shape/README.md new file mode 100644 index 0000000000..6e5979f56a --- /dev/null +++ b/specs/issues/FIX-1790/poc/key-shape/README.md @@ -0,0 +1,35 @@ +# FIX-1790 · key-shape — can a per-org key name an old cell, or another person's? + +**Question.** D1 adds no runtime guard against reading a cell an older release wrote. That is +safe only if no new key can ever equal an old one. And the new keys must stay injective for ids +carrying the delimiter, the escape, or the marker text itself. + +**What it does.** Mirrors the engine's component escaping (`\` and `:` get a backslash) and builds: + +- every legacy key: `` and `:` +- every proposed key: `:~org:` and `:~org::` + +over every id of length 1–3 from `a : \ ~ o`, plus `~org`, `~org:a` and `a:~org` (158 ids per +part). Any two different tuples sharing a key fail it. It also confirms the shared key equals +FIX-1538's owner-pinned cell for every (person, org). + +**Run** (from the repo root, no install needed, about 20 seconds): + +```bash +node specs/issues/FIX-1790/poc/key-shape/check.mjs # PASS, exit 0 +node specs/issues/FIX-1790/poc/key-shape/check.mjs --plant # negative control: FAIL, exit 1 +``` + +**What it showed (2026-10-06).** PASS: 25,122 legacy keys and 3,969,276 proposed keys, all +distinct; the shared key equals the pinned cell in 24,964 of 24,964 cases. + +`--plant` drops the `~org` marker from the flow-isolated form (`::`). It +FAILED with 24,964 collisions, the first being person `~org`, org `~org`, flow `~org` against the +shared key of person `~org`, org `~org`, and exited 1. The marker is what keeps the four-part form +apart. + +**The premise held.** An old cell is unreachable by construction, so D1 needs no fallback guard +and no refusal path. Nothing in the design moved. + +Retained design evidence, a sketch of the shapes and not the implementation. Not wired into CI +or any default discovery. diff --git a/specs/issues/FIX-1790/poc/key-shape/check.mjs b/specs/issues/FIX-1790/poc/key-shape/check.mjs new file mode 100644 index 0000000000..22eb65d9c3 --- /dev/null +++ b/specs/issues/FIX-1790/poc/key-shape/check.mjs @@ -0,0 +1,108 @@ +#!/usr/bin/env node +/** + * FIX-1790 key-shape check: can the per-(user, org) keys ever name a cell an + * older release wrote, or each other's? + * + * Two premises the plan rests on: + * + * 1. Disjoint from the past. No new key equals a key an older release wrote + * (the one-part cross-org key, the two-part flow-isolated key). If that + * holds, an old cell is unreachable by construction: the runtime needs no + * guard, no fallback and no refusal to keep it from reading in two orgs. + * 2. Injective. Two different (user, org) or (user, org, flow) tuples never + * share a key, including ids carrying `:`, `\` or the literal `~org`. + * + * And one convenience it confirms: the shared key is byte-identical to the + * owner-pinned cell FIX-1538 already writes, so a seat's shared data does not + * move. + * + * The encoder below mirrors `encodeScopeKeyComponent` in + * `packages/engine/src/stores/scope-keys.ts` (escape `\` and `:` with `\`). It + * is a sketch of the proposed shapes, not the implementation. + * + * Exhaustive over every id of length 0–3 from a small alphabet that includes + * the delimiter, the escape and the marker's characters, plus the marker itself + * as a whole id. + * + * Negative control: `--plant` joins the flow-isolated key without the marker + * (`::`). That form must collide with the shared form for an + * org id of `~org`, and the check must FAIL. + * + * Retained design evidence. Not wired into CI or any default discovery. Run + * from the repo root: + * + * node specs/issues/FIX-1790/poc/key-shape/check.mjs + * node specs/issues/FIX-1790/poc/key-shape/check.mjs --plant # must exit 1 + */ + +const enc = (value) => value.replace(/[\\:]/g, "\\$&"); + +/** Today's forms, which stay on disk until an operator moves them. */ +const legacyShared = (user) => enc(user); +const legacyIsolated = (user, flow) => `${enc(user)}:${enc(flow)}`; +/** FIX-1538's owner-pinned cell, written today for hired seats. */ +const pinnedShared = (user, org) => `${enc(user)}:~org:${enc(org)}`; + +/** The proposed forms. */ +const plant = process.argv.includes("--plant"); +const nextShared = (user, org) => `${enc(user)}:~org:${enc(org)}`; +const nextIsolated = plant + ? (user, org, flow) => `${enc(user)}:${enc(org)}:${enc(flow)}` + : (user, org, flow) => `${enc(user)}:~org:${enc(org)}:${enc(flow)}`; + +const ALPHABET = ["a", ":", "\\", "~", "o"]; +const ids = new Set(["", "~org", "~org:a", "a:~org"]); +const grow = (prefix, depth) => { + ids.add(prefix); + if (depth === 0) return; + for (const ch of ALPHABET) grow(prefix + ch, depth - 1); +}; +grow("", 3); +const nonEmpty = [...ids].filter((id) => id.length > 0); + +const problems = []; +const owner = new Map(); +const claim = (key, who) => { + const prior = owner.get(key); + if (prior !== undefined && prior !== who) { + if (problems.length < 5) problems.push(`COLLISION ${JSON.stringify(key)}: ${prior} vs ${who}`); + else problems.push(null); + } + owner.set(key, who); +}; + +// Every key an older release could have written. +for (const u of nonEmpty) { + claim(legacyShared(u), `legacy-shared(${JSON.stringify(u)})`); + for (const f of nonEmpty) claim(legacyIsolated(u, f), `legacy-isolated(${JSON.stringify(u)},${JSON.stringify(f)})`); +} +const legacyCount = owner.size; + +// Every key the proposed forms write. A collision with a legacy key or with +// another tuple is a failure. +let pinnedSame = 0; +let pinnedChecked = 0; +for (const u of nonEmpty) { + for (const o of nonEmpty) { + const shared = nextShared(u, o); + claim(shared, `shared(${JSON.stringify(u)},${JSON.stringify(o)})`); + pinnedChecked += 1; + if (pinnedShared(u, o) === shared) pinnedSame += 1; + for (const f of nonEmpty) { + claim(nextIsolated(u, o, f), `isolated(${JSON.stringify(u)},${JSON.stringify(o)},${JSON.stringify(f)})`); + } + } +} + +console.log(`ids per component: ${nonEmpty.length}`); +console.log(`legacy keys: ${legacyCount}; proposed keys: ${owner.size - legacyCount}`); +console.log(`shared key equals FIX-1538's pinned cell: ${pinnedSame}/${pinnedChecked}`); + +if (pinnedSame !== pinnedChecked) problems.push("the shared key differs from the pinned cell for some (user, org)"); +if (problems.length > 0) { + const shown = problems.filter((p) => p !== null); + console.log(`\nFAIL — ${problems.length} problem(s):`); + for (const p of shown) console.log(` ${p}`); + process.exit(1); +} +console.log("\nPASS — no proposed key names a legacy cell or another tuple's cell."); diff --git a/specs/issues/FIX-1790/poc/key-sites/README.md b/specs/issues/FIX-1790/poc/key-sites/README.md new file mode 100644 index 0000000000..1a3d4b8a99 --- /dev/null +++ b/specs/issues/FIX-1790/poc/key-sites/README.md @@ -0,0 +1,39 @@ +# FIX-1790 · key-sites — where a user key comes from, and which org each site has + +**Question.** Does putting the org into the one key derivation reach every flow? The plan rests +on an enumerated fact: every production read and write of user-scoped storage takes its key from +`packages/engine/src/stores/scope-keys.ts`, and each such site already holds an org it can pass. + +**What it does.** Scans every `packages//src` TypeScript file (tests and the store adapters +excluded) for key derivations and for user-scope or variable-scope store calls, and asserts that +every file with a hit is classified with its exact counts. A new site fails until someone +classifies it. Each site's org source is read from the code by hand and printed beside it. + +**Run** (from the repo root, no install needed): + +```bash +node specs/issues/FIX-1790/poc/key-sites/check.mjs # PASS, exit 0 +node specs/issues/FIX-1790/poc/key-sites/check.mjs --plant # negative control: FAIL, exit 1 +``` + +**What it showed, at `fbecfe6f2` (2026-10-06).** PASS: 1,044 source files, 9 touch user-scoped keys. + +| Class | Files | The org it has | +|---|---|---| +| convergence | `engine/src/stores/scope-keys.ts` | takes it as an argument after S1 | +| derives | `engine/src/context/createExecutionContext.ts` | the admitted run's, required since FIX-1442 | +| derives | `engine/src/resources/internal.ts`, `engine/src/routes/state-routes.ts` | the stored session's | +| derives | `scheduled/src/createResourceCollectionScheduleResolver.ts` | **none** before the read: the schedule id names only the user | +| consumes | `engine/src/context/resource-registry.ts` | handed a derived id | +| session-only | `engine/src/routes/resource-routes.ts` | n/a | +| harness | `testing/src/runtime/createTestContext.ts`, `testing/src/test-utilities/testFlow.ts` | the harness org, but both seed the bare user id | + +`--plant` adds one unclassified `content.get("user", …)` call; the check reported +`UNCLASSIFIED … derive=0 store=1` and exited 1. + +**The premise held**, with one gap: the schedule resolver has no org until the dispatch names +one, which is plan surface S4. The two harness files bypass the derivation, which is S5. Same +nine files FIX-1538's `cell-sites` found; nothing new has appeared since. + +Retained design evidence. Not wired into CI or any default discovery. A grep can miss a call +written through an alias; the required org argument in S1 is what catches those at build time. diff --git a/specs/issues/FIX-1790/poc/key-sites/check.mjs b/specs/issues/FIX-1790/poc/key-sites/check.mjs new file mode 100644 index 0000000000..42dec626d8 --- /dev/null +++ b/specs/issues/FIX-1790/poc/key-sites/check.mjs @@ -0,0 +1,170 @@ +#!/usr/bin/env node +/** + * FIX-1790 factual-base check: where does a user-scoped storage key come from, + * and does each site have an organization in hand? + * + * The plan rests on one enumerated fact: every production read and write of + * user-scoped storage takes its key from the derivation in + * `packages/engine/src/stores/scope-keys.ts`, so putting the organization into + * that derivation reaches every flow, with the exceptions the plan names. This + * script re-derives the site list from the source instead of from memory. The + * organization each site has in hand is recorded beside it (`org`), read by + * hand from the code; the plan's surfaces follow from that column. + * + * It scans every `packages//src/**\/*.ts` (tests excluded, and the store + * adapters excluded, because they store what they are handed rather than + * deriving a key) for two families of call: + * + * derive — `resolveUserStorageKey(`, `resolveResourceScopeId(`, `resourceScopeIds(` + * store — `stores.user.(`, and any `resourceState.*` / + * `content.*` call whose scope argument is not a literal + * `"session"`, `"org"` or `"lineage"` + * + * Totality, not a spot check: every file with a hit must appear in `SITES` with + * the exact count of each family. A new file, or a new call in a listed file, + * fails until someone classifies it. + * + * Negative control: `--plant` adds one synthetic source file carrying an + * unclassified user-scope read. The check must FAIL on it. + * + * Retained design evidence. Not wired into CI or any default discovery. Run + * from the repo root: + * + * node specs/issues/FIX-1790/poc/key-sites/check.mjs + * node specs/issues/FIX-1790/poc/key-sites/check.mjs --plant # must exit 1 + * + * Adapted from FIX-1538's `poc/cell-sites/check.mjs`, which asked the same + * question about the owner pin. + */ +import { readdirSync, readFileSync, statSync } from "node:fs"; +import { join, relative } from "node:path"; + +const ROOT = process.cwd(); + +/** + * Every file that derives or touches a user-scoped key. + * + * convergence — the derivation itself; the change lands here + * derives — calls the derivation; `org` says where its organization comes from + * consumes — stores a scopeId it was handed by a `derives` site + * session-only — variable scope argument, but only ever session or lineage + * harness — test seeding helpers that build a bare user key themselves + */ +const SITES = { + "packages/engine/src/stores/scope-keys.ts": { + derive: 5, store: 0, class: "convergence", org: "takes it as an argument after this change", + }, + "packages/engine/src/context/createExecutionContext.ts": { + derive: 3, store: 11, class: "derives", org: "the admitted run (options.orgId, required since FIX-1442)", + }, + "packages/engine/src/resources/internal.ts": { + derive: 2, store: 4, class: "derives", org: "the stored session (session.orgId)", + }, + "packages/engine/src/routes/state-routes.ts": { + derive: 1, store: 1, class: "derives", org: "the stored session (session.orgId)", + }, + "packages/scheduled/src/createResourceCollectionScheduleResolver.ts": { + derive: 1, store: 1, class: "derives", org: "NONE before the read: the schedule id names only the user", + }, + "packages/engine/src/context/resource-registry.ts": { + derive: 0, store: 4, class: "consumes", org: "n/a, handed a derived id", + }, + "packages/engine/src/routes/resource-routes.ts": { + derive: 0, store: 10, class: "session-only", org: "n/a", + }, + "packages/testing/src/runtime/createTestContext.ts": { + derive: 0, store: 2, class: "harness", org: "options.orgId, optional; seeds the bare user id", + }, + "packages/testing/src/test-utilities/testFlow.ts": { + derive: 0, store: 4, class: "harness", org: "options.orgId, optional; seeds the bare user id", + }, +}; + +const DERIVE = /\b(resolveUserStorageKey|resolveResourceScopeId|resourceScopeIds)\(/g; +const STORE_USER = /\bstores\.user\.(get|set|delete|list)\(/g; +const STORE_SCOPED = + /\b(resourceState|content)\.(get|getAll|getByPrefix|set|delete|deleteAll|purgeTombstones)\(\s*(?!"(session|org|lineage)")/g; + +const EXCLUDED = [ + /\.test\.ts$/, + /\/test\//, + /^packages\/engine\/src\/stores\/(memory|filesystem)\//, + /^packages\/store-(sqlite|postgres)\//, +]; + +function walk(dir, out) { + for (const name of readdirSync(dir)) { + if (name === "node_modules" || name === "dist") continue; + const path = join(dir, name); + if (statSync(path).isDirectory()) walk(path, out); + else if (name.endsWith(".ts")) out.push(path); + } + return out; +} + +/** Drop comment lines so a doc comment quoting a call is not a call. */ +function code(text) { + return text + .split("\n") + .filter((line) => !/^\s*(\*|\/\/|\/\*)/.test(line)) + .join("\n"); +} + +function count(text, pattern) { + return [...text.matchAll(pattern)].length; +} + +const files = []; +for (const pkg of readdirSync(join(ROOT, "packages"))) { + const src = join(ROOT, "packages", pkg, "src"); + try { + if (statSync(src).isDirectory()) walk(src, files); + } catch { + // a package with no src/ has nothing to classify + } +} + +const sources = files + .map((path) => ({ rel: relative(ROOT, path), text: code(readFileSync(path, "utf8")) })) + .filter(({ rel }) => !EXCLUDED.some((rule) => rule.test(rel))); + +if (process.argv.includes("--plant")) { + sources.push({ + rel: "packages/engine/src/routes/planted-negative-control.ts", + text: 'const row = await ctx.stores.content.get("user", session.userId, key);', + }); +} + +const found = {}; +for (const { rel, text } of sources) { + const derive = count(text, DERIVE); + const store = count(text, STORE_USER) + count(text, STORE_SCOPED); + if (derive + store > 0) found[rel] = { derive, store }; +} + +const problems = []; +for (const [rel, hit] of Object.entries(found)) { + const want = SITES[rel]; + if (want === undefined) { + problems.push(`UNCLASSIFIED ${rel}: derive=${hit.derive} store=${hit.store}`); + } else if (want.derive !== hit.derive || want.store !== hit.store) { + problems.push( + `CHANGED ${rel}: expected derive=${want.derive} store=${want.store}, found derive=${hit.derive} store=${hit.store}` + ); + } +} +for (const rel of Object.keys(SITES)) { + if (found[rel] === undefined) problems.push(`GONE ${rel}: classified but has no hits`); +} + +console.log(`scanned ${sources.length} source files; ${Object.keys(found).length} touch user-scoped keys`); +for (const [rel, site] of Object.entries(SITES)) { + console.log(` ${site.class.padEnd(12)} ${rel}\n org: ${site.org}`); +} + +if (problems.length > 0) { + console.log("\nFAIL — the site inventory no longer matches the source:"); + for (const problem of problems) console.log(` ${problem}`); + process.exit(1); +} +console.log("\nPASS — every user-scoped key site is classified, and the counts match.");