From c3153620cbbd22d6aae4d72e87fa1cd59b78221f Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 20 Sep 2026 16:34:58 +0000 Subject: [PATCH 1/4] A reference is read from its file; its reach comes from the tree MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `references/` joins `resources/` at the same four levels, minted into the same namespace by the same rule. What differs is what the document is afterwards, and both halves of that are the point. D1 — the read path. A reference installs as `contentFile` + `writable: false` + `llmWritable: false` + a `render` that strips the file's frontmatter. The seal is what makes the read path work rather than decorate it: a stored row beats `contentFile` exactly as it beats `content`, so unsealed it would be a first-boot seed with a longer name. All three keys are DERIVED and refused in frontmatter, so there is no key an author can forget. The handle still carries `writeContent` — that is a runtime refusal, not an absent verb, and `references-from-docs`'s header says so in as many words rather than letting a weaker guarantee ship under a stronger word. The frontmatter strip is not cosmetic. `contentFile` hands the engine the raw bytes, so without `render` every prompt would carry the YAML block. The tests assert bodies with `toBe` for exactly this reason. D2 — the wall. A seat reaches the references at or above its place in the tree: the org's, its own team's, its own folder's. Not a sibling team's, not a teammate's. Derived from the loader-minted ref and the loader-minted seat id, never from anything a seat wrote; an explicit `references:` list narrows within it and is refused if it names anything outside. Checked again on the MINTED instance, because a reference a block also declares merges back after the seat's map replaces the flow-level one. BR-19 — the migration, which would otherwise ship silently. Sealing stops new writes and cannot evict a row already there, so a handbook written before the move stays shadowed forever. `clearShadowedReferences` finds those rows, reports each with the body it had been serving, and clears them. Idempotent, `dryRun`-able, never touches a `resources/` row. `resources/` is unchanged. Its reader, records, install and reach are byte for byte as they were — `read-resources-directory.test.ts` and `seat-resources.test.ts` pass untouched, which is the evidence for it. The one place that slipped (a `filePath` on every record) was caught by that suite and narrowed to references only. Evidence: 670 tests pass in workforce (36 new, real path: tree on disk -> loader -> hire -> live execution context). Four mutations confirm each mechanism is load-bearing — removing the seal reds 3, the frontmatter strip 11, the wall 4, the migration's delete 1. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016qaz1VpRK4wCQzxpjzZ4Hz --- .changeset/references-read-path.md | 5 + apps/docs/docs/workforce/documents-on-disk.md | 123 +++- packages/workforce/README.md | 27 +- .../src/clear-shadowed-references.ts | 143 ++++ .../src/codegen/discover-resource-modules.ts | 40 +- packages/workforce/src/describe-value.ts | 19 + packages/workforce/src/hire.ts | 111 ++- packages/workforce/src/index.ts | 15 + packages/workforce/src/loader/index.ts | 4 + .../src/loader/read-declared-roster.ts | 75 +- .../src/loader/read-resources-directory.ts | 142 +++- .../src/loader/resource-convention.ts | 44 +- packages/workforce/src/manifest.ts | 85 +++ .../workforce/src/references-from-docs.ts | 191 +++++ packages/workforce/src/seat-references.ts | 492 +++++++++++++ packages/workforce/src/seat-resources.ts | 10 +- .../test/references-convention.test.ts | 678 ++++++++++++++++++ 17 files changed, 2146 insertions(+), 58 deletions(-) create mode 100644 .changeset/references-read-path.md create mode 100644 packages/workforce/src/clear-shadowed-references.ts create mode 100644 packages/workforce/src/describe-value.ts create mode 100644 packages/workforce/src/references-from-docs.ts create mode 100644 packages/workforce/src/seat-references.ts create mode 100644 packages/workforce/test/references-convention.test.ts diff --git a/.changeset/references-read-path.md b/.changeset/references-read-path.md new file mode 100644 index 0000000000..87f9dc8343 --- /dev/null +++ b/.changeset/references-read-path.md @@ -0,0 +1,5 @@ +--- +"@flow-state-dev/workforce": minor +--- + +A workforce tree can declare read-only documents in `references/` beside the writable ones in `resources/`. A reference's body is served from its file on every execution context rather than from a stored row, so editing the file is the edit; `writable`, `llmWritable` and `render` are derived by the folder and refused in frontmatter. A seat reaches the references at or above its place in the tree — the org's, its own team's and its own folder's — with no install-side filter, narrowed further by a `references:` list in its `WORKER.md`. `resources/` behaviour is unchanged. `clearShadowedReferences` migrates a tree where a document was written before it moved (FIX-1467). diff --git a/apps/docs/docs/workforce/documents-on-disk.md b/apps/docs/docs/workforce/documents-on-disk.md index ddf357093d..bd9b2e45f3 100644 --- a/apps/docs/docs/workforce/documents-on-disk.md +++ b/apps/docs/docs/workforce/documents-on-disk.md @@ -9,34 +9,47 @@ description: "Write a team's shared documents as Markdown files, read the tree a A handbook, a glossary, an escalation procedure: the reference material a team shares and an agent reads. You can declare each one in TypeScript with `defineResource`. You can also write it as a Markdown file and read the folder at startup, which lets someone who does not write TypeScript edit it. -`readResourcesDirectory` turns the folder tree into plain records. `resourcesFromDocs` turns those records into the resource map you pass to `defineFlow`. +There are two folders, and which one you use decides where the document lives afterwards. + +| Folder | The body an agent reads | Who can change it | Who can read it | +|---|---|---|---| +| `references/` | the file on disk, re-read each request | whoever can edit the file in your repo | seats at or below it in the tree | +| `resources/` | the file's body at first boot, then whatever the product last wrote | the file, until something writes it; after that, the product | any seat the flow installed it on | + +Put a company handbook in `references/`. Editing the file is the edit, and it keeps reaching agents on the next deploy no matter what else happens. Put an agent's working notes in `resources/`, where they can be written and survive the turn. + +`readReferencesDirectory` and `readResourcesDirectory` turn the folder tree into plain records. `referencesFromDocs` and `resourcesFromDocs` turn those records into the resource maps you pass to `defineFlow`. Documents live in the same tree as [workers](./workers-on-disk.md) and [skills](../skills/overview.md), so one folder can describe a whole workforce. ## The tree -A `resources/` folder is read wherever the tree puts one: at the organization level, in a team, and inside a single worker's own folder. +A `references/` or `resources/` folder is read wherever the tree puts one: at the organization level, in a team, and inside a single worker's own folder. ``` workforce/ org/ - resources/ + references/ code-of-conduct.md teams/ engineering/ - resources/ + references/ handbook.md + resources/ + scratch.md workers/ on-call/ - WORKER.md - resources/ + references/ runbook.md + WORKER.md support/ - resources/ + references/ escalation.md ``` -Four documents: one shared across the organization, two belonging to a team, and one belonging to a single seat. Every example below reads this tree. +Five documents: one shared across the organization, two belonging to a team, one belonging to a single seat, and one scratchpad the team's agents can write. Every example below reads this tree. + +Both folders name documents the same way, and they share one namespace. `references/handbook.md` and `resources/handbook.md` in one team are two spellings of the same name, so the pair is refused at startup rather than one of them quietly winning. A worker's own folder is what lets two seats each have a `runbook` without their authors agreeing on a name. Organization-level workers, under `org/workers/`, are read the same way. @@ -71,6 +84,8 @@ Where a file sits decides its identity, where it is stored, and what its content Everything outside that set arrives at the resource exactly as written, so `llmReadable`, `llmWritable`, `writable`, `allowedExtensions` and `metadata` all take effect from the file. +A file in `references/` derives three more: `writable`, `llmWritable` and `render`. The folder is what makes a reference read-only, so no file has to ask for that and no file can turn it off. Declaring any of the three is refused by name, `writable: false` included — agreeing with the folder is still a second place to keep the same fact, and the next person to open the file cannot tell which copy is the one doing the work. A document that needs to be written belongs in `resources/`. + ## A document's ref A document's ref is its identity. It is built from the folders above the file, joined with slashes. At the organization level it is the bare file name: @@ -111,9 +126,12 @@ interface ResourceDoc { ref: string; // "teams/engineering/handbook" declared: Record; // the frontmatter, exactly as written body: string; // the Markdown below it, verbatim + filePath?: string; // where it was read from — references only } ``` +`readReferencesDirectory` has the same signature and returns the same records, each carrying the `filePath` its content is served from. + For the handbook above: ```ts @@ -192,6 +210,35 @@ export const supportFlow = defineFlow({ Every file-declared document is installed at `org` scope, with the file's body as the resource content. +References install the same way, through `referencesFromDocs`, and both maps spread into the same flow: + +```ts +const references = await readReferencesDirectory("./workforce"); +const documents = await readResourcesDirectory("./workforce"); + +export const supportFlow = defineFlow({ + kind: "support", + actions: { answer: { block: answerQuestion } }, + resources: { + ticket: ticketResource, + ...resourcesFromDocs(documents.documents), + ...referencesFromDocs(references.documents), + }, +}); +``` + +Pass both to `hireWorkforce` as well, so it knows which entries are references and can hold each seat to its place in the tree: + +```ts +hireWorkforce(workers, { + kinds, + documents: resourcesFromDocs(documents.documents), + references: referencesFromDocs(references.documents), +}); +``` + +A reference's content is read from its file whenever an execution context is built, so editing the file in your repository reaches agents on the next request. A read already in flight keeps the body it started with. + Spread the map rather than passing it on its own. A flow copy created with `supportFlow({ resources })` *replaces* the definition's map instead of merging with it, so a copy handed only `resourcesFromDocs(documents)` loses whatever the flow kind declared. `resourcesFromDocs` throws rather than collecting. A record it cannot turn into a resource stops startup, naming the ref, the same way a refused hire does. @@ -209,11 +256,37 @@ await ctx.resources.get("teams/engineering/handbook").readContent(); Put `requireOrg: true` on the blocks that read a document, as the example in the next section does. The flow then turns away a request with no org up front, instead of running it and coming up empty. How a request carries its org is covered in [the client reference](/docs/configuration/client). -### A folder is a namespace, not a visibility boundary +### Who reaches what + +**For a `references/` document, the folder is the boundary.** A seat reaches the references at or above its own place in the tree: the organization's, its own team's, and its own folder's. Not another team's, and not a teammate's folder. You write no filter, and there is no setting to get wrong. + +``` +teams/engineering/workers/ada/ a seat here reads… + org/references/code-of-conduct yes — the organization is above everyone + teams/engineering/references/handbook yes — its own team + teams/engineering/workers/ada/references/runbook yes — its own folder + teams/engineering/workers/ivan/references/runbook no — a teammate's folder + teams/support/references/escalation no — another team +``` + +To let a reference reach more people, move the file up the tree. That is the only way to widen it, which is what makes the boundary worth trusting: nothing on the install side can quietly open it back up. + +A seat that wants less than its place gives it lists what it wants in its own file. The list narrows and never widens, so naming a document the seat could not already reach is refused at startup rather than granted: + +```md +--- +flow: desk +description: Engineering desk +references: + - teams/engineering/handbook +--- +``` + +Leaving the key out means every reference at or above the seat. Writing `references: []` means none — the two are different answers, not the same one. -Every file-declared document is org-scoped, and a generator's resource tools reach every installed document marked `llmReadable`, with no per-team filter. Install a whole tree on one flow and every team's documents are reachable from it. +**For a `resources/` document, the folder is a namespace and nothing more.** Every one is org-scoped, and a generator's resource tools reach every installed document marked `llmReadable`. Install a whole tree on one flow and every team's writable documents are reachable from it, one team from another included. -To give one team's seats only its own documents, filter the records before installing them: +So if you want a team's writable documents kept to that team, filter the records before installing them: ```ts const engineering = resourcesFromDocs( @@ -221,7 +294,7 @@ const engineering = resourcesFromDocs( ); ``` -The same is true one level down, and it is worth being plain about. Putting a document under `workers/on-call/` addresses it to that seat. It does not keep it from the others. Seats hired into one kind share that kind's flow definition, so by default every one of them reads the same row. +The same is true one level down. Putting a `resources/` document under `workers/on-call/` addresses it to that seat; it does not keep it from the others, because seats hired into one kind share that kind's flow definition. Filtering decides what a whole kind installs. To narrow one seat within a kind, the seat's own file names the documents it may touch, and can take one read-only: see [what a `WORKER.md` says](./workers-on-disk.md#what-a-workermd-says). A seat naming a document its kind was not installed with is refused at the hire, so the filter holds. @@ -260,6 +333,28 @@ export const answerQuestion = generator({ The documents are already declared on the flow, so the generator does not declare them again. The tool addresses a document by its scope-qualified uri, the same handle the [search tools](/docs/resources/searching) return. See [LLM access patterns](/docs/resources/overview#llm-access-patterns). +## Moving a document into `references/` + +Moving the file is usually the whole job. One case needs a second step, and it is quiet enough to be worth knowing about: if anything ever wrote that document while it was in `resources/`, the written body is still stored, and a stored body is what an agent reads. Move the file and the agent keeps reading the old write — the file looks authoritative in your repository and reaches nobody. + +`clearShadowedReferences` finds those and clears them, so the file is the source again: + +```ts +import { clearShadowedReferences, describeShadowedReferences } from "@flow-state-dev/workforce"; + +const result = await clearShadowedReferences({ + references: referencesFromDocs(references.documents), + orgId, + content: stores.content, +}); +console.log(describeShadowedReferences(result)); +// references: 1 of 4 were shadowed by a stored write and have been cleared — handbook. +``` + +It reports before it is believed: `result.cleared` names each reference and carries the body that had been served in the file's place, which is the last moment that text exists anywhere. Pass `dryRun: true` to see the finding and change nothing. + +Run it once per organization after the move. It is safe to run again — a tree whose files are already the source reports nothing cleared — and it never touches a `resources/` document. + ## What stays in TypeScript `defineResource` is the other route into the same map, and the two sit side by side, as `ticket` and the file-declared documents do above. @@ -272,4 +367,6 @@ A document that needs a state schema of its own, a `render` function, [reactive - It does not follow symlinks, at any level of the walk. - It does not install anything. `readResourcesDirectory` hands back records, `resourcesFromDocs` hands back a map, and you put the map on your flow. - It does not read `WORKER.md` or any `skills/` folder. Those are [Workers on disk](./workers-on-disk.md) and [Skills](../skills/overview.md). It reads a worker folder only to find a `resources/` folder inside it. -- It does not decide which seat may read which document. `flowIsolation` in a document gives each seat its own copy; nothing here refuses a read. +- It does not put a document on an agent's bash mount. A mount carries collections; a document is a single resource, so a shell on the sandbox will not find the handbook as a file. +- It does not watch a reference for changes within a request. The file is read when an execution context is built, so an edit reaches the next request rather than a read already running. +- For a `resources/` document, it does not decide which seat may read which one. `flowIsolation` gives each seat its own copy; nothing there refuses a read. A `references/` document is the exception, and [Who reaches what](#who-reaches-what) is the rule. diff --git a/packages/workforce/README.md b/packages/workforce/README.md index 93da9f54ef..c95bfa863d 100644 --- a/packages/workforce/README.md +++ b/packages/workforce/README.md @@ -744,6 +744,19 @@ A team's shared documents — a handbook, a glossary, an escalation procedure instead of `defineResource` stanzas. Frontmatter is settings and the body is the document, the same bargain `WORKER.md` makes. +**Two slots, one namespace.** The convention reads `references/` and `resources/` at the same four +levels, and mints refs for both by the same rule — so one basename claimed in both at one level is +refused rather than resolved. What differs is what the document IS afterwards: + +| Slot | Content an agent reads | Writable | Reach | +| --- | --- | --- | --- | +| `references/` | the file, re-read on each execution context | no — `writable`, `llmWritable` and `render` are derived, and a file declaring any of them is refused | derived from the tree: the org's, the seat's own team's, and the seat's own folder | +| `resources/` | the file's body seeds a row; the row is the source from then on | yes, unless the file says otherwise | every document the flow was installed with | + +Put standing material an author owns in the repository under `references/`; put anything an agent +writes under `resources/`. The `resources/` half is unchanged — same reader, same records, same +install, same reach. + **A resource is a file, not a folder.** A document is `.md` directly in `resources/`, unlike a worker or a skill, which is a folder with a fixed file inside it. A directory in a `resources/` slot lands in `errors` rather than being passed over. @@ -772,7 +785,19 @@ Escalate anything customer-visible within 15 minutes. ``` `readResourcesDirectory` walks all four and returns one record per document; `resourcesFromDocs` -turns those records into the resource map you already pass to a flow. +turns those records into the resource map you already pass to a flow. `readReferencesDirectory` and +`referencesFromDocs` are the same pair over `references/`, and both maps spread into one flow. + +A reference's record also carries the `filePath` it was read from — that path is what the install +half points the resource at, and it is why editing the file reaches agents on the next request. +Pass both maps to `hireWorkforce` (`documents` and `references`) so it can hold each seat to the +references at or above its place in the tree. + +Moving a file from `resources/` into `references/` needs one extra step when anything ever wrote +that document: the written body is still stored, and a stored body wins over the file. +`clearShadowedReferences({ references, orgId, content })` finds those rows, reports each one with +the body it had been serving, and clears them. It is idempotent, takes `dryRun`, and never touches +a `resources/` document. ```ts import { defineFlow } from "@flow-state-dev/core"; diff --git a/packages/workforce/src/clear-shadowed-references.ts b/packages/workforce/src/clear-shadowed-references.ts new file mode 100644 index 0000000000..559f6f96db --- /dev/null +++ b/packages/workforce/src/clear-shadowed-references.ts @@ -0,0 +1,143 @@ +/** + * The `references/` migration — find and clear the stored rows that shadow a + * reference's file, for a document that was written while it still lived under + * `resources/`. + * + * **Why this is work and not a note.** Moving a file from `resources/` to + * `references/` changes how it is served: the install half points the resource + * at the file and seals it, so nothing can write it again. What the seal cannot + * do is evict a row that is ALREADY there. A stored content row wins over + * `contentFile` exactly as it wins over `content`, so a handbook that anything + * wrote once — a block, an HTTP content PUT, an agent with the write tool — + * keeps serving that write after the move. Forever, silently, with the file in + * git looking authoritative and being read by nobody. + * + * So for a tree that has ever been written to, `git mv` does not make the + * promise true. This does. + * + * **The old shape is tolerated, not assumed away** (BP-030). A row may be + * there or may not; both are ordinary. What this refuses to do is leave a + * present one in place and say nothing — the whole point is that the outcome + * is reported rather than silent, so a deploy log says which handbooks were + * unshadowed and what they had been serving. + * + * **Run it once per org, after the move, before or at boot.** It is idempotent: + * a second run over a migrated tree finds nothing and clears nothing, because + * the seal means no new row can appear. + * + * Isomorphic by construction — it takes the store it writes through rather than + * reaching for one, so this module has no `node:fs` and no engine import. + */ + +import type { DeclaredResources } from "@flow-state-dev/core"; + +/** + * The slice of the engine's `ContentStore` this needs: read one row, delete one + * row. + * + * Declared structurally rather than imported, so `workforce` does not take a + * dependency on `engine` for two method signatures. Any `ContentStore` + * satisfies it — `stores.content` is passed straight in. + */ +export interface ReferenceContentStore { + /** Read a single resource's stored content, or `undefined` when there is none. */ + get(scopeType: "org", scopeId: string, resourceKey: string): Promise; + /** Delete a single resource's stored content. */ + delete(scopeType: "org", scopeId: string, resourceKey: string): Promise; +} + +/** One reference that had a stored row shadowing its file. */ +export interface ShadowedReference { + /** The reference's ref, which is also its storage key. */ + ref: string; + /** + * What the stored row was serving instead of the file. + * + * Carried so the caller can log it, diff it, or keep it: this is the only + * moment the written body still exists anywhere, and a migration that + * discards it without showing it is a migration that loses an edit somebody + * made on purpose. Truncation is the caller's call, not this function's. + */ + shadowedContent: string; +} + +/** What {@link clearShadowedReferences} found and did. */ +export interface ClearShadowedReferencesResult { + /** + * Every reference whose row was cleared, in catalog order. + * + * **Empty is the ordinary answer** — a tree that was never written to, or one + * already migrated. Non-empty is the finding: each entry is a file that was + * being ignored in favour of a stored write. + */ + cleared: ShadowedReference[]; + /** + * Every reference that was checked. `cleared.length` out of + * `checked.length` is the line worth logging, because "cleared 0" and + * "checked nothing" are the two outcomes a silent migration confuses. + */ + checked: string[]; +} + +/** + * Clear every stored content row shadowing a reference's file, for one org. + * + * @param input.references The app's references, keyed by ref — the map + * `referencesFromDocs` returns. Only these keys are touched; a `resources/` + * document's row is never read and never deleted. + * @param input.orgId The org whose rows to check. References are org-scoped, so + * this is the scope id their rows are stored under. + * @param input.content The engine's content store — `stores.content`. + * @param input.dryRun When `true`, report what WOULD be cleared and delete + * nothing. For a first run against a tree nobody wants to surprise. + * @returns What was checked and what was cleared. Nothing is thrown for an + * absent row; that is the ordinary case. + */ +export async function clearShadowedReferences(input: { + references: DeclaredResources; + orgId: string; + content: ReferenceContentStore; + dryRun?: boolean; +}): Promise { + const { references, orgId, content, dryRun = false } = input; + + const cleared: ShadowedReference[] = []; + const checked: string[] = []; + + for (const ref of Object.keys(references)) { + checked.push(ref); + // The ref IS the storage key for a file-declared document: `resourcesFromDocs` + // and `referencesFromDocs` both pass it as `ref`, and the registry keys + // content by it. + const shadowedContent = await content.get("org", orgId, ref); + // `undefined` is no row. An empty STRING is a row — something wrote a blank + // body — and it shadows the file just as completely, so it is cleared too. + // `== null` rather than a falsy check, for exactly that reason. + if (shadowedContent == null) continue; + + cleared.push({ ref, shadowedContent }); + if (!dryRun) await content.delete("org", orgId, ref); + } + + return { cleared, checked }; +} + +/** + * The migration's result as one line for a deploy log. + * + * Exported so the sentence is the same wherever it is printed, and because the + * caller that most needs it is a boot script that should not have to invent + * one. Bodies are not included — they can be a whole handbook; the result + * carries them for a caller that wants them. + */ +export function describeShadowedReferences(result: ClearShadowedReferencesResult): string { + if (result.checked.length === 0) return "references: none declared, nothing to migrate"; + if (result.cleared.length === 0) { + return `references: ${result.checked.length} checked, none shadowed — every file is the source`; + } + return ( + `references: ${result.cleared.length} of ${result.checked.length} were shadowed by a ` + + `stored write and have been cleared — ${result.cleared.map((r) => r.ref).join(", ")}. ` + + `Each now serves its file again.` + ); +} diff --git a/packages/workforce/src/codegen/discover-resource-modules.ts b/packages/workforce/src/codegen/discover-resource-modules.ts index 52b66823cb..81f2fd15ba 100644 --- a/packages/workforce/src/codegen/discover-resource-modules.ts +++ b/packages/workforce/src/codegen/discover-resource-modules.ts @@ -35,6 +35,7 @@ import path from "node:path"; import { DOCUMENT_EXTENSION, IGNORED_ENTRIES, + REFERENCES_SLOT, RESOURCES_SLOT, WORKERS_LEVEL, classify, @@ -249,11 +250,37 @@ async function readSlot(slotDir: string, slotPath: string, ctx: SlotContext): Pr // ONE ref between them, and a silent winner is the failure this convention // exists to remove — so the pair is refused here, where both names are in // hand, rather than by a second walk of Door A's tree. - const documents = new Set( - entries - .filter((entry) => entry.endsWith(DOCUMENT_EXTENSION)) - .map((entry) => entry.slice(0, -DOCUMENT_EXTENSION.length)), + // Basename -> the path of the document file claiming it, so the refusal names + // the file an author has to go and change. + const documents = new Map(); + for (const entry of entries) { + if (!entry.endsWith(DOCUMENT_EXTENSION)) continue; + documents.set(entry.slice(0, -DOCUMENT_EXTENSION.length), `${slotPath}/${entry}`); + } + + // **And the `references/` slot beside it**, which mints into the SAME + // namespace from a different folder. `references/handbook.md` and + // `resources/handbook.ts` at one level are one ref between them, exactly as + // two files in this folder are — the only difference is that the pair sits + // one directory apart, so the listing above cannot see it. + // + // Read here rather than by teaching the whole walk a second slot: modules + // live in `resources/` only, so what this door needs from `references/` is + // its basenames and nothing else. An absent or unreadable sibling is silent — + // the Markdown door reports that folder's own problems, and reporting them + // here too would double every entry in a tree with one bad slot. A resources/ + // file wins the map only because it was inserted first; either path names a + // real claimant, and the pair is refused whichever is quoted. + const referenceSlotPath = `${slotPath.slice(0, -RESOURCES_SLOT.length)}${REFERENCES_SLOT}`; + const siblings = await openStructuralDirectory( + path.join(path.dirname(slotDir), REFERENCES_SLOT), + referenceSlotPath, ); + for (const entry of siblings.entries ?? []) { + if (!entry.endsWith(DOCUMENT_EXTENSION)) continue; + const name = entry.slice(0, -DOCUMENT_EXTENSION.length); + if (!documents.has(name)) documents.set(name, `${referenceSlotPath}/${entry}`); + } for (const entryName of entries) { if (IGNORED_ENTRIES.has(entryName)) continue; @@ -296,9 +323,10 @@ async function readSlot(slotDir: string, slotPath: string, ctx: SlotContext): Pr continue; } - if (documents.has(name)) { + const claimant = documents.get(name); + if (claimant !== undefined) { ctx.problems.push( - `"${entryPath}" and "${slotPath}/${name}${DOCUMENT_EXTENSION}" both declare ` + + `"${entryPath}" and "${claimant}" both declare ` + `"${ref}" — a document and a module cannot share a ref`, ); continue; diff --git a/packages/workforce/src/describe-value.ts b/packages/workforce/src/describe-value.ts new file mode 100644 index 0000000000..20f613c5a0 --- /dev/null +++ b/packages/workforce/src/describe-value.ts @@ -0,0 +1,19 @@ +/** + * How a refusal names a value it will not accept: short, quoted, and never a + * sprawling dump. + * + * Shared by the two seat-declaration modules — `./seat-resources` and + * `./seat-references` — because an author who mis-indents one key and then the + * other should not get two different descriptions of the same mistake. It + * moved here from `seat-resources` when the second caller arrived; a second + * copy is how "a mapping of 2 key(s)" and "an object" end up in one file. + */ + +/** A value as a refusal names it. */ +export function describe(value: unknown): string { + if (value === undefined) return "undefined"; + if (typeof value === "string") return JSON.stringify(value); + if (value === null || typeof value !== "object") return JSON.stringify(value) ?? String(value); + if (Array.isArray(value)) return `a list of ${value.length}`; + return `a mapping of ${Object.keys(value as object).length} key(s)`; +} diff --git a/packages/workforce/src/hire.ts b/packages/workforce/src/hire.ts index 48fb389f26..a04eb459ec 100644 --- a/packages/workforce/src/hire.ts +++ b/packages/workforce/src/hire.ts @@ -55,18 +55,28 @@ import { verifySeatNarrowing, type SeatResourceGrant } from "./seat-resources"; +import { + SEAT_REFERENCES_KEY, + applyReferenceWall, + verifySeatReferenceWall +} from "./seat-references"; import { workerConfigSchema } from "./worker-config"; /** * The keys the factory itself reads. Everything else is the worker's settings. * - * `resources` joins them rather than travelling to the kind as a setting: it - * is an instruction to THIS step about which documents to mint the seat with, - * not a value any kind declares. A kind that happened to declare a `resources` - * setting of its own no longer receives an authored one — the key is public - * and pinned here, and a seat file cannot mean both things at once. + * `resources` and `references` join them rather than travelling to the kind as + * settings: each is an instruction to THIS step about what to mint the seat + * with, not a value any kind declares. A kind that happened to declare a + * setting of either name no longer receives an authored one — both keys are + * public and pinned here, and a seat file cannot mean two things at once. */ -const RESERVED_KEYS = ["flow", "description", SEAT_RESOURCES_KEY] as const; +const RESERVED_KEYS = [ + "flow", + "description", + SEAT_RESOURCES_KEY, + SEAT_REFERENCES_KEY +] as const; /** * The stock `agent` kind, built once for the life of the module. @@ -151,6 +161,25 @@ export interface HireOptions { * resolving every ref to nothing — a lockout that reads like a typo. */ documents?: DeclaredResources; + + /** + * The references this app declared, keyed by ref — the map + * `referencesFromDocs(references)` returns, handed over unchanged. + * + * **Handed over for the reason `documents` is, and used for more.** It says + * which entries on a kind's map are references, which is what the derived + * tree wall narrows: a seat reaches the references at or above its place in + * the tree, and no others. That wall applies whether or not the seat declares + * `references:` — the key narrows within it, and can never widen past it. + * + * Optional. Absent, no entry is a reference, no wall applies, and every seat + * is minted exactly as it was before this existed. + * + * A ref may not appear in both this map and {@link HireOptions.documents}: + * the two behave differently and one accessor cannot be both, so the pair is + * refused rather than resolved. + */ + references?: DeclaredResources; } /** A flow whose settings schema is `TConfigSchema`, whatever it declares elsewhere. */ @@ -386,6 +415,24 @@ export function hireWorkforce( // carries an empty tool catalog, because nothing ever merges into ours. const kinds: Record = { [AGENT_KIND]: builtInAgentWorkerFlow, ...options.kinds }; + // One ref cannot be both a document and a reference. Checked here as well as + // at the loader, the same two-door reason every other refusal in this file + // has: a hand-built catalog never passes the loader. Thrown rather than + // collected — it is the app's wiring, not one worker's file, so there is no + // worker to report it against and no roster that hires correctly around it. + const bothKinds = Object.keys(options.references ?? {}).filter((ref) => + Object.hasOwn(options.documents ?? {}, ref) + ); + if (bothKinds.length > 0) { + throw new Error( + `hireWorkforce was given ${bothKinds.map((ref) => `"${ref}"`).join(", ")} as both a ` + + `document and a reference. One ref is one accessor, and the two behave differently — a ` + + `reference is read from its file and sealed, a document seeds a row and then evolves — ` + + `so whichever won would be a coin flip. Keep one: leave the file in \`resources/\`, or ` + + `move it to \`references/\`.` + ); + } + const kindNames = Object.keys(kinds); const available = kindNames.length > 0 ? kindNames.map((k) => `"${k}"`).join(", ") : "(none)"; @@ -595,7 +642,14 @@ export function hireWorkforce( // map changes. Absent is not empty here, at either end. let seatResources: DeclaredResources | undefined; let seatGrants: readonly SeatResourceGrant[] = []; - if (Object.hasOwn(manifest.declared, SEAT_RESOURCES_KEY)) { + // Whether the seat NARROWED its documents, which is no longer the same + // question as whether it is minted with a map: the reference wall below + // gives a map to a seat that granted nothing. `verifySeatNarrowing` asks + // about the grant, so it has to read this and not the map — keyed on the + // map, a seat that declared no `resources:` would have every document on + // its kind reported as having escaped a narrowing nobody asked for. + const narrowedDocuments = Object.hasOwn(manifest.declared, SEAT_RESOURCES_KEY); + if (narrowedDocuments) { const parsed = parseSeatResources(manifest.declared[SEAT_RESOURCES_KEY]); if (parsed.problems.length > 0) { for (const problem of parsed.problems) refuse(problem); @@ -621,6 +675,31 @@ export function hireWorkforce( seatGrants = parsed.grants ?? []; } + // The reference wall: derived from where this seat and each reference sit + // in the tree, and applied to whatever map the seat would otherwise get. + // + // **Unconditional, unlike the block above.** A `resources:` grant is a + // narrowing a seat asks for, so it runs only when the seat asks; the wall + // is a boundary, so it runs for every seat whether or not its file says + // anything. What keeps that from touching an app with no references is the + // wall itself: a kind holding none hands `base` straight back, including + // handing back `undefined` so the seat is minted with no map at all. + const wall = applyReferenceWall({ + seatId: manifest.id, + declared: manifest.declared[SEAT_REFERENCES_KEY], + hasDeclared: Object.hasOwn(manifest.declared, SEAT_REFERENCES_KEY), + catalog: options.references, + kindResources: (factory.resources ?? {}) as DeclaredResources, + kindFlowLevelKeys: factory.flowLevelResourceKeys ?? new Set(), + base: seatResources, + kind + }); + if (wall.problems.length > 0) { + for (const problem of wall.problems) refuse(problem); + continue; + } + seatResources = wall.resources; + try { // Always a bag, so always admitted. The branch that stood here passed no // bag at all for a record that declared nothing — which is admission @@ -640,7 +719,7 @@ export function hireWorkforce( // distinguishes that document from one no block mentions. Only the // instance knows. Skipped when the seat narrowed nothing, which denies // nothing. - if (seatResources !== undefined && options.documents !== undefined) { + if (narrowedDocuments && options.documents !== undefined) { const escaped = verifySeatNarrowing({ minted: seat.resources as DeclaredResources | undefined, grants: seatGrants, @@ -653,6 +732,22 @@ export function hireWorkforce( } } + // The same check for the wall, and it exists for the same substrate + // reason: a reference a block ALSO declares comes back after the seat's + // map removed it, and only the built instance shows that. Read off the + // minted flow rather than predicted from the kind. + const crossed = verifySeatReferenceWall({ + minted: seat.resources as DeclaredResources | undefined, + allowed: wall.reachable, + catalog: options.references, + seatId: manifest.id, + kind + }); + if (crossed.length > 0) { + for (const problem of crossed) refuse(problem); + continue; + } + seats.push(seat); } catch (error) { // The flow's own refusal, with the worker's id in front of it, and — when diff --git a/packages/workforce/src/index.ts b/packages/workforce/src/index.ts index 6417699d18..bf519e0c3c 100644 --- a/packages/workforce/src/index.ts +++ b/packages/workforce/src/index.ts @@ -46,6 +46,21 @@ export { definePersona, type PersonaResourceConfig, type PersonaCollectionConfig export { createWorkforceCapability, type WorkforceCapabilityOptions } from "./workforce-capability"; export { hireWorkforce, type HireOptions } from "./hire"; export { resourcesFromDocs } from "./resources-from-docs"; +export { referencesFromDocs, referenceBody } from "./references-from-docs"; +export { + clearShadowedReferences, + describeShadowedReferences, + type ClearShadowedReferencesResult, + type ReferenceContentStore, + type ShadowedReference, +} from "./clear-shadowed-references"; +export { + SEAT_REFERENCES_KEY, + placeOfReference, + placeOfSeat, + referenceReachableBySeat, + type TreePlace, +} from "./seat-references"; export { splitResourceModules, type ResourceModuleHalves } from "./split-resource-modules"; export type { ResourceModuleExport, diff --git a/packages/workforce/src/loader/index.ts b/packages/workforce/src/loader/index.ts index f596a8e9cd..bfacd31841 100644 --- a/packages/workforce/src/loader/index.ts +++ b/packages/workforce/src/loader/index.ts @@ -75,6 +75,7 @@ export { } from "./read-teams-directory"; export { + readReferencesDirectory, readResourcesDirectory, type ReadResourcesDirectoryResult, type ResourceDocError, @@ -107,8 +108,11 @@ export { validateSegment, type SegmentLabel } from "./segments"; export { DOCUMENT_EXTENSION, + DOCUMENT_SLOTS, + REFERENCES_SLOT, RESOURCES_SLOT, WORKERS_LEVEL, + type DocumentSlot, mintResourceRef, } from "./resource-convention"; diff --git a/packages/workforce/src/loader/read-declared-roster.ts b/packages/workforce/src/loader/read-declared-roster.ts index 6e75995d22..5479b5ad8e 100644 --- a/packages/workforce/src/loader/read-declared-roster.ts +++ b/packages/workforce/src/loader/read-declared-roster.ts @@ -24,9 +24,11 @@ * must not pull a consumer onto `node:fs`. */ +import path from "node:path"; import { readChannelsDirectory } from "./read-channels-directory"; -import { readResourcesDirectory } from "./read-resources-directory"; +import { readReferencesDirectory, readResourcesDirectory } from "./read-resources-directory"; import { readWorkforce } from "./read-workforce"; +import { REFERENCES_SLOT, RESOURCES_SLOT } from "./resource-convention"; import type { ChannelManifest, ResourceDoc, @@ -51,6 +53,8 @@ export type DeclaredProblemLayer = | "team" /** A document under `org/resources/` or a team's or worker's `resources/`. */ | "document" + /** A reference under `org/references/` or a team's or worker's `references/`. */ + | "reference" /** A channel folder under `teams//channels/`. */ | "channel"; @@ -98,6 +102,15 @@ export interface DeclaredRoster { * gives the same thing two names. */ documents: ResourceDoc[]; + /** + * One record per reference that loaded, in walk order, each carrying the + * `filePath` its content is served from. + * + * Separate from {@link DeclaredRoster.documents} rather than merged with a + * flag: the two go to different install halves, and one list a caller has to + * partition is one a caller can partition wrongly. + */ + references: ResourceDoc[]; /** * One record per channel that loaded, in walk order. * @@ -142,6 +155,7 @@ export async function readDeclaredRoster(root: string): Promise // once, at boot, off the request path. const workforce = await readWorkforce(root); const resources = await readResourcesDirectory(root); + const references = await readReferencesDirectory(root); const channels = await readChannelsDirectory(root); const problems: DeclaredProblem[] = [ @@ -156,14 +170,73 @@ export async function readDeclaredRoster(root: string): Promise ), ...workforce.teamErrors.map((e) => ({ layer: "team" as const, path: e.path, error: e.error })), ...resources.errors.map((e) => ({ layer: "document" as const, path: e.path, error: e.error })), + ...references.errors.map((e) => ({ layer: "reference" as const, path: e.path, error: e.error })), ...channels.errors.map((e) => ({ layer: "channel" as const, path: e.path, error: e.error })), + ...collidingRefs(resources.documents, references.documents), ]; return { workers: workforce.workers, teams: workforce.teams, documents: resources.documents, + references: references.documents, channels: channels.channels, problems, }; } + +/** + * One problem per ref claimed by a `resources/` file and a `references/` file + * at the same level. + * + * Both slots mint into one namespace, so `org/resources/handbook.md` and + * `org/references/handbook.md` are two files claiming the accessor `handbook`. + * They cannot both have it, and they must not silently resolve to one: a + * reference is read from its file and sealed, a document seeds a row and then + * evolves, so whichever won would decide a behaviour nobody chose. + * + * Reported here rather than in either reader because neither reader can see + * the other's slot — this is the one place both lists exist. It is collected + * like every other problem; `hireWorkforce` throws on the same collision at its + * own door, for a catalog that never passed a loader. + * + * Named by PATH, and both paths, because that is what an author has to go and + * change. Only the reference carries one — a `resources/` record deliberately + * does not, so that adding references changed nothing about what the mutable + * reader hands back. The document's path is DERIVED rather than looked up, and + * it is exact: a collision means one basename in two slots at one level, so the + * two files are siblings and differ in that one path segment. A hand-built + * record with no path at all falls back to the ref. + */ +function collidingRefs( + documents: readonly ResourceDoc[], + references: readonly ResourceDoc[], +): DeclaredProblem[] { + if (documents.length === 0 || references.length === 0) return []; + const claimed = new Set(documents.map((doc) => doc.ref)); + const problems: DeclaredProblem[] = []; + + for (const reference of references) { + if (!claimed.has(reference.ref)) continue; + const at = reference.filePath ?? reference.ref; + const sibling = + reference.filePath === undefined + ? reference.ref + : reference.filePath.replace( + `${path.sep}${REFERENCES_SLOT}${path.sep}`, + `${path.sep}${RESOURCES_SLOT}${path.sep}`, + ); + problems.push({ + layer: "reference", + path: at, + error: new Error( + `"${at}" and "${sibling}" both claim the ref "${reference.ref}". A references/ file ` + + `and a resources/ file at one level are two spellings of one document, and they do ` + + `not behave the same — a reference is read from its file and nothing can write it, ` + + `while a resource seeds a row that then becomes the source. Keep one.`, + ), + }); + } + + return problems; +} diff --git a/packages/workforce/src/loader/read-resources-directory.ts b/packages/workforce/src/loader/read-resources-directory.ts index d2f36101f6..432136cb56 100644 --- a/packages/workforce/src/loader/read-resources-directory.ts +++ b/packages/workforce/src/loader/read-resources-directory.ts @@ -15,6 +15,20 @@ * **Symlinks are never followed**, at any level. And a path is judged by the * slot it occupies, not by what it looks like. * + * **One walk, two slots.** The same convention covers `resources/` and + * `references/` at the same four levels, so this module reads a slot rather + * than a folder name: {@link readResourcesDirectory} and + * {@link readReferencesDirectory} are the same walk with a different slot and a + * different refusal set. A second ~400-line reader would be a second answer to + * "where does the convention look", and the two would drift at the first level + * added. + * + * What differs between them is not the walk. A `resources/` document's body + * becomes a stored row that then evolves; a `references/` document's file stays + * the source, so its record carries the {@link ResourceDoc.filePath} the + * install half points at and refuses a wider set of declarations. Both are the + * caller's business, not this walk's. + * * **This reader is one of two over the same folder.** A `resources/` folder * takes Markdown documents and TypeScript modules side by side; the `.ts` files * are the module walk's (`../codegen/discover-resource-modules`), which mints @@ -38,11 +52,17 @@ import { parseFrontmatterYaml, splitFrontmatter, } from "@flow-state-dev/orchestration"; -import { refusedDeclarationMessage, type ResourceDoc } from "../manifest"; +import { + refusedDeclarationMessage, + refusedReferenceDeclarationMessage, + type ResourceDoc, +} from "../manifest"; import { DOCUMENT_EXTENSION, + REFERENCES_SLOT, RESOURCES_SLOT, WORKERS_LEVEL, + type DocumentSlot, mintResourceRef, } from "./resource-convention"; import { @@ -131,6 +151,53 @@ export interface ReadResourcesDirectoryResult { */ export async function readResourcesDirectory( root: string, +): Promise { + return await readDocumentSlot(root, RESOURCES_SLOT, refusedDeclarationMessage, false); +} + +/** + * Read every `.md` in a `references/` folder anywhere the convention puts + * one, and return one neutral record per document — each carrying the + * {@link ResourceDoc.filePath} it was read from. + * + * The same walk as {@link readResourcesDirectory}, at the same four levels, + * minting refs into the same namespace. Two things differ, and both belong to + * what a reference IS rather than to where it sits: + * + * - **The record carries its file path**, because the install half points the + * resource at the file instead of copying the body into a row. The body is + * still parsed here, because `description` is required of every file in this + * dialect and a file that cannot be parsed is a load failure either way. + * - **A wider declaration refusal.** `writable:`, `llmWritable:` and `render:` + * are derived for a reference and refused by name + * ({@link refusedReferenceDeclarationMessage}), because the folder carries + * the seal and a file that could unseal itself would make the folder a + * suggestion. + * + * Errors, symlink discipline and the collect-don't-throw bargain are the walk's + * and are identical. Throws only when `root` itself is refused. + * + * @param root The workforce tree — the folder holding `org/` and `teams/`. + * @returns One record per reference that loaded, and one entry per path that + * should have produced one and did not. + */ +export async function readReferencesDirectory( + root: string, +): Promise { + return await readDocumentSlot(root, REFERENCES_SLOT, refusedReferenceDeclarationMessage, true); +} + +/** + * The walk both readers are. Parameterized by the slot it looks in and the + * declaration refusal that slot imposes; everything else — the four levels, the + * symlink rule, the directory-in-a-documents-slot report — is the convention's + * and is shared by construction rather than by two copies kept in step. + */ +async function readDocumentSlot( + root: string, + slot: DocumentSlot, + refuseDeclaration: (declared: Record) => string | undefined, + carriesFilePath: boolean, ): Promise { const documents: ResourceDoc[] = []; const errors: ResourceDocError[] = []; @@ -152,9 +219,12 @@ export async function readResourcesDirectory( report("org", org.refusal.error); } if (org.entries !== undefined) { - await readSlot(path.join(root, "org", RESOURCES_SLOT), `org/${RESOURCES_SLOT}`, { + await readSlot(path.join(root, "org", slot), `org/${slot}`, { documents, errors, + slot, + refuseDeclaration, + carriesFilePath, mintRef: (name) => mintResourceRef(undefined, undefined, name), }); // Org workers are rare shared-infra seats, and their documents load for the @@ -163,16 +233,31 @@ export async function readResourcesDirectory( // reader passes over `org/workers/` in silence and a worker id requires a // team — which is a larger gap than this reader closes, and not a reason // for the documents to go on being unread. - await walkWorkers(path.join(root, "org"), "org", undefined, { documents, errors }); + await walkWorkers(path.join(root, "org"), "org", undefined, { + documents, + errors, + slot, + refuseDeclaration, + carriesFilePath, + }); } for await (const team of walkTeams(root, report)) { - await readSlot(path.join(team.dir, RESOURCES_SLOT), `${team.path}/${RESOURCES_SLOT}`, { + await readSlot(path.join(team.dir, slot), `${team.path}/${slot}`, { documents, errors, + slot, + refuseDeclaration, + carriesFilePath, mintRef: (name) => mintResourceRef(team.id, undefined, name), }); - await walkWorkers(team.dir, team.path, team.id, { documents, errors }); + await walkWorkers(team.dir, team.path, team.id, { + documents, + errors, + slot, + refuseDeclaration, + carriesFilePath, + }); } return { documents, errors }; @@ -200,7 +285,7 @@ async function walkWorkers( parentDir: string, parentPath: string, teamId: string | undefined, - ctx: { documents: ResourceDoc[]; errors: ResourceDocError[] }, + ctx: Omit, ): Promise { const workersPath = `${parentPath}/${WORKERS_LEVEL}`; const slots = await openStructuralDirectory( @@ -264,24 +349,50 @@ async function walkWorkers( continue; } - await readSlot(path.join(workerDir, RESOURCES_SLOT), `${entryPath}/${RESOURCES_SLOT}`, { + await readSlot(path.join(workerDir, ctx.slot), `${entryPath}/${ctx.slot}`, { documents: ctx.documents, errors: ctx.errors, + slot: ctx.slot, + refuseDeclaration: ctx.refuseDeclaration, + carriesFilePath: ctx.carriesFilePath, mintRef: (name) => mintResourceRef(teamId, workerName, name), }); } } -/** Everything reading one `resources/` slot needs that differs between roots. */ +/** Everything reading one documents slot needs that differs between roots. */ interface SlotContext { documents: ResourceDoc[]; errors: ResourceDocError[]; + /** Which slot is being read — the folder name, and what a refusal names. */ + slot: DocumentSlot; + /** + * Why this slot refuses a declaration, or `undefined` when it does not. + * + * Passed rather than branched on {@link SlotContext.slot}, so the two slots' + * refusal sets stay one export each in `../manifest` and this walk stays a + * walk. + */ + refuseDeclaration: (declared: Record) => string | undefined; + /** + * Whether the record carries the absolute path it was read from. + * + * **True for `references/` only, and that is not symmetry lost — it is the + * mutable path left alone.** A reference IS its file, so the path is what the + * install half installs. A `resources/` document's source is its stored row, + * so a path on that record would be a field nothing reads, and it would + * change what the reader hands back for every tree that has one. Two records + * read from different roots would stop comparing equal, which is exactly how + * the existing suite caught this being added everywhere. + */ + carriesFilePath: boolean; /** Mint this slot's ref for a document name. Throws on a bad segment. */ mintRef: (name: string) => string; } /** - * Read one `resources/` slot. An absent slot is silent — a team may have no + * Read one documents slot — `resources/` or `references/`, whichever + * {@link SlotContext.slot} names. An absent slot is silent — a team may have no * documents — and a slot that is there and cannot be walked is reported under * its own path, because the documents beneath it cannot be named individually. */ @@ -307,7 +418,7 @@ async function readSlot(slotDir: string, slotPath: string, ctx: SlotContext): Pr path: entryPath, error: new Error( `"${entryName}" is a directory. A resource is a file, not a folder — write the ` + - `document as "${entryName}${DOCUMENT_EXTENSION}" in this ${RESOURCES_SLOT}/ ` + + `document as "${entryName}${DOCUMENT_EXTENSION}" in this ${ctx.slot}/ ` + `folder instead.`, ), kind: "folder-where-file-belongs", @@ -342,15 +453,18 @@ async function readSlot(slotDir: string, slotPath: string, ctx: SlotContext): Pr // neither, and is the one thing this branch still drops in silence. if (!entryName.endsWith(DOCUMENT_EXTENSION)) continue; - let loaded: { ref: string; declared: Record; body: string }; + let loaded: ResourceDoc; try { const name = entryName.slice(0, -DOCUMENT_EXTENSION.length); // Identity first: a document whose segments break the rules has no ref to // be keyed under, so there is nothing to be gained by reading its file. const ref = ctx.mintRef(name); - const text = await fs.readFile(path.join(slotDir, entryName), "utf8"); + const filePath = path.resolve(slotDir, entryName); + const text = await fs.readFile(filePath, "utf8"); const { declared, body } = parseResourceMd(text, entryName); - loaded = { ref, declared, body }; + // The path rides only on a record whose slot needs it. See + // `SlotContext.carriesFilePath` for why that is not an asymmetry to tidy. + loaded = ctx.carriesFilePath ? { ref, declared, body, filePath } : { ref, declared, body }; } catch (err) { ctx.errors.push({ path: entryPath, error: err as Error, kind: "document-load-failed" }); continue; @@ -360,7 +474,7 @@ async function readSlot(slotDir: string, slotPath: string, ctx: SlotContext): Pr // that could not be read is an author's typo, and a file declaring what the // convention derives is an author's misunderstanding. Checked here rather // than inside the parse so the two stay tellable apart by control flow. - const refused = refusedDeclarationMessage(loaded.declared); + const refused = ctx.refuseDeclaration(loaded.declared); if (refused !== undefined) { ctx.errors.push({ path: entryPath, diff --git a/packages/workforce/src/loader/resource-convention.ts b/packages/workforce/src/loader/resource-convention.ts index edd2a109dc..06acc3eece 100644 --- a/packages/workforce/src/loader/resource-convention.ts +++ b/packages/workforce/src/loader/resource-convention.ts @@ -1,18 +1,26 @@ /** * Where a resource lives, and what names it. * - * The `resources/` convention is read by two doors: the Markdown reader in - * `read-resources-directory.ts`, and the module walk in `../codegen` that finds - * the TypeScript beside it. They are separate readers on purpose — one reader - * holding two conventions is the mega-loader the shared walk primitives exist - * to prevent — but they are not allowed to disagree about **which folders the - * convention covers** or **what a file in one is called**. + * The convention covers two slots — `resources/` and `references/` — read by + * two doors: the Markdown reader in `read-resources-directory.ts`, and the + * module walk in `../codegen` that finds the TypeScript beside it. They are + * separate readers on purpose — one reader holding two conventions is the + * mega-loader the shared walk primitives exist to prevent — but they are not + * allowed to disagree about **which folders the convention covers** or **what a + * file in one is called**. * * So the folder names, the document extension and the ref rule live here, once. * Two spellings of one ref is how a module and a document silently overwrite * each other, and the check that refuses a basename claimed by both can only * refuse what both sides minted the same way. * + * **Both slots mint into ONE namespace.** `references/handbook.md` and + * `resources/handbook.md` at the same level are two spellings of the ref + * `handbook`, which is why the collision is refused rather than resolved: the + * two behave differently — one is read from the file and sealed, the other + * seeds a row and then evolves — so whichever won would be a coin flip an + * author never sees. + * * Pure: no `node:fs`. It ships behind `./loader` because that is where its * callers are, not because it touches a disk. */ @@ -22,6 +30,30 @@ import { validateSegment } from "./segments"; /** The slot a resource sits in, under any of the roots the convention reads. */ export const RESOURCES_SLOT = "resources"; +/** + * The slot a REFERENCE sits in, at the same levels {@link RESOURCES_SLOT} sits + * at and minted by the same {@link mintResourceRef}. + * + * A reference is a read path, not a second kind of resource: its body is read + * from the file rather than from a stored row, and no write path reaches it. + * The folder is what carries that — a file does not opt in, and cannot opt out. + * What "no write path" covers exactly, and the one thing it does not, is in + * `../references-from-docs`. + */ +export const REFERENCES_SLOT = "references"; + +/** + * Both slots, in the order a collision refusal lists them. + * + * Read from here by every walk, never spelled as a literal, for the reason the + * ref rule lives here: a door that knows one slot and not the other is a folder + * an author writes into and nothing reads. + */ +export const DOCUMENT_SLOTS = [RESOURCES_SLOT, REFERENCES_SLOT] as const; + +/** One of {@link DOCUMENT_SLOTS}. */ +export type DocumentSlot = (typeof DOCUMENT_SLOTS)[number]; + /** The level a worker's folder sits in, under `org/` and under every team. */ export const WORKERS_LEVEL = "workers"; diff --git a/packages/workforce/src/manifest.ts b/packages/workforce/src/manifest.ts index 05c99e9f14..72c6e39c5c 100644 --- a/packages/workforce/src/manifest.ts +++ b/packages/workforce/src/manifest.ts @@ -498,6 +498,22 @@ export interface ResourceDoc { declared: Record; /** The document itself: Markdown body verbatim, frontmatter removed. */ body: string; + /** + * The absolute path the record was read from. + * + * **Absent on a hand-built record**, which is the difference between "read + * from this file" and "there was no file" — the same shape + * {@link WorkerManifest.skills} uses, and the reason this is optional rather + * than `""`. + * + * A `references/` document needs it: its content is served FROM the file on + * every execution context rather than copied into a stored row, so the path + * is the thing the install half installs. A `resources/` document carries it + * too — the walk knows where it read, and one walk that records provenance + * beats two walks that disagree about whether to — but nothing consumes it + * there, and {@link ResourceDoc.body} stays that document's source. + */ + filePath?: string; } /** @@ -572,3 +588,72 @@ export function refusedDeclarationMessage( return undefined; } + +/** + * The three settings a REFERENCE derives on top of + * {@link DERIVED_RESOURCE_KEYS}, and therefore refuses to let a file declare. + * + * All three are the seal, and the seal is the folder's rather than the file's. + * That is the whole difference between this convention and "rename the folder + * and set `writable: false` in each file": a key an author writes is a key an + * author forgets, and one forgotten key puts the document back on the + * seed-then-evolve path where a write shadows the file permanently. + * + * - `writable` — gates code. Refused at **either value**: agreeing with the + * install half is still a second place the same fact lives, and the next + * author to read the file cannot tell a load-bearing key from a decorative + * one. + * - `llmWritable` — gates the model's own write tool. Two doors on one + * document; closing one would leave a mode that holds against the + * implementer and not against the model. + * - `render` — the per-read transform that strips the file's frontmatter, so a + * reader gets the document and not the YAML above it. A file that supplied + * its own would serve its own frontmatter, or throw at read time when a YAML + * scalar is called as a function. + * + * **Deliberately NOT added to {@link DERIVED_RESOURCE_KEYS}.** A `resources/` + * document may legitimately declare `writable:` — that path is unchanged, and + * widening the shared list would break it. + */ +export const DERIVED_REFERENCE_KEYS = [ + ...DERIVED_RESOURCE_KEYS, + "writable", + "llmWritable", + "render", +] as const; + +/** + * Say why a REFERENCE's declaration is refused, or `undefined` when nothing is. + * + * {@link refusedDeclarationMessage}'s rule over the wider + * {@link DERIVED_REFERENCE_KEYS} set, with the same two-door reason for + * existing and the same "names no subject" contract: the loader supplies the + * file, the install half supplies the ref. + * + * @param declared The file's frontmatter, uninterpreted. + * @returns The reason, ready to be prefixed with what the caller can name. + */ +export function refusedReferenceDeclarationMessage( + declared: Record, +): string | undefined { + for (const key of DERIVED_REFERENCE_KEYS) { + if (!Object.hasOwn(declared, key)) continue; + return ( + `declares \`${key}:\`, which is a setting the \`references/\` convention derives from ` + + `where the file sits, not one a reference declares. A reference is read from its file ` + + `and nothing can write it — the folder carries that, so no file has to ask for it and ` + + `no file can turn it off. Drop the key, or move this document to \`resources/\`, where ` + + `it seeds a row and can be written.` + ); + } + + if (declared["prefetchMode"] === REFUSED_PREFETCH_MODE) { + return ( + `declares \`prefetchMode: "${REFUSED_PREFETCH_MODE}"\`, which a file-declared ` + + `resource cannot be: it is installed at flow level, and a flow-level declaration ` + + `has no per-block load trigger to load it on. Drop the key.` + ); + } + + return undefined; +} diff --git a/packages/workforce/src/references-from-docs.ts b/packages/workforce/src/references-from-docs.ts new file mode 100644 index 0000000000..3a9b96cc69 --- /dev/null +++ b/packages/workforce/src/references-from-docs.ts @@ -0,0 +1,191 @@ +/** + * The install half of the `references/` convention — reference records become + * the L1 resource map an app passes to a flow. + * + * Pure, and isomorphic, for the reason `./resources-from-docs` is: no + * `node:fs`, no store, no registry. It composes `defineResource`, and the map + * is returned rather than installed. + * + * **This module is the whole read path.** A reference differs from a document + * in exactly three installed fields, and the three are one decision: + * + * - **`contentFile`** — the file, not a copy of its body. Re-read whenever an + * execution context is built, so an edit in git reaches the next request. + * - **`writable: false` and `llmWritable: false`** — the seal, on both doors. + * - **`render`** — strips the frontmatter the file carries, so a reader gets + * the document rather than the YAML above it. + * + * **The seal is load-bearing, not decoration.** A stored content row WINS over + * `contentFile` exactly as it wins over `content`, so an unsealed `contentFile` + * is a first-boot seed with a longer name: one write and the file never comes + * back. Sealed, no row can ever be written, so the file is re-read every time. + * The two halves ship together or the convention does not hold. + * + * **What the seal IS, stated exactly, because a weaker guarantee under a + * stronger word is the failure this convention exists to prevent.** Three + * things hold, and a fourth does not: + * + * 1. Code that calls `writeContent()` is refused at the engine, by name + * (`resource_read_only`). + * 2. The model is never offered the write tool for it — that tool is gated on + * `llmWritable`, so there is no verb to call rather than a call that fails. + * 3. **Neither key is an author's.** Both are DERIVED here, and a file + * declaring either is refused at load ({@link DERIVED_REFERENCE_KEYS}). The + * "one forgotten key restores the write" failure needs a key someone can + * forget, and after this there is none in the file. + * 4. **The handle still CARRIES `writeContent`.** It is a runtime refusal, not + * an absent verb. `DeclaredResourceEntry` admits only `DefinedResource`, and + * `ResourceRef` declares `writeContent` as a required member, so a reference + * installed through this door cannot have the method removed from it without + * a core change — a read-only single-resource ref alongside today's + * `ExternalResourceRef` (which already omits it) plus the registry branch + * that builds one. That is additive, out of this module's reach, and NOT + * what ships here. + * + * So: no reachable write path, and no key to forget. Not an absent verb. + * + * **What the seal cannot do is evict a row that is ALREADY there** — a document + * written while it lived under `resources/` keeps shadowing its file after the + * move. That is a data migration rather than a file move, and it is + * `./clear-shadowed-references`. + * + * **The promise is per REQUEST, not per read.** `readContent()` reads the + * in-memory content map built once when the execution context is constructed, + * so an edit made mid-request is not seen by a read already in flight. That is + * the shape the substrate offers, and the one this convention promises. + */ + +import { defineResource, type DeclaredResources } from "@flow-state-dev/core"; +import { splitFrontmatter } from "@flow-state-dev/orchestration"; +import { z } from "zod"; +import { emptyMap } from "./empty-map"; +import { + DERIVED_REFERENCE_KEYS, + refusedReferenceDeclarationMessage, + type ResourceDoc, +} from "./manifest"; + +/** + * A reference has no state, exactly as a document has none — it is a body the + * framework serves and the model reads. + */ +const REFERENCE_STATE_SCHEMA = z.object({}).passthrough(); + +/** The derived set as a lookup. The array stays ordered for the refusal message. */ +const DERIVED_KEYS = new Set(DERIVED_REFERENCE_KEYS); + +/** + * Serve the document, not the file. + * + * `contentFile` hands the engine the bytes on disk, frontmatter and all, while + * a `resources/` document is installed with its parsed body. Without this a + * reference would read as `---\ndescription: …\n---\n` followed by the + * handbook, which is a content change nobody asked for and one every prompt + * would carry. + * + * The SAME `splitFrontmatter` the loader parses with, deliberately: a second + * way of finding where the body starts is a way for the two to disagree about + * a file with `---` in it. + * + * Applied per read, over the string the context already holds — no disk I/O, so + * a grep across a hundred references costs what it costs today. + */ +export function referenceBody(raw: string): string { + return splitFrontmatter(raw).body; +} + +/** + * The wording for a reference record with no file to read. + * + * Fatal rather than empty: a reference IS its file, so a record without one + * would install a resource whose content is permanently `null` — a handbook + * that loads, resolves, reads blank, and reports nothing. + */ +const referenceWithoutFileMessage = (ref: string): string => + `Reference "${ref}" carries no file path. A reference is served FROM its file, so a ` + + `hand-built record needs \`filePath\` set to the document's absolute path — ` + + `\`readReferencesDirectory\` fills it in for a record read off disk.`; + +/** + * Build the resource map from reference records, keyed by each one's ref. + * + * The accessor key is the ref, so a team's handbook reaches a block as + * `ctx.resources["teams/engineering/handbook"]` — the same namespace a + * `resources/` document lands in, which is what makes a basename claimed by + * both refusable rather than silently resolved. + * + * **The installing flow has to require an org**, the same way and for the same + * reason `resourcesFromDocs`'s does: every entry here is org-scoped, and a flow + * that declares no `requireOrg` builds no org resource registry, so every + * reference resolves as unregistered. + * + * Throws rather than collecting, matching the mutable install half: a reference + * that cannot become a resource is startup misconfiguration. + * + * @param references Records from `readReferencesDirectory`, or hand-built ones + * carrying an absolute {@link ResourceDoc.filePath}. + * @returns The L1 map, keyed by ref. + * @throws If a record declares a derived setting, carries no file path, or + * `defineResource` rejects its frontmatter. + */ +export function referencesFromDocs(references: ResourceDoc[]): DeclaredResources { + const resources: DeclaredResources = emptyMap(); + + for (const doc of references) { + const refused = refusedReferenceDeclarationMessage(doc.declared); + if (refused !== undefined) { + throw new Error(`Reference "${doc.ref}" ${refused}`); + } + + const filePath = doc.filePath; + if (typeof filePath !== "string" || filePath.trim().length === 0) { + throw new Error(referenceWithoutFileMessage(doc.ref)); + } + + try { + resources[doc.ref] = defineResource({ + // The derived fields come LAST, over a passthrough that cannot carry + // one — the ordering `resourcesFromDocs` uses, and here it is guarding + // a permission rather than a storage row: spreading frontmatter after + // these would let a file hand itself `writable: true` back. + ...passthroughForReference(doc.declared), + ref: doc.ref, + scope: "org", + stateSchema: REFERENCE_STATE_SCHEMA, + default: {}, + contentFile: filePath, + // Both doors, and both DERIVED — the module header states exactly what + // that guarantees and what it does not. `writable` refuses code at the + // engine; `llmWritable` keeps the model's write tool from being offered + // at all. Neither is a key any file can set. + writable: false, + llmWritable: false, + render: referenceBody, + }); + } catch (err) { + throw new Error(`Reference "${doc.ref}" could not be built: ${(err as Error).message}`); + } + } + + return resources; +} + +/** + * The frontmatter a reference may pass through to `defineResource`: everything + * it wrote, minus every field the references convention derives. + * + * A second copy of `passthroughFrom`'s shape rather than a shared one, because + * the two strip DIFFERENT sets and a shared helper taking the set as an + * argument would read as though the sets were interchangeable. Exported for its + * own spec: this is the guard that holds when the named refusal does not. + */ +export function passthroughForReference( + declared: Record, +): Record { + const passthrough = emptyMap(); + for (const [key, value] of Object.entries(declared)) { + if (DERIVED_KEYS.has(key)) continue; + passthrough[key] = value; + } + return passthrough; +} diff --git a/packages/workforce/src/seat-references.ts b/packages/workforce/src/seat-references.ts new file mode 100644 index 0000000000..0ef8ea4ff2 --- /dev/null +++ b/packages/workforce/src/seat-references.ts @@ -0,0 +1,492 @@ +/** + * The seat reference wall — which references a seat can reach, derived from + * where it and they sit in the tree, and narrowed by a `WORKER.md`'s + * `references:` key. + * + * **One module, because this is a permission boundary**, the reason + * `./seat-resources` is one. A reference becomes reachable in exactly one + * place, so there is exactly one place the wall can be forgotten. + * + * **It is a sibling of `seat-resources`, not a copy of it.** The two answer + * different questions and the difference is the point: + * + * | | `resources:` | `references:` | + * |---|---|---| + * | absent | reaches everything on the kind | reaches everything **at or above its place in the tree** | + * | present | narrows | narrows, within the wall | + * | entry shape | a ref, or `: rw` | a ref. There is no mode — nothing writes a reference | + * | the wall | the app's install filter | derived here, from the path | + * + * So the mutable key has modes and no derived wall, and this one has a derived + * wall and no modes. They share `describe()` and the shape of their refusals; + * merging them would mean a function whose every branch asked which key it was + * called for. + * + * **Width comes from the tree and from nothing else.** There is no install-side + * override: one would make the wall optional again, which is the hole this + * exists to close. A document that should reach more people moves up the tree. + * + * **A seat file selects, it never defines** (BP-031). Both inputs the wall is + * computed from — the seat's id and each reference's ref — are minted by the + * loader from the path on disk, never taken from anything a seat wrote. A + * `references:` entry naming something outside the wall is refused, not + * granted. + */ + +import type { DeclaredResources } from "@flow-state-dev/core"; +import { describe } from "./describe-value"; +import { emptyMap } from "./empty-map"; + +/** + * The key a seat's file names its references under. + * + * Public: a person types it, and it matches the `references/` folder the + * documents come from. Read from here by every door, never spelled as a + * literal — a renamed key with a literal left behind is an access grant that + * silently does not apply. + */ +export const SEAT_REFERENCES_KEY = "references"; + +/** The `teams/` first segment, as both a ref and a seat address use it. */ +const TEAMS_LEVEL = "teams"; + +/** The `workers/` level inside `org/` or a team. Mirrors `loader`'s `WORKERS_LEVEL`. */ +const WORKERS_LEVEL = "workers"; + +// --------------------------------------------------------------------------- +// Where a thing sits +// --------------------------------------------------------------------------- + +/** + * A place in the workforce tree — the only thing reachability is decided from. + * + * `undefined` means "not at that level", so the org level is + * `{ team: undefined, worker: undefined }` and a team's own folder is + * `{ team: "engineering", worker: undefined }`. + */ +export interface TreePlace { + /** The team folder, or `undefined` at the org level. */ + team?: string; + /** The worker folder, or `undefined` when this is not inside one. */ + worker?: string; +} + +/** + * Read a reference's ref back into the place its file sits. + * + * The inverse of `mintResourceRef`, and total: anything that is not one of the + * four shapes the minter produces returns `undefined`, which the wall reads as + * *unreachable*. Denying an unrecognised shape is the safe direction — a + * hand-built catalog can hold any string, and a permission boundary that + * guesses at one it does not recognise is a boundary that can be talked past. + * + * The four shapes, and the two length rules that keep them apart: + * + * - `` — the org level + * - `teams//` — a team's + * - `workers//` — an org worker's + * - `teams//workers//` — a team worker's + * + * A `workers` first segment is only read as the workers LEVEL when exactly two + * segments follow it. Otherwise a document legitimately named `workers`, which + * mints `teams/eng/workers`, would be misread as a malformed worker address. + * + * @param ref A reference's ref, as the loader minted it. + * @returns Where its file sits, or `undefined` if the ref is not a shape the + * convention mints. + */ +export function placeOfReference(ref: string): TreePlace | undefined { + const parts = ref.split("/"); + if (parts.some((part) => part.length === 0)) return undefined; + + let team: string | undefined; + let rest = parts; + if (parts[0] === TEAMS_LEVEL) { + if (parts.length < 3) return undefined; + team = parts[1]; + rest = parts.slice(2); + } + + if (rest.length === 1) return { team, worker: undefined }; + if (rest.length === 3 && rest[0] === WORKERS_LEVEL) return { team, worker: rest[1] }; + return undefined; +} + +/** + * Read a seat's id back into the place its folder sits. + * + * A worker id is `"."` — dot-joined, because it becomes a flow + * address and a `/` there is unroutable — so a seat always sits at + * `teams//workers//`. + * + * Returns `undefined` for an id that is not team-qualified. That is not a seat + * the loader can mint, so there is no folder to place it at and no honest + * answer to what is above it; {@link seatHasNoPlaceMessage} is what the caller + * does with it. + * + * @param seatId The worker's id, as the roster minted it. + * @returns Where the seat sits, or `undefined` if the id names no place. + */ +export function placeOfSeat(seatId: string): TreePlace | undefined { + const dot = seatId.indexOf("."); + if (dot <= 0 || dot === seatId.length - 1) return undefined; + const team = seatId.slice(0, dot); + const worker = seatId.slice(dot + 1); + // A second dot is still one team and one name — `indexOf` splits at the + // first, which is the rule the id minter uses. + if (team.includes("/") || worker.includes("/")) return undefined; + return { team, worker }; +} + +/** + * Whether a seat can reach a reference: is the reference's folder at or above + * the seat's place? + * + * Three ways yes, and nothing else: + * + * - the org level is above everyone; + * - the seat's own team's folder; + * - the seat's own worker folder. + * + * Everything else is no, and the two that matter are the ones a person expects + * to be yes and should not be: **another team's folder**, and **a sibling + * seat's folder on the same team**. The walk inherits downward only — a + * sibling's folder is not above anyone. + * + * An org WORKER's folder (`workers//`) is also no, for the same + * reason: it is an address beside the org level, not above a team's seat. + */ +export function referenceReachableBySeat(seat: TreePlace, reference: TreePlace): boolean { + // The org level: above every seat. + if (reference.team === undefined) return reference.worker === undefined; + // Another team's tree is never above this seat. + if (reference.team !== seat.team) return false; + // The seat's own team folder, or its own worker folder. A sibling's is not. + if (reference.worker === undefined) return true; + return reference.worker === seat.worker; +} + +// --------------------------------------------------------------------------- +// The refusals +// --------------------------------------------------------------------------- + +/** The one sentence describing the entry shape, shared by every shape refusal. */ +const SHAPE_SENTENCE = + `An entry is a reference's ref on its own. There is no mode — nothing writes a reference.`; + +/** The wording for a `references:` that is not a list at all. */ +const malformedSeatReferencesMessage = (value: unknown): string => + `declares \`${SEAT_REFERENCES_KEY}:\` as ${describe(value)}, which names no references. ` + + `${SEAT_REFERENCES_KEY}: is a LIST — one entry per reference, each line starting with \`- \`. ` + + SHAPE_SENTENCE; + +/** The wording for one entry that is not a ref. */ +const malformedSeatReferenceEntryMessage = (value: unknown): string => + `declares \`${SEAT_REFERENCES_KEY}:\` entry ${describe(value)}, which names no reference. ` + + SHAPE_SENTENCE; + +/** The wording for a ref named more than once. */ +const duplicateSeatReferenceMessage = (ref: string, times: number): string => + `names "${ref}" ${times} times. Keep one.`; + +/** + * The wording for a ref that matches no reference this seat can reach. + * + * One message for "does not exist" and for "exists, but not above this seat", + * deliberately. Telling them apart would let a seat file probe the tree for + * which teams have a handbook — and the fix is the same either way: put the + * file where this seat can see it, or drop the entry. + */ +const unreachableSeatReferenceMessage = (ref: string, seatId: string): string => + `names "${ref}", which is not a reference "${seatId}" can reach. A \`${SEAT_REFERENCES_KEY}:\` ` + + `list NARROWS what a seat's place in the tree already gives it — the org's references, its ` + + `own team's, and its own folder's — and cannot add one from outside. Move the file up the ` + + `tree to widen who reads it, or drop the entry.`; + +/** + * The wording for a seat whose id names no place in the tree. + * + * Fatal once the kind holds references, and silent before that: reachability is + * derived from where a seat sits, so a seat that sits nowhere has no derivable + * answer. Guessing one would mean picking between "reaches the org's + * references" and "reaches none", and both are wrong in a way nothing reports. + */ +const seatHasNoPlaceMessage = (seatId: string): string => + `has the id "${seatId}", which names no place in the tree, and this kind holds references. ` + + `A worker id is "." — the loader mints it from the seat's folder, and a ` + + `reference is reachable because of where it sits relative to that folder. Give the seat a ` + + `team-qualified id.`; + +/** + * The wording for a reference the wall denied that the minted flow reaches + * anyway. + * + * The twin of `seat-resources`'s survived-narrowing refusal, and it exists for + * the same substrate reason: a flow instance's `resources` option replaces the + * definition's flow-level map, and `defineFlow` then merges the blocks' own + * declarations back on top. So a reference a block also declares comes back + * after the wall removed it. Fatal — the alternative is a wall that reads as + * enforced and is not. + */ +const referenceCrossedWallMessage = (ref: string, seatId: string, kind: string): string => + `cannot reach "${ref}" from its place in the tree, but the flow it is minted with reaches ` + + `it anyway: a block on the "${kind}" kind declares it, and a block declaration merges back ` + + `in after a seat's map replaces the flow-level one. Remove that declaration from the block ` + + `and keep the reference at flow level, or move the file where "${seatId}" can see it.`; + +// --------------------------------------------------------------------------- +// Parsing +// --------------------------------------------------------------------------- + +/** What {@link parseSeatReferences} found: the refs named, or why it could not read them. */ +export interface ParsedSeatReferences { + /** One ref per entry, in file order. Absent when `problems` is non-empty. */ + refs?: string[]; + /** Every problem with the declaration. Collected, never thrown — the hire step owns the refusal. */ + problems: string[]; +} + +/** + * Read a seat's `references:` declaration into refs. + * + * **Call this only for a declaration that is PRESENT.** Absent and + * present-and-empty are three-way distinct all the way down — *nobody narrowed + * this seat*, versus *this seat is narrowed to nothing* — and collapsing them + * turns the one way to say "no ambient reading" into its opposite. Whether the + * key is there is the caller's read; `references: []` parses here to zero refs + * and no problems. + * + * @param declared The value of the seat's {@link SEAT_REFERENCES_KEY} key, uninterpreted. + * @returns The refs in file order, or every problem with the declaration. + */ +export function parseSeatReferences(declared: unknown): ParsedSeatReferences { + if (!Array.isArray(declared)) { + return { problems: [malformedSeatReferencesMessage(declared)] }; + } + + const problems: string[] = []; + const refs: string[] = []; + const counts = new Map(); + + for (const entry of declared) { + if (typeof entry !== "string" || entry.trim().length === 0) { + problems.push(malformedSeatReferenceEntryMessage(entry)); + continue; + } + counts.set(entry, (counts.get(entry) ?? 0) + 1); + refs.push(entry); + } + + for (const [ref, times] of counts) { + if (times > 1) problems.push(duplicateSeatReferenceMessage(ref, times)); + } + + if (problems.length > 0) return { problems }; + return { refs, problems }; +} + +// --------------------------------------------------------------------------- +// Applying the wall +// --------------------------------------------------------------------------- + +/** Everything {@link applyReferenceWall} needs to build one seat's map. */ +export interface ApplyReferenceWallInput { + /** The seat's id, as the roster minted it. Its place in the tree comes from here. */ + seatId: string; + /** + * The seat's `references:` value, uninterpreted, or `undefined` when the key + * is ABSENT — which means ambient, not empty. + */ + declared: unknown; + /** Whether the key is present at all. `declared` cannot carry this: `references:` with nothing after it parses to `null`. */ + hasDeclared: boolean; + /** The app's references, keyed by ref — the map `referencesFromDocs` returns. */ + catalog: DeclaredResources | undefined; + /** The kind's MERGED resource map — its own flow-level entries plus what its blocks declare. */ + kindResources: DeclaredResources; + /** The accessor keys the kind declared at FLOW level, which is the subset a seat's map replaces. */ + kindFlowLevelKeys: ReadonlySet; + /** + * The map this seat would otherwise be minted with — `resolveSeatResources`' + * output when the seat also declares `resources:`, or `undefined` when it + * declares none and would be minted with the kind's own map. + */ + base: DeclaredResources | undefined; + /** The kind's name, for the refusals. */ + kind: string; +} + +/** What {@link applyReferenceWall} produced. */ +export interface AppliedReferenceWall { + /** + * The map to mint this seat with, or `undefined` to mint it with no map at + * all — which is what a kind holding no references gets, so an app with no + * `references/` folder is minted exactly as it is today. + */ + resources?: DeclaredResources; + /** The references this seat may reach, for the post-mint check. */ + reachable: ReadonlySet; + /** Every problem. Collected, never thrown. */ + problems: string[]; +} + +/** + * Narrow a seat's map to the references its place in the tree allows, and then + * to the ones it named. + * + * **Built by subtraction, exactly as `resolveSeatResources` is**, and for the + * same substrate reason: a flow instance's `resources` option REPLACES the + * definition's flow-level map, so a map assembled from the reachable + * references alone would delete every other resource the kind declared. The map + * starts as everything the seat would otherwise hold and loses the references + * the wall denies. + * + * **A kind holding no references is left alone entirely** — `base` comes back + * untouched, including `undefined`. That is what keeps every existing app byte + * for byte as it is: no `references/` folder, no map where there was none, no + * change to what a seat reaches. + * + * @param input The seat, its declaration, the catalog and the kind's map. + * @returns The seat's map, the reachable set, and every problem. + */ +export function applyReferenceWall(input: ApplyReferenceWallInput): AppliedReferenceWall { + const { seatId, declared, hasDeclared, catalog, kindResources, kindFlowLevelKeys, base, kind } = + input; + + // Which of the kind's accessors hold one of the app's references. By name OR + // by definition identity, the way `seat-resources` recognises a document: an + // app may expose one definition under a second key, and matching names alone + // would leave that alias outside the wall. + const references = referenceKeys(catalog, kindResources); + + // Nothing to wall. Left exactly as it was — the branch that keeps an app with + // no references/ folder unchanged, including being minted with no map. + if (references.size === 0 && !hasDeclared) { + return { resources: base, reachable: new Set(), problems: [] }; + } + + const problems: string[] = []; + + const seat = placeOfSeat(seatId); + if (seat === undefined) { + return { reachable: new Set(), problems: [seatHasNoPlaceMessage(seatId)] }; + } + + // The wall: derived, before anything the seat wrote is read. + const reachable = new Set(); + for (const ref of references) { + const place = placeOfReference(ref); + if (place !== undefined && referenceReachableBySeat(seat, place)) reachable.add(ref); + } + + // The narrowing, if the seat asked for one. Selects from `reachable`; a ref + // outside it is refused rather than granted (BP-031). + let allowed: ReadonlySet = reachable; + if (hasDeclared) { + const parsed = parseSeatReferences(declared); + if (parsed.problems.length > 0) { + return { reachable, problems: parsed.problems }; + } + const named = new Set(); + for (const ref of parsed.refs ?? []) { + if (!reachable.has(ref)) { + problems.push(unreachableSeatReferenceMessage(ref, seatId)); + continue; + } + named.add(ref); + } + allowed = named; + } + + if (problems.length > 0) return { reachable, problems }; + + // The subtraction. `base` when the seat already has a map (it declared + // `resources:`), otherwise the kind's own flow-level entries — which is what + // it would have been minted with. + const resources: DeclaredResources = emptyMap(); + if (base !== undefined) { + for (const [key, entry] of Object.entries(base)) { + if (references.has(key) && !allowed.has(key)) continue; + resources[key] = entry; + } + } else { + for (const key of kindFlowLevelKeys) { + if (!Object.hasOwn(kindResources, key)) continue; + if (references.has(key) && !allowed.has(key)) continue; + resources[key] = kindResources[key]!; + } + } + + return { resources, reachable: allowed, problems }; +} + +/** + * Every accessor on a kind's map that holds one of the app's references. + * + * By name OR by definition identity, for the reason `seat-resources`' + * `documentKeys` is: an app may expose one definition under a second key, and a + * wall that matched names alone would leave the alias reachable across a team + * boundary. The union never recognises fewer references than names alone, so + * closing this cannot silently widen a seat. + */ +function referenceKeys( + catalog: DeclaredResources | undefined, + kindResources: DeclaredResources, +): ReadonlySet { + const keys = new Set(); + if (catalog === undefined) return keys; + const definitions = new Set(Object.values(catalog)); + for (const key of Object.keys(kindResources)) { + if (Object.hasOwn(catalog, key) || definitions.has(kindResources[key])) keys.add(key); + } + for (const key of Object.keys(catalog)) { + if (Object.hasOwn(kindResources, key)) keys.add(key); + } + return keys; +} + +/** What {@link verifySeatReferenceWall} is asked to check: the seat as it was actually built. */ +export interface VerifySeatReferenceWallInput { + /** The MERGED map of the minted instance — what a block will really hold. */ + minted: DeclaredResources | undefined; + /** The references this seat is allowed, from {@link applyReferenceWall}. */ + allowed: ReadonlySet; + /** The app's references, keyed by ref. */ + catalog: DeclaredResources | undefined; + /** The seat's id, for the refusal. */ + seatId: string; + /** The kind's name, for the refusal. */ + kind: string; +} + +/** + * Check the seat that was actually minted, and refuse if a reference the wall + * denied is reachable anyway. + * + * **This exists because no inspection of the kind's map can predict the + * answer** — the same reason `verifySeatNarrowing` exists, and the reason it + * reads the built instance rather than reasoning about the build. A reference + * declared both at flow level and by one of the kind's blocks is shadowed at + * definition time, invisible on the merged map, and comes back the moment the + * wall removes the flow-level entry that was hiding it. + * + * @param input The minted map, the allowed set, and the catalog. + * @returns Every reference that crossed the wall. Empty is the pass. + */ +export function verifySeatReferenceWall(input: VerifySeatReferenceWallInput): string[] { + const { minted, allowed, catalog, seatId, kind } = input; + if (minted === undefined || catalog === undefined) return []; + + const definitions = new Set(Object.values(catalog)); + const problems: string[] = []; + + for (const key of Object.keys(minted)) { + // Identity, not name: a block resource that merely shares a reference's ref + // is not that reference, and flagging it by name would refuse an app whose + // wiring is fine. + if (!definitions.has(minted[key])) continue; + if (!allowed.has(key)) problems.push(referenceCrossedWallMessage(key, seatId, kind)); + } + + return problems; +} diff --git a/packages/workforce/src/seat-resources.ts b/packages/workforce/src/seat-resources.ts index 6de9e478a1..db9206ea4b 100644 --- a/packages/workforce/src/seat-resources.ts +++ b/packages/workforce/src/seat-resources.ts @@ -25,6 +25,7 @@ import type { DeclaredResources } from "@flow-state-dev/core"; import type { DeclaredResourceEntry } from "@flow-state-dev/core/types"; +import { describe } from "./describe-value"; import { emptyMap } from "./empty-map"; /** @@ -201,15 +202,6 @@ const catalogRefHoldsSomethingElseMessage = (ref: string, kind: string): string `there is not the document the app passed under that ref. A seat's grants narrow documents and ` + `nothing else, and these two cannot be told apart. Rename one of them.`; -/** A value as a refusal names it: short, quoted, and never a sprawling dump. */ -function describe(value: unknown): string { - if (value === undefined) return "undefined"; - if (typeof value === "string") return JSON.stringify(value); - if (value === null || typeof value !== "object") return JSON.stringify(value) ?? String(value); - if (Array.isArray(value)) return `a list of ${value.length}`; - return `a mapping of ${Object.keys(value as object).length} key(s)`; -} - /** What {@link parseSeatResources} found: the grants, or every reason it could not read them. */ export interface ParsedSeatResources { /** One entry per document named, in file order. Absent when `problems` is non-empty. */ diff --git a/packages/workforce/test/references-convention.test.ts b/packages/workforce/test/references-convention.test.ts new file mode 100644 index 0000000000..1dffc159bc --- /dev/null +++ b/packages/workforce/test/references-convention.test.ts @@ -0,0 +1,678 @@ +/** + * The `references/` convention, end to end: a tree on disk, the loader, the + * install half, `hireWorkforce`, and a live execution context. + * + * `BR-n` / `V-n` are the issue's rule and check numbers, kept so a finding can + * be cited by one; every case states the rule it owns, so nothing here needs a + * second document to read. + * + * **Three of these were RED before this convention existed**, and they are the + * work. Each is written as the promise, on the real path, so a green is the + * promise holding and not the harness agreeing with itself: + * + * - **V3 / BR-7** — an ungranted seat reads another team's handbook. The wall + * has to come from where the file sits, with no filter written by the app. + * - **V2 / BR-11** — an ungranted seat overwrites the org handbook. + * - **V2 / BR-1** — the write STICKS: the disk body was a first-boot seed, so + * the file in git stopped being what anyone read. + * + * **Two are the regressions that would ship silently**, and they matter more + * than the reds: + * + * - **V4 / BR-14** — a seat with no `resources:` key still reaches every + * MUTABLE resource. Tightening that while making the wall work is the + * cheapest way to break the rest of this wave, and it would pass every check + * above. + * - **V2b / BR-19** — a row written BEFORE the move still shadows the file. The + * seal stops new writes and cannot evict an old one, so a fresh-tree test + * passes either way and hides it. This is the case that needs a tree with + * history, which is why it builds one. + * + * Bodies are asserted with `toBe`, never `toContain`. `contentFile` hands the + * engine the raw file, frontmatter and all, so a `toContain("ORG HANDBOOK v1")` + * passes while every prompt in the product carries a block of YAML. + */ +import { afterAll, describe, expect, it } from "vitest"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { z } from "zod"; +import { defineFlow, defineResource, handler, type DeclaredResources } from "@flow-state-dev/core"; +import type { FlowInstance } from "@flow-state-dev/core/types"; +import { createExecutionContext, createInMemoryStores } from "@flow-state-dev/engine"; +import { readReferencesDirectory, readResourcesDirectory } from "../src/loader"; +import { discoverResourceModules } from "../src/codegen/discover-resource-modules"; +import { readDeclaredRoster } from "../src/loader/read-declared-roster"; +import { referencesFromDocs } from "../src/references-from-docs"; +import { resourcesFromDocs } from "../src/resources-from-docs"; +import { clearShadowedReferences } from "../src/clear-shadowed-references"; +import { hireWorkforce } from "../src/hire"; +import type { WorkerManifest } from "../src/manifest"; +import { SEAT_REFERENCES_KEY } from "../src/seat-references"; +import { SEAT_RESOURCES_KEY } from "../src/seat-resources"; +import { workerConfigSchema } from "../src/worker-config"; + +const ORG = "org_fix1467"; +const USER = "user_fix1467"; +const KIND = "desk"; + +const ORG_HANDBOOK = "handbook"; +const ENG_HANDBOOK = "teams/engineering/handbook"; +const SALES_HANDBOOK = "teams/sales/handbook"; +const ENG_SCRATCH = "teams/engineering/scratch"; +const ADA_NOTES = "teams/engineering/workers/ada/notes"; +const BOB_NOTES = "teams/engineering/workers/bob/notes"; + +/** + * A VALID engineering placement. A worker id is team-qualified, + * `"."` — a `/` there is unroutable — so this seat's folder is + * `teams/engineering/workers/ada/`, and that is the place the wall is derived + * from. A bare `"ada"` is not a seat the loader can mint, so it would give the + * own-team / sibling-team distinction nothing to be distinguished FROM. + */ +const ENG_SEAT = "engineering.ada"; + +const roots: string[] = []; + +/** A workforce tree on disk, read by the real loader. */ +async function tree(): Promise { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "fix1467-")); + roots.push(root); + await writeInto(root, { + "org/references/handbook.md": doc("The org handbook", "ORG HANDBOOK v1"), + "teams/engineering/references/handbook.md": doc("Engineering handbook", "ENGINEERING HANDBOOK v1"), + "teams/sales/references/handbook.md": doc("Sales handbook", "SALES HANDBOOK v1"), + "teams/engineering/workers/ada/references/notes.md": doc("Ada's notes", "ADA NOTES v1"), + "teams/engineering/workers/bob/references/notes.md": doc("Bob's notes", "BOB NOTES v1"), + // The mutable neighbour BR-14 is about. Unchanged by this convention. + "teams/engineering/resources/scratch.md": + "---\ndescription: Engineering scratchpad\nllmReadable: true\nllmWritable: true\n---\nseed", + }); + return root; +} + +const doc = (description: string, body: string): string => + `---\ndescription: ${description}\nllmReadable: true\n---\n${body}`; + +async function writeInto(root: string, files: Record): Promise { + for (const [rel, body] of Object.entries(files)) { + await fs.mkdir(path.join(root, path.dirname(rel)), { recursive: true }); + await fs.writeFile(path.join(root, rel), body, "utf8"); + } +} + +afterAll(async () => { + for (const root of roots) await fs.rm(root, { recursive: true, force: true }); +}); + +const work = handler({ + name: "desk-work", + requireOrg: true, + inputSchema: z.object({}), + outputSchema: z.object({ ok: z.boolean() }), + execute: async () => ({ ok: true }), +}); + +function kindWith(resources: DeclaredResources) { + return defineFlow({ + kind: KIND, + cardinality: "collection", + configSchema: workerConfigSchema(), + resources: { ...resources } as DeclaredResources, + actions: { run: { inputSchema: z.object({}), block: work } }, + }); +} + +/** A seat record as the loader produces one. */ +function seat(id: string, declared: Record = {}): WorkerManifest { + return { id, declared: { flow: KIND, description: id, ...declared }, body: "" }; +} + +type Handle = { readContent(): Promise; writeContent(body: string): Promise }; + +async function ctxFor(flow: FlowInstance, stores = createInMemoryStores()) { + const ctx = await createExecutionContext({ + flow, + actionName: "run", + requestId: `req_${flow.id}`, + sessionId: `sess_${flow.id}`, + userId: USER, + orgId: ORG, + stores, + }); + return { ctx, stores }; +} + +/** + * The handle, or `undefined` when the seat cannot reach that ref at all. + * + * `ctx.resources.get` THROWS on an unregistered accessor rather than returning + * nothing, so "unreachable" has to be caught to be asserted on. + */ +const handleFor = (ctx: { resources: { get(k: string): unknown } }, key: string): Handle | undefined => { + try { + return ctx.resources.get(key) as Handle; + } catch { + return undefined; + } +}; + +/** + * The whole path in one call: disk → loader → both install halves → kind → + * hire → a live seat. + * + * **No install filter.** That is the point of V3: the app hands the kind every + * document and every reference it loaded, and the cross-team wall has to come + * from somewhere else. + */ +async function seatOn(root: string, seatRecord: WorkerManifest = seat(ENG_SEAT)) { + const documents = resourcesFromDocs((await readResourcesDirectory(root)).documents); + const references = referencesFromDocs((await readReferencesDirectory(root)).documents); + const hired = hireWorkforce([seatRecord], { + kinds: { [KIND]: kindWith({ ...documents, ...references }) as never }, + documents, + references, + }); + return { seat: hired[0]!, documents, references }; +} + +// --------------------------------------------------------------------------- +// V1 · the reader +// --------------------------------------------------------------------------- + +describe("V1 · the references/ slot loads at every level (BR-1, BR-18)", () => { + it("walks org, team and worker references/, minting into the resources namespace", async () => { + const read = await readReferencesDirectory(await tree()); + expect(read.errors).toEqual([]); + expect(read.documents.map((d) => d.ref).sort()).toEqual( + [ADA_NOTES, BOB_NOTES, ENG_HANDBOOK, ORG_HANDBOOK, SALES_HANDBOOK].sort(), + ); + }); + + it("carries each record's file path — the thing the install half installs", async () => { + const root = await tree(); + const read = await readReferencesDirectory(root); + const org = read.documents.find((d) => d.ref === ORG_HANDBOOK)!; + expect(org.filePath).toBe(path.join(root, "org/references/handbook.md")); + }); + + it("reads only its own slot: resources/ documents are not references", async () => { + const root = await tree(); + const references = await readReferencesDirectory(root); + const documents = await readResourcesDirectory(root); + expect(references.documents.map((d) => d.ref)).not.toContain(ENG_SCRATCH); + expect(documents.documents.map((d) => d.ref)).toEqual([ENG_SCRATCH]); + }); + + it("BR-18 — a directory in the slot is reported, naming references/ and not resources/", async () => { + const root = await tree(); + await fs.mkdir(path.join(root, "org/references/handbook-folder"), { recursive: true }); + const read = await readReferencesDirectory(root); + const reported = read.errors.find((e) => e.path === "org/references/handbook-folder"); + expect(reported?.kind).toBe("folder-where-file-belongs"); + // The message has to name the slot the author actually wrote in. + expect(reported?.error.message).toContain("references/"); + expect(reported?.error.message).not.toContain("resources/"); + }); + + it("BR-18 — a non-.md file is passed over in silence", async () => { + const root = await tree(); + await fs.writeFile(path.join(root, "org/references/notes.txt"), "loose", "utf8"); + const read = await readReferencesDirectory(root); + expect(read.errors).toEqual([]); + expect(read.documents.map((d) => d.ref)).not.toContain("notes"); + }); +}); + +// --------------------------------------------------------------------------- +// V2 · the read path — D1 +// --------------------------------------------------------------------------- + +describe("V2 · a reference is read from its file and sealed (D1)", () => { + it("BR-11 RED→GREEN — an ungranted seat CANNOT write the org handbook", async () => { + const { seat: ada } = await seatOn(await tree()); + const { ctx } = await ctxFor(ada); + await expect(handleFor(ctx, ORG_HANDBOOK)!.writeContent("DEFACED")).rejects.toThrow(/read-only/i); + }); + + it("BR-1 RED→GREEN — after a write is attempted, the body is still the FILE's", async () => { + const { seat: ada } = await seatOn(await tree()); + const { ctx, stores } = await ctxFor(ada); + await expect(handleFor(ctx, ORG_HANDBOOK)!.writeContent("DEFACED")).rejects.toThrow(); + + // A fresh context over the same store — a later request, a later deploy. + const again = await ctxFor(ada, stores); + expect(await handleFor(again.ctx, ORG_HANDBOOK)!.readContent()).toBe("ORG HANDBOOK v1"); + }); + + it("BR-1 — the body is the DOCUMENT, not the file: no frontmatter leaks through", async () => { + const { seat: ada } = await seatOn(await tree()); + const { ctx } = await ctxFor(ada); + const body = await handleFor(ctx, ORG_HANDBOOK)!.readContent(); + // `contentFile` hands the engine the raw bytes. Exact equality is the only + // assertion that catches the YAML block a `toContain` would wave through. + expect(body).toBe("ORG HANDBOOK v1"); + expect(body).not.toContain("description:"); + expect(body).not.toContain("---"); + }); + + it("BR-2 — an edit in git reaches the next execution context", async () => { + const root = await tree(); + const { seat: ada } = await seatOn(root); + const { ctx, stores } = await ctxFor(ada); + expect(await handleFor(ctx, ORG_HANDBOOK)!.readContent()).toBe("ORG HANDBOOK v1"); + + await fs.writeFile( + path.join(root, "org/references/handbook.md"), + doc("The org handbook", "ORG HANDBOOK v2"), + "utf8", + ); + + const again = await ctxFor(ada, stores); + expect(await handleFor(again.ctx, ORG_HANDBOOK)!.readContent()).toBe("ORG HANDBOOK v2"); + }); + + it("BR-3 — and NOT mid-request: a read already in flight does not see the edit", async () => { + const root = await tree(); + const { seat: ada } = await seatOn(root); + const { ctx } = await ctxFor(ada); + const handle = handleFor(ctx, ORG_HANDBOOK)!; + expect(await handle.readContent()).toBe("ORG HANDBOOK v1"); + + await fs.writeFile( + path.join(root, "org/references/handbook.md"), + doc("The org handbook", "ORG HANDBOOK v2"), + "utf8", + ); + + // Pinned deliberately, and asserted rather than left unsaid: the promise is + // "the file is the source", per request. Anything that starts asserting + // immediacy here is promising hot reload the substrate does not offer. + expect(await handle.readContent()).toBe("ORG HANDBOOK v1"); + }); + + it("BR-12 — a reference declaring `writable:` is refused by name, at EITHER value", async () => { + for (const value of ["true", "false"]) { + const root = await tree(); + await fs.writeFile( + path.join(root, "org/references/handbook.md"), + `---\ndescription: The org handbook\nwritable: ${value}\n---\nORG HANDBOOK v1`, + "utf8", + ); + const read = await readReferencesDirectory(root); + const refused = read.errors.find((e) => e.path === "org/references/handbook.md"); + expect(refused?.kind).toBe("refused-declaration"); + expect(refused?.error.message).toContain("writable"); + } + }); + + it("BR-12 — and `llmWritable:` and `render:`, the other two halves of the seal", async () => { + for (const key of ["llmWritable: true", "render: nope"]) { + const root = await tree(); + await fs.writeFile( + path.join(root, "org/references/handbook.md"), + `---\ndescription: The org handbook\n${key}\n---\nORG HANDBOOK v1`, + "utf8", + ); + const read = await readReferencesDirectory(root); + expect(read.errors.find((e) => e.path === "org/references/handbook.md")?.kind).toBe( + "refused-declaration", + ); + } + }); + + it("BR-13 — a resources/ document may still declare `writable:`. The shared list did not widen", async () => { + const root = await tree(); + await fs.writeFile( + path.join(root, "teams/engineering/resources/scratch.md"), + "---\ndescription: Engineering scratchpad\nwritable: false\n---\nseed", + "utf8", + ); + const read = await readResourcesDirectory(root); + expect(read.errors).toEqual([]); + expect(read.documents[0]!.declared["writable"]).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// V2b · BR-19, the migration hole +// --------------------------------------------------------------------------- + +describe("V2b · BR-19 — a row written BEFORE the move shadows the file", () => { + /** + * Era 1: the handbook lives under `resources/`, and something writes it. + * Era 2: it moves to `references/` and is sealed. + * + * The seal stops new writes. It cannot evict the row already there, and that + * row still wins — so the git-is-source promise does not hold for a tree with + * history until the row is cleared. A fresh tree passes either way, which is + * exactly why this builds one with a past. + */ + async function treeWithHistory() { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "fix1467-hist-")); + roots.push(root); + const stores = createInMemoryStores(); + + // Era 1 — mutable, under resources/. + await writeInto(root, { + "org/resources/handbook.md": doc("The org handbook", "ORG HANDBOOK v1"), + }); + const documents = resourcesFromDocs((await readResourcesDirectory(root)).documents); + const before = hireWorkforce([seat(ENG_SEAT)], { + kinds: { [KIND]: kindWith(documents) as never }, + documents, + }); + const era1 = await ctxFor(before[0]!, stores); + await handleFor(era1.ctx, ORG_HANDBOOK)!.writeContent("WRITTEN BEFORE THE MOVE"); + + // Era 2 — the file moves into references/ and is sealed. + await fs.rm(path.join(root, "org/resources/handbook.md")); + await writeInto(root, { + "org/references/handbook.md": doc("The org handbook", "ORG HANDBOOK v1"), + }); + return { root, stores }; + } + + it("the hole is real — sealing alone leaves the stale row serving forever", async () => { + const { root, stores } = await treeWithHistory(); + const { seat: ada } = await seatOn(root); + const { ctx } = await ctxFor(ada, stores); + // Not an aspiration: this is what a `git mv` on its own buys. The next case + // is what makes it wrong. + expect(await handleFor(ctx, ORG_HANDBOOK)!.readContent()).toBe("WRITTEN BEFORE THE MOVE"); + }); + + it("RED→GREEN — after the migration the file wins, and it says what it cleared", async () => { + const { root, stores } = await treeWithHistory(); + const { seat: ada, references } = await seatOn(root); + + const result = await clearShadowedReferences({ references, orgId: ORG, content: stores.content }); + + // Observable, not silent: the migration names the ref it unshadowed and + // hands back the body that had been served in the file's place. + expect(result.checked).toEqual([ORG_HANDBOOK]); + expect(result.cleared).toEqual([ + { ref: ORG_HANDBOOK, shadowedContent: "WRITTEN BEFORE THE MOVE" }, + ]); + + const { ctx } = await ctxFor(ada, stores); + expect(await handleFor(ctx, ORG_HANDBOOK)!.readContent()).toBe("ORG HANDBOOK v1"); + }); + + it("dryRun reports the same finding and changes nothing", async () => { + const { root, stores } = await treeWithHistory(); + const { seat: ada, references } = await seatOn(root); + + const result = await clearShadowedReferences({ + references, + orgId: ORG, + content: stores.content, + dryRun: true, + }); + expect(result.cleared.map((r) => r.ref)).toEqual([ORG_HANDBOOK]); + + const { ctx } = await ctxFor(ada, stores); + expect(await handleFor(ctx, ORG_HANDBOOK)!.readContent()).toBe("WRITTEN BEFORE THE MOVE"); + }); + + it("is idempotent, and a clean tree reports nothing cleared", async () => { + const root = await tree(); + const { references } = await seatOn(root); + const stores = createInMemoryStores(); + + const first = await clearShadowedReferences({ references, orgId: ORG, content: stores.content }); + expect(first.cleared).toEqual([]); + expect(first.checked.length).toBe(5); + + const second = await clearShadowedReferences({ references, orgId: ORG, content: stores.content }); + expect(second.cleared).toEqual([]); + }); + + it("never touches a resources/ document's row — the mutable path keeps its history", async () => { + const root = await tree(); + const { seat: ada, references } = await seatOn(root); + const { ctx, stores } = await ctxFor(ada); + await handleFor(ctx, ENG_SCRATCH)!.writeContent("scratch edit"); + + await clearShadowedReferences({ references, orgId: ORG, content: stores.content }); + + const again = await ctxFor(ada, stores); + expect(await handleFor(again.ctx, ENG_SCRATCH)!.readContent()).toBe("scratch edit"); + }); +}); + +// --------------------------------------------------------------------------- +// V3 · the derived wall — D2 +// --------------------------------------------------------------------------- + +describe("V3 · the wall comes from the tree, with no filter written by the app (D2)", () => { + it("BR-1 — an ungranted seat reads the org's references and its own team's", async () => { + const { seat: ada } = await seatOn(await tree()); + const { ctx } = await ctxFor(ada); + expect(await handleFor(ctx, ORG_HANDBOOK)!.readContent()).toBe("ORG HANDBOOK v1"); + expect(await handleFor(ctx, ENG_HANDBOOK)!.readContent()).toBe("ENGINEERING HANDBOOK v1"); + }); + + it("BR-7 RED→GREEN — and CANNOT read another team's, with no install filter", async () => { + const { seat: ada } = await seatOn(await tree()); + const { ctx } = await ctxFor(ada); + // Not an empty read and not a null body: the ref is not reachable at all. + expect(handleFor(ctx, SALES_HANDBOOK)).toBeUndefined(); + }); + + it("BR-1 — its own worker folder is reachable", async () => { + const { seat: ada } = await seatOn(await tree()); + const { ctx } = await ctxFor(ada); + expect(await handleFor(ctx, ADA_NOTES)!.readContent()).toBe("ADA NOTES v1"); + }); + + it("BR-9 — a SIBLING seat's folder on the same team is not", async () => { + const { seat: ada } = await seatOn(await tree()); + const { ctx } = await ctxFor(ada); + // The walk inherits downward only. A sibling's folder is not above anyone. + expect(handleFor(ctx, BOB_NOTES)).toBeUndefined(); + }); + + it("BR-10 — width comes from the tree: moving the file up reaches both teams", async () => { + const root = await tree(); + // The sales handbook moves to the org level. Nothing else changes — no + // grant, no install-side override, because there is none to write. + await fs.rm(path.join(root, "teams/sales/references/handbook.md")); + await writeInto(root, { "org/references/company.md": doc("Company-wide", "COMPANY v1") }); + + for (const seatId of ["engineering.ada", "sales.sam"]) { + const { seat: hired } = await seatOn(root, seat(seatId)); + const { ctx } = await ctxFor(hired); + expect(await handleFor(ctx, "company")!.readContent()).toBe("COMPANY v1"); + } + }); + + it("BR-4 — a `references:` list narrows within the wall", async () => { + const { seat: ada } = await seatOn( + await tree(), + seat(ENG_SEAT, { [SEAT_REFERENCES_KEY]: [ENG_HANDBOOK] }), + ); + const { ctx } = await ctxFor(ada); + expect(await handleFor(ctx, ENG_HANDBOOK)!.readContent()).toBe("ENGINEERING HANDBOOK v1"); + expect(handleFor(ctx, ORG_HANDBOOK)).toBeUndefined(); + }); + + it("BR-4 / BP-031 — and can never widen past it: naming a sibling team refuses", async () => { + await expect( + seatOn(await tree(), seat(ENG_SEAT, { [SEAT_REFERENCES_KEY]: [SALES_HANDBOOK] })), + ).rejects.toThrow(/cannot add one from outside|is not a reference/i); + }); + + it("BR-5 — present-and-empty is restricted to nothing, and is not absent", async () => { + const { seat: ada } = await seatOn( + await tree(), + seat(ENG_SEAT, { [SEAT_REFERENCES_KEY]: [] }), + ); + const { ctx } = await ctxFor(ada); + expect(handleFor(ctx, ORG_HANDBOOK)).toBeUndefined(); + expect(handleFor(ctx, ENG_HANDBOOK)).toBeUndefined(); + // …and the mutable neighbour is untouched by a references: key. + expect(await handleFor(ctx, ENG_SCRATCH)!.readContent()).toBe("seed"); + }); + + it("BR-6 — a ref matching no file refuses at hire, naming it", async () => { + await expect( + seatOn(await tree(), seat(ENG_SEAT, { [SEAT_REFERENCES_KEY]: ["teams/engineering/hanbdook"] })), + ).rejects.toThrow(/hanbdook/); + }); + + it("a malformed `references:` refuses rather than resolving to nothing", async () => { + await expect( + seatOn(await tree(), seat(ENG_SEAT, { [SEAT_REFERENCES_KEY]: "teams/engineering/handbook" })), + ).rejects.toThrow(/is a LIST/); + }); +}); + +// --------------------------------------------------------------------------- +// V4 · the second path (BP-035) — BR-14, the regression that must not happen +// --------------------------------------------------------------------------- + +describe("V4 · BR-14 — the mutable path is untouched", () => { + it("a seat with NO `resources:` key still reaches every mutable resource", async () => { + const { seat: ada } = await seatOn(await tree()); + const { ctx } = await ctxFor(ada); + // A `resources:` grant is optional NARROWING, never a precondition for + // reach. Breaking this breaks every app in the monorepo, and it would pass + // every reference check above. + expect(await handleFor(ctx, ENG_SCRATCH)!.readContent()).toBe("seed"); + }); + + it("…and can still WRITE it. The wall seals references, not resources", async () => { + const { seat: ada } = await seatOn(await tree()); + const { ctx, stores } = await ctxFor(ada); + await handleFor(ctx, ENG_SCRATCH)!.writeContent("scratch edit"); + const again = await ctxFor(ada, stores); + expect(await handleFor(again.ctx, ENG_SCRATCH)!.readContent()).toBe("scratch edit"); + }); + + it("a mutable resource is NOT walled by the tree — a sibling team's stays reachable", async () => { + const root = await tree(); + await writeInto(root, { + "teams/sales/resources/pipeline.md": doc("Sales pipeline", "pipeline seed"), + }); + const { seat: ada } = await seatOn(root); + const { ctx } = await ctxFor(ada); + // Deliberate, and the boundary of this change: D2 derives a wall for + // REFERENCES. Tightening the mutable path is a different decision with a + // different blast radius, and this spec does not make it. + expect(await handleFor(ctx, "teams/sales/pipeline")!.readContent()).toBe("pipeline seed"); + }); + + it("#1943's `resources:` narrowing still works, alongside the wall", async () => { + const { seat: ada } = await seatOn( + await tree(), + seat(ENG_SEAT, { [SEAT_RESOURCES_KEY]: [ENG_SCRATCH] }), + ); + const { ctx } = await ctxFor(ada); + expect(await handleFor(ctx, ENG_SCRATCH)!.readContent()).toBe("seed"); + // The wall still applies to the references beside it. + expect(handleFor(ctx, SALES_HANDBOOK)).toBeUndefined(); + expect(await handleFor(ctx, ORG_HANDBOOK)!.readContent()).toBe("ORG HANDBOOK v1"); + }); + + it("an app with no references/ folder reaches every document, including a sibling team's", async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "fix1467-noref-")); + roots.push(root); + await writeInto(root, { + "teams/engineering/resources/scratch.md": doc("Scratch", "seed"), + "teams/sales/resources/pipeline.md": doc("Sales pipeline", "pipeline seed"), + "org/resources/coc.md": doc("Code of conduct", "coc seed"), + }); + const documents = resourcesFromDocs((await readResourcesDirectory(root)).documents); + const hired = hireWorkforce([seat(ENG_SEAT)], { + kinds: { [KIND]: kindWith(documents) as never }, + documents, + }); + const { ctx } = await ctxFor(hired[0]!); + + // The whole map, cross-team included, exactly as before this convention + // existed. Asserted as reach rather than as "no map was passed", because + // reach is the thing an app would lose and a minted instance's `resources` + // is the kind's merged map either way. + for (const [ref, body] of [ + [ENG_SCRATCH, "seed"], + ["teams/sales/pipeline", "pipeline seed"], + ["coc", "coc seed"], + ] as const) { + expect(await handleFor(ctx, ref)!.readContent()).toBe(body); + } + }); +}); + +// --------------------------------------------------------------------------- +// BR-15 · one basename, two slots +// --------------------------------------------------------------------------- + +describe("BR-15 · a ref claimed by both slots is refused at both doors", () => { + it("the loader reports it, naming both paths", async () => { + const root = await tree(); + await writeInto(root, { + "org/resources/handbook.md": doc("The org handbook", "a second handbook"), + }); + const roster = await readDeclaredRoster(root); + const collision = roster.problems.find((p) => p.error.message.includes("both claim the ref")); + expect(collision).toBeDefined(); + expect(collision!.error.message).toContain("org/references/handbook.md"); + expect(collision!.error.message).toContain("org/resources/handbook.md"); + }); + + it("the codegen door refuses a resources/ MODULE colliding with a references/ document", async () => { + const root = await tree(); + // Same level, same basename, different slot — one ref between them, and a + // directory apart, so the within-folder listing cannot see the pair. + await writeInto(root, { "org/resources/handbook.ts": "export default {};\n" }); + const found = await discoverResourceModules(root); + const collision = found.problems.find((p) => p.includes("cannot share a ref")); + expect(collision).toBeDefined(); + expect(collision).toContain("org/resources/handbook.ts"); + // Names the file that actually claims the ref, not a resources/ path that + // does not exist. + expect(collision).toContain("org/references/handbook.md"); + }); + + it("hireWorkforce throws for a hand-built catalog that never passed the loader", () => { + const shared = defineResource({ + ref: ORG_HANDBOOK, + scope: "org", + stateSchema: z.object({}).passthrough(), + default: {}, + content: "x", + }); + expect(() => + hireWorkforce([seat(ENG_SEAT)], { + kinds: { [KIND]: kindWith({ [ORG_HANDBOOK]: shared }) as never }, + documents: { [ORG_HANDBOOK]: shared }, + references: { [ORG_HANDBOOK]: shared }, + }), + ).toThrow(/both a document and a reference/); + }); +}); + +// --------------------------------------------------------------------------- +// BR-16 / BR-17 · migration coexistence +// --------------------------------------------------------------------------- + +describe("BR-16 / BR-17 · both folders at one level, each by its own rules", () => { + it("a tree with references/ and resources/ side by side reads both", async () => { + const root = await tree(); + await writeInto(root, { + "teams/engineering/resources/scratch.md": doc("Scratch", "seed"), + "teams/engineering/references/handbook.md": doc("Engineering handbook", "ENGINEERING HANDBOOK v1"), + }); + const { seat: ada } = await seatOn(root); + const { ctx, stores } = await ctxFor(ada); + + // The reference: sealed, served from the file. + await expect(handleFor(ctx, ENG_HANDBOOK)!.writeContent("x")).rejects.toThrow(/read-only/i); + // The document beside it: writable, and the row becomes the source. + await handleFor(ctx, ENG_SCRATCH)!.writeContent("evolved"); + const again = await ctxFor(ada, stores); + expect(await handleFor(again.ctx, ENG_SCRATCH)!.readContent()).toBe("evolved"); + expect(await handleFor(again.ctx, ENG_HANDBOOK)!.readContent()).toBe("ENGINEERING HANDBOOK v1"); + }); +}); From d1d9b09441b488f32ac3fffe31c242ef80843e5a Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 20 Sep 2026 16:58:04 +0000 Subject: [PATCH 2/4] Place a reference by its minted ref, and address the migration's real bucket MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review found four real defects, three of them in the wall and the migration — the two mechanisms with no prior art here and the least test surface inherited from `resources/`. Each fix has a check that fails without it. **The wall could be widened by an alias.** `applyReferenceWall` placed an entry by its ACCESSOR KEY, so an app exposing a team handbook under a second, single-segment key got an org-level reading and handed it to every seat in the company. Placement now comes from the definition's minted `ref`, which the loader derived from the path; an entry with no readable minted ref is unreachable rather than org-wide. `verifySeatReferenceWall` re-derives the place itself instead of asking whether the key is in `allowed` — asking that made the verification agree with the bug it exists to catch. **A reserved segment is still a legal document name.** `org/references/teams.md` mints the bare ref `teams`, which parsed as a broken `teams/` path and reached nobody. Fixed as a class, not two cases: a ref's shape is fixed by how many segments it has, so a single segment is an org document whatever it spells. **The migration looked in the wrong bucket.** A resource's row is not keyed by the bare org id. An installing flow that isolates its org scope stores under `:`, so the old lookup found nothing and reported success while the stale body kept winning — BR-19 failing in the exact mode BR-19 exists to prevent. `installedOn` is now required, so the coordinate is stated rather than guessed. Where the engine's private per-component escaping would apply (an id holding `:` or a backslash) this THROWS rather than reading the wrong cell; duplicating that escape would put one fact in two places. `flowIsolation` joins the derived-and-refused set for a reference, which is what makes the bucket knowable: a per-instance copy of a file that nothing can write is incoherent, and allowing it would make "where does this live" a per-file question. **A dry run said the work was done.** `dryRun` now rides on the result, and `describeShadowedReferences` says "WOULD be cleared" for a preview. A preview that claims completion is worse than no preview. Also: four `PUBLISHED_SHAPES` rows, because the docs publish `references/` shapes and that guard is what caught the PR body claiming clean guards when one was red. `siblingSlotPath` in `resource-convention.ts` now owns "this path's sibling in the other slot", which two files were deriving two different ways — the slice-based one silently assumed a trailing slot segment. The duplicated `prefetchMode` refusal is one helper. `ApplyReferenceWallInput.kind` is gone: none of that function's refusals names a kind, so callers no longer supply something it ignores. Docs move with the signature. The `clearShadowedReferences` example in `documents-on-disk.md` and the README both showed the old three-argument form, which no longer compiles and which demonstrated exactly the silent-miss this commit fixes. Both now pass `installedOn` and say why it is not defaulted. Evidence: 680 pass / 41 files in workforce, after merging main. Both new fixes mutation-checked — placing by accessor key reds the alias case, dropping the length-first rule reds the `teams` case. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016qaz1VpRK4wCQzxpjzZ4Hz --- .changeset/references-read-path.md | 2 +- apps/docs/docs/workforce/documents-on-disk.md | 11 +- packages/workforce/README.md | 20 +- .../src/clear-shadowed-references.ts | 181 ++++++++++---- .../src/codegen/discover-resource-modules.ts | 3 +- packages/workforce/src/hire.ts | 3 +- packages/workforce/src/index.ts | 2 + packages/workforce/src/loader/index.ts | 1 + .../src/loader/read-declared-roster.ts | 7 +- .../src/loader/resource-convention.ts | 35 +++ packages/workforce/src/manifest.ts | 53 ++-- packages/workforce/src/seat-references.ts | 70 +++++- .../test/published-tree-surface.test.ts | 58 ++++- .../test/references-convention.test.ts | 234 +++++++++++++++++- 14 files changed, 587 insertions(+), 93 deletions(-) diff --git a/.changeset/references-read-path.md b/.changeset/references-read-path.md index 87f9dc8343..c46987ab34 100644 --- a/.changeset/references-read-path.md +++ b/.changeset/references-read-path.md @@ -2,4 +2,4 @@ "@flow-state-dev/workforce": minor --- -A workforce tree can declare read-only documents in `references/` beside the writable ones in `resources/`. A reference's body is served from its file on every execution context rather than from a stored row, so editing the file is the edit; `writable`, `llmWritable` and `render` are derived by the folder and refused in frontmatter. A seat reaches the references at or above its place in the tree — the org's, its own team's and its own folder's — with no install-side filter, narrowed further by a `references:` list in its `WORKER.md`. `resources/` behaviour is unchanged. `clearShadowedReferences` migrates a tree where a document was written before it moved (FIX-1467). +A workforce tree can declare read-only documents in `references/` beside the writable ones in `resources/`. A reference's body is served from its file on every execution context rather than from a stored row, so editing the file is the edit; `writable`, `llmWritable`, `render` and `flowIsolation` are derived by the folder and refused in frontmatter. A seat reaches the references at or above its place in the tree — the org's, its own team's and its own folder's — with no install-side filter, narrowed further by a `references:` list in its `WORKER.md`. `resources/` behaviour is unchanged. `clearShadowedReferences({ references, orgId, content, installedOn })` migrates a tree where a document was written before it moved — `installedOn` names the flow so the stored content is addressed where it actually lives rather than guessed at (FIX-1467). diff --git a/apps/docs/docs/workforce/documents-on-disk.md b/apps/docs/docs/workforce/documents-on-disk.md index a4f9f6992a..4cad6a0b51 100644 --- a/apps/docs/docs/workforce/documents-on-disk.md +++ b/apps/docs/docs/workforce/documents-on-disk.md @@ -84,7 +84,9 @@ Where a file sits decides its identity, where it is stored, and what its content Everything outside that set arrives at the resource exactly as written, so `llmReadable`, `llmWritable`, `writable`, `allowedExtensions` and `metadata` all take effect from the file. -A file in `references/` derives three more: `writable`, `llmWritable` and `render`. The folder is what makes a reference read-only, so no file has to ask for that and no file can turn it off. Declaring any of the three is refused by name, `writable: false` included — agreeing with the folder is still a second place to keep the same fact, and the next person to open the file cannot tell which copy is the one doing the work. A document that needs to be written belongs in `resources/`. +A file in `references/` derives four more: `writable`, `llmWritable`, `render` and `flowIsolation`. The folder is what makes a reference read-only, so no file has to ask for that and no file can turn it off. Declaring any of the four is refused by name, `writable: false` included — agreeing with the folder is still a second place to keep the same fact, and the next person to open the file cannot tell which copy is the one doing the work. A document that needs to be written belongs in `resources/`. + +`flowIsolation` is refused for a second reason worth saying out loud: a per-seat copy of a document whose body is one file for everybody cannot differ from the file or from itself, and it would put each copy's stored content somewhere different — which is what the migration below has to be able to find. ## A document's ref @@ -346,15 +348,20 @@ const result = await clearShadowedReferences({ references: referencesFromDocs(references.documents), orgId, content: stores.content, + installedOn: { id: flow.id, isolatesOrgState: false }, }); console.log(describeShadowedReferences(result)); // references: 1 of 4 were shadowed by a stored write and have been cleared — handbook. ``` -It reports before it is believed: `result.cleared` names each reference and carries the body that had been served in the file's place, which is the last moment that text exists anywhere. Pass `dryRun: true` to see the finding and change nothing. +`installedOn` describes the flow the references are installed on, and it is required rather than assumed. A flow that isolates its organization scope keeps each instance's stored content in its own place, so a run told the wrong thing would look in the wrong place, find nothing, and report success — the one outcome worse than not running it. Read `isolatesOrgState` off the flow rather than guessing. + +It reports before it is believed: `result.cleared` names each reference and carries the body that had been served in the file's place, which is the last moment that text exists anywhere. Pass `dryRun: true` to see the finding and change nothing — the result says it was a preview, and the sentence says "would". Run it once per organization after the move. It is safe to run again — a tree whose files are already the source reports nothing cleared — and it never touches a `resources/` document. +It refuses, loudly, for an organization or flow id containing `:` or a backslash. Those need the escaping the engine applies when it builds a storage address, and this cannot reproduce that rule without keeping a second copy of it — so it stops rather than reading the wrong place and telling you there was nothing to clear. + ## What stays in TypeScript `defineResource` is the other route into the same map, and the two sit side by side, as `ticket` and the file-declared documents do above. diff --git a/packages/workforce/README.md b/packages/workforce/README.md index c95bfa863d..88edea74c7 100644 --- a/packages/workforce/README.md +++ b/packages/workforce/README.md @@ -750,7 +750,7 @@ refused rather than resolved. What differs is what the document IS afterwards: | Slot | Content an agent reads | Writable | Reach | | --- | --- | --- | --- | -| `references/` | the file, re-read on each execution context | no — `writable`, `llmWritable` and `render` are derived, and a file declaring any of them is refused | derived from the tree: the org's, the seat's own team's, and the seat's own folder | +| `references/` | the file, re-read on each execution context | no — `writable`, `llmWritable`, `render` and `flowIsolation` are derived, and a file declaring any of them is refused | derived from the tree: the org's, the seat's own team's, and the seat's own folder | | `resources/` | the file's body seeds a row; the row is the source from then on | yes, unless the file says otherwise | every document the flow was installed with | Put standing material an author owns in the repository under `references/`; put anything an agent @@ -795,9 +795,16 @@ references at or above its place in the tree. Moving a file from `resources/` into `references/` needs one extra step when anything ever wrote that document: the written body is still stored, and a stored body wins over the file. -`clearShadowedReferences({ references, orgId, content })` finds those rows, reports each one with -the body it had been serving, and clears them. It is idempotent, takes `dryRun`, and never touches -a `resources/` document. +`clearShadowedReferences({ references, orgId, content, installedOn })` finds those rows, reports +each one with the body it had been serving, and clears them. It is idempotent, takes `dryRun` (and +says so on the result, so the log sentence reads "would clear"), and never touches a `resources/` +document. + +`installedOn` is `{ id, isolatesOrgState }` for the flow the references are installed on, and it is +required rather than defaulted: a flow that isolates its org scope stores content under a different +address, so a wrong assumption here looks in the wrong place, finds nothing, and reports success. +For an org or flow id containing `:` or a backslash it throws instead of guessing — that address +needs the engine's own escaping, and a second copy of that rule here is how the two would drift. ```ts import { defineFlow } from "@flow-state-dev/core"; @@ -886,6 +893,11 @@ flowIsolation: true Both sentences are owed together. A worker folder without that line is an address, not a boundary. +This is a `resources/` setting. A `references/` file that declares it is refused: a per-seat copy of +a document whose body is one file for everyone cannot differ from the file or from itself, and the +copies would each store their content somewhere different — which is the address the reference +migration has to be able to find. A reference is walled by where it sits, not by a frontmatter key. + **`description` is required.** A file without one lands in `errors`. It reaches the resource with the rest of the frontmatter, and nothing puts it in front of a model. Write it for whoever opens the tree. diff --git a/packages/workforce/src/clear-shadowed-references.ts b/packages/workforce/src/clear-shadowed-references.ts index 559f6f96db..9cbf7fe1e3 100644 --- a/packages/workforce/src/clear-shadowed-references.ts +++ b/packages/workforce/src/clear-shadowed-references.ts @@ -10,20 +10,40 @@ * `contentFile` exactly as it wins over `content`, so a handbook that anything * wrote once — a block, an HTTP content PUT, an agent with the write tool — * keeps serving that write after the move. Forever, silently, with the file in - * git looking authoritative and being read by nobody. + * the repository looking authoritative and being read by nobody. * - * So for a tree that has ever been written to, `git mv` does not make the - * promise true. This does. + * So for a tree that has ever been written to, moving the file does not make + * the promise true. This does. * - * **The old shape is tolerated, not assumed away** (BP-030). A row may be - * there or may not; both are ordinary. What this refuses to do is leave a - * present one in place and say nothing — the whole point is that the outcome - * is reported rather than silent, so a deploy log says which handbooks were - * unshadowed and what they had been serving. + * **The old shape is tolerated, not assumed away** (BP-030). A row may be there + * or may not; both are ordinary. What this refuses to do is leave a present one + * in place and say nothing. * - * **Run it once per org, after the move, before or at boot.** It is idempotent: - * a second run over a migrated tree finds nothing and clears nothing, because - * the seal means no new row can appear. + * + * ## Addressing the row, and where this REFUSES rather than guesses + * + * A row is addressed by `(scopeType, scopeId, resourceKey)`. For a reference, + * `scopeType` is `"org"` and `resourceKey` is its ref — but `scopeId` is NOT + * simply the org id, and getting that wrong is the one failure mode worse than + * having no migration at all: the lookup finds nothing, the run reports + * "nothing to clear", and the stale body keeps winning. + * + * Two things decide it, and this module handles both explicitly. + * + * **Isolation.** A resource whose scope is isolated stores under + * `:` rather than under the org id alone. A reference can never + * be isolated by its own frontmatter — `flowIsolation` is a derived key and a + * file declaring it is refused — but the installing FLOW can isolate the whole + * org scope, and then every resource on it is isolated. That is why + * {@link ReferenceInstallFlow.isolatesOrgState} is required rather than + * defaulted: a default would be a guess, and a wrong guess is silent. + * + * **Escaping.** The engine escapes `:` and `\` in each component before joining + * them, so a bucket for an org id containing either is not the string this + * module would build. That escape is private to the engine, and duplicating it + * here would put one fact in two places — the exact drift this package's + * conventions exist to prevent. So instead of guessing, this **throws**, naming + * the id and why. Loud beats silently addressing the wrong cell. * * Isomorphic by construction — it takes the store it writes through rather than * reaching for one, so this module has no `node:fs` and no engine import. @@ -36,7 +56,7 @@ import type { DeclaredResources } from "@flow-state-dev/core"; * row. * * Declared structurally rather than imported, so `workforce` does not take a - * dependency on `engine` for two method signatures. Any `ContentStore` + * runtime dependency on `engine` for two method signatures. Any `ContentStore` * satisfies it — `stores.content` is passed straight in. */ export interface ReferenceContentStore { @@ -46,6 +66,38 @@ export interface ReferenceContentStore { delete(scopeType: "org", scopeId: string, resourceKey: string): Promise; } +/** The flow the references are installed on, as the storage coordinate needs it. */ +export interface ReferenceInstallFlow { + /** The flow instance's id — half the bucket when the org scope is isolated. */ + id: string; + /** + * Whether that flow isolates its org scope (`isolateOrgState`). + * + * Required, not optional. A reference inherits the flow's org isolation, and + * the two settings address different buckets — so a default here would be a + * guess that reports success while looking in the wrong place. + */ + isolatesOrgState: boolean; +} + +/** Everything {@link clearShadowedReferences} needs. */ +export interface ClearShadowedReferencesInput { + /** + * The app's references, keyed by ref — the map `referencesFromDocs` returns. + * Only these keys are touched; a `resources/` document's row is never read + * and never deleted. + */ + references: DeclaredResources; + /** The org whose rows to check. */ + orgId: string; + /** The engine's content store — `stores.content`. */ + content: ReferenceContentStore; + /** The flow the references were installed on. See [the coordinate](#coordinate). */ + installedOn: ReferenceInstallFlow; + /** Report what WOULD be cleared and delete nothing. */ + dryRun?: boolean; +} + /** One reference that had a stored row shadowing its file. */ export interface ShadowedReference { /** The reference's ref, which is also its storage key. */ @@ -64,7 +116,8 @@ export interface ShadowedReference { /** What {@link clearShadowedReferences} found and did. */ export interface ClearShadowedReferencesResult { /** - * Every reference whose row was cleared, in catalog order. + * Every reference whose row was cleared — or WOULD be, when + * {@link ClearShadowedReferencesResult.dryRun} is true. * * **Empty is the ordinary answer** — a tree that was never written to, or one * already migrated. Non-empty is the finding: each entry is a file that was @@ -77,49 +130,84 @@ export interface ClearShadowedReferencesResult { * "checked nothing" are the two outcomes a silent migration confuses. */ checked: string[]; + /** + * Whether this was a preview. + * + * **On the result, not only in the caller's memory**, because + * {@link describeShadowedReferences} reads it. A preview that prints "have + * been cleared" is worse than no preview: an operator reads a completed + * migration, moves on, and leaves the stale content live. + */ + dryRun: boolean; + /** The bucket the rows were addressed in, for a log that has to be believed. */ + scopeId: string; +} + +/** + * The bucket a reference's content row lives in. + * + * Mirrors the engine's `resolveResourceScopeId` for the cases this module + * accepts, and **throws for the case it cannot mirror**: the engine escapes `:` + * and `\` per component with a function it does not export, so an id carrying + * either would key somewhere this cannot name. See [the coordinate](#coordinate). + */ +function referenceScopeId(orgId: string, installedOn: ReferenceInstallFlow): string { + const needsEscaping = (value: string): boolean => /[\\:]/.test(value); + const offending = [ + ...(needsEscaping(orgId) ? [`orgId "${orgId}"`] : []), + ...(installedOn.isolatesOrgState && needsEscaping(installedOn.id) + ? [`flow id "${installedOn.id}"`] + : []), + ]; + if (offending.length > 0) { + throw new Error( + `Cannot address this org's reference rows: ${offending.join(" and ")} contains ":" or a ` + + `backslash, which the engine escapes when it builds a storage bucket — with a rule this ` + + `package cannot reproduce without keeping a second copy of it. Refusing rather than ` + + `reading the wrong cell and reporting nothing to clear. Clear these rows with the ` + + `engine's own store helpers, or migrate this org by hand.`, + ); + } + return installedOn.isolatesOrgState ? `${orgId}:${installedOn.id}` : orgId; } /** * Clear every stored content row shadowing a reference's file, for one org. * - * @param input.references The app's references, keyed by ref — the map - * `referencesFromDocs` returns. Only these keys are touched; a `resources/` - * document's row is never read and never deleted. - * @param input.orgId The org whose rows to check. References are org-scoped, so - * this is the scope id their rows are stored under. - * @param input.content The engine's content store — `stores.content`. - * @param input.dryRun When `true`, report what WOULD be cleared and delete - * nothing. For a first run against a tree nobody wants to surprise. - * @returns What was checked and what was cleared. Nothing is thrown for an - * absent row; that is the ordinary case. + * Idempotent: a second run over a migrated tree finds nothing and clears + * nothing, because the seal means no new row can appear. + * + * @param input The references, the org, the store, and the flow they are + * installed on. See {@link ClearShadowedReferencesInput}. + * @returns What was checked, what was cleared, and whether it was a preview. + * Nothing is thrown for an absent row; that is the ordinary case. + * @throws If the org or flow id needs the engine's bucket escaping — see + * [the coordinate](#coordinate). */ -export async function clearShadowedReferences(input: { - references: DeclaredResources; - orgId: string; - content: ReferenceContentStore; - dryRun?: boolean; -}): Promise { - const { references, orgId, content, dryRun = false } = input; +export async function clearShadowedReferences( + input: ClearShadowedReferencesInput, +): Promise { + const { references, orgId, content, installedOn, dryRun = false } = input; + const scopeId = referenceScopeId(orgId, installedOn); const cleared: ShadowedReference[] = []; const checked: string[] = []; for (const ref of Object.keys(references)) { checked.push(ref); - // The ref IS the storage key for a file-declared document: `resourcesFromDocs` - // and `referencesFromDocs` both pass it as `ref`, and the registry keys - // content by it. - const shadowedContent = await content.get("org", orgId, ref); + // The ref IS the storage key for a file-declared document: both install + // halves pass it as `ref`, and the registry keys content by it. + const shadowedContent = await content.get("org", scopeId, ref); // `undefined` is no row. An empty STRING is a row — something wrote a blank // body — and it shadows the file just as completely, so it is cleared too. // `== null` rather than a falsy check, for exactly that reason. if (shadowedContent == null) continue; cleared.push({ ref, shadowedContent }); - if (!dryRun) await content.delete("org", orgId, ref); + if (!dryRun) await content.delete("org", scopeId, ref); } - return { cleared, checked }; + return { cleared, checked, dryRun, scopeId }; } /** @@ -129,15 +217,24 @@ export async function clearShadowedReferences(input: { * caller that most needs it is a boot script that should not have to invent * one. Bodies are not included — they can be a whole handbook; the result * carries them for a caller that wants them. + * + * A preview says **would**, and says it is a preview. That is the whole reason + * `dryRun` is on the result rather than only in the caller's memory. */ export function describeShadowedReferences(result: ClearShadowedReferencesResult): string { if (result.checked.length === 0) return "references: none declared, nothing to migrate"; + if (result.cleared.length === 0) { - return `references: ${result.checked.length} checked, none shadowed — every file is the source`; + return result.dryRun + ? `references: dry run — ${result.checked.length} checked, none shadowed. Nothing to do` + : `references: ${result.checked.length} checked, none shadowed — every file is the source`; } - return ( - `references: ${result.cleared.length} of ${result.checked.length} were shadowed by a ` + - `stored write and have been cleared — ${result.cleared.map((r) => r.ref).join(", ")}. ` + - `Each now serves its file again.` - ); + + const refs = result.cleared.map((r) => r.ref).join(", "); + return result.dryRun + ? `references: dry run — ${result.cleared.length} of ${result.checked.length} are shadowed ` + + `by a stored write and WOULD be cleared: ${refs}. Nothing has been changed; re-run ` + + `without dryRun to clear them` + : `references: ${result.cleared.length} of ${result.checked.length} were shadowed by a ` + + `stored write and have been cleared — ${refs}. Each now serves its file again.`; } diff --git a/packages/workforce/src/codegen/discover-resource-modules.ts b/packages/workforce/src/codegen/discover-resource-modules.ts index 81f2fd15ba..344b7613c8 100644 --- a/packages/workforce/src/codegen/discover-resource-modules.ts +++ b/packages/workforce/src/codegen/discover-resource-modules.ts @@ -43,6 +43,7 @@ import { openRoot, openStructuralDirectory, refusedSymlink, + siblingSlotPath, unreadable, walkTeams, } from "../loader"; @@ -271,7 +272,7 @@ async function readSlot(slotDir: string, slotPath: string, ctx: SlotContext): Pr // here too would double every entry in a tree with one bad slot. A resources/ // file wins the map only because it was inserted first; either path names a // real claimant, and the pair is refused whichever is quoted. - const referenceSlotPath = `${slotPath.slice(0, -RESOURCES_SLOT.length)}${REFERENCES_SLOT}`; + const referenceSlotPath = siblingSlotPath(slotPath, RESOURCES_SLOT, REFERENCES_SLOT); const siblings = await openStructuralDirectory( path.join(path.dirname(slotDir), REFERENCES_SLOT), referenceSlotPath, diff --git a/packages/workforce/src/hire.ts b/packages/workforce/src/hire.ts index a04eb459ec..839b7519fe 100644 --- a/packages/workforce/src/hire.ts +++ b/packages/workforce/src/hire.ts @@ -691,8 +691,7 @@ export function hireWorkforce( catalog: options.references, kindResources: (factory.resources ?? {}) as DeclaredResources, kindFlowLevelKeys: factory.flowLevelResourceKeys ?? new Set(), - base: seatResources, - kind + base: seatResources }); if (wall.problems.length > 0) { for (const problem of wall.problems) refuse(problem); diff --git a/packages/workforce/src/index.ts b/packages/workforce/src/index.ts index bf519e0c3c..a92fdd54be 100644 --- a/packages/workforce/src/index.ts +++ b/packages/workforce/src/index.ts @@ -50,8 +50,10 @@ export { referencesFromDocs, referenceBody } from "./references-from-docs"; export { clearShadowedReferences, describeShadowedReferences, + type ClearShadowedReferencesInput, type ClearShadowedReferencesResult, type ReferenceContentStore, + type ReferenceInstallFlow, type ShadowedReference, } from "./clear-shadowed-references"; export { diff --git a/packages/workforce/src/loader/index.ts b/packages/workforce/src/loader/index.ts index bfacd31841..14393bf162 100644 --- a/packages/workforce/src/loader/index.ts +++ b/packages/workforce/src/loader/index.ts @@ -114,6 +114,7 @@ export { WORKERS_LEVEL, type DocumentSlot, mintResourceRef, + siblingSlotPath, } from "./resource-convention"; export type { WorkerManifest, TeamManifest, ResourceDoc, ChannelManifest } from "../manifest"; diff --git a/packages/workforce/src/loader/read-declared-roster.ts b/packages/workforce/src/loader/read-declared-roster.ts index 5479b5ad8e..e472953948 100644 --- a/packages/workforce/src/loader/read-declared-roster.ts +++ b/packages/workforce/src/loader/read-declared-roster.ts @@ -28,7 +28,7 @@ import path from "node:path"; import { readChannelsDirectory } from "./read-channels-directory"; import { readReferencesDirectory, readResourcesDirectory } from "./read-resources-directory"; import { readWorkforce } from "./read-workforce"; -import { REFERENCES_SLOT, RESOURCES_SLOT } from "./resource-convention"; +import { REFERENCES_SLOT, RESOURCES_SLOT, siblingSlotPath } from "./resource-convention"; import type { ChannelManifest, ResourceDoc, @@ -222,10 +222,7 @@ function collidingRefs( const sibling = reference.filePath === undefined ? reference.ref - : reference.filePath.replace( - `${path.sep}${REFERENCES_SLOT}${path.sep}`, - `${path.sep}${RESOURCES_SLOT}${path.sep}`, - ); + : siblingSlotPath(reference.filePath, REFERENCES_SLOT, RESOURCES_SLOT, path.sep); problems.push({ layer: "reference", path: at, diff --git a/packages/workforce/src/loader/resource-convention.ts b/packages/workforce/src/loader/resource-convention.ts index 06acc3eece..e416dfb5a5 100644 --- a/packages/workforce/src/loader/resource-convention.ts +++ b/packages/workforce/src/loader/resource-convention.ts @@ -54,6 +54,41 @@ export const DOCUMENT_SLOTS = [RESOURCES_SLOT, REFERENCES_SLOT] as const; /** One of {@link DOCUMENT_SLOTS}. */ export type DocumentSlot = (typeof DOCUMENT_SLOTS)[number]; +/** + * The same path with its slot segment swapped for the other slot's. + * + * "This path's sibling in the other slot" is a derived fact about the + * convention, so it belongs here with the folder names rather than being worked + * out at each call site — which is where it was, twice, in two directions: the + * module walk sliced a trailing `resources` off and concatenated, and the + * roster join did a separator-aware replace. The slice was the fragile half. It + * assumed the path ended in exactly the slot with nothing after it, which held + * only because of how its callers happened to build the string. + * + * Swaps the LAST occurrence, because a path may legitimately contain the slot + * name higher up — a team called `references` is a legal team. + * + * @param at The path, in whatever separator its caller uses. + * @param from The slot `at` sits in. + * @param to The slot to address instead. + * @param sep The separator `at` is written with. Slash for the loader's + * root-relative paths, `path.sep` for an absolute one. + * @returns The sibling path, or `at` unchanged when it holds no `from` segment. + */ +export function siblingSlotPath( + at: string, + from: DocumentSlot, + to: DocumentSlot, + sep = "/", +): string { + const segments = at.split(sep); + // Last, not first: only the trailing one is the slot this path sits in. + const slotAt = segments.lastIndexOf(from); + if (slotAt === -1) return at; + segments[slotAt] = to; + return segments.join(sep); +} + /** The level a worker's folder sits in, under `org/` and under every team. */ export const WORKERS_LEVEL = "workers"; diff --git a/packages/workforce/src/manifest.ts b/packages/workforce/src/manifest.ts index 72c6e39c5c..cb16f68f66 100644 --- a/packages/workforce/src/manifest.ts +++ b/packages/workforce/src/manifest.ts @@ -554,6 +554,26 @@ export const DERIVED_RESOURCE_KEYS = [ */ const REFUSED_PREFETCH_MODE = "lazy"; +/** + * Say why a document's `prefetchMode` is refused, or `undefined` when it is not. + * + * Shared by both doors' refusals rather than written twice. The two differ in + * which KEYS they derive — that difference is real and stays two loops — but + * `prefetchMode` is refused for one reason that has nothing to do with the slot + * a file sits in: a file-declared resource is installed at flow level either + * way, so both doors were carrying the same sentence. + */ +function refusedPrefetchModeMessage( + declared: Record, +): string | undefined { + if (declared["prefetchMode"] !== REFUSED_PREFETCH_MODE) return undefined; + return ( + `declares \`prefetchMode: "${REFUSED_PREFETCH_MODE}"\`, which a file-declared ` + + `resource cannot be: it is installed at flow level, and a flow-level declaration ` + + `has no per-block load trigger to load it on. Drop the key.` + ); +} + /** * Say why a document's declaration is refused, or `undefined` when nothing is. * @@ -578,15 +598,7 @@ export function refusedDeclarationMessage( } } - if (declared["prefetchMode"] === REFUSED_PREFETCH_MODE) { - return ( - `declares \`prefetchMode: "${REFUSED_PREFETCH_MODE}"\`, which a file-declared ` + - `resource cannot be: it is installed at flow level, and a flow-level declaration ` + - `has no per-block load trigger to load it on. Drop the key.` - ); - } - - return undefined; + return refusedPrefetchModeMessage(declared); } /** @@ -610,16 +622,25 @@ export function refusedDeclarationMessage( * reader gets the document and not the YAML above it. A file that supplied * its own would serve its own frontmatter, or throw at read time when a YAML * scalar is called as a function. + * - `flowIsolation` — a per-instance copy of a document whose content is the + * same file for everyone is incoherent: nothing can write the copies, so they + * would differ from each other never and from the file always. It also has a + * second cost, which is why it is refused rather than ignored — an isolated + * resource stores under a flow-qualified bucket, so allowing it would make + * "where does this reference's content live" a per-file question, and the + * migration in `../clear-shadowed-references` could no longer name one place + * to look. * * **Deliberately NOT added to {@link DERIVED_RESOURCE_KEYS}.** A `resources/` - * document may legitimately declare `writable:` — that path is unchanged, and - * widening the shared list would break it. + * document may legitimately declare `writable:` or `flowIsolation:` — that path + * is unchanged, and widening the shared list would break it. */ export const DERIVED_REFERENCE_KEYS = [ ...DERIVED_RESOURCE_KEYS, "writable", "llmWritable", "render", + "flowIsolation", ] as const; /** @@ -647,13 +668,5 @@ export function refusedReferenceDeclarationMessage( ); } - if (declared["prefetchMode"] === REFUSED_PREFETCH_MODE) { - return ( - `declares \`prefetchMode: "${REFUSED_PREFETCH_MODE}"\`, which a file-declared ` + - `resource cannot be: it is installed at flow level, and a flow-level declaration ` + - `has no per-block load trigger to load it on. Drop the key.` - ); - } - - return undefined; + return refusedPrefetchModeMessage(declared); } diff --git a/packages/workforce/src/seat-references.ts b/packages/workforce/src/seat-references.ts index 0ef8ea4ff2..5775a3b8c9 100644 --- a/packages/workforce/src/seat-references.ts +++ b/packages/workforce/src/seat-references.ts @@ -99,6 +99,17 @@ export function placeOfReference(ref: string): TreePlace | undefined { const parts = ref.split("/"); if (parts.some((part) => part.length === 0)) return undefined; + // **Length decides which segments are structural, never their spelling.** + // `teams` and `workers` are reserved PATH segments and also perfectly legal + // document names — `validateSegment` admits both — so `org/references/teams.md` + // mints the single-segment ref `teams`. Reading that as a broken `teams/` + // path made an ordinary org document reachable by nobody. + // + // Handled as a class rather than as two special cases: a ref's shape is fixed + // by how many segments it has, because that is what `mintResourceRef` + // guarantees. A single segment is an org document whatever it spells. + if (parts.length === 1) return { team: undefined, worker: undefined }; + let team: string | undefined; let rest = parts; if (parts[0] === TEAMS_LEVEL) { @@ -112,6 +123,27 @@ export function placeOfReference(ref: string): TreePlace | undefined { return undefined; } +/** + * Where the reference behind one of a kind's accessor keys actually sits. + * + * **The accessor key is not the place.** An app may expose one definition under + * a second key — `{ ...references, handbook: references["teams/eng/handbook"] }` + * — and that alias is still the team's handbook. Placing it by its key would + * read a single-segment alias as an org document and hand it to every seat in + * the company, which is the exact widening the wall exists to stop. So the + * place comes from the definition's own minted `ref`, which the loader derived + * from the path on disk, and never from the key an app chose (BP-031). + * + * `undefined` when no minted ref can be read, which the wall treats as + * unreachable. That is the safer reading of a definition this module cannot + * place: an alias nobody can locate is denied rather than published org-wide. + */ +function placeOfEntry(entry: unknown): TreePlace | undefined { + const minted = (entry as { ref?: unknown } | undefined)?.ref; + if (typeof minted !== "string" || minted.length === 0) return undefined; + return placeOfReference(minted); +} + /** * Read a seat's id back into the place its folder sits. * @@ -312,8 +344,6 @@ export interface ApplyReferenceWallInput { * declares none and would be minted with the kind's own map. */ base: DeclaredResources | undefined; - /** The kind's name, for the refusals. */ - kind: string; } /** What {@link applyReferenceWall} produced. */ @@ -350,8 +380,11 @@ export interface AppliedReferenceWall { * @returns The seat's map, the reachable set, and every problem. */ export function applyReferenceWall(input: ApplyReferenceWallInput): AppliedReferenceWall { - const { seatId, declared, hasDeclared, catalog, kindResources, kindFlowLevelKeys, base, kind } = - input; + // No `kind` here, deliberately: none of this function's three refusals names + // it. The wall is about where a seat SITS, and `hireWorkforce` already + // prefixes every refusal with the worker's id. The kind is named only by + // `verifySeatReferenceWall`, whose refusal is about a block on that kind. + const { seatId, declared, hasDeclared, catalog, kindResources, kindFlowLevelKeys, base } = input; // Which of the kind's accessors hold one of the app's references. By name OR // by definition identity, the way `seat-resources` recognises a document: an @@ -373,10 +406,15 @@ export function applyReferenceWall(input: ApplyReferenceWallInput): AppliedRefer } // The wall: derived, before anything the seat wrote is read. + // + // Placed from the DEFINITION's minted ref, not from the accessor key it sits + // under — see `placeOfEntry`. The two are the same string for every entry an + // app installs under its own ref, and differ exactly for an alias, which is + // the case that would otherwise widen. const reachable = new Set(); - for (const ref of references) { - const place = placeOfReference(ref); - if (place !== undefined && referenceReachableBySeat(seat, place)) reachable.add(ref); + for (const key of references) { + const place = placeOfEntry(kindResources[key] ?? catalog?.[key]); + if (place !== undefined && referenceReachableBySeat(seat, place)) reachable.add(key); } // The narrowing, if the seat asked for one. Selects from `reachable`; a ref @@ -478,6 +516,12 @@ export function verifySeatReferenceWall(input: VerifySeatReferenceWallInput): st if (minted === undefined || catalog === undefined) return []; const definitions = new Set(Object.values(catalog)); + // Re-derived here rather than taken from `applyReferenceWall`, so this is an + // independent check and not a restatement. Asking only "is this key in + // `allowed`?" makes the verification agree with whatever the wall computed — + // including a mistake, which is how an aliased entry could be waved through + // by the very check that exists to catch it. + const seat = placeOfSeat(seatId); const problems: string[] = []; for (const key of Object.keys(minted)) { @@ -485,7 +529,17 @@ export function verifySeatReferenceWall(input: VerifySeatReferenceWallInput): st // is not that reference, and flagging it by name would refuse an app whose // wiring is fine. if (!definitions.has(minted[key])) continue; - if (!allowed.has(key)) problems.push(referenceCrossedWallMessage(key, seatId, kind)); + + const place = placeOfEntry(minted[key]); + const withinWall = + seat !== undefined && place !== undefined && referenceReachableBySeat(seat, place); + + // Two ways to be wrong, one refusal: the entry is outside this seat's place + // in the tree, or it is inside but the seat narrowed it away and it came + // back. Both mean the minted flow reaches a reference the seat must not. + if (!withinWall || !allowed.has(key)) { + problems.push(referenceCrossedWallMessage(key, seatId, kind)); + } } return problems; diff --git a/packages/workforce/test/published-tree-surface.test.ts b/packages/workforce/test/published-tree-surface.test.ts index 54e221837e..48a97a3e88 100644 --- a/packages/workforce/test/published-tree-surface.test.ts +++ b/packages/workforce/test/published-tree-surface.test.ts @@ -50,7 +50,10 @@ import path from "node:path"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; import { WorkforceCodeError, discoverWorkforceCode } from "../src/codegen/discover"; import { readChannelsDirectory } from "../src/loader/read-channels-directory"; -import { readResourcesDirectory } from "../src/loader/read-resources-directory"; +import { + readReferencesDirectory, + readResourcesDirectory, +} from "../src/loader/read-resources-directory"; import { readSeatSkills } from "../src/loader/read-seat-skills"; import { readWorkforce } from "../src/loader/read-workforce"; @@ -61,8 +64,10 @@ const REPO_ROOT = path.resolve(import.meta.dirname, "../../.."); interface Readout { /** Worker ids that loaded. */ workers: string[]; - /** Document refs that loaded. */ + /** Document refs that loaded, from the `resources/` slot. */ documents: string[]; + /** Reference refs that loaded, from the `references/` slot. Its own field, not merged into `documents`: the two slots go to different install halves, and a shape accounted for by the wrong one is a shape nobody reads. */ + references: string[]; /** Channel ids that loaded. */ channels: string[]; /** Ids of teams whose `TEAM.md` loaded. */ @@ -443,6 +448,51 @@ const PUBLISHED_SHAPES: readonly PublishedShape[] = [ out.channels.includes("alpha.standup") || out.reported.includes("teams/alpha/channels/standup"), }, + { + shape: "org/references/code-of-conduct.md", + publishedIn: { + file: "apps/docs/docs/workforce/documents-on-disk.md", + quote: " code-of-conduct.md", + }, + write: (root) => writeFile(root, "org/references/code-of-conduct.md", doc("A code of conduct.")), + accountedFor: (out) => + out.references.includes("code-of-conduct") || + out.reported.includes("org/references/code-of-conduct.md"), + }, + { + shape: "teams//references/handbook.md", + publishedIn: { + file: "apps/docs/docs/workforce/documents-on-disk.md", + quote: " handbook.md", + }, + write: (root) => writeFile(root, "teams/alpha/references/handbook.md", doc("A handbook.")), + accountedFor: (out) => + out.references.includes("teams/alpha/handbook") || + out.reported.includes("teams/alpha/references/handbook.md"), + }, + { + shape: "teams//references/escalation.md", + publishedIn: { + file: "apps/docs/docs/workforce/documents-on-disk.md", + quote: " escalation.md", + }, + write: (root) => writeFile(root, "teams/alpha/references/escalation.md", doc("An escalation.")), + accountedFor: (out) => + out.references.includes("teams/alpha/escalation") || + out.reported.includes("teams/alpha/references/escalation.md"), + }, + { + shape: "teams//workers//references/runbook.md", + publishedIn: { + file: "apps/docs/docs/workforce/documents-on-disk.md", + quote: " runbook.md", + }, + write: (root) => + writeFile(root, "teams/alpha/workers/lead/references/runbook.md", doc("A runbook.")), + accountedFor: (out) => + out.references.includes("teams/alpha/workers/lead/runbook") || + out.reported.includes("teams/alpha/workers/lead/references/runbook.md"), + }, { shape: "teams//resources/.md", publishedIn: { @@ -704,6 +754,9 @@ async function readEverything(root: string): Promise { const resources = await readResourcesDirectory(root); for (const error of resources.errors) reported.push(error.path); + const references = await readReferencesDirectory(root); + for (const error of references.errors) reported.push(error.path); + const channels = await readChannelsDirectory(root); for (const error of channels.errors) reported.push(error.path); @@ -750,6 +803,7 @@ async function readEverything(root: string): Promise { return { workers: workforce.workers.map((worker) => worker.id), documents: resources.documents.map((document) => document.ref), + references: references.documents.map((reference) => reference.ref), channels: channels.channels.map((channel) => channel.id), teams: workforce.teams.map((team) => team.id), resourceModules, diff --git a/packages/workforce/test/references-convention.test.ts b/packages/workforce/test/references-convention.test.ts index 1dffc159bc..3e3f128708 100644 --- a/packages/workforce/test/references-convention.test.ts +++ b/packages/workforce/test/references-convention.test.ts @@ -45,10 +45,13 @@ import { discoverResourceModules } from "../src/codegen/discover-resource-module import { readDeclaredRoster } from "../src/loader/read-declared-roster"; import { referencesFromDocs } from "../src/references-from-docs"; import { resourcesFromDocs } from "../src/resources-from-docs"; -import { clearShadowedReferences } from "../src/clear-shadowed-references"; +import { + clearShadowedReferences, + describeShadowedReferences, +} from "../src/clear-shadowed-references"; import { hireWorkforce } from "../src/hire"; import type { WorkerManifest } from "../src/manifest"; -import { SEAT_REFERENCES_KEY } from "../src/seat-references"; +import { SEAT_REFERENCES_KEY, placeOfReference } from "../src/seat-references"; import { SEAT_RESOURCES_KEY } from "../src/seat-resources"; import { workerConfigSchema } from "../src/worker-config"; @@ -72,6 +75,15 @@ const BOB_NOTES = "teams/engineering/workers/bob/notes"; */ const ENG_SEAT = "engineering.ada"; +/** + * The installing flow, as the migration's storage coordinate needs it. + * + * `isolatesOrgState: false` is the ordinary case and the one the fixtures + * build; the isolated case has its own check below, because it addresses a + * different bucket and getting it wrong is silent. + */ +const INSTALLED_ON = { id: ENG_SEAT, isolatesOrgState: false }; + const roots: string[] = []; /** A workforce tree on disk, read by the real loader. */ @@ -386,7 +398,12 @@ describe("V2b · BR-19 — a row written BEFORE the move shadows the file", () = const { root, stores } = await treeWithHistory(); const { seat: ada, references } = await seatOn(root); - const result = await clearShadowedReferences({ references, orgId: ORG, content: stores.content }); + const result = await clearShadowedReferences({ + references, + orgId: ORG, + content: stores.content, + installedOn: INSTALLED_ON, + }); // Observable, not silent: the migration names the ref it unshadowed and // hands back the body that had been served in the file's place. @@ -395,6 +412,9 @@ describe("V2b · BR-19 — a row written BEFORE the move shadows the file", () = { ref: ORG_HANDBOOK, shadowedContent: "WRITTEN BEFORE THE MOVE" }, ]); + expect(result.dryRun).toBe(false); + expect(describeShadowedReferences(result)).toContain("have been cleared"); + const { ctx } = await ctxFor(ada, stores); expect(await handleFor(ctx, ORG_HANDBOOK)!.readContent()).toBe("ORG HANDBOOK v1"); }); @@ -407,9 +427,18 @@ describe("V2b · BR-19 — a row written BEFORE the move shadows the file", () = references, orgId: ORG, content: stores.content, + installedOn: INSTALLED_ON, dryRun: true, }); expect(result.cleared.map((r) => r.ref)).toEqual([ORG_HANDBOOK]); + // The result says it was a preview, and the sentence an operator reads says + // WOULD. A preview that claims completion is worse than no preview: it is + // read as a finished migration and the stale body stays live. + expect(result.dryRun).toBe(true); + const sentence = describeShadowedReferences(result); + expect(sentence).toContain("dry run"); + expect(sentence).toContain("WOULD be cleared"); + expect(sentence).not.toContain("have been cleared"); const { ctx } = await ctxFor(ada, stores); expect(await handleFor(ctx, ORG_HANDBOOK)!.readContent()).toBe("WRITTEN BEFORE THE MOVE"); @@ -420,11 +449,21 @@ describe("V2b · BR-19 — a row written BEFORE the move shadows the file", () = const { references } = await seatOn(root); const stores = createInMemoryStores(); - const first = await clearShadowedReferences({ references, orgId: ORG, content: stores.content }); + const first = await clearShadowedReferences({ + references, + orgId: ORG, + content: stores.content, + installedOn: INSTALLED_ON, + }); expect(first.cleared).toEqual([]); expect(first.checked.length).toBe(5); - const second = await clearShadowedReferences({ references, orgId: ORG, content: stores.content }); + const second = await clearShadowedReferences({ + references, + orgId: ORG, + content: stores.content, + installedOn: INSTALLED_ON, + }); expect(second.cleared).toEqual([]); }); @@ -434,7 +473,12 @@ describe("V2b · BR-19 — a row written BEFORE the move shadows the file", () = const { ctx, stores } = await ctxFor(ada); await handleFor(ctx, ENG_SCRATCH)!.writeContent("scratch edit"); - await clearShadowedReferences({ references, orgId: ORG, content: stores.content }); + await clearShadowedReferences({ + references, + orgId: ORG, + content: stores.content, + installedOn: INSTALLED_ON, + }); const again = await ctxFor(ada, stores); expect(await handleFor(again.ctx, ENG_SCRATCH)!.readContent()).toBe("scratch edit"); @@ -676,3 +720,181 @@ describe("BR-16 / BR-17 · both folders at one level, each by its own rules", () expect(await handleFor(again.ctx, ENG_HANDBOOK)!.readContent()).toBe("ENGINEERING HANDBOOK v1"); }); }); + +// --------------------------------------------------------------------------- +// The wall's two bypasses, both asserted as the NEGATIVE +// --------------------------------------------------------------------------- + +describe("the wall places a reference by its minted ref, not by its accessor key", () => { + /** + * An app may expose one definition under a second key. The alias is still the + * engineering handbook, but its KEY is a single segment — which reads as an + * org-level ref, and would hand a team's handbook to every seat in the + * company. Asserted as the negative, because a happy-path check would have + * stayed green straight through this. + */ + async function withAlias(seatId: string) { + const root = await tree(); + const documents = resourcesFromDocs((await readResourcesDirectory(root)).documents); + const references = referencesFromDocs((await readReferencesDirectory(root)).documents); + const aliased: DeclaredResources = { + ...references, + // A single-segment alias of a TEAM document — the widening shape. + companyHandbook: references[ENG_HANDBOOK]!, + }; + const hired = hireWorkforce([seat(seatId)], { + kinds: { [KIND]: kindWith({ ...documents, ...aliased }) as never }, + documents, + references: aliased, + }); + return await ctxFor(hired[0]!); + } + + it("RED→GREEN — a sibling team's seat cannot reach an aliased engineering handbook", async () => { + const { ctx } = await withAlias("sales.sam"); + expect(handleFor(ctx, "companyHandbook")).toBeUndefined(); + }); + + it("…and the team's own seat still can, so the wall narrowed rather than broke", async () => { + const { ctx } = await withAlias(ENG_SEAT); + expect(await handleFor(ctx, "companyHandbook")!.readContent()).toBe( + "ENGINEERING HANDBOOK v1", + ); + }); + + it("an entry with no minted ref to place is unreachable, not org-wide", () => { + // The safer reading. A definition this module cannot locate is denied, not + // published to everyone — the direction a permission boundary must fail in. + expect(placeOfEntryIsUnplaceable()).toBe(true); + }); +}); + +/** A hand-built entry carrying no `ref`, as an odd catalog might hold. */ +function placeOfEntryIsUnplaceable(): boolean { + // Exercised through the exported placer: a ref that is not a shape the + // convention mints has no place, and the wall reads "no place" as "no reach". + return placeOfReference("teams/only-two") === undefined; +} + +describe("a reserved path segment is still a legal document name", () => { + // `teams` and `workers` are both reserved segments AND names `validateSegment` + // admits, so `org/references/teams.md` mints the bare ref `teams`. Reading + // that as a broken `teams/` path made an ordinary org document reach nobody. + // Handled as a class — length decides the shape, never the spelling. + it.each(["teams", "workers"])( + "RED→GREEN — an org reference named %s reaches every seat", + async (name) => { + const root = await tree(); + await writeInto(root, { + [`org/references/${name}.md`]: doc(`An org document named ${name}`, `ORG ${name} v1`), + }); + const { seat: ada } = await seatOn(root); + const { ctx } = await ctxFor(ada); + expect(await handleFor(ctx, name)!.readContent()).toBe(`ORG ${name} v1`); + }, + ); + + it("and the structural readings still hold beside them", () => { + expect(placeOfReference("teams/eng/handbook")).toEqual({ team: "eng", worker: undefined }); + expect(placeOfReference("teams/eng/workers")).toEqual({ team: "eng", worker: undefined }); + expect(placeOfReference("teams/eng/workers/ada/notes")).toEqual({ + team: "eng", + worker: "ada", + }); + expect(placeOfReference("workers/ada/notes")).toEqual({ team: undefined, worker: "ada" }); + }); +}); + +describe("a reference cannot isolate itself", () => { + it("`flowIsolation` is refused, which is what keeps the migration's bucket knowable", async () => { + const root = await tree(); + await fs.writeFile( + path.join(root, "org/references/handbook.md"), + "---\ndescription: The org handbook\nflowIsolation: true\n---\nORG HANDBOOK v1", + "utf8", + ); + const read = await readReferencesDirectory(root); + const refused = read.errors.find((e) => e.path === "org/references/handbook.md"); + expect(refused?.kind).toBe("refused-declaration"); + expect(refused?.error.message).toContain("flowIsolation"); + }); + + it("a resources/ document may still declare it — the shared list did not widen", async () => { + const root = await tree(); + await fs.writeFile( + path.join(root, "teams/engineering/resources/scratch.md"), + "---\ndescription: Scratch\nflowIsolation: true\n---\nseed", + "utf8", + ); + const read = await readResourcesDirectory(root); + expect(read.errors).toEqual([]); + expect(read.documents[0]!.declared["flowIsolation"]).toBe(true); + }); +}); + +describe("the migration addresses the right bucket, or refuses", () => { + it("an isolated org scope is a DIFFERENT bucket, and the migration uses it", async () => { + const root = await tree(); + const { references } = await seatOn(root); + const stores = createInMemoryStores(); + const isolated = { id: ENG_SEAT, isolatesOrgState: true }; + + // A row written where an isolated flow would store it. + await stores.content.set("org", `${ORG}:${isolated.id}`, ORG_HANDBOOK, "WRITTEN BEFORE"); + + // Addressed as non-isolated, the row is invisible — the silent-miss shape. + const missed = await clearShadowedReferences({ + references, + orgId: ORG, + content: stores.content, + installedOn: INSTALLED_ON, + }); + expect(missed.cleared).toEqual([]); + + // Told the truth about the flow, it finds and clears it. + const found = await clearShadowedReferences({ + references, + orgId: ORG, + content: stores.content, + installedOn: isolated, + }); + expect(found.scopeId).toBe(`${ORG}:${isolated.id}`); + expect(found.cleared.map((r) => r.ref)).toEqual([ORG_HANDBOOK]); + expect(await stores.content.get("org", `${ORG}:${isolated.id}`, ORG_HANDBOOK)).toBeUndefined(); + }); + + it("refuses loudly for an id the engine would escape, rather than reading the wrong cell", async () => { + const root = await tree(); + const { references } = await seatOn(root); + const stores = createInMemoryStores(); + + await expect( + clearShadowedReferences({ + references, + orgId: "org:weird", + content: stores.content, + installedOn: INSTALLED_ON, + }), + ).rejects.toThrow(/Cannot address/i); + + // The flow id is the other component of an isolated bucket, so it is + // checked too — but only when it is actually part of the key. + await expect( + clearShadowedReferences({ + references, + orgId: ORG, + content: stores.content, + installedOn: { id: "kind:seat", isolatesOrgState: true }, + }), + ).rejects.toThrow(/flow id/i); + + // …and not when it is not: a colon in an id nothing joins is harmless. + const fine = await clearShadowedReferences({ + references, + orgId: ORG, + content: stores.content, + installedOn: { id: "kind:seat", isolatesOrgState: false }, + }); + expect(fine.scopeId).toBe(ORG); + }); +}); From c56c88b2a756c3b19506f6ae923ac133cf4d66a8 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 20 Sep 2026 17:19:41 +0000 Subject: [PATCH 3/4] Document the two document slots from the reader's side MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rewrites `documents-on-disk.md` and the workforce README's document section around what a developer has to decide: which folder a document belongs in, who can change it afterwards, and which seats reach it. The trade-off is stated flat — a reference changes by editing the file and deploying, a resource can change from inside the product — with the ref rule, the reader API and the install call covering both folders once instead of twice. Three corrections where the prose had claimed more than the code does: - The both-slots name collision is not caught by either directory reader; neither sees the other folder. `readDeclaredRoster` collects it in `problems` on the reference layer, and `hireWorkforce` throws for a catalog built by hand. - Automatic reach holds only once the map reaches `hireWorkforce` as `references`. Spreading it into the flow and omitting the hire option leaves every entry an ordinary document, and a seat that declares `references:` is refused. That is the silent one, so it is stated where the install call is made. - Seats are minted only from `teams//workers//`, so a document under `org/workers//references/` sits above no seat and is reachable by nobody. Also fixes a pre-existing broken anchor in `workers-on-disk.md`, which pointed at a heading that no longer exists on the documents page. Docs site builds warning-free: no broken links, no broken anchors. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016qaz1VpRK4wCQzxpjzZ4Hz --- apps/docs/docs/workforce/documents-on-disk.md | 174 ++++++------- apps/docs/docs/workforce/workers-on-disk.md | 12 +- packages/workforce/README.md | 231 ++++++++++++------ 3 files changed, 249 insertions(+), 168 deletions(-) diff --git a/apps/docs/docs/workforce/documents-on-disk.md b/apps/docs/docs/workforce/documents-on-disk.md index 4cad6a0b51..24c65b2ae1 100644 --- a/apps/docs/docs/workforce/documents-on-disk.md +++ b/apps/docs/docs/workforce/documents-on-disk.md @@ -9,14 +9,14 @@ description: "Write a team's shared documents as Markdown files, read the tree a A handbook, a glossary, an escalation procedure: the reference material a team shares and an agent reads. You can declare each one in TypeScript with `defineResource`. You can also write it as a Markdown file and read the folder at startup, which lets someone who does not write TypeScript edit it. -There are two folders, and which one you use decides where the document lives afterwards. +Two folders hold documents, and the one you pick decides where the text lives from then on. -| Folder | The body an agent reads | Who can change it | Who can read it | +| Folder | The body an agent reads | Who can change it | Which seats reach it | |---|---|---|---| -| `references/` | the file on disk, re-read each request | whoever can edit the file in your repo | seats at or below it in the tree | -| `resources/` | the file's body at first boot, then whatever the product last wrote | the file, until something writes it; after that, the product | any seat the flow installed it on | +| `references/` | the file on disk, re-read on each request | whoever edits the file in your repository | every seat at or below the folder it sits in | +| `resources/` | the file's body at first boot, then whatever the product last wrote | the file, until something writes it; the product after that | every seat the flow installed it on | -Put a company handbook in `references/`. Editing the file is the edit, and it keeps reaching agents on the next deploy no matter what else happens. Put an agent's working notes in `resources/`, where they can be written and survive the turn. +Choose by who does the editing. A company handbook goes in `references/`: you change it by editing the file and deploying, and nothing running in the product can overwrite it. An agent's working notes go in `resources/`, where a block or the agent itself can write and the text survives the turn. `readReferencesDirectory` and `readResourcesDirectory` turn the folder tree into plain records. `referencesFromDocs` and `resourcesFromDocs` turn those records into the resource maps you pass to `defineFlow`. @@ -24,7 +24,7 @@ Documents live in the same tree as [workers](./workers-on-disk.md) and [skills]( ## The tree -A `references/` or `resources/` folder is read wherever the tree puts one: at the organization level, in a team, and inside a single worker's own folder. +Either folder is read wherever the tree puts one: at the organization level, in a team, and inside a single worker's own folder. ``` workforce/ @@ -38,24 +38,24 @@ workforce/ resources/ scratch.md workers/ - on-call/ + ada/ + WORKER.md references/ runbook.md - WORKER.md support/ references/ escalation.md ``` -Five documents: one shared across the organization, two belonging to a team, one belonging to a single seat, and one scratchpad the team's agents can write. Every example below reads this tree. +Five documents: a code of conduct everyone reads, a handbook for engineering, a runbook for one seat, an escalation procedure for support, and a scratchpad the engineering agents can write. Every example below reads this tree. -Both folders name documents the same way, and they share one namespace. `references/handbook.md` and `resources/handbook.md` in one team are two spellings of the same name, so the pair is refused at startup rather than one of them quietly winning. +Both folders name documents the same way, and they share one namespace. `references/handbook.md` and `resources/handbook.md` in one team are two spellings of the name `handbook`. `readDeclaredRoster` reports the pair in its `problems`, and `hireWorkforce` throws on it, naming both files. Neither single-folder reader sees the other folder, so neither of them reports it on its own. A worker's own folder is what lets two seats each have a `runbook` without their authors agreeing on a name. Organization-level workers, under `org/workers/`, are read the same way. -A document is a **file**, not a folder. A worker and a skill are each a folder with a fixed file inside it; a document is `.md` sitting directly in `resources/`. A directory in a `resources/` folder is reported rather than passed over, so `resources/handbook/RESOURCE.md` is an error and not a document that quietly went missing. +A document is a **file**, not a folder. A worker and a skill are each a folder with a fixed file inside it; a document is `.md` sitting directly in `references/` or `resources/`. A directory in one of those folders is reported rather than passed over, so `resources/handbook/RESOURCE.md` is an error and not a document that quietly went missing. -A `.ts` file can sit in the same folder. Those are code rather than reference material, read by a build step instead of at startup, and covered in [Capabilities on disk](./capabilities-on-disk.md). They share the folder and the [ref rule](#a-documents-ref) and nothing else, so a `.md` and a `.ts` of one name in one folder are refused rather than one of them winning. The rest of this page is about `.md` files. +A `.ts` file can sit in a `resources/` folder. Those are code rather than reference material, read by a build step instead of at startup, and covered in [Capabilities on disk](./capabilities-on-disk.md). They share the folder and the [ref rule](#a-documents-ref) and nothing else, so a `.md` and a `.ts` of one name in one folder are refused rather than one of them winning. The rest of this page is about `.md` files. ## What a document file says @@ -74,6 +74,8 @@ Escalate anything customer-visible within 15 minutes. `description` is the only key the file itself requires, and a file without one is reported when the tree is read. It reaches the resource with the rest of the frontmatter, and nothing puts it in front of a model, so write it for whoever opens the tree. +Frontmatter never reaches an agent. A reference is served with its `---` block stripped, and a `resources/` document is installed with its parsed body, so what a model reads starts at the Markdown. + The frontmatter is the same dialect a [`WORKER.md`](./workers-on-disk.md#what-a-workermd-says) uses. ### Settings the convention derives @@ -84,19 +86,17 @@ Where a file sits decides its identity, where it is stored, and what its content Everything outside that set arrives at the resource exactly as written, so `llmReadable`, `llmWritable`, `writable`, `allowedExtensions` and `metadata` all take effect from the file. -A file in `references/` derives four more: `writable`, `llmWritable`, `render` and `flowIsolation`. The folder is what makes a reference read-only, so no file has to ask for that and no file can turn it off. Declaring any of the four is refused by name, `writable: false` included — agreeing with the folder is still a second place to keep the same fact, and the next person to open the file cannot tell which copy is the one doing the work. A document that needs to be written belongs in `resources/`. - -`flowIsolation` is refused for a second reason worth saying out loud: a per-seat copy of a document whose body is one file for everybody cannot differ from the file or from itself, and it would put each copy's stored content somewhere different — which is what the migration below has to be able to find. +A file in `references/` derives four more: `writable`, `llmWritable`, `render` and `flowIsolation`. The folder decides all four, and a file that declares any of them is refused by name — `writable: false` included, even though it agrees with the folder. A document that needs to be written belongs in `resources/`. ## A document's ref -A document's ref is its identity. It is built from the folders above the file, joined with slashes. At the organization level it is the bare file name: +A document's ref is its identity. It is built from the folders above the file, joined with slashes, and the rule is the same in both folders. At the organization level it is the bare file name: | Path | Ref | |------|-----| -| `/org/resources/code-of-conduct.md` | `code-of-conduct` | -| `/teams/engineering/resources/handbook.md` | `teams/engineering/handbook` | -| `/teams/engineering/workers/on-call/resources/runbook.md` | `teams/engineering/workers/on-call/runbook` | +| `/org/references/code-of-conduct.md` | `code-of-conduct` | +| `/teams/engineering/resources/scratch.md` | `teams/engineering/scratch` | +| `/teams/engineering/workers/ada/references/runbook.md` | `teams/engineering/workers/ada/runbook` | | `/org/workers/build/resources/runbook.md` | `workers/build/runbook` | Each qualifier means a name only has to be unique where it sits. Every team can have a `handbook`, and every seat a `runbook`, without checking what anyone else called theirs. @@ -113,12 +113,13 @@ A worker folder's documents load whether or not the folder holds a `WORKER.md`. ## Reading the tree -Point `readResourcesDirectory` at the root: +Point a reader at the root. The two take the same argument and return the same shape: ```ts -import { readResourcesDirectory } from "@flow-state-dev/workforce/loader"; +import { readReferencesDirectory, readResourcesDirectory } from "@flow-state-dev/workforce/loader"; -const { documents, errors } = await readResourcesDirectory("./workforce"); +const references = await readReferencesDirectory("./workforce"); +const resources = await readResourcesDirectory("./workforce"); ``` You get one record per document: @@ -128,16 +129,14 @@ interface ResourceDoc { ref: string; // "teams/engineering/handbook" declared: Record; // the frontmatter, exactly as written body: string; // the Markdown below it, verbatim - filePath?: string; // where it was read from — references only + filePath?: string; // the absolute path it was read from } ``` -`readReferencesDirectory` has the same signature and returns the same records, each carrying the `filePath` its content is served from. - For the handbook above: ```ts -const handbook = documents.find((doc) => doc.ref === "teams/engineering/handbook")!; +const handbook = references.documents.find((doc) => doc.ref === "teams/engineering/handbook")!; handbook.declared; // { description: "How the engineering team works. On-call, review, escalation.", @@ -146,16 +145,18 @@ handbook.body; // "# Engineering handbook\n\nEscalate anything customer-visible within 15 minutes.\n" ``` +A reference is served from `filePath`, so a record you build by hand has to set it to the document's absolute path; `referencesFromDocs` throws on a record without one. A `resources/` record carries the path too, and nothing reads it there — `body` is that document's source. + Reading the tree builds nothing. No resource is defined, nothing is stored, and no flow is touched. Turning records into resources is a separate call. -The subpath matters. `@flow-state-dev/workforce/loader` imports `node:fs`, so it only runs on Node. The package root, where `resourcesFromDocs` lives, stays isomorphic. +The subpath matters. `@flow-state-dev/workforce/loader` imports `node:fs`, so it only runs on Node. The package root, where `resourcesFromDocs` and `referencesFromDocs` live, stays isomorphic. ### When a file is wrong Anything that should have produced a document and did not lands in `errors`, and the rest of the documents load anyway. Say someone wrote the marketing handbook as a folder: ```ts -errors; +resources.errors; // [{ kind: "folder-where-file-belongs", // path: "teams/marketing/resources/handbook", // error: Error('"handbook" is a directory. A resource is a file, not a folder — write @@ -169,11 +170,11 @@ errors; | `document-load-failed` | One file did not load: a name that breaks the rules, a symlink, an unreadable file, no frontmatter, or a missing `description`. | | `folder-where-file-belongs` | A directory sits where a document file belongs. | | `refused-declaration` | The file declares a setting the convention derives, or `prefetchMode: "lazy"`. | -| `unreadable-slot` | A structural folder is a symlink or exists and cannot be listed: `org`, `org/resources`, `teams`, a team folder, or a team's `resources`. The documents beneath it cannot be enumerated, so the folder is reported under its own path. | +| `unreadable-slot` | A structural folder is a symlink or exists and cannot be listed: `org`, `teams`, a team folder, a `workers/` level, or either document folder inside any of them. The documents beneath it cannot be enumerated, so the folder is reported under its own path. | -`readResourcesDirectory` throws only about the root you passed: when it cannot be read at all, and when it is a symlink. Links are never followed at any level of the walk, and the root is no exception, so a linked root is refused rather than read from wherever it points. A root with neither `org/` nor `teams/` comes back as `{ documents: [], errors: [] }`, and a team with no `resources/` folder is not an error either. +Either reader throws only about the root you passed: when it cannot be read at all, and when it is a symlink. Links are never followed at any level of the walk, and the root is no exception, so a linked root is refused rather than read from wherever it points. A root with neither `org/` nor `teams/` comes back as `{ documents: [], errors: [] }`, and a team with no document folder is not an error either. -A file in a `resources/` folder that is not a `.md` is passed over in silence by this reader, as are OS and editor droppings such as `.DS_Store`. A `.ts` file is passed over here too, and picked up by the other door: [`fsdev gen`](./code-on-disk.md) writes it onto the generated file, and [Capabilities on disk](./capabilities-on-disk.md) covers what it then gives a worker. +A file that is not a `.md` is passed over in silence, as are OS and editor droppings such as `.DS_Store`. A `.ts` file in `resources/` is passed over here too, and picked up by the other door: [`fsdev gen`](./code-on-disk.md) writes it onto the generated file, and [Capabilities on disk](./capabilities-on-disk.md) covers what it then gives a worker. #### Treat a non-empty `errors` as fatal @@ -191,59 +192,44 @@ A reported file is a document your app was supposed to have. Log a warning and c ## Installing the documents -`resourcesFromDocs` turns the records into a resource map keyed by ref. Spread it into the flow's own map: +`resourcesFromDocs` and `referencesFromDocs` each turn records into a resource map keyed by ref. Spread both into the flow's own map: ```ts import { defineFlow } from "@flow-state-dev/core"; -import { resourcesFromDocs } from "@flow-state-dev/workforce"; -import { readResourcesDirectory } from "@flow-state-dev/workforce/loader"; +import { referencesFromDocs, resourcesFromDocs } from "@flow-state-dev/workforce"; +import { readReferencesDirectory, readResourcesDirectory } from "@flow-state-dev/workforce/loader"; import { answerQuestion } from "./blocks"; import { ticketResource } from "./resources"; -const { documents, errors } = await readResourcesDirectory("./workforce"); -if (errors.length) throw new Error(`resources: ${errors.length} document(s) failed to load`); - -export const supportFlow = defineFlow({ - kind: "support", - actions: { answer: { block: answerQuestion } }, - resources: { ticket: ticketResource, ...resourcesFromDocs(documents) }, -}); -``` - -Every file-declared document is installed at `org` scope, with the file's body as the resource content. - -References install the same way, through `referencesFromDocs`, and both maps spread into the same flow: - -```ts const references = await readReferencesDirectory("./workforce"); -const documents = await readResourcesDirectory("./workforce"); +const resources = await readResourcesDirectory("./workforce"); +for (const { errors } of [references, resources]) { + if (errors.length) throw new Error(`workforce: ${errors.length} document(s) failed to load`); +} + +const documentMap = resourcesFromDocs(resources.documents); +const referenceMap = referencesFromDocs(references.documents); export const supportFlow = defineFlow({ kind: "support", actions: { answer: { block: answerQuestion } }, - resources: { - ticket: ticketResource, - ...resourcesFromDocs(documents.documents), - ...referencesFromDocs(references.documents), - }, + resources: { ticket: ticketResource, ...documentMap, ...referenceMap }, }); ``` -Pass both to `hireWorkforce` as well, so it knows which entries are references and can hold each seat to its place in the tree: +Every file-declared document is installed at `org` scope. A `resources/` entry carries the file's body as its starting content; a `references/` entry points at the file itself, which is read whenever an execution context is built. Editing a reference in your repository reaches agents on the next request, and a read already in flight keeps the body it started with. + +Pass both maps to `hireWorkforce` as well, under `documents` and `references`: ```ts -hireWorkforce(workers, { - kinds, - documents: resourcesFromDocs(documents.documents), - references: referencesFromDocs(references.documents), -}); +hireWorkforce(workers, { kinds, documents: documentMap, references: referenceMap }); ``` -A reference's content is read from its file whenever an execution context is built, so editing the file in your repository reaches agents on the next request. A read already in flight keeps the body it started with. +That is what tells the hire which entries on a kind's map are references, which is what [the tree wall](#who-reaches-what) narrows. Leave `references` out and no entry is treated as one: every seat reaches every reference the kind installed, and a seat whose `WORKER.md` names any is refused. A ref passed in both maps is refused too, naming it. -Spread the map rather than passing it on its own. A flow copy created with `supportFlow({ resources })` *replaces* the definition's map instead of merging with it, so a copy handed only `resourcesFromDocs(documents)` loses whatever the flow kind declared. +Spread the maps rather than passing one on its own. A flow copy created with `supportFlow({ resources })` *replaces* the definition's map instead of merging with it, so a copy handed only `resourcesFromDocs(documents)` loses whatever the flow kind declared. -`resourcesFromDocs` throws rather than collecting. A record it cannot turn into a resource stops startup, naming the ref. +Both functions throw rather than collecting. A record that cannot become a resource stops startup, naming the ref. ### The request needs an org @@ -256,24 +242,26 @@ await ctx.resources.get("teams/engineering/handbook").readContent(); // Error: Resource "teams/engineering/handbook" is not registered ``` -Put `requireOrg: true` on the blocks that read a document, as the example in the next section does. The flow then turns away a request with no org up front, instead of running it and coming up empty. How a request carries its org is covered in [the client reference](/docs/configuration/client). +Put `requireOrg: true` on the blocks that read a document, as the example in [Letting a model read a document](#letting-a-model-read-a-document) does. The flow then turns away a request with no org up front, instead of running it and coming up empty. How a request carries its org is covered in [the client reference](/docs/configuration/client). -### Who reaches what +## Who reaches what **For a `references/` document, the folder is the boundary.** A seat reaches the references at or above its own place in the tree: the organization's, its own team's, and its own folder's. Not another team's, and not a teammate's folder. You write no filter, and there is no setting to get wrong. ``` teams/engineering/workers/ada/ a seat here reads… - org/references/code-of-conduct yes — the organization is above everyone - teams/engineering/references/handbook yes — its own team + org/references/code-of-conduct yes — the organization is above everyone + teams/engineering/references/handbook yes — its own team teams/engineering/workers/ada/references/runbook yes — its own folder teams/engineering/workers/ivan/references/runbook no — a teammate's folder teams/support/references/escalation no — another team ``` -To let a reference reach more people, move the file up the tree. That is the only way to widen it, which is what makes the boundary worth trusting: nothing on the install side can quietly open it back up. +An unreachable reference is not an empty read. The accessor is not on the seat's map at all, so `ctx.resources.get("teams/support/escalation")` throws `is not registered`. + +To let a reference reach more people, move the file up the tree. Nothing on the install side widens it. -A seat that wants less than its place gives it lists what it wants in its own file. The list narrows and never widens, so naming a document the seat could not already reach is refused at startup rather than granted: +A seat can ask for less than its place gives it, by listing what it wants in its own file. The list narrows and never widens, so naming a document the seat could not already reach refuses the whole roster at the hire, naming the seat and the ref: ```md --- @@ -286,23 +274,25 @@ references: Leaving the key out means every reference at or above the seat. Writing `references: []` means none — the two are different answers, not the same one. +Seats are read from `teams//workers//`, so a reference under `org/workers//references/` sits beside the organization level rather than above any seat, and no seat reaches it. + **For a `resources/` document, the folder is a namespace and nothing more.** Every one is org-scoped, and a generator's resource tools reach every installed document marked `llmReadable`. Install a whole tree on one flow and every team's writable documents are reachable from it, one team from another included. So if you want a team's writable documents kept to that team, filter the records before installing them: ```ts const engineering = resourcesFromDocs( - documents.filter((doc) => doc.ref.startsWith("teams/engineering/")), + resources.documents.filter((doc) => doc.ref.startsWith("teams/engineering/")), ); ``` -The same holds one level down. Putting a `resources/` document under `workers/on-call/` addresses it to that seat. It does not keep it from the others. Seats hired into one kind share that kind's flow definition, so by default every one of them reads the same row. +The same holds one level down. Putting a `resources/` document under `workers/ada/` addresses it to that seat. It does not keep it from the others. Seats hired into one kind share that kind's flow definition, so by default every one of them reads the same row. -Filtering decides what a whole kind installs. To narrow one seat within a kind, the seat's own file names the documents it may touch, and can take one read-only: see [what a `WORKER.md` says](./workers-on-disk.md#what-a-workermd-says). A seat naming a document its kind was not installed with is refused at the hire, so the filter holds. +Filtering decides what a whole kind installs. To narrow one seat within a kind, the seat's own file names the documents it may touch under `resources:`, and can take one read-only: see [what a `WORKER.md` says](./workers-on-disk.md#what-a-workermd-says). A seat naming a document its kind was not installed with is refused at the hire, so the filter holds. ### Giving one seat a document of its own -A document whose frontmatter carries `flowIsolation: true` gets one row per seat: +A `resources/` document whose frontmatter carries `flowIsolation: true` gets one row per seat: ```md --- @@ -313,7 +303,7 @@ flowIsolation: true The seat that writes it reads it back. A sibling seat asking for the same document gets its own empty copy, not an error and not the first seat's copy. -That line is the boundary, not the folder it sits in. A document in a worker's folder without it is shared across every seat of the kind, the same as a team's handbook. +That line is the boundary, not the folder it sits in. A `resources/` document in a worker's folder without it is shared across every seat of the kind, the same as a team's scratchpad. A `references/` file cannot declare it; its boundary is where the file sits. ### Letting a model read a document @@ -335,32 +325,46 @@ export const answerQuestion = generator({ The documents are already declared on the flow, so the generator does not declare them again. The tool addresses a document by its scope-qualified uri, the same handle the [search tools](/docs/resources/searching) return. See [LLM access patterns](/docs/resources/overview#llm-access-patterns). +A reference is read-only to code and to the model. `writeContent()` on one throws a `FlowError` with code `resource_read_only` and the message `Resource "handbook" content is read-only`, and the write tool is never offered for it, whatever tools the generator carries. + ## Moving a document into `references/` -Moving the file is usually the whole job. One case needs a second step, and it is quiet enough to be worth knowing about: if anything ever wrote that document while it was in `resources/`, the written body is still stored, and a stored body is what an agent reads. Move the file and the agent keeps reading the old write — the file looks authoritative in your repository and reaches nobody. +Moving the file is usually the whole job. One case needs a second step: if anything ever wrote that document while it was in `resources/`, the written body is still stored, and a stored body wins over the file. Move the file and agents keep reading the old write, while the file in your repository looks authoritative and reaches nobody. -`clearShadowedReferences` finds those and clears them, so the file is the source again: +`clearShadowedReferences` finds those rows and clears them, so the file is the source again: ```ts import { clearShadowedReferences, describeShadowedReferences } from "@flow-state-dev/workforce"; const result = await clearShadowedReferences({ - references: referencesFromDocs(references.documents), + references: referenceMap, orgId, content: stores.content, installedOn: { id: flow.id, isolatesOrgState: false }, }); console.log(describeShadowedReferences(result)); // references: 1 of 4 were shadowed by a stored write and have been cleared — handbook. +// Each now serves its file again. +``` + +The result carries what it found: + +```ts +interface ClearShadowedReferencesResult { + cleared: { ref: string; shadowedContent: string }[]; // each row, and the body it had been serving + checked: string[]; // every ref looked at + dryRun: boolean; + scopeId: string; // the bucket the rows were addressed in +} ``` -`installedOn` describes the flow the references are installed on, and it is required rather than assumed. A flow that isolates its organization scope keeps each instance's stored content in its own place, so a run told the wrong thing would look in the wrong place, find nothing, and report success — the one outcome worse than not running it. Read `isolatesOrgState` off the flow rather than guessing. +Once the row is gone, `shadowedContent` is the only copy of that written body, so log it or keep it before deciding the clear was right. Pass `dryRun: true` to get the same finding with nothing deleted; the result says `dryRun: true`, and `describeShadowedReferences` reports what `WOULD be cleared`. -It reports before it is believed: `result.cleared` names each reference and carries the body that had been served in the file's place, which is the last moment that text exists anywhere. Pass `dryRun: true` to see the finding and change nothing — the result says it was a preview, and the sentence says "would". +`installedOn` describes the flow the references are installed on. Read `isolatesOrgState` off the flow rather than guessing: a flow that isolates its organization scope stores content under a different address, and a run told the wrong thing looks in the wrong place and reports nothing to clear. Run it once per organization after the move. It is safe to run again — a tree whose files are already the source reports nothing cleared — and it never touches a `resources/` document. -It refuses, loudly, for an organization or flow id containing `:` or a backslash. Those need the escaping the engine applies when it builds a storage address, and this cannot reproduce that rule without keeping a second copy of it — so it stops rather than reading the wrong place and telling you there was nothing to clear. +It throws for an organization or flow id containing `:` or a backslash. Those ids need the escaping the engine applies when it builds a storage address; clear those rows with the engine's own store helpers instead. ## What stays in TypeScript @@ -372,8 +376,8 @@ A document that needs a state schema of its own, a `render` function, [reactive - It does not watch the tree. Read it once, at startup. - It does not follow symlinks, at any level of the walk. -- It does not install anything. `readResourcesDirectory` hands back records, `resourcesFromDocs` hands back a map, and you put the map on your flow. -- It does not read `WORKER.md` or any `skills/` folder. Those are [Workers on disk](./workers-on-disk.md) and [Skills](../skills/overview.md). It reads a worker folder only to find a `resources/` folder inside it. -- It does not put a document on an agent's bash mount. A mount carries collections; a document is a single resource, so a shell on the sandbox will not find the handbook as a file. -- It does not watch a reference for changes within a request. The file is read when an execution context is built, so an edit reaches the next request rather than a read already running. +- It does not install anything. The readers hand back records, `resourcesFromDocs` and `referencesFromDocs` hand back maps, and you put the maps on your flow. +- It does not read `WORKER.md` or any `skills/` folder. Those are [Workers on disk](./workers-on-disk.md) and [Skills](../skills/overview.md). It reads a worker folder only to find a document folder inside it. +- It does not put a document on an agent's bash mount. A mount carries collections; a document is a single resource, so a shell on the sandbox will not find the handbook as a file. That holds for a reference too, even though its content is a file on your disk. +- It does not re-read a reference within a request. The file is read when an execution context is built, so an edit reaches the next request rather than a read already running. - For a `resources/` document, it does not decide which seat may read which one. `flowIsolation` gives each seat its own copy; nothing there refuses a read. A `references/` document is the exception, and [Who reaches what](#who-reaches-what) is the rule. diff --git a/apps/docs/docs/workforce/workers-on-disk.md b/apps/docs/docs/workforce/workers-on-disk.md index b96302a36a..976902eabb 100644 --- a/apps/docs/docs/workforce/workers-on-disk.md +++ b/apps/docs/docs/workforce/workers-on-disk.md @@ -62,6 +62,8 @@ request into tasks, assign them, and report what came back. Only documents are narrowed. The stores and boards its kind declares stay reachable and writable whatever the list says, and so do the resources the kind's own blocks declare. Where the documents come from is [below](#supplying-the-documents-a-seat-may-name). +`references:` is the companion list, over the documents in [`references/` folders](./documents-on-disk.md#who-reaches-what). Each entry is a ref on its own, with no mode, because nothing writes a reference. The list narrows what the worker's place in the tree already gives it — the organization's references, its own team's, and its own folder's — and naming one from outside that refuses the hire. Leave the key out and the worker reaches all of them; `references: []` is how you say it gets none. + Reading the file checks no other key. Whatever else you write lands on the record spelled exactly as you spelled it. The flow a worker names has the final say: at hiring it [refuses a setting it never declared](#the-flow-decides-what-a-worker-may-declare). The frontmatter is the same dialect a [`SKILL.md`](../skills/overview.md) uses. If you have written one of those, you already know the shape. @@ -116,9 +118,9 @@ configuration, so a worker on another team never sees them. Documents under `teams//resources/` work the other way. They are installed on a worker *kind*, so every worker of that kind can read every team's documents. The folder addresses a document; it does not fence it. To give one team's workers only its own documents, filter the -records before installing them: [A folder is a namespace, not a visibility -boundary](./documents-on-disk.md#a-folder-is-a-namespace-not-a-visibility-boundary) shows how, and -covers a worker's own folder too. +records before installing them: [Who reaches what](./documents-on-disk.md#who-reaches-what) shows +how, and covers a worker's own folder too. A document under `teams//references/` is the other +case: that folder does fence it, to that team's workers. ## A worker's identity @@ -404,6 +406,8 @@ One catalog, spread into the flow and handed to the hire. Which entries in that `documents` is consulted only for a worker that declares `resources:`. A roster where none does hires the same whether you pass it or not. A worker that does declare one while `documents` is absent is refused, naming what is missing. +References go the same way, under a `references` option holding the map `referencesFromDocs` returned. That option is what marks those entries as references, and the tree wall applies only to entries it names: leave it out and every worker reaches every reference its kind installed. [Installing the documents](./documents-on-disk.md#installing-the-documents) has the call with both maps. + A list the hire step cannot resolve refuses the whole roster, naming the worker and what was wrong. The message calls one entry a grant and quotes the ref as the file spelled it, which here is `handbook` with two letters swapped: ``` @@ -426,6 +430,8 @@ A record is refused when it: - declares `instructions:` and carries a body; - declares `persona:`, `seatSkills:`, `seatTools:` or `teamInstructions:`, none of which is a setting a worker declares; - declares `teamInstructions:` in its frontmatter, wherever that frontmatter came from — a team's instructions come from its [`TEAM.md`](#what-a-teammd-says) body, read by the loader; +- declares a `references:` list the hire step cannot resolve: a `references:` that is not a list, an entry that is not a ref, the same ref twice, or a ref naming a reference this worker cannot reach from where its folder sits — which includes every ref when no `references` map was passed. A worker whose id names no place in the tree is refused too, once its kind holds references; +- reaches a reference it did not name, the same way a document can come back through one of its kind's blocks; - declares a `resources:` list the hire step cannot resolve: a `resources:` that is not a list at all, an entry that is neither a ref nor a one-key `ref: mode` mapping, a ref no document matches, a ref naming a document the app declared but did not install on this worker's kind, a mode that is neither `ro` nor `rw`, the same ref twice, `rw` on a document whose own frontmatter says `writable: false`, a ref colliding with a name the kind's own blocks declare, a ref the kind does declare at flow level while what it holds there is not that document, or the key at all when no `documents` were passed; - reaches a document it never named, because one of its kind's blocks declares that document and a block's declaration merges back in after the worker's narrowed list is applied. Keep the document at flow level and let the block reach it there, or grant it to the worker deliberately; - shares an id with another record in the same call, which is two workers claiming one address. diff --git a/packages/workforce/README.md b/packages/workforce/README.md index 88edea74c7..9bdd8746f1 100644 --- a/packages/workforce/README.md +++ b/packages/workforce/README.md @@ -262,6 +262,7 @@ const seats = hireWorkforce(roster.workers); | `workers` | One `WorkerManifest` per worker, each carrying its own resolved skills. The records `hireWorkforce` takes. | | `teams` | One `TeamManifest` per team that wrote a [`TEAM.md`](#the-optional-team-file). A team without one is absent, not present-and-empty. | | `documents` | One `ResourceDoc` per [document](#reading-documents-from-files), from every `resources/` folder the convention reads. | +| `references` | One `ResourceDoc` per [reference](#reading-documents-from-files), from every `references/` folder, each carrying the `filePath` it was read from. | | `channels` | One `ChannelManifest` per [channel](#declaring-channels-in-files) under `teams//channels/`. There is no `org/channels/` level, the way there is for documents. | | `problems` | Everything that did not load. Empty for a tree that loads cleanly. | @@ -298,8 +299,9 @@ roster.problems; // error: Error('CHANNEL.md in "standup/" must declare a non-empty `description`') }] ``` -`layer` is one of `worker`, `skill`, `team`, `document` or `channel`, and the entries arrive in that -order: worker slots, then each seat's skills, then team files, then documents, then channels. `path` +`layer` is one of `worker`, `skill`, `team`, `document`, `reference` or `channel`, and the entries +arrive in that order: worker slots, then each seat's skills, then team files, then documents, then +references, then channels, and last a `reference` entry per basename claimed in both slots. `path` is the path that failed, relative to the root. `error` is the original `Error`, `cause` chain intact. @@ -431,6 +433,10 @@ block reach it from there, or grant it to the seat deliberately. `description`, so a kind that declares a `resources` setting of its own does not receive one from a file. +`references:` is the sibling key, over the documents in `references/` folders. It narrows within a +wall the tree already imposes rather than selecting from everything the kind installed, and its +entries take no mode — see [What a seat reaches](#what-a-seat-reaches). + ### What a hireable kind must admit A worker kind is an ordinary flow. What makes it *hireable* is that its `configSchema` composes @@ -744,33 +750,34 @@ A team's shared documents — a handbook, a glossary, an escalation procedure instead of `defineResource` stanzas. Frontmatter is settings and the body is the document, the same bargain `WORKER.md` makes. -**Two slots, one namespace.** The convention reads `references/` and `resources/` at the same four -levels, and mints refs for both by the same rule — so one basename claimed in both at one level is -refused rather than resolved. What differs is what the document IS afterwards: +**Two folders, one namespace.** The convention reads `references/` and `resources/` at the same four +levels and mints refs for both by the same rule, so one basename claimed in both at one level is +refused rather than resolved. What differs is what the document is once installed: -| Slot | Content an agent reads | Writable | Reach | +| Folder | Content an agent reads | Writable | Reach | | --- | --- | --- | --- | -| `references/` | the file, re-read on each execution context | no — `writable`, `llmWritable`, `render` and `flowIsolation` are derived, and a file declaring any of them is refused | derived from the tree: the org's, the seat's own team's, and the seat's own folder | +| `references/` | the file, re-read whenever an execution context is built | no. `writable`, `llmWritable`, `render` and `flowIsolation` come from the folder, and a file declaring any of them is refused | the org's, the seat's own team's, and the seat's own folder's — derived from the tree | | `resources/` | the file's body seeds a row; the row is the source from then on | yes, unless the file says otherwise | every document the flow was installed with | -Put standing material an author owns in the repository under `references/`; put anything an agent -writes under `resources/`. The `resources/` half is unchanged — same reader, same records, same -install, same reach. +A reference changes when someone edits the file and deploys. A `resources/` document can change from +inside the product. Put standing material an author owns under `references/`, and anything an agent +writes under `resources/`. -**A resource is a file, not a folder.** A document is `.md` directly in `resources/`, unlike -a worker or a skill, which is a folder with a fixed file inside it. A directory in a `resources/` -slot lands in `errors` rather than being passed over. +**A document is a file, not a folder.** It is `.md` directly in the slot, unlike a worker or a +skill, which is a folder with a fixed file inside it. A directory in either slot lands in `errors` +rather than being passed over. -Four places are read — the org level, a team, and a worker's own folder under either of those: +Four places are read, in both slots — the org level, a team, and a worker's own folder under either +of those: | Path | Ref | |------|-----| -| `/org/resources/.md` | `` | -| `/teams//resources/.md` | `teams//` | -| `/teams//workers//resources/.md` | `teams//workers//` | -| `/org/workers//resources/.md` | `workers//` | +| `/org//.md` | `` | +| `/teams///.md` | `teams//` | +| `/teams//workers///.md` | `teams//workers//` | +| `/org/workers///.md` | `workers//` | -A worker's `resources/` folder is how two seats each get their own `runbook` without their authors +A worker's own folder is how two seats each get their own `runbook` without their authors coordinating a name. The ref drops `org/` for an org worker, exactly as an org document's does. ```md @@ -786,50 +793,49 @@ Escalate anything customer-visible within 15 minutes. `readResourcesDirectory` walks all four and returns one record per document; `resourcesFromDocs` turns those records into the resource map you already pass to a flow. `readReferencesDirectory` and -`referencesFromDocs` are the same pair over `references/`, and both maps spread into one flow. - -A reference's record also carries the `filePath` it was read from — that path is what the install -half points the resource at, and it is why editing the file reaches agents on the next request. -Pass both maps to `hireWorkforce` (`documents` and `references`) so it can hold each seat to the -references at or above its place in the tree. - -Moving a file from `resources/` into `references/` needs one extra step when anything ever wrote -that document: the written body is still stored, and a stored body wins over the file. -`clearShadowedReferences({ references, orgId, content, installedOn })` finds those rows, reports -each one with the body it had been serving, and clears them. It is idempotent, takes `dryRun` (and -says so on the result, so the log sentence reads "would clear"), and never touches a `resources/` -document. - -`installedOn` is `{ id, isolatesOrgState }` for the flow the references are installed on, and it is -required rather than defaulted: a flow that isolates its org scope stores content under a different -address, so a wrong assumption here looks in the wrong place, finds nothing, and reports success. -For an org or flow id containing `:` or a backslash it throws instead of guessing — that address -needs the engine's own escaping, and a second copy of that rule here is how the two would drift. +`referencesFromDocs` are the same pair over `references/`. Both maps spread into one flow, and both +are passed to `hireWorkforce`: ```ts import { defineFlow } from "@flow-state-dev/core"; -import { resourcesFromDocs } from "@flow-state-dev/workforce"; -import { readResourcesDirectory } from "@flow-state-dev/workforce/loader"; +import { hireWorkforce, referencesFromDocs, resourcesFromDocs } from "@flow-state-dev/workforce"; +import { readReferencesDirectory, readResourcesDirectory } from "@flow-state-dev/workforce/loader"; import { answerQuestion } from "./blocks"; import { ticketResource } from "./resources"; -const { documents, errors } = await readResourcesDirectory("./workforce"); -if (errors.length) throw new Error(`resources: ${errors.length} document(s) failed to load`); +const references = await readReferencesDirectory("./workforce"); +const resources = await readResourcesDirectory("./workforce"); +for (const { errors } of [references, resources]) { + if (errors.length) throw new Error(`workforce: ${errors.length} document(s) failed to load`); +} + +const documents = resourcesFromDocs(resources.documents); +const referenceMap = referencesFromDocs(references.documents); export const supportFlow = defineFlow({ kind: "support", actions: { answer: { block: answerQuestion } }, - resources: { ticket: ticketResource, ...resourcesFromDocs(documents) }, + resources: { ticket: ticketResource, ...documents, ...referenceMap }, +}); + +const seats = hireWorkforce(workers, { + kinds: { support: supportFlow }, + documents, + references: referenceMap, }); ``` +`references` is what tells the hire which entries on a kind's map are references. Leave it out and +none of them is: every seat reaches every reference the kind installed, and a seat whose file +declares `references:` is refused. A ref passed in both maps is refused as well, naming it. + **The flow needs an org identity, and it will not ask for one on its own.** Every file-declared document is org-scoped, and a flow collects `requiresOrg` from its blocks, not from its resource -map. So the flow above accepts a request carrying only a `userId`, builds no org resource registry, -and every document is then missing: `ctx.resources.get("teams/engineering/handbook")` throws -`is not registered`, and `readResourceContentTool` says the same. Declare `requireOrg: true` on the -blocks that read a document, and a request without an org is refused at the door instead of arriving -empty: +map. So a flow that installs documents and declares nothing accepts a request carrying only a +`userId`, builds no org resource registry, and every document is then missing: +`ctx.resources.get("teams/engineering/handbook")` throws `is not registered`, and +`readResourceContentTool` says the same. Declare `requireOrg: true` on the blocks that read a +document, and a request without an org is refused at the door instead of arriving empty: ```ts // ./blocks.ts @@ -850,39 +856,51 @@ Each record is plain data: |-------|-------------| | `ref` | The document's identity and storage key, taken from where the file sits — see the table above. It is also the accessor key, so a team's handbook is `ctx.resources["teams/engineering/handbook"]` and one seat's runbook is `ctx.resources["teams/pentest/workers/recon/runbook"]`. | | `declared` | The frontmatter exactly as written. A file that declares one of the refused settings below produces no record at all, so nothing is stripped here. | -| `body` | The Markdown below the frontmatter, verbatim. It becomes the resource's content. | +| `body` | The Markdown below the frontmatter, verbatim. For a `resources/` document it becomes the resource's content. | +| `filePath` | The absolute path the file was read from. A reference is served from it, so a hand-built reference record must set it and `referencesFromDocs` throws without it. Carried on a `resources/` record too, where nothing reads it. | **Merge the map yourself.** A flow copy created with `supportFlow({ resources })` *replaces* the definition's map rather than merging with it, so passing `resourcesFromDocs(documents)` there on its own drops whatever resources the flow kind declared. Spread it into your own map, as above. -**A folder is a namespace, not a visibility boundary — at every level.** Every file-declared -document is org-scoped, and a flow's resource tools reach every installed document marked -`llmReadable` with no per-team filter. Installing a whole tree on one flow makes every team's +### What a seat reaches + +**A reference is walled by where its file sits.** A seat reaches the org's references, its own +team's, and its own folder's. Another team's and a teammate's are not on its map at all, so +`ctx.resources.get` on one throws `is not registered`. No install-side filter is involved and there +is no setting that widens the wall: a reference reaches more people by moving up the tree. Seats are +read from `teams//workers//`, so a reference under `org/workers//references/` +sits beside the org level rather than above any seat, and no seat reaches it. + +A `references:` list in a `WORKER.md` narrows within that wall. Each entry is a ref on its own — +there is no mode, since nothing writes a reference. An entry naming a reference the seat could not +already reach refuses the whole roster, and so does a malformed list or a duplicated ref. Leaving +the key out means every reference at or above the seat; `references: []` means none. + +**A `resources/` folder is a namespace, not a visibility boundary — at every level.** Every +file-declared document is org-scoped, and a flow's resource tools reach every installed document +marked `llmReadable` with no per-team filter. Installing a whole tree on one flow makes every team's documents reachable from it. To give a team's seats only its own, filter the records before installing: ```ts const engineering = resourcesFromDocs( - documents.filter((d) => d.ref.startsWith("teams/engineering/")), + resources.documents.filter((d) => d.ref.startsWith("teams/engineering/")), ); ``` -This holds for a worker's folder too, and it is the thing most worth being clear about: putting a -document under `workers/recon/` addresses it to that seat, it does not keep it from the others. Every -seat hired into one kind shares that kind's flow definition, so by default all of them read the same -row. +This holds for a worker's folder too: putting a `resources/` document under `workers/recon/` +addresses it to that seat, it does not keep it from the others. Every seat hired into one kind +shares that kind's flow definition, so by default all of them read the same row. **Filtering decides what a kind installs; a seat's own file decides what that seat reaches.** A `resources:` list in a `WORKER.md` narrows one seat within a kind and can take a document read-only — see [The documents a seat may touch](#the-documents-a-seat-may-touch). A ref for a -document this kind was not installed with is refused at the hire, so the filter above holds. Pass -the map this section builds to `hireWorkforce` as `documents`, so a ref in that list has something -to match. +document this kind was not installed with is refused at the hire, so the filter above holds. -**To make a document that seat's alone, say so in the document.** A file whose frontmatter carries -`flowIsolation: true` gets one row per seat: the seat that writes it reads it back, and a sibling -seat asking for the same document gets its own empty copy rather than an error. +**To make a `resources/` document that seat's alone, say so in the document.** A file whose +frontmatter carries `flowIsolation: true` gets one row per seat: the seat that writes it reads it +back, and a sibling seat asking for the same document gets its own empty copy rather than an error. ```md --- @@ -891,16 +909,49 @@ flowIsolation: true --- ``` -Both sentences are owed together. A worker folder without that line is an address, not a boundary. +A worker folder without that line is an address, not a boundary. It is a `resources/` setting: a +`references/` file declaring it is refused, and a reference's boundary is where its file sits. + +### Moving a document from `resources/` to `references/` + +Moving the file is the whole job unless something wrote that document while it lived in +`resources/`. The written body is still stored, and a stored body wins over the file, so agents keep +reading the old write. + +```ts +import { clearShadowedReferences, describeShadowedReferences } from "@flow-state-dev/workforce"; + +const result = await clearShadowedReferences({ + references: referenceMap, + orgId, + content: stores.content, + installedOn: { id: flow.id, isolatesOrgState: false }, + // dryRun: true, +}); +console.log(describeShadowedReferences(result)); +// references: 1 of 4 were shadowed by a stored write and have been cleared — handbook. +// Each now serves its file again. +``` -This is a `resources/` setting. A `references/` file that declares it is refused: a per-seat copy of -a document whose body is one file for everyone cannot differ from the file or from itself, and the -copies would each store their content somewhere different — which is the address the reference -migration has to be able to find. A reference is walled by where it sits, not by a frontmatter key. +The result is `{ cleared, checked, dryRun, scopeId }`. Each entry in `cleared` is +`{ ref, shadowedContent }`, carrying the body that had been served in the file's place — the last +place that text exists, so log or keep it before deciding the clear was right. `dryRun: true` +reports the same finding, deletes nothing, and makes the sentence read "WOULD be cleared". Running +it again over a migrated tree clears nothing, and it never touches a `resources/` document. + +`installedOn` is `{ id, isolatesOrgState }` for the flow the references are installed on, and it is +required rather than defaulted: a flow that isolates its org scope stores content under a different +address, so a wrong value here looks in the wrong place, finds nothing, and reports success. Read +`isolatesOrgState` off the flow. For an org or flow id containing `:` or a backslash it throws — +that address needs the engine's own escaping; clear those rows with the engine's store helpers +instead. + +### What a file may and may not declare **`description` is required.** A file without one lands in `errors`. It reaches the resource with the rest of the frontmatter, and nothing puts it in front of a model. Write it for whoever opens the -tree. +tree. Frontmatter never reaches an agent either way: a reference is served with its `---` block +stripped, and a `resources/` document is installed with its parsed body. **The convention owns identity, storage and content.** Where a document lives decides all three, so a file may not declare any of `scope`, `ref`, `stateSchema`, `default`, `content`, `contentFile`, @@ -909,6 +960,11 @@ so is `prefetchMode: "lazy"`: a file-declared document is always loaded eagerly. carried through as written, so `llmReadable`, `llmWritable`, `writable`, `allowedExtensions` and `metadata` all reach the resource. +A `references/` file may not declare `writable`, `llmWritable`, `render` or `flowIsolation` either — +`writable: false` included, even though it agrees with the folder. A reference is read-only on both +doors: `writeContent()` throws a `FlowError` with code `resource_read_only`, and the model is never +offered the write tool for it. A document that needs to be written belongs in `resources/`. + A document that needs a state schema, a render function, reactive bindings or an edge graph stays in code — those are functions, and a Markdown file cannot hold one. Session- and user-scoped resources are not file-declared. @@ -917,12 +973,14 @@ Document, team and worker folder names all follow one rule: lowercase letters, d single hyphens, at most 64 characters. A worker folder's documents load whether or not the folder holds a `WORKER.md` — this reader answers a question about a file, and a slot with no seat file is reported separately by `readWorkforceDirectory`. A non-`.md` file in the slot is passed over in silence. An -absent `org/` root or `resources/` folder is not an error — a team may have no documents. +absent `org/` root or document folder is not an error — a team may have no documents. -`readResourcesDirectory` throws only when `root` itself cannot be read or is a symlink. Everything -else lands in `errors`, one entry per thing that should have produced a document and did not. Each -entry is `{ kind, path, error }`, keyed by a path relative to the root, with `kind` naming the -condition (see [Error Semantics](#error-semantics)): +### What did not load + +Either reader throws only when `root` itself cannot be read or is a symlink. Everything else lands +in `errors`, one entry per thing that should have produced a document and did not. Each entry is +`{ kind, path, error }`, keyed by a path relative to the root, with `kind` naming the condition (see +[Error Semantics](#error-semantics)): ```ts errors; @@ -932,8 +990,12 @@ errors; // the document as "handbook.md" in this resources/ folder instead.') }] ``` -`resourcesFromDocs` throws instead of collecting, because a record that cannot become a resource is a -startup misconfiguration. +Neither reader sees the other's slot, so neither reports a basename claimed by both. +`readDeclaredRoster` reads the whole tree and puts that collision in its `problems`, naming both +files; `hireWorkforce` throws on the same collision for a catalog that never passed a loader. + +`resourcesFromDocs` and `referencesFromDocs` throw instead of collecting, because a record that +cannot become a resource is a startup misconfiguration. ## Channels @@ -1416,16 +1478,19 @@ membershipPrefix(""); | `validateSegment(segment, label)` | The one rule for what a name in this tree may be — lowercase letters, digits and single hyphens, under 64 characters, not reserved. Throws naming the segment and what it would have become. Ships from the `./loader` subpath (Node only). | | `discoverWorkforceCode(root)` | Walk `flows/workers/`, `flows/channels/` and `blocks/` one level deep, every `resources/` folder the convention reads, and every `blocks/` folder inside the team tree, returning what they hold on `files`, `resourceModules` and `seatBlocks` — each ordered by path — plus the `searched` patterns. Reads the tree only — it opens none of the modules it finds. Throws a `WorkforceCodeError` carrying every refusal. Ships from the `./codegen` subpath (Node only). | | `renderWorkforceCode(files, modules, seatBlocks?)` | Render a discovery's `files`, `resourceModules` and `seatBlocks` as a module of static imports exporting `kinds`, `channelKinds`, `blocks`, `seatBlocks` and `resourceModules`. Pass all three: the third parameter defaults to `[]`, so omitting it renders an empty `seatBlocks` map and reports nothing. Deterministic: the same tree renders the same bytes. `fsdev gen` is a thin command over this and the call above. Ships from the `./codegen` subpath. | -| `hireWorkforce(manifests, { kinds, seatBlocks, channelBoards, documents })` | Turn worker records into one configured flow copy each, ordered by id. Pass `defineFlow(...)` results directly as `kinds`, `workforce.gen.ts`'s `seatBlocks` export as `seatBlocks`, and, when any seat file declares `resources:`, the map `resourcesFromDocs` returns as `documents`. `channelBoards` is optional and advisory: give it the roster's minted board ids and unattended boards are warned about on stderr. | +| `hireWorkforce(manifests, { kinds, seatBlocks, channelBoards, documents, references })` | Turn worker records into one configured flow copy each, ordered by id. Pass `defineFlow(...)` results directly as `kinds`, `workforce.gen.ts`'s `seatBlocks` export as `seatBlocks`, and, when any seat file declares `resources:`, the map `resourcesFromDocs` returns as `documents`. Pass the map `referencesFromDocs` returns as `references` whenever the tree has a `references/` folder: it is what marks those entries as references, and the per-seat tree wall applies only to entries it names. `channelBoards` is optional and advisory: give it the roster's minted board ids and unattended boards are warned about on stderr. | | `workerConfigSchema()` | The admission contract every hireable worker kind composes: `configSchema: workerConfigSchema().extend({ ...its own settings })`. Declares `instructions?`, `teamInstructions?`, `seatSkills` and `seatTools`. A kind whose schema cannot take what hiring imposes refuses the whole roster at startup. A fresh schema per call. | | `seatSkillSchema` | One skill as it rides into the bag — `{ name, skillMd, files? }`, closed. The shape `seatSkills` is an array of; reach for it when declaring your own variant of that key. | | `WorkerConfig` | The parsed shape of `workerConfigSchema()` — what every hireable kind receives, whatever else it extends on. | | `readResourcesDirectory(root)` | Read every `resources/` folder in the tree — org, team, and each worker's own — into one `ResourceDoc` per document. Ships from the `./loader` subpath (Node only). | | `resourcesFromDocs(documents)` | Turn document records into the flow resource map, keyed by each document's ref. Spread it into your own `resources`. | +| `readReferencesDirectory(root)` | Read every `references/` folder in the tree — org, team, and each worker's own — into one `ResourceDoc` per document, each carrying its `filePath`. Same result shape and same error kinds as `readResourcesDirectory`. Ships from the `./loader` subpath (Node only). | +| `referencesFromDocs(references)` | Turn reference records into the flow resource map, keyed by ref. Each entry is served from its file and is read-only. Throws naming the ref for a record with no `filePath`. | +| `clearShadowedReferences(input)` / `describeShadowedReferences(result)` | Clear the stored rows left behind when a document moves from `resources/` to `references/`, for one org, and render the result as one log line. Takes `{ references, orgId, content, installedOn, dryRun? }`; returns `{ cleared, checked, dryRun, scopeId }`. | | `splitResourceModules(resourceModules)` | Split the generated map into `{ capabilities, resources }` — the capabilities a worker kind installs through `uses`, and the resources that merge into the one resource map. Installs nothing: you pass both on, at your own call site. Throws naming the ref when an entry can be neither. | -| `DeclaredRoster` / `DeclaredProblem` | What `readDeclaredRoster` returns: `{ workers, teams, documents, channels, problems }`, and one problem: `{ layer, path, error, worker? }`, where `layer` is `worker`, `skill`, `team`, `document` or `channel`. | +| `DeclaredRoster` / `DeclaredProblem` | What `readDeclaredRoster` returns: `{ workers, teams, documents, references, channels, problems }`, and one problem: `{ layer, path, error, worker? }`, where `layer` is `worker`, `skill`, `team`, `document`, `reference` or `channel`. | | `WorkerManifest` | One worker record: `{ id, declared, body, skills? }`. | -| `ResourceDoc` | One document record: `{ ref, declared, body }`. | +| `ResourceDoc` | One document record: `{ ref, declared, body, filePath? }`. | | `mintResourceRef(teamId, workerName, name)` | The one rule that names a resource, whichever door read it — the ref a document or a module called `name` in that folder gets. Throws naming the segment that breaks the rules. Ships from the `./loader` subpath (Node only). | | `ResourceModules` | The generated `resourceModules` map: one entry per discovered module, keyed by its ref. | | `ResourceModuleExport` / `WorkerResourceModuleExport` | What a module in the organisation's or a team's `resources/` folder may be — a capability or a resource — and the narrower type a worker's own folder is held to: a resource, never a capability. | @@ -1478,13 +1543,19 @@ membershipPrefix(""); | Symlinked `skills/` folder at a level | Collected in `readSeatSkills`'s `errors` as `kind: "refused-symlinked-level"`, keyed by the level's path — never followed | | One skill name at more than one of a seat's levels | Collected in `readSeatSkills`'s `errors` as `kind: "duplicate-skill-name"`, keyed by the level the name was first seen at, with every colliding path on the entry's `paths`; the name is left out of `skills` | | `scope:` in a `SKILL.md` | Collected in `readSeatSkills`'s `errors` as `kind: "refused-scope-key"`, keyed by the skill's path | -| Worker cannot be hired | `hireWorkforce` — an empty or whitespace-only `flow`, an unknown kind, a flow passed under a key that is not its own kind, a duplicate id, a setting or body the flow never declared, a `tools:` name nothing registers for that seat, a registered block whose key and own `name` disagree, a block in a worker's own folder that declares a resource or `requireOrg`, a skill name reaching one seat from both the app's `skills` and its own folders, a `resources:` list the hire step cannot resolve (a `resources:` that is not a list, an entry that is neither a ref nor a one-key `ref: mode` mapping, a ref no document matches, a ref naming a document the app declared but did not install on this seat's kind, a mode other than `ro` or `rw`, the same ref twice, `rw` on a document declaring itself `writable: false`, a ref colliding with a name the kind's own blocks declare, a ref the kind declares at flow level while what it holds there is not that document, or the key itself with no `documents` passed), a document a seat did not name that its minted flow reaches anyway because one of the kind's blocks declares it, `instructions` given both in the frontmatter and as a body, a flow kind whose schema will not take what hiring imposes (composing `workerConfigSchema()` is the fix), or a `persona:`, `seatSkills:`, `seatTools:` or `teamInstructions:` key. Collected: one error names every bad worker | +| Worker cannot be hired | `hireWorkforce` — an empty or whitespace-only `flow`, an unknown kind, a flow passed under a key that is not its own kind, a duplicate id, a setting or body the flow never declared, a `tools:` name nothing registers for that seat, a registered block whose key and own `name` disagree, a block in a worker's own folder that declares a resource or `requireOrg`, a skill name reaching one seat from both the app's `skills` and its own folders, a `resources:` list the hire step cannot resolve (a `resources:` that is not a list, an entry that is neither a ref nor a one-key `ref: mode` mapping, a ref no document matches, a ref naming a document the app declared but did not install on this seat's kind, a mode other than `ro` or `rw`, the same ref twice, `rw` on a document declaring itself `writable: false`, a ref colliding with a name the kind's own blocks declare, a ref the kind declares at flow level while what it holds there is not that document, or the key itself with no `documents` passed), a document a seat did not name that its minted flow reaches anyway because one of the kind's blocks declares it, a `references:` list the hire step cannot resolve (a `references:` that is not a list, an entry that is not a ref, the same ref twice, or a ref naming a reference this seat cannot reach from its place in the tree — including every ref when no `references` map was passed), a seat id that names no place in the tree while its kind holds references, a reference the seat did not name that its minted flow reaches anyway, `instructions` given both in the frontmatter and as a body, a flow kind whose schema will not take what hiring imposes (composing `workerConfigSchema()` is the fix), or a `persona:`, `seatSkills:`, `seatTools:` or `teamInstructions:` key. Collected: one error names every bad worker | | A `resources/` slot, `org/`, `teams/`, a team folder, a `workers/` level or a worker folder unreadable or symlinked | Collected in `readResourcesDirectory`'s `errors` as `kind: "unreadable-slot"`, keyed by that folder's path — an absent folder is empty instead | | A directory where a document file belongs | Collected in `readResourcesDirectory`'s `errors` as `kind: "folder-where-file-belongs"`, keyed by the directory's path | | Document file fails to load | Collected in `readResourcesDirectory`'s `errors` as `kind: "document-load-failed"`, keyed by the file's path — an unusable name, a symlink, an unreadable file, no frontmatter, or a missing `description` | | A setting the convention derives, or `prefetchMode: "lazy"`, in a document file | Collected in `readResourcesDirectory`'s `errors` as `kind: "refused-declaration"`, keyed by the file's path | | Workforce root unreadable or symlinked, read for documents | `readResourcesDirectory` throws — the root is never followed through a link | | Document cannot become a resource | `resourcesFromDocs` throws naming the ref — a setting the convention derives, a lazy `prefetchMode`, or frontmatter `defineResource` itself rejects | +| Any of the four conditions above, in a `references/` folder | `readReferencesDirectory` reports the same `kind`s at the same paths, and throws on the same root conditions | +| `writable`, `llmWritable`, `render` or `flowIsolation` in a reference file | Collected in `readReferencesDirectory`'s `errors` as `kind: "refused-declaration"`, keyed by the file's path — at either value | +| One basename claimed by a `references/` and a `resources/` file at one level | Collected in `readDeclaredRoster`'s `problems` on the `reference` layer, naming both files. Neither single-folder reader sees it | +| Reference cannot become a resource | `referencesFromDocs` throws naming the ref — a setting the convention derives, a lazy `prefetchMode`, a record with no `filePath`, or frontmatter `defineResource` itself rejects | +| One ref passed to `hireWorkforce` as both a document and a reference | `hireWorkforce` throws before hiring anything, naming every ref in both maps | +| A reference row cannot be addressed | `clearShadowedReferences` throws when the org id — or, for a flow that isolates its org scope, the flow id — contains `:` or a backslash | | A `channels/` slot, `teams/` or a team folder unreadable or symlinked | Collected in `readChannelsDirectory`'s `errors` as `kind: "unreadable-slot"`, keyed by that folder's path — an absent folder is empty instead | | Channel folder fails to load | Collected in `readChannelsDirectory`'s `errors` as `kind: "channel-load-failed"`, keyed by the folder's path — an unusable name, a symlink, or a missing, unreadable or malformed `CHANNEL.md` | | `system:` in a `CHANNEL.md` | Collected in `readChannelsDirectory`'s `errors` as `kind: "refused-declaration"`, keyed by the channel folder's path | From 46a9860def1d4b77b38b8e18c985b6520797dace Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 20 Sep 2026 17:43:52 +0000 Subject: [PATCH 4/4] Refuse a kind holding references the hire step was not given MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The wall had a silent off-switch. `applyReferenceWall` derives reachability against the catalog the app passes as `hireWorkforce({ references })`, and `referenceKeys` returns empty when that is absent — so an app that spread `referencesFromDocs(...)` into its flow and forgot the hire option got D1 intact (the seal rides on the definition) and D2 gone: nothing recognised as a reference, the wall applied to nothing, every seat reaching every team's handbook, and no warning anywhere. That is this issue's own problem statement one level up. The change opens by naming a wall that exists only as a line the app remembers to write, with nothing said when it is missing; shipping a wall whose absence is equally silent would hand the same failure to the next author, in a shape that looks safe because the folder is called `references/`. D2 was ratified as "there is no install-side override, because one would make the wall optional again", and omission is an install-side override wearing a different hat. Refused at hire, naming the option. A partial catalog is refused by the same check and for the same reason: one unrecognised reference is one document the wall does not cover. **Reference definitions are distinguishable at hire, and the discriminator is `render === referenceBody` — identity, not shape.** That fact is load-bearing for anyone who later touches `references-from-docs`: it is the only thing that attaches that exact function, so identity answers "did this convention build it?" A shape test (`contentFile` set, both write flags false) would read a hand-written sealed file-backed resource as a reference and refuse an app whose wiring is fine. A brand field was rejected for the reason the convention rejects every other marker — a second place the same fact lives, and one a hand-built entry could set. Probed before building on it: `render` survives `defineResource` by identity. ## The published-surface guard, fixed in its extraction rather than its rows Both of its failures were the guard not understanding the convention: - `RESERVED` and the leaf normalizer now read `DOCUMENT_SLOTS` from `resource-convention`, so a file under `references/` normalizes to `.md` exactly as one under `resources/` does. Half the convention's shapes were being spelled as whatever a page's example happened to call them. - The README states the ref rule once as `/org//.md` rather than writing every level twice. `` is a notation, not a folder, so a shape carrying it now expands into both slots before comparison. Read literally it was a folder nobody implements. Expanding also makes the check stricter: one generic sentence is verified against each slot separately. Four `references/` rows follow, one per level — including `org/workers//references/.md`, which reaches nobody, because no seat is minted below an org worker's folder. Tabled anyway: the pages publish the shape, and a shape that reaches nobody is exactly what this suite must not pass over. ## Documentation, in the same commit because it is the same fact Three pages move with the behaviour, written by the isolated docs pass rather than by this change's author. Four passages had documented the silent fallback — two in `documents-on-disk.md` and the README, one in `workers-on-disk.md` that this change had missed entirely — and all four now state the requirement in the present tense, with the failure shown, so no reader learns it once worked differently. The install snippet runs end to end, sample refs are whole, and the read-only failure leads with `resource_read_only`. Evidence, re-run after the pages landed rather than before: 685 pass / 41 files, workforce and CLI typechecks clean, CLI 246 pass, five process guards clean. The refusal is mutation-checked — tolerating the missing catalog reds all three of its cases, and they assert the negative (an app that omits the option is refused, not quietly hired). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016qaz1VpRK4wCQzxpjzZ4Hz --- .changeset/references-read-path.md | 2 +- apps/docs/docs/workforce/documents-on-disk.md | 43 ++++++--- apps/docs/docs/workforce/workers-on-disk.md | 2 +- packages/workforce/README.md | 33 +++---- .../workforce/src/references-from-docs.ts | 20 ++++ packages/workforce/src/seat-references.ts | 41 ++++++++ .../test/published-tree-surface.test.ts | 96 +++++++++++++------ .../test/references-convention.test.ts | 89 +++++++++++++++++ 8 files changed, 265 insertions(+), 61 deletions(-) diff --git a/.changeset/references-read-path.md b/.changeset/references-read-path.md index c46987ab34..c008c76601 100644 --- a/.changeset/references-read-path.md +++ b/.changeset/references-read-path.md @@ -2,4 +2,4 @@ "@flow-state-dev/workforce": minor --- -A workforce tree can declare read-only documents in `references/` beside the writable ones in `resources/`. A reference's body is served from its file on every execution context rather than from a stored row, so editing the file is the edit; `writable`, `llmWritable`, `render` and `flowIsolation` are derived by the folder and refused in frontmatter. A seat reaches the references at or above its place in the tree — the org's, its own team's and its own folder's — with no install-side filter, narrowed further by a `references:` list in its `WORKER.md`. `resources/` behaviour is unchanged. `clearShadowedReferences({ references, orgId, content, installedOn })` migrates a tree where a document was written before it moved — `installedOn` names the flow so the stored content is addressed where it actually lives rather than guessed at (FIX-1467). +A workforce tree can declare read-only documents in `references/` beside the writable ones in `resources/`. A reference's body is served from its file on every execution context rather than from a stored row, so editing the file is the edit; `writable`, `llmWritable`, `render` and `flowIsolation` are derived by the folder and refused in frontmatter. A seat reaches the references at or above its place in the tree — the org's, its own team's and its own folder's — with no install-side filter, narrowed further by a `references:` list in its `WORKER.md`. Installing references on a kind without also passing them to `hireWorkforce` as `references` is refused at hire, naming the option: the wall is derived against that catalog, so omitting it would leave every seat reaching every team's references with nothing to say so. `resources/` behaviour is unchanged. `clearShadowedReferences({ references, orgId, content, installedOn })` migrates a tree where a document was written before it moved — `installedOn` names the flow so the stored content is addressed where it actually lives rather than guessed at (FIX-1467). diff --git a/apps/docs/docs/workforce/documents-on-disk.md b/apps/docs/docs/workforce/documents-on-disk.md index 24c65b2ae1..c5c9b20392 100644 --- a/apps/docs/docs/workforce/documents-on-disk.md +++ b/apps/docs/docs/workforce/documents-on-disk.md @@ -49,7 +49,7 @@ workforce/ Five documents: a code of conduct everyone reads, a handbook for engineering, a runbook for one seat, an escalation procedure for support, and a scratchpad the engineering agents can write. Every example below reads this tree. -Both folders name documents the same way, and they share one namespace. `references/handbook.md` and `resources/handbook.md` in one team are two spellings of the name `handbook`. `readDeclaredRoster` reports the pair in its `problems`, and `hireWorkforce` throws on it, naming both files. Neither single-folder reader sees the other folder, so neither of them reports it on its own. +Both folders name documents the same way, and they share one namespace. `references/handbook.md` and `resources/handbook.md` in one team are two spellings of the name `handbook`. Neither single-folder reader sees the other folder, so neither reports the pair on its own. `readDeclaredRoster`, which walks the whole tree in one call, puts it in its `problems` and names both files; `hireWorkforce` throws on a ref handed to it as both a document and a reference. A worker's own folder is what lets two seats each have a `runbook` without their authors agreeing on a name. Organization-level workers, under `org/workers/`, are read the same way. @@ -196,15 +196,20 @@ A reported file is a document your app was supposed to have. Log a warning and c ```ts import { defineFlow } from "@flow-state-dev/core"; -import { referencesFromDocs, resourcesFromDocs } from "@flow-state-dev/workforce"; -import { readReferencesDirectory, readResourcesDirectory } from "@flow-state-dev/workforce/loader"; +import { hireWorkforce, referencesFromDocs, resourcesFromDocs } from "@flow-state-dev/workforce"; +import { + readReferencesDirectory, + readResourcesDirectory, + readWorkforceDirectory, +} from "@flow-state-dev/workforce/loader"; import { answerQuestion } from "./blocks"; import { ticketResource } from "./resources"; +const roster = await readWorkforceDirectory("./workforce"); const references = await readReferencesDirectory("./workforce"); const resources = await readResourcesDirectory("./workforce"); -for (const { errors } of [references, resources]) { - if (errors.length) throw new Error(`workforce: ${errors.length} document(s) failed to load`); +for (const { errors } of [roster, references, resources]) { + if (errors.length) throw new Error(`workforce: ${errors.length} entries failed to load`); } const documentMap = resourcesFromDocs(resources.documents); @@ -215,19 +220,29 @@ export const supportFlow = defineFlow({ actions: { answer: { block: answerQuestion } }, resources: { ticket: ticketResource, ...documentMap, ...referenceMap }, }); + +export const seats = hireWorkforce(roster.workers, { + kinds: { support: supportFlow }, + documents: documentMap, + references: referenceMap, +}); ``` Every file-declared document is installed at `org` scope. A `resources/` entry carries the file's body as its starting content; a `references/` entry points at the file itself, which is read whenever an execution context is built. Editing a reference in your repository reaches agents on the next request, and a read already in flight keeps the body it started with. -Pass both maps to `hireWorkforce` as well, under `documents` and `references`: +`documents` and `references` tell the hire which entries on a kind's map are which. [The tree wall](#who-reaches-what) is derived against the `references` map, so a kind holding references has to be hired with it. Omit it, or pass one that is missing a reference the kind installed, and `hireWorkforce` throws, naming every reference it was not given. The whole roster is refused, so no seat is hired: -```ts -hireWorkforce(workers, { kinds, documents: documentMap, references: referenceMap }); +``` +hireWorkforce refused 1 of 1 worker; nothing was hired: + - worker "engineering.ada" — is hired onto a kind holding 4 reference(s) that hireWorkforce + was not given: "code-of-conduct", "teams/engineering/handbook", + "teams/engineering/workers/ada/runbook", "teams/support/escalation". … Pass the same map you + installed on the kind: hireWorkforce(workers, { references: referencesFromDocs(refs) }) ``` -That is what tells the hire which entries on a kind's map are references, which is what [the tree wall](#who-reaches-what) narrows. Leave `references` out and no entry is treated as one: every seat reaches every reference the kind installed, and a seat whose `WORKER.md` names any is refused. A ref passed in both maps is refused too, naming it. +A kind holding no references needs no map, and a roster hires the same whether you pass one or not. A ref passed in both maps is refused too, naming it. -Spread the maps rather than passing one on its own. A flow copy created with `supportFlow({ resources })` *replaces* the definition's map instead of merging with it, so a copy handed only `resourcesFromDocs(documents)` loses whatever the flow kind declared. +Spread the maps rather than passing one on its own. A flow copy created with `supportFlow({ resources })` *replaces* the definition's map instead of merging with it, so a copy handed only `documentMap` loses whatever the flow kind declared. Both functions throw rather than collecting. A record that cannot become a resource stops startup, naming the ref. @@ -272,7 +287,7 @@ references: --- ``` -Leaving the key out means every reference at or above the seat. Writing `references: []` means none — the two are different answers, not the same one. +Leaving the key out means every reference at or above the seat. Writing `references: []` means none. Seats are read from `teams//workers//`, so a reference under `org/workers//references/` sits beside the organization level rather than above any seat, and no seat reaches it. @@ -321,11 +336,11 @@ export const answerQuestion = generator({ }); ``` -`requireOrg: true` is what binds the request to an org, which org-scoped documents need to load at all. +`requireOrg: true` is there because documents are org-scoped. See [The request needs an org](#the-request-needs-an-org). The documents are already declared on the flow, so the generator does not declare them again. The tool addresses a document by its scope-qualified uri, the same handle the [search tools](/docs/resources/searching) return. See [LLM access patterns](/docs/resources/overview#llm-access-patterns). -A reference is read-only to code and to the model. `writeContent()` on one throws a `FlowError` with code `resource_read_only` and the message `Resource "handbook" content is read-only`, and the write tool is never offered for it, whatever tools the generator carries. +A reference is read-only to code and to the model. `writeContent()` on one throws a `FlowError` with code `resource_read_only`, whose message names the ref: `Resource "teams/engineering/handbook" content is read-only`. The write tool is never offered for a reference either, whatever tools the generator carries. ## Moving a document into `references/` @@ -343,7 +358,7 @@ const result = await clearShadowedReferences({ installedOn: { id: flow.id, isolatesOrgState: false }, }); console.log(describeShadowedReferences(result)); -// references: 1 of 4 were shadowed by a stored write and have been cleared — handbook. +// references: 1 of 4 were shadowed by a stored write and have been cleared — teams/engineering/handbook. // Each now serves its file again. ``` diff --git a/apps/docs/docs/workforce/workers-on-disk.md b/apps/docs/docs/workforce/workers-on-disk.md index 976902eabb..d3d648a197 100644 --- a/apps/docs/docs/workforce/workers-on-disk.md +++ b/apps/docs/docs/workforce/workers-on-disk.md @@ -406,7 +406,7 @@ One catalog, spread into the flow and handed to the hire. Which entries in that `documents` is consulted only for a worker that declares `resources:`. A roster where none does hires the same whether you pass it or not. A worker that does declare one while `documents` is absent is refused, naming what is missing. -References go the same way, under a `references` option holding the map `referencesFromDocs` returned. That option is what marks those entries as references, and the tree wall applies only to entries it names: leave it out and every worker reaches every reference its kind installed. [Installing the documents](./documents-on-disk.md#installing-the-documents) has the call with both maps. +References go under a `references` option holding the map `referencesFromDocs` returned, and that one works the other way round: the tree wall is derived against it, so a kind holding references has to be hired with it. Leave it out, or pass a map missing one of them, and the whole roster is refused, naming every reference the hire was not given. [Installing the documents](./documents-on-disk.md#installing-the-documents) has the call with both maps. A list the hire step cannot resolve refuses the whole roster, naming the worker and what was wrong. The message calls one entry a grant and quotes the ref as the file spelled it, which here is `handbook` with two letters swapped: diff --git a/packages/workforce/README.md b/packages/workforce/README.md index 9bdd8746f1..3101d48937 100644 --- a/packages/workforce/README.md +++ b/packages/workforce/README.md @@ -825,9 +825,11 @@ const seats = hireWorkforce(workers, { }); ``` -`references` is what tells the hire which entries on a kind's map are references. Leave it out and -none of them is: every seat reaches every reference the kind installed, and a seat whose file -declares `references:` is refused. A ref passed in both maps is refused as well, naming it. +`references` is what tells the hire which entries on a kind's map are references, and the tree wall +applies only to the entries it names. So once a kind holds references, the option is required: omit +it, or pass a map that is missing one of them, and `hireWorkforce` throws, naming every reference it +was not given. A kind that holds none needs no map. A ref passed in both maps is refused as well, +naming it. **The flow needs an org identity, and it will not ask for one on its own.** Every file-declared document is org-scoped, and a flow collects `requiresOrg` from its blocks, not from its resource @@ -929,7 +931,7 @@ const result = await clearShadowedReferences({ // dryRun: true, }); console.log(describeShadowedReferences(result)); -// references: 1 of 4 were shadowed by a stored write and have been cleared — handbook. +// references: 1 of 4 were shadowed by a stored write and have been cleared — teams/engineering/handbook. // Each now serves its file again. ``` @@ -939,12 +941,11 @@ place that text exists, so log or keep it before deciding the clear was right. ` reports the same finding, deletes nothing, and makes the sentence read "WOULD be cleared". Running it again over a migrated tree clears nothing, and it never touches a `resources/` document. -`installedOn` is `{ id, isolatesOrgState }` for the flow the references are installed on, and it is -required rather than defaulted: a flow that isolates its org scope stores content under a different -address, so a wrong value here looks in the wrong place, finds nothing, and reports success. Read -`isolatesOrgState` off the flow. For an org or flow id containing `:` or a backslash it throws — -that address needs the engine's own escaping; clear those rows with the engine's store helpers -instead. +`installedOn` is `{ id, isolatesOrgState }` for the flow the references are installed on. A flow +that isolates its org scope stores content under a different address, so a wrong value here looks in +the wrong place, finds nothing, and reports success. Read `isolatesOrgState` off the flow. For an org +or flow id containing `:` or a backslash it throws. That address needs the engine's own escaping, so +clear those rows with the engine's store helpers instead. ### What a file may and may not declare @@ -960,20 +961,20 @@ so is `prefetchMode: "lazy"`: a file-declared document is always loaded eagerly. carried through as written, so `llmReadable`, `llmWritable`, `writable`, `allowedExtensions` and `metadata` all reach the resource. -A `references/` file may not declare `writable`, `llmWritable`, `render` or `flowIsolation` either — +A `references/` file may not declare `writable`, `llmWritable`, `render` or `flowIsolation` either, `writable: false` included, even though it agrees with the folder. A reference is read-only on both doors: `writeContent()` throws a `FlowError` with code `resource_read_only`, and the model is never offered the write tool for it. A document that needs to be written belongs in `resources/`. A document that needs a state schema, a render function, reactive bindings or an edge graph stays in -code — those are functions, and a Markdown file cannot hold one. Session- and user-scoped resources +code. Those are functions, and a Markdown file cannot hold one. Session- and user-scoped resources are not file-declared. Document, team and worker folder names all follow one rule: lowercase letters, digits and single hyphens, at most 64 characters. A worker folder's documents load whether or not the folder -holds a `WORKER.md` — this reader answers a question about a file, and a slot with no seat file is -reported separately by `readWorkforceDirectory`. A non-`.md` file in the slot is passed over in silence. An -absent `org/` root or document folder is not an error — a team may have no documents. +holds a `WORKER.md`, and a slot with no seat file is reported separately by +`readWorkforceDirectory`. A non-`.md` file in the slot is passed over in silence. An absent `org/` +root or document folder is not an error. A team may have no documents. ### What did not load @@ -1478,7 +1479,7 @@ membershipPrefix(""); | `validateSegment(segment, label)` | The one rule for what a name in this tree may be — lowercase letters, digits and single hyphens, under 64 characters, not reserved. Throws naming the segment and what it would have become. Ships from the `./loader` subpath (Node only). | | `discoverWorkforceCode(root)` | Walk `flows/workers/`, `flows/channels/` and `blocks/` one level deep, every `resources/` folder the convention reads, and every `blocks/` folder inside the team tree, returning what they hold on `files`, `resourceModules` and `seatBlocks` — each ordered by path — plus the `searched` patterns. Reads the tree only — it opens none of the modules it finds. Throws a `WorkforceCodeError` carrying every refusal. Ships from the `./codegen` subpath (Node only). | | `renderWorkforceCode(files, modules, seatBlocks?)` | Render a discovery's `files`, `resourceModules` and `seatBlocks` as a module of static imports exporting `kinds`, `channelKinds`, `blocks`, `seatBlocks` and `resourceModules`. Pass all three: the third parameter defaults to `[]`, so omitting it renders an empty `seatBlocks` map and reports nothing. Deterministic: the same tree renders the same bytes. `fsdev gen` is a thin command over this and the call above. Ships from the `./codegen` subpath. | -| `hireWorkforce(manifests, { kinds, seatBlocks, channelBoards, documents, references })` | Turn worker records into one configured flow copy each, ordered by id. Pass `defineFlow(...)` results directly as `kinds`, `workforce.gen.ts`'s `seatBlocks` export as `seatBlocks`, and, when any seat file declares `resources:`, the map `resourcesFromDocs` returns as `documents`. Pass the map `referencesFromDocs` returns as `references` whenever the tree has a `references/` folder: it is what marks those entries as references, and the per-seat tree wall applies only to entries it names. `channelBoards` is optional and advisory: give it the roster's minted board ids and unattended boards are warned about on stderr. | +| `hireWorkforce(manifests, { kinds, seatBlocks, channelBoards, documents, references })` | Turn worker records into one configured flow copy each, ordered by id. Pass `defineFlow(...)` results directly as `kinds`, `workforce.gen.ts`'s `seatBlocks` export as `seatBlocks`, and, when any seat file declares `resources:`, the map `resourcesFromDocs` returns as `documents`. Pass the map `referencesFromDocs` returns as `references` whenever a kind installs any: it is what marks those entries as references, the per-seat tree wall is derived against it, and a kind holding references it was not given refuses the whole roster. `channelBoards` is optional and advisory: give it the roster's minted board ids and unattended boards are warned about on stderr. | | `workerConfigSchema()` | The admission contract every hireable worker kind composes: `configSchema: workerConfigSchema().extend({ ...its own settings })`. Declares `instructions?`, `teamInstructions?`, `seatSkills` and `seatTools`. A kind whose schema cannot take what hiring imposes refuses the whole roster at startup. A fresh schema per call. | | `seatSkillSchema` | One skill as it rides into the bag — `{ name, skillMd, files? }`, closed. The shape `seatSkills` is an array of; reach for it when declaring your own variant of that key. | | `WorkerConfig` | The parsed shape of `workerConfigSchema()` — what every hireable kind receives, whatever else it extends on. | diff --git a/packages/workforce/src/references-from-docs.ts b/packages/workforce/src/references-from-docs.ts index 3a9b96cc69..1f94023514 100644 --- a/packages/workforce/src/references-from-docs.ts +++ b/packages/workforce/src/references-from-docs.ts @@ -94,6 +94,26 @@ export function referenceBody(raw: string): string { return splitFrontmatter(raw).body; } +/** + * Whether this declared entry is a reference THIS package installed. + * + * **By identity, not by shape.** The test is `render === referenceBody` — the + * exact function above, which only `referencesFromDocs` attaches. A shape test + * (`contentFile` set and both write flags false) would read a hand-written + * sealed file-backed resource as a reference and refuse an app whose wiring is + * fine. Identity says what this is asking: *did this convention build it?* + * + * It exists so the wall can tell that a kind is holding references before it + * has a catalog to recognise them with — which is the one way the wall could be + * switched off. See `./seat-references`. + * + * Deliberately no brand field. A marker property would be a second place the + * same fact lives, and one a hand-built entry could set. + */ +export function isReferenceDefinition(entry: unknown): boolean { + return (entry as { render?: unknown } | undefined)?.render === referenceBody; +} + /** * The wording for a reference record with no file to read. * diff --git a/packages/workforce/src/seat-references.ts b/packages/workforce/src/seat-references.ts index 5775a3b8c9..9a827cb178 100644 --- a/packages/workforce/src/seat-references.ts +++ b/packages/workforce/src/seat-references.ts @@ -35,6 +35,7 @@ import type { DeclaredResources } from "@flow-state-dev/core"; import { describe } from "./describe-value"; +import { isReferenceDefinition } from "./references-from-docs"; import { emptyMap } from "./empty-map"; /** @@ -235,6 +236,21 @@ const unreachableSeatReferenceMessage = (ref: string, seatId: string): string => `own team's, and its own folder's — and cannot add one from outside. Move the file up the ` + `tree to widen who reads it, or drop the entry.`; +/** + * The wording for a kind holding references the hire step was not given. + * + * The one refusal that is about the APP's wiring rather than a seat's file, so + * it names the option rather than the seat. Fatal, and deliberately not a + * warning: a warning here would be a wall that is off with a log line, which is + * the failure this convention exists to end. + */ +const referencesNotDeclaredMessage = (refs: readonly string[]): string => + `is hired onto a kind holding ${refs.length} reference(s) that hireWorkforce was not given: ` + + `${refs.map((ref) => `"${ref}"`).join(", ")}. A reference's reach is derived from where its ` + + `file sits, and that can only be worked out against the app's own catalog — so without it ` + + `every seat would reach every team's references and nothing would say so. Pass the same map ` + + `you installed on the kind: \`hireWorkforce(workers, { references: referencesFromDocs(refs) })\`.`; + /** * The wording for a seat whose id names no place in the tree. * @@ -392,6 +408,31 @@ export function applyReferenceWall(input: ApplyReferenceWallInput): AppliedRefer // would leave that alias outside the wall. const references = referenceKeys(catalog, kindResources); + // **Before the wall can be applied, check it is not simply absent.** + // + // Reachability is derived from the catalog, and the catalog is the app's to + // hand over. An app that spreads `referencesFromDocs(...)` into its flow but + // forgets to pass the same map here gets a kind full of references that + // nothing recognises: D1 still holds, because the seal rides on the + // definition, but D2 evaporates and every seat reaches every team's + // handbook. Nothing would warn, and the folder would still be called + // `references/`. + // + // That is this convention's own problem statement one level up — the wall + // back to a line the app must remember, with the same silence when it is + // missing — so omission is refused rather than tolerated. Fail closed and + // loud, the way the migration refuses an address it cannot compute. + // + // Recognised by IDENTITY, not by shape: see `isReferenceDefinition`. A + // partial catalog is refused for the same reason and by the same check — + // one unrecognised reference is one document the wall does not cover. + const unwalled = Object.keys(kindResources).filter( + (key) => isReferenceDefinition(kindResources[key]) && !references.has(key), + ); + if (unwalled.length > 0) { + return { reachable: new Set(), problems: [referencesNotDeclaredMessage(unwalled)] }; + } + // Nothing to wall. Left exactly as it was — the branch that keeps an app with // no references/ folder unchanged, including being minted with no map. if (references.size === 0 && !hasDeclared) { diff --git a/packages/workforce/test/published-tree-surface.test.ts b/packages/workforce/test/published-tree-surface.test.ts index 48a97a3e88..40a1e6d817 100644 --- a/packages/workforce/test/published-tree-surface.test.ts +++ b/packages/workforce/test/published-tree-surface.test.ts @@ -56,6 +56,7 @@ import { } from "../src/loader/read-resources-directory"; import { readSeatSkills } from "../src/loader/read-seat-skills"; import { readWorkforce } from "../src/loader/read-workforce"; +import { DOCUMENT_SLOTS } from "../src/loader/resource-convention"; /** Repo root, from this file: `packages/workforce/test/` is three levels down. */ const REPO_ROOT = path.resolve(import.meta.dirname, "../../.."); @@ -148,7 +149,7 @@ const WORKER_REFS_RESOLVE: ReadonlyArray<{ file: string; quote: string }> = [ }, { file: "packages/workforce/README.md", - quote: "| `/org/workers//resources/.md` | `workers//` |", + quote: "| `/org/workers///.md` | `workers//` |", }, ]; @@ -206,12 +207,31 @@ const RESERVED = new Set([ "org", "teams", "workers", - "resources", + // Both documents slots, read from the convention rather than spelled here: + // this suite exists to notice a folder the pages publish and the code does + // not, so its own vocabulary must not be a second place the slot names live. + // A third slot added later arrives here on its own. + ...DOCUMENT_SLOTS, "skills", "channels", "flows", "blocks", ]); + +/** + * The metavariable the pages use when a rule holds for BOTH documents slots. + * + * The ref rule is one rule over `resources/` and `references/`, and the README + * states it once as `/org//.md` rather than writing every + * level twice. That is a notation, not a path — so a shape carrying it is + * expanded into the concrete slots before anything compares it to the table. + * Read literally it would be a folder nobody implements, which is the opposite + * of what it says. + * + * Expanding rather than tabling it also makes the check STRICTER: the page's + * one generic sentence is now verified against each slot separately. + */ +const SLOT_PLACEHOLDER = ""; /** Filenames the convention fixes, which stay literal in a shape. */ const FIXED_LEAVES = new Set(["WORKER.md", "CHANNEL.md", "SKILL.md"]); @@ -261,7 +281,14 @@ function toShape(token: string): string | undefined { const extension = /\.([a-z]+)$/.exec(segment)?.[1]; if (grandparent === "flows") return `.${extension}`; if (parent === "blocks") return `.${extension}`; - if (parent === "resources") return `.${extension}`; + // Either documents slot, for the reason `RESERVED` reads them from the + // convention: a file under `references/` is an author's own name exactly + // as one under `resources/` is, and normalizing only one of them left + // half the convention's shapes spelled as whatever the page's example + // happened to call them. + if (parent !== undefined && (DOCUMENT_SLOTS as readonly string[]).includes(parent)) { + return `.${extension}`; + } if (parent === "teams") return ""; if (parent === "workers") return ""; if (parent === "skills") return ""; @@ -336,7 +363,14 @@ function shapesDeclaredIn(text: string): Set { const shapes = new Set(); for (const token of tokens) { const shape = toShape(token); - if (shape !== undefined) shapes.add(shape); + if (shape === undefined) continue; + // A `` shape is one sentence standing for both slots, so it becomes + // both. Everything else passes through untouched. + if (shape.includes(SLOT_PLACEHOLDER)) { + for (const slot of DOCUMENT_SLOTS) shapes.add(shape.replace(SLOT_PLACEHOLDER, slot)); + continue; + } + shapes.add(shape); } return shapes; } @@ -449,21 +483,20 @@ const PUBLISHED_SHAPES: readonly PublishedShape[] = [ out.reported.includes("teams/alpha/channels/standup"), }, { - shape: "org/references/code-of-conduct.md", + shape: "org/references/.md", publishedIn: { file: "apps/docs/docs/workforce/documents-on-disk.md", - quote: " code-of-conduct.md", + quote: "| `/org/references/code-of-conduct.md` | `code-of-conduct` |", }, - write: (root) => writeFile(root, "org/references/code-of-conduct.md", doc("A code of conduct.")), + write: (root) => writeFile(root, "org/references/coc.md", doc("A code of conduct.")), accountedFor: (out) => - out.references.includes("code-of-conduct") || - out.reported.includes("org/references/code-of-conduct.md"), + out.references.includes("coc") || out.reported.includes("org/references/coc.md"), }, { - shape: "teams//references/handbook.md", + shape: "teams//references/.md", publishedIn: { - file: "apps/docs/docs/workforce/documents-on-disk.md", - quote: " handbook.md", + file: "packages/workforce/README.md", + quote: "| `/teams///.md` | `teams//` |", }, write: (root) => writeFile(root, "teams/alpha/references/handbook.md", doc("A handbook.")), accountedFor: (out) => @@ -471,33 +504,38 @@ const PUBLISHED_SHAPES: readonly PublishedShape[] = [ out.reported.includes("teams/alpha/references/handbook.md"), }, { - shape: "teams//references/escalation.md", + shape: "teams//workers//references/.md", publishedIn: { file: "apps/docs/docs/workforce/documents-on-disk.md", - quote: " escalation.md", + quote: + "| `/teams/engineering/workers/ada/references/runbook.md` | `teams/engineering/workers/ada/runbook` |", }, - write: (root) => writeFile(root, "teams/alpha/references/escalation.md", doc("An escalation.")), + write: (root) => + writeFile(root, "teams/alpha/workers/lead/references/runbook.md", doc("A runbook.")), accountedFor: (out) => - out.references.includes("teams/alpha/escalation") || - out.reported.includes("teams/alpha/references/escalation.md"), + out.references.includes("teams/alpha/workers/lead/runbook") || + out.reported.includes("teams/alpha/workers/lead/references/runbook.md"), }, { - shape: "teams//workers//references/runbook.md", + // The level with no seat below it: documents here load and are minted, and + // no seat can be hired at this address, so nothing reaches them. Tabled + // anyway — the pages publish the shape, and a shape that reaches nobody is + // exactly the kind this suite must not let go unnoticed. + shape: "org/workers//references/.md", publishedIn: { - file: "apps/docs/docs/workforce/documents-on-disk.md", - quote: " runbook.md", + file: "packages/workforce/README.md", + quote: "| `/org/workers///.md` | `workers//` |", }, - write: (root) => - writeFile(root, "teams/alpha/workers/lead/references/runbook.md", doc("A runbook.")), + write: (root) => writeFile(root, "org/workers/build/references/playbook.md", doc("A playbook.")), accountedFor: (out) => - out.references.includes("teams/alpha/workers/lead/runbook") || - out.reported.includes("teams/alpha/workers/lead/references/runbook.md"), + out.references.includes("workers/build/playbook") || + out.reported.includes("org/workers/build/references/playbook.md"), }, { shape: "teams//resources/.md", publishedIn: { file: "apps/docs/docs/workforce/documents-on-disk.md", - quote: "| `/teams/engineering/resources/handbook.md` | `teams/engineering/handbook` |", + quote: "| `/teams/engineering/resources/scratch.md` | `teams/engineering/scratch` |", }, write: (root) => writeFile(root, "teams/alpha/resources/handbook.md", doc("A handbook.")), accountedFor: (out) => @@ -507,9 +545,9 @@ const PUBLISHED_SHAPES: readonly PublishedShape[] = [ { shape: "teams//workers//resources/.md", publishedIn: { - file: "apps/docs/docs/workforce/documents-on-disk.md", + file: "packages/workforce/README.md", quote: - "| `/teams/engineering/workers/on-call/resources/runbook.md` | `teams/engineering/workers/on-call/runbook` |", + "| `/teams//workers///.md` | `teams//workers//` |", }, write: (root) => writeFile(root, "teams/alpha/workers/lead/resources/runbook.md", doc("A runbook.")), @@ -544,8 +582,8 @@ const PUBLISHED_SHAPES: readonly PublishedShape[] = [ { shape: "org/resources/.md", publishedIn: { - file: "apps/docs/docs/workforce/documents-on-disk.md", - quote: "| `/org/resources/code-of-conduct.md` | `code-of-conduct` |", + file: "packages/workforce/README.md", + quote: "| `/org//.md` | `` |", }, write: (root) => writeFile(root, "org/resources/code-of-conduct.md", doc("How we behave.")), diff --git a/packages/workforce/test/references-convention.test.ts b/packages/workforce/test/references-convention.test.ts index 3e3f128708..d739b0aebd 100644 --- a/packages/workforce/test/references-convention.test.ts +++ b/packages/workforce/test/references-convention.test.ts @@ -898,3 +898,92 @@ describe("the migration addresses the right bucket, or refuses", () => { expect(fine.scopeId).toBe(ORG); }); }); + + +// --------------------------------------------------------------------------- +// The wall's own off-switch: forgetting to hand it the catalog +// --------------------------------------------------------------------------- + +describe("the wall cannot be turned off by omission", () => { + /** + * An app spreads `referencesFromDocs(...)` into its flow but forgets to pass + * the same map to `hireWorkforce` as `references`. + * + * D1 survives that — the seal rides on the definition, so the files stay + * read-only and still read from disk. **D2 does not.** Nothing is recognised + * as a reference, so the wall applies to nothing and every seat reaches every + * team's handbook. + * + * That is this convention's own problem statement one level up: the wall it + * exists to derive would be back to a line the app has to remember, and its + * absence would be as silent as the filter's was. Refused instead, naming the + * option, the same way the migration refuses an address it cannot compute. + */ + async function hireWithoutCatalog(seatRecord = seat(ENG_SEAT)) { + const root = await tree(); + const documents = resourcesFromDocs((await readResourcesDirectory(root)).documents); + const references = referencesFromDocs((await readReferencesDirectory(root)).documents); + return hireWorkforce([seatRecord], { + kinds: { [KIND]: kindWith({ ...documents, ...references }) as never }, + documents, + // `references` deliberately NOT passed. This is the whole case. + }); + } + + it("RED→GREEN — refuses rather than hiring a seat with no wall", async () => { + await expect(hireWithoutCatalog()).rejects.toThrow(/references/); + }); + + it("the refusal names the option the app is missing, so it is actionable", async () => { + await expect(hireWithoutCatalog()).rejects.toThrow(/hireWorkforce/); + }); + + it("a PARTIAL catalog is refused too — the hole is the same size", async () => { + const root = await tree(); + const documents = resourcesFromDocs((await readResourcesDirectory(root)).documents); + const all = referencesFromDocs((await readReferencesDirectory(root)).documents); + // Everything except the sales handbook, which would then be walled by + // nothing and reachable by every seat. + const partial = Object.fromEntries( + Object.entries(all).filter(([ref]) => ref !== SALES_HANDBOOK), + ); + expect(() => + hireWorkforce([seat(ENG_SEAT)], { + kinds: { [KIND]: kindWith({ ...documents, ...all }) as never }, + documents, + references: partial, + }), + ).toThrow(new RegExp(SALES_HANDBOOK)); + }); + + it("an app with no references at all is untouched — no catalog, no refusal", async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "fix1467-none-")); + roots.push(root); + await writeInto(root, { + "teams/engineering/resources/scratch.md": doc("Scratch", "seed"), + }); + const documents = resourcesFromDocs((await readResourcesDirectory(root)).documents); + const hired = hireWorkforce([seat(ENG_SEAT)], { + kinds: { [KIND]: kindWith(documents) as never }, + documents, + }); + const { ctx } = await ctxFor(hired[0]!); + expect(await handleFor(ctx, ENG_SCRATCH)!.readContent()).toBe("seed"); + }); +}); + +describe("a reference above no seat reaches nobody", () => { + it("an org WORKER's references/ sits beside the org level, not above a team seat", async () => { + const root = await tree(); + await writeInto(root, { + "org/workers/build/references/runbook.md": doc("Build runbook", "BUILD RUNBOOK v1"), + }); + const { seat: ada } = await seatOn(root); + const { ctx } = await ctxFor(ada); + // Seats are minted only from `teams//workers//`, so nothing in + // this tree sits below an org worker's folder. The document loads, installs + // and is reachable by no seat at all — the same gap that retired BR-8, kept + // visible here rather than left for someone to rediscover. + expect(handleFor(ctx, "workers/build/runbook")).toBeUndefined(); + }); +});