Skip to content

fix(engine): scope state reads with its schema's defaults (FIX-2554) - #4770

Merged
jhoffner merged 5 commits into
mainfrom
contact/fix-2554-scope-state-applies-its-schemas-defaults-on-load
Nov 5, 2026
Merged

jhoffner merged 5 commits into
mainfrom
contact/fix-2554-scope-state-applies-its-schemas-defaults-on-load

Conversation

@jhoffner

@jhoffner jhoffner commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

The problem

A flow declares its session, user or org state with a schema, and a field with a default (count: z.number().default(0)) is typed as always present. At runtime it wasn't: the stored state was handed to blocks as-is, so on a brand-new user, a record stored before the field existed, or right after a concurrent write, the field read undefined. Every flow either wrote its own ?? default (about 20 of them across workforce, kitchen-sink and two labs) or trusted the type and risked a crash. Kitchen-sink's web-search gate read features.search with no fallback.

What this does

Whenever a run reads a stored scope record, each field of the flow's scope stateSchema that the record doesn't hold now reads as its default. That covers the load at request start, the record a write or a write conflict hands back mid-run, the /state endpoint's clientData, and the session listing.

It only fills gaps. A value already stored is read exactly as stored, and keys the schema doesn't declare are kept, at the top level and inside declared objects. That matters because a user or org record is one blob shared by every flow on the server: a strip-mode parse would have deleted other flows' fields on the next save. Each top-level field is checked on its own, so one field that fails its schema (a session created half-filled on purpose) doesn't block the others from getting their defaults.

The workarounds for missing defaults are removed in the same change.

What's asked of you

Need your sign-off

Should a block's own state declaration also supply defaults at runtime, or only the flow's?

  • Plain terms: this change applies the defaults in the flow's state schema. A block or capability can also declare the state fields it uses (sessionStateSchema on the block). Today those declarations only shape the types; nothing applies their defaults when the program runs, and this PR leaves it that way. So a field that only a block declares, with a default, still reads undefined until something writes it. Our user docs (Fundamentals → State and scopes) describe block declarations as "collected and merged" into the flow, which reads like they'd behave the same way.
  • Trade-off: flow-only (this PR) is one rule in one place, and every flow that declares its state at the top gets the fix now. Applying block declarations too means each block gets a view of state shaped by its own schema. It's a bigger change to how a block sees shared state, and it would land later.
  • My recommendation: ship flow-only now. The docs paragraph is corrected in this PR, so it says block declarations type ctx.session.state and the flow's schema supplies the runtime defaults (the architecture doc in this PR already says so). One kitchen-sink fallback stays for exactly this reason (listed below).
  • What would change my mind: if adopters are meant to declare state only on blocks, as the docs example suggests, then flow-only leaves the main documented path broken, and block-level defaults should come next rather than a docs fix.
  • If I'm wrong: cheap. Flow-level defaults are a strict improvement either way, and adding block-level defaults later changes nothing this PR does.

Parts worth reviewing closely

  • packages/workforce/src/worker-flow-contract.ts: hireWorkforce refuses a manager or coordinator flow whose session stateSchema doesn't default the fields its board and delegates read. The library pieces read those fields without a fallback.

  • packages/engine/src/context/scope-state-defaults.ts: the fill-only merge. It parses per field, keeps stored values and unknown keys, and recurses into objects and same-length arrays.

  • packages/engine/src/stores/scope-persist.ts: a one-field write adopts the record the store returns, and a conflict adopts the store's current record. Both now go through the same read, or the scope would lose its defaults mid-run. The regression test covers each path. An incState / pushState on a field the stored record lacks but the scope reads as a default goes through the version-checked full set, because the store's delta would start it at 0 / [].

  • The session readonly guard now compares against the loaded state, with defaults applied (createExecutionContext.ts, createdWith). Otherwise a defaulted readonly field missing from an old record would look changed on the first write.

  • Intentional: kitchen-sink's flow session schema now declares activeSkills. The skills capability writes it, and a capability's own schema gets no defaults on read, so the modeStatus projection reads it without a fallback. The projection's features now follows the schema default (crawl: false). The removed literal said crawl: true, which disagreed with what the tool gate actually did.

Spec: none, bug on the direct route · Linear: FIX-2554 · Epic: FIX-2447 (specs/epics/FIX-2447/) · Merges: yes, after review

How to review this — altitude, what's in scope, what's deliberately unsettled

How to review this. This is a bug fix on the direct route: there is no spec and no
spec PR, because the hard part was the diagnosis and it happened in code. This PR is the
only gate
, so unlike a spec-backed change, the approach is in scope — nothing here was
signed off upstream.

Most valuable here: the diagnosis chain — does the repro actually reproduce the reported
failure, is the named cause the real one, does the fix address that cause rather than the
symptom, and does the regression test fail without the fix? Then the second path (the legacy
shape, the null boundary, the concurrent case, the cancel path).

In scope to challenge: where the guard was placed, any behaviour change a user could
notice, and whether this should have been a spec'd feature instead of a fix.

Engineering calls
  • Fill-only, not a passthrough parse. The issue suggested passthrough(). That keeps only top-level unknown keys: a nested z.object still strips, so another flow's prefs.lang would vanish when this flow declares prefs: { theme }. A passthrough parse would also run transforms and coercions on values that are already stored. Taking only the values the parse fills in avoids both.
  • Defaults are not written back on their own. They reach the store with the next full-record write. A one-field delta write stores only that field.
  • Request state is unchanged. request.stateSchema defaults were never applied and aren't in this issue.
  • No new public surface. The helper is engine-internal. Core's unwrapToZodObject isn't exported, and this PR doesn't export it.
Everything else — the cleanup checklist, verification

Cleanup checklist (from the issue)

workforce

  • coordinator-flow.ts:1017, :1259, :1447, :1497: fallbacks removed. roundsIn kept, reduced to its BP-030 read: it still drops the retired ended field from rounds saved before the rename. Its ?? [] and begun ?? 0 went (begun defaults inside the array element now).
  • task-settled.ts:200, :355
  • delegate-list.ts currentDelegates: the partial parse and ?? null are gone, and seeding from config is kept.
  • worker-delegates.ts:171
  • test agent-worker-flow.test.ts:604

orchestration

  • activation-store.ts readFieldEntries: kept, type guard included. The field is author-named (activeState: { scope, field }) and may live at request scope or in block state. The library can't assume the author's flow schema defaults it.
  • test apply-skill-activation.test.ts:88. Its mock already seeds activeSkills: [].

apps/kitchen-sink

  • flow.ts modeStatus: the activeSkills, mode, requestCount and features fallbacks are gone (activeSkills is now declared on the session schema). source ?? "tool" stays because source is optional with no default.
  • run/steps.ts:71
  • background-work.ts:186 / alreadyReportedTaskIds: kept. reportedSideChainTaskIds is declared only on the pipeline block's sessionStateSchema, not the flow's, so no default applies (the sign-off above). The legacy-key union stays either way.
  • app/flow-state/page.tsx:360, :362: kept. They cover clientData that hasn't loaded yet (userPrefs is undefined), not a missing default.
  • client-data-bar.tsx:31: kept. Confirmed it's a projection prop, undefined while loading.
  • comments in shared/schemas.ts and flow.ts that said nothing parses session state on load
  • test thinking-styles.test.ts: the comment is fixed, and the projection tests now pass state the way a run loads it.
  • features.ts:181-186: fixed by the engine change. The nested features.search case is in the regression test.

labs/claude-code-style

labs/fsd-coding-skill

  • src/flow.ts:61: optional chaining on harnessSessions removed.

Verification

  • packages/engine/test/scope-state-defaults.test.ts (9 tests): fresh user/session/org, another flow's keys kept through a load and a full-record save, defaults kept after a one-field delta write, a failing field not blocking its siblings, the CAS re-read, the /state projection, and the helper's fill-only rules.
  • Red state: with defaults disabled, 7 of the 8 runtime and helper tests fail. The CAS test reads count: NaN. With only the /state user projection reverted, the projection test fails.
  • Suites: engine 4372 passed, workforce 1778, orchestration 2285, kitchen-sink 472, claude-code-style 172, fsd-coding-skill 39, integration-tests 152.
  • Typecheck is clean for engine, kitchen-sink, claude-code-style and fsd-coding-skill. Workforce shows 11 errors locally, none in files this PR touches; they're in project-source.ts and missing node types.
  • org-record-adoption.test.ts: the seeded org now reads { tier: "free" }, and the stored row is still {}. That's what its comment already claimed ("defaults are applied on read, not baked into the row").

Docs: docs/architecture/state-and-scopes.md → State Operations and apps/docs/.../fundamentals/state-and-scopes.md → Creating sessions. Changeset: @flow-state-dev/engine patch.


Generated by Claude Code

ctx.session/user/org.state is typed z.infer of the flow's scope stateSchema,
but the runtime never parsed it, so a defaulted field read undefined on a fresh
user, an older record, or after a concurrent write. Fill in the defaults on
every read of a stored record: the load, the record a delta write or a CAS
conflict hands back, the /state clientData projection, and the session
listing. Only missing values are filled; stored values and undeclared keys
(another flow's, on a shared user/org blob) are kept as stored.

Removes the fallbacks flows kept for those defaults in workforce,
kitchen-sink, and the claude-code-style and fsd-coding-skill labs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UNkLC9NknMcaqpK1h2mVK
@vercel

vercel Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
flow-state-dev-build-site Ready Ready Preview Nov 5, 2026 2:57am UTC
3 Skipped Deployments
Project Deployment Actions Updated
flow-state-dev-docs Ignored Ignored Nov 5, 2026 2:57am UTC
flow-state-dev-kitchen-sink Ignored Ignored Nov 5, 2026 2:57am UTC
flow-state-dev-ui Ignored Ignored Preview Nov 5, 2026 2:57am UTC

Request Review

@linear-code

linear-code Bot commented Nov 5, 2026

Copy link
Copy Markdown

FIX-2554

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-05T03:00:24.771536Z 8f9bf63 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

jhoffner commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review — main concern: packages/engine/src/context/scope-state-defaults.ts fills defaults by parsing each top-level field and merging only missing values back. Please check it never drops or rewrites a stored value (nested objects, arrays, transforms), and that every path that installs a stored record into a scope container (load, delta-write return, CAS conflict, reread in scope-persist.ts) goes through it.


Generated by Claude Code

jhoffner commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Second look + architecture pass (read-only; for the agent that picks this up)

Verdict: the approach is sound. Fill-only beats passthrough() for the shared user/org blob, and applying the defaults at the read boundary is the right place. Nothing blocks the direction. The items below are worth fixing or consciously declining before merge. I read the diff and spot-checked the code. I did not run the suites.

Worth acting on

  1. Library helpers lost their tolerance for a missing field (BP-030 / BP-035 null boundary).

    • delegate-list.ts currentDelegates and worker-delegates.ts seeded now do === null / !== null on state[DELEGATES_STATE]. Before, undefined was handled. delegateStateShape is declared per block (blockBase = { sessionStateSchema: delegateSessionStateSchema } in worker-delegates.ts:259 and project-coordinator.ts:194) and gets into the flow schema only if the host flow installs it (the coordinator flow and session-board do, via installation.session(...)).
    • This PR's own sign-off says block-level schemas apply no runtime defaults. So any flow that uses defineWorkerDelegates or the project coordinator without installing the shape now gets delegates: undefined, or skips options.seed, where it used to work. project-coordinator.ts:249 still uses == null, so the two disagree.
    • Suggested change: keep == null in these exported helpers. They take an arbitrary state and don't own the flow's schema. Also keep the ended strip in roundsIn, as the PR already does. Removing fallbacks fits app code that owns its schema (kitchen-sink, labs). It does not fit library code that is handed someone else's state.
    • The same applies to task-settled.ts (state[TASK_NOTICES_STATE] as string[] then .includes) and the coordinator-flow.ts ROUNDS_STATE reads. They are safe only if every host flow installs those shapes. Either confirm that with a test, or leave the ?? []. A crash on undefined.includes is a worse failure than a redundant fallback.
  2. modeStatus dropped legacy tolerance it was written for. currentMode: modeSchema.parse(ctx.state.mode ?? "ask") became ctx.state.mode, and Number(requestCount ?? 0) became requestCount. The default fills a missing mode, but a stored garbage or retired mode value used to be coerced or rejected and now reaches the browser raw. The thinkingStyle fold right beside it is the pattern to follow (BP-030). Keep the modeSchema.parse, or say why a bad stored mode can't occur.

  3. Block-level schemas: the sign-off question hides an inconsistency.

    • The kitchen-sink comment this PR removes said capability schemas are "merged at flow registration". The PR now says block and capability schemas are never applied at runtime. The activeSkills addition to the kitchen-sink session schema works around exactly that: the skills capability already declares it.
    • I agree with the recommendation (flow-only now, fix the docs). But the workaround is a smell that will repeat for every capability that owns a field. The follow-up should ask whether the flow-registration merge (capability/merge.ts, which already produces the merged sessionStateSchema) can feed flow.session.stateSchema for defaults. That gives one schema per scope, and block views stay type-only. File the follow-up so the docs-only fix isn't the end of it.
    • Until then, please add the doc correction in the same PR if it's one paragraph. The current user docs now contradict the new "Creating sessions" paragraph.

Architecture: one choke point, not three call sites

  1. The change threads readState through createScopePersist, createScopeReread, and three readXState closures in createExecutionContext.ts, so every place that adopts a stored record has to remember to call it. The comment itself says "or the scope would lose its defaults mid-run". That is the shape that regresses the next time someone adds an adoption path.

    • Alternative: give the state container (or createScopeStateOps) a single normalize option applied wherever it adopts a record (initial, post-persist, CAS re-read). Then scope-persist.ts stays unchanged and the user/session/org closures collapse to one option each.
    • This is a refactor that falls out of the feature, not scope creep. If the container is too deep to touch here, say so and leave the threading as is.
  2. Cost on the write path. Every delta write and CAS re-read now runs safeParse on every top-level field of the whole record, including large arrays such as rounds and notices. If a schema has transforms, refinements, or .default(() => ...) factories, they run on each read. Consider skipping the parse for a field whose stored value is present and whose schema has no nested defaults, or at least note the cost in the helper's header. fillMissing also throws away the parsed result for any present scalar, so the work is wasted exactly where the record is largest.

Smaller

  1. thinking-styles.test.ts builds its own loaded() helper (sessionStateSchema.parse({}) spread), which duplicates the engine rule instead of calling it. It would pass even if applyScopeStateDefaults regressed. Use applyScopeStateDefaults(sessionStateSchema, stored) so the test shares the real rule.
  2. activeSkills: (ctx.state.activeSkills as Array<...>): the schema now declares activeSkillsArraySchema, so the cast should be unnecessary. If it still needs the cast, the types don't line up and that's worth knowing.
  3. Test gaps, per the second-path checklist: no test for a union or discriminated-union field, or a record of defaulted objects. The helper silently leaves these as stored when they fail to parse. A one-line note in the header would set expectations.
  4. Pre-existing, not for this PR: activation-store.ts readFieldEntries is kept for the right reason (author-named field). request.stateSchema defaults stay unapplied, which is a separate gap worth an issue.

Not worth changing

  • Engine-internal helper, no public export, patch changeset: correct.
  • Not writing defaults back is the right call for a shared blob.
  • The crawl: true → false projection fix is a real bug fix, and good to have called out.

Generated by Claude Code

jhoffner commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Architect: not approving yet, with two edits.

  1. The approach holds. One fill function covers load, delta-write and conflict records, the reread, /state, and the session listing, so the browser gets filled values too. Flow-only is the right pick. Block and capability sessionStateSchema is merged only for types today (core/src/capability/merge.ts:509), and the engine never reads it. Merging at runtime would need a rule for blocks whose defaults conflict, which is more mechanism for no gain.
  2. Library entries lose their safety net. taskSettledEntry, the coordinator rounds, and the delegate helpers now crash (task-settled.ts:200, undefined.includes) when an adopter's flow doesn't spread workforce's schema into its own stateSchema. Removing the fallbacks in app and lab code is fine. For library code, either refuse loudly at registration when the flow's schema lacks the entry's keys, or keep the fallbacks there. Jake's goal is to delete the workarounds, so the loud refusal is the closer fit. Please don't leave it crashing silently.
  3. Fix apps/docs/docs/fundamentals/state-and-scopes.md:100 in this PR. Its "collected and merged… without repeating" now contradicts the paragraph this PR adds on the same page. Block declarations type ctx.session.state, and the flow's stateSchema supplies runtime defaults.

Optional: add one line in docs/architecture/state-and-scopes.md saying resources normalize on load while scope state only fills, and why.

— FSD Architect

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/simplify review (FIX-2554)

Problem this PR solves: Flow scope stateSchema types promise defaulted fields are always present, but the engine historically adopted stored JSON with bare casts — so blocks, clientData, and tests needed scattered ?? fallbacks and still missed nested defaults (e.g. features.search inside a partial {}).

Verdict on the engine approach: The shape of the fix is the simple one: one read-path projection wired through initial load, CAS persist/reread, /state clientData, and session listing. That is fewer moving parts than pushing default-fill into every app or block.

The custom applyScopeStateDefaults + fillMissing walk is not over-engineering given constraints this PR documents well:

  • Root schema.parse(stored) would strip keys other flows wrote on shared user/org blobs.
  • Resource-style normalizeResourceState is the opposite contract (whole-row parse or substitute default).
  • Per-field safeParse + merge preserves invalid siblings and avoids re-coercing stored values.

I do not see a materially simpler algorithm that keeps those invariants; the main simplification wins are dedup and PR scope, not deleting the helper.


Where this went farther than necessary

  1. Unrelated epic spec bundle (~780 lines + SVGs under specs/epics/FIX-2447/) landed on the same branch as the engine fix (commits d9d86ad90 / 840e7f6be ahead of the FIX-2554 commit). That makes review, bisect, and rollback noisy. Recommendation: rebase this PR to only the FIX-2554 implementation (+ its changeset/docs/tests/app cleanups), and keep FIX-2447 on its own PR/branch (or merge from main after that epic PR lands separately).

  2. Kitchen-sink activeSkills on flow sessionStateSchema is a real fix (capability-only schemas are not run through applyScopeStateDefaults), but it is a second concern bundled with engine work. Fine as a follow-up commit on the same PR after scope is clean, or called out explicitly in the description so reviewers do not treat it as accidental scope creep.


Targeted simplifications (low risk, same behavior)

Item Suggestion
objectShapeOf Replace with existing unwrapToZodObject + getZodObjectShape from @flow-state-dev/core/helpers (same pattern as sequencer-state, transient-slot).
Comments Module header in scope-state-defaults.ts is good; trim repeated FIX-2554 paragraphs in createExecutionContext / scope-persist JSDoc.
Tests Prefer behavioral describe titles over ticket ids (BP-006); align thinking-styles.test.ts loaded() with applyScopeStateDefaults instead of parse({}) spread so tests match nested-default semantics.
Performance Not a merge blocker for typical runAction load (3× per context). Watch session list paths that call toClientSession per row — optional fast-path only if profiling shows pain.

Alternative approaches considered (and why I would not take them)

  • Leave defaults in apps (?? everywhere): Does not match z.infer, fails nested partial objects, and duplicates flow schema truth.
  • Single root Zod parse on load: Loses cross-flow keys on shared records; tests explicitly guard this.
  • Move algorithm to core now: Reasonable follow-up once a second consumer exists; not required to ship FIX-2554.

Verification

Ran pnpm --filter @flow-state-dev/engine test on this branch — 4373 passed (includes scope-state-defaults integration coverage).


Ask: Drop the FIX-2447 spec commits from this branch (or split PRs), then the engine slice is in good shape. Happy to re-review and approve once the diff matches the FIX-2554 title.

Open in Web View Automation 

Sent by Cursor Automation: Review For Best Approach

* The shape of the `z.object` under a schema's wrappers (`.default({})`,
* `.optional()`, effects), or `undefined` when there is none.
*/
function objectShapeOf(schema: ZodTypeAny): Record<string, ZodTypeAny> | undefined {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

objectShapeOf duplicates unwrapToZodObject in @flow-state-dev/core/helpers (used in sequencer-state, define-capability, transient-slot). Swapping to that helper removes ~10 lines and avoids a third copy of the unwrap loop if another read-path helper appears.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept. unwrapToZodObject isn't exported from @flow-state-dev/core/helpers (only getZodInnerType, getZodObjectShape, getZodArrayElement are), and this PR doesn't widen core's exports. The helper now shares one local unwrap loop between the object and array lookups, so there's still one copy in this file.


Generated by Claude Code


Generated by Claude Code

// type promises. Nothing else of the stored blob changes. Every read of a
// stored record goes through these: the load here, and the records a write
// or a conflict hands back (`createScopePersist`, `createScopeReread`).
const readUserState = (state: unknown) =>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three read*State closures are the right seam for persist/reread — keep them. Optional micro-DRY: a tiny local bindRead(schema) if you touch this again; not worth exporting.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No change, as you suggest: three one-line closures read clearly enough here.


Generated by Claude Code


Generated by Claude Code

* another action's file.
*/
import { z } from "zod";
import { activeSkillsArraySchema } from "@flow-state-dev/orchestration";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving activeSkills onto the flow sessionStateSchema is the correct fix for FIX-2554 (capability sessionStateSchema alone is not default-filled on load). Worth a one-line PR note so reviewers see this as intentional schema wiring, not drive-by kitchen-sink churn.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a line to the PR body saying the activeSkills declaration on the kitchen-sink flow schema is intentional.


Generated by Claude Code


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 65ee4c565e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ok: true,
version: result.version,
record: next.state as TState
record: readState(next.state)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor schema-default baselines in native delta writes

When a stored record lacks a field with a nonzero numeric or nonempty array default, the container now reads that default but the native delta writer still operates on the raw, absent field. Using the real in-memory store, count.default(10) reads as 10 but becomes 1 after incState({ count: 1 }); an array default of ["seed"] becomes ["new"] after pushState("tags", "new"). The store starts missing fields at 0/[], and this adoption preserves its incorrect result. Make delta writes honor the defaulted baseline through concurrency-safe initialization or a version-checked full-write path; otherwise mutations produce different results depending on whether the adapter advertises delta verbs.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced and fixed in 9d9e75c. With the in-memory store, count.default(10) + incState({ count: 1 }) read 1 and tags.default(["seed"]) + pushState("tags", "new") read ["new"]. Now, when the stored record lacks the field but the scope reads it as a default, incField / pushToArray fall through to the version-checked full set, which carries the defaulted value. A conflict re-reads and retries as usual, and once the field is stored, later deltas take the native path again. Tests: native delta writes start from the defaulted value in packages/engine/test/scope-state-defaults.test.ts (both red before the fix).


Generated by Claude Code


Generated by Claude Code

Comment on lines +72 to +75
if (Array.isArray(stored) && Array.isArray(parsed) && stored.length === parsed.length) {
let out: unknown[] | undefined;
stored.forEach((value, i) => {
const filled = fillMissing(value, parsed[i]);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid merging transformed arrays by positional index

When an array field has a transform that reorders elements, equal lengths do not mean the parsed and stored elements correspond. For example, an array of { id: z.string(), flag: z.boolean().default(false) } followed by .transform(rows => rows.reverse()), applied to [{ id: "a", flag: true }, { id: "b" }], makes this helper return [{ id: "a", flag: true }, { id: "b", flag: true }]: the second element inherits the first element's stored flag instead of its own default false. This was reproduced against the helper and can subsequently be persisted by a full-record write. Fill array-element defaults without using the reordered transform output as a positional source.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced and fixed in 9d9e75c. Your example returned [{ id: "a", flag: true }, { id: "b", flag: true }]. Arrays are now filled one element at a time from the element schema (unwrapped through .transform / .default), and the parsed array is never used as a positional source. An array with no element schema to read is left as stored. Test: fills each array element from its own schema, not a transform's reordered output (red before the fix).


Generated by Claude Code


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Snob — already tight. No POC.

Head 65ee4c565e9512e0ec3f34d0a08ade34aa8eb649. Read packages/engine/src/context/scope-state-defaults.ts, packages/engine/src/stores/scope-persist.ts, packages/engine/src/context/createExecutionContext.ts, packages/engine/src/routes/state-routes.ts, packages/engine/src/routes/client-session.ts, packages/engine/test/scope-state-defaults.test.ts, and the fallback removals in workforce, kitchen-sink, labs/claude-code-style/workforce/flows/pins.mts, and labs/fsd-coding-skill/src/flow.ts.

A rewrite would not shrink the contract:

  • fillMissing is the fill-only merge. deepMerge overwrites a stored value and replaces an array, so tags: [{}, { on: false }] would lose the element default on: true. Parse the field, then copy only the keys that were missing. Walking element schemas to skip that merge is longer.
  • Per-field safeParse stays, so one field that fails its schema still lets its siblings get their defaults. The other branch is that same merge for a root with no object shape.
  • objectShapeOf stays. It is the unwrapToZodObject walk plus getZodObjectShape. That helper is not on @flow-state-dev/core/helpers, and this PR already declined exporting it.
  • readUserState, readSessionState, and readOrgState stay. Request state is left raw. The load, the commutative store record, the conflict record, and reread are four reads of one function. A generic around the three closures is the same three calls.
  • /state and the session listing call it again. Those routes read the store on their own.
  • Copy-on-write stays. The stored object comes back when nothing was missing, and a copy comes back when a default is filled, so the store record is left as stored.
  • The fallback deletions are the shrink. roundsIn still drops retired ended. currentDelegates still seeds on null. coalesceThinkingStyle still folds a removed style. source ?? "tool" stays. Block-only fields stay with the sign-off on this PR. pinsUserStateSchema is already one schema, on the flow and the handlers.
  • The file header restates the fill. Cutting it leaves the same functions.
Open in Web View Automation 

Sent by Cursor Automation: Code Snob

…ire refuses an undefaulted manager

- incState/pushState on a field the stored record lacks go through the
  version-checked full set, so they build on the schema default.
- Array elements are filled from the element schema, never positionally
  from a transform's output.
- hireWorkforce refuses a manager or coordinator flow whose session schema
  doesn't default the fields its board and delegates read.
- Docs: block state declarations type state; only the flow schema supplies
  runtime defaults.
- modeStatus keeps parsing the stored mode (BP-030).
- Ticket ids dropped from code comments and test titles.

Refs FIX-2554

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UNkLC9NknMcaqpK1h2mVK
…e-state-applies-its-schemas-defaults-on-load

jhoffner commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Review round 1: what changed in 9d9e75c

Declined: dropping the FIX-2447 spec commits (Cursor's review). They aren't in this diff. The epic spec is on main already (840e7f6), and git diff origin/main...HEAD touches only this fix's files. The branch now also merges current main.

Fixed

  • Delta writes start from the default (Codex P2). Reproduced, fixed, and covered by a test. The details are in the inline reply.
  • Array elements are filled from their own schema (Codex P2). Reproduced, fixed, and covered by a test. The details are in the inline reply.
  • Library pieces refuse at hire instead of crashing on undefined (second look, item 1). hireWorkforce now refuses a worker flow registered as manager or coordinator when its session stateSchema doesn't default the fields those pieces read: delegates, fallback, taskNotices, and, for a coordinator, its rounds, hold and cursor. The refusal names the fields and the fix (session: installation.session(board.sessionStateShape)). The check runs only once the role's entries pass, so a flow gets one sentence per problem. Test: refuses a manager flow whose session schema doesn't default the fields its board reads, naming them in packages/workforce/test/worker-registration.test.ts. The role goal goals/workers/roles-checked-at-hire still passes, including every repo manager and coordinator.
  • The user docs no longer contradict themselves (item 3). "Schema bubbling" now says block declarations type ctx.session.state and only the flow's stateSchema supplies defaults on read. "When to declare state at the flow level" lists that case.
  • modeStatus parses the stored mode again (item 2). A retired mode is refused rather than sent to the browser raw (BP-030).
  • Ticket ids are gone from code comments and test titles. They stay in the changeset.
  • The helper header notes the per-read parse cost and the union/record behaviour (items 5, 8).

Kept, with reasons

  • One normalize on the state container (item 4). I didn't do this refactor here. The three adoption paths each go through readState and are tested, and the delta-write fix needed readState inside scope-persist anyway.
  • thinking-styles.test.ts calling applyScopeStateDefaults (item 6). The helper is internal to the engine, and exporting it would add public surface.
  • The activeSkills cast (item 7). client.derived's ctx.state is typed as JSON, not the schema's type. Removing the cast fails to typecheck.

Verification: engine 4387 passed · workforce 1785 · kitchen-sink 472 · integration-tests 152 · engine, workforce and kitchen-sink typecheck clean.


Generated by Claude Code


Generated by Claude Code

jhoffner commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review: one concern about 9d9e75c. packages/engine/src/stores/scope-persist.ts now sends an incState / pushState through the version-checked full set when the stored record lacks the field but the scope reads it as a default. Please check that a concurrent writer can't lose an increment or push on that path (a conflict re-reads and retries), and that the per-element array fill in scope-state-defaults.ts never rewrites a stored value.


Generated by Claude Code


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d9e75ca19

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

return (readState(ref.current.state) as Record<string, unknown>)[field] !== undefined;
};

if (hint.kind === "incField" && typeof store.incField === "function" && !readsDefault(hint.path[0])) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retry conflicts when defaulted deltas fall back to full writes

When two contexts increment or push onto the same absent, defaulted field, this fallback can silently drop one mutation. The hint remains commutative, so runDurableMutation in state-container.ts dispatches to runCommutative, which persists only once and returns false on conflict without refreshing the container or retrying. Reproduced with the in-memory store: two concurrent increments against count.default(10) return [true, false] and store 11 instead of 12; two pushes against tags.default(["seed"]) likewise retain only one appended value. Route this version-checked fallback through a refresh-and-reapply retry path rather than the one-shot commutative path.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced and fixed in 746b3a7. Two runs incrementing the same absent count.default(10) returned [true, false] and stored 11. Two pushes onto tags.default(["seed"]) lost one value the same way.

The persist callback now retries this case itself instead of going back to the general CAS path. An increment or push is fully described by its hint, so on a conflict it applies the hint again to the store's current record and writes again at that record's version. After the winner's write the field is usually stored, so the retry takes the atomic incField / pushToArray against "any". The retry is capped at 4 attempts. runCommutative is unchanged: it still persists once, and the retry stays inside createScopePersist, which is the only place that knows it chose a version-checked write for a commutative hint.

Tests: two runs incrementing the same defaulted counter both count and two runs pushing onto the same defaulted array both append in packages/engine/test/scope-state-defaults.test.ts. Both failed before the fix and pass after it. The full engine suite passes.


Generated by Claude Code


Generated by Claude Code

The first incState / pushState onto a field the record lacks but the scope
reads as its default goes through a version-checked full set. The caller's
commutative path does not retry, so a concurrent run's write could be
dropped (two increments landed as one). The persist callback now re-applies
the hint to the store's current record on a conflict; once the field is
stored the atomic delta takes over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UNkLC9NknMcaqpK1h2mVK

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 746b3a77ca

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +107 to +108
for (const [key, value] of Object.entries(parsed)) {
const filled = fillMissing(shape?.[key], stored[key], value);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Fill nested object defaults from their original field schemas

When an object transform swaps child objects, this merge fills missing fields from the wrong child's transformed values. For example, let row = z.object({ flag: z.boolean().default(false) }) and declare obj: z.object({ a: row, b: row }).transform(({ a, b }) => ({ a: b, b: a })). Reading stored { obj: { a: { flag: true }, b: {} } } produces b.flag: true instead of its default false. This was reproduced against the helper and can be persisted by a subsequent full-record write. Parse existing children through their original field schemas, as the array branch does, rather than sourcing defaults from the object's transformed output.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced and fixed in 8f9bf63. Your example returned b.flag: true.

The fill no longer copies anything out of a parse. It walks the declared schemas instead: objects by key, arrays by element, records by value, through .optional() / .nullable() / .readonly() / .default(). A .default() supplies its declared value, and a transform fills only through its input schema. So swapping, reordering or resizing in a transform at any level can't change what goes into the record. One side effect: an absent field declared z.string().transform(f).default("anon") now fills with "anon", the stored input form, not f("anon"). That matches how present values are already kept.

Tests: fills a nested object from its own field schemas, not a transform's swapped output and a transform's default fills in its declared value, never the transformed one in packages/engine/test/scope-state-defaults.test.ts. Both failed before the fix.


Generated by Claude Code


Generated by Claude Code

Comment on lines +91 to +92
const element = arrayElementOf(schema);
if (element === undefined) return stored;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Resolve array schemas inside records and union variants

Defaults inside existing array elements are silently skipped when the array is reached through a record or union. For example, z.object({ rows: z.record(z.array(z.object({ flag: z.boolean().default(false) }))) }) applied to { rows: { x: [{}] } } returns the unchanged object, leaving flag undefined even though parsing succeeds and its inferred type requires a boolean. The record recursion loses the value schema through shape?.[key], and a union containing an array similarly cannot yield an element schema here. Preserve the record value schema and resolve the matching union variant before filling array elements so these supported schema compositions receive their defaults.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Record case reproduced and fixed in 8f9bf63. { rows: { x: [{}] } } came back unchanged. Now a z.record is walked through its value schema, so the array elements inside it get their defaults. Test: fills array elements reached through a record's value schema (failed before the fix).

The union case is left as stored on purpose. We're not adding compositions one at a time, so the rule is fixed: the fill walks only objects, arrays, records and their optional/nullable/readonly/default wrappers (a transform through its input). A union, discriminated union, intersection or any other composition is never given a value. Picking a variant means trusting a parse to say which branch matched, and that is the same parse-output path that caused the transform bugs. The cost is that a default inside a union variant can stay unfilled. The gain is that it can never be filled wrong. This is now one line in docs/architecture/state-and-scopes.md, and leaves a union value as stored, even where a variant declares a default pins it.


Generated by Claude Code


Generated by Claude Code

… transform's output (FIX-2554)

Objects, arrays, records and their optional/nullable/readonly/default
wrappers are walked; a transform fills only through its input schema.
Unions and other compositions are left as stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UNkLC9NknMcaqpK1h2mVK

jhoffner commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Review round 3: what changed in 8f9bf63

Both of Codex's P2s on 746b3a7 are fixed under one bounded rule instead of one composition at a time:

  • The fill never takes a value from a transform's output, at any level. applyScopeStateDefaults now walks only the declared schemas: objects by key, arrays by element, z.record by value, through .optional() / .nullable() / .readonly() / .default(). A .default() supplies its declared value, and a transform fills only through its input schema. This covers the object-transform swap and the array inside a record. Both were reproduced red first.
  • Unions, intersections and other compositions are left as stored. A default inside one may stay unfilled, but it is never filled wrong. One line in docs/architecture/state-and-scopes.md says so, and a test pins it.
  • Behaviour change to know about: an absent field shaped X.transform(f).default(d) now fills with d (the stored input form), not f(d).

Engine suite: 330 files, 4393 tests pass. Engine typecheck is clean.

@codex review: one concern about 8f9bf63. fillDefaults in packages/engine/src/context/scope-state-defaults.ts replaces the parse-and-merge with a schema walk. Check whether its wrapper handling (ZodOptional stopping on undefined, ZodNullable on null, ZodDefault filling then walking its inner type) ever fills a value that field.safeParse of the stored record would reject, or misses a default inside a supported object, array or record. Union and other compositions are out of scope by design.


Generated by Claude Code


Generated by Claude Code

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 8f9bf63d0b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

jhoffner commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

CI: Test (3/3) failed on labs/teammates → "shows a coding run as plain steps, the same on every harness". The assertion was Starting on Cursor…, and this PR does not touch it. The same assertion fails on an unrelated branch (claude/project-thread-i8owvt, run 41945982546). Locally on 8f9bf63d0 it passes 5/5 in isolation and 3/3 across the full teammates suite (191 tests).

The likely cause is the test, not this PR. The assertion catches a short-lived empty state: the popup shows its "Starting on …" placeholder only until the run reports its first step. On a busy runner the fake adapter can report steps before the popup renders. I re-ran the failed jobs. The flake is out of scope here and should get its own fix.


Generated by Claude Code

@jhoffner

jhoffner commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

I'm not happy with the split (works for flows, not for blocks) but I get that only type information bubble up to the flow.

I think we need to create a separate POC PR with various ways of solving this. The idea that what comes to mind is that we have a schema registry in the Frameworks Foundation, probably the core package. That registry is responsible for holding its bond to all of the schemas that are being registered either at the flow or the block level. Every time a block is checked, it must register the schema in the registry. That way, we can check if there are any issues, such as incompatible schemas or different default types. Schemas could union so that if one provides new fields, but they don't collide, then that is okay.

An advantage here is that it also makes it possible for us to just register schemas directly, as opposed to having to do it through a block or a flow, if that need ever arises. However, how much overhead this is and how well we're able to reconcile schema paths, I don't know. So I think we would need to explore this with a proof of concept.

Merged via the queue into main with commit 1e4724f Nov 5, 2026
30 of 32 checks passed
@jhoffner
jhoffner deleted the contact/fix-2554-scope-state-applies-its-schemas-defaults-on-load branch November 5, 2026 19:57

jhoffner commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

from: epic-coordinator
session: fix-2447-coordinator
kind: reply

Re the schema-registry ask: the POC is in #4788. The choice is option B: at defineFlow, each flow unions the scope schemas of the blocks and capabilities it reaches. A field declared with a different type or default in two places is refused, and the error names both. A server-admission check covers shared user/org fields. There is no process-wide registry, because the POC showed it leaks defaults across flows and raises false conflicts. This PR merges as-is. B only changes which schema the three read closures pass to the fill, and it's tracked as FIX-2576, which also retires the "only the flow's own stateSchema counts" line and the hireWorkforce restatement refusal.


Generated by Claude Code

jhoffner pushed a commit that referenced this pull request Nov 5, 2026
…IX-2576

FIX-2554 (#4770) shipped the flow stateSchema fill; FIX-2576 owns block-
and capability-declared defaults and the conflict refusal at defineFlow,
and retires the hire refusal. Drop the stale "as FIX-2554 refuses a hire
at registration" clause and name both contributors in the cell and the
manifest figure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UNkLC9NknMcaqpK1h2mVK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants