Skip to content

fix(core,workforce): evaluators work on gateway 4.0.108+, and best fit's record says when its evaluator failed (FIX-1925) - #3188

Merged
jhoffner merged 4 commits into
mainfrom
claude/project-thread-gwlxsh
Oct 12, 2026
Merged

jhoffner merged 4 commits into
mainfrom
claude/project-thread-gwlxsh

Conversation

@jhoffner

@jhoffner jhoffner commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Requested by Jake · project thread

Before: in an app that installed a current @ai-sdk/gateway (4.0.108 or later), every evaluator call threw TypeError: state.map is not a function. Under best fit, the language-model retry then picked the worker, and the routing record said by: "evaluated" with nothing showing the evaluator had failed. Run 3 of the FIX-1797 closure hit this in J1.

After: evaluators work with those gateway versions. A best-fit record whose evaluator call failed, and which asked again as a language model, carries evaluationFailed with the evaluator's error.

Why it broke. Gateway 4.0.108 and later read an evaluation's state as decision parts. Only ai 7.0.133 and later send parts. Core accepted ai 7.0.103, and the repo resolved 7.0.114, which passes the state object as is. Core loads the gateway from the app, so the app's newer gateway met the repo's older ai.

How

  • Raised the ai range to ^7.0.133 everywhere it is declared (core, engine, tools, kitchen-sink, hello-chat). The lockfile now resolves 7.0.138.
  • runEvaluation sends an array state as one json part. Without that, the new SDK reads an array as parts and the model never sees the array.
  • mockEvaluationModel gives its answers function and calls the value the evaluator evaluated, not the SDK's parts, so existing scripts keep working. Three hand-rolled test models in engine and memory now read the text part.
  • Workforce: when best fit's evaluator call fails and it asks again, a tap keeps the error on the post's request state, and recordOf adds evaluationFailed to every record for that post. Documented in coordinators.md and the workforce README.

Evidence

  • New core test with a model shaped like the gateway's decision model. On ai 7.0.114 it fails with the same call.state.map is not a function. After the change it passes for string, object and array states.
  • Scratch run against the real createGateway decision model, with a stubbed fetch: the gateway posts stateParts: [{type:"json", value:{recent, post}}] and the evaluator returns its pick.
  • Routing tests assert evaluationFailed on the language-route records and its absence on a normal evaluated record. With the tap removed, both language-route tests fail.
  • Suites green: core 2359, testing 160, workforce 1238, engine, tools 481, memory 547, orchestration 1749, integration-tests 153. The workforce typecheck shows the same three errors on a clean origin/main worktree, and nothing new. The one shift-manager failure (command.test.ts, asset path) also fails on main.

Changesets: core and testing patch (the ai floor and the mock's state), workforce patch (the record field).

🤖 Generated with Claude Code

https://claude.ai/code/session_012k4ApttNtvbmHmX8SvjdyS


Generated by Claude Code


Note

Medium Risk
Changes the evaluator→SDK state wire format and coordinator routing records; misalignment with app ai/gateway versions could still break evaluation, but scope is narrow and well-tested.

Overview
Fixes evaluator calls against Vercel AI Gateway 4.0.108+, which expect evaluation state as decision parts (state.map), while older ai versions passed raw objects and triggered state.map is not a function.

Core/testing: Raises the ai floor to ^7.0.133 (core, engine, tools, examples). In runEvaluation, array evaluator state is sent as a single { type: "json", value } part so list-shaped state still reaches the model intact. mockEvaluationModel unwraps text/json parts on calls and for answers callbacks so tests keep seeing the logical evaluated value.

Workforce: When best-fit’s evaluator fails and routing retries as a language model, a tap persists the error on post state and every coordinator-route record can include optional evaluationFailed (documented in coordinators docs and README). Routing tests assert the field on LM-fallback paths and its absence on a normal evaluated route.

Apps/deps: Kitchen-sink bumps @ai-sdk/gateway to ^4.0.108; docs note the new minimum gateway version for evaluation.

Reviewed by Cursor Bugbot for commit bc4f0a3. Configure here.

… and best fit's record says when its evaluator failed (FIX-1925)

@ai-sdk/gateway 4.0.108 and later read an evaluation's state as decision
parts, which only ai 7.0.133 and later send. Core allowed ai 7.0.103, so an
app with a current gateway got "state.map is not a function" from every
evaluator call. Raise the ai floor to 7.0.133 across the workspace, and send
an array state as one json part so the SDK doesn't read it as parts.

mockEvaluationModel hands its answers function, and records, the state the
evaluator evaluated rather than the SDK's parts.

A best-fit routing record now carries evaluationFailed, the evaluator's
error, when that call failed and best fit asked again as a language model.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012k4ApttNtvbmHmX8SvjdyS
@vercel

vercel Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
flow-state-dev-docs Ignored Ignored Oct 12, 2026 9:50pm UTC
flow-state-dev-kitchen-sink Ignored Ignored Oct 12, 2026 9:50pm UTC
flow-state-dev-ui Ignored Ignored Preview Oct 12, 2026 9:50pm UTC

Request Review

@linear-code

linear-code Bot commented Oct 12, 2026

Copy link
Copy Markdown

FIX-1925

@jhoffner jhoffner self-assigned this Oct 12, 2026
@cursor

cursor Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: d77310be-3d72-4103-9303-d95959902e39)

@jhoffner

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-12T21:06:40.841013Z dd80e01 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copy link
Copy Markdown
Collaborator Author

Second look + architecture notes (read-only review; no code changed)

Overall the diagnosis is sound and the diff is small and well-evidenced. A few points for the implementing agent to weigh, most important first.

1. Alternative approach: send parts explicitly instead of version-gating on ai. (the big one)
The root cause is that core hands the SDK an implicit state shape and the SDK's interpretation changed. This PR fixes it by raising the ai floor in 5 packages and special-casing arrays in runEvaluation. The special case is the tell: core now knows about the SDK's array-means-parts rule for one shape but still relies on the SDK to wrap strings and objects. Normalising all three in runEvaluation (string → {type:"text"}, object/array → {type:"json"}) would make core's contract with the SDK explicit and version-independent, and then:

  • the Array.isArray branch disappears (one rule, not two);
  • the ai floor bump, and the lockfile churn (new gateway/provider/provider-utils majors) may be unnecessary. Caveat I could not verify: whether ai <7.0.133 passes an array of parts through untouched. node_modules isn't installed in this run. The author has a 7.0.114 worktree and can answer in a minute; the existing new core test fails on 7.0.114 for the old shape, so running it with explicit parts is the check.
    If 7.0.114 passes parts through, I'd prefer this shape. If it re-wraps them (double parts), the bump + array special case is the right fix and this comment can be ignored.

2. mockEvaluationModel.evaluatedState is a lossy inverse.
It unwraps only a lone text/json part. A state that is legitimately a single-element array of {type:"text",...}-shaped objects would be mis-unwrapped; more practically, three hand-rolled models (engine faceted-collection, memory capture-evaluator ×3) now each re-implement "read the text part" (textOf, inline cast). That is the duplicate-helper pattern CLAUDE.md warns against. Since the mock already owns the unwrap, export it from @flow-state-dev/testing (e.g. evaluatedState) and have those tests use it, rather than three local casts. Small, and it keeps the next SDK shape change a one-file fix.

3. evaluationFailed can go stale across rounds.
recordOf(post) adds evaluationFailed to every record for that post, and the tap writes it to POST_STATE. Confirm that POST_STATE is reset per round (or per post) and not carried into a later round where the evaluator succeeded; otherwise a later by: "evaluated" record would wrongly claim a failure. The tests only cover single-round posts. Worth one multi-round assertion (fail in round 0, succeed in round 1, no evaluationFailed on the second record).

4. Minor.

  • noteEvaluationFailed is a handler used purely for state mutation; per BP-012 .tap() is the intended shape, which the PR does, good. The as never casts on patchState match neighbouring code, so I'd leave them.
  • tapIf(evaluationFailed, …) and stepIf(evaluationFailed, …) parse the same value twice. Acceptable; only worth folding if a third use appears.
  • Changeset wording for core mentions "sends an evaluator's state the way those versions read it": fine for consumers. The workforce changeset cites FIX-1925 in a published package changelog; BP-006 and the user-docs rule keep tracking IDs out of published prose, though changesets are listed as an allowed place for them in CLAUDE.md, so no change needed.
  • Pre-existing, not for this PR: the workforce typecheck has 3 errors on main and shift-manager command.test.ts fails on main; both out of scope here.

Verdict: correct fix for the stated bug. Items 1 and 3 are worth resolving before merge; 2 is a cheap cleanup that rides this PR's testing-package change.


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplify review (FIX-1925)

Verdict: The core fix is about as small as it can be for the two failures this PR targets. I’m approving; what follows is consolidation and clarity, not a different architecture.

What broke and why this shape fits

  1. Gateway 4.0.108+ decision models expect evaluator state as decision parts; older ai + raw state produced state.map is not a function on the gateway path.
  2. FSD still allows array EvaluationInput meaning “one logical JSON array to evaluate,” which the SDK/gateway can treat as multiple parts unless you send a single json part.

The PR’s split is right:

  • Bump ai to ≥7.0.133 — lets the SDK normalize string/object state into parts (verified: evaluator.test.ts BR-4 + decision-model test pass on ai@7.0.138).
  • Array-only wrap in runEvaluation — one line at the seam that owns experimental_evaluate; bumping alone would not preserve “array = one payload.”

I don’t see a cleaner third-party escape hatch: experimental_evaluate still passes state through to doEvaluate/doDecide; there’s no public “state → parts” helper to reuse instead of the array wrap.

What I’d not do differently

  • Workforce evaluationFailed on route records — tapIf + patchState + recordOf is idiomatic here (BP-012). fit alone can’t carry “evaluator blew up, then LM succeeded” when by: "evaluated" still applies. A separate optional field is the straightforward audit surface.
  • Coupling FIX-1925 evaluator fix + route observability in one PR — reasonable if both ship for the same incident; not extra machinery on the core path.

Optional simplifications (non-blocking)

  1. Export the mock’s evaluatedState (or a thin wrapper) from @flow-state-dev/testing and use it in capture-evaluator.test.ts / faceted-collection.test.ts instead of local textOf / inline [0]?.text casts — same behavior today, less drift when json-part bodies show up.
  2. Rename the local predicate evaluationFailed in coordinator-flow.ts to isFailedEvaluation so it doesn’t shadow the PostState / record field name.
  3. Name the array wrap in evaluate.ts (e.g. stateForSdkEvaluation) so the asymmetry “SDK handles string/object; we pre-wrap arrays” is obvious at the call site.
  4. Two mock contracts are intentional — core’s hand-rolled mock records SDK wire shape; @flow-state-dev/testing denormalizes for scripts. Worth one cross-link in comments so future edits don’t “fix” BR-4 by unwrapping in the wrong place.

Checks

pnpm install + pnpm --filter @flow-state-dev/core test -- evaluator.test.ts and pnpm --filter @flow-state-dev/workforce test -- coordinator-routing.test.ts — all green on this branch.

No changeset concerns from this review angle; the fragments match the public surfaces.

Open in Web View Automation 

Sent by Cursor Automation: Review For Best Approach

…acets guide and two goal controls (FIX-1925)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012k4ApttNtvbmHmX8SvjdyS
@cursor

cursor Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: e491ffe7-338f-4ede-aea8-821cd99a5d16)

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Snob. Already tight on dd80e01. No POC.

Files: packages/core/src/models/evaluate.ts, packages/core/src/blocks/evaluator.ts, packages/core/test/evaluator.test.ts, packages/testing/src/mocks/mockEvaluationModel.ts, packages/engine/test/faceted-collection.test.ts, packages/memory/test/capture-evaluator.test.ts, packages/workforce/src/best-fit.ts, packages/workforce/src/coordinator/coordinator-flow.ts, packages/workforce/src/coordinator/coordinator-route.ts, packages/workforce/test/coordinator-routing.test.ts.

The array wrap is the one ternary in runEvaluation. A named helper around it is a new function. The decision-model test's .map is the gateway call (state.map is not a function). doEvaluate on that model is what isEvaluationModel checks. Folding the three states into the BR-4 mock drops that throw.

evaluatedState stays private. textOf is one line used three times in the memory tests. The engine cast is one line. Exporting the unwrap and calling it from both is the same length.

noteEvaluationFailed is the tap that writes the error onto the post before the language-model ask, and recordOf spreads it the way it spreads note. failedEvaluationSchema is { failed: string }, so putting that field on the ask's output makes a successful pick read as a failure.

Open in Web View Automation 

Sent by Cursor Automation: Code Snob

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dd80e01dcf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/kitchen-sink/package.json Resolved
….133 pairs with (FIX-1925)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012k4ApttNtvbmHmX8SvjdyS
@cursor

cursor Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 63db89c5-f020-43ef-9563-849c9c38c417)

Copy link
Copy Markdown
Collaborator Author

Test (1/3) failed on 2a4a22e in one test: shift-manager › task-screen.test.tsx › "marks nothing as waiting on a task whose run asks nothing". findByTestId("shell") timed out while the page still showed "Reading the Lab…".

I don't think this PR caused it. The same shard passed on dd80e01, and the only changes since then are kitchen-sink's gateway range, one docs line and a local rename in the workforce coordinator flow, none of which touch shift-manager's task screen. The file also passes 3 times out of 3 locally on this head (20/20). I'll re-run the failed job once when this run finishes. If it fails again, I'll treat it as real and fix it here.


Generated by Claude Code

@jhoffner

Copy link
Copy Markdown
Collaborator Author

@claude merge conflicts

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012k4ApttNtvbmHmX8SvjdyS

Copy link
Copy Markdown
Collaborator Author

I merged main into the branch in bc4f0a3. The only conflict was in packages/workforce/README.md. I kept both edits: the evaluationFailed sentence from this PR and main's paragraph on reading records through listSessionRequests. The lockfile still installs with --frozen-lockfile and resolves ai 7.0.138. The core, testing and workforce tests pass on the merged head. CI is re-running.


Generated by Claude Code

@cursor

cursor Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 80f39395-fcef-468e-a4dc-7b64438b53ec)

@jhoffner
jhoffner merged commit 19bcc39 into main Oct 12, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants