Repository navigation
fix(core,workforce): evaluators work on gateway 4.0.108+, and best fit's record says when its evaluator failed (FIX-1925) - #3188
Conversation
… and best fit's record says when its evaluator failed (FIX-1925) @ai-sdk/gateway 4.0.108 and later read an evaluation's state as decision parts, which only ai 7.0.133 and later send. Core allowed ai 7.0.103, so an app with a current gateway got "state.map is not a function" from every evaluator call. Raise the ai floor to 7.0.133 across the workspace, and send an array state as one json part so the SDK doesn't read it as parts. mockEvaluationModel hands its answers function, and records, the state the evaluator evaluated rather than the SDK's parts. A best-fit routing record now carries evaluationFailed, the evaluator's error, when that call failed and best fit asked again as a language model. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012k4ApttNtvbmHmX8SvjdyS
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: d77310be-3d72-4103-9303-d95959902e39) |
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Second look + architecture notes (read-only review; no code changed)Overall the diagnosis is sound and the diff is small and well-evidenced. A few points for the implementing agent to weigh, most important first. 1. Alternative approach: send parts explicitly instead of version-gating on
2. 3. 4. Minor.
Verdict: correct fix for the stated bug. Items 1 and 3 are worth resolving before merge; 2 is a cheap cleanup that rides this PR's testing-package change. Generated by Claude Code |
There was a problem hiding this comment.
Simplify review (FIX-1925)
Verdict: The core fix is about as small as it can be for the two failures this PR targets. I’m approving; what follows is consolidation and clarity, not a different architecture.
What broke and why this shape fits
- Gateway 4.0.108+ decision models expect evaluator
stateas decision parts; olderai+ raw state producedstate.map is not a functionon the gateway path. - FSD still allows array
EvaluationInputmeaning “one logical JSON array to evaluate,” which the SDK/gateway can treat as multiple parts unless you send a single json part.
The PR’s split is right:
- Bump
aito ≥7.0.133 — lets the SDK normalize string/object state into parts (verified:evaluator.test.tsBR-4 + decision-model test pass onai@7.0.138). - Array-only wrap in
runEvaluation— one line at the seam that ownsexperimental_evaluate; bumping alone would not preserve “array = one payload.”
I don’t see a cleaner third-party escape hatch: experimental_evaluate still passes state through to doEvaluate/doDecide; there’s no public “state → parts” helper to reuse instead of the array wrap.
What I’d not do differently
- Workforce
evaluationFailedon route records —tapIf+patchState+recordOfis idiomatic here (BP-012).fitalone can’t carry “evaluator blew up, then LM succeeded” whenby: "evaluated"still applies. A separate optional field is the straightforward audit surface. - Coupling FIX-1925 evaluator fix + route observability in one PR — reasonable if both ship for the same incident; not extra machinery on the core path.
Optional simplifications (non-blocking)
- Export the mock’s
evaluatedState(or a thin wrapper) from@flow-state-dev/testingand use it incapture-evaluator.test.ts/faceted-collection.test.tsinstead of localtextOf/ inline[0]?.textcasts — same behavior today, less drift when json-part bodies show up. - Rename the local predicate
evaluationFailedincoordinator-flow.tstoisFailedEvaluationso it doesn’t shadow thePostState/ record field name. - Name the array wrap in
evaluate.ts(e.g.stateForSdkEvaluation) so the asymmetry “SDK handles string/object; we pre-wrap arrays” is obvious at the call site. - Two mock contracts are intentional — core’s hand-rolled mock records SDK wire shape;
@flow-state-dev/testingdenormalizes for scripts. Worth one cross-link in comments so future edits don’t “fix” BR-4 by unwrapping in the wrong place.
Checks
pnpm install + pnpm --filter @flow-state-dev/core test -- evaluator.test.ts and pnpm --filter @flow-state-dev/workforce test -- coordinator-routing.test.ts — all green on this branch.
No changeset concerns from this review angle; the fragments match the public surfaces.
Sent by Cursor Automation: Review For Best Approach
…acets guide and two goal controls (FIX-1925) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012k4ApttNtvbmHmX8SvjdyS
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: e491ffe7-338f-4ede-aea8-821cd99a5d16) |
There was a problem hiding this comment.
Code Snob. Already tight on dd80e01. No POC.
Files: packages/core/src/models/evaluate.ts, packages/core/src/blocks/evaluator.ts, packages/core/test/evaluator.test.ts, packages/testing/src/mocks/mockEvaluationModel.ts, packages/engine/test/faceted-collection.test.ts, packages/memory/test/capture-evaluator.test.ts, packages/workforce/src/best-fit.ts, packages/workforce/src/coordinator/coordinator-flow.ts, packages/workforce/src/coordinator/coordinator-route.ts, packages/workforce/test/coordinator-routing.test.ts.
The array wrap is the one ternary in runEvaluation. A named helper around it is a new function. The decision-model test's .map is the gateway call (state.map is not a function). doEvaluate on that model is what isEvaluationModel checks. Folding the three states into the BR-4 mock drops that throw.
evaluatedState stays private. textOf is one line used three times in the memory tests. The engine cast is one line. Exporting the unwrap and calling it from both is the same length.
noteEvaluationFailed is the tap that writes the error onto the post before the language-model ask, and recordOf spreads it the way it spreads note. failedEvaluationSchema is { failed: string }, so putting that field on the ask's output makes a successful pick read as a failure.
Sent by Cursor Automation: Code Snob
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dd80e01dcf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
….133 pairs with (FIX-1925) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012k4ApttNtvbmHmX8SvjdyS
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 63db89c5-f020-43ef-9563-849c9c38c417) |
|
I don't think this PR caused it. The same shard passed on dd80e01, and the only changes since then are kitchen-sink's gateway range, one docs line and a local rename in the workforce coordinator flow, none of which touch shift-manager's task screen. The file also passes 3 times out of 3 locally on this head (20/20). I'll re-run the failed job once when this run finishes. If it fails again, I'll treat it as real and fix it here. Generated by Claude Code |
|
@claude merge conflicts |
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012k4ApttNtvbmHmX8SvjdyS
|
I merged main into the branch in bc4f0a3. The only conflict was in Generated by Claude Code |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 80f39395-fcef-468e-a4dc-7b64438b53ec) |


Requested by Jake · project thread
Before: in an app that installed a current
@ai-sdk/gateway(4.0.108 or later), every evaluator call threwTypeError: state.map is not a function. Under best fit, the language-model retry then picked the worker, and the routing record saidby: "evaluated"with nothing showing the evaluator had failed. Run 3 of the FIX-1797 closure hit this in J1.After: evaluators work with those gateway versions. A best-fit record whose evaluator call failed, and which asked again as a language model, carries
evaluationFailedwith the evaluator's error.Why it broke. Gateway 4.0.108 and later read an evaluation's state as decision parts. Only
ai7.0.133 and later send parts. Core acceptedai7.0.103, and the repo resolved 7.0.114, which passes the state object as is. Core loads the gateway from the app, so the app's newer gateway met the repo's olderai.How
airange to^7.0.133everywhere it is declared (core, engine, tools, kitchen-sink, hello-chat). The lockfile now resolves 7.0.138.runEvaluationsends an array state as one json part. Without that, the new SDK reads an array as parts and the model never sees the array.mockEvaluationModelgives itsanswersfunction andcallsthe value the evaluator evaluated, not the SDK's parts, so existing scripts keep working. Three hand-rolled test models in engine and memory now read the text part.recordOfaddsevaluationFailedto every record for that post. Documented incoordinators.mdand the workforce README.Evidence
ai7.0.114 it fails with the samecall.state.map is not a function. After the change it passes for string, object and array states.createGatewaydecision model, with a stubbed fetch: the gateway postsstateParts: [{type:"json", value:{recent, post}}]and the evaluator returns its pick.evaluationFailedon the language-route records and its absence on a normal evaluated record. With the tap removed, both language-route tests fail.origin/mainworktree, and nothing new. The one shift-manager failure (command.test.ts, asset path) also fails on main.Changesets: core and testing patch (the
aifloor and the mock's state), workforce patch (the record field).🤖 Generated with Claude Code
https://claude.ai/code/session_012k4ApttNtvbmHmX8SvjdyS
Generated by Claude Code
Note
Medium Risk
Changes the evaluator→SDK state wire format and coordinator routing records; misalignment with app
ai/gateway versions could still break evaluation, but scope is narrow and well-tested.Overview
Fixes evaluator calls against Vercel AI Gateway 4.0.108+, which expect evaluation state as decision parts (
state.map), while olderaiversions passed raw objects and triggeredstate.map is not a function.Core/testing: Raises the
aifloor to ^7.0.133 (core, engine, tools, examples). InrunEvaluation, array evaluator state is sent as a single{ type: "json", value }part so list-shaped state still reaches the model intact.mockEvaluationModelunwraps text/json parts oncallsand foranswerscallbacks so tests keep seeing the logical evaluated value.Workforce: When best-fit’s evaluator fails and routing retries as a language model, a tap persists the error on post state and every
coordinator-routerecord can include optionalevaluationFailed(documented in coordinators docs and README). Routing tests assert the field on LM-fallback paths and its absence on a normal evaluated route.Apps/deps: Kitchen-sink bumps
@ai-sdk/gatewayto ^4.0.108; docs note the new minimum gateway version for evaluation.Reviewed by Cursor Bugbot for commit bc4f0a3. Configure here.