Repository navigation
Check a coordinator's pick of lead against the work, and give the DevTeam a general lead (FIX-1920) - #3263
Conversation
…a general lead to the DevTeam (FIX-1920) A worker's openWorkstream now asks one small model call whether the lead's description says it does the work, and is refused lead-doesnt-fit when it doesn't. The DevTeam gains a general lead and a researcher, so research and planning have a lead that fits. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9
…r it does the work itself (FIX-1920) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9
…the fit check without retired terms (FIX-1920) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: ddc8cc16-670e-4907-8e40-bad405472769) |
|
@codex review |
Second-look + architecture review (read-only, from the diff)The "Both" call is already made, so I'm not re-opening it. These are findings about how the check is built and whether it is proven. Ordered by what I'd act on. Worth fixing in this PR
Smaller / structural
Not flaggedChangeset (patch, workforce only), docs, README and refusal-reason additions are consistent with each other. The Generated by Claude Code |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
Simplify review (FIX-1920)
Problem: When the project coordinator opens a workstream, it often picks the only (or default) lead even when that worker’s description says it does a different kind of work—e.g. research under the EM.
What this PR does: Adds optional leadFit on defineWorkstreamBlocks, exposes openWorkstreamTool (worker/catalog path) with a small structured generator gate, new refusal lead-doesnt-fit, plus general-lead / researcher workers and devteam wiring. User openWorkstream actions stay unchecked. Tests cover refuse / accept / model-failure pass-through / app bypass.
Verdict: The FSD shape (sequencer + generator + .rescue fail-open + ProjectRefusedError) matches existing workforce patterns (noCoordinatorYet, bestFitEvaluationFailed abort semantics). I would not call this over-abstracted for the framework layer. The open question is product layering: whether a public leadFit API is the smallest fix for the devteam failure mode, or whether most of the win could have come from roster + coordinator prose alone, with deterministic metadata only if misfires persist.
Simpler approaches worth weighing (not necessarily better)
| Approach | Fit for devteam (EM vs general-lead) | Fit for arbitrary installations |
|---|---|---|
| This PR — LLM check on worker open | Strong; gives structured retry + lead-doesnt-fit |
Strong when descriptions are free text |
Structured worker metadata (e.g. workstreamKinds: [research, feature] on WORKER.md) + rule or filter on listLeads |
Likely enough for a fixed roster; no per-open token/latency | Needs schema, hire validation, migration story |
| Prompt / WORKER.md only (no new block) | Weak when only one lead is offered—the bug you fixed | Same |
Reuse coordinator placeBestFit / route evaluator |
Wrong shape (multi-option routing, not binary gate before a write) | N/A |
I’m not asking you to rip this out without evidence—but if FIX-1920’s scope was “make devteam pick general-lead for research,” a tags + deterministic match path would be a materially smaller diff and ongoing cost. Keeping optional leadFit for installations that live on prose-only descriptions is a reasonable compromise if you expect consumers beyond shift-manager.
What looks good
- Splitting
openWorkstream(user action) vsopenWorkstreamTool(worker catalog) is clear and tested. stepIf(description !== null)avoids pointless model calls when there’s nothing to judge.- Fail-open on generator failure preserves availability; abort still propagates via
fitUnanswered. - Refusal lands on the existing
ProjectRefusalReasonsurface; docs/changeset/tests move together.
Nits / optional tighten-ups (non-blocking)
openWorkstreamToolneeds anas WorkstreamBlocks[...]cast because the sequenced block doesn’t unify with the plain handler—worth a follow-up typing pass so consumers don’t inheritZodTypeAnyleakage.fitCase+checkOpenboth callrosterWorkerfor the same lead on the happy path—probably fine if the resource is request-cached; otherwise a one-line comment or threading the resolved lead would reduce duplicate work.- Fail-open when
descriptionis null (missing description or lead already failingleadProblem) means “no description” and “model down” both bypass fit—intentional per docs, but worth confirming that’s what you want for leads withoutdescription:in YAML. - Devteam enables
leadFiton every lab open—expected cost is one mini call per coordinator open attempt (retries in the integration test hit the check twice). Fine if deliberate; document for operators if not already obvious outside the diff.
No changeset concerns from this review; the fragment matches the published API shift.
Recommendation: Safe to merge from a correctness/convention standpoint if you’re comfortable committing to the LLM gate as the generic workforce story. If the epic intent was narrower, consider a follow-up issue to explore structured lead kinds and whether leadFit should stay devteam-only for a while.
Sent by Cursor Automation: Review For Best Approach
|
Architect: not approving yet — your second-look items 1–3 are still open (fit check runs before structural refusals / can refuse an idempotent re-open; |
There was a problem hiding this comment.
Code Snob — already tight. No POC.
Head 3c254d7dd9ddca9d4d98baaa659dbe2cbfd8673d.
I read packages/workforce/src/projects/workstream-writes.ts, project-refusal.ts, packages/shift-manager/teams/devteam/host.mts, the general-lead, researcher, and project-coordinator worker files (and the EM file they sit next to), packages/workforce/test/project-coordinator.test.ts, packages/shift-manager/test/project-workstreams.test.tsx, packages/workforce/README.md, apps/docs/docs/workforce/projects.md, and .changeset/workstream-lead-fit.md.
fitCase is the one lookup that skips the model when the lead has no description or cannot lead. judgeFit is the one call. fitUnanswered rethrows a cancel and otherwise answers nothing, which is the fail-open. fitVerdict refuses only when fits is false. stepIf is why that handler's input is unknown, and getBlockOutput(fitCase) is the open when the judge did not run. You already kept the factory (no fit block unless leadFit is set), the cast, and that read.
Inlining the generator builds it for every lab. A handler that calls the model loses the workstream-lead-fit name the tests mock. Putting the check inside checkOpen changes which refusal wins, and whether opening the same workstream again is checked. A lead-fit.ts moves the lines. The three coordinator tests repeat boot the way the rest of that file does. generate and generateStep on the scripted fit model return the same object; a helper is the same length. The README and the projects page are the two readers. The general lead's last paragraph matches the EM's because each worker file is the whole prompt. The changeset is one sentence.
Left as recorded: the factory, the cast, getBlockOutput, fail-open with no description, and the check on every Lab.
Sent by Cursor Automation: Code Snob
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3c254d7dd9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| }); | ||
|
|
||
| const judgeFit = (model: string) => | ||
| generator({ |
There was a problem hiding this comment.
Model the lead-fit classification as an evaluator
When leadFit is enabled, this binary model result directly controls whether the write is refused, making it the classification/branching use case assigned to the evaluator block by docs/architecture/overview.md. Implementing it as a generator gives the decision the wrong runtime and DevTool identity and creates a second classification shape beside Workforce's existing evaluator-based best-fit routing. Use an evaluator question and branch on its answer, retaining the intended rescue behavior for evaluation failures.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Keeping the generator. The convention is right in principle, so I measured both on real models before deciding.
The test set was 12 cases (EM and general lead × 3 features and 3 research/planning items), each run twice:
- Evaluator (
typesafe-ai/jev, the evaluation model the DevTeam already routes with), onebooleanquestion with the same instruction: 14/24 at p ≥ 0.5.- The EM scored 0.64–0.74 on research it should refuse, against 0.75–0.82 on features.
- The only threshold that separates the two groups is 0.745, a margin of 0.01. That would be overfit to this set.
- Current generator (
openai/gpt-5.4-mini,{ fits }): 21/24.
Workforce's best-fit routing hits the same gap: its evaluator falls back to a generator call (coordinator-route-ask). A fit check that can't tell research from a feature would refuse good opens or pass bad ones, which is worse than the convention cost. If an evaluation model separates these cases later, swapping the judge only touches lead-fit.ts.
Generated by Claude Code
…cord why
The lead-fit check moves to its own leaf module and runs only once the
open has passed every structural refusal, so a bad id or a lead that can't
lead never costs a model call. A re-open of a workstream already open with
the same lead isn't checked again. The tool's answer now carries leadFit:
{checked:true}, or {checked:false, why} for already-open, no-description or
an unanswered model call, so a failing check is no longer silent.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9
|
Second-look items 1–3 are fixed in 5df5e0c, along with 4, 5 and 7.
Negative controls: checking re-opens fails the re-open test, and dropping the unanswered record fails the failed-check test. Both were run and the code restored. Workforce passes 24/24. The shift-manager workstream, goal-labs and tool-name suites pass, and both packages typecheck. The docs (README and projects.md) cover the new ordering, the re-open skip and the Item 6 (the ablation arm and a committed eval) can wait for a follow-up so it doesn't hold this PR. The real-model numbers from the earlier run stand: right lead 11 of 12, judge alone 21 of 24. Generated by Claude Code |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 0525fef9-ce4b-496e-a6b3-f68284fbb1b0) |
…he general lead The checked open tells a refused worker to pick another lead from listLeads, but the chief of staff's tools line didn't name it. It does now, and its instructions say to pick the lead from it and what to do on a lead-doesnt-fit refusal. The general lead's description advertised any work that isn't building software, but its one delegate, the researcher, only writes up findings. It now leads research and planning, where the result is a write-up. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: f100fe40-bf20-445b-8832-724b43690ae7) |
…d-terms check Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: f9729a9e-5eec-4486-a8fc-21bc1d3ba3d1) |


Before: asked to open a research workstream, the DevTeam project coordinator opened it under the EM, the only lead on the roster, even though the EM's description says it files features for the coder. Telling it in its instructions to open nothing when no lead fits didn't change that.
After: research and planning open under a new general lead, and feature work still opens under the EM. When a worker picks a lead whose description says it does other work, the open is refused with
lead-doesnt-fitand nothing is written. A user's own open from the app is never checked.This is the follow-up to #3230, built from Jake's "Both" answer on the lead-fit decision: check fit in code and add a general lead.
How:
@flow-state-dev/workforce:defineWorkstreamBlockstakes an optionalleadFit: { model }and returnsopenWorkstreamTool. Before opening, the tool makes one small model call that compares the lead's description with the workstream's title and objectives. It refuseslead-doesnt-fiton a no. If the call fails, or the lead has no description, the open goes through. WithoutleadFit, the tool is the same block asopenWorkstream.leadFitand gives the chief of staff and the project coordinatoropenWorkstreamTool.general-lead, onagent, which delegates toresearcher, andresearcher, onagent.lead-doesnt-fitrefusal.leadFit,openWorkstreamTooland the new refusal.Evidence:
project-coordinator.test.ts):project-workstreams.test.tsx): the coordinator's research open under the EM is refused, and it then opens under the general lead and hands the ask in. The test fails withleadFitremoved from the host, and fails with the general lead removed.openai/gpt-5.4-mini, 3 runs each of four asks):split-down-the-chainin workforce, andtask-screenin Shift Manager, which passes run alone.check-retired-terms, the changeset check and the docs build pass.🤖 Generated with Claude Code
https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9
Generated by Claude Code
Note
Medium Risk
Introduces a model-dependent gate on coordinator workstream opens (fail-open on errors) and changes DevTeam routing; core project write paths are extended but user
openWorkstreamactions are unchanged.Overview
Adds optional lead-fit gating when coordinators open workstreams, plus DevTeam workers so research/planning no longer default to the EM.
@flow-state-dev/workforce:defineWorkstreamBlocksnow acceptsleadFit: { model }and returnsopenWorkstreamTool(worker/catalog path) alongside unchangedopenWorkstream(user/app actions). WithleadFit, the tool runs one structured model check after other validations and before any write; a clear mismatch refuseslead-doesnt-fitand recordsleadFiton success (checked/why). Failed checks, missing lead descriptions, and re-opens skip or bypass the judge; direct user opens are never checked. WithoutleadFit,openWorkstreamToolequalsopenWorkstream.DevTeam: Host enables
leadFitand wires chief of staff / project coordinator toopenWorkstreamTool+listLeads. Newgeneral-lead(research/planning) andresearcherworkers; coordinator and chief-of-staff instructions describe picking leads and recovering from refusals.Docs & tests: Projects docs, workforce README, and changeset document the API; workforce and Shift Manager tests cover refusal, pass-through on outage, idempotent re-open, and app opens bypassing the check.
Reviewed by Cursor Bugbot for commit d18c4b2. Configure here.