Skip to content

Check a coordinator's pick of lead against the work, and give the DevTeam a general lead (FIX-1920) - #3263

Merged
jhoffner merged 7 commits into
mainfrom
claude/project-thread-v9ry2p
Oct 13, 2026
Merged

jhoffner merged 7 commits into
mainfrom
claude/project-thread-v9ry2p

Conversation

@jhoffner

@jhoffner jhoffner commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Before: asked to open a research workstream, the DevTeam project coordinator opened it under the EM, the only lead on the roster, even though the EM's description says it files features for the coder. Telling it in its instructions to open nothing when no lead fits didn't change that.

After: research and planning open under a new general lead, and feature work still opens under the EM. When a worker picks a lead whose description says it does other work, the open is refused with lead-doesnt-fit and nothing is written. A user's own open from the app is never checked.

This is the follow-up to #3230, built from Jake's "Both" answer on the lead-fit decision: check fit in code and add a general lead.

How:

  • @flow-state-dev/workforce: defineWorkstreamBlocks takes an optional leadFit: { model } and returns openWorkstreamTool. Before opening, the tool makes one small model call that compares the lead's description with the workstream's title and objectives. It refuses lead-doesnt-fit on a no. If the call fails, or the lead has no description, the open goes through. Without leadFit, the tool is the same block as openWorkstream.
  • DevTeam:
    • The host passes leadFit and gives the chief of staff and the project coordinator openWorkstreamTool.
    • New org workers: general-lead, on agent, which delegates to researcher, and researcher, on agent.
    • The project coordinator's instructions name the general lead for research and planning, and say what to do after a lead-doesnt-fit refusal.
  • Docs:
    • The workforce README and the Projects page document leadFit, openWorkstreamTool and the new refusal.
    • A changeset for the workforce package is included.

Evidence:

  • Workforce tests (project-coordinator.test.ts):
    • a misfit lead is refused and nothing is written;
    • a fitting lead opens;
    • a failed check lets the open through;
    • a user's open from the app is never checked.
    • Making the check never refuse fails the misfit test. Making it fail closed fails the failed-check test.
  • Shift Manager test (project-workstreams.test.tsx): the coordinator's research open under the EM is refused, and it then opens under the general lead and hands the ask in. The test fails with leadFit removed from the host, and fails with the general lead removed.
  • Real model, end to end (DevTeam, openai/gpt-5.4-mini, 3 runs each of four asks):
    • 11 of 12 asks opened a workstream, every one under the right lead: research and the Q3 roadmap under the general lead, guest checkout under the EM.
    • In the one other run, the coordinator answered with project status instead of opening anything.
  • Real model, the check alone (exact prompt, 3 runs each pairing):
    • the EM is refused for research 6 of 6 times and passes for guest checkout 3 of 3;
    • the general lead passes for research and the roadmap 9 of 9 and is refused for guest checkout 3 of 3.
    • Known gap: the check lets the EM through for a "Q3 roadmap" open. In practice the coordinator picked the general lead for that ask every time.
  • Local suites:
    • Workforce and Shift Manager suites are green apart from tests that are also red on main: split-down-the-chain in workforce, and task-screen in Shift Manager, which passes run alone.
    • Typecheck, check-retired-terms, the changeset check and the docs build pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9


Generated by Claude Code


Note

Medium Risk
Introduces a model-dependent gate on coordinator workstream opens (fail-open on errors) and changes DevTeam routing; core project write paths are extended but user openWorkstream actions are unchanged.

Overview
Adds optional lead-fit gating when coordinators open workstreams, plus DevTeam workers so research/planning no longer default to the EM.

@flow-state-dev/workforce: defineWorkstreamBlocks now accepts leadFit: { model } and returns openWorkstreamTool (worker/catalog path) alongside unchanged openWorkstream (user/app actions). With leadFit, the tool runs one structured model check after other validations and before any write; a clear mismatch refuses lead-doesnt-fit and records leadFit on success (checked / why). Failed checks, missing lead descriptions, and re-opens skip or bypass the judge; direct user opens are never checked. Without leadFit, openWorkstreamTool equals openWorkstream.

DevTeam: Host enables leadFit and wires chief of staff / project coordinator to openWorkstreamTool + listLeads. New general-lead (research/planning) and researcher workers; coordinator and chief-of-staff instructions describe picking leads and recovering from refusals.

Docs & tests: Projects docs, workforce README, and changeset document the API; workforce and Shift Manager tests cover refusal, pass-through on outage, idempotent re-open, and app opens bypassing the check.

Reviewed by Cursor Bugbot for commit d18c4b2. Configure here.

…a general lead to the DevTeam (FIX-1920)

A worker's openWorkstream now asks one small model call whether the lead's
description says it does the work, and is refused lead-doesnt-fit when it
doesn't. The DevTeam gains a general lead and a researcher, so research and
planning have a lead that fits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9
…r it does the work itself (FIX-1920)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9
…the fit check without retired terms (FIX-1920)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9
@vercel

vercel Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
flow-state-dev-docs Ignored Ignored Oct 13, 2026 10:13pm UTC
flow-state-dev-kitchen-sink Ignored Ignored Oct 13, 2026 10:13pm UTC
flow-state-dev-ui Ignored Ignored Preview Oct 13, 2026 10:13pm UTC

Request Review

@linear-code

linear-code Bot commented Oct 13, 2026

Copy link
Copy Markdown

FIX-1920

@cursor

cursor Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: ddc8cc16-670e-4907-8e40-bad405472769)

@jhoffner

Copy link
Copy Markdown
Collaborator Author

@codex review

Copy link
Copy Markdown
Collaborator Author

Second-look + architecture review (read-only, from the diff)

The "Both" call is already made, so I'm not re-opening it. These are findings about how the check is built and whether it is proven. Ordered by what I'd act on.

Worth fixing in this PR

  1. The check runs before the open's own validation, and on idempotent re-opens. workstream-open-checked is fitCase → judgeFit → fitVerdict → openWorkstream, so the model call happens before invalid-workstream-id, not-a-member, no-such-worker and too-many-workstreams are evaluated. Two consequences:

    • Error precedence changes. A bad id plus a misfit lead now returns lead-doesnt-fit, not invalid-workstream-id. Every doomed open also pays for a model call.
    • Re-opening a workstream that already exists with the same lead is documented as a no-op (opened: false). With the check in front, that retry can now be refused if the verdict flips. The check is nondeterministic and the lead's description can change. A retry of a workstream that was already accepted should not be able to fail on a re-roll.

    Suggest running the fit check after the cheap structural refusals and skipping it when the entry already exists with that lead. That likely means placing it inside openWorkstream's write path, behind a flag, rather than in front of it. That is also less code than the current sequencer of three handlers plus a generator.

  2. fitVerdict passes the "skipped" case by accident. When stepIf skips judgeFit (no description), fitVerdict receives the previous step's output, a FitCase. It lets the open through only because fitVerdictSchema.safeParse(FitCase) fails and .data?.fits is undefined. If anyone adds a fits field to FitCase, or loosens the schema, the skipped path starts behaving differently. It also reads open back through ctx.getBlockOutput(fitCase) rather than carrying it along. Make the "no verdict" path explicit (for example the skipped branch yields { fits: null }) and thread open through instead of reaching into earlier output.

  3. A failed check is silent. .rescue([{ block: fitUnanswered }]) swallows every error, including schema and config errors such as a bad model id, and the generator is itemVisibility: { client: false, history: false }. If the model is misconfigured, the feature turns itself off with no trace anywhere (BP-035, cost and observability path). Fail-open is a defensible call, so keep it, but emit something (a log or a hidden-but-recorded item) saying "fit check unanswered: ". Otherwise the first sign of a dead check is the original bug coming back.

Smaller / structural

  1. workstream-writes.ts is now doing two jobs. About 110 lines of fit-check machinery sit inside defineWorkstreamBlocks. The file header already carries four separate contracts. A leaf module (lead-fit.ts) exporting withLeadFit(open, { model }) would keep workstream-writes.ts about the write and make the check unit-testable without booting the whole installation. This is the one place where this PR is a natural opportunity to refactor, so I'd do it only if item 1 is reworked anyway.

  2. The roster lookup happens twice per open. fitCase calls installation.rosterWorker and leadProblem, then openWorkstream does the same. If item 1 is taken, the second read goes away. If not, at least share the lookup.

  3. Evidence gaps.

    • No ablation separates the two changes. The 11/12 real-model result is with both leadFit and the general lead present. The report doesn't say how often the coordinator picks the right lead with only the general lead plus the prompt change (no leadFit). If that is already near 12/12, the check is mostly insurance, and the user should know that is the trade (one extra model call per open, a known false-pass on "Q3 roadmap"). One run of that arm would settle it.
    • The shift-manager test stubs the verdict with !prompt.includes("eng.em"). That proves wiring, not judgement, and breaks if a description ever mentions another worker's id. The real-model numbers live only in the PR body, so nothing guards the prompt against regression. A small committed eval of the 4 asks × 2 leads would keep the check honest when the prompt or model changes.
  4. Nit: prompt vs. the known gap. The judge prompt tells the model to refuse "only when its description says it handles a different sort of work", and the PR already shows the EM passes for a roadmap. That leniency is by design, but the coordinator WORKER.md now has to carry the other half (general lead for planning). Worth a one-line comment in judgeFit saying the check is a backstop for clear misfits, not a router, so nobody tightens it into one.

Not flagged

Changeset (patch, workforce only), docs, README and refusal-reason additions are consistent with each other. The openWorkstreamTool === openWorkstream identity when leadFit is unset keeps existing hosts unchanged.


Generated by Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-13T21:51:28.953860Z 3c254d7 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplify review (FIX-1920)

Problem: When the project coordinator opens a workstream, it often picks the only (or default) lead even when that worker’s description says it does a different kind of work—e.g. research under the EM.

What this PR does: Adds optional leadFit on defineWorkstreamBlocks, exposes openWorkstreamTool (worker/catalog path) with a small structured generator gate, new refusal lead-doesnt-fit, plus general-lead / researcher workers and devteam wiring. User openWorkstream actions stay unchecked. Tests cover refuse / accept / model-failure pass-through / app bypass.

Verdict: The FSD shape (sequencer + generator + .rescue fail-open + ProjectRefusedError) matches existing workforce patterns (noCoordinatorYet, bestFitEvaluationFailed abort semantics). I would not call this over-abstracted for the framework layer. The open question is product layering: whether a public leadFit API is the smallest fix for the devteam failure mode, or whether most of the win could have come from roster + coordinator prose alone, with deterministic metadata only if misfires persist.

Simpler approaches worth weighing (not necessarily better)

Approach Fit for devteam (EM vs general-lead) Fit for arbitrary installations
This PR — LLM check on worker open Strong; gives structured retry + lead-doesnt-fit Strong when descriptions are free text
Structured worker metadata (e.g. workstreamKinds: [research, feature] on WORKER.md) + rule or filter on listLeads Likely enough for a fixed roster; no per-open token/latency Needs schema, hire validation, migration story
Prompt / WORKER.md only (no new block) Weak when only one lead is offered—the bug you fixed Same
Reuse coordinator placeBestFit / route evaluator Wrong shape (multi-option routing, not binary gate before a write) N/A

I’m not asking you to rip this out without evidence—but if FIX-1920’s scope was “make devteam pick general-lead for research,” a tags + deterministic match path would be a materially smaller diff and ongoing cost. Keeping optional leadFit for installations that live on prose-only descriptions is a reasonable compromise if you expect consumers beyond shift-manager.

What looks good

  • Splitting openWorkstream (user action) vs openWorkstreamTool (worker catalog) is clear and tested.
  • stepIf(description !== null) avoids pointless model calls when there’s nothing to judge.
  • Fail-open on generator failure preserves availability; abort still propagates via fitUnanswered.
  • Refusal lands on the existing ProjectRefusalReason surface; docs/changeset/tests move together.

Nits / optional tighten-ups (non-blocking)

  • openWorkstreamTool needs an as WorkstreamBlocks[...] cast because the sequenced block doesn’t unify with the plain handler—worth a follow-up typing pass so consumers don’t inherit ZodTypeAny leakage.
  • fitCase + checkOpen both call rosterWorker for the same lead on the happy path—probably fine if the resource is request-cached; otherwise a one-line comment or threading the resolved lead would reduce duplicate work.
  • Fail-open when description is null (missing description or lead already failing leadProblem) means “no description” and “model down” both bypass fit—intentional per docs, but worth confirming that’s what you want for leads without description: in YAML.
  • Devteam enables leadFit on every lab open—expected cost is one mini call per coordinator open attempt (retries in the integration test hit the check twice). Fine if deliberate; document for operators if not already obvious outside the diff.

No changeset concerns from this review; the fragment matches the published API shift.

Recommendation: Safe to merge from a correctness/convention standpoint if you’re comfortable committing to the LLM gate as the generic workforce story. If the epic intent was narrower, consider a follow-up issue to explore structured lead kinds and whether leadFit should stay devteam-only for a while.

Open in Web View Automation 

Sent by Cursor Automation: Review For Best Approach

@jhoffner

Copy link
Copy Markdown
Collaborator Author

Architect: not approving yet — your second-look items 1–3 are still open (fit check runs before structural refusals / can refuse an idempotent re-open; fitVerdict treats a skipped judgeFit as pass only because safeParse fails; fail-open is silent with no recorded reason). Fix those three in this PR (or say they can slip to a follow-up) and I'll label once CI is also green.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Snob — already tight. No POC.

Head 3c254d7dd9ddca9d4d98baaa659dbe2cbfd8673d.

I read packages/workforce/src/projects/workstream-writes.ts, project-refusal.ts, packages/shift-manager/teams/devteam/host.mts, the general-lead, researcher, and project-coordinator worker files (and the EM file they sit next to), packages/workforce/test/project-coordinator.test.ts, packages/shift-manager/test/project-workstreams.test.tsx, packages/workforce/README.md, apps/docs/docs/workforce/projects.md, and .changeset/workstream-lead-fit.md.

fitCase is the one lookup that skips the model when the lead has no description or cannot lead. judgeFit is the one call. fitUnanswered rethrows a cancel and otherwise answers nothing, which is the fail-open. fitVerdict refuses only when fits is false. stepIf is why that handler's input is unknown, and getBlockOutput(fitCase) is the open when the judge did not run. You already kept the factory (no fit block unless leadFit is set), the cast, and that read.

Inlining the generator builds it for every lab. A handler that calls the model loses the workstream-lead-fit name the tests mock. Putting the check inside checkOpen changes which refusal wins, and whether opening the same workstream again is checked. A lead-fit.ts moves the lines. The three coordinator tests repeat boot the way the rest of that file does. generate and generateStep on the scripted fit model return the same object; a helper is the same length. The README and the projects page are the two readers. The general lead's last paragraph matches the EM's because each worker file is the whole prompt. The changeset is one sentence.

Left as recorded: the factory, the cast, getBlockOutput, fail-open with no description, and the check on every Lab.

Open in Web View Automation 

Sent by Cursor Automation: Code Snob

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3c254d7dd9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

});

const judgeFit = (model: string) =>
generator({

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Model the lead-fit classification as an evaluator

When leadFit is enabled, this binary model result directly controls whether the write is refused, making it the classification/branching use case assigned to the evaluator block by docs/architecture/overview.md. Implementing it as a generator gives the decision the wrong runtime and DevTool identity and creates a second classification shape beside Workforce's existing evaluator-based best-fit routing. Use an evaluator question and branch on its answer, retaining the intended rescue behavior for evaluation failures.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping the generator. The convention is right in principle, so I measured both on real models before deciding.

The test set was 12 cases (EM and general lead × 3 features and 3 research/planning items), each run twice:

  • Evaluator (typesafe-ai/jev, the evaluation model the DevTeam already routes with), one boolean question with the same instruction: 14/24 at p ≥ 0.5.
    • The EM scored 0.64–0.74 on research it should refuse, against 0.75–0.82 on features.
    • The only threshold that separates the two groups is 0.745, a margin of 0.01. That would be overfit to this set.
  • Current generator (openai/gpt-5.4-mini, { fits }): 21/24.

Workforce's best-fit routing hits the same gap: its evaluator falls back to a generator call (coordinator-route-ask). A fit check that can't tell research from a feature would refuse good opens or pass bad ones, which is worse than the convention cost. If an evaluation model separates these cases later, swapping the judge only touches lead-fit.ts.


Generated by Claude Code

…cord why

The lead-fit check moves to its own leaf module and runs only once the
open has passed every structural refusal, so a bad id or a lead that can't
lead never costs a model call. A re-open of a workstream already open with
the same lead isn't checked again. The tool's answer now carries leadFit:
{checked:true}, or {checked:false, why} for already-open, no-description or
an unanswered model call, so a failing check is no longer silent.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9

Copy link
Copy Markdown
Collaborator Author

Second-look items 1–3 are fixed in 5df5e0c, along with 4, 5 and 7.

  1. Ordering. The open is now split into a check step (every structural refusal: id, member, roster, cannot-lead, lead-differs, too-many) and a make step. The fit check sits between them and takes its case from the check step's output, so it can't run before a refusal. A re-open of a workstream already open with the same lead skips the check. Tests: a bad id is refused invalid-workstream-id with no model call, and a re-open makes no second call.
  2. Skipped judge counted as a pass. The verdict no longer infers anything from safeParse. The judge's rescue returns an explicit { unanswered }, and the verdict branches on it.
  3. Silent fail-open. The tool's answer now carries leadFit: { checked: true }, or { checked: false, why } where why is already-open, no-description or unanswered: <error>. A test checks for the unanswered: record.
  4. The check now lives in its own leaf module, projects/lead-fit.ts.
  5. The split also removed the double roster lookup.
  6. There's a comment saying the check is a backstop, not a router.

Negative controls: checking re-opens fails the re-open test, and dropping the unanswered record fails the failed-check test. Both were run and the code restored. Workforce passes 24/24. The shift-manager workstream, goal-labs and tool-name suites pass, and both packages typecheck. The docs (README and projects.md) cover the new ordering, the re-open skip and the leadFit record.

Item 6 (the ablation arm and a committed eval) can wait for a follow-up so it doesn't hold this PR. The real-model numbers from the earlier run stand: right lead 11 of 12, judge alone 21 of 24.


Generated by Claude Code

@cursor

cursor Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 0525fef9-ce4b-496e-a6b3-f68284fbb1b0)

…he general lead

The checked open tells a refused worker to pick another lead from
listLeads, but the chief of staff's tools line didn't name it. It does now,
and its instructions say to pick the lead from it and what to do on a
lead-doesnt-fit refusal.

The general lead's description advertised any work that isn't building
software, but its one delegate, the researcher, only writes up findings.
It now leads research and planning, where the result is a write-up.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9
@cursor

cursor Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: f100fe40-bf20-445b-8832-724b43690ae7)

…d-terms check

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJZMq1XGNsUeycFS4dXqh9
@cursor

cursor Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: f9729a9e-5eec-4486-a8fc-21bc1d3ba3d1)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants