Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/lead-fit-unsaid-work.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@flow-state-dev/workforce": patch
---

`openWorkstreamTool`'s lead-fit check also asks whether a workstream's title and objectives say what sort of work it is, and no longer refuses work they don't, such as a bare name (FIX-2019).
2 changes: 1 addition & 1 deletion apps/docs/docs/workforce/projects.md
Original file line number Diff line number Diff line change
Expand Up @@ -459,7 +459,7 @@ When `openWorkstream` runs in your own coordinator's turn, the new workstream is

`listLeads` returns every worker that can lead, whether or not it suits the work. A coordinator with a single lead tends to pick it even when its description says it does something else. Set `leadFit` to have the open refuse that pick.

With `leadFit: { model }` set, `openWorkstreamTool` makes one call to that model before it opens, asking whether the lead's description (from its worker file, as `listLeads` shows it) says it does the work in the workstream's title and objectives. When the answer is no, the open is refused with `lead-doesnt-fit` and nothing is written. The refusal's message tells the worker to pick another lead from `listLeads`, or, if none does the work, to open nothing and say a worker for it has to be hired first.
With `leadFit: { model }` set, `openWorkstreamTool` makes one call to that model before it opens, asking whether the lead's description (from its worker file, as `listLeads` shows it) says it does the work in the workstream's title and objectives. When the answer is no, the open is refused with `lead-doesnt-fit` and nothing is written. The same call asks whether the title and objectives say what sort of work it is at all, and work they don't, such as a bare name, passes the check. The refusal's message tells the worker to pick another lead from `listLeads`, or, if none does the work, to open nothing and say a worker for it has to be hired first.
Comment thread
jhoffner marked this conversation as resolved.
Resolved

The check runs only once every other refusal has passed, so a bad id or a lead that can't lead is refused for that reason without a model call. Opening a workstream that is already open, with the same lead, isn't checked again.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -454,8 +454,22 @@ export async function legsAB(o: LegOptions): Promise<LegFacts> {
if (!sameSet(roster1.alice, roster0.alice) || !sameSet(roster1.bob, roster0.bob)) r.fail("b2", `a roster changed: ${show(roster0)} → ${show(roster1)}`);
else r.saw("b2", "nothing was added to either roster");
for (const [person, owner, key] of [[A, true, "alice"], [B, false, "bob"]] as const) {
const drawn = await projectWorkstreams(o.browser, origin(), person, owner, r, `b2-${key}-workstreams`, P);
const entries = await entriesOf(key === "alice" ? alice : bob, P);
// The leads keep working after b2's asks and may edit their objectives while the screen is read, so
// the screen is graded against a store that held still across the read.
const who = key === "alice" ? alice : bob;
let entries = await entriesOf(who, P);
let drawn!: Awaited<ReturnType<typeof projectWorkstreams>>;
let still = false;
for (let tries = 0; tries < 3 && !still; tries++) {
const before = entries;
drawn = await projectWorkstreams(o.browser, origin(), person, owner, r, `b2-${key}-workstreams`, P);
entries = await entriesOf(who, P);
still = JSON.stringify(before) === JSON.stringify(entries);
}
if (!still) {
r.fail("b2", `${person.label}'s P workstreams changed during each of 3 screen reads, so the screen can't be graded against them`);
continue;
}
const want = Object.values(workstreams).map((w) => w.id);
if (drawn.value === undefined) r.fail("b2", `${person.label}'s P Workstreams tab failed: ${drawn.errors.join(" | ")}`);
else {
Expand Down
4 changes: 2 additions & 2 deletions packages/shift-manager/test/project-workstreams.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -87,10 +87,10 @@ function scripted(): ModelResolver {
const fit: GeneratorModel = {
modelId: "test/fit",
async generate(options: unknown) {
return { structuredOutput: { fits: leadFits(JSON.stringify(options)) }, finishReason: "stop" };
return { structuredOutput: { says: true, fits: leadFits(JSON.stringify(options)) }, finishReason: "stop" };
},
async generateStep(options: unknown) {
return { structuredOutput: { fits: leadFits(JSON.stringify(options)) }, finishReason: "stop" };
return { structuredOutput: { says: true, fits: leadFits(JSON.stringify(options)) }, finishReason: "stop" };
},
};
return Object.assign(
Expand Down
8 changes: 5 additions & 3 deletions packages/workforce/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1540,8 +1540,10 @@ with a single lead tends to pick it. To have the open refuse that pick, pass `de
leadFlows, leadFit: { model: "openai/gpt-5.4-mini" } })`: `openWorkstreamTool` then makes one call
to that model asking whether the lead's description (as `listLeads` shows it) says it does the work in
the workstream's title and objectives. If the answer is no, the open is refused with `lead-doesnt-fit` and nothing
is written; the message tells the worker to pick another lead from `listLeads`, or, if none does the work, to open
nothing and say a worker for it has to be hired first. A failed call, or a lead with no description,
is written. The same call asks whether the title and objectives say what sort of work it is at all, and work
they don't, such as a bare name, passes the check. A refusal's message tells the worker to pick another lead
from `listLeads`, or, if none does the work, to open nothing and say a worker for it has to be hired first.
A failed call, or a lead with no description,
lets the open through. The check runs after every other refusal, and an open of a workstream that is
already open with the same lead isn't checked again. The tool's answer carries `leadFit`: `{ checked: true }`,
or `{ checked: false, why }` with `why` one of `already-open`, `no-description` or `unanswered: <error>`.
Expand Down Expand Up @@ -2016,7 +2018,7 @@ the root exports, and reaches no Node built-in.
| `projectMcpTools(servers, { operator?, env? })` / `ProjectMcpServers` / `ProjectMcpToolsResolver` | The resolver a coding harness calls once per run for its project's MCP tools, relayed through FSD's own connection with the run owner's key. See [Projects](#projects). |
| `defineMcpCredentialsCollection()` / `mcpCredentialsCapability` / `mcpCredentialProblem(value)` / `MCP_CREDENTIALS_PREFIX` | Each user's MCP server keys, `mcp-credentials/*` at user scope (client read: `setAt` only), the capability the resolving block holds, and the check a key route runs on a value. |
| `checkedMcpServers(names, offered)` / `isMcpServerName(value)` | The check both project writes run on `mcpServers`. |
| `defineWorkstreamBlocks({ installation, leadFlows, leadFit? })` | Returns `{ openWorkstream, openWorkstreamTool, updateWorkstream, updateWorkstreamTool, editWorkstreamTool, listLeadsTool, actions }`. `editWorkstreamTool` (input `editWorkstreamInputSchema`) is a project coordinator's change to one of the caller's own workstreams by id. `openWorkstreamTool` is the open for a worker; with `leadFit: { model }`, it refuses with `lead-doesnt-fit` when the lead's description doesn't say it does the work. Each lead flow declares `workstreamOpenedEntry()` under `internal.actions[WORKSTREAM_OPENED_ENTRY]`, as the built-in `agent` flow does; one that doesn't throws. See [Projects](#projects). |
| `defineWorkstreamBlocks({ installation, leadFlows, leadFit? })` | Returns `{ openWorkstream, openWorkstreamTool, updateWorkstream, updateWorkstreamTool, editWorkstreamTool, listLeadsTool, actions }`. `editWorkstreamTool` (input `editWorkstreamInputSchema`) is a project coordinator's change to one of the caller's own workstreams by id. `openWorkstreamTool` is the open for a worker; with `leadFit: { model }`, it refuses with `lead-doesnt-fit` when the work says what sort it is and the lead's description doesn't say it does that work. Each lead flow declares `workstreamOpenedEntry()` under `internal.actions[WORKSTREAM_OPENED_ENTRY]`, as the built-in `agent` flow does; one that doesn't throws. See [Projects](#projects). |
| `readProject` / `projectProgress(entries, now)` / `STALE_AFTER_MS` / `DONE_STATUS` | A project's row, entries and progress, and the progress function itself (also from `./browser`): counts by status, objectives met, the next due date, and the entries not done and unchanged for `STALE_AFTER_MS` (seven days). `"done"` is the one status with a meaning. |
| `parseWorkstreamEntry(path, state)` | One entry read off its storage key and checked against `workstreamEntrySchema`, with its `project`, `owner` and `id`; `undefined` for a key that isn't an entry. Also from `./browser`, for a browser reading entries through the resource routes. |
| `projectCoordinatorTools` | `{ readProject }`, the project coordinator's tool: put it in `defineCoordinatorFlow({ agent: { catalog } })` and name it in the coordinator's `tools:`. Add `setBrief` and `setRepository` from `defineProjectBlocks()` for it to manage its own project. |
Expand Down
16 changes: 11 additions & 5 deletions packages/workforce/src/projects/lead-fit.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,10 @@
*
* It is a backstop for clear misfits, not a router: it answers yes unless the
* description says the lead handles a different sort of work, so a lead whose
* description is broad passes. Picking the best lead stays with whoever calls
* `listLeads`.
* description is broad passes. Work that doesn't say what sort it is, such as
* a bare name, can't be a clear misfit, so it passes whatever the judge makes
* of the lead: the judge says whether the work says, and only work that does
* is refused. Picking the best lead stays with whoever calls `listLeads`.
*
* A check that can't answer lets the open through, so a model outage never
* stops a workstream opening, and says why in the open's record (`leadFit`),
Expand Down Expand Up @@ -46,8 +48,11 @@ export type LeadFitRecord = z.infer<typeof leadFitRecordSchema>;

export const leadFitStateSchema = z.object({ [LEAD_FIT_STATE]: leadFitRecordSchema.optional() });

/** The judge's answer: whether the work says what sort it is, and whether the lead fits it. */
const answerSchema = z.object({ says: z.boolean(), fits: z.boolean() });

/** The judge's answer, or why it gave none. */
const judgedSchema = z.union([z.object({ fits: z.boolean() }), z.object({ unanswered: z.string() })]);
const judgedSchema = z.union([answerSchema, z.object({ unanswered: z.string() })]);

type Judged = z.infer<typeof judgedSchema>;

Expand All @@ -73,14 +78,15 @@ export function defineLeadFitCheck(model: string) {
name: "workstream-lead-fit",
model,
inputSchema: leadFitCaseSchema,
outputSchema: z.object({ fits: z.boolean() }),
outputSchema: answerSchema,
Comment thread
jhoffner marked this conversation as resolved.
Resolved
itemVisibility: { client: false, history: false },
repair: { coerce: false },
// A backstop for clear misfits, not a router: don't tighten it into picking the best lead.
prompt: () =>
[
"You check whether a worker is the right one to lead a workstream.",
"A lead never does the work itself: it hands the work to the workers under it. So judge only what sort of work this is.",
'Answer `says: false` when the work doesn\'t say what sort of work it is, such as a bare name or "the requested work", and `says: true` when it does.',
"Answer `fits: true` when the work, such as a feature, research or planning, is the sort the worker's description says it leads or handles.",
"Answer `fits: false` only when its description says it handles a different sort of work."
].join("\n"),
Expand All @@ -100,7 +106,7 @@ export function defineLeadFitCheck(model: string) {
await ctx.request.patchState({ [LEAD_FIT_STATE]: { checked: false, why: `unanswered: ${judged.unanswered}` } });
return {};
}
if (!judged.fits) {
if (judged.says && !judged.fits) {
Comment thread
jhoffner marked this conversation as resolved.
Resolved
Comment thread
jhoffner marked this conversation as resolved.
Resolved
const fitCase = leadFitCaseSchema.parse(ctx.parent?.input);
throw new ProjectRefusedError(
"lead-doesnt-fit",
Expand Down
3 changes: 2 additions & 1 deletion packages/workforce/src/projects/workstream-writes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,8 @@ export type WorkstreamBlocks = {
/**
* `openWorkstream` for a worker to call: with `leadFit` set, it first checks
* the lead's description says it does the work, and refuses `lead-doesnt-fit`
* when it doesn't. Without `leadFit`, it is `openWorkstream`.
* when it doesn't. Work that doesn't say what sort it is is never refused.
* Without `leadFit`, it is `openWorkstream`.
*/
openWorkstreamTool: BlockDefinition<
typeof openWorkstreamInputSchema,
Expand Down
18 changes: 15 additions & 3 deletions packages/workforce/test/project-coordinator.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -572,8 +572,8 @@ describe("opening a workstream from the coordinator (BR-38, BR-39, BR-40, BR-42)
});

describe("checking its pick of lead against the work", () => {
/** The fit check's one call, answering `fits`. */
const fit = (fits: boolean) => mockGenerator({ script: [{ structuredOutput: { fits } }] });
/** The fit check's one call, answering `fits` for work that says (by default) what sort it is. */
const fit = (fits: boolean, says = true) => mockGenerator({ script: [{ structuredOutput: { says, fits } }] });
Comment thread
jhoffner marked this conversation as resolved.
Resolved
const research = { id: "research", title: "Research: what to build next", lead: "alice-lead", objectives: ["Rank three ideas"] };
/** What the open tool answered in a conversation's turns. */
const opens = async (h: Awaited<ReturnType<typeof boot>>, sessionId: string) =>
Expand Down Expand Up @@ -616,6 +616,18 @@ describe("opening a workstream from the coordinator (BR-38, BR-39, BR-40, BR-42)
expect(await opens(h, id)).toContain('"leadFit":{"checked":true}');
});

it("opens it when the work doesn't say what sort it is, whatever the judge makes of the lead: that is no clear misfit", async () => {
// The work names no sort, so no lead can be the wrong sort for it.
const unsaid = { id: "garnet", title: "garnet delta 6965", lead: "alice-lead", objectives: [] };
const h = await boot({ judgment: scripted([{ toolName: "openWorkstream", args: unsaid }]), leadFit: fit(false, false) });
await hireOn(h, "alice", "alice-lead", "lead", ["alice-lead.helper"]);
const id = await h.coordinator("alice");
await h.act("alice", COORDINATOR_FLOW, id, "run", { message: "Open a workstream for garnet delta 6965." });
await h.settled();
expect(await opens(h, id)).not.toContain("lead-doesnt-fit");
expect(await workstreams(h)).toEqual(["garnet"]);
});

it("opens it when the check can't answer, so a failed model call never stops an open, and says why", async () => {
const h = await boot({ judgment: scripted([{ toolName: "openWorkstream", args: research }]), leadFit: mockGenerator({ script: [] }) });
await hireOn(h, "alice", "alice-lead", "lead", ["alice-lead.helper"]);
Expand All @@ -642,7 +654,7 @@ describe("opening a workstream from the coordinator (BR-38, BR-39, BR-40, BR-42)

it("doesn't check a workstream opened again: it was accepted once, and a second verdict can't refuse it", async () => {
// The first open is judged a fit; asked again, the judge would say no, but it is never asked.
const leadFit = mockGenerator({ script: [{ structuredOutput: { fits: true } }, { structuredOutput: { fits: false } }] });
const leadFit = mockGenerator({ script: [{ structuredOutput: { says: true, fits: true } }, { structuredOutput: { says: true, fits: false } }] });
const h = await boot({
judgment: scripted([
{ toolName: "openWorkstream", args: research },
Expand Down
Loading