Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/calm-owls-research.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"@flow-state-dev/patterns": minor
---

Add an installable `research()` kit that plans multi-step web research, uses search/fetch/crawl tools, retries isolated task failures, and returns a provenance-checked cited report with its typed task results (FIX-2822).

Retrieval provenance and attempt counts reset for each run. Set `maxAttemptsPerTask: 1` or `maxTotalRetries: 0` to disable task retries.
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
import { createFlowState, inMemoryStores } from "@flow-state-dev/engine";
import { handler } from "@flow-state-dev/core";
import { research } from "@flow-state-dev/patterns/research";
import { createMockModelResolver, mockGenerator } from "@flow-state-dev/testing";
import { z } from "zod";

const url = "https://fixture.example/research";
let fetchAttempts = 0;
const search = handler({ name: "search", inputSchema: z.any(), execute: (input) => ({ query: input.query, results: [{ title: "Fixture", url, snippet: "Evidence", source: "tavily" }] }) });
const fetch = handler({
name: "fetch", inputSchema: z.any(),
execute: (input) => {
fetchAttempts += 1;
if (fetchAttempts === 1 || process.env.GOAL_CONTROL === "no-rescue") throw new Error("injected flaky fetch");
return { url: input.url, title: "Fixture", markdown: "The fixture documents the result.", metadata: { statusCode: 200, contentType: "text/html", wordCount: 6 }, source: "builtin" };
},
});
const crawl = handler({ name: "crawl", inputSchema: z.any(), execute: (input) => ({
rootUrl: input.url,
pages: [{ url: input.url, title: "Fixture section", markdown: "More fixture evidence.", metadata: { statusCode: 200, contentType: "text/html", wordCount: 3 }, source: "builtin" }],
totalPages: 1, crawlDepth: 1, source: "builtin",
}) });

const invented = process.env.GOAL_CONTROL === "invent-citation";
const models = createMockModelResolver({
generators: {
"research-planner": mockGenerator({ name: "planner", script: [{ structuredOutput: { tasks: [{ id: "evidence", title: null, goal: "Find and read the fixture evidence", context: null }] } }] }),
"research-executor": mockGenerator({ name: "executor", script: [
{ toolCalls: [{ toolCallId: "s1", toolName: "search", args: { query: "fixture evidence", maxResults: 5, topic: "general" } }, { toolCallId: "f1", toolName: "fetch", args: { url } }] },
{ toolCalls: [{ toolCallId: "s2", toolName: "search", args: { query: "fixture evidence", maxResults: 5, topic: "general" } }, { toolCallId: "f2", toolName: "fetch", args: { url } }, { toolCallId: "c2", toolName: "crawl", args: { url, maxPages: 2, maxDepth: 1 } }] },
{ structuredOutput: { summary: "The fixture documents the result.", success: true, reason: "", sources: [{ title: "Fixture", url: process.env.GOAL_CONTROL === "invent-executor-source" ? "https://invented.example/executor" : url, method: "fetch" }] } },
] }),
"research-evaluate-llm": mockGenerator({ name: "evaluator", script: [{ structuredOutput: { decision: "complete", reasoning: "enough evidence" } }] }),
"research-synthesizer": mockGenerator({ name: "synthesizer", script: [{ structuredOutput: {
question: "What does the fixture document?", summary: "The fixture documents the result.",
findings: [{ claim: "The fixture documents the result.", citations: [invented ? "https://invented.example/report" : url] }],
sources: [{ title: "Fixture", url: invented ? "https://invented.example/report" : url }],
} }] }),
},
policy: "error",
});

export default createFlowState({
kits: [research({ model: "fixture/model", toolOverrides: { search, fetch, crawl }, maxAttemptsPerTask: 2 })],
root: import.meta.url,
stores: { default: { primary: inMemoryStores() } },
modelResolver: models,
});
12 changes: 12 additions & 0 deletions goals/kits/research-returns-a-cited-report/goal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# kits › research returns a cited report

**Outcome:** An app installs `research()` and receives a cited report from a planned run that searches, fetches, crawls, and survives one flaky fetch.
**Input:** The fixture app asks “What does the fixture document?”. Its search, page and crawl are deterministic retrieval fixtures; the first fetch throws and the retry succeeds.
**Signal:** `fsdev run` returns a report with a non-empty finding and citation list, every citation occurs in the returned retrieved-source list, and the preserved task state has one completed task and its typed result.
**Anti-game:** The check grades the running app's captured action output, not the kit factory. The first fetch always fails, so a pass proves task rescue. Every graded list is floored with `tell.over`.
**Model:** Scripted generators drive planning, tool calls, evaluation and synthesis; no vendor key or live web is used.
**Run:** `pnpm tsx goals/kits/research-returns-a-cited-report/run.mts`
**Controls:**
- `GOAL_CONTROL=invent-citation` makes synthesis cite an unobserved URL. Citation validation must fail the run.
- `GOAL_CONTROL=invent-executor-source` makes the executor claim an unobserved URL. Runtime retrieval provenance must keep it out of a valid report, so the goal must fail.
- `GOAL_CONTROL=no-rescue` makes every fetch attempt fail. The task cannot complete, so the goal must fail.
41 changes: 41 additions & 0 deletions goals/kits/research-returns-a-cited-report/run.mts
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
import { spawnSync } from "node:child_process";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { goalTmpDir, readCapture, REPO_ROOT, runGoal } from "../../lib/index.mts";

const here = fileURLToPath(new URL(".", import.meta.url));
const capture = join(goalTmpDir("research-returns-a-cited-report"), "run.json");
const fsdev = join(REPO_ROOT, "packages/fsdev/bin/fsdev.ts");
const tsx = join(REPO_ROOT, "packages/fsdev/node_modules/.bin/tsx");
const config = join(here, "fixture/fsdev.config.mts");

await runGoal((failures, tell) => {
const ran = spawnSync(tsx, [fsdev, "run", "research", "run", "--input", JSON.stringify({ question: "What does the fixture document?" }), "--config", config, "--capture", capture], {
cwd: join(here, "fixture"), env: process.env, encoding: "utf8",
});
if (ran.status !== 0) {
failures.push(`run: research app failed (exit ${ran.status}): ${`${ran.stderr}${ran.stdout}`.slice(-800)}`);
return { failures, evidence: "the running app did not return a report" };
}
const output = readCapture(capture).result.output as {
summary?: string; findings?: Array<{ claim: string; citations: string[] }>; sources?: Array<{ url: string }>;
retrieval?: { urls?: string[]; attempts?: Record<string, number> };
tasks?: Array<{ status: string; result?: { summary?: string; sources?: Array<{ url: string }> }; error?: string }>; completedSteps?: number;
};
const findings = tell.over("report:findings", output.findings ?? []);
const sources = tell.over("report:sources", output.sources ?? []);
const tasks = tell.over("state:tasks", output.tasks ?? []);
const citations = tell.over("report:citations", findings.flatMap((finding) => finding.citations));
const retrieved = tell.over("retrieval:urls", output.retrieval?.urls ?? []);
if (!citations.every((citation) => sources.some((source) => source.url === citation))) failures.push(`report:citations · citations are not all retrieved sources: ${JSON.stringify(citations)}`);
if (!citations.every((citation) => retrieved.includes(citation))) failures.push(`retrieval:urls · citation was not observed in a tool result: ${JSON.stringify({ citations, retrieved })}`);
const task = tasks[0];
if (output.completedSteps !== 1 || task?.status !== "completed" || task.result?.summary !== output.summary || task.result?.sources?.[0]?.url !== citations[0]) {
failures.push(`state:tasks · typed task evidence did not feed the report: ${JSON.stringify(output)}`);
}
const attempts = output.retrieval?.attempts ?? {};
if (attempts.search !== 2 || attempts.fetch !== 2 || attempts.crawl !== 1 || task?.error !== "injected flaky fetch") {
failures.push(`retry:attempts · expected search=2, fetch=2, crawl=1 and retained first error, got ${JSON.stringify({ attempts, error: task?.error })}`);
}
return { failures, evidence: `report ${JSON.stringify(findings)}\nretrieval ${JSON.stringify(output.retrieval)}\ntasks ${JSON.stringify(tasks)}` };
});
9 changes: 9 additions & 0 deletions packages/patterns/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@
"types": "./dist/plan-and-execute/index.d.ts",
"default": "./src/plan-and-execute/index.ts"
},
"./research": {
"types": "./dist/research/index.d.ts",
"default": "./src/research/index.ts"
},
"./supervisor": {
"types": "./dist/supervisor/index.d.ts",
"default": "./src/supervisor/index.ts"
Expand Down Expand Up @@ -83,6 +87,10 @@
"types": "./dist/plan-and-execute/index.d.ts",
"default": "./dist/plan-and-execute/index.js"
},
"./research": {
"types": "./dist/research/index.d.ts",
"default": "./dist/research/index.js"
},
"./supervisor": {
"types": "./dist/supervisor/index.d.ts",
"default": "./dist/supervisor/index.js"
Expand Down Expand Up @@ -120,6 +128,7 @@
"dependencies": {
"@flow-state-dev/core": "workspace:*",
"@flow-state-dev/orchestration": "workspace:*",
"@flow-state-dev/tools": "workspace:*",
"zod": "^3.24.1"
},
"peerDependencies": {
Expand Down
Loading
Loading