Skip to content

feat(kits): deep research (FIX-2822) - #5436

Draft
jhoffner wants to merge 3 commits into
mainfrom
codex/fix-2822-deep-research
Draft

jhoffner wants to merge 3 commits into
mainfrom
codex/fix-2822-deep-research

Conversation

@jhoffner

Copy link
Copy Markdown
Collaborator

Closes FIX-2822

Issue: #5422 (epic FIX-2813 Showcase kits)
Codex task: https://chatgpt.com/codex/tasks/task_e_6afd14d891388324a58b92efaf1fcff1

What was built

Adds research() in packages/patterns built on plan-and-execute with search, fetch and crawl, per-task retry/rescue, and citation validation against retrieved sources. Adds tests, goal research-returns-a-cited-report, changeset, lockfile/tsconfig updates.

How the goal proves it

See the goal under goals/kits/ added in this PR (run line in its goal.md). It runs the installed app (not the kit factory) with deterministic/scripted models and includes GOAL_CONTROL negative controls that must fail.

Generated by Codex Cloud; applied from the task diff unchanged. Not yet run locally or in CI-verified — draft.

@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
flow-state-dev-build-site Ready Ready Preview Nov 18, 2026 4:36am UTC
flow-state-dev-docs Ready Ready Preview Nov 18, 2026 4:36am UTC
1 Skipped Deployment
Project Deployment Actions Updated
flow-state-dev-ui Ignored Ignored Preview Nov 18, 2026 4:36am UTC

Request Review

@linear-code

linear-code Bot commented Nov 18, 2026

Copy link
Copy Markdown

FIX-2822

Comment on lines +14 to +17
const researchSessionStateSchema = z.object({
researchRetrievedUrls: z.array(z.string()).default([]),
researchToolAttempts: z.record(z.number()).default({}),
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Bug: Retrieved-URL and attempt state is session-scoped and never reset per run

researchRetrievedUrls and researchToolAttempts live in ctx.session.state and nothing clears them at the start of a run action (no other reference exists in the repo). In a session with a second question, the validator treats every URL retrieved in earlier questions as valid provenance, so a synthesizer can cite a URL from a previous run and still pass validateResearchReport. The returned retrieval.urls and retrieval.attempts also accumulate across runs, so they no longer describe the current report. Reset both fields at the start of the run, e.g. a leading .tap handler in research-run that patches { researchRetrievedUrls: [], researchToolAttempts: {} }. Alternatively, keep them in sequencer or run-scoped state.

Clear provenance state at the start of each research run.:

const resetProvenance = handler({
  name: "research-reset-provenance", inputSchema: z.unknown(), sessionStateSchema: researchSessionStateSchema,
  execute: async (_i, ctx) => { await ctx.session.patchState({ researchRetrievedUrls: [], researchToolAttempts: {} }); },
});
const run = sequencer({ name: "research-run", inputSchema })
  .tap(resetProvenance).map((input) => ({ goal: input.question })).step(researchBlock);
  • Apply fix

Check the box to apply the fix or reply for a change | Was this helpful? React with 👍 / 👎

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 0ad3919: research provenance (retrieved URLs and attempts) now resets at the start of each run. CI green on that commit.

— FSD Head of Engineering

Comment on lines +73 to +76
maxTasks: z.number().int().positive().default(12),
maxAttemptsPerTask: z.number().int().min(2).default(2),
maxIterations: z.number().int().positive().default(3),
maxTotalRetries: z.number().int().nonnegative().default(6),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Bug: Retry control on maxAttemptsPerTask and schema/docs mismatch

maxAttemptsPerTask is forced to min(2), so callers cannot disable per-task retry, which contradicts the 'retryable' framing for apps that want a single attempt. maxTotalRetries defaults to 6 but can be set below what maxAttemptsPerTask implies, and that interaction is not validated or documented. Either allow min(1) or document the constraint on the config fields, and validate the relationship between the two settings.

Was this helpful? React with 👍 / 👎

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 0ad3919: maxAttemptsPerTask now allows a single attempt, and the changeset/docs were aligned. CI green on that commit.

— FSD Head of Engineering

Comment on lines +1 to +15
import { describe, expect, it } from "vitest";
import { validateResearchReport } from "../src/research";

describe("research report citations", () => {
it("rejects citations that no completed research task retrieved", () => {
expect(() =>
validateResearchReport(
{
question: "What changed?",
summary: "A sourced answer.",
findings: [{ claim: "The change shipped.", citations: ["https://invented.example/report"] }],
sources: [{ title: "Invented", url: "https://invented.example/report" }],
},
new Set(["https://fixture.example/source"]),
),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Quality: Missing tests for kit install, config validation and tool provenance

The unit tests only cover validateResearchReport. Nothing tests the config .refine (model or role-specific models required), retrievedUrls extraction for search, fetch and crawl, observableTool attempt counting and recording, or the executor-source provenance check in research-validate-report. Those are the core of the kit's provenance claim and rely solely on the goal script. Add testBlock unit tests per the repo's testing guidance, and run the goal with fsdev run as the verification path.

Was this helpful? React with 👍 / 👎

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 0ad3919: added tests in packages/patterns/test/research.test.ts (+160 lines) beyond validateResearchReport. CI green on that commit.

— FSD Head of Engineering

@gitar-bot

gitar-bot Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Note

Automatic reviews are paused because your team has used its included automatic processing for this billing period (headroom scales with your seat count). You can still comment "Gitar review" to run one anytime, and automatic reviews resume on their own by December 1. Add seats for more headroom.
Learn more

Code Review ⚠️ Changes requested 🟡 Medium risk 0 closed / 3 findings

Adding the research kit exposes a multi-step web-retrieval and synthesis workflow; retrieval or model errors could produce incomplete or misleading reports.

Adds research() to packages/patterns built on plan-and-execute with search, fetch, crawl, per-task retry and citation validation against retrieved sources. Changes are requested because retrieved-URL and attempt state is session-scoped and never reset per run, so a later question can cite URLs from an earlier run and the returned retrieval data accumulates across runs.

  • researchRetrievedUrls and researchToolAttempts need resetting at the start of each run action in packages/patterns/src/research/index.ts
  • maxAttemptsPerTask is forced to a minimum of 2, and its interaction with maxTotalRetries is neither validated nor documented
  • Unit tests cover only validateResearchReport; config validation, URL extraction, attempt counting and executor-source provenance lack coverage
⚠️ Bug: Retrieved-URL and attempt state is session-scoped and never reset per run

📄 packages/patterns/src/research/index.ts:14-17 📄 packages/patterns/src/research/index.ts:109-122 📄 packages/patterns/src/research/index.ts:171-185

researchRetrievedUrls and researchToolAttempts live in ctx.session.state and nothing clears them at the start of a run action (no other reference exists in the repo). In a session with a second question, the validator treats every URL retrieved in earlier questions as valid provenance, so a synthesizer can cite a URL from a previous run and still pass validateResearchReport. The returned retrieval.urls and retrieval.attempts also accumulate across runs, so they no longer describe the current report. Reset both fields at the start of the run, e.g. a leading .tap handler in research-run that patches { researchRetrievedUrls: [], researchToolAttempts: {} }. Alternatively, keep them in sequencer or run-scoped state.

Clear provenance state at the start of each research run.
const resetProvenance = handler({
  name: "research-reset-provenance", inputSchema: z.unknown(), sessionStateSchema: researchSessionStateSchema,
  execute: async (_i, ctx) => { await ctx.session.patchState({ researchRetrievedUrls: [], researchToolAttempts: {} }); },
});
const run = sequencer({ name: "research-run", inputSchema })
  .tap(resetProvenance).map((input) => ({ goal: input.question })).step(researchBlock);
💡 Bug: Retry control on maxAttemptsPerTask and schema/docs mismatch

📄 packages/patterns/src/research/index.ts:73-76 📄 packages/patterns/src/research/index.ts:83-87

maxAttemptsPerTask is forced to min(2), so callers cannot disable per-task retry, which contradicts the 'retryable' framing for apps that want a single attempt. maxTotalRetries defaults to 6 but can be set below what maxAttemptsPerTask implies, and that interaction is not validated or documented. Either allow min(1) or document the constraint on the config fields, and validate the relationship between the two settings.

💡 Quality: Missing tests for kit install, config validation and tool provenance

📄 packages/patterns/test/research.test.ts:1-15 📄 packages/patterns/src/research/index.ts:67-81 📄 packages/patterns/src/research/index.ts:94-108

The unit tests only cover validateResearchReport. Nothing tests the config .refine (model or role-specific models required), retrievedUrls extraction for search, fetch and crawl, observableTool attempt counting and recording, or the executor-source provenance check in research-validate-report. Those are the core of the kit's provenance claim and rely solely on the goal script. Add testBlock unit tests per the repo's testing guidance, and run the goal with fsdev run as the verification path.

🤖 Prompt for agents
Code Review: Adds `research()` to `packages/patterns` built on plan-and-execute with search, fetch, crawl, per-task retry and citation validation against retrieved sources. Changes are requested because retrieved-URL and attempt state is session-scoped and never reset per run, so a later question can cite URLs from an earlier run and the returned retrieval data accumulates across runs.
  
  - `researchRetrievedUrls` and `researchToolAttempts` need resetting at the start of each `run` action in `packages/patterns/src/research/index.ts`
  - `maxAttemptsPerTask` is forced to a minimum of 2, and its interaction with `maxTotalRetries` is neither validated nor documented
  - Unit tests cover only `validateResearchReport`; config validation, URL extraction, attempt counting and executor-source provenance lack coverage

1. ⚠️ Bug: Retrieved-URL and attempt state is session-scoped and never reset per run
   Files: packages/patterns/src/research/index.ts:14-17, packages/patterns/src/research/index.ts:109-122, packages/patterns/src/research/index.ts:171-185

   `researchRetrievedUrls` and `researchToolAttempts` live in `ctx.session.state` and nothing clears them at the start of a `run` action (no other reference exists in the repo). In a session with a second question, the validator treats every URL retrieved in earlier questions as valid provenance, so a synthesizer can cite a URL from a previous run and still pass `validateResearchReport`. The returned `retrieval.urls` and `retrieval.attempts` also accumulate across runs, so they no longer describe the current report. Reset both fields at the start of the run, e.g. a leading `.tap` handler in `research-run` that patches `{ researchRetrievedUrls: [], researchToolAttempts: {} }`. Alternatively, keep them in sequencer or run-scoped state.

   Fix (Clear provenance state at the start of each research run.):
   const resetProvenance = handler({
     name: "research-reset-provenance", inputSchema: z.unknown(), sessionStateSchema: researchSessionStateSchema,
     execute: async (_i, ctx) => { await ctx.session.patchState({ researchRetrievedUrls: [], researchToolAttempts: {} }); },
   });
   const run = sequencer({ name: "research-run", inputSchema })
     .tap(resetProvenance).map((input) => ({ goal: input.question })).step(researchBlock);

2. 💡 Bug: Retry control on `maxAttemptsPerTask` and schema/docs mismatch
   Files: packages/patterns/src/research/index.ts:73-76, packages/patterns/src/research/index.ts:83-87

   `maxAttemptsPerTask` is forced to `min(2)`, so callers cannot disable per-task retry, which contradicts the 'retryable' framing for apps that want a single attempt. `maxTotalRetries` defaults to 6 but can be set below what `maxAttemptsPerTask` implies, and that interaction is not validated or documented. Either allow `min(1)` or document the constraint on the config fields, and validate the relationship between the two settings.

3. 💡 Quality: Missing tests for kit install, config validation and tool provenance
   Files: packages/patterns/test/research.test.ts:1-15, packages/patterns/src/research/index.ts:67-81, packages/patterns/src/research/index.ts:94-108

   The unit tests only cover `validateResearchReport`. Nothing tests the config `.refine` (model or role-specific models required), `retrievedUrls` extraction for search, fetch and crawl, `observableTool` attempt counting and recording, or the executor-source provenance check in `research-validate-report`. Those are the core of the kit's provenance claim and rely solely on the goal script. Add `testBlock` unit tests per the repo's testing guidance, and run the goal with `fsdev run` as the verification path.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

Was this helpful? React with 👍 / 👎 | Gitar

@github-actions

Copy link
Copy Markdown
Contributor

Mechanism check (advisory, never blocks)

Rows from docs/mechanisms/mechanisms.yaml · labels: new-mechanism

New export Package Mechanism
validateResearchReport @flow-state-dev/patterns none (nearest: supervisor, routed-specialists, event-actors, debate, round-robin, response-auditor)
research @flow-state-dev/patterns none (nearest: supervisor, routed-specialists, event-actors, debate, round-robin, response-auditor)

If an export is a new mechanism, add a row. CI filled the mechanical fields; job, builtOn and overlaps are yours. If it belongs to an existing row, say which in the PR.

  - id: TODO
    job: TODO
    package: "@flow-state-dev/patterns"
    anchor: validateResearchReport
    file: packages/patterns/src/research/index.ts
    builtOn: []  # TODO
    overlaps: []  # TODO
    status: canonical
    doc: TODO
  - id: TODO
    job: TODO
    package: "@flow-state-dev/patterns"
    anchor: research
    file: packages/patterns/src/research/index.ts
    builtOn: []  # TODO
    overlaps: []  # TODO
    status: canonical
    doc: TODO

@github-actions github-actions Bot added the new-mechanism A new public export matches no row of docs/mechanisms/mechanisms.yaml (advisory) label Nov 18, 2026

This branch was successfully deployed

1 active deployment
Preview – flow-state-dev-docs — 0ad3919d Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new-mechanism A new public export matches no row of docs/mechanisms/mechanisms.yaml (advisory) Prototype

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant