Repository navigation
spec(FIX-1185): a coding agent driven through the in-process SDK never invokes its plan tools; the CLI does #3166
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| # FIX-1185 · Business rules | ||
|
|
||
| [Spec](SPEC.md) · [Decisions](DECISIONS.md) · **Rules** · [Plan](PLAN.md) · [Docs](DOCS.md) | ||
|
|
||
| The cases, written as rules. Each says what a host or the run does and what happens. The | ||
| *proved by* column is the check the plan runs. | ||
|
|
||
| ## What the run starts from | ||
|
|
||
| | # | When | Then | Proved by | | ||
| |---|---|---|---| | ||
| | BR-1 | A host gives no `systemPrompt` | The SDK is asked for Claude Code's own prompt (the `claude_code` preset), with nothing appended | CI, on the options the SDK receives · goal check | | ||
| | BR-2 | A host gives a string, including a long one | The string is forwarded as is and replaces Claude Code's prompt, as today | CI | | ||
| | BR-3 | A host gives `""` | Forwarded as `""`: the run starts blank, exactly as every run does on today's `main` | CI · the goal check's control | | ||
| | BR-4 | A host gives the preset with `append` | Forwarded as is: Claude Code's prompt plus the host's text | CI | | ||
| | BR-5 | A host gives an array of prompt blocks, or any other shape the SDK accepts | Forwarded as is. The SDK validates it, as it does `sandbox` | CI, one array case | | ||
| | BR-6 | The block is attached as a capability with any of the above | The same as the block built directly | CI | | ||
|
|
||
| ## Every run, not only the first | ||
|
|
||
| | # | When | Then | Proved by | | ||
| |---|---|---|---| | ||
| | BR-7 | A detached run resumes an earlier session | The same prompt rule applies on the resumed run; its plan rows are its own | CI, on the resume path | | ||
| | BR-8 | A run is in-session (not detached) | The same prompt rule applies | CI | | ||
| | BR-9 | `recordWork` is off | The prompt rule is the same. Recording never changes what the agent is told | CI | | ||
|
|
||
| ## The plan record | ||
|
|
||
| | # | When | Then | Proved by | | ||
| |---|---|---|---| | ||
| | BR-10 | A real run, no prompt given, works a multi-step job that names no tool | The plan record holds the agent's own to-do items, read back from state, and at least one reaches `completed` | Goal check, 3 of 3 runs | | ||
| | BR-11 | The same run with `""` | No plan rows | Goal check control, 2 of 2 runs | | ||
| | BR-12 | A run reports no plan at all | Nothing is invented. The record stays empty and the plan panel says the harness reported none ([ER-5](../../epics/FIX-1907/BUSINESS-RULES.md#what-a-team-gets-and-what-it-doesnt), FIX-1915's panel) | Existing recorder tests | | ||
|
|
||
| ## Failure taxonomy | ||
|
|
||
| Nothing here is fatal. A prompt shape the SDK rejects fails the run the way any bad SDK option | ||
| does today, as a `ClaudeAgentRunError`. A run that plans nothing is not an error: the empty record | ||
| is the honest answer. | ||
|
|
||
| ## Acceptance criteria this issue owns | ||
|
|
||
| [The goal](SPEC.md#the-goal-and-how-well-know-its-met): on a real model, a run with no prompt | ||
| given and a job that names no tool fills the plan record on 3 of 3 runs, and the same check fails | ||
| under `GOAL_CONTROL=empty-prompt`. This satisfies the epic's | ||
| [ER-7](../../epics/FIX-1907/BUSINESS-RULES.md#what-a-team-gets-and-what-it-doesnt) on the fix | ||
| branch, not the finding branch. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| # FIX-1185 · Decisions | ||
|
|
||
| [Spec](SPEC.md) · **Decisions** · [Rules](BUSINESS-RULES.md) · [Plan](PLAN.md) · [Docs](DOCS.md) | ||
|
|
||
| What was considered, what was chosen, and what each choice locks in. Two decisions are the | ||
| sign-off surface. They sit under the epic's [D2](../../epics/FIX-1907/DECISIONS.md#d2) (one | ||
| contract, harness differences live in the block's options) and | ||
| [D3](../../epics/FIX-1907/DECISIONS.md#d3) (each harness writes its own plan; FIX-1185 makes | ||
| Claude Code's fill). | ||
|
|
||
| ## The tree | ||
|
|
||
| ```mermaid | ||
| flowchart TD | ||
| I["FIX-1185"] --> D1["D1 · no prompt given means Claude Code's own prompt"] | ||
| D1 -.->|"rejected"| X1["keep the empty default · add a plan line when recording"] | ||
| D1 -.->|"rejected"| X2["change nothing · write the finding · each host adds the line"] | ||
| I --> D2["D2 · a caller's string still replaces · adding is explicit"] | ||
| D2 -.->|"rejected"| X3["every string becomes an addition"] | ||
| ``` | ||
|
|
||
| Solid edges are what you're signing. Dashed edges lost, and the label says why. | ||
|
|
||
| <a name="d1"></a> | ||
| ## D1 · With no prompt given, a run starts from Claude Code's own prompt, not the SDK's empty one | ||
|
|
||
| | | | | ||
| |---|---| | ||
| | **Instead of** | Keeping the empty default and having `recordWork` add one line telling the agent to use its to-do tools · or changing nothing and documenting the finding, so each host adds the line itself | | ||
| | **Because** | The premise of a harness block is the same agent under our orchestration (tenet 1, the epic's goal). The CLI and the SDK run the same binary; the measured difference is that the CLI starts from Claude Code's prompt and our block starts from nothing ([POC](#settled)). A plan line fixes one surface the blank prompt loses and leaves the rest. A recorder that injects its own instruction would record a plan we asked for, then call it the agent's own. Documenting it leaves the gap in the package, which [ER-11](../../epics/FIX-1907/BUSINESS-RULES.md#what-no-child-may-do) forbids patching downstream | | ||
| | **Locks in** | Every host that omits the prompt changes behaviour on upgrade: Kitchen Sink, the coding-skill lab, the conductor lab, the workspace capability, and any app built on the block. Each run carries a larger prompt and follows Claude Code's own conventions. Our default now moves when Anthropic changes Claude Code's prompt | | ||
|
|
||
|  | ||
|
|
||
| It comes down to whose plan it is: a line we inject records our instruction, not the agent's account. | ||
|
|
||
| **What would change my mind:** the goal check showing Claude Code's own prompt still yields no | ||
| plan rows. Then the blank prompt was not the whole cause, and the plan line is the fallback, said | ||
| in the docs as an instruction we add. | ||
|
|
||
| <a name="d2"></a> | ||
| ## D2 · A caller's own string still replaces that prompt; adding to it is a separate, explicit form | ||
|
|
||
| | | | | ||
| |---|---| | ||
| | **Instead of** | Treating every string as an addition to Claude Code's prompt | | ||
| | **Because** | The option is documented as forwarded to the SDK, and in the SDK a string replaces. Turning it into an addition silently changes every host that wrote a full prompt to narrow the agent on purpose. The SDK already has the addition form (`preset` with `append`); forwarding it costs one type, not a new option (Step 3.5 question 7: a parameter on the existing entry) | | ||
| | **Locks in** | Two spellings to teach. A host that wrote one line of role text, and meant it as an addition, keeps an empty plan record until it moves to the added form. In this repo that is Shift Manager's harness and one devforce goal | | ||
|
|
||
|  | ||
|
|
||
| It comes down to the host that wrote a full prompt on purpose: an addition would grow it unasked. | ||
|
|
||
| **What would change my mind:** finding that every known caller's string was one line of role | ||
| text meant to sit on top. Then replacement protects nobody and the addition is the better default. | ||
|
|
||
| ## Decided, not asked | ||
|
|
||
| - **An empty string means today's blank start.** It is the escape hatch, and it is the goal | ||
| check's control. The SDK already reads `""` that way. | ||
| - **The option forwards the SDK's own prompt shape**, the way `sandbox` forwards its settings. | ||
| No FSD spelling of `preset`. | ||
| - **No warning when a string replaces the prompt with recording on.** A caller may want a plain | ||
| run; the docs say what it gives up, and the plan record's own emptiness is the signal. | ||
| - **The capability takes the same option.** It forwards the agent's options today. | ||
|
|
||
| ## Considered and dropped | ||
|
|
||
| | Alternative | Why not | | ||
| |---|---| | ||
| | Drive one goal run through the CLI to exercise the plan half once (the issue's second option) | Proves the recorder on a path Kitchen Sink and every host don't use. The epic needs the SDK path to fill | | ||
| | Put the plan line in each host's job prompt | Shown to work, but every host copies a line that only exists to undo our default | | ||
| | Add `TaskCreate` and `TaskUpdate` to `allowedTools`, or raise `maxTurns` | Measured: no change on 8 runs | | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
On Opus 4.8, Sonnet 5, Fable 5, Mythos 5, and newer models, changing only the system prompt cannot satisfy BR-10: the locked Agent SDK 0.3.234 includes the 0.3.233 change that removes the task-tracking tools from the default surface unless they are named through Useful? React with 👍 / 👎. |
||
| | A `planGuidance: true` option | A new option for one line the default prompt already carries | | ||
|
|
||
| <a name="settled"></a> | ||
| ## Settled | ||
|
|
||
| - **The two drivers differ at prompt assembly, before the model sees anything** — **CONFIRMED**, | ||
| model-free. Driving the real SDK `query()` with a stubbed agent process, the first message | ||
| it sends carries `systemPrompt: [""]` when none is given, the same as `""`. With the | ||
| `claude_code` preset it sends no custom prompt and only the appended text, so the agent | ||
| process keeps Claude Code's own. A planted fault (the omitted arm sent as the preset) turns | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Settled claims a planted fault turns the POC red; |
||
| the check red. Run: `pnpm tsx specs/issues/FIX-1185/poc/what-the-sdk-sends/run.mts`. | ||
| - **An instruction to use the tools restores the calls on the SDK path** — **CONFIRMED** once, | ||
| 0 → 9 calls and 3 plan rows read back from state | ||
| ([the Aug 19 measurement](https://linear.app/fixpoint-labs/issue/FIX-1185)). One run, not a | ||
| paired control; the goal check repeats it under the control. | ||
|
|
||
| ## How it got here | ||
|
|
||
| - **Draft** — framed as a prompt-assembly gap, not a driver gap; the block's default moves to | ||
| Claude Code's own prompt and forwards the SDK's addition form; one PR in the package, proved by | ||
| a real-model goal check reading the plan record from state. | ||
|
|
||
| **Open: none.** | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| # FIX-1185 · Documentation draft | ||
|
|
||
| [Spec](SPEC.md) · [Decisions](DECISIONS.md) · [Rules](BUSINESS-RULES.md) · [Plan](PLAN.md) · **Docs** | ||
|
|
||
| Two destinations, both extended, no new page. The epic's shared narrative (the harness picker, | ||
| the plan panel) belongs to FIX-1914 and FIX-1915 and is not drafted here. Voice rules most at | ||
| risk for this topic: no internal issue numbers in `apps/docs`, few em-dashes, and introduce | ||
| "system prompt" in plain terms on first use. | ||
|
|
||
| ## UPDATE · `apps/docs/docs/tools/claude-code-sdk.md` · "As a sequencer step" | ||
|
|
||
| Replace the example's `systemPrompt` line so the first example a reader copies keeps the run's | ||
| to-do list: | ||
|
|
||
| ```ts | ||
| const agent = claudeCodeAgent({ | ||
| systemPrompt: { | ||
| type: "preset", | ||
| preset: "claude_code", | ||
| append: "You are a careful refactoring assistant.", | ||
| }, | ||
| allowedTools: ["Read", "Grep", "Edit"], | ||
| }); | ||
| ``` | ||
|
|
||
| ## UPDATE · `apps/docs/docs/tools/claude-code-sdk.md` · "Configuring the run", new subsection before `settingSources` | ||
|
|
||
| ### The prompt the run starts from | ||
|
|
||
| A system prompt is the standing instruction the agent reads before your request. Leave | ||
| `systemPrompt` out and the run starts from Claude Code's own, the same one the `claude` command | ||
| uses. That prompt is what tells the agent to keep a to-do list as it works, among much else, so | ||
| a run started this way behaves like the agent you know from the terminal. | ||
|
|
||
| To add your own instructions on top, append them: | ||
|
|
||
| ```ts | ||
| claudeCodeAgent({ | ||
| systemPrompt: { | ||
| type: "preset", | ||
| preset: "claude_code", | ||
| append: "Commit your work with git before you finish.", | ||
| }, | ||
| }); | ||
| ``` | ||
|
|
||
| A plain string replaces Claude Code's prompt entirely. The agent then knows only what your string | ||
| says. It will still have its tools, but nothing tells it to keep a to-do list, so with | ||
| `recordWork` on the plan record usually stays empty. Use a string when you want that narrow | ||
| agent; use `append` when you want to add a line. | ||
|
|
||
| `""` starts the run with no system prompt at all. The value is forwarded to the Agent SDK as | ||
| given, so any shape the SDK accepts works here, and the SDK rejects one it doesn't. | ||
|
|
||
| ## UPDATE · `apps/docs/docs/tools/claude-code-sdk.md` · "Reading it back", after "The plan record reads the same way" | ||
|
|
||
| The plan record fills only when the agent keeps a to-do list. It does when the run starts from | ||
| Claude Code's own prompt, with or without `append`. A run given its own prompt string usually | ||
| writes its plan as prose instead, and the record stays empty. That is the honest answer, not a | ||
| recording failure, and nothing appears in `observed-gaps` for it. | ||
|
|
||
| ## UPDATE · `packages/claude-code/README.md` · "Recording what a run did", after the code block | ||
|
|
||
| The plan half fills when the agent keeps a to-do list, which it does when the run starts from | ||
| Claude Code's own prompt. That is the default when `systemPrompt` is left out. To add | ||
| instructions, use `systemPrompt: { type: "preset", preset: "claude_code", append: "…" }`. A | ||
| plain string replaces Claude Code's prompt and usually leaves `observed-plan` empty; `""` starts | ||
| the run with no system prompt. | ||
|
|
||
| ## UPDATE · `.changeset/*.md` · `@flow-state-dev/claude-code` minor | ||
|
|
||
| > `claudeCodeAgent` now starts a run from Claude Code's own system prompt when `systemPrompt` is | ||
| > left out, matching the `claude` CLI, and accepts the Agent SDK's preset form to add | ||
| > instructions with `append`. Runs that relied on starting with no system prompt pass | ||
| > `systemPrompt: ""`. | ||
|
|
||
| ## Publication ownership | ||
|
|
||
| FIX-1185 publishes these four operations after V2 and the goal check pass. It does not edit | ||
| `coding-agents.md`: the harness contract is unchanged. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| # FIX-1185 · Plan | ||
|
|
||
| [Spec](SPEC.md) · [Decisions](DECISIONS.md) · [Rules](BUSINESS-RULES.md) · **Plan** · [Docs](DOCS.md) | ||
|
|
||
| Written for the implementing agent. IDs cross-reference [BUSINESS-RULES.md](BUSINESS-RULES.md) | ||
| (BR-n) and [DECISIONS.md](DECISIONS.md) (D-n). `diagnose` for the cause (already settled, see | ||
| the POC line), then `tdd` for the default. One PR. | ||
|
|
||
| ## Surfaces | ||
|
|
||
| | ID | Package · role | Change | Rules | | ||
| |---|---|---|---| | ||
| | S1 | `claude-code` · the agent options type and the SDK query-options type | `systemPrompt` takes the SDK's own union (string, string array, the `claude_code` preset with optional `append`), forwarded verbatim (D2) | BR-2 BR-4 BR-5 | | ||
| | S2 | `claude-code` · where the block builds the SDK query options | `undefined` becomes the preset with nothing appended (D1). Every other value, `""` included, passes through untouched | BR-1 BR-3 BR-7 BR-8 BR-9 | | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. S1 widening to the full SDK union is the right long-term shape (D2 + |
||
| | S3 | `claude-code` · the option's doc comment, the README's options section | Say what omitting, a string, `""` and the preset each give, and that a string drops the to-do guidance | BR-2 BR-3 | | ||
| | S4 | `goals/claude-code-harness/it-keeps-its-own-plan/` | **New** goal: `goal.md`, `run.mts`, a held-out job fixture, `GOAL_CONTROL=empty-prompt` | BR-10 BR-11 | | ||
| | S5 | Docs | Publish [DOCS.md](DOCS.md) · one `minor` changeset for `@flow-state-dev/claude-code` (a default changes for every host) | — | | ||
|
|
||
| Nothing is removed. | ||
|
|
||
| ## Sequence | ||
|
|
||
| ```mermaid | ||
| flowchart TD | ||
| S1["S1 · widen the type"] --> S2["S2 · the default"] | ||
| S2 --> S3["S3 · doc comment and README"] | ||
| S2 --> S4["S4 · the goal check"] | ||
| S3 --> S5["S5 · docs and changeset"] | ||
| S4 --> S5 | ||
| ``` | ||
|
|
||
| ## Checks | ||
|
|
||
| | ID | Runs after | Passes when | | ||
| |---|---|---| | ||
| | V1 | S1 | Type test: each SDK prompt shape is accepted by the block and the capability; a non-prompt object is a type error | | ||
| | V2 | S2 | Agent spec: the options the stubbed SDK receives carry the preset when none is given (BR-1); the string, `""`, the preset and an array pass through byte-identical (BR-2 to BR-5). Red first: on today's `main` the BR-1 case fails | | ||
| | V3 | S2 | The second path (BP-035): the same assertions on a detached resumed run, an in-session run, `recordWork` off, and through the capability (BR-6 to BR-9) | | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. V2 + V3 as separate checklist rows duplicate the same assertions across detached / in-session / |
||
| | VG | S4 | [The goal](SPEC.md#the-goal-and-how-well-know-its-met): `pnpm tsx goals/claude-code-harness/it-keeps-its-own-plan/run.mts` PASSES 3 of 3 on a real model, after the same run FAILED 2 of 2 under `GOAL_CONTROL=empty-prompt` on *at least two plan rows* | | ||
| | V4 | S3 S5 | `pnpm --filter @flow-state-dev/claude-code test` and `typecheck` green; the docs build | | ||
|
|
||
| D1's check is V2's BR-1 case plus VG; D2's is V2's BR-2 case. | ||
|
|
||
| ## Pinned names | ||
|
|
||
| | Where | Name | Why pinned | | ||
| |---|---|---| | ||
| | Goal folder | `goals/claude-code-harness/it-keeps-its-own-plan/` | The epic's closure cites it | | ||
| | Control | `GOAL_CONTROL=empty-prompt` | `SPEC.md` and the closure name it | | ||
|
|
||
| The option keeps its name, `systemPrompt`. Everything else is yours to name. | ||
|
|
||
| ## Guardrails | ||
|
|
||
| | Rule | Because | | ||
| |---|---| | ||
| | The default is decided in one place, where the query options are built | The capability and every resolver path reach the SDK through it. A second place is how one path keeps the blank prompt (tenet 5) | | ||
| | Forward the SDK's shape; never translate or validate it | The package forwards `sandbox` the same way. A copy of the SDK's union drifts the day the SDK adds a field | | ||
| | The recorder is not touched | D1 rests on the plan being the agent's own. Any instruction the recorder adds would make the record ours | | ||
| | The goal check reads the plan record from state, through the resource route | The issue's verification asks for rows read back from state. A count of tool calls in the stream passes while the record stays empty | | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When the goal reuses a session to cover repeated or resumed runs, reading the session-scoped Useful? React with 👍 / 👎. |
||
| | The goal's job names no tool and does not mention planning | Otherwise the PASS proves the Aug 19 result again, not the default | | ||
| | `""` stays the blank start, and the control uses it | A host that wants today's behaviour needs one spelling, and the control must be exactly today's behaviour | | ||
|
|
||
| ## Docs | ||
|
|
||
| Reconcile [DOCS.md](DOCS.md) against the shipped option and VG's verdict, then publish it with | ||
| S5. The README section (S3) and the docs page say the same thing in their own register. | ||
|
|
||
| ## Sketch · pseudocode, illustrative | ||
|
|
||
| ``` | ||
| where the block builds the SDK's query options: | ||
| prompt ← the host's systemPrompt | ||
| if prompt is absent: ← the whole fix (D1) | ||
| prompt ← Claude Code's own (the preset), nothing appended | ||
| forward prompt unchanged (D2: a string still replaces; "" still blank) | ||
| ``` | ||
|
|
||
| **POC:** `poc/what-the-sdk-sends/` inside this spec. Model-free: it drives the real SDK `query()` | ||
| with a stubbed agent process and prints the first message the SDK sends. It showed the premise | ||
| holds: no prompt and `""` both send `systemPrompt: [""]`; the preset sends no custom prompt, so | ||
| the agent process keeps Claude Code's own. A planted fault turned it red. It does not show that | ||
| Claude Code's prompt makes the agent plan; VG does that on a real model. | ||
|
|
||
| ## At implement time | ||
|
|
||
| - FIX-1915 may have moved the plan record's shape into `@flow-state-dev/core` first. Rebase on | ||
| it; neither PR changes the other's half ([epic seam](../../epics/FIX-1907/PLAN.md#coordination-seams-to-watch)). | ||
| - Check the SDK version the workspace resolves. If its default for an absent prompt is no longer | ||
| empty, S2 is a no-op: say so, keep S1, S3 and S4. | ||
| - The goal needs a host where Claude Code is signed in. A cloud session can't run it; hand it to | ||
| a local host and record where it ran. | ||
|
|
||
| ## Follow-ups | ||
|
|
||
| - **Shift Manager's harness and the devforce goal pass a one-line string**, so under D2 their | ||
| plan record stays empty. Moving them to the added form is their owners' call. Flagged to the | ||
| epic coordinator; not filed here. | ||
| - **A run asserted that a sub-agent wrote a file it wrote itself** (same Aug 19 run). A separate | ||
| problem, noted on the issue, not filed. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BR-12 is “existing recorder tests” — not in PLAN’s V1–VG matrix and not owned by this issue. Consider moving to SPEC “What stays as it is” so acceptance criteria stay BR-10/11 only.