Repository navigation
spec(FIX-1185): a coding agent driven through the in-process SDK never invokes its plan tools; the CLI does - #3166
Conversation
The SDK path sends an empty system prompt when the block gives none, so the agent never sees the guidance that makes the CLI use its to-do tools. The spec moves the block's default to Claude Code's own prompt, forwards the SDK's append form, and proves it with a real-model goal check that reads the plan record from state. Includes a model-free POC showing what the SDK sends for each prompt shape. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017pp7wCd87NW4BH7XVuF9Gv
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Second look (direction only)The diagnosis and the lean shape are right. One option's default moves, the SDK's own prompt union is forwarded, and the recorder is left alone. I found no better approach. The items below are things to settle or watch, ranked. 1. The "3 of 3 runs, at least two rows" bar may be flaky by construction. 2. D1 widens beyond the plan. The spec prices "a larger, different agent" but doesn't measure it. 3. Interaction with 4. A type-level breaking change is hidden under "minor". 5. The "Follow-ups" leave two in-repo callers silently degraded. 6. Smaller, and not blocking.
On the architecture lens: nothing here calls for a refactor. The default is decided in one place (the query options in Items 1, 2 and 5 are the ones I'd have the implementing agent resolve before building. Generated by Claude Code |
There was a problem hiding this comment.
Problem
In-process Claude Code runs start with an empty system prompt while the CLI uses Claude Code’s own, so plan/to-do tooling never fires unless hosts patch prompts themselves.
Verdict
Approve direction. D1 (omit → SDK claude_code preset) plus D2 (string still replaces; preset + append for additions) is the smallest honest fix: one branch at query-option build time, reuse the Agent SDK’s existing preset shape (same philosophy as sandbox), and leave the recorder alone so the plan stays the agent’s account. The rejected paths (recorder-injected plan line, per-host job text, planGuidance flag) would be simpler on paper but recreate “whose plan is this?” and ER-11 downstream patching.
The spec set is much heavier than the implementation (~784 lines for a one-line default). That is not wrong for epic closure, but it is where /simplify has the most leverage before implementation lands.
What went further than necessary (optional tightening)
- Control wording — “FAIL on at least two plan rows” reads like the control should have two rows. You mean “does not meet the PASS signal (fewer than two rows / none).” Same phrase in
SPEC.mdandPLAN.mdVG; easy to mis-implement. - Omitted vs
mainvsGOAL_CONTROL— POC shows omitted and""both become[""]today, but they are different host inputs. One sentence on which armmainexercises (omitted default) vs the control env (""escape hatch) would prevent someone from running only one path. - Goal bar — 3/3 PASS, second held-out job, 2/2 control, FAIL-before-PASS on
main, all on a real signed-in host is expensive for a wiring fix already backed by Aug 19 + model-free POC + V2 BR-1. Consider collapsing to BUSINESS-RULES as canonical numbers and dropping the second job unless FIX-1907 closure truly requires it. - BR-2–BR-5 — five “forward verbatim” rows could be one rule plus BR-1 as the special case.
- Settled / POC — “planted fault turns the check red” is not in
run.mts(only omitted/preset assertions). Either add aFIX1185_PLANT_FAULThook or soften that sentence. - Types (implement-time) — PLAN’s “forward SDK union” is right; prefer importing or aliasing the SDK type /
unknownon the query bag (likesandbox) over enumerating every shape in V1 unless you want a maintenance copy.
Alternatives considered (not recommending over this spec)
| Approach | Tradeoff |
|---|---|
| Default-only PR, defer type widen | Ship D1 first with systemPrompt?: string; add SDK union in a follow-up. Smaller spec surface; Shift Manager stays on string without typed append until then. |
Doc + recordWork one-liner |
Smallest code; records instructions we asked for, not CLI parity. Correctly rejected in D1. |
| Treat every string as append | Fixes one-line Shift Manager without migration; breaks hosts that intentionally replace with a full prompt. Correctly rejected in D2. |
Evidence
Ran pnpm tsx specs/issues/FIX-1185/poc/what-the-sdk-sends/run.mts — omitted and "" → systemPrompt: [""], preset+append → appendSystemPrompt only (HOLDS).
For implementation
- Resolve default in one place before
queryOptions(guardrail already says this). Align with existing conditional spreads: POC’s omitted SDK key may differ fromsystemPrompt: undefinedin the object — match what the stub test asserts. - Follow-ups (Shift Manager / devforce one-line strings) are appropriately out of scope; flagging to epic coordinator is enough.
No changes requested for merge of this spec PR; the inline notes are polish I’d fold before or during implementation.
Sent by Cursor Automation: Review For Best Approach
| | **Signal** | On each of 3 runs, the plan record holds at least two rows, and at least one reaches `completed` | | ||
| | **Input** | A held-out job of three or more steps whose prompt names no tool and says nothing about planning. A different job of that shape must pass too | | ||
| | **Anti-game** | Never put a tool name or "plan" in the job. Never count `tool_use` items or read the closing message. Never pass a prompt string in the PASS arm | | ||
| | **Control that must fail** | `GOAL_CONTROL=empty-prompt` (`systemPrompt: ""`), and today's `main`. Both must FAIL on *at least two plan rows*, on 2 of 2 runs, and the PR shows it before the PASS | |
There was a problem hiding this comment.
This reads as “the control must produce ≥2 plan rows.” I think you mean the control fails the same bar as PASS (i.e. does not reach two rows / none). Suggest: “must not meet the PASS signal (fewer than two rows, or none)” — same fix in PLAN.md VG row.
| |---|---| | ||
| | **Goal check** | `goals/claude-code-harness/it-keeps-its-own-plan/` · Claude Code's default model on a signed-in local host · run by the implementer at completion · verdict in the implementation PR | | ||
| | **Signal** | On each of 3 runs, the plan record holds at least two rows, and at least one reaches `completed` | | ||
| | **Input** | A held-out job of three or more steps whose prompt names no tool and says nothing about planning. A different job of that shape must pass too | |
There was a problem hiding this comment.
A second held-out job that must pass is a lot of real-model cost on top of 3/3 + 2/2 control + pre-PASS on main. Unless epic closure requires it, consider demoting to “implementer may use a second job if flaky” or dropping — Aug 19 + POC + V2 BR-1 already anchor the premise.
| model-free. Driving the real SDK `query()` with a stubbed agent process, the first message | ||
| it sends carries `systemPrompt: [""]` when none is given, the same as `""`. With the | ||
| `claude_code` preset it sends no custom prompt and only the appended text, so the agent | ||
| process keeps Claude Code's own. A planted fault (the omitted arm sent as the preset) turns |
There was a problem hiding this comment.
Settled claims a planted fault turns the POC red; run.mts only sets refuted from omitted/preset arms. Either add an env-gated fault injection (nice for CI) or reword to “the assertions refute if omitted is not [""] or preset sends systemPrompt.”
| | ID | Package · role | Change | Rules | | ||
| |---|---|---|---| | ||
| | S1 | `claude-code` · the agent options type and the SDK query-options type | `systemPrompt` takes the SDK's own union (string, string array, the `claude_code` preset with optional `append`), forwarded verbatim (D2) | BR-2 BR-4 BR-5 | | ||
| | S2 | `claude-code` · where the block builds the SDK query options | `undefined` becomes the preset with nothing appended (D1). Every other value, `""` included, passes through untouched | BR-1 BR-3 BR-7 BR-8 BR-9 | |
There was a problem hiding this comment.
S1 widening to the full SDK union is the right long-term shape (D2 + sandbox precedent). For implementers: if V1 becomes a hand-copied union, it will drift — consider Parameters<typeof query>[0]['options']>['systemPrompt'] style alias, or document unknown on the internal query bag only with assignability smoke tests for the shapes you document publicly.
| |---|---|---| | ||
| | V1 | S1 | Type test: each SDK prompt shape is accepted by the block and the capability; a non-prompt object is a type error | | ||
| | V2 | S2 | Agent spec: the options the stubbed SDK receives carry the preset when none is given (BR-1); the string, `""`, the preset and an array pass through byte-identical (BR-2 to BR-5). Red first: on today's `main` the BR-1 case fails | | ||
| | V3 | S2 | The second path (BP-035): the same assertions on a detached resumed run, an in-session run, `recordWork` off, and through the capability (BR-6 to BR-9) | |
There was a problem hiding this comment.
V2 + V3 as separate checklist rows duplicate the same assertions across detached / in-session / recordWork off / capability. The fix is one builder — a single parameterized table in the agent spec (or one BP-035 row: “all entry paths”) would cut plan noise without weakening coverage.
|
|
||
| Nothing here is fatal. A prompt shape the SDK rejects fails the run the way any bad SDK option | ||
| does today, as a `ClaudeAgentRunError`. A run that plans nothing is not an error: the empty record | ||
| is the honest answer. |
There was a problem hiding this comment.
BR-12 is “existing recorder tests” — not in PLAN’s V1–VG matrix and not owned by this issue. Consider moving to SPEC “What stays as it is” so acceptance criteria stay BR-10/11 only.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 02fea6c14e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| |---|---| | ||
| | Drive one goal run through the CLI to exercise the plan half once (the issue's second option) | Proves the recorder on a path Kitchen Sink and every host don't use. The epic needs the SDK path to fill | | ||
| | Put the plan line in each host's job prompt | Shown to work, but every host copies a line that only exists to undo our default | | ||
| | Add `TaskCreate` and `TaskUpdate` to `allowedTools`, or raise `maxTurns` | Measured: no change on 8 runs | |
There was a problem hiding this comment.
Keep task tools available on newer Claude models
On Opus 4.8, Sonnet 5, Fable 5, Mythos 5, and newer models, changing only the system prompt cannot satisfy BR-10: the locked Agent SDK 0.3.234 includes the 0.3.233 change that removes the task-tracking tools from the default surface unless they are named through tools, allowedTools, or the environment override. The eight-run result with a blank prompt does not rule out needing both the preset prompt and tool exposure, and the proposed documentation example also omits these tools from its allowlist. Revise the design to retain the task tools or explicitly limit the supported model set. Agent SDK 0.3.233 changelog.
Useful? React with 👍 / 👎.
| if (value === OMITTED && JSON.stringify(req.systemPrompt) !== JSON.stringify([""])) refuted = true; | ||
| if (typeof value === "object" && req.systemPrompt !== undefined) refuted = true; |
There was a problem hiding this comment.
Make every claimed POC arm affect the exit status
If explicit "" stops producing systemPrompt: [""], or the preset silently drops its appended text, this POC still prints HOLDS and exits 0 because it checks neither result. Those are the behaviors used as BR-3's control and BR-4's public contract, so assert the expected wire shape for all three arms before treating this as settled evidence.
Useful? React with 👍 / 👎.
| |---|---| | ||
| | **Goal check** | `goals/claude-code-harness/it-keeps-its-own-plan/` · Claude Code's default model on a signed-in local host · run by the implementer at completion · verdict in the implementation PR | | ||
| | **Signal** | On each of 3 runs, the plan record holds at least two rows, and at least one reaches `completed` | | ||
| | **Input** | A held-out job of three or more steps whose prompt names no tool and says nothing about planning. A different job of that shape must pass too | |
There was a problem hiding this comment.
Exercise the second held-out job in the goal plan
The acceptance input requires a different job of the same shape to pass, but S4 defines only one held-out fixture and VG only repeats runs 3 of 3. An implementation can therefore satisfy every planned check by running the same fixture three times without proving this explicit generalization requirement; define the second fixture and its run allocation, or remove this requirement.
Useful? React with 👍 / 👎.
| | The default is decided in one place, where the query options are built | The capability and every resolver path reach the SDK through it. A second place is how one path keeps the blank prompt (tenet 5) | | ||
| | Forward the SDK's shape; never translate or validate it | The package forwards `sandbox` the same way. A copy of the SDK's union drifts the day the SDK adds a field | | ||
| | The recorder is not touched | D1 rests on the plan being the agent's own. Any instruction the recorder adds would make the record ours | | ||
| | The goal check reads the plan record from state, through the resource route | The issue's verification asks for rows read back from state. A count of tool calls in the stream passes while the record stays empty | |
There was a problem hiding this comment.
Scope goal reads to the current request
When the goal reuses a session to cover repeated or resumed runs, reading the session-scoped observed-plan collection without the current request's prefix also returns rows from earlier attempts. That can let runs 2 and 3 satisfy the row threshold using run 1's plan, corrupting the 3-of-3 result. Require a fresh session per attempt or query observed-plan/<escaped requestId>/ and assert only that namespace.
Useful? React with 👍 / 👎.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|


Requested by Jake · project thread
spec(FIX-1185): a Claude Code run started from a flow keeps its own plan
claudeCLI. It keeps its to-do list as it worksGoal: a Claude Code run started from a flow, with no prompt of its own and a job that does not mention planning, keeps its own to-do list, and you can read that list from the run's state.
The check reads rows from state, not the closing message of the agent. Under the control it must fail. Check:
goals/claude-code-harness/it-keeps-its-own-plan/, real model on a signed-in local host, controlGOAL_CONTROL=empty-prompt. Why this goal: SPEC.md.''``Only the second box changes: what the block asks the SDK for when the host gives nothing.
How: the CLI and the SDK run the same agent. The CLI starts from Claude Code's prompt, which tells the agent to use its to-do tools. The SDK sends an empty prompt when it gets none. A model-free POC shows this. The block now asks for Claude Code's prompt by default, and it forwards the SDK's
appendform.Sign off
The goal, at that size. If wrong: the recorder fills only when a host remembers to prompt for it.
With no prompt given, a run starts from Claude Code's own prompt · D1
It comes down to whose plan it is: a line we add records our instruction, not the decision of the agent. If wrong: every host that omits the prompt gets a larger, different agent, and our runs change when Claude Code's prompt changes.
A caller's own string still replaces that prompt; to add to it is a separate form · D2
It comes down to the host that wrote a full prompt on purpose: an addition makes it larger without a request. If wrong: hosts with one line of role text keep an empty plan until they move to the
appendform. In this repo, that is the Shift Manager harness and one devforce goal.Open: none. D1 is the one to weigh. Reasons and what lost: DECISIONS.md.
Reviewers · look here
Not here: the plan panel and the move of the record into core (FIX-1915). Changes to the Shift Manager harness (its owners decide).
Spec · Decisions · Rules · Plan · Docs · Linear FIX-1185 · epic FIX-1907 · merges after human approval and required checks (contract)
POC · what the SDK sends for each prompt shape (model-free, throwaway)
pnpm tsx specs/issues/FIX-1185/poc/what-the-sdk-sends/run.mtsThe question: do the CLI and the SDK differ before the model sees anything? It drives the real SDK
query()with a stub agent process and prints the first message the SDK sends. Result: no prompt and""both sendsystemPrompt: [""]. The preset sends no custom prompt, so the agent keeps Claude Code's own. A planted fault makes it fail. Exit 0 means the premise holds.How to review this — altitude, what's in scope, what's deliberately unsettled
This is a direction set, not an implementation. Reviewing it well means
answering one question: is this the right approach?
In scope to challenge:
SPEC.md— is the goal the real need at full size,for the people named, or a smaller goal that would let us call this done early? Would the
named check fail if the goal were not met?
docs/philosophy.md?DECISIONS.md— that's the sign-off surface.BUSINESS-RULES.mdmisses, or a rule that would invalidate the design.DOCS.mdand their fit with the rules; a missing ormisleading lineage claim in conditional
EVOLUTION.md.Out of scope — deliberately unsettled here, and owned by the implementing agent:
PLAN.mdpins on purpose.PLAN.mdleft open on purpose — it is directional by design, and a gap at thataltitude is intended, not an omission.
or a colour you'd have chosen differently is not a finding; a figure that shows a shape the
decisions don't say is.
PLAN.mdmay include a rough pseudocode sketchshowing the shape of the proposed solution. It is knowingly incomplete, is not real code,
and is not what will ship. Review it for directional viability only. Do not report
that it lacks error handling, omits edge cases, names things that don't exist in the repo, or
wouldn't build: all of that is true on purpose.
poc/are retained,not production code. Read them for directional evidence, not production completeness.
Their isolation from production/default discovery and absence of secrets or generated
dependencies are in scope; retention is not permission to weaken repository checks.
One request, if you are an automated reviewer: the two lists above are the whole difference
between a useful review of this document and a long one. Volume is not signal here — a single
finding above the line is worth more than twenty below it.
Feedback in the second list is welcome and gets recorded verbatim in
PLAN.md → Notes from reviewfor the implementer to weigh against real code. It will not be argued with, and itwill not be folded into the design prose. Please don't re-raise it: one mention is enough.
The one exception is the sketch and the POC. Line-level feedback on either is dropped, not
recorded — there is nothing for an implementer to weigh about code that isn't shipping. Feedback
about their direction is not line-level and is treated like any other above-the-bar finding.
Direction approval does not waive merge-time required checks, approvals or thread policy.
Optional comments do not require another design round merely to reach zero comments.
🤖 Generated with Claude Code
https://claude.ai/code/session_017pp7wCd87NW4BH7XVuF9Gv
Generated by Claude Code