Skip to content

spec(FIX-1185): amend after implementation: parity default, narrowed goal - #3193

Merged
jhoffner merged 1 commit into
mainfrom
spec/FIX-1185-amend-parity
Oct 12, 2026
Merged

jhoffner merged 1 commit into
mainfrom
spec/FIX-1185-amend-parity

Conversation

@jhoffner

Copy link
Copy Markdown
Collaborator

Requested by Jake · project thread

This is a post-merge amendment to the FIX-1185 spec (original review: #3166), following the "ship parity" direction. That direction is with Jake, and this stays a draft until he confirms it.

The problem

The spec's goal was "a Claude Code run with no prompt of its own plans on its own." The implementation (#3190) ran that check on Claude Code's current default model (claude-sonnet-5, Agent SDK 0.3.234), and it failed 0 of 3. The empty-prompt control failed the same way. There are two reasons:

What this changes

  • Goal (SPEC.md): narrowed to CLI parity plus a recorder proved on a real run. A default arm checks that the SDK is handed Claude Code's own prompt. A planning arm, with the tools named and the job asking for them, checks that the plan lands in the record. The controls are empty-prompt and no-record. The pinned folder name is unchanged.
  • D1 (DECISIONS.md): records its outcome. "What would change my mind" fired, and D1 still ships, now for CLI parity alone. The fallback (a plan line the block owns) is not adopted here and is left to the epic.
  • Settled: the Codex P1 is marked CONFIRMED with the evidence, alongside "offering the tools is not enough" and a re-measured Aug 19 claim.
  • BR-10 and BR-11, VG and S4 (BUSINESS-RULES.md, PLAN.md): rewritten to match the narrowed goal.
  • DOCS.md: no longer promises that the plan fills by default, and matches what fix(claude-code): start a run from Claude Code's own prompt by default (FIX-1185) #3190 now publishes.

What's asked of you

Confirm the narrowed goal and the parity framing of D1. If Jake picks a block-owned plan instruction instead, this amendment changes and #3190 grows.

Parts worth reviewing closely

  • SPEC.md's amended goal table and its "Bigger, and dropped after measurement" row
  • DECISIONS.md → D1's Outcome and the new Settled entries

Implementation: #3190 · Linear FIX-1185 · epic FIX-1907

🤖 Generated with Claude Code

https://claude.ai/code/session_017pp7wCd87NW4BH7XVuF9Gv


Generated by Claude Code

…goal

The goal check failed on Claude Code's current default model (0 of 3; the
empty-prompt control failed the same way). Records D1's outcome, settles
the Codex P1 from #3166 as confirmed (to-do tools off by default on newer
models), narrows the goal to CLI parity plus a recorder proved on a real
run, and rewrites BR-10/BR-11, VG and the docs draft to match.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017pp7wCd87NW4BH7XVuF9Gv
@linear-code

linear-code Bot commented Oct 12, 2026

Copy link
Copy Markdown

FIX-1185

@vercel

vercel Bot commented Oct 12, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
flow-state-dev-docs Ignored Ignored Oct 12, 2026 10:13pm UTC
flow-state-dev-kitchen-sink Ignored Ignored Oct 12, 2026 10:13pm UTC
flow-state-dev-ui Ignored Ignored Oct 12, 2026 10:13pm UTC

Request Review

@jhoffner
jhoffner marked this pull request as ready for review October 12, 2026 23:07
@jhoffner
jhoffner merged commit 623b512 into main Oct 12, 2026
19 checks passed

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/simplify review — spec amendment (FIX-1185)

Verdict: The engineering call in this amendment looks right: measure on the real default model, stop promising autonomous planning, ship D1 as CLI parity, and prove the recorder on an explicit planning arm with anti-game controls. That is simpler and more honest than keeping the original “3/3 plan rows with no prompt” goal or adopting the D1 “plan line” fallback (which would still be our plan, not the agent’s).

I am not approving in this pass because several contradictory remnants and duplicated amendment narrative make the spec harder to implement and review than it needs to be. None of these require re-opening D1; they are tighten-ups in the same PR.

Where the approach may be heavier than necessary

Area What the PR does Simpler alternative (if you want less machinery)
Goal harness Two jobs × two arms × two controls; default arm also prints ungraded “did it plan?” Grade only prompt:preset + empty-prompt on one held-out job (V2/POC already cover stubbed SDK). Keep one real-model run for plan:rows + no-record on the planning arm only. Matches sign-off (“CLI parity + recorder proved once”) with fewer sessions (~2–3 vs up to ~6).
Spec structure Large SPEC blockquote + parallel restatement in DECISIONS / BR / DOCS / figure footnote Peers (FIX-1589, FIX-1663) use EVOLUTION.md for post-measurement goal narrows; FIX-1916-style DECISIONS-only history is the lighter option if you skip Evolution. One canonical home, forward SPEC reads as today’s truth.
Epic closure Child spec now admits panel usually empty on default model Consider a small FIX-1907 EVOLUTION row so ER-7 / “fills” language does not fight this amendment at epic wrap (out of scope here, but worth tracking).

What I would fix before merge (recommended)

  1. D1 “What would change my mind” vs Outcome — still promises the plan-line fallback; Outcome correctly rejects it. Reconcile so implementers do not read two futures.
  2. SPEC fold (lines 15–18) — still says the CLI prompt “tells the agent to track its work with its to-do tools” as the core story; contradicts Settled + narrowed goal. Reframe as prompt assembly / parity; tool gating belongs in Settled or the planning arm.
  3. DECISIONS opening (“FIX-1185 makes Claude Code’s fill”) — same drift; narrow to “default prompt parity + recorder path proved when tools are offered and used.”
  4. Goal table row label — “Smaller, and rejected” now describes the adopted planning arm; rename (e.g. “Recorder leg (not default behaviour)”).
  5. DOCS.md editor note (L12–13) — “keeps the run’s to-do list” fights the amended user-facing body; aim readers at preset shape, not planning outcome.
  6. PLAN guardrail (recorder / “D1 rests on the plan being the agent’s own”) — after amendment, D1 is prompt-only; plan proof is BR-11 / planning arm only.

What I would not change

  • Keeping D1 (preset default) despite failed autonomous planning — correct tradeoff vs injecting plan guidance in the block.
  • Planning arm + no-record control — needed if this issue must close with a real-model recorder proof (cannot mock without weakening the claim).
  • Settled Codex P1 / tools-off-by-default bullets — high value, keep; optional merge into one structured finding for readability.

Alternative approach (only if epic re-scopes)

If FIX-1907 owns “panel fills on default hosts,” this issue could shrink to D1 + docs + V2 only and move the real-model recorder goal to a sibling issue or epic leg — smaller FIX-1185, but splits closure evidence. I would not do that without an explicit epic decision; the current dual-arm design is defensible if FIX-1185 must own recorder proof.

Happy to approve after the contradiction cleanups above (especially D1 + SPEC lede); the narrowed substance does not need a rethink.

Open in Web View Automation 

Sent by Cursor Automation: Review For Best Approach

@@ -38,6 +38,14 @@ It comes down to whose plan it is: a line we inject records our instruction, not
plan rows. Then the blank prompt was not the whole cause, and the plan line is the fallback, said
in the docs as an instruction we add.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/simplify · contradiction: This paragraph still commits to the plan-line fallback, but Outcome (below) says that path fired and is not adopted. Suggest rewriting WWCMM as historical (“condition met; we chose CLI parity + epic follow-up instead”) so D1 has one story.

@@ -38,6 +38,14 @@ It comes down to whose plan it is: a line we inject records our instruction, not
plan rows. Then the blank prompt was not the whole cause, and the plan line is the fallback, said

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/simplify: After the amendment, “makes Claude Code’s fill” overstates what FIX-1185 closes. Consider “ships default prompt parity and proves the recorder when the agent uses to-do tools” — aligns with sign-off and avoids fighting Settled (“CLI plans, SDK doesn’t” doesn’t reproduce on current default).

|---|---|
| **The real need** | "The agent's own account of what it planned to do is present via the CLI and absent via the SDK" ([the issue](https://linear.app/fixpoint-labs/issue/FIX-1185)). The epic needs Claude Code's plan panel to fill ([ER-7](../../epics/FIX-1907/BUSINESS-RULES.md#what-a-team-gets-and-what-it-doesnt), [D3](../../epics/FIX-1907/DECISIONS.md#d3)) |
| **Smaller, and rejected** | "A run whose job prompt says *use TaskCreate* fills the record." Already shown once. It proves the recorder, not that the agent plans on its own, and every host would have to copy the line |
| **Smaller, and rejected** | "A run whose job prompt says *use TaskCreate* fills the record", on its own. It proves the recorder but says nothing about the default. It is now the goal's second arm, beside the default check |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/simplify · labeling: This row is now the adopted planning arm, but the label still says “Smaller, and rejected.” Renaming (e.g. “Recorder leg (not default behaviour)”) removes a forced mental undo for readers.

| **Anti-game** | Never put a tool name or "plan" in the job. Never count `tool_use` items or read the closing message. Never pass a prompt string in the PASS arm |
| **Control that must fail** | `GOAL_CONTROL=empty-prompt` (`systemPrompt: ""`), and today's `main`. Both must FAIL on *at least two plan rows*, on 2 of 2 runs, and the PR shows it before the PASS |
| **Goal check** | `goals/claude-code-harness/it-keeps-its-own-plan/` (the name is pinned; the claim is the narrowed one above) · Claude Code's default model · run by the implementer at completion · verdict in the implementation PR |
| **Signal** | Default arm, each job: the SDK was handed `{ type: "preset", preset: "claude_code" }`. Planning arm, each job: at least two plan rows, one `completed`. Whether the default arm planned unasked is printed and not graded |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/simplify · verification cost (optional): Two held-out jobs for the default arm mostly doubles real-model time for prompt:preset, which is job-independent. One graded job + second fixture only in the verdict log (or planning arm only) would preserve honesty with fewer runs — worth stating in PLAN if you keep two fixtures.

![Two rows, the same run. Today the block asks for nothing, the SDK sends an empty prompt and the plan record stays empty. After, the block asks for Claude Code's own prompt and the record fills. A full prompt string or an empty one still passes through unchanged](figures/what-changes.svg)

Only the left box moves: what the block asks the SDK for when the caller gave nothing.
Only the left box moves: what the block asks the SDK for when the caller gave nothing. The figure predates the [amendment](#amended-after-implementation): the record fills only when the agent keeps a to-do list, which on today's default model it usually doesn't.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/simplify: Good disclaimer. For a cleaner fold, consider shortening the figure alt text itself (still says “the record fills”) so readers do not need the footnote to correct the first impression.

@@ -29,8 +29,8 @@ const agent = claudeCodeAgent({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/simplify · contradicts amended DOCS body: This editor instruction still says the first example should “keep the run’s to-do list.” After the amendment, preset + recordWork do not imply that. Suggest: first example should match CLI-default prompt shape (preset + append), not planning outcome.

@@ -58,7 +58,7 @@ The option keeps its name, `systemPrompt`. Everything else is yours to name.
| Forward the SDK's shape; never translate or validate it | The package forwards `sandbox` the same way. A copy of the SDK's union drifts the day the SDK adds a field |
| The recorder is not touched | D1 rests on the plan being the agent's own. Any instruction the recorder adds would make the record ours |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/simplify: “D1 rests on the plan being the agent’s own” no longer matches D1 Outcome (CLI parity, no promise the agent plans). Suggest: D1 is prompt default only; plan rows are proved in the planning arm / BR-11, not by changing the default prompt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants