Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(goals): stop the capture and the grader from going blind (LAB-136…
… review round 2)

Two more instances of the failure this goal keeps finding: a check that cannot
see what it claims to measure, reporting fine.

1. The capture could lose the brief it was measuring. Each model invocation
   overwrote `capture.messages`, so a second call — from a runtime regression or
   a retry — silently discarded the first handoff and left `briefsAgree` and the
   content checks grading only the final stable call. A malformed or
   pass-dependent first brief was invisible.

   Asserted at exactly one invocation rather than accumulating. This recipe is
   tool-free and terminal at step 0, so a second call is a regression by
   definition: the assertion is both the guard and its own canary, and
   accumulating would grade a case the design says cannot happen.

2. The 15-contribution grader checked presence, not placement. It asked whether
   each scalar appeared somewhere in its half, so swapping `linesAdded` and
   `linesRemoved` left every leaf present and every signal green while the brief
   reported the wrong change size. A held-out world with two equal values hid an
   omitted field the same way.

   The ambiguous fields are now graded on their labelled fragment — `+47`, `−12`,
   `3 files`, `#101` — built from the rendered form rather than the bare value.
   The labels mirror `renderTail` and sit beside it. The graded SET is still
   derived by walking the fixture, so a field added to the world and never
   rendered still fails; only the expectation for ambiguous fields is tightened.

Both verified by breaking them: swapping the two metrics goes red naming the
expected fragments (it was fully green before), and giving the recipe a tool with
a tool call scripted on step 0 produces two invocations and goes red.

No new abstraction: +25 lines of code, no grading framework, no shared helper.
Comment share unchanged at 22%.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
  • Loading branch information
claude committed Aug 18, 2026
commit cd07cdcd7fb85e8dcc75367e29e6c2b758dc45d7
14 changes: 10 additions & 4 deletions goals/context-supply/assembles-the-same-brief-every-pass/goal.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,18 @@
**Signal:**
- **Eight passes, one issue.** Eight assemblies of the recipe, each over a fresh copy of the world, render to the same bytes under one named comparator (`briefsAgree`).
- **The precondition.** Exactly one leading `system` message and none after it — which is what makes "the leading system run *is* the standing half" exact for this recipe rather than a heuristic. If it fails, the boundary derivation is void and the standing-half signal is withdrawn, not reinterpreted.
- **Every contribution the brief claims to carry is present.** The expected set is derived by *walking the fixture world*, never hand-listed — the phase instructions, the project grounding, the review standards and the objective in the standing half; the issue's id, title, body, PR metrics, diff and verification statement in the changing half. A contribution added to the world but never rendered fails here, and the list cannot drift out of agreement with what the brief promises. Without this the *shape* could stay perfectly stable while most of the *content* silently vanished: eight passes would still agree, the split would still hold, the canary would still fire.
- **Exactly one model invocation per pass.** Each invocation overwrites the capture, so a second call would silently discard the first handoff and leave every check below grading the *last* brief rather than the one the recipe assembled. This recipe is tool-free and terminal at step 0, so a second call is a regression by definition — asserting one is both the guard and its own canary. Accumulating instead would grade something the design says cannot happen.
- **Every contribution the brief claims to carry is present *and correctly placed*.** The expected set is derived by *walking the fixture world*, never hand-listed — the phase instructions, the project grounding, the review standards and the objective in the standing half; the issue's id, title, body, PR metrics, diff and verification statement in the changing half. A contribution added to the world but never rendered fails here, and the list cannot drift out of agreement with what the brief promises. Fields whose bare value cannot identify them (the PR metrics) are graded on their **labelled fragment** — `+47`, `−12`, `3 files`, `#101` — not on the digits alone, because presence cannot tell `linesAdded` from `linesRemoved`: swap them and every leaf is still somewhere in the tail while the brief reports the wrong change size. The labels mirror `renderTail` and are kept beside it. Without all of this the *shape* could stay perfectly stable while the *content* silently vanished or transposed: eight passes would still agree, the split would still hold, the canary would still fire.
- **Two different issues.** The standing half is byte-identical between issue A and issue B; the changing half **differs**; each carries its own contributions; and neither carries the other's identifying facts (id, title, PR number).

**Anti-game:** A hollow pass here looks like one of five things, and each has a check that closes it.
**Anti-game:** A hollow pass here looks like one of seven things, and each has a check that closes it. Five of the six are the same failure at different depths — *an instrument that cannot see what it claims to measure, reporting fine* — which is worth naming, because it is the shape every defect found on this issue has taken.
1. *A comparator that could never return false.* → A variant recipe whose grounding appends a **per-call counter** must make the same named `briefsAgree` return false. A counter, never the clock: in-process assemblies can land in one tick, and a flaky check is worse than none. Two assemblies suffice for the canary — a per-call counter differs on the second; the primary loop runs eight because that is the claim's language.
2. *A brief so thin nothing in it could drift.* → Every fact graded is pulled from the fixture; the assembled brief is printed on every run so a human sees what was actually measured.
3. *A stable shape over draining content.* → The derived contribution assertions above. This is the one that makes the goal a regression check rather than a shape check, because the human read that would otherwise catch it is a one-time verdict-log row that `goal:all` never re-runs. (Verified: dropping the diff and the verification statement, then the standards and the objective summary, turns it red and names the missing fixture paths — while every other signal stays green.)
4. *A standing-half comparison between two briefs that are secretly the same brief.* → Requiring the changing halves to differ and to carry their own fixture's facts. Without those two assertions a recipe that assembled issue A twice passes every other signal. (Verified: forcing exactly that turns three assertions red.)
5. *Measuring the wrong mechanism and passing green.* → The model stub is step-capable and its legacy `generate` **throws**. A `generate()`-only stub drives the SDK-owned compatibility path, which contains no `emitGeneratorStep` call sites and which production never takes. (Verified: removing `generateStep` routes the run into `generate()` and the guard fires.)
4. *Content present but transposed.* → Labelled-fragment grading for the fields whose bare value is ambiguous. (Verified: swapping `linesAdded` and `linesRemoved` in `renderTail` turns it red naming `+47` and `−12`; before this it was fully green, since both digits were still in the tail.)
5. *A capture that quietly lost the brief it was measuring.* → The one-invocation assertion. (Verified: giving the recipe a tool and scripting a tool call on the first step produces two invocations and turns it red.)
6. *A standing-half comparison between two briefs that are secretly the same brief.* → Requiring the changing halves to differ and to carry their own fixture's facts. Without those two assertions a recipe that assembled issue A twice passes every other signal. (Verified: forcing exactly that turns three assertions red.)
7. *Measuring the wrong mechanism and passing green.* → The model stub is step-capable and its legacy `generate` **throws**. A `generate()`-only stub drives the SDK-owned compatibility path, which contains no `emitGeneratorStep` call sites and which production never takes. (Verified: removing `generateStep` routes the run into `generate()` and the guard fires.)

The stub implements `generateStep` and **not** `streamStep`. Both are optional on `GeneratorModel` (only `modelId` and `generate` are required), and this recipe is tool-free and identity-less, so `canStream` is false and the streaming branch is unreachable — a `streamStep` here would be dead surface. Adding tools or `itemVisibility` later cannot silently reroute to a legacy path either: `canStream` also requires `stream` or `streamStep` to exist, and neither does.

Expand Down Expand Up @@ -49,6 +52,7 @@ Spelled that way alone, the step is unsound. The constraint check has to run **i
| 2026-08-18 | f56c6b216 (+goal) | n/a | FAIL (human read) | First read of the assembled brief against the standards it carries. Assembly was correct and all machine signals were green, but the brief asked a reviewer to check that "every claim has a re-runnable evidence path" while giving them nothing to check it against — no tests, no statement of what the author ran. Gap was in the recipe's world, not in the framework's assembly. Fixed by adding a `verification` field per pull request and rendering it in the changing half. Recorded because the human gate is only real if its red state is. |
| 2026-08-18 | f56c6b216 (+goal) | n/a | PASS | 8 passes over `ISSUE-1`, fresh world each, byte-identical (2153 bytes; standing half 1258). Precondition held: exactly one leading system message. `ISSUE-2` left the standing half unchanged, differed in the changing half, and carried its own facts and none of `ISSUE-1`'s. Anti-game: the per-call-counter variant made `briefsAgree` return false and localized to the **recipe**. Constraint guard: a nested context formatter looks stable to the raw slot signal (the hazard), is caught by the guard, and localization consequently refuses to blame the seam. **No framework finding** — the assembly seam was deterministic across every pass. |
| 2026-08-18 | f56c6b216 (+goal, review round 1) | n/a | PASS | Re-run after review. The check now grades **all 15 contributions** the brief claims to carry, derived by walking the fixture — previously only 3 (id, title, PR number), which left the body, diff, verification, PR metrics, project grounding, standards and objective ungraded and a content-drop invisible to every automated signal. Also: substring-overlap guard at setup so the held-out claim survives a colliding replacement world; per-pass `goalSessionId` slugs (the stamp is per-process, so a bare slug would share one session across all eight passes and accumulate history); canary reduced to 2 assemblies; the always-on constraint check removed as redundant with `localize`'s ordering and the nested canary; `streamStep` removed from the stub as unreachable on this non-streaming path. Same result: byte-identical across eight passes, standing half survives a change of issue, no framework finding. |
| 2026-08-18 | f56c6b216 (+goal, review round 2) | n/a | PASS | Two more blind-instrument holes closed. (1) The capture itself could lose data and report green: each invocation overwrote `capture.messages`, so a second call would discard the first handoff and leave everything grading the last brief — now asserted at exactly one invocation, which for a tool-free step-0-terminal recipe is guard and canary in one. (2) The 15-contribution grader checked presence, not placement: swapping `linesAdded` and `linesRemoved` left every leaf present and every signal green while the brief reported the wrong change size — the ambiguous fields are now graded on their labelled fragment (`+47`, `−12`, `3 files`, `#101`). Both verified by breaking them. Same result: byte-identical across eight passes, standing half survives a change of issue, no framework finding. |

### Falsification runs behind that PASS
Each was performed by breaking the check on purpose and confirming it went red, then reverting. A green from a check nobody has seen fail is decoration.
Expand All @@ -63,3 +67,5 @@ Each was performed by breaking the check on purpose and confirming it went red,
| The diff and the verification statement dropped from the tail | RED — named `issue(ISSUE-1).pullRequest.diff` and `.verification`, for both issues. Every other signal stayed green, which is the point |
| The standards and the objective summary dropped from the grounding | RED — named `grounding.standards`, `objective.summary` |
| A replacement world whose two issues are `ISSUE-1` and `ISSUE-10` | RED at setup — the leak check uses substring containment, so overlapping identifiers are rejected at the boundary rather than reported as a false leak |
| `linesAdded` and `linesRemoved` swapped in `renderTail` | RED — named `+47` and `−12` as the expected fragments. Fully green before labelled-fragment grading, since both digits were still present in the tail |
| A tool added and a tool call scripted on step 0, forcing two invocations | RED — "called 2 time(s), expected exactly 1. The capture is void." Before the assertion the second call would have overwritten the first brief and everything downstream would have graded the wrong one |
58 changes: 51 additions & 7 deletions goals/context-supply/assembles-the-same-brief-every-pass/run.mts
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,8 @@ type SlotName = "instructions" | "grounding" | "tail";
interface Capture {
messages: unknown[];
slots: Partial<Record<SlotName, unknown>>;
/** Step-method invocations this pass. Must be exactly 1 — see `assemble`. */
calls: number;
}

/** The changing half: this issue, this pull request, this diff. */
Expand Down Expand Up @@ -115,7 +117,7 @@ async function assemble(
grounding: (w: World) => unknown,
passLabel: string,
): Promise<Capture> {
const capture: Capture = { messages: [], slots: {} };
const capture: Capture = { messages: [], slots: {}, calls: 0 };
const record = <T,>(slot: SlotName, value: T): T => {
capture.slots[slot] = value;
return value;
Expand All @@ -136,6 +138,7 @@ async function assemble(
// a legacy path either: `canStream` also requires `stream` or `streamStep` to
// exist, and neither does.
async generateStep(options: GeneratorModelCallOptions) {
capture.calls += 1;
capture.messages = options.messages;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Record and validate every model invocation

If a runtime regression or retry invokes this tool-free generator more than once, each call overwrites capture.messages, discarding every earlier model handoff. An earlier malformed or pass-dependent brief can therefore be ignored while briefsAgree and the content checks grade only the final stable call, allowing the goal to pass even though the model received non-identical briefs; accumulate all invocations or assert that exactly one occurred.

Useful? React with 👍 / 👎.

return { text: "", finishReason: "stop" };
},
Expand Down Expand Up @@ -171,7 +174,18 @@ async function assemble(
} as never,
});
if (result.error) throw new Error(`assembly run failed: ${result.error.message}`);
if (capture.messages.length === 0) throw new Error("the model stub was never called");
// Exactly one, not "at least one". Each invocation OVERWRITES `capture.messages`,
// so on a second call every earlier handoff is silently discarded and the checks
// below grade only the last one — a malformed or pass-dependent first brief would
// be invisible. This recipe is tool-free and terminal at step 0, so a second call
// is a regression by definition: asserting it is both the guard and the canary.
if (capture.calls !== 1) {
throw new Error(
`the model stub was called ${capture.calls} time(s), expected exactly 1. The capture keeps ` +
`only the last handoff, so every brief graded below would be the final call rather than ` +
`the one the recipe assembled. The capture is void.`,
);
}
return capture;
}

Expand Down Expand Up @@ -253,15 +267,43 @@ const standingContributions = (): Leaf[] => [
/** Contributions one issue's changing half claims to carry. */
const changingContributions = (id: string): Leaf[] => leaves(issueOf(WORLD, id), `issue(${id})`);

/**
* How a field is PLACED in the brief, for the fields whose bare value cannot
* identify them. Mirrors the labels in `renderTail` — keep the two together.
*
* Presence alone grades the wrong thing here: swap `linesAdded` and
* `linesRemoved` and every leaf is still somewhere in the tail, so a brief
* reporting the wrong metrics passes. A held-out world with two equal numbers
* hides an omitted field the same way. Grading the labelled fragment ties each
* value to the field it was rendered for. Leaves absent from this list are
* distinctive prose and are graded on their text.
*/
const PLACED: Array<[string, (v: string) => string]> = [
["pullRequest.number", (v) => `#${v}`],
["pullRequest.filesChanged", (v) => `${v} files`],
["pullRequest.linesAdded", (v) => `+${v}`],
["pullRequest.linesRemoved", (v) => `−${v}`],
];

const expectedFragment = (leaf: Leaf): string => {
const placed = PLACED.find(([suffix]) => leaf.path.endsWith(suffix));
return placed === undefined ? leaf.text : placed[1](leaf.text);
Comment on lines +304 to +306

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bind every contribution to its rendered field

Fresh evidence beyond the prior placement comment is that the new PLACED table fixes only the four PR scalars; this fallback still grades every other leaf by unscoped substring containment across the entire half. For example, swapping reviewGrounding's project and standards assignments leaves both fixture strings present, so every signal remains green while the model receives each value under the wrong XML field, contradicting the goal's “correctly placed” guarantee. Bind the remaining structured leaves to their tags or labeled regions as well.

Useful? React with 👍 / 👎.

};

/** Context values are XML-escaped on the way into the system message; prompt and
* user values are not. Accept either form so the check works for both slots. */
const escapeXml = (s: string): string =>
s.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
const carries = (haystack: string, leaf: string): boolean =>
haystack.includes(leaf) || haystack.includes(escapeXml(leaf));
const carries = (haystack: string, fragment: string): boolean =>
haystack.includes(fragment) || haystack.includes(escapeXml(fragment));
Comment on lines +313 to +314

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Distinguish contributions instead of matching substrings

Fresh evidence after the earlier overlap fix is that only cross-issue identity collisions are guarded; arbitrary leaves in the same region can still overlap. For example, with a valid replacement fixture whose body repeats its title, removing the title from renderTail still passes because this includes() finds the title inside the body, so the goal reports all contributions present despite the regression. Reject all same-region leaf overlaps during setup or grade structured, field-specific fragments.

Useful? React with 👍 / 👎.


const missingFrom = (haystack: string, expected: Leaf[]): string[] =>
expected.filter((l) => !carries(haystack, l.text)).map((l) => l.path);
expected
.filter((l) => !carries(haystack, expectedFragment(l)))
.map((l) => {
const fragment = expectedFragment(l);
return fragment === l.text ? l.path : `${l.path} (expected ${JSON.stringify(fragment)})`;
});

// --- identity, for the leak check and the setup guard ----------------------

Expand Down Expand Up @@ -498,8 +540,10 @@ await runGoal(async () => {
`${PASSES} passes of the review recipe, each over a fresh copy of the fixture world, handed the ` +
`model a byte-identical brief (${renderBrief(a.messages).length} bytes, standing half ` +
`${standingHalf(a).length}); the boundary precondition held (exactly one leading system ` +
`message); all ${graded} contributions the brief claims to carry were present, derived by ` +
`walking the fixture rather than hand-listed; assembling ${ISSUE_B} instead left the standing ` +
`message); each pass called the model exactly once, so no handoff was overwritten; all ` +
`${graded} contributions the brief claims to carry were present, derived by walking the ` +
`fixture rather than hand-listed and graded on their rendered placement, not bare presence, ` +
`so a value under the wrong label fails; assembling ${ISSUE_B} instead left the standing ` +
`half unchanged while the changing half differed, carried its own contributions and none of ` +
`${ISSUE_A}'s identity. The comparator is proved able to fire: the same named comparator over ` +
`a grounding that appends a per-call counter returned false, and localization attributed ` +
Expand Down
Loading