Repository navigation
Fixture proof: one phase's context recipe, byte-identical across eight passes (LAB-136) - #1325
Conversation
…asses (LAB-136) Adds a model-free goal under `goals/context-supply/` that assembles one phase's context recipe — review — over a hand-built fixture world and checks that the framework hands the model the same bytes on every pass, and that the brief's standing half survives a change of issue. Nothing reusable ships: no assembly machinery, no recipe format, no store, no second phase. Three files in one goal folder; zero workspace members, zero config edits, no `packages/*` changes. Three things the check does deliberately, each of which fails GREEN if done the obvious way instead: - The model stub is step-capable and its legacy `generate` throws. A `generate()`-only stub drives the SDK-owned compatibility path production never takes; measuring the brief there passes while testing the wrong mechanism. Verified by removing `generateStep` and watching the guard fire. - The two-issue check requires the changing halves to DIFFER and to carry their own fixture's facts, not just that the standing halves match. Verified by assembling issue A twice: three assertions go red. - The drift canary uses a per-call counter, never the clock, and calls the same named comparator the real signals use. One correction to the spec's localization step, found by building it. Capturing slot outputs and reading "slots agree" as an assembly-seam finding is unsound: `resolveSlotValues` resolves only top-level slot functions, so a function nested inside object-form context resolves later and never enters the capture. That recipe drifts the brief while the capture reports the slot stable, and the step emits a confident false framework finding — the expensive direction, since it pauses the epic. The fix is to run the recipe constraint INSIDE the attribution: an unresolved slot means the capture is blind, and blindness is not evidence of stability. Both behaviours are recorded in the verdict log. No framework finding: the assembly seam was deterministic across every pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|
| Internal only — no published package changes. | ||
|
|
||
| Adds one model-free goal, `goals/context-supply/assembles-the-same-brief-every-pass`, proving that a phase's context recipe hands the model a byte-identical brief on every pass and that its standing half survives a change of issue. Nothing reusable ships: no assembly machinery, no recipe format, no persistence, no second phase. Three files in one goal folder; no workspace member, no config edit, no change to any package. | ||
|
|
||
| The recipe is a real generator run through the real `runAction`, so what is measured is the framework's own message assembly rather than a helper written for the test. The stub is step-capable and its legacy `generate` throws, because a `generate()`-only stub drives the SDK-owned compatibility path production never takes — and would have passed green while measuring the wrong mechanism. | ||
|
|
||
| The assembly seam was deterministic across every pass, so there is no framework finding to file. |
There was a problem hiding this comment.
Populated internal-only Changeset
This change is explicitly internal-only, but the empty-package Changeset contains release-note prose, causing internal implementation details to enter the release-note pipeline. Internal-only changes should use an empty fragment.
| Internal only — no published package changes. | |
| Adds one model-free goal, `goals/context-supply/assembles-the-same-brief-every-pass`, proving that a phase's context recipe hands the model a byte-identical brief on every pass and that its standing half survives a change of issue. Nothing reusable ships: no assembly machinery, no recipe format, no persistence, no second phase. Three files in one goal folder; no workspace member, no config edit, no change to any package. | |
| The recipe is a real generator run through the real `runAction`, so what is measured is the framework's own message assembly rather than a helper written for the test. The stub is step-capable and its legacy `generate` throws, because a `generate()`-only stub drives the SDK-owned compatibility path production never takes — and would have passed green while measuring the wrong mechanism. | |
| The assembly seam was deterministic across every pass, so there is no framework finding to file. |
Context Used: CLAUDE.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b0094f5771
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const factsOf = (id: string): string[] => { | ||
| const issue = issueOf(WORLD, id); | ||
| return [issue.id, issue.title, `#${issue.pullRequest.number}`]; |
There was a problem hiding this comment.
Assert every required brief contribution
If a later recipe or framework regression deterministically drops the issue body, diff, verification, PR metrics, project grounding, standards, or objective, this goal still reports PASS: the equality and split checks remain green, while factsOf() grades only the ID, title, and PR number. The verdict log's one-time human read is not rerun by goal:all, so add fixture-derived assertions for every contribution the goal claims the reviewer receives.
Useful? React with 👍 / 👎.
| if (missing.length > 0) { | ||
| failures.push(`${id}'s changing half is missing its own fixture's facts: ${missing.join(", ")}`); | ||
| } | ||
| const leaked = factsOf(other).filter((f) => tail.includes(f)); |
There was a problem hiding this comment.
Compare held-out facts without substring collisions
When a valid replacement fixture contains overlapping facts such as ISSUE-1/ISSUE-10, #10/#101, or one title containing another, the correct longer issue tail is falsely reported as leaking the shorter issue because includes() performs a substring match. The setup guard rejects only exactly equal facts, so this breaks the stated guarantee that another valid world can be swapped in; compare structured fields or reject substring overlaps during setup.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Simplify review (LAB-136 goal check)
Ran pnpm tsx goals/context-supply/assembles-the-same-brief-every-pass/run.mts — PASS in ~1.3s. Direction is sound; I would not swap this for an assembleMessages unit test or @flow-state-dev/testing's mockGenerator (legacy generate path only — your falsification table already shows that false-greens).
The execution path looks like the right minimum for "what the framework hands the model on the step path." What reads heavy is mostly presentation and attribution machinery, not a wrong architectural choice.
What I'd trim (behavior-preserving)
-
Comment duplication — Roughly half of
run.mtsrestatesgoal.md(recipe constraint, localization ordering, anti-game, decision 3). The goal doc is excellent; the runner could keep a short header (run command + pointer togoal.md) and one-line section labels. That alone is probably ~80–120 lines without weakening the proof. -
Session id pattern — Per-pass isolation is load-bearing.
Math.random()works, but the corpus pattern isgoalSessionIdwith a per-pass slug (e.g.goalSessionId(lab136-pass-${i})). BaregoalSessionId("lab136")would reuse one id for the whole process becauseRUN_STAMPis per-run, not per-pass. -
Micro hoist —
defineFlow/ recipe shell inside everyassemble()is identical except thegroundingclosure; hoisting the static shell is a small win on 19runActioncalls (wall-clock, not correctness).
What I would not cut
- Eight passes (signal + canary) — contract in
goal.md; clock-based drift is explicitly out of scope. - Localization + nested-formatter guard — verdict log shows a real false "report and stop — seam" before the ordering fix. Heavy, but tied to decision 3, not speculative abstraction.
- Two-issue standing/changing checks — closes the "assembled issue A twice" vacuous-green hole; already falsified in your table.
- Step-capable stub with throwing
generate()— correct guardrail; aligns withstepModel()in engine tests but shouldn't be imported from test internals.
Simpler approaches considered
| Alternative | Verdict |
|---|---|
assembleMessages unit test (message-assembly.test.ts) |
Wrong surface — grades helper return, not model input |
testFlow + mockGenerator |
Wrong path — no generateStep |
| Drop anti-game / localization | Weaker proof; decision 3 failure mode returns |
| Third-party stable-stringify | Doesn't buy byte identity of the messages array |
Follow-ups (optional, not blocking)
- If a second goal needs the recording stub,
createMessageRecorderModel()ingoals/lib/model.mts(README bar: third consumer). @flow-state-dev/testingmockGeneratorwithgenerateStepwould help future goals — out of scope here.Promise.allon the two eight-pass loops could cut elapsed time without changing pass count.
Verdict
Comment, not approve — not because the approach is wrong, but because ~500 lines of runner with heavy goal.md duplication will be expensive to extend. A comment-trim pass (and goalSessionId alignment) would make this an easy approve without changing what the goal proves.
goal.md coverage limits (eight in-process passes ≠ production "round eight") are stated honestly — good; don't let a green here become the epic headline claim.
Sent by Cursor Automation: Review For Best Approach
| userId: "goal-user", | ||
| // A fresh session per pass: a shared one would accumulate history and make | ||
| // pass 2 legitimately differ from pass 1, which is not the drift we're after. | ||
| sessionId: `lab136_${Math.random().toString(16).slice(2)}`, |
There was a problem hiding this comment.
Per-pass session isolation is load-bearing (comment above is right). For corpus consistency, consider goalSessionId(lab136-pass-${i}) in the S1/canary loops instead of Math.random() — not bare goalSessionId("lab136"), which would reuse one id for the whole process because RUN_STAMP is per-run.
| user: () => record("tail", renderTail(world, issueId)), | ||
| }); | ||
|
|
||
| const flow = defineFlow({ |
There was a problem hiding this comment.
The defineFlow / generator shell is identical on every assemble() except the grounding closure. Hoisting the static flow definition (and only varying grounding + issue id per call) would shave setup on 19 runAction invocations. Cosmetic for correctness; minor wall-clock win.
| * Model-free (`Model: n/a`), real assembly path, out of CI. See goal.md for the | ||
| * contract; this header covers only what a reader needs to follow the code. | ||
| * | ||
| * What makes this a goal check rather than a dressed-up unit test: |
There was a problem hiding this comment.
Simplify opportunity: this header + several section blocks (~localization, recipe constraint, anti-game) largely duplicate goal.md, which is already thorough. Trimming to "contract in goal.md" + one-line section pointers would cut ~80–120 lines without weakening the proof — the runnable file should grade, not re-teach decision 3.
| * epic. An unresolved slot is therefore not evidence of stability; it is evidence | ||
| * that the capture is blind, and blindness is never grounds for escalating. | ||
| */ | ||
| function localize(captures: Capture[]): Attribution { |
There was a problem hiding this comment.
This localization block is load-bearing (verdict log: false seam attribution before the ordering fix). I would not drop it — but localize + slotOutputsAgree + containsFunction could be one attributeDrift(captures) helper to shrink the runner without changing behavior.
|
|
||
| Spelled that way alone, the step is unsound. The constraint check has to run **inside** the attribution, first: an unresolved slot means the capture is blind, and blindness is not evidence of stability. Before that ordering was in place, a nested recipe formatter produced a confident *"report and stop — the drift is in the assembly seam"*. It now produces *"ours to fix here, NOT a framework finding."* Both outputs are recorded in the verdict log. | ||
|
|
||
| **Coverage this does not have** (stated rather than papered over): |
There was a problem hiding this comment.
These coverage limits are exactly the kind of honesty that keeps a green goal from overselling the epic. Worth keeping prominent if you trim run.mts comments — that's the news a reader shouldn't have to dig for.
jhoffner
left a comment
There was a problem hiding this comment.
Second-look + architecture pass
Verdict: appropriately scoped, nothing to change. No bloat/YAGNI findings clear the calibration gate, and there's no architecture-improvement occasion here — this is a single throwaway goal folder (3 files + changeset), touches no packages/*, adds no workspace member, and the PR body is explicit that no reusable machinery ships. A broader refactor suggestion would itself be the kind of unrequested expansion CLAUDE.md §2/§3 argues against, so none is offered.
I did verify the two things the PR body specifically asked a reviewer to check, against the actual framework source rather than taking the claims at face value:
-
localize()'s ordering (run.mts:434-451) is correct, and the two-stage resolution gap it depends on is real.resolveSlotValues(packages/core/src/blocks/generator.ts:685-698) only calls top-level slot functions;aggregateContextEntries(packages/core/src/blocks/context-aggregator.ts:1-42) is what resolves a function nested inside an object-form context entry, and it runs afterresolveSlotValuesin the pipeline (packages/core/src/blocks/internal/message-assembly.ts:172). So the capture point atrun.mts:365genuinely sits before a nesteddrift: () => …would get invoked — the hazard is mechanically real, not a dramatization, and running the constraint guard (unresolvedSlots) beforeslotOutputsAgreeis the only sound order. I didn't find a case where reordering or dropping the guard would still produce a correct verdict. -
The
generateStep-removal falsification checks out.packages/core/src/blocks/generator.ts:3055(generation = model.generateStep !== undefined) is exactly the branch that falsification run exercises — pullinggenerateStepoff the stub routes into legacygenerate(), which the stub makes throw (run.mts:307-312) specifically to keep this from passing green while measuring the wrong path.
Also spot-checked run.mts's 497 lines against the rest of the goals/ corpus (105–1794 lines, median ~190, several single-behavior goals already at 250–450) — size is in-family, not an outlier, and the decision to keep containsFunction/unresolvedSlots inline rather than extracting to goals/lib matches the documented policy in goals/README.md ("only when a third goal needs it") rather than being a judgment call worth second-guessing.
Nothing here blocks merge.
Generated by Claude Code
There was a problem hiding this comment.
Grok version of this simplify review (parallel pass to the other automation reviewer — treat as a second opinion, not a duplicate bot).
Verdict
Not approving. The mechanism for the real claim is excellent and should stay. The approach still carries more product-shape law and self-test surface than an eight-pass determinism proof needs. Cleanest merge would shrink that shell; if Decision 2 stays locked from the approved LAB-136 spec, say so explicitly and treat the rest of this as optional polish rather than blockers.
What went right (keep)
- Real path, not a helper:
generator→runAction→ captureoptions.messagesongenerateStep. Correctly avoids callingassembleMessagesdirectly. - Wrong-path guard: step-capable stub + legacy
generate()throws.mockGeneratorhas nogenerateStep— reusing it would fail green on the compatibility path. Custom stub is justified. - Same-run localization with unresolved-first ordering: the nested-formatter hazard against
resolveSlotValues/aggregateContextEntriesis real; blaming the seam from a blind capture would pause the epic under Decision 3. That ordering stays. - Counter canary (not clock) + empty
briefsAgree([]): right anti-games. - Home and reuse:
goals/context-supply/…layout is clean; correctly usesgoals/lib(loadFixture/runGoal/silentLogger); no package scaffolding, no cross-imports, empty changeset. Spec-PR concern aboutlabs/conductorwas fixed by landing here instead. - No meaningful perf issue for a hand-run model-free goal; no useful 3rd-party package to swap in.
Where the approach went too far
1. Decision 2 is still the main complexity dial (same finding as Grok on spec PR #1321)
Standing-half-survives-issue-change forces: role-scan precondition, hardcoded slice(0,1), two-issue assembly, fact leak suite, and a large share of goal.md / comments. That is recipe/product layout (what a phase brief should look like for caching), not “assembly is byte-identical across passes.”
A cleaner change for the epic’s reliability claim:
- N assemblies of one issue → same bytes
- On red → localize (unresolved-first → slot drift → seam)
- Counter canary + nested-formatter canary
Drop cross-issue standing/changing unless conductor layers that consume a standing half already exist. I’m uncertain whether product already re-ratified Decision 2 after the #1321 feedback — if yes, downgrade this to “scope bloat we accepted”; if not, this is the biggest simplify lever left.
2. Constraint / attribution covered three times
G1 always-on + localize’s first step + A3/A4 nested canary all police the same hazard. Keep localize-first and the nested canary (that’s what found the false seam). The always-on G1 when bytes already agree is optional duplication.
3. Prose weight
~500-line run.mts (~29% comment lines) and a ~9k goal.md retell the soundness story several times. Sibling inherits-parent-conversation is ~180 / ~3k. Prefer one contract in goal.md and short call-site comments; the evidence string at the end also re-narrates the thesis.
4. Small leftovers (not blockers)
streamStepis never exercised on this path — generateStep-only stub is enough unless the model contract requires both.- Canary loop at
PASSES=8is theatrical; N=2 already proves the comparator can go red. - Prefer
goalSessionId(...)fromgoals/liboverlab136_${Math.random()}(sibling goal already does).
Alternative shape (if rewriting were allowed)
assemble → Capture (generateStep only; generate throws)
8× same issue → briefsAgree
on red → localize(unresolved-first)
2× counter canary → !agree && recipe
2× nested fn → !agree && raw slots “stable” && localize === recipe
print one brief for the human gate
That still proves Decision 1 + Decision 3 + behaviours that catch vacuous greens — without smuggling standing-half product law into the proof.
Disposition
COMMENT, not APPROVE. Core seam proof is the right approach; I would not ask to rewrite it onto unit assembleMessages or mockGenerator. I also would not merge while calling Decision 2 “necessary for determinism” without an explicit product call — that is where complexity was created beyond the claim.
If Decision 2 is locked and intentional, a short reply to that effect is enough for me to treat findings 2–4 as optional cleanup and reconsider approval on a follow-up.
Sent by Cursor Automation: Grok Review For Best Approach
| // S3 — two DIFFERENT issues. The standing half must hold; the changing half | ||
| // must actually change and carry its own issue's facts. Without the last two, | ||
| // a recipe that silently assembled issue A twice would pass every other signal. | ||
| const a = eight[0]!; | ||
| const b = await assemble(freshWorld(), ISSUE_B, reviewGrounding); | ||
| // Gated on the precondition: with the boundary void, every conclusion below is | ||
| // drawn from a split that does not mean what it claims, and reporting them | ||
| // alongside the real cause just buries it. | ||
| if (preFail !== undefined) { | ||
| failures.push("the two-issue signals were not evaluated — the boundary precondition failed above"); | ||
| } else { | ||
| if (standingHalf(a) !== standingHalf(b)) { | ||
| failures.push("the standing half changed between two different issues — the split does not hold"); | ||
| } | ||
| if (changingHalf(a) === changingHalf(b)) { | ||
| failures.push( | ||
| "the changing half is IDENTICAL for two different issues — issue B was never really assembled, " + | ||
| "so the standing-half comparison above compared a brief with itself", | ||
| ); | ||
| } | ||
| for (const [id, capture, other] of [ | ||
| [ISSUE_A, a, ISSUE_B], | ||
| [ISSUE_B, b, ISSUE_A], | ||
| ] as const) { | ||
| const tail = changingHalf(capture); | ||
| const missing = factsOf(id).filter((f) => !tail.includes(f)); | ||
| if (missing.length > 0) { | ||
| failures.push(`${id}'s changing half is missing its own fixture's facts: ${missing.join(", ")}`); | ||
| } | ||
| const leaked = factsOf(other).filter((f) => tail.includes(f)); | ||
| if (leaked.length > 0) { | ||
| failures.push(`${id}'s changing half carries ${other}'s facts: ${leaked.join(", ")}`); | ||
| } | ||
| } |
There was a problem hiding this comment.
Grok / simplify — Decision 2 surface (main overshoot).
This whole S3 block (plus the standing/changing helpers above) is most of the extra machinery beyond “eight passes, same bytes.” It encodes that a review brief should split into a standing half that survives an issue change — recipe/product layout — not that the assembly seam is deterministic.
Same concern as the Grok review on spec PR #1321. A thinner goal keeps S1 + localize + canaries and drops cross-issue standing/changing until something actually consumes a standing-half cache key.
Uncertain: if Decision 2 was deliberately re-ratified after that spec feedback, this stays as accepted scope — but then please don’t narrate it as required for the determinism claim.
| function localize(captures: Capture[]): Attribution { | ||
| const unresolved = [...new Set(captures.flatMap(unresolvedSlots))]; | ||
| if (unresolved.length > 0) { | ||
| return { | ||
| verdict: "recipe", | ||
| detail: | ||
| `slot(s) [${unresolved.join(", ")}] returned an unresolved function, so their per-call values ` + | ||
| `never entered the capture. Attribution to the seam is unavailable — fix the recipe first`, | ||
| }; | ||
| } | ||
| const { agree, drifted } = slotOutputsAgree(captures); | ||
| return agree | ||
| ? { verdict: "seam", detail: "every slot returned the same value on every pass" } | ||
| : { verdict: "recipe", detail: `slot output varies between passes: ${drifted.join(", ")}` }; | ||
| } |
There was a problem hiding this comment.
Grok / simplify — keep the ordering; trim the surrounding belt.
Unresolved-first inside localize is load-bearing (false seam → epic pause). Keep it, and keep the nested-formatter canary that proves it.
What I’d drop or collapse: the always-on G1 check when bytes already agree — it retells the same constraint localize already enforces on red. Triple coverage of one hazard is how a 150-line proof became ~500.
| async *streamStep(options: GeneratorModelCallOptions) { | ||
| capture.messages = options.messages; | ||
| yield { | ||
| type: "finish" as const, | ||
| finishReason: "stop", | ||
| fullResult: { text: "", finishReason: "stop" }, | ||
| }; | ||
| }, |
There was a problem hiding this comment.
Grok / simplify (low). streamStep is never hit on this non-streaming path; generateStep + throwing generate already pin the mechanism. Unless the GeneratorModel contract requires both methods present, this is dead surface in a throwaway goal.
| // A1/A2 — anti-game. A variant whose grounding appends a PER-CALL COUNTER | ||
| // (never the clock: eight in-process assemblies can land in one tick). The | ||
| // same named comparator must return false, and localization — reading the slot | ||
| // outputs captured on these same runs — must attribute it to the recipe. | ||
| let calls = 0; | ||
| const countingGrounding = (w: World): unknown => ({ | ||
| ...(reviewGrounding(w) as Record<string, unknown>), | ||
| drift: `call ${(calls += 1)}`, | ||
| }); | ||
| const canary: Capture[] = []; | ||
| for (let i = 0; i < PASSES; i += 1) { | ||
| canary.push(await assemble(freshWorld(), ISSUE_A, countingGrounding)); | ||
| } |
There was a problem hiding this comment.
Grok / simplify (low / uncertain). Primary PASSES = 8 matches the claim language; fine. The canary does not need eight assemblies — a per-call counter makes briefsAgree false at N=2 (nested variant already uses 2). Shrinking only the canary loop would cut ~6 runActions without weakening the anti-game.
| **Signal:** | ||
| - **Eight passes, one issue.** Eight assemblies of the recipe, each over a fresh copy of the world, render to the same bytes under one named comparator (`briefsAgree`). | ||
| - **The precondition.** Exactly one leading `system` message and none after it — which is what makes "the leading system run *is* the standing half" exact for this recipe rather than a heuristic. If it fails, the boundary derivation is void and the standing-half signal is withdrawn, not reinterpreted. | ||
| - **Two different issues.** The standing half is byte-identical between issue A and issue B; the changing half **differs**; and each changing half carries its own fixture's facts (id, title, PR number) and **none** of the other's. |
There was a problem hiding this comment.
Grok / simplify — contract shape.
Signal bullets 1 + anti-game (counter, wrong-path stub) carry the epic claim. Bullet 3 (two different issues / standing half) is Decision 2 and is what forces the precondition + slice + fact-leak suite in run.mts.
If the simpler approach wins, this bullet becomes “out of scope / later phase,” not a machine signal — and goal.md can shed a large share of the standing/changing prose below.
|
Decision 2 is locked and intentional. Answering the question the Grok pass flagged as uncertain, since only the gate-holder can. It was challenged at the spec gate by two reviewers on #1321, deliberately kept, and recorded in §12 as settled specifically so it wouldn't be reopened blind. The reasoning, briefly, because it's the part that wasn't written down clearly enough the first time: Eight passes over one issue pass by construction. Nothing moved between them, so they hold whatever the recipe put wherever. They rule out per-call nondeterminism and nothing else. The cross-issue assertion is the only signal in this goal that can distinguish "the assembly is stable" from "the whole brief happens to be constant" — which means dropping it doesn't trim scope, it removes the part that can fail. On whether it's product law smuggled into a determinism proof: that was the original objection and it was fair against the spec as written, because the spec asserted the dependency instead of arguing it. The epic's L4 claim is the standing/changing split — a stable prefix identical every run by construction plus a small changing tail. The determinism claim and the split are the same claim stated at two altitudes, and only the higher one is falsifiable. That argument was added in round 2 rather than the decision being re-asserted. What it is not is a rule binding future recipes. Round 2 also cut that framing — §6 now records it as an observation about this recipe, not an invariant over L4, and §12 leaves open whether a later phase legitimately needs an issue-specific standing instruction. So please treat findings 2–4 as optional cleanup, per your own disposition. Two of them are being folded anyway and are good catches: the always-on constraint check duplicating what One correction worth making, on size. And agreed on the thing you closed with: a green here is narrower than the epic's headline. Eight in-process passes seconds apart say nothing about restarts, machines, or locales. That's stated in Generated by Claude Code |
…36 review round 1) The check could report PASS while the brief lost most of its content. It graded three facts — issue id, title, PR number — so a deterministic drop of the issue body, the diff, the verification statement, the PR metrics, the project grounding, the standards or the objective left every automated signal green: eight passes still agreed, the standing/changing split still held, the canary still fired. That made it a check on the SHAPE being stable, not the content. The expected set is now derived by walking the fixture world rather than hand-listed, so it cannot drift from what the brief promises: 15 contributions graded instead of 3. Verified by dropping the diff and verification statement (RED, named by fixture path) and separately the standards and objective summary (RED) — with every other signal staying green, which was the point. Also from round 1: - Substring-overlap guard at setup. The leak check uses `includes`, so a replacement world holding `ISSUE-1` and `ISSUE-10` would have reported a false leak. Rejected at the boundary instead, keeping the held-out claim honest. Verified by renaming a fixture issue. - Per-pass `goalSessionId` slugs replace `Math.random()`. The per-pass part is load-bearing, not cosmetic: the stamp is per-process, so a bare slug would hand all eight passes one session and accumulate history between them. - Canary reduced from 8 assemblies to 2 — a per-call counter differs on the second. The primary loop stays at eight because that is the claim's language. - The always-on recipe-constraint check is removed. The hazard is policed where it changes an outcome: `localize` consults the constraint before it may say "seam", and the nested-formatter canary proves that guard fires. - `streamStep` removed from the stub. Checked rather than assumed: both step methods are optional on `GeneratorModel` (only `modelId` and `generate` are required), and this recipe is tool-free and identity-less, so `canStream` is false and the streaming branch is unreachable. Adding tools or `itemVisibility` later cannot silently reroute to a legacy path either, since `canStream` also requires `stream` or `streamStep` to exist. - Comment trim where `run.mts` restated `goal.md`; the coverage-limits block stays in the header, where a reader meets it before the code. Not folded: consolidating `localize` / `slotOutputsAgree` / `containsFunction` into one helper, and hoisting the flow shell. Local structure is out of scope on this PR's review contract, and the adjacent pull was already resisted deliberately under the spec's §4 practice 2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e5aa167c47
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // a legacy path either: `canStream` also requires `stream` or `streamStep` to | ||
| // exist, and neither does. | ||
| async generateStep(options: GeneratorModelCallOptions) { | ||
| capture.messages = options.messages; |
There was a problem hiding this comment.
Record and validate every model invocation
If a runtime regression or retry invokes this tool-free generator more than once, each call overwrites capture.messages, discarding every earlier model handoff. An earlier malformed or pass-dependent brief can therefore be ignored while briefsAgree and the content checks grade only the final stable call, allowing the goal to pass even though the model received non-identical briefs; accumulate all invocations or assert that exactly one occurred.
Useful? React with 👍 / 👎.
| const carries = (haystack: string, leaf: string): boolean => | ||
| haystack.includes(leaf) || haystack.includes(escapeXml(leaf)); |
There was a problem hiding this comment.
Bind graded values to their rendered fields
Unlike the earlier identity-only leak path, the newly added S3 all-leaf grader checks only whether each scalar occurs somewhere in the half, not whether it is rendered under the correct field. With the committed fixture, swapping linesAdded and linesRemoved in renderTail leaves every leaf present and every other signal green while the brief reports incorrect metrics; a held-out fixture with overlapping values can similarly hide an omitted field. Grade field-aware rendered fragments rather than unscoped substrings.
Useful? React with 👍 / 👎.
… review round 2) Two more instances of the failure this goal keeps finding: a check that cannot see what it claims to measure, reporting fine. 1. The capture could lose the brief it was measuring. Each model invocation overwrote `capture.messages`, so a second call — from a runtime regression or a retry — silently discarded the first handoff and left `briefsAgree` and the content checks grading only the final stable call. A malformed or pass-dependent first brief was invisible. Asserted at exactly one invocation rather than accumulating. This recipe is tool-free and terminal at step 0, so a second call is a regression by definition: the assertion is both the guard and its own canary, and accumulating would grade a case the design says cannot happen. 2. The 15-contribution grader checked presence, not placement. It asked whether each scalar appeared somewhere in its half, so swapping `linesAdded` and `linesRemoved` left every leaf present and every signal green while the brief reported the wrong change size. A held-out world with two equal values hid an omitted field the same way. The ambiguous fields are now graded on their labelled fragment — `+47`, `−12`, `3 files`, `#101` — built from the rendered form rather than the bare value. The labels mirror `renderTail` and sit beside it. The graded SET is still derived by walking the fixture, so a field added to the world and never rendered still fails; only the expectation for ambiguous fields is tightened. Both verified by breaking them: swapping the two metrics goes red naming the expected fragments (it was fully green before), and giving the recipe a tool with a tool call scripted on step 0 produces two invocations and goes red. No new abstraction: +25 lines of code, no grading framework, no shared helper. Comment share unchanged at 22%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cd07cdcd7f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| }); | ||
|
|
||
| const flow = defineFlow({ | ||
| kind: "lab-136-review-brief", |
There was a problem hiding this comment.
Remove the tracking ID from runtime identifiers
Every execution creates a flow kind named after the Linear ticket, and the same tracking label also drives the session slug at line 169. This violates BP-006 in docs/contributing/best-practices/process.md, which explicitly prohibits Linear references as identifiers in code or test logic because they couple runtime artifacts to transient planning state; use stable, behavior-based names such as review-brief-goal instead.
Useful? React with 👍 / 👎.
| const expectedFragment = (leaf: Leaf): string => { | ||
| const placed = PLACED.find(([suffix]) => leaf.path.endsWith(suffix)); | ||
| return placed === undefined ? leaf.text : placed[1](leaf.text); |
There was a problem hiding this comment.
Bind every contribution to its rendered field
Fresh evidence beyond the prior placement comment is that the new PLACED table fixes only the four PR scalars; this fallback still grades every other leaf by unscoped substring containment across the entire half. For example, swapping reviewGrounding's project and standards assignments leaves both fixture strings present, so every signal remains green while the model receives each value under the wrong XML field, contradicting the goal's “correctly placed” guarantee. Bind the remaining structured leaves to their tags or labeled regions as well.
Useful? React with 👍 / 👎.
…ds from runtime identifiers (LAB-136) Placement grading covered four named fields; every other contribution still graded by unscoped substring across the whole half. Swapping `project` and `standards` in the recipe's grounding left both strings present and every signal green while the model read each under the other's heading — the "correctly placed" guarantee did not hold for the fields the table didn't name. Each contribution is now graded inside its own tag's content. The tags are read off the recipe's own context object — its keys ARE the tags the framework renders — so this is derived from the recipe rather than declared beside it, and a recipe that grows a fourth grounding key is covered without anyone remembering to update the check. The labelled fragments were KEPT rather than replaced, and that is a tested decision rather than a cautious one. Region scoping cannot see a transposition INSIDE a region: all four PR scalars render into the same untagged tail, so `linesAdded` and `linesRemoved` swap without leaving it. Emptying the table and relying on region scoping alone was run, and the transposed brief goes GREEN. The two mechanisms cover two different hazards and neither subsumes the other; the pair is the tested minimum. Both directions are recorded in goal.md's falsification table, including the green, so the redundancy is not mistaken for belt-and-braces later. Also, BP-006: the flow kind and session slug were named after the Linear ticket, coupling runtime artifacts to transient planning state. Renamed to `review-brief-goal` and `review-brief-pass-N`; the per-pass structure is unchanged, so the session-isolation property is unaffected. The only remaining mention is the file header's provenance line, which matches the sibling goal (`FIX-920`) and goal.md's `Issue:` field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5dfc63933f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ] as const) { | ||
| // The changing half is one untagged region, so placement within it is what | ||
| // the labelled fragments in `PLACED` carry. | ||
| const tailMissing = missingFrom(changingContributions(id), () => changingText(capture)); |
There was a problem hiding this comment.
Bind changing-half contributions to their rendered fields
Fresh evidence after the earlier placement feedback is that the new scoped regions cover only the standing half: every changing contribution is still searched across the entire tail here. If renderTail swaps issue.body with pullRequest.verification, both strings remain present, so S3 and every other signal stay green while the brief assigns the content to the wrong fields; bind these contributions to their labels or positional regions instead.
Useful? React with 👍 / 👎.
Region scoping landed on the standing half only; every changing contribution was still searched across the whole tail. Swapping `issue.body` with the verification statement left both strings present and every signal green while the brief assigned each to the wrong field — the same hole, in the half the previous commit did not reach. The tail gets its OWN mechanism rather than the standing half's stretched to fit, because the two are shaped differently and pretending otherwise would be worse than having two: the standing half is tag-structured, so its regions come from the recipe's context keys (which ARE the tags the framework renders); the tail is prose with labels and has no tags, so it is sliced at the labels `renderTail` writes — `Issue `, `PR #`, `Verification: ` — kept beside the renderer. A label that goes missing voids the tail's scoping loudly instead of silently widening the search back to the whole half, which is how this hole appeared. Verified both ways: the body/verification swap goes red naming both fields, and the metric transposition still goes red afterwards, so region scoping and the labelled fragments compose rather than one masking the other. Known limit, now recorded in goal.md rather than left implicit: two UNLABELLED fields transposed inside a single region are not detected. Regions are the finest scope the render offers, and within one region only the labelled fragments distinguish fields — `issue.id` has `Issue `, the line counts have `+`/`−`, so those are covered. Closing it further would mean labelling the brief for the test's benefit, which is the tail wagging the dog. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b14dfc3556
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| function tailRegion(text: string, index: number): string { | ||
| const starts = TAIL_ANCHORS.map((a) => text.indexOf(a)); |
There was a problem hiding this comment.
Reject fixture text that collides with tail anchors
When a valid replacement fixture contains a line beginning with PR # in an issue body, or Verification: in the body or diff, these indexOf calls select the embedded text instead of the labels emitted by renderTail. The correct recipe then fails contribution grading because its regions are sliced at the wrong offsets; for example, appending \nPR #999 is mentioned here to the first issue body makes this goal report issue.body missing. Either reject anchor collisions during setup, as is already done for identifying facts, or derive unambiguous boundaries.
Useful? React with 👍 / 👎.
…el (LAB-136) The setup guard was half-applied. It already refuses to run on a fixture whose two issues have overlapping identifying facts, because that would break the leak check's own comparison. Anchor collision is the same class — fixture content silently breaking this check's slicing — and was not covered. `tailRegion` and `standingRegion` locate regions with `indexOf`, so a fixture containing one of the labels sends the search to the embedded occurrence, cuts the regions at the wrong offsets, and reports a contribution missing from a perfectly correct recipe. Now rejected at setup in the same voice as the existing guard, naming which label collided and where. The current fixture is clean — verified independently, with the instrument sanity-checked against a string known to be present — so the standing PASS and the verdict log are unaffected. This closes the door, it does not withdraw a result. The guarded list is deliberately asymmetric, and the asymmetry is now backed by a run rather than a source read: grounding and objective values reach the standing half through `renderTaggedContext`, which escapes `<` and `>`, so a value containing `<project>` cannot collide with a tag. Confirmed by putting exactly that into a grounding value and watching the goal pass rather than be rejected — the guard does not over-reject a valid world. The prompt and the whole tail are written verbatim, so those are what is checked. Proved able to fire in both branches, then reverted: embedding "PR #" and "Verification: " in an issue body is rejected naming both, and embedding "<project>" in the unescaped prompt is rejected naming that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
The run behind this row was made on a clean tree at e380252, the commit that implements the anchor-collision guard — the row cites that SHA rather than the branch base, so a later reader can check out exactly what was run. Adds the three falsification rows behind the guard, including the one that PASSES: a grounding value containing `<project>` is XML-escaped and cannot collide, so it is correctly not rejected. That row is the evidence for the guard's asymmetric list, and it is recorded as a pass on purpose — a guard that over-rejects valid worlds would break the held-out property this goal depends on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5367e513c1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| context: () => record("grounding", grounding(world)), | ||
| user: () => record("tail", renderTail(world, issueId)), |
There was a problem hiding this comment.
Move structured brief data into context slots
This recipe puts the complete issue, PR, diff, and verification dump in user while returning all grounding from one context function. BP-017 in docs/contributing/best-practices/generators.md requires structured data in context: { tagName: fn } and reserves user for a short trailing instruction; because this goal becomes the reusable regression record for brief assembly, it currently validates and documents a prompt shape the repository explicitly prohibits. Split these contributions into tagged context slots and keep user short, or surface the grounding conflict instead of canonizing this exception.
AGENTS.md reference: AGENTS.md:L11-L15
Useful? React with 👍 / 👎.
| const carries = (haystack: string, fragment: string): boolean => | ||
| haystack.includes(fragment) || haystack.includes(escapeXml(fragment)); |
There was a problem hiding this comment.
Distinguish contributions instead of matching substrings
Fresh evidence after the earlier overlap fix is that only cross-issue identity collisions are guarded; arbitrary leaves in the same region can still overlap. For example, with a valid replacement fixture whose body repeats its title, removing the title from renderTail still passes because this includes() finds the title inside the body, so the goal reports all contributions present despite the regression. Reject all same-region leaf overlaps during setup or grade structured, field-specific fragments.
Useful? React with 👍 / 👎.
| TAIL_ANCHORS.map((a) => a.replace(/^\n/, "")) | ||
| .filter((a) => l.text.includes(a)) |
There was a problem hiding this comment.
Preserve newlines when checking tail-anchor collisions
The new setup guard strips the leading newline from "\nPR #" and "\nVerification: ", so a valid fixture that merely mentions PR #123 inline in a title or body is rejected even though tailRegion() searches for the newline-prefixed anchor and that inline text cannot alter its boundary. This makes held-out fixture substitution fail for realistic content; test collisions against the exact anchors rather than their stripped substrings.
Useful? React with 👍 / 👎.
…136) The guard added in e380252 stripped the leading newline off each anchor before testing, but `tailRegion` searches for the newline-prefixed form. A fixture that merely mentioned "PR #123" mid-sentence was rejected as invalid even though that text cannot move a region boundary. Reproduced before fixing. Tested against the RENDERED tail rather than per-leaf, because neither per-leaf test is exact and the collision is a property of the joined text: - matching the bare `PR #` rejects an inline mention that cannot move a boundary (the over-rejection above); - matching the newline-prefixed anchor against a leaf misses a body that STARTS with it and picks the newline up from the join. Verified that second case rather than reasoning about it: a body beginning "PR #999 tracked the earlier attempt." does not contain "\nPR #" as a leaf, yet produces a genuine duplicate in the rendered tail. An anchor occurring exactly once is the precise condition, and `renderTail` is a pure function of the fixture, so setup can just ask it. Verified in both directions: the inline mention now passes, and both a body line starting with an anchor and one containing it are rejected. The asymmetry between escaped and verbatim renders is unchanged and still holds. Also adds a one-line note above the recipe recording why BP-017 does not apply: it governs production recipes, where structured material belongs in `context`; this recipe must populate all three slots and cover both the escaped and verbatim paths, so routing the tail through `context` would make the check test less. Written down so the next reviewer does not re-raise it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
…ing (LAB-136) The run behind this row was made on a clean tree at 5a44782, the commit that fixes the anchor guard's over-rejection — the row cites that SHA, not the branch base. Also records the limit being tracked on FIX-1183 rather than folded: grading is substring matching over rendered text, and every guard in this check exists to protect that approach against arbitrary substituted content. The fourth guard was introduced by the third one's fix, which is the signal that the approach is the hole rather than any individual gap. Written into the coverage list so a reader meets it as a known ceiling, not as an oversight. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2262c9e452
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| // S2 — the precondition the standing/changing derivation depends on. | ||
| const preFail = preconditionFailure(eight[0]!.messages); |
There was a problem hiding this comment.
Check the held-out brief's message boundary
When an assembly regression emits a later system message only for issue B, this checks only issue A's message shape; S3/S4 then slice B after index 1, treating the extra system message as changing content while contribution, standing-half, and leak checks can still pass. The goal therefore reports PASS despite its explicit precondition of one leading system message and none afterward being false for one of the two briefs; validate b.messages before evaluating the two-issue signals.
Useful? React with 👍 / 👎.
| // and `renderTail` is a pure function of the fixture, so setup can just ask. | ||
| ...WORLD.issues.flatMap((issue) => { | ||
| const rendered = renderTail(WORLD, issue.id); | ||
| return TAIL_ANCHORS.filter((a) => rendered.indexOf(a) !== rendered.lastIndexOf(a)).map( |
There was a problem hiding this comment.
Allow harmless later Issue anchors
Fresh evidence after the earlier newline-anchor fix is that the new exact-once guard also applies to "Issue ". Because renderTail() always emits that anchor at offset 0 and tailRegion() uses indexOf, a second occurrence in a valid title or body—for example, “The previous Issue was closed”—cannot move any boundary, yet this setup check rejects the held-out fixture before exercising the correct recipe. This breaks the goal's promised fixture swappability; reject only occurrences capable of preceding or moving the canonical boundary.
Useful? React with 👍 / 👎.
Removes the guard added in e380252 and corrected in 5a44782, in full. This backs out a decision, not a mistake in carrying it out. The guard rejected a fixture whose content contained a region label, on the argument that the setup validation was otherwise half-applied. Two attempts over-rejected valid worlds in opposite directions: - matching a bare `PR #` refused a body that merely mentions "PR #123" mid-sentence, which cannot move a boundary; - requiring each anchor to occur exactly once refused a title or body containing the word "Issue", even though `renderTail` always emits that anchor at offset 0 and the slicing takes the first match, so no boundary can move there either. Both breaks damaged the fixture swappability the guard existed to protect, and it was never protecting anything real: the fixture in the repo has no collisions, so every recorded verdict was sound throughout. When completing a mechanism costs more than the mechanism is worth, the answer is to back it out rather than keep paying. Verified the revert restores what the guard broke: a world whose body mentions `PR #123` and whose text contains "Issue" now passes, and the goal is green on the real fixture. The limit itself is not lost — it is recorded in goal.md's coverage list alongside the two others of the same family (bare-substring matching, and the message-boundary precondition being asserted for issue A only). All three are one problem: grading by substring over rendered text cannot reliably localize a contribution, and each guard bolted on is a patch on that. Tracked as its own issue; the fix is structured, field-specific comparison. Stale falsification rows for the removed guard are deleted rather than left describing behaviour that no longer exists, and replaced by the one that now matters: those worlds must PASS. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
The run behind this row was made on a clean tree at 172d472, the commit that backs out the anchor-collision guard — the row cites that SHA, not the branch base. The two verdict rows for the guard's own commits stay: they record runs that genuinely happened, and a verdict log is a history rather than a current-state summary. This row says plainly that what they describe has since been removed, so a later reader is not misled into thinking the guard is still there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3d9cf7e042
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const first = JSON.stringify(captures[0]?.slots[name]); | ||
| return captures.some((c) => JSON.stringify(c.slots[name]) !== first); |
There was a problem hiding this comment.
Treat missing slot invocations as seam failures
If the framework skips a top-level slot callback on one pass, that capture has no corresponding slots property; this comparison treats the missing value as ordinary slot-output drift, so localize() reports recipe even though assembly caused the divergence. That sends the goal down its “fix here, do not file a framework finding” path for exactly the seam regression it is meant to localize; validate that every expected slot was invoked in every capture and classify an absent invocation as seam drift or void attribution before comparing values.
Useful? React with 👍 / 👎.
| if (messages.slice(lead).some((m) => roleOf(m) === "system")) { | ||
| return `a system message appears after the leading run (roles: ${messages.map(roleOf).join(", ")})`; |
There was a problem hiding this comment.
Require the changing message to retain the user role
If message assembly regresses and wraps the user tail as an assistant or tool message, this precondition still passes because it rejects only later system messages. The contribution checks discard roles through textOf(), and stable role corruption also passes the equality checks, so the goal can report PASS while the model interprets the issue, diff, and verification as prior conversation rather than the request; for this history-free recipe, assert the expected system, user role sequence.
Useful? React with 👍 / 👎.


The problem
We tell people that a phase of work gets exactly the context it needs, assembled the same way on every pass. Nothing tested that. If the assembly quietly varies between passes — a timestamp here, a differently-ordered list there — then round eight does not behave like round one, and the reliability the whole system is being built for is not there.
Why now. The two layers that would make the assembly observable are deliberately last in the epic's build order. On that plan the claim stays untested until everything sits on top of it. Proving it here costs one goal check. Discovering it after the substrate is built costs the substrate.
What this does
Hand-builds a small world as literal fixtures — two issues, their pull requests, the diffs, the objective they sit under — with no database and no event feed. Builds one phase's context recipe against it (review), then checks two things: that a person reading the resulting brief agrees it is what a reviewer needs, and that the machine gets the same bytes eight times running.
The brief is shaped as a standing half and a small changing half. Only the bottom moves between runs:
Result: it holds. Eight passes were byte-identical (2153 bytes; standing half 1258). Assembling a second, different issue left the standing half untouched. No framework finding — the assembly was deterministic across every pass.
Nothing reusable ships: no assembly machinery, no recipe format, no store, no second phase. Three files in one goal folder, zero workspace members, zero config edits, no changes to any package. Throwaway was an acceptable outcome; the written finding is the deliverable.
The human read went red once, and that is in the log. The brief hands a reviewer a standard that asks them to check "every claim has a re-runnable evidence path", and the first version gave them nothing to check it against — no tests, no statement of what the author ran. Fixed by adding that to the world. Recorded because a gate whose red state never fires isn't a gate.
What's asked of you
Nothing here is yours to decide — this implements the approved direction in the LAB-136 spec, and its §12 records what was settled during review so it isn't reopened.
One thing is worth your awareness rather than your decision: a green here is narrower than the epic's headline claim. Eight passes seconds apart in one process rule out per-call nondeterminism. They say nothing about drift across restarts, machines, or locales. If the epic later needs cross-machine identity, that is its own issue. Please don't let this get written up as the headline claim having been met.
Parts worth reviewing closely
Every defect found on this issue has been the same shape, and it is worth stating once because it is what the check is now built to resist: an instrument that cannot see what it claims to measure, reporting fine. Five instances, each a few lines to fix — the boundary derivation, localization's ordering, contribution grading, the capture overwrite, and field placement.
+47,−12,3 files,#101) rather than the digits, because presence alone can't telllinesAddedfromlinesRemoved. Worth checking the derivation really covers what the brief claims, and thatcarries()'s XML-escape tolerance isn't hiding a mismatch.localize(). The one mechanism the spec flagged as having had no adversarial pass. Building it surfaced a real defect in §8 step 7:resolveSlotValuesresolves only top-level slot functions, so a function nested inside object-form context resolves later and never enters the capture. That recipe drifts the brief while the capture reports the slot perfectly stable — and the step as specified then emits a confident "the drift is in the assembly seam", a false framework finding that pauses the epic under decision 3. The fix is ordering: the constraint check runs inside the attribution, because an unresolved slot means the capture is blind, and blindness is not evidence of stability.localize, and via the nested-formatter canary), andgoal.mdsays why, because the spec PR never merges and someone extending this recipe needs to know.goal.md; each was performed by breaking the check on purpose and confirming red.Links
Spec: LAB-136 Linear document · spec PR #1321 (closes unmerged) · Issue LAB-136 · Epic LAB-68 · This PR merges.
How to review this — altitude, what's in scope, what's deliberately unsettled
How to review this. This implements an approved spec — the approach and the numbered Decisions in its §6 are already signed off by a human, so please review the code against that direction, not the direction itself. The spec lives on the issue's Linear document (the durable copy) and on the spec PR (#1321), which keeps its review history. Don't expect
spec/LAB-136.mdto be in this diff — the spec PR closes unmerged at approval and the spec never lands onmain(BP-037).Most valuable here: correctness on the second path (the legacy shape, the null boundary, the concurrent case, the cancel path), anything deriving an auth or routing decision from caller-controllable input, and a behaviour the tests assert around rather than on.
Already settled upstream: the approach and every §6 Decision (human-approved — if one is wrong, say so as a spec finding and it gets folded back; don't re-argue it inline), and the conventions in
docs/contributing/best-practices.md.Out of scope on this contract: local structure — which helper exists, how a function is decomposed, where a shell is hoisted. Two such suggestions were declined for that reason, not because they were wrong in the abstract.
Engineering calls — decided along the way, no product sign-off needed
goals/README.mddescribes, because the human read that would catch a content drop is a one-time verdict-log rowgoal:allnever re-runs.seamverdict and a separate constraint failure — and the seam bullet came first, which is the one that triggers an escalation. Cost: a false framework finding pauses the epic.streamStepomitted from the stub. Checked, not assumed: both step methods are optional onGeneratorModel(onlymodelIdandgenerateare required) andcanStreamis false for a tool-free, identity-less recipe, so the streaming branch is unreachable. A later recipe that adds tools oritemVisibilitystill cannot silently reroute to a legacy path, becausecanStreamalso requiresstreamorstreamStepto exist.goalSessionId's stamp is per-process, so a bare slug hands all eight passes one session; the accumulated history would make pass 2 legitimately differ from pass 1 — real drift, but not the drift under test.briefsAgree([])returns false.[].every()is true, and a vacuous green is the exact failure this goal exists to prevent.containsFunction/unresolvedSlotsis a check a second phase's recipe would want too, and the instinct was to put it ingoals/lib. I kept it inline inrun.mts.goals/README.mdsays to reach for a shared helper only when a third goal needs it, and this is the layer the issue exists in order not to build.Verification
Falsification runs — each performed by breaking the check on purpose, confirming red, then reverting. A green from a check nobody has seen fail is decoration.
generateStepremoved from the stubgenerate()and the guard threwgrounding.standards,objective.summaryISSUE-1andISSUE-10linesAddedandlinesRemovedswapped inrenderTail+47and−12as expected fragments. Fully green before labelled-fragment grading, since both digits were still in the tailCoverage this does not have, stated rather than papered over: the
seamverdict is reachable but has never fired on a real drift (producing one needs the framework itself made nondeterministic, i.e. editingpackages/*, a non-goal); a clock read inside one tick is not reliably detected; fixture mutation is not detected either way; cross-machine/process/locale variance is out of scope.Files
goals/context-supply/assembles-the-same-brief-every-pass/goal.md— the contract, the recipe constraint and why it's load-bearing, the verdict log (the human-read FAIL, the PASS, and the two review-round re-runs), and the falsification table.goals/context-supply/assembles-the-same-brief-every-pass/run.mts— the recipe, the recording step-capable stub, one named comparator used by every check, the boundary precondition, the derived placement-aware contribution assertions, the constraint guard, and localization.goals/context-supply/assembles-the-same-brief-every-pass/fixtures/world.json— the hand-built world. Invented identifiers only (ISSUE-1,#101), never real tracker ids (BP-006)..changeset/lab-136-brief-byte-identical.md— empty frontmatter (no package bumps), with the short internal-only note the repo's other hand-named internal changesets carry.