Repository navigation
Conversation
…routes Model-free probe of the persistence + route round-trip, on the real emitter, real SQLite and the shipped routes. Three premises the spec rests on: 1. a persisted tool_output carries toolCall.name and parseable toolCall.arguments, so a file path is a FIELD not a substring — CONFIRMED 2. the ordering key is itemIndex; `seq` does not exist on a persisted item, and the values carry a duplicate and a gap — CONFIRMED (FIX-1183's shape, reproduced deliberately) 3. the collection-state route returns clientData (not state), addresses a collection by its accessor key, pages at 50 with nextCursor, and 403s without client.state.read — CONFIRMED Scoped to persistence and readback: it measures nothing about what the harness emits, which LAB-134's POC measured and this takes as given. Throwaway. Never merges. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
The epic's Proof and its kill line. A third goal check in the same library: dispatch a real coding run, hand a reader nothing but the HTTP routes, have it DERIVE a structured account of the run, and only then compare that account field by field against the held-out expectation. Deriving before comparing is the whole delta from the two checks we have, which already hold the answer and search the record for it — a record that kept a fraction of a run passes that. No judge, no score, no eval framework: the objective cut those and the instrument is the goal-check library that already exists. Three decisions: derive-then-compare; the proof rests on what the run DID while the plan half is graded separately and may come back unmeasured (half of measured real runs keep no plan); and the run is a seeded throwaway job rather than the live backlog issue the ticket names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|
|
Answered — reviewers need not re-answer. The "where I'm unsure" line above (assertion 2: the item stream and the file-op collection must name the same mutations, both ways) has a settled answer from the coordinator. It is not folded into the spec yet, on purpose — it will be applied together with round 1 so the doc gets one pass, not two. The answer. Assertion 2 as written is over-tight and would go red for a non-defect, because LAB-134's decision 3 guarantees the recorder may skip: watching the work must never break the work, so a path it can't key is skipped rather than thrown. A legitimate skip puts a mutation in the item stream and not in the collection. The fix keeps the assertion and makes it precise: disagreement between the two surfaces fails unless the collection carries a corresponding recorded gap for that mutation. Then it says what it was meant to say — the graph lost something without admitting it — rather than the two surfaces differ. And a finding that has to go to LAB-134 before that fix is buildable. The escape hatch needs a gap signal a reader can see, and LAB-134's currently cannot be seen after the run.
So after the run a skipped mutation is indistinguishable from one that never happened. That is the exact blindness LAB-134's own §10 grades as a failure ("a recorder that silently skips everything looks exactly like a run that did nothing"), and it is this epic's lesson 1 again: the rule is stated, and the named mechanism cannot deliver it. What LAB-134 needs: a durable gap signal the reader can correlate to a path — most naturally a row in the file-op collection carrying a Routing that to LAB-134 rather than weakening this assertion, per the coordinator's instruction. Generated by Claude Code |
…ce under decision 2
Three changes, all measured rather than reasoned.
1. Assertion 2 sharpens from "any disagreement between the item stream and the
file-op collection fails" to "...unless a gap row accounts for it". The
recorder is ALLOWED to skip (watching the work must never break the work);
what it may not do is skip silently. LAB-134 now ships `observed-gaps`,
keyed by run id + ordinal with the raw path on the row — a collection this
spec's finding caused to exist.
2. A per-path unmeasured arm for shell-driven work. `allowedTools` is a
permission allowlist, not an availability filter — a measured run used
`Bash` while absent from the list — and shell edits are invisible to the
recorder by design. Missing path + no shell call = FAIL; missing path +
shell call = unmeasured, named. All paths unmeasured = the run proved
nothing.
3. Decision 2 gains its evidence: zero plan-tool invocations across eight
consecutive real runs on the in-process SDK path this check uses. Splitting
the proof so the kill line rides on what the run DID is load-bearing, not
cautious — under a single verdict this issue could not ship.
Also corrects a claim I made on the PR: a status item CAN persist
(`runAction.ts` filters on `item.transient !== true`, and `{transient:false}`
opts out). The conclusion held for a better reason — `emit.status` drops a
message identical to the slot's, so repeated skips collapse to one item.
Filed the items.md self-contradiction as a follow-up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
|
Round 1 folded ( I was wrong that a
|
|
Spec approved — proceeding to implementation. Round 1 folded; no reviewer engaged in the ~45 minutes the PR was open, so the review opportunity elapsed rather than being spent. Three things settled here are worth naming, because each began as a claim someone held confidently and ended up corrected by the code: Assertion 2 now says what it was reaching for. Disagreement between the item stream and the file-op collection fails unless a gap row accounts for it. My original version rested on a durable gap signal that did not exist — this spec's author found that, which is why LAB-134 now ships an A missing path is two different things, not one. A status note could never have carried the gap record. Not because it isn't persisted — it can be, and the claim that it can't was wrong — but because Two calls sit in §6 for overrule: the account is derived from state before it is compared, never searched for; and the proof rests on what the run did, with what it planned graded separately and allowed to come back unmeasured. The second now carries its evidence — zero plan-tool invocations across eight consecutive runs — rather than resting on judgment. Known and accepted: nothing exercises LAB-134's plan recorder end to end, because the plan tools never fire on the driver we use. This check will report that unmeasured on essentially every run, which is the honest answer and cannot fail the kill line. Filed separately. This PR does not merge — Linear holds the durable copy. Implementation waits on #1332's in-flight fixes, since this spec's assertions read exactly the fields being corrected there. Generated by Claude Code |
|
Closing this spec PR unmerged — the implementation is open at #1334, and the goal it specified passes. The proof this issue owes: a real coding run is dispatched, a reader is handed nothing but the shipped HTTP routes, and the account it derives from state alone stands up field by field against the job the run was given. PASS on nine consecutive real runs. The plan half reports UNMEASURED on every one, exactly as Decision 2 anticipated — the run writes its to-do list as prose in its own messages and invokes no plan tool at all. That decision, splitting the proof so the kill line rides on what a run did rather than what it planned, is the reason this issue could ship instead of stalling on a surface the harness makes optional. Two things this spec got right that are worth recording, because both were live questions when it was written:
Two review rounds found six ways the check could have passed on a wrong account, all fixed and each regressed on purpose. Four of its guards exist only because a mutation stayed green — including one that reported itself proven while the branch it named had been deleted. Spec PRs never merge; Linear holds the durable copy. Generated by Claude Code |
The problem
Three layers of this epic now record what a coding agent did — its messages, the files it touched, the plan it kept. Nobody has checked whether what they record is enough.
The two checks we have can't tell us. Both already know the file the run was told to write, and search the stored record for that name. A record that kept a fraction of the run passes that, because the fraction it kept is the part we asked about. Nobody has put the question the other way round: hand a reader only the stored state, ask it what the run did, and see whether the answer holds up.
Why now. This is the epic's kill line. If state alone can't say what a run did, the layer above has nothing to read and the one above that has nothing to file, and the two layers already built are worth throwing away.
What this does
One more entry in the goal-check library we already run, on the same real host, database and coding run the previous two layers proved themselves on.
It dispatches a real coding job, hands a reader nothing but the system's own web routes, and has it produce an account of the run — which files were touched and how each turned out, in what order, what the run said, and what it thought its job was (or, when it kept no plan, that it kept none, with the evidence). Only then does it compare that account against the held-out expectation, field by field.
Deriving the answer before comparing is the whole difference from the checks we have.
No judge, no score, no eval framework — the objective cut those and the cut stands. It never grades whether the agent did a good job, and never asserts how a run was settled. And it may come back FAIL, which is what makes it a proof rather than a formality.
What's asked of you
Three calls, numbered in the spec as §6 Decisions. No open questions.
The account is derived from state first, then compared — never searched for
The proof rests on what the run did; what it thought its job was is graded separately and may come back unmeasured
The run is a seeded throwaway job, not the live backlog issue the ticket names
Parts worth reviewing closely
The rest of this is for whoever reviews the code.
1. §7 — the account as a layer. The check derives a structured record and then grades it, rather than asserting one property at a time straight off the routes. The question: does that extra layer earn itself? A wrong answer here is the whole design — everything in §8 and §10 hangs off it.
2. Decision 2 and §10's verdict shape. Two arms report rather than fail — the plan half when the run never planned, and a per-path arm when the run did its work through the shell, which the record does not cover by design. Is "report, name it, log it, and refuse to call it a pass" the right treatment, or should either one fail? Getting it wrong means either an epic signed off on unexercised halves, or a kill-line check that goes red for stated gaps.
3. §7 — matching a path across the two surfaces. The collection's key is run-namespaced and canonicalized; the item stream carries the raw tool input. The spec matches on whole trailing path segments and rejects fixtures with colliding basenames, rather than re-implementing the recorder's canonicalization. Is that the right trade, given FIX-1184 is a warning about exactly this class of matching?
Where I'm unsure — now answered, kept for the record. Whether assertion 2 (the two surfaces must name the same mutations) was a genuine sufficiency signal or an over-tight coupling to LAB-134's recorder. It was over-tight: the recorder is allowed to skip. It now reads disagreement fails unless a gap row accounts for it, and LAB-134 ships the durable gap collection that makes that readable. See the round-1 comment below.
Deliberately absent: any judge, scorer or corpus; grading the agent's output quality; settlement (FIX-1182); and re-asserting what the two existing goal checks already cover — §10 lists those explicitly so the three stay non-overlapping.
Links
Spec doc:
spec/LAB-135.md· Linear: LAB-135 · Epic: LAB-68 · Depends on PR #1329 (LAB-134), which this branch is cut from and which this PR targets, extended by PR #1332 (the gap collection), and transitively on PR #1327 (LAB-133). None is merged.This PR never merges. It closes unmerged once the direction is approved; Linear keeps the durable copy.
POC on this branch — what survives to the reader
Throwaway, never merges. Model-free — no credential, no coding run, a few seconds. It drives the real translation and emission layers with SDK messages transcribed from LAB-134's measured runs, persists through real SQLite, and reads back over the shipped routes. So it measures persistence and readback only; it says nothing about what the harness emits, which LAB-134's POC measured and this takes as given.
All three premises held, and two of them changed §9 rather than the direction.
toolCall.nameand a parseabletoolCall.argumentssurvive the round trip, with acompleted/failedstatus beside them. This is what makes structured grading (FIX-1184) possible at all — had it come back prose-only, the design would have had to move onto LAB-134's collections alone.itemIndex;seqdoes not exist on a persisted item.seqvalues came back length 0 — FIX-1183's sharpest instance, reproduced deliberately. The measureditemIndexvalues carry a duplicate and a gap ([0,1,3,4,4,5]), both legitimate, so the assertion is non-decreasing and never contiguous.clientData(notstate); addressed by accessor key, so a wrong ref is a 404 distinct from the 403; 50 rows per page withnextCursor; and 403 withoutclient.state.read.Two traps carried into §7:
item.blockNameis the tool name whileitem.provenance.blockNameis the FSD block, and a tool result'soutputis the prose shown to the model rather than a structured result — nothing in the reader parses it.Per the contract below, please don't review these files as code.
How to review this — altitude, what's in scope, what's deliberately unsettled
This is a direction document, not an implementation. Reviewing it well means answering one question: is this the right approach?
In scope to challenge:
docs/philosophy.md?Out of scope — deliberately unsettled here, and owned by the implementing agent:
spec-poc). None of it ships. Read it to judge whether the shape holds; do not review it as code. Its quality is not a finding, and its findings are summarized in §7 — the summary is what a reviewer reacts to.One request, if you are an automated reviewer: the two lists above are the whole difference between a useful review of this document and a long one. Volume is not signal here — a single finding above the line is worth more than twenty below it.
Feedback in the second list is welcome and gets recorded verbatim in §13 for the implementer to weigh against real code. It will not be argued with, and it will not be folded into the design prose — that would pretend the spec can settle something it can't. Please don't re-raise it: one mention is enough for it to land in §13.
The one exception is the sketch and the POC. Line-level feedback on either is dropped, not recorded — there is nothing for an implementer to weigh about code that isn't shipping, and a §13 note would imply otherwise. Feedback about their direction is not line-level and is treated like any other above-the-bar finding.
The rest of the case — tradeoffs, focus practices, the account's shape
Alternative weighed: extend the existing goal check instead of adding a third. It already has the host, the store and the dispatch. Rejected because the two ask different questions and would grade at different strengths under one verdict — the existing one proves capture at its own layer and is cheaper to diagnose, this one proves sufficiency across both. Folded together, a failure tells you less than either does apart.
The simpler approach: assert the account is non-empty and stop. Tempting, and it catches the catastrophic case. Insufficient because "non-empty" is the exact bar a lossy graph clears — the epic's failure mode is a record that keeps some of a run.
Where the complexity is: not the derivation. It is keeping the check honest about its own blind spots — every assertion has to fail (or declare itself unmeasured) when the set it reads is empty; the reader has to be structurally unable to reach the transcript rather than merely instructed not to; and the grading must not quietly become a test of whether the agent was any good.
Focus practices: every assertion has a can't-tell branch (FIX-1183) · grade structured field-specific fragments, never substrings over rendered text (FIX-1184) · scope every claim to exactly what was measured · sanity-check the instrument against a known case before trusting its sweep · prove every guard fires by breaking it and watching it go red.
What the account looks like, derived from the routes alone:
Non-goals: grading the coding agent's output quality; asserting how a run was settled (FIX-1182); re-asserting what the two existing goal checks already cover; any reusable judge, scorer, harness or corpus.
Size: Small — 0 production files · 1 new goal check (4 files) · 8 graded assertions plus 2 preconditions · 0 doc surfaces · 1 PR.
Part II — technical design, sequence, edge cases, testing strategy, docs plan — is in the committed doc diff.