Repository navigation
fix(memory): capture with an evaluator marks read only the messages it judged - #2836
Conversation
…t judged The read cursor was a position in the session item log, but each request sees its own view of that log: prior completed requests sorted by (ts, per-request itemIndex), then its own live items. When two turns of one session overlap, a skip's boundary, taken from its own view, lands on a different item in a later turn's view, so it can cover a turn whose evaluator call failed (BR-6 broken: the failed turn is never observed) or leave a skipped turn unread. In CI the overlapping turns shared a millisecond, the itemIndex tie-break sorted turn C's message to position 0, and turn B's late skip marked it read. Capture with an evaluator now records the judged messages by id in a new `readMessages` field on the memorySystem state and reads a message only while its id is missing there. The first such capture carries over what the legacy watermark covered; capture without an evaluator clears the field so re-adding an evaluator resumes from the watermark; entries older than everything in the capture's session view are dropped. The watermark still moves as before, and capture without an evaluator is unchanged. The flaky test now pins the clock to the one-millisecond interleaving that failed in CI, so it fails deterministically without the fix; a second race test covers a skip that lands while an earlier turn is still running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014pGvtq33naugpfJRZN9cDW
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: eb8b3b3a-4288-4e09-a529-5c817569031c) |
The process guard requires every changeset to name its Linear issue; this fix restores FIX-1555's BR-6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014pGvtq33naugpfJRZN9cDW
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 34fa2164-5d59-4815-be99-7dc13d17c448) |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 43ab1de3-3816-467b-919e-9d32dccad09a) |
There was a problem hiding this comment.
Problem
When capture uses an evaluator, “read” was expressed only as lastProcessedIndex — a position in the full session item log. Overlapping turns (same millisecond, or a slow evaluator while another turn finishes) can interleave messages in that log. A late skip could advance the watermark past a sibling turn whose evaluator failed, so that turn was never judged again.
Did this go too far?
I treated this as a /simplify pass: look for a smaller fix before accepting more state and plumbing.
Watermark-only tweaks (single-flight capture, deferring the watermark until the evaluator returns, boundary from “this turn’s items only”) do not fully repair the interleaved-log case your fake-timer / racingModel tests encode. Ephemeral sequencer-only tracking would not survive skip/reflect persisting read state across turns or evaluator on/off. Pushing read flags onto session items would be a core contract change, not a memory-package fix.
The added surface area is real (readMessages on MemorySystemState, judged on the judge input, markJudgedRead, readJudged instead of readBoundary, optional messages on readObserveWindow). It is still the minimal durable model I could find for “mark read exactly the messages this capture judged, by id.” Keeping lastProcessedIndex in sync (including clearing readMessages when capture runs without an evaluator) is migration semantics, not accidental duplication.
Simpler directions worth considering (non-blocking)
These are follow-ups, not merge blockers:
- DRY the watermark branch filter (
idx > lastProcessedIndex && type === 'message') shared bybuildObserveContextandunreadMessages. - One
{ id, ts }Zod schema exported frommemory-system.tsand reused injudgedWindowSchema(and the localReadMessagetype). - Align text extraction between
formatMessagesandextractItemTextso payload/content fallback policy cannot drift. - Optional later: prune
readMessagesby “id still visible in currentall()” instead ofts >= oldest(allItems)— might allow id-only storage if you re-prove the history-window test; I have not checked equivalence. - Perf (only if profiling complains): evaluator capture already dominates; still, two
items.all()calls per capture and Θ(N) scans are the scaling knobs — reuse one snapshot per capture, or incremental id sets on state.
Verification
pnpm --filter @flow-state-dev/memory test — 536 tests passed on this branch.
Verdict
Approve. The complexity matches the concurrency bug; tests document the contract (evaluator add/drop/re-add, history-window pruning, racing skip vs failed peer). Inline notes are nits / optional cleanups.
Sent by Cursor Automation: Review For Best Approach
There was a problem hiding this comment.
Code Snob — already tight. No POC.
I read packages/memory/src/memory-system-blocks.ts, packages/memory/src/memory-system.ts, packages/memory/test/capture-evaluator.test.ts, packages/memory/test/memory-system.test.ts, and .changeset/memory-evaluator-marks-read-by-id.md (head 202bf37c).
A rewrite would not shrink the contract:
unreadMessagesandmarkJudgedReadare the fix: oncereadMessagesis set, selection is by id; the first evaluator capture carries the watermark; pruning is by timestamp against this capture's view so a concurrent mark stays;lastProcessedIndexstill advances so dropping the evaluator has a cursor.formatMessagesand the optionalmessagesargument are what let the window step reusereadObserveWindowinstead of copying its source and raw-input fallback.toReadMessageis the snapshot both the skip path and reflect consume. Inlining it duplicates the field pick.- Clearing
readMessagesonly when it is set is the no-evaluator "write the old shape" branch. Folding that intomarkJudgedReadmoves the lines. racingModelis the shared hold/fail/skip harness for the two overlap tests.
Left alone: the same positional cursor on the no-evaluator path, which this PR already split off. Not re-litigating the persisted readMessages field.
Sent by Cursor Automation: Code Snob
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 202bf37c87
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| * something, and after one without an evaluator: the watermark then says | ||
| * what is read. | ||
| */ | ||
| readMessages: z.array(z.object({ id: z.string(), ts: z.number() })).nullable().default(null), |
There was a problem hiding this comment.
Re-gate the persisted-state change
Adding readMessages reverses the approved FIX-1555 decision that the watermark and counters retain their existing persisted shape (specs/issues/FIX-1555/DECISIONS.md:76-89). Because this is a material design change rather than an implementation detail, pause the implementation until a follow-up amendment records the new compatibility and migration trade-offs and receives renewed approval.
AGENTS.md reference: AGENTS.md:L52-L56
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
I don't think this needs a re-gate, but the call belongs to the spec's owner, so I've put it to them and am leaving this thread open.
FIX-1555 listed "No persisted state changes" under Decided, not asked, as an implementer note about that change set, not as an approved direction. The approved direction (D2, BR-4, BR-6: a skip is final, and an error leaves the turn unread) is unchanged, and this PR is what makes BR-6 hold when turns overlap. The positional watermark can't express "read exactly what I judged" across overlapping turns, so keeping the old shape would mean leaving BR-6 broken. Per orchestration.md, an amendment needs renewed approval only for a materially changed direction. If the owner wants it on record anyway, the follow-up is a short amendment to specs/issues/FIX-1555/DECISIONS.md from main.
Generated by Claude Code
Pruning `readMessages` took its bound from the capture's whole session view, which for a request is the completed history it loaded plus its own live items. A turn that finished while this one was judging is in neither, so when two skips overlapped, the later one dropped the earlier one's mark and the next capture judged that turn again. The bound now comes from the completed history alone, and nothing is dropped when there is none. Also: the changeset is `minor` (the exported MemorySystemState gains a required field) and one user-facing sentence; the watermark filter is one helper; the read-message type is shared; the `source` case is commented. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014pGvtq33naugpfJRZN9cDW
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 4d825e3c-c079-4d4b-a697-0eef8e3d0444) |


The problem
Memory capture with an
evaluatorcan lose a turn for good. Two turns of one session overlap. The first is judgedskip, but its answer comes back late. Meanwhile the second turn's evaluator call fails. The late skip then marks the failed turn as read, so memory never observes it. The spec forbids this (FIX-1555 BR-6: "The turn stays unread and the next capture's evaluator sees it"), and the docs promise the same.This showed up as an intermittent CI failure on a spec-only PR, #2831 (
Test (3/3), run 37539012212, at 636e905). The failing test wascapture-evaluator.test.ts› "a delayed skip racing an evaluator failure never marks the failed turn read", withexpected '[message] I work at Acme, turn D' to contain 'turn C'.It is a real race in read-marking, not a fragile test. The test already waits for the background capture:
runActiondrains side-chain work beforetestFlowresolves.Root cause
The read cursor (
memorySystem.lastProcessedIndex) is a position inctx.session.items.all(). That list is not the same for every request. Each request builds its own view:(ts, itemIndex), whereitemIndexis counted per request;So a position taken in one request's view points at a different item in a later request's view. In the test:
ts. On a fast runner both turns fall in one millisecond. Ties then break on the per-requestitemIndex, so C's message (index 0) sorts to position 0, ahead of B's items. Cursor 4 covers it, and turn C is lost.How I reproduced it:
Dateso all items share one millisecond makes it fail every time, with exactly the CI message.What this does
Capture with an evaluator now records the messages it judged by id, in a new
readMessagesfield on thememorySystemstate. It reads a message only while its id is missing from that field, wherever the message sits in the log. A capture marks read exactly what it judged.readMessages, and the watermark decides what is read again. Re-adding an evaluator picks up from the watermark.readMessagesbehind.What's asked of you
Two decisions:
1. Does FIX-1555's spec need re-approval before this merges?
Codex flagged that adding
readMessagesoverturns FIX-1555's "No persisted state changes". Its thread onmemory-system.tsis still open.specs/issues/FIX-1555/DECISIONS.mdin a follow-up PR frommain.2. Fix the same positional cursor on the no-evaluator path too?
Every app running memory without an evaluator still uses the positional watermark, which has the same weakness. I checked one case: a history window of 2 and 6 sequential turns. From turn 4 on, the window found no new messages. The observer saw only the raw input (
[user] turn 4instead of[message] turn 4), so the turn's other messages, such as the assistant's reply, were skipped. Overlapping turns would hit the same race as here.I can file it in Linear if you want.
Parts worth reviewing closely
markJudgedReadandunreadMessagesinpackages/memory/src/memory-system-blocks.ts: the carry-over from the legacy watermark and the pruning bound (completed history only, throughitems.all({ includeInFlight: false })).memorySystemReflectbranch clearsreadMessagesonly when it is set, so apps without an evaluator write exactly the state they did before.memorySystemReflect's optional second parameter changed fromreadBoundary(returns a number) toreadJudged(returns{ boundary, judged }). Only memory's own capture passes it.Links: CI failure in #2831 · FIX-1555 spec at
specs/issues/FIX-1555/· merges on its own.How to review this
This is a bug fix on the direct route: there is no spec and no spec PR, because the hard part was the diagnosis and it happened in code. This PR is the only gate, so unlike a spec-backed change, the approach is in scope: nothing here was signed off upstream.
Most valuable here: the diagnosis chain. Does the repro actually reproduce the reported failure? Is the named cause the real one? Does the fix address that cause rather than the symptom? Does the regression test fail without the fix? Then the second path: the legacy shape, the null boundary, the concurrent case, the cancel path.
In scope to challenge: where the guard was placed, any behaviour change a user could notice, and whether this should have been a spec'd feature instead of a fix.
Everything else: verification
Tests in
packages/memory/test/capture-evaluator.test.ts:Each mutation was confirmed red: no carry-over, no reset, no pruning, pruning against the whole view, and marking nothing read.
Results:
@flow-state-dev/memory: 537/537 tests pass.turbo typecheckfor memory and kitchen-sink is clean.fd2f873.Changeset:
@flow-state-dev/memoryminor, because the exportedMemorySystemStategains a requiredreadMessagesfield. No docs change is needed:apps/docs/docs/memory/configuration.mdalready describes the corrected behaviour.🤖 Generated with Claude Code
https://claude.ai/code/session_014pGvtq33naugpfJRZN9cDW