Skip to content

fix(orchestration): a wait that finds its task ended is the filer's one hearing of it (FIX-2112) - #3697

Merged
jhoffner merged 4 commits into
mainfrom
fix/FIX-2112
Oct 23, 2026
Merged

jhoffner merged 4 commits into
mainfrom
fix/FIX-2112

Conversation

@jhoffner

@jhoffner jhoffner commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

The problem

A worker files a task and waits on it with getTask({ id, wait: true }) in the same turn. If the task finishes before the wait starts (a fast delegate), the worker hears the ending twice. The wait returns it, and then an onTaskSettled turn runs for the same ending. That second turn is another model call, and its answer replaces the worker's own result. FIX-2087's closure walk hit this on 2 of 2 runs (j1:read-once).

What this does

When a wait finds its task already ended and the waiting session filed the task, the wait's answer counts as the delivery. The notice that ending owed gets dropped. The onTaskSettled request still starts, because it was sent the moment the task ended, before anyone could know a wait was coming. In a Workforce session it waits behind the running turn. When it runs it finds nothing owed, so it does nothing: no turn and no model call. A notice that was already acted on gets the same treatment today.

A caller's patchMetadata can't clear a notice marker (BP-031: Workforce's board strips those keys), so the drop is a new server-owned write. It is optional on TaskCollectionRef as dropNoticeReadByWait(id), and both built-in backings implement it.

What's asked of you

Nothing here is yours to decide. One heads-up for FIX-2087 is under Engineering calls.

Parts worth reviewing closely

  • packages/orchestration/src/tasks/helpers/wait-on-task.ts: ended() → deliverEndedBeforeWait. It checks that the reader is the filer and never fails the call.
  • packages/orchestration/src/tasks/notice/task-notice.ts deliveredByWait: it drops only the marker for the ending the wait returned (current attempt, terminal status). It leaves alone a held marker (a lapsed wait's, FIX-2082) and an earlier attempt's question.
  • The ordering this relies on: the notice entry waits behind the running turn (AFTER_REPLY_CONCURRENCY / REPLY_CONCURRENCY) and then re-checks isNoticeOwed.

Linear: FIX-2112 · epic FIX-2056 · found by FIX-2087 (#3692) · merges to main

How to review this

How to review this. This is a bug fix on the direct route: there is no spec and no spec PR, because the hard part was the diagnosis and it happened in code. This PR is the only gate, so unlike a spec-backed change, the approach is in scope: nothing here was signed off upstream.

Most valuable here: the diagnosis chain. Does the repro actually reproduce the reported failure, is the named cause the real one, does the fix address that cause rather than the symptom, and does the regression test fail without the fix? Then the second path (the legacy shape, the null boundary, the concurrent case, the cancel path).

In scope to challenge: where the guard was placed, any behaviour change a user could notice, and whether this should have been a spec'd feature instead of a fix.

Engineering calls
  • Only a wait delivers, not a plain read. A plain getTask (no wait) of an ended task still leaves the notice owed. The tool tells the model "read it without them; you hear when it ends", and FIX-2078 gives a non-waiting filer its notice. A wait is the case FIX-2078/FIX-2082 already treat as the delivery ("a request waiting on the task is resumed; no onTaskSettled runs for it"). This extends that to a wait that started after the ending.
  • A new optional ref write, not patchMetadata. Workforce's board strips notice keys from caller metadata on purpose (BP-031), so the drop is server-owned, like releaseWait / clearResumeOwed. A ref without the method keeps today's behaviour.
  • For FIX-2087's j1:read-once: that grade counts every onTaskSettled request in editor's session. After this fix, one request can still start in the race, and it admits nothing ({"owed": false}, no turn). This is the same no-op a resent copy produces today. The grade should count notice requests that delivered something (result.output.owed === true), which is what this PR's model-free case counts. As written, it will keep flagging a request that heard nothing.
  • A flow whose reply holds no line (no hold on the turn's action, such as the hand-built desk.respond in the waits goal) can still run the notice alongside the turn, before the wait reads. That ordering predates this PR. Workforce's own respond and work entries hold the line.
  • Docs: I wrote the one-sentence apps/docs and README additions myself. The docs-writer / docs-editor agents couldn't be dispatched from this session.
Everything else: verification

Unit, red → green. packages/orchestration/test/tasks/wait-on-task.test.ts › "a row that ended before the wait is heard once, through the wait". The run goes through the host's dispatch, so the turn holds its line as Workforce's does.

  • Without the fix: AssertionError: expected [ { boardId: 'waits', … ending: 'completed', output: 'already done' } ] to deeply equal []
  • With it: 18/18 pass. Plus the deliveredByWait rule in task-notice.test.ts.

Goal, red → green. goals/task-board/waits-on-a-task-it-filed, leg a gains the issue's model-free case: a task turn files a piece, the piece completes, then the turn waits.

  • Without the fix: FAIL, a:an ending read by the filer's wait starts no notice turn — 1 onTaskSettled delivered it too and a:the waiting turn's answer is the task's result — output "heard"
  • With it: all legs PASS (verdict log updated).

Closure journey j1 (FIX-2087's goal from fix/FIX-2087, real model, GOAL_LEGS=a, one attempt each):

  • Run 1: j1:read-once green, but research ended after the wait parked (no marker was written), so the race wasn't hit.
  • Run 2: the race was hit (read 57 ms after the ending). j1:read-once still reports 1 onTaskSettled turn(s). The store shows that request's output was {"owed":false}. Editor's task output is its own work answer ("its battery lasts 541 hours on one charge") and noticeOwed:1:completed was cleared. j1:answer, j1:waits-in-request, j1:once and j1:tools are green. The other failures in leg a are FIX-2110/2111/2113.

Checks: pnpm typecheck ✓, pnpm lint ✓, pnpm fallow:ci ✓, node scripts/check-retired-terms.mjs ✓, orchestration tests 2069/2069 ✓, workforce tests 1549/1549 ✓ (one run under CPU load had 3 timing failures in one file; the clean rerun passed). FIX-2087's task-surface sweep shows the same 9 live doc lines as main (FIX-2119), none from this change.

Second paths (BP-035): a legacy row with no filer → no drop, the notice is sent as before. A drop that throws → logged, and the notice stays owed. A held marker for a lapsed wait → untouched (releaseWait owns it). Another session → refused at the wait's filer check. A ref without the method → old behaviour.

Changeset: @flow-state-dev/orchestration patch.


Generated by Claude Code


Note

Medium Risk
Changes task-notice delivery semantics for same-turn wait-after-end races; incorrect drops could suppress legitimate onTaskSettled or leave stale owed markers, though failures are non-fatal and scoped to the filer session.

Overview
Fixes double delivery when a filer calls getTask({ id, wait: true }) in the same turn after a child task already finished: the wait still returns the ended task immediately, but a pending onTaskSettled no longer runs a second model turn that overwrites the worker’s answer.

Orchestration: On the instant-return path in wait-on-task, if the waiting session is the filer, the code drops the ending’s notice metadata via a new server-owned dropNoticeReadByWait write (deliveredByWait / noticeReadByWaitWrite), implemented on both collection backings. Refs without the method keep the old behavior (notice may still fire).

Tests & goals: Unit test for notice-after-turn ordering; goal leg a adds the nested create/wait race (FIX-2112). Docs/README/changeset describe one hearing through the wait when notices defer behind the filing turn (Workforce-style).

Reviewed by Cursor Bugbot for commit efdaea3. Configure here.

…ne hearing of it (FIX-2112)

A task that ended before its filer's getTask wait was heard twice: the
wait returned it, and the notice its ending owed still ran an
onTaskSettled turn, whose answer replaced the task's result.

The wait now drops that notice when the waiting session is the filer,
through a new server-owned ref write (dropNoticeReadByWait), since a
caller's patchMetadata can't clear a notice marker (BP-031).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A
@vercel

vercel Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
flow-state-dev-docs Ignored Ignored Oct 23, 2026 11:00am UTC
flow-state-dev-kitchen-sink Ignored Ignored Oct 23, 2026 11:00am UTC
flow-state-dev-ui Ignored Ignored Preview Oct 23, 2026 11:00am UTC

Request Review

@linear-code

linear-code Bot commented Oct 23, 2026

Copy link
Copy Markdown

FIX-2112

@cursor

cursor Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 78ab398c-5d62-41a5-8d6d-6bc639a4ac42)

@jhoffner jhoffner left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stand-in review for Bugbot. I covered correctness, coherence and restraint, with FIX-2112 as the spec.

Verdict: nothing blocking. The fix addresses the cause the issue names: a notice the ending already owed was never paid off by the wait's read. It does that the same way the existing held-ending read does (releaseWait({ read: true })).

What I checked myself

  • Red/green. In a scratch worktree at 6708b3d, wait-on-task.test.ts and task-notice.test.ts pass (42/42). With only wait-on-task.ts reverted to main, the new test fails with expected [ { boardId: 'waits', …(4) } ] to deeply equal []. The test does fail without the fix.
  • The ending still arrives when the read never happens. The drop only happens in ended(). A deadline, a stop (endStopped), a cancel of the waiter's own task (endQuietly), or a refused wait leaves the notice owed. A failed drop logs and leaves it owed, so the failure mode is hearing it twice, never zero times.
  • The drop is narrow. deliveredByWait only drops the marker for the current attempt with a terminal status. It leaves a held marker alone (a lapsed wait's, which releaseWait owns), an earlier attempt's park question, and a row whose ending wrote resumeOwed instead. It goes through patchRef/patchOne, which recompute from the committed row, so a concurrent acknowledge just makes it unchanged.
  • Legacy rows (BP-030). Rows from before FIX-2078 have no filer, so the wait is refused wait_not_filer and nothing is dropped. The marker key format is unchanged. A ref without the optional method keeps the old behaviour, and no in-repo wrapper drops the method.
  • Restart and replay. A re-driven waiter replays getTask, finds the row still terminal, and returns it again. The drop is idempotent.
  • Coherence and restraint. It's a server-owned write rather than patchMetadata, which matches BP-031 and its sibling wait writes. Making only a wait the delivery, not a plain read, fits FIX-2078. The diff is small and adds no extra surface.

Findings

  • 🟡 Exactly-once only holds when the notice is admitted after the read. Admission isn't a claim: the marker stays sendable until acknowledge. With the default taskSettledEntry concurrency, I reproduced the double hearing at head by dropping noticeAfterTurn from the new test. In Workforce, a hold turn that starts while a deferred notice turn is running can still double up. Neither is a regression (inline on wait-on-task.ts:503).
  • 🟡 The docs, README and changeset make that promise with no conditions (inline on task-board.md:850; also README.md:991 and the changeset).
  • 🟣 A crash between the drop and the tool result settling, on a path that never parked, can lose the ending. FIX-2082's held read has the same tradeoff (inline on wait-on-task.ts:494).

Not run: the model-free goal leg a, because it would mean building the workspace into the scratch tree, and the real-model j1. The PR body's red→green evidence for both looks consistent with the code. The changeset is present, patch, names FIX-2112, and is on a published package. Its only problem is the overclaim above. I agree with the author's heads-up for FIX-2087: j1:read-once should count notice requests with owed: true, not every onTaskSettled request.


Generated by Claude Code

…s behind the turn (FIX-2112)

Review 5627158643 on #3697.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A
@cursor

cursor Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: f319ccd3-b229-4660-8518-0c930281dd72)

jhoffner pushed a commit that referenced this pull request Oct 23, 2026
…ard-once

FIX-2112 (#3697) stops the second notice turn when a task ends before its
filer's getTask wait. The ending still starts one onTaskSettled request,
which finds the notice no longer owed and runs no turn (output
{ owed: false }). j1:read-once now counts only the requests that admitted
the notice (owed: true), as goals/task-board/waits-on-a-task-it-filed does.
A new structural j1:heard-once holds that to its claim: the ending is heard
exactly once, by editor's getTask read in one request, and by no notice
turn. The coordinators store reader carries each request's result output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A
@cursor

cursor Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 94624168-1188-445b-9e1a-44e719ef3155)

@jhoffner
jhoffner marked this pull request as ready for review October 23, 2026 11:09
@jhoffner
jhoffner merged commit b72c5f8 into main Oct 23, 2026
21 checks passed
@jhoffner
jhoffner deleted the fix/FIX-2112 branch October 23, 2026 11:09
@cursor

cursor Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: e9c31656-1ee6-4734-a6c9-43032c4cd705)

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem this PR solves

When a task ends in the same turn that filed it, but before getTask({ wait: true }) runs, Workforce still schedules onTaskSettled after the turn. The wait correctly returns the ended row, but the deferred notice can deliver the same ending again and overwrite the waiter’s answer. FIX-2112 makes the wait’s return the filer’s one hearing by dropping the owed notice on the row (server-owned write), mirroring how FIX-2082 already treats resumed waits.

The same PR also lands FIX-2111 (follow-up runs where the root ran: copy session / target / targetSessionId at filing, refuse follow_up_takes_no_target, skip workstream resolution in withTarget). That is a separate product fix, but it shares the filing path.

Simplify take: is this the cleanest shape?

FIX-2112 — yes, with minor nits. I looked for a smaller fix (only suppress in the notice pass, or caller patchMetadata) and those don’t hold: notices are server-owned, and the race is real because onTaskSettled is deferred until the turn ends (noticeAfterTurn / Workforce hold+defer). deliveredByWait + noticeReadByWaitWrite + optional dropNoticeReadByWait follows the same layering as releaseWait / droppedHeldNotices / clearResumeOwed. I would not collapse this into a one-off Workforce special case without duplicating the invariant elsewhere.

FIX-2111 — right layer, noisy bundle. Persisting placement on the row (not only sessionKeyOf at runtime) is what dispatch, workstream holds, and non-Workforce readers need; withFollowUpPlacement next to filing rules is coherent. The simpler-but-wrong alternative is Workforce-only session routing without writing targetSessionId on the follow-up row. If you want a cleaner review story next time, splitting 2112 vs 2111 would help; shipping together is defensible because routed follow-ups are broken without both ledger copy and withTarget early return.

Alternatives considered (and why I’m not asking for them here):

  • Notice admission only (owed: false without clearing metadata): still leaves a resend path and doesn’t match how other wait paths clear markers.
  • Require dropNoticeReadByWait whenever markWait exists: stricter contract, but breaking for optional ref surfaces; document for custom backings is enough for now.
  • Shared followUpRoot helper across filing.ts, task-tools-capability.ts, and filerFor: good DRY follow-up, out of scope for this bugfix.

Verification

Ran pnpm --filter @flow-state-dev/orchestration test on this branch (full package suite): 2073 tests passed, including the new wait-on-task / task-notice / filing-settings cases.

Approval

No major concerns. The added API surface is proportional to the invariant you’re pinning. Inline notes are optional polish, not blockers.

Open in Web View Automation 

Sent by Cursor Automation: Review For Best Approach

await clearMarker(collection, taskId);
const task = collection.get(taskId)!;
await deliverHeldEnding(ctx, collection, task);
await deliverEndedBeforeWait(ctx, collection, collection.get(taskId) ?? task);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small readability win: ended() calls collection.get(taskId) three times in a row. After deliverHeldEnding, one fresh read into a let row would be enough for both deliverEndedBeforeWait and the return value (metadata may change across the writes). Not a behavior issue.

* Optional: both built-in backings implement it. On a ref without it the
* filer hears the ending again, as `onTaskSettled`.
*/
dropNoticeReadByWait?(id: string): Promise<TaskWriteOutcome>;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional dropNoticeReadByWait means a custom TaskCollectionRef that implements waits but not this method keeps pre-2112 double-delivery on the fast path. The doc comment calls that out — worth a one-line note in package README’s “implementing a collection” area if such hosts exist, so it’s not only discoverable from the type.

await until(state, requestId, "completed", 10_000);
expect(toolResults(seen[1]!.messages)).toContain("already done");
// Past the notice's due time: the fast path's copy and the pass's resend have both had their chance.
await new Promise((r) => setTimeout(r, 2_500));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fixed 2_500 ms sleep matches NOTICE_DUE_MS + slack and mirrors the goal leg — fine for integration realism. If this file’s runtime becomes painful, driving the notice pass explicitly (or a test-only due hook) would be simpler than more wall-clock sleeps; optional, not blocking.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Snob — already tight. No POC.

Head efdaea374. I read wait-on-task.ts (ended, deliverEndedBeforeWait, check, deliverHeldEnding, clearMarker), task-notice.ts (deliveredByWait, droppedHeldNotices, markerKey, isSendable), internal.ts (noticeReadByWaitWrite next to releaseWaitWrite), types.ts, both backings' dropNoticeReadByWait and their patch helpers, wait-fixture.ts, wait-on-task.test.ts, task-notice.test.ts, and leg a of goals/task-board/waits-on-a-task-it-filed/run.mts (plus the README, task-board, and changeset sentences).

A rewrite does not shrink this contract.

  • deliveredByWait stays in task-notice.ts. markerKey and isSendable are private there, and a held marker has to stay. noticeReadByWaitWrite is the { patch } | undefined adapter releaseWaitWrite already is. Moving the rule into internal.ts exports those helpers or copies them. Putting it in both backings copies it.
  • The caller's deliveredByWait check is not the write's check again. patchRef still persists when the updater returns the current state, so skipping the call is what leaves a row that owes nothing unwritten. The write's empty check is the fresh row inside the CAS.
  • The filer check in deliverEndedBeforeWait is the ownBoard drain path, which check skips. Threading a boolean through waitChecked adds a parameter.
  • deliverEndedBeforeWait is the same shape as clearMarker: try, warn, never fail the wait. Inlining it into ended saves a signature.
  • Both backings' one-liners match releaseWait. A shared registrar would be the first abstraction for one method.
  • Folding the drop into releaseWait does not fit. markWait already declined terminal, so there is no lapsed slot for releaseWaitWrite to match. A no-slot mode makes that function bigger. patchMetadata is the recorded decline.
  • The three tests pin different results: which marker drops, that the block is not called when the notice waits behind the turn, and that the turn's answer is the task's result. noticeAfterTurn flips the two concurrency slots together; the other heard cases need them off.

The header, ended, deliverEndedBeforeWait, the write, and the ref method all say the same sentence. The README, the task-board page, and the changeset say it again, which is the wording review already narrowed. Cutting those lines does not shrink the contract, so no POC.

Open in Web View Automation 

Sent by Cursor Automation: Code Snob

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants