From 5c8a13035e0c35ba33cd134dcbd29a7370dba7d4 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 14:53:02 +0000 Subject: [PATCH 1/8] fix(orchestration): a task's ending reaches its filer once, on the defaults too (FIX-2123) A notice was admitted by a read, not a claim, so a filer's wait and its onTaskSettled could both deliver one ending. Now: - A row records the request that filed it (server-side, from the engine's request identity, only in the filer's own session). Its notice is turned away, as `defer` turns one away, while that request still runs, and the request sends it as it ends (the request-end seam), past any backoff. A wait in the filing request reads the ending first and drops the notice. - taskSettledEntry takes a notice in one conditional write before it runs the block (`takeNotice`, both backings). A wait's drop and a take can't both go through. A block that throws leaves the notice owed, and the next copy takes it over once the failed request has ended. - A wait that finds the notice taken (still owed, or its request still running) answers `{ ok, taskId, arrivesAsNotice, message }`, naming the task, instead of the ending. Tests: the defaults path and Workforce `hold` path, red first, on the in-memory and SQLite stores, and on both backings for the take; a turn that never waits still hears its ending once, as it ends; a failed block's notice is taken over. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A --- .changeset/a-notice-is-taken-once.md | 5 + packages/orchestration/README.md | 17 +- .../src/skills/task-tools-capability.ts | 5 + .../src/tasks/collection/get-or-create.ts | 3 + .../src/tasks/collection/internal.ts | 24 ++- .../src/tasks/collection/resource-backed.ts | 52 +++++- .../src/tasks/collection/state-backed.ts | 6 + .../src/tasks/collection/types.ts | 20 +- .../src/tasks/helpers/wait-on-task.ts | 71 +++++-- .../src/tasks/notice/delivery.ts | 27 ++- .../src/tasks/notice/task-notice.ts | 94 +++++++++- .../src/tasks/notice/task-settled-entry.ts | 53 +++++- .../orchestration/src/tasks/schema/task.ts | 8 +- .../test/collection/notice-take.test.ts | 89 +++++++++ .../test/tasks/notice-claim.test.ts | 175 ++++++++++++++++++ .../test/tasks/notice-defer.test.ts | 36 +++- .../orchestration/test/tasks/wait-fixture.ts | 28 ++- .../test/conversation-board-wait.test.ts | 60 ++++++ 18 files changed, 730 insertions(+), 43 deletions(-) create mode 100644 .changeset/a-notice-is-taken-once.md create mode 100644 packages/orchestration/test/collection/notice-take.test.ts create mode 100644 packages/orchestration/test/tasks/notice-claim.test.ts diff --git a/.changeset/a-notice-is-taken-once.md b/.changeset/a-notice-is-taken-once.md new file mode 100644 index 0000000000..17c6445692 --- /dev/null +++ b/.changeset/a-notice-is-taken-once.md @@ -0,0 +1,5 @@ +--- +"@flow-state-dev/orchestration": patch +--- + +A task's ending reaches the session that filed it once, on the defaults too: a `getTask({ id, wait: true })` that reads a task already ended no longer races that task's `onTaskSettled`. A notice now waits until the request that filed its task has ended (that request sends it as it ends), and `taskSettledEntry` takes a notice in one conditional write before it runs your block. A wait that finds the notice already taken returns `{ ok: true, taskId, arrivesAsNotice: true, message }` instead of the task. Task collection refs gain an optional `takeNotice(id, notice, take)`, which both built-in backings implement; a ref without it admits notices as before (FIX-2123). diff --git a/packages/orchestration/README.md b/packages/orchestration/README.md index 5c6eac33eb..4f1bc72f49 100644 --- a/packages/orchestration/README.md +++ b/packages/orchestration/README.md @@ -507,6 +507,13 @@ the filer waited on (`getTask` with `wait`), nor a cancel the filer's own model task tool in its turn. The block runs once per ending; if it throws, the notice stays owed and is sent again. The entry queues on the session unless `concurrency` says otherwise. +A notice waits for the request that filed its task. While that request is still running (or +parked), the entry holds the notice back the way `defer` does (below), and the request sends it as +it ends. A wait in the filing request therefore reads the ending first. Before it runs `block`, +the entry takes the notice in one conditional write (the board's optional `takeNotice`, which both +built-in backings implement), so a wait that reads the task and the notice never both deliver the +same ending. A notice whose `block` threw stays owed, and the next copy takes it over. + **Holding a notice back.** Give `taskSettledEntry` a `defer(ctx, notice)` when the session can be busy with something a notice must not run beside, such as a turn that is still under way. When it answers `true`, the entry runs nothing and acknowledges nothing: the notice stays owed, and the @@ -998,10 +1005,12 @@ See [Per-generator binding](https://flow-state.dev/docs/skills/binding) for the `getTask({ id, wait: true, deadline? })` parks the calling turn on a task its session filed, until the task ends or the deadline passes, and returns the task as `getTask` always does. The deadline is milliseconds from the call, five minutes by default, within the host's -`waitDeadlines` bounds. A task that already ended is returned at once, and no `onTaskSettled` -follows for it when the flow's `onTaskSettled` waits behind the turn that filed the task, as -Workforce's `respond` and `work` turns make it: the request still runs, but your block isn't -called. At the deadline the task runs on and nothing is cancelled. If it ends before the +`waitDeadlines` bounds. A task that already ended is returned at once, and your `onTaskSettled` +block isn't called for it: a task's notice waits until the request that filed it has ended, so a +wait in that request reads the ending first. If the notice is already being handled when the wait +reads the task (another request filed it, or its turn had ended), the wait returns +`{ ok: true, taskId, arrivesAsNotice: true, message }` instead of the task: the notice is how the +session hears that ending. At the deadline the task runs on and nothing is cancelled. If it ends before the waiting request does, that request's next `getTask` on it returns the ending. If the request ends without reading it, the ending goes to `onTaskSettled`, once. Waits nest: a task's own turn may wait too. A wait is refused when it would make a chain of more than five diff --git a/packages/orchestration/src/skills/task-tools-capability.ts b/packages/orchestration/src/skills/task-tools-capability.ts index 791f6e174b..b9e55f8b57 100644 --- a/packages/orchestration/src/skills/task-tools-capability.ts +++ b/packages/orchestration/src/skills/task-tools-capability.ts @@ -84,6 +84,7 @@ import { boardResolverOf } from "../task-board/board-resolver"; // The one task-turn test (FIX-1816 BR-5a), owned by the task-board. import { isTaskTurn } from "../task-board/task-turn"; import { + arrivesAsNoticeAnswer, canHoldWait, deliverHeldEnding, toolCallOf, @@ -1487,6 +1488,8 @@ function buildTaskTools(resolve: TaskCollectionResolver, options: BuildTaskTools .optional(), }), }), + /** A wait whose task's ending arrives as a notice, which the conversation hears on its own (FIX-2123). */ + z.object({ ok: z.literal(true), taskId: z.string(), arrivesAsNotice: z.literal(true), message: z.string() }), z.object({ ok: z.literal(false), error: z.string(), taskId: z.string().optional() }), ]), execute: async (raw, ctx) => { @@ -1541,6 +1544,8 @@ function buildTaskTools(resolve: TaskCollectionResolver, options: BuildTaskTools }; } if (!waited.ok) return { ok: false as const, taskId: t.id, error: `${waited.error}: ${waited.message}` }; + // The filer's onTaskSettled took its ending first: that notice is the one hearing of it. + if (waited.arrivesAsNotice === true) return arrivesAsNoticeAnswer(waited.task); t = waited.task; } else { // The ending held for this request, when its wait on the task lapsed: diff --git a/packages/orchestration/src/tasks/collection/get-or-create.ts b/packages/orchestration/src/tasks/collection/get-or-create.ts index f54eccc228..66b23b689a 100644 --- a/packages/orchestration/src/tasks/collection/get-or-create.ts +++ b/packages/orchestration/src/tasks/collection/get-or-create.ts @@ -217,10 +217,13 @@ export async function getOrCreateTaskCollection( ...(status === "cancelled" && init.causedBy !== undefined ? { causedBy: init.causedBy } : {}), ...(createdBy !== undefined ? { createdBy } : {}), // Server-written by the backing (FIX-2078), never from `init`. - ...(filer !== undefined ? { filer: { sessionId: filer.sessionId } } : {}), + ...(filer !== undefined ? { filer: { sessionId: filer.sessionId, ...(filer.requestId !== undefined ? { requestId: filer.requestId } : {}) } } : {}), createdAt: now, updatedAt: now, }; @@ -284,6 +284,8 @@ export function releaseWaitWrite( * (`TaskCollectionRef.dropNoticeReadByWait`, FIX-2112): the wait's answer * delivered the ending, so the notice the ending owes the filer is dropped. * + * A notice already taken is left: its notice is the delivery (FIX-2123). + * * @returns The patch to write, or `undefined` when the row owes no notice for its ending. */ export function noticeReadByWaitWrite(task: Task): { patch: Partial } | undefined { @@ -292,6 +294,24 @@ export function noticeReadByWaitWrite(task: Task): { patch: Partial } | un return { patch: { metadata: { ...(task.metadata ?? {}), ...dropped } } }; } +/** + * The one rule for taking a notice to deliver it + * (`TaskCollectionRef.takeNotice`, FIX-2123): decided on the row as stored, so + * a wait's drop and a take can't both go through. + * + * @returns The patch to write, or `undefined` when the row doesn't owe the + * notice, or another request holds it. + */ +export function noticeTakeWrite( + task: Task, + notice: Pick, + take: { readonly by: string; readonly at: number; readonly replacing?: string } +): { patch: Partial } | undefined { + const taken = takenNoticePatch(task, notice, take); + if (Object.keys(taken).length === 0) return undefined; + return { patch: { metadata: { ...(task.metadata ?? {}), ...taken } } }; +} + /** * The fields a filing may change until the task first starts (FIX-2080, * ER-4; the brief's fields and the assignee since FIX-2081). Each field's rule diff --git a/packages/orchestration/src/tasks/collection/resource-backed.ts b/packages/orchestration/src/tasks/collection/resource-backed.ts index 2b94534e47..b405f3658e 100644 --- a/packages/orchestration/src/tasks/collection/resource-backed.ts +++ b/packages/orchestration/src/tasks/collection/resource-backed.ts @@ -136,6 +136,7 @@ import { markWaitWrite, releaseWaitWrite, noticeReadByWaitWrite, + noticeTakeWrite, settingsWrite, settingsChangeKind, type TaskSettingsPatch, @@ -147,7 +148,7 @@ import { generateId } from "../generate-id"; import { steerWrite, takeSteersPatch, untakenSteers } from "../steers"; import type { TaskEndingRecorder } from "./ending"; import type { DeliveryHooks } from "../notice/delivery"; -import { NOTICE_DUE_MS, LAPSED_SLOT_CHECK_MS } from "../notice/task-notice"; +import { NOTICE_DUE_MS, LAPSED_SLOT_CHECK_MS, owesUntakenNotice, resendNowPatch } from "../notice/task-notice"; import type { TaskChangeEvent, TaskChangeKind } from "./change-event"; import { createTaskChangeEmitter } from "./change-event"; import { assertDepsFileable, assertFilingSettings, withFollowUpPlacement, type TaskSettingsSupport } from "./filing"; @@ -273,7 +274,12 @@ export interface ResourceBackedOptions { * the hooks run around every write that may end a row. Omitted on a ledger * no board hands rows out from. */ - delivery?: { ownSession: string; hooks: DeliveryHooks }; + delivery?: { + ownSession: string; + /** The running request, by its engine identity (FIX-2123): stamped as the filing request of each row it files in its own session. */ + ownRequest?: string; + hooks: DeliveryHooks; + }; } /** @@ -709,13 +715,42 @@ export async function createResourceBackedTaskCollection(); + + /** + * Note a row this request filed with itself as the filing request + * (FIX-2123), and, on the first, ask to be called as the request ends: a + * notice it turned away while it ran is sent then, past any backoff, rather + * than at the pass's next resend. Not at a park: the request hasn't ended. + */ + function noteFiled(row: Task): void { + if (delivery === undefined || row.filer?.requestId === undefined || row.filer.requestId !== delivery.ownRequest) return; + const first = filedHere.size === 0; + filedHere.add(row.id); + if (!first) return; + delivery.hooks.whenRequestEnds?.(async () => { + for (const id of filedHere) { + await ref.refresh?.(id); + const row = lookupRow(id); + if (row === undefined || !owesUntakenNotice(row)) continue; + const sooner = resendNowPatch(row); + if (Object.keys(sooner).length > 0) await ref.patchMetadata(id, sooner); + await deliverAfter(lookupRow(id) ?? row); + } + }); } /** @@ -839,6 +874,7 @@ export async function createResourceBackedTaskCollection; return task; }, @@ -860,6 +896,7 @@ export async function createResourceBackedTaskCollection noticeReadByWaitWrite(task as Task)?.patch as Partial> | undefined); }, + async takeNotice(id, notice, take) { + const at = now(); + return patchRef(id, "metadata_changed", (task) => noticeTakeWrite(task as Task, notice, { ...take, at })?.patch as Partial> | undefined); + }, + async refresh(id) { const taskRef = mirror.get(id); if (taskRef !== undefined) await readCommitted(taskRef, () => undefined); diff --git a/packages/orchestration/src/tasks/collection/state-backed.ts b/packages/orchestration/src/tasks/collection/state-backed.ts index a6ab6188c7..d8f2d4c3bb 100644 --- a/packages/orchestration/src/tasks/collection/state-backed.ts +++ b/packages/orchestration/src/tasks/collection/state-backed.ts @@ -68,6 +68,7 @@ import { markWaitWrite, releaseWaitWrite, noticeReadByWaitWrite, + noticeTakeWrite, settingsWrite, settingsChangeKind, type EndingRecording, @@ -1074,6 +1075,11 @@ export function createStateBackedTaskCollection noticeReadByWaitWrite(task as Task)?.patch as Partial> | undefined); }, + async takeNotice(id, notice, take) { + const at = now(); + return patchOne(id, "metadata_changed", (task) => noticeTakeWrite(task as Task, notice, { ...take, at })?.patch as Partial> | undefined); + }, + async payOwedCascades() { return payOwedCascades(cascadeWrites, now()); }, diff --git a/packages/orchestration/src/tasks/collection/types.ts b/packages/orchestration/src/tasks/collection/types.ts index e02b8ef3c5..ccfb59410d 100644 --- a/packages/orchestration/src/tasks/collection/types.ts +++ b/packages/orchestration/src/tasks/collection/types.ts @@ -13,6 +13,7 @@ import type { Task, TaskRunLink, TaskStatus } from "../schema/task"; import type { TaskInit, TaskFilter } from "../schema/task-init"; import type { TaskSettingsPatch } from "./internal"; import type { SteerInput } from "../steers"; +import type { NoticeEnding as TaskNoticeEnding } from "../notice/task-notice"; /** * How a cancel was made (FIX-2081), as `Task.cancelledVia` records it: @@ -1044,13 +1045,30 @@ export interface TaskCollectionRef { /** * The filer's wait found the row already ended and returned it (FIX-2112): * drop the notice the ending owes the filer, which that answer delivered. - * `unchanged` when the row owes none for its ending. Server-owned, as the + * `unchanged` when the row owes none for its ending, or its notice is taken + * ({@link takeNotice}): that notice is the delivery. Server-owned, as the * other wait writes are: a caller's `patchMetadata` can't clear a notice. * * Optional: both built-in backings implement it. On a ref without it the * filer hears the ending again, as `onTaskSettled`. */ dropNoticeReadByWait?(id: string): Promise; + /** + * Take a notice the row owes, for the `onTaskSettled` request `take.by` + * that is about to deliver it (FIX-2123), in one conditional write: a wait's + * drop ({@link dropNoticeReadByWait}) and a take never both go through. + * `unchanged` when the row no longer owes it, or another request holds it, + * other than `take.replacing` (a taker the caller found has ended). The + * request that took it clears the marker once its block has run. + * + * Optional: both built-in backings implement it. On a ref without it a + * notice is admitted with no take, as before. + */ + takeNotice?( + id: string, + notice: { readonly attempt: number; readonly ending: TaskNoticeEnding }, + take: { readonly by: string; readonly replacing?: string } + ): Promise; /** * A touch of the board: pay every cascade owed past its due time * (`Task.cascadeOwed`, FIX-2080 D2), so one a crash left owed is paid. diff --git a/packages/orchestration/src/tasks/helpers/wait-on-task.ts b/packages/orchestration/src/tasks/helpers/wait-on-task.ts index 8bffa00b3a..7bba8c2d40 100644 --- a/packages/orchestration/src/tasks/helpers/wait-on-task.ts +++ b/packages/orchestration/src/tasks/helpers/wait-on-task.ts @@ -17,7 +17,9 @@ * that waits again at its deadline counts again before each new park, * except a drain on a row it handed over, whose wait already exists. * A task already ended is returned at once, and the notice its ending - * owes the filer is dropped: this answer delivered it (FIX-2112). + * owes the filer is dropped: this answer delivered it (FIX-2112). Unless + * `onTaskSettled` has already taken that notice (FIX-2123): then the notice + * is the delivery, and the answer says the ending arrives as one. * 2. **Take the row's wait slot** in one conditional write (`markWait`): a * second request waiting on the same task is refused. * 3. **Park** on the one wait gate (`parkOnWait`), whose deadline the engine @@ -67,7 +69,7 @@ import type { TaskCollectionRef } from "../collection/types"; import type { TaskClaimTicket } from "../claim-ticket"; import { resolveResourceCollection } from "../collection/resolve-resource-collection"; import { getOrCreateTaskCollection } from "../collection/get-or-create"; -import { deliveredByWait, holdsAnEnding, LAPSED_SLOT_CHECK_CAP_MS, LAPSED_SLOT_CHECK_MS } from "../notice/task-notice"; +import { deliveredByWait, holdsAnEnding, LAPSED_SLOT_CHECK_CAP_MS, LAPSED_SLOT_CHECK_MS, noticeDelivery } from "../notice/task-notice"; // The running request's own task, as the claim and the gate know it. Reached by // path, not through the task-board barrel, which imports this module. import { currentWorkerClaim } from "../../task-board/flow-policy-wiring"; @@ -132,9 +134,12 @@ export type TaskWaitRefusal = export type TaskWaitResult = /** * The task once it ended (`ended: true`), or as it stands at the deadline - * (`ended: false`, still running). The caller returns it as `getTask` would. + * (`ended: false`, still running). The caller returns it as `getTask` would, + * except with `arrivesAsNotice` (FIX-2123): the filer's `onTaskSettled` + * had already taken the ending's notice, which is the filer's one hearing of + * it, so the caller answers {@link arrivesAsNoticeAnswer} instead. */ - | { readonly ok: true; readonly task: Task; readonly ended: boolean } + | { readonly ok: true; readonly task: Task; readonly ended: boolean; readonly arrivesAsNotice?: true } | { readonly ok: false; readonly error: TaskWaitRefusal; readonly message: string } /** * The request is over: a person stopped it while it waited, or its own task @@ -485,30 +490,68 @@ function claimBoardOf(ctx: BlockContext, collectionId: string) { /** * The waited-on row ended: clear what it owes the resume, and return it. An * ending that came before the wait owes its filer a notice; when the filer is - * the one waiting, this answer delivers it, so it is dropped (FIX-2112). + * the one waiting, this answer delivers it, so it is dropped (FIX-2112). One + * the filer's `onTaskSettled` already took is its delivery instead (FIX-2123). */ async function ended(ctx: BlockContext, collection: TaskCollectionRef, taskId: string): Promise { await clearMarker(collection, taskId); const task = collection.get(taskId)!; await deliverHeldEnding(ctx, collection, task); - await deliverEndedBeforeWait(ctx, collection, collection.get(taskId) ?? task); - return { ok: true, task: collection.get(taskId) ?? task, ended: true }; + const arrivesAsNotice = await deliverEndedBeforeWait(ctx, collection, collection.get(taskId) ?? task); + return { ok: true, task: collection.get(taskId) ?? task, ended: true, ...(arrivesAsNotice ? { arrivesAsNotice: true as const } : {}) }; } /** * Drop the notice an ending owes its filer, when the filer's own wait read it * (FIX-2112). Never fails the call: a drop that fails leaves the notice owed, * and the filer hears the ending a second time. + * + * @returns Whether the filer's `onTaskSettled` took the notice first + * (FIX-2123), read off the row as stored after the drop: the notice is the + * delivery while it is owed or the request that took it still runs, and the + * wait says so. */ -async function deliverEndedBeforeWait(ctx: BlockContext, collection: TaskCollectionRef, task: Task): Promise { +async function deliverEndedBeforeWait(ctx: BlockContext, collection: TaskCollectionRef, task: Task): Promise { const filer = collection.filerOf?.(task) ?? task.filer?.sessionId; - if (filer === undefined || filer !== ctx.session.identity.id) return; - if (Object.keys(deliveredByWait(task)).length === 0) return; - try { - await collection.dropNoticeReadByWait?.(task.id); - } catch (error) { - console.warn(`[orchestration] wait "${task.id}": its ending's notice stays owed: ${(error as Error).message}`); + if (filer === undefined || filer !== ctx.session.identity.id) return false; + let row = task; + if (Object.keys(deliveredByWait(task)).length > 0) { + try { + await collection.dropNoticeReadByWait?.(task.id); + // A drop declined by a take another request committed first. + await collection.refresh?.(task.id); + row = collection.get(task.id) ?? task; + } catch (error) { + console.warn(`[orchestration] wait "${task.id}": its ending's notice stays owed: ${(error as Error).message}`); + return false; + } } + const delivery = noticeDelivery(row, ctx.request.identity.id); + if (delivery === undefined) return false; + if (delivery.owed) return true; + const ended = ctx.requestHost?.waits?.ended; + return ended !== undefined && (await ended([delivery.by]))[delivery.by] !== true; +} + +/** + * What a wait answers when the task's ending arrives as a notice (FIX-2123): + * the task, by name, and that its ending reaches the conversation as a notice + * in its own right, so a model doesn't read it as "no result" and wait again. + */ +export function arrivesAsNoticeAnswer(task: Pick): { + ok: true; + taskId: string; + arrivesAsNotice: true; + message: string; +} { + return { + ok: true, + taskId: task.id, + arrivesAsNotice: true, + message: + `Task "${task.title ?? task.goal}" (${task.id}) has ended. Its ending is arriving as a notice, ` + + `which you'll read on its own; don't wait on this task again.` + }; } /** diff --git a/packages/orchestration/src/tasks/notice/delivery.ts b/packages/orchestration/src/tasks/notice/delivery.ts index 41d909db45..842f83f7e9 100644 --- a/packages/orchestration/src/tasks/notice/delivery.ts +++ b/packages/orchestration/src/tasks/notice/delivery.ts @@ -22,7 +22,7 @@ * undeliverable, recorded on the row, and handed to the board's * `onUndeliverable`. Nothing here fails the write that already committed. */ -import { dispatchThroughSeam } from "@flow-state-dev/core/types"; +import { dispatchThroughSeam, onRequestEnded } from "@flow-state-dev/core/types"; import type { BlockContext, DispatchOutcome, SessionTarget } from "@flow-state-dev/core/types"; import type { Task } from "../schema/task"; import { clearNotice, NOTICE_DUE_MS, owedNotices, UNDELIVERABLE_PREFIX, type TaskNotice } from "./task-notice"; @@ -104,6 +104,14 @@ export interface DeliveryHooks { rows: readonly Task[], patchMetadata: (id: string, patch: Record) => Promise ): Promise; + /** + * Run `send` once the running request has ended (FIX-2123), on the + * engine's request-end seam: not when it parks, which isn't its end. A + * failure is logged, never thrown; what `send` didn't send stays owed for + * the filer's pass. Nothing runs on a context with no seam. Optional, so + * hooks built by hand keep compiling: without it, the pass sends instead. + */ + whenRequestEnds?(send: () => Promise): void; } /** The refusal that says the filer's session is gone: it can never be told. */ @@ -159,9 +167,24 @@ export function createDelivery( filerOf, beforeEnding: (row) => owePassAt(row, Date.now() + NOTICE_DUE_MS), owePassAt, - afterEnding + afterEnding, + whenRequestEnds }; + function whenRequestEnds(send: () => Promise): void { + onRequestEnded(ctx, async (ending) => { + if (ending.status === "suspended") return; + try { + await send(); + } catch (error) { + console.warn("[orchestration] a notice held for the request that filed its task wasn't sent as it ended: the filer's pass sends it", { + boardId: board.collectionId, + error: (error as Error).message + }); + } + }); + } + async function owePassAt(row: Task, dueAt: number): Promise { if (owesNothing(row)) return; const filer = filerOf(row); diff --git a/packages/orchestration/src/tasks/notice/task-notice.ts b/packages/orchestration/src/tasks/notice/task-notice.ts index 5ea143236c..0aeb8007e7 100644 --- a/packages/orchestration/src/tasks/notice/task-notice.ts +++ b/packages/orchestration/src/tasks/notice/task-notice.ts @@ -35,6 +35,19 @@ * filer's wait on it was made owes one at its ending, and the wait that finds * it ended drops it ({@link deliveredByWait}): the wait's answer delivered it. * + * ## A notice is taken before it runs (FIX-2123) + * + * The filer's `onTaskSettled` takes a notice before it runs the flow's block, + * in one conditional write that records its request beside the marker + * ({@link takenNoticePatch}). A wait's drop and that take are writes to the + * same row, each conditional on the other not having happened, so exactly one + * goes through: a wait leaves a taken marker alone ({@link deliveredByWait}) + * and answers that the ending arrives as a notice ({@link noticeDelivery}). A + * taken marker is still owed, and the filer's pass still sends it again: a + * copy finds it taken and runs nothing while the request that took it runs, + * and takes it over once that request has ended without clearing it (its + * block failed, or its process died). + * * ## An ending held for the request that waited (FIX-2082) * * A wait's deadline lapses the row's wait slot rather than clearing it. An @@ -146,6 +159,15 @@ interface OwedMarker { readonly held?: true; } +/** + * Where the `onTaskSettled` request delivering an owed notice records that it + * took it (FIX-2123), by its request id, server-written, and when. Kept beside + * the marker, as a resend record is, and left when the marker clears: a wait + * reads it to tell that the request delivering the ending still runs. Absent + * for a notice no request has taken, and for every notice owed before FIX-2123. + */ +const TAKEN_PREFIX = "noticeTaken:"; + function markerKey(attempt: number, ending: NoticeEnding): string { return `${OWED_PREFIX}${attempt}:${ending}`; } @@ -217,6 +239,70 @@ function isSendable(key: string, value: unknown): value is OwedMarker { return isMarker(key, value) && value.held !== true; } +/** + * The request that took a notice (FIX-2123), off its take record, read + * tolerantly: anything but a well-formed record reads as none. + */ +function takerOf(row: Task, attempt: number, ending: NoticeEnding): string | undefined { + const taken = (row.metadata ?? {})[`${TAKEN_PREFIX}${attempt}:${ending}`] as { by?: unknown } | null | undefined; + return taken != null && typeof taken.by === "string" ? taken.by : undefined; +} + +/** + * The request that took `notice` to deliver it (FIX-2123), or `undefined` when + * `row` no longer owes it, or owes it and no request has taken it. + */ +export function noticeTakenBy(row: Task, notice: Pick): string | undefined { + const key = markerKey(notice.attempt, notice.ending); + return isSendable(key, (row.metadata ?? {})[key]) ? takerOf(row, notice.attempt, notice.ending) : undefined; +} + +/** + * The metadata patch that takes `notice` for the request `take.by` (FIX-2123): + * its take record, naming that request. Empty when `row` doesn't owe it, or + * another request holds it: one other than `take.replacing`, the taker its + * caller found had ended. Run inside the write that commits it, on the row as + * stored. + */ +export function takenNoticePatch( + row: Task, + notice: Pick, + take: { readonly by: string; readonly at: number; readonly replacing?: string } +): Record { + const key = markerKey(notice.attempt, notice.ending); + if (!isSendable(key, (row.metadata ?? {})[key])) return {}; + const taker = takerOf(row, notice.attempt, notice.ending); + if (taker !== undefined && taker !== take.by && taker !== take.replacing) return {}; + return { [`${TAKEN_PREFIX}${notice.attempt}:${notice.ending}`]: { by: take.by, at: take.at } }; +} + +/** + * Who delivers the ending `row` holds as a notice (FIX-2123): the request, + * other than `requestId` (the one asking), that took the notice owed for that + * ending, and whether the notice is still owed. `undefined` when no other + * request took it. A wait answers that the ending arrives as a notice while + * the notice is owed, or that request still runs: the notice is the filer's + * one hearing of it. + */ +export function noticeDelivery(row: Task, requestId: string): { readonly by: string; readonly owed: boolean } | undefined { + if (row.status !== "completed" && row.status !== "errored" && row.status !== "cancelled") return undefined; + const by = takerOf(row, row.attempts, row.status); + if (by === undefined || by === requestId) return undefined; + const key = markerKey(row.attempts, row.status); + return { by, owed: isSendable(key, (row.metadata ?? {})[key]) }; +} + +/** + * Whether `row` owes a notice no request has taken (FIX-2123): one that a + * send now would deliver. What the request that filed `row` sends as it ends. + */ +export function owesUntakenNotice(row: Task): boolean { + for (const [key, value] of Object.entries(row.metadata ?? {})) { + if (isSendable(key, value) && takerOf(row, value.attempt, value.ending) === undefined) return true; + } + return false; +} + /** * The metadata patch that drops every held marker on `row` (FIX-2082): the * request it was held for read the row, which delivered the ending. @@ -234,12 +320,13 @@ export function droppedHeldNotices(row: Task): Record { * the row found it already ended (FIX-2112): the wait's answer is the * ending's delivery, as it is for a wait the ending resumes. Empty when the * row hasn't ended or owes nothing for it. A marker held for a request whose - * wait lapsed is that request's to deliver ({@link droppedHeldNotices}). + * wait lapsed is that request's to deliver ({@link droppedHeldNotices}). A + * taken marker is left too: its notice is the delivery (FIX-2123). */ export function deliveredByWait(row: Task): Record { if (row.status !== "completed" && row.status !== "errored" && row.status !== "cancelled") return {}; const key = markerKey(row.attempts, row.status); - return isSendable(key, (row.metadata ?? {})[key]) ? { [key]: null } : {}; + return isSendable(key, (row.metadata ?? {})[key]) && takerOf(row, row.attempts, row.status) === undefined ? { [key]: null } : {}; } /** @@ -537,7 +624,8 @@ export function withoutNoticeMarkers( !key.startsWith(OWED_PREFIX) && !key.startsWith(UNDELIVERABLE_PREFIX) && !key.startsWith(RESENT_PREFIX) && - !key.startsWith(DEFERRED_PREFIX) + !key.startsWith(DEFERRED_PREFIX) && + !key.startsWith(TAKEN_PREFIX) ) ); } diff --git a/packages/orchestration/src/tasks/notice/task-settled-entry.ts b/packages/orchestration/src/tasks/notice/task-settled-entry.ts index d615c148b7..cae6b6b2fa 100644 --- a/packages/orchestration/src/tasks/notice/task-settled-entry.ts +++ b/packages/orchestration/src/tasks/notice/task-settled-entry.ts @@ -13,10 +13,17 @@ * ``` * * One request per ending. The entry admits a notice only while its row still - * owes it, runs the flow's block with it, and then acknowledges it (clears the - * row's marker), so a block that fails leaves the notice owed and the filer's - * next pass sends it again. Queued on the session by default, so a second - * copy of one notice waits behind the first and finds it no longer owed. + * owes it, takes it (FIX-2123: one conditional write, so a wait's drop and this + * delivery never both go through), runs the flow's block with it, and then + * acknowledges it (clears the row's marker), so a block that fails leaves the + * notice owed and the filer's next pass sends it again, taken over from the + * request that failed. Queued on the session by default, so a second copy of + * one notice waits behind the first and finds it no longer owed. + * + * While the request that filed the task still runs, its notice is turned away + * as `defer` turns one away (FIX-2123): a wait in that request reads the + * ending first and drops it, and the request sends what is still owed as it + * ends. * * Woken with an empty input, by the wake a board's delivery owes when it * names this entry as its pass (the default): it pays what this session's @@ -25,7 +32,7 @@ import { handler, sequencer } from "@flow-state-dev/core"; import type { BlockContext, BlockDefinition, ConcurrencyConfig } from "@flow-state-dev/core/types"; import { z } from "zod"; -import { clearNotice, deferredNoticePatch, isNoticeOwed, type TaskNotice } from "./task-notice"; +import { clearNotice, deferredNoticePatch, isNoticeOwed, noticeTakenBy, type TaskNotice } from "./task-notice"; import { filerResolver, TASK_SETTLED_ENTRY } from "./delivery"; import { delegatingBoardsOf, payBoardDebts, settlePassWake, type BoardDebts } from "./pass"; @@ -74,6 +81,17 @@ export interface TaskSettledEntryOptions { readonly defer?: (ctx: BlockContext, notice: TaskSettledInput) => Promise; } +/** + * Whether a request of this session has stopped running (FIX-2123), as the + * engine records it. A host that keeps no such record answers `true`: there a + * notice is admitted as it was before. + */ +async function hasEnded(ctx: BlockContext, requestId: string): Promise { + const ended = ctx.requestHost?.waits?.ended; + if (ended === undefined) return true; + return (await ended([requestId]))[requestId] === true; +} + /** The running session's board a notice names, at this session's partition. */ async function boardOf(ctx: BlockContext, boardId: string) { return (await delegatingBoardsOf(ctx, boardId)).find((board) => board.ref.collectionId === boardId); @@ -114,11 +132,32 @@ export function taskSettledEntry(options: TaskSettledEntryOptions) { // A notice its row doesn't owe was delivered already, or never owed. if (row === undefined || !isNoticeOwed(row, notice)) return { owed: false }; // Not now: left owed, for the pass to send again, with the time it was - // turned away, so its age (which ends it as undeliverable) counts from then. - if (options.defer !== undefined && (await options.defer(ctx, notice))) { + // turned away, so its age (which ends it as undeliverable) counts from + // then. Turned away while the request that filed the task runs too + // (FIX-2123): a wait in it reads the ending first, and that request + // sends what is still owed as it ends. + const own = ctx.request.identity.id; + const filing = row.filer?.requestId; + const turnedAway = + (options.defer !== undefined && (await options.defer(ctx, notice))) || + (filing !== undefined && filing !== own && !(await hasEnded(ctx, filing))); + if (turnedAway) { await board!.ref.patchMetadata(row.id, deferredNoticePatch(row, notice, board!.ref.now())); return { owed: false }; } + // Taken in one conditional write before the block runs (FIX-2123), so a + // wait's drop and this delivery can't both go through. Taken by a request + // still running: it is delivering it. Taken by one that ended without + // clearing it (its block failed): this copy takes it over. + const taker = noticeTakenBy(row, notice); + if (taker !== undefined && taker !== own && !(await hasEnded(ctx, taker))) return { owed: false }; + if (board!.ref.takeNotice !== undefined) { + const took = await board!.ref.takeNotice(row.id, notice, { + by: own, + ...(taker !== undefined && taker !== own ? { replacing: taker } : {}) + }); + if (took.outcome !== "recorded") return { owed: false }; + } return { owed: true, notice }; } }); diff --git a/packages/orchestration/src/tasks/schema/task.ts b/packages/orchestration/src/tasks/schema/task.ts index a493bcd59f..850f9a1fde 100644 --- a/packages/orchestration/src/tasks/schema/task.ts +++ b/packages/orchestration/src/tasks/schema/task.ts @@ -564,8 +564,14 @@ export const taskSchema = z.object({ * is delivered. Its flow is read from the session's own record when needed. * **Absent** on a row filed before it (BP-030): a composing layer may name a * fallback, and with none its notice is settled undeliverable. + * + * `requestId` is the request that filed it, when that request runs in the + * filer's session (FIX-2123), from the engine's request identity: the row's + * notice waits until that request ends, so a wait in it reads the ending + * first. Absent on a row filed before it, and on a follow-up filed outside + * the filer's session: its notice is admitted at once, as before. */ - filer: z.object({ sessionId: z.string() }).optional(), + filer: z.object({ sessionId: z.string(), requestId: z.string().optional() }).optional(), /** * The task whose ending cancelled this row (FIX-2080): set by a cascade's diff --git a/packages/orchestration/test/collection/notice-take.test.ts b/packages/orchestration/test/collection/notice-take.test.ts new file mode 100644 index 0000000000..26949e1703 --- /dev/null +++ b/packages/orchestration/test/collection/notice-take.test.ts @@ -0,0 +1,89 @@ +/** + * Taking a notice (FIX-2123): the filer's `onTaskSettled` takes a notice in + * one conditional write before it runs its block, and the filer's wait drops + * a notice it read in another. The two are decided on the row as stored, so + * exactly one goes through, whichever comes first. The same rules on both + * backings. + */ +import { describe, expect, it } from "vitest"; +import { + createResourceBackedTaskCollection, + createStateBackedTaskCollection, + type Task, + type TaskCollectionRef, +} from "../../src/tasks"; +import { createFakeResourceCollection, createFakeSequencerState } from "../helpers"; + +async function sequencerBacking(): Promise { + const sequencer = createFakeSequencerState<{ tasks: Record }>({ tasks: {} }); + return createStateBackedTaskCollection({ collectionId: "tasks", state: sequencer, now: () => 1000 }); +} + +async function resourceBacking(): Promise { + return createResourceBackedTaskCollection({ collectionId: "tasks", collection: createFakeResourceCollection(), now: () => 1000 }); +} + +const BACKINGS = [ + ["sequencer-backed", sequencerBacking], + ["resource-backed", resourceBacking], +] as const; + +const KEY = "noticeOwed:1:completed"; +const NOTICE = { attempt: 1, ending: "completed" } as const; + +/** A row that completed and owes its filer the notice, written as the ledger's recorder would. */ +async function endedOwing(tasks: TaskCollectionRef): Promise { + await tasks.addTask({ id: "t", goal: "g", assignee: "ops" }); + const claimed = await tasks.claim("w", { leaseDurationMs: 10_000 }); + await tasks.complete(claimed!.id, "done"); + await tasks.patchMetadata("t", { [KEY]: { attempt: 1, ending: "completed", due: 1000 } }); +} + +const marker = (tasks: TaskCollectionRef) => (tasks.get("t") as Task).metadata?.[KEY]; +const take = (tasks: TaskCollectionRef) => (tasks.get("t") as Task).metadata?.["noticeTaken:1:completed"]; + +describe.each(BACKINGS)("taking a notice (%s)", (_name, makeBacking) => { + it("a take first leaves the wait's drop nothing to drop: the notice is the delivery", async () => { + const tasks = await makeBacking(); + await endedOwing(tasks); + expect(await tasks.takeNotice!("t", NOTICE, { by: "r-notice" })).toEqual({ outcome: "recorded" }); + expect(take(tasks)).toEqual({ by: "r-notice", at: 1000 }); + expect(marker(tasks)).toEqual({ attempt: 1, ending: "completed", due: 1000 }); + + expect(await tasks.dropNoticeReadByWait!("t")).toEqual({ outcome: "unchanged" }); + expect(marker(tasks)).toEqual({ attempt: 1, ending: "completed", due: 1000 }); + expect(take(tasks)).toMatchObject({ by: "r-notice" }); + }); + + it("a drop first leaves no notice to take: the wait was the delivery", async () => { + const tasks = await makeBacking(); + await endedOwing(tasks); + expect(await tasks.dropNoticeReadByWait!("t")).toEqual({ outcome: "recorded" }); + expect(await tasks.takeNotice!("t", NOTICE, { by: "r-notice" })).toEqual({ outcome: "unchanged" }); + expect(marker(tasks)).toBeNull(); + }); + + it("a notice taken by one request is refused to another, unless that one names it as the taker that ended", async () => { + const tasks = await makeBacking(); + await endedOwing(tasks); + await tasks.takeNotice!("t", NOTICE, { by: "r1" }); + expect(await tasks.takeNotice!("t", NOTICE, { by: "r2" })).toEqual({ outcome: "unchanged" }); + expect(take(tasks)).toMatchObject({ by: "r1" }); + // r1's block failed and its request ended: the pass's copy takes it over. + expect(await tasks.takeNotice!("t", NOTICE, { by: "r2", replacing: "r1" })).toEqual({ outcome: "recorded" }); + expect(take(tasks)).toMatchObject({ by: "r2" }); + // The same request taking it again (its replay) keeps it. + expect(await tasks.takeNotice!("t", NOTICE, { by: "r2" })).toEqual({ outcome: "recorded" }); + }); + + it("takes a marker stored before takes existed, and never one held for a request whose wait lapsed", async () => { + const tasks = await makeBacking(); + await endedOwing(tasks); + // Stored before FIX-2078: no due time, and no take. + await tasks.patchMetadata("t", { [KEY]: { attempt: 1, ending: "completed" } }); + expect(await tasks.takeNotice!("t", NOTICE, { by: "r1" })).toEqual({ outcome: "recorded" }); + + await tasks.patchMetadata("t", { [KEY]: { attempt: 1, ending: "completed", due: 1000, held: true } }); + expect(await tasks.takeNotice!("t", NOTICE, { by: "r2" })).toEqual({ outcome: "unchanged" }); + }); +}); diff --git a/packages/orchestration/test/tasks/notice-claim.test.ts b/packages/orchestration/test/tasks/notice-claim.test.ts new file mode 100644 index 0000000000..a5a7d228ab --- /dev/null +++ b/packages/orchestration/test/tasks/notice-claim.test.ts @@ -0,0 +1,175 @@ +/** + * A task's ending reaches the session that filed it once (FIX-2123): through + * the filer's wait, or through `onTaskSettled`, never both. + * + * - The turn that filed the task is still running: its notice waits for it to + * end, so a wait in that turn reads the ending first and drops the notice. + * On a flow's defaults, where the turn holds no line. + * - A turn that never waits hears it from `onTaskSettled`, sent as the turn ends. + * - A notice already running (taken) when another turn waits on the task: the + * wait answers that the ending arrives as a notice, and the notice is the one + * delivery. + * + * Each runs on the in-memory stores and on SQLite. + */ +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import type { FlowInstance, GeneratorModelCallOptions } from "@flow-state-dev/core/types"; +import { inMemoryStores, type StoreAdapter } from "@flow-state-dev/engine"; +import { sqliteStores } from "@flow-state-dev/store-sqlite"; +import type { TaskSettledInput } from "../../src"; +import { + fileCall, + finalAnswer, + rows, + runtimeFor, + SESSION, + stepModel, + toolResults, + until, + USER, + waitAgain, + waitCall, + waitFlow, + waitedTaskId +} from "./wait-fixture"; + +const dirs: string[] = []; +afterEach(async () => { + for (const dir of dirs.splice(0)) await rm(dir, { recursive: true, force: true }); +}); + +const STORES: ReadonlyArray Promise]> = [ + ["in-memory", async () => inMemoryStores()], + [ + "sqlite", + async () => { + const dir = await mkdtemp(join(tmpdir(), "fsd-notice-claim-")); + dirs.push(dir); + return sqliteStores({ filename: join(dir, "store.db") }); + } + ] +]; + +/** The notice markers a row still owes (a cleared marker is `null`). */ +function owedMarkers(row: { metadata?: Record }): string[] { + return Object.entries(row.metadata ?? {}) + .filter(([key, value]) => key.startsWith("noticeOwed:") && value !== null) + .map(([key]) => key); +} + +/** Start a turn through the host's dispatch, which applies the action's concurrency. */ +async function startTurn(state: ReturnType): Promise { + const router = await state.getRouter(); + const path = ["wait-board", SESSION, "actions", "run"]; + const sent = await router.POST( + new Request(`http://localhost/api/flows/${path.join("/")}`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ userId: USER, input: {} }) + }), + { params: { path } } + ); + expect(sent.status).toBe(202); + return ((await sent.json()) as { request: { id: string } }).request.id; +} + +async function eventually(check: () => boolean, withinMs: number): Promise { + const deadline = Date.now() + withinMs; + while (!check()) { + if (Date.now() > deadline) throw new Error(`not within ${withinMs}ms`); + await new Promise((r) => setTimeout(r, 10)); + } +} + +const settle = (ms: number) => new Promise((r) => setTimeout(r, ms)); + +describe.each(STORES)("a task's ending is heard once by its filer (%s)", { timeout: 30_000 }, (_name, stores) => { + it("on the defaults, the filing turn's wait is the delivery: no onTaskSettled block runs beside it", async () => { + // The turn action holds no line and the notice entry queues: the notice is + // sent the moment the task ends, while the turn that filed it still runs. + const notices: TaskSettledInput[] = []; + const { model, seen } = stepModel([waitCall("c1"), finalAnswer]); + const flow = waitFlow(model, { endsOnFiling: "already done", heard: notices }); + const state = runtimeFor(flow, await stores()); + try { + const turn = await startTurn(state); + await until(state, turn, "completed", 10_000); + expect(toolResults(seen[1]!.messages)).toContain("already done"); + // Past the notice's due time: the copy sent at the ending, the one sent + // as the turn ended and the pass's resend have all had their chance. + await settle(2_500); + expect(notices).toEqual([]); + expect(owedMarkers((await rows(state, flow))[0]!)).toEqual([]); + } finally { + await state.dispose(); + } + }); + + it("a turn that files a task and never waits on it hears its ending once, from onTaskSettled, as the turn ends", async () => { + const notices: TaskSettledInput[] = []; + const { model } = stepModel([fileCall("f1"), finalAnswer]); + const flow = waitFlow(model, { endsOnFiling: "already done", heard: notices }); + const state = runtimeFor(flow, await stores()); + try { + const turn = await startTurn(state); + await until(state, turn, "completed", 10_000); + // Sent as the turn ends, not a resend's backoff later. + await eventually(() => notices.length > 0, 1_500); + expect(notices).toEqual([expect.objectContaining({ ending: "completed", output: "already done" })]); + await settle(2_500); + expect(notices).toHaveLength(1); + expect(owedMarkers((await rows(state, flow))[0]!)).toEqual([]); + } finally { + await state.dispose(); + } + }); + + it("a wait that finds the notice already running answers that the ending arrives as a notice, and the notice is heard once", async () => { + // As in a Workforce session: the turn holds the line and the notice defers + // on it, so the notice runs once the filing turn ends. A second turn + // (`hold` never waits) starts beside that running notice and waits on the task. + const notices: TaskSettledInput[] = []; + let release!: () => void; + const gate = new Promise((resolve) => (release = resolve)); + const seen: GeneratorModelCallOptions[] = []; + let taskId = ""; + const { model } = stepModel( + [ + fileCall("f1"), + (opts) => { + taskId = waitedTaskId([opts], 0); + return finalAnswer(opts); + }, + waitAgain("w1", () => taskId), + finalAnswer + ], + seen + ); + const flow: FlowInstance = waitFlow(model, { endsOnFiling: "already done", heard: notices, noticeAfterTurn: true, heardGate: () => gate }); + const state = runtimeFor(flow, await stores()); + try { + const filing = await startTurn(state); + await until(state, filing, "completed", 10_000); + // The notice's block is running, held open. + await eventually(() => notices.length === 1, 5_000); + + const second = await startTurn(state); + await until(state, second, "completed", 10_000); + const answer = toolResults(seen[3]!.messages); + expect(answer).toContain(taskId); + expect(answer).toContain("arriving as a notice"); + expect(answer).not.toContain("already done"); + + release(); + await settle(2_500); + expect(notices).toHaveLength(1); + expect(owedMarkers((await rows(state, flow))[0]!)).toEqual([]); + } finally { + release(); + await state.dispose(); + } + }); +}); diff --git a/packages/orchestration/test/tasks/notice-defer.test.ts b/packages/orchestration/test/tasks/notice-defer.test.ts index a5661fe82d..57338bf863 100644 --- a/packages/orchestration/test/tasks/notice-defer.test.ts +++ b/packages/orchestration/test/tasks/notice-defer.test.ts @@ -22,8 +22,9 @@ async function boardOf(ctx: BlockContext): Promise { return getOrCreateTaskCollection({ ctx, backing: "resource", collectionId: BOARD, collection }); } -function host(defer: () => boolean) { +function host(defer: () => boolean, failFirst = false) { const heard: TaskSettledInput[] = []; + let failed = false; const seed = handler({ name: "seed", inputSchema: z.object({ id: z.string() }), @@ -54,6 +55,10 @@ function host(defer: () => boolean) { inputSchema: z.any(), outputSchema: z.any(), execute: (notice: TaskSettledInput) => { + if (failFirst && !failed) { + failed = true; + throw new Error("the handling failed"); + } heard.push(notice); return {}; } @@ -114,3 +119,32 @@ describe("a notice its session turns away (defer)", () => { } }); }); + +describe("a notice whose block fails (FIX-2123)", () => { + it("stays owed, taken by the request that failed, and the next copy takes it over and is heard once", async () => { + const h = host(() => false, true); + try { + // The request that filed the task sends its notice as it ends; that copy's block fails. + await h.act("seed", { id: "t1" }); + const takenBy = async () => + ((await h.act("read", { id: "t1" })) as Record)["noticeTaken:1:completed"]?.by; + for (let i = 0; i < 200 && (await takenBy()) === undefined; i += 1) await new Promise((r) => setTimeout(r, 10)); + expect(await takenBy()).toEqual(expect.any(String)); + expect(h.heard).toEqual([]); + + // The filer's pass sends it again: once the request that took it has ended, a copy takes it over. + for (let i = 0; i < 200 && h.heard.length === 0; i += 1) { + await h.act("onTaskSettled", notice, true); + if (h.heard.length === 0) await new Promise((r) => setTimeout(r, 10)); + } + expect(h.heard).toEqual([expect.objectContaining({ taskId: "t1", ending: "completed" })]); + const metadata = (await h.act("read", { id: "t1" })) as Record; + expect(metadata["noticeOwed:1:completed"] ?? null).toBeNull(); + // A later copy finds it delivered. + await h.act("onTaskSettled", notice, true); + expect(h.heard).toHaveLength(1); + } finally { + await h.dispose(); + } + }); +}); diff --git a/packages/orchestration/test/tasks/wait-fixture.ts b/packages/orchestration/test/tasks/wait-fixture.ts index 785687699d..e642b9bec2 100644 --- a/packages/orchestration/test/tasks/wait-fixture.ts +++ b/packages/orchestration/test/tasks/wait-fixture.ts @@ -21,7 +21,7 @@ import { resolveResourceCollection, type TaskCollectionRef } from "../../src/tasks"; -import { deliverHeldEnding, payOwedResumes, toolCallOf, waitOnTask } from "../../src/tasks/helpers/wait-on-task"; +import { arrivesAsNoticeAnswer, deliverHeldEnding, payOwedResumes, toolCallOf, waitOnTask } from "../../src/tasks/helpers/wait-on-task"; import { taskSettledEntry, type TaskSettledInput } from "../../src"; export const USER = "u_asker"; @@ -70,6 +70,12 @@ export const waitAgain = (callId: string, id: () => string, deadline?: number): finishReason: "tool-calls" }); +/** A step that files a task and doesn't wait on it, as `createTask` alone does. */ +export const fileCall = (id: string, question = "Is ACME's SOC 2 current?"): StepFn => (_options) => ({ + toolCalls: [{ toolCallId: id, toolName: "file_task", args: { question } }], + finishReason: "tool-calls" +}); + /** A step that reads a task without waiting, as a plain `getTask({ id })` does. */ export const readTask = (callId: string, id: () => string): StepFn => (_options) => ({ toolCalls: [{ toolCallId: callId, toolName: "read_task", args: { id: id() } }], @@ -114,6 +120,8 @@ export function waitFlow( * does: `run` holds the session's line and `onTaskSettled` defers on it. */ noticeAfterTurn?: boolean; + /** Awaited by the `heard` block after it records a notice: holds the notice's request open. */ + heardGate?: () => Promise; /** The board's scope: `session` (the default), or `user`, which every session of the user reaches. */ scope?: "session" | "user"; } = {} @@ -193,6 +201,7 @@ export function waitFlow( ...(input.deadline !== undefined ? { deadlineMs: input.deadline } : {}) }); if (!waited.ok) return { ok: false, taskId, error: `${waited.error}: ${waited.message}` }; + if (waited.arrivesAsNotice === true) return arrivesAsNoticeAnswer(waited.task); const { status, output, error } = waited.task; return { ok: true, taskId, status, ...(output !== undefined ? { output } : {}), ...(error !== undefined ? { error } : {}) }; } @@ -217,9 +226,21 @@ export function waitFlow( step: call.stepKey, ...(input.deadline !== undefined ? { deadlineMs: input.deadline } : {}) }); + if (waited.ok && waited.arrivesAsNotice === true) return arrivesAsNoticeAnswer(waited.task); return waited.ok ? shown(waited.task) : { ok: false, taskId: input.id, error: `${waited.error}: ${waited.message}` }; } }); + const fileTool = handler({ + name: "file_task", + inputSchema: z.object({ question: z.string() }), + outputSchema: z.any(), + execute: async (input, ctx) => { + const board = await waitBoard(ctx); + const call = toolCallOf(ctx)!; + const taskId = await ctx.runOnce!(`fixture.file:${call.logicalId}`, async () => (await board.addTask({ goal: input.question, assignee: "researcher" })).id); + return { ok: true, taskId }; + } + }); const readTaskTool = handler({ name: "read_task", inputSchema: z.object({ id: z.string() }), @@ -275,7 +296,7 @@ export function waitFlow( actions: { run: { block: sequencer({ name: "turn" }).step( - generator({ name: "agent", model, prompt: "p", tools: [wait, waitAgainTool, readTaskTool] }) + generator({ name: "agent", model, prompt: "p", tools: [wait, waitAgainTool, readTaskTool, fileTool] }) ), inputSchema: z.object({}).passthrough(), ...(options.noticeAfterTurn === true ? { concurrency: "hold" as const } : {}) @@ -293,8 +314,9 @@ export function waitFlow( name: "record-heard", inputSchema: z.any(), outputSchema: z.any(), - execute: (notice: TaskSettledInput) => { + execute: async (notice: TaskSettledInput) => { options.heard!.push(notice); + await options.heardGate?.(); return {}; } }), diff --git a/packages/workforce/test/conversation-board-wait.test.ts b/packages/workforce/test/conversation-board-wait.test.ts index e98e663449..a1e3df55f2 100644 --- a/packages/workforce/test/conversation-board-wait.test.ts +++ b/packages/workforce/test/conversation-board-wait.test.ts @@ -364,3 +364,63 @@ describe("a task session's turn that waits on a task it filed (FIX-2082)", { tim } }); }); + +describe("a wait beside the notice turn of the task it waits on (FIX-2123)", { timeout: 30_000 }, () => { + it("answers that the task's ending arrives as a notice, and the conversation hears the ending once, from that notice", async () => { + // `respond` holds the conversation's line and never waits, so a second + // reply starts beside the notice turn that is already reading the ending. + let noticeTurns = 0; + let openNotice!: () => void; + const noticeHeld = new Promise((resolve) => (openNotice = resolve)); + let filed: string | undefined; + let secondAnswer: string | undefined; + const judgment: GeneratorModel = { + modelId: "beside-the-notice", + async generate() { + throw new Error("driven step by step"); + }, + async generateStep(options) { + const messages = (options.messages ?? []) as Array<{ role: string; content: unknown }>; + const lastUser = messages.map((m) => m.role).lastIndexOf("user"); + const asked = JSON.stringify(messages[lastUser]?.content ?? ""); + const since = toolResults(messages.slice(lastUser + 1)); + if (asked.includes("completed by eng.tasker")) { + noticeTurns += 1; + await noticeHeld; + return { text: "Nine laptops, I hear.", finishReason: "stop" }; + } + if (asked.includes("check again")) { + if (since === "") return { toolCalls: [{ toolCallId: "w1", toolName: "getTask", args: { id: filed, wait: true } }], finishReason: "tool-calls" }; + secondAnswer = since; + return { text: "Noted.", finishReason: "stop" }; + } + if (since === "") { + return { toolCalls: [{ toolCallId: "f1", toolName: "createTask", args: { goal: "Count laptops", assignee: "eng.tasker" } }], finishReason: "tool-calls" }; + } + filed = filedId(since); + return { text: "Filed.", finishReason: "stop" }; + } + }; + const host = await waitingHost(judgment); + try { + const conv = await host.conversation("alice", "lead"); + const first = await host.post("alice", conv, "respond", { message: "count laptops" }); + expect(await until(async () => (await statusOf(host, first)) === "completed")).toBe(true); + // The task ends, and its notice turn runs once the filing reply has ended: held open here. + expect(await until(() => noticeTurns === 1, 10_000)).toBe(true); + + const second = await host.post("alice", conv, "respond", { message: "check again" }); + expect(await until(async () => (await statusOf(host, second)) === "completed", 10_000)).toBe(true); + expect(secondAnswer).toContain(filed!); + expect(secondAnswer).toContain("arriving as a notice"); + expect(secondAnswer).not.toContain("eng.tasker did"); + + openNotice(); + await host.settled(); + expect(noticeTurns).toBe(1); + } finally { + openNotice(); + await host.dispose(); + } + }); +}); From 7e5b3c9646099c2e81e314a5c2ffde63a9969a7a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 15:00:53 +0000 Subject: [PATCH 2/8] docs(orchestration): when a task's ending reaches its filer (FIX-2123) Update "Waiting on a task" and "Hearing how a task ended": a wait on an already-ended task takes the ending whatever the entry's options, a wait while the notice is being handled gets arrivesAsNotice, and a notice is held until the request that filed the task has ended. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A --- apps/docs/docs/orchestration/task-board.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/apps/docs/docs/orchestration/task-board.md b/apps/docs/docs/orchestration/task-board.md index bbc64316e0..487758acdd 100644 --- a/apps/docs/docs/orchestration/task-board.md +++ b/apps/docs/docs/orchestration/task-board.md @@ -796,9 +796,11 @@ When a task the board handed out ends, the session that filed it is told, once. | `question` | On `parked`: what the worker asked. Answer it with [`answerTask`](#when-a-task-stops-on-a-question) | | `cancelled` | On `errored` or `cancelled`: the ids of the dependents this ending [cancelled](#when-a-dependency-fails) that the same session filed. Those dependents send no notice of their own | +A notice is held until the request that filed the task has ended, including while that request is paused on a [`getTask` wait](#waiting-on-a-task), and it's sent as that request ends. A turn that files a task and never waits on it hears the ending once that turn ends. + A task that fails and is retried sends nothing until its last attempt ends. These endings send no notice either: -- A task the filer [waited on](#waiting-on-a-task) with `getTask({ id, wait: true })` while it ended. The waiting call gets the result instead. An ending after the wait's deadline goes to the waiting request's next `getTask` while it runs, and comes here only if that request ends without reading it. +- A task the filer [waited on](#waiting-on-a-task) with `getTask({ id, wait: true })`, either while it ended or after it ended but before an `onTaskSettled` request started handling its notice. The waiting call gets the result instead. Once a request has started handling the notice, the ending comes from the notice, and a wait while that request is in progress gets [`arrivesAsNotice`](#waiting-on-a-task). An ending after the wait's deadline goes to the waiting request's next `getTask` while it runs, and comes here only if that request ends without reading it. - A cancel the filer's own model made with its `updateTask` tool (`status: "cancelled"`), in its own turn. The tool's result already told it, and names the dependents the cancel took along. A cancel from anywhere else does send one: a person, an app calling the board's actions, or a cascade from a task another session filed. @@ -846,9 +848,21 @@ getTask({ id: "task_8f2", wait: true }) You can wait in the same turn or in a later one. Your turn pauses until the task ends, then carries on with the task as `getTask` always returns it: its status, and its output once it completed. Nothing holds a request open while it waits, so a server restart loses nothing. -A task that already ended is returned at once, and no `onTaskSettled` follows for it, as long -as your `onTaskSettled` waits behind the turn that filed the task. A Workforce conversation or -task turn works that way. The request still runs, but your block isn't called. +A task that already ended is returned at once. If no `onTaskSettled` request has started +handling that ending's notice yet, the wait takes the ending and your block isn't called for it. +A wait after your block has handled the notice returns the task the same way. + +If an `onTaskSettled` request has started handling the notice and hasn't finished, for example +a second turn waits on the task while that request is in progress, `getTask` returns this +instead of the task: + +```ts +{ ok: true, taskId: "task_8f2", arrivesAsNotice: true, message: "…" } +``` + +`message` names the task, says its ending is arriving as a notice, and tells the model not to +wait on it again. +The session hears that ending once, from the notice. A wait has a deadline, five minutes unless you pass `deadline` in milliseconds. At the deadline your turn carries on with the task as it stands, usually still `in_progress`. Nothing is From 0501d15228db0cbc4473d2182bb398c448f212ed Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 15:25:47 +0000 Subject: [PATCH 3/8] fix(orchestration): hold no request-end hook where no notice can be held (FIX-2123) The send at the filing request's end registered a request-end hook at every filing on a board that delivers. A context built outside a request's run (testBlock's) never runs its hooks, and the engine keeps owed hooks process-wide: every later host shutdown in the process then waited out its 10 s drain budget. On CI's shared module cache that made wait-on-task.test.ts time out test by test. The hook is registered only where a notice can be held for the request: a host that records which requests have ended (`requestHost.waits.ended`). Anywhere else nothing is held, so nothing needs sending at its end. Regression test: a row filed under testBlock, then a host's dispose finishes in under 2 s (it took the 10 s budget before). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A --- .../src/tasks/notice/delivery.ts | 6 +++ .../test/tasks/notice-claim.test.ts | 37 +++++++++++++++++++ .../test/tasks/notice-defer.test.ts | 16 +++----- 3 files changed, 49 insertions(+), 10 deletions(-) diff --git a/packages/orchestration/src/tasks/notice/delivery.ts b/packages/orchestration/src/tasks/notice/delivery.ts index 842f83f7e9..e97967043f 100644 --- a/packages/orchestration/src/tasks/notice/delivery.ts +++ b/packages/orchestration/src/tasks/notice/delivery.ts @@ -172,6 +172,12 @@ export function createDelivery( }; function whenRequestEnds(send: () => Promise): void { + // Only where a notice can be held for this request: a host that records + // which requests have ended. Anywhere else nothing is held for it, and a + // hook would be owed for nothing: a context built outside a request's + // run (a test harness's) never runs its hooks, and a host's shutdown + // waits out its drain budget for every hook still owed in the process. + if (ctx.requestHost?.waits?.ended === undefined) return; onRequestEnded(ctx, async (ending) => { if (ending.status === "suspended") return; try { diff --git a/packages/orchestration/test/tasks/notice-claim.test.ts b/packages/orchestration/test/tasks/notice-claim.test.ts index a5a7d228ab..de6aba7a7a 100644 --- a/packages/orchestration/test/tasks/notice-claim.test.ts +++ b/packages/orchestration/test/tasks/notice-claim.test.ts @@ -19,7 +19,12 @@ import { afterEach, describe, expect, it } from "vitest"; import type { FlowInstance, GeneratorModelCallOptions } from "@flow-state-dev/core/types"; import { inMemoryStores, type StoreAdapter } from "@flow-state-dev/engine"; import { sqliteStores } from "@flow-state-dev/store-sqlite"; +import { dispatcher, handler } from "@flow-state-dev/core"; +import { testBlock } from "@flow-state-dev/testing"; +import { z } from "zod"; import type { TaskSettledInput } from "../../src"; +import { defineTaskCollection, type TaskCollectionRef, type TaskWorker } from "../../src/tasks"; +import { taskBoard } from "../../src/task-board"; import { fileCall, finalAnswer, @@ -173,3 +178,35 @@ describe.each(STORES)("a task's ending is heard once by its filer (%s)", { timeo } }); }); + +describe("a row filed by a block run outside a request (FIX-2123)", () => { + it("leaves nothing for a host's shutdown to wait on: its context records no ended requests, so no notice is held for it", async () => { + // `testBlock` builds a request's context but never ends the request. A + // request-end hook registered there stays owed in the process, and every + // host's shutdown in it waits out its drain budget for it. + const board = taskBoard({ + name: "outside-run", + boardId: "outside-run", + mode: "delegation", + collection: defineTaskCollection({ id: "outside-run-ledger", scope: "user" }), + workers: { implement: dispatcher({ name: "outside-impl", type: "task", action: "implement", session: "per-task" }) as unknown as TaskWorker } + }); + const file = handler({ + name: "outside-file", + inputSchema: z.unknown(), + uses: [board.capability], + execute: async (_input, ctx) => { + const tasks = await (ctx.cap as Record Promise }>)["outside-run"].tasks(); + await tasks.addTask({ goal: "work", assignee: "implement" }); + return {}; + } + }); + expect((await testBlock(file, { input: undefined })).error).toBeNull(); + + const state = runtimeFor(waitFlow(stepModel([]).model)); + await state.getRouter(); + const started = Date.now(); + await state.dispose(); + expect(Date.now() - started).toBeLessThan(2_000); + }); +}); diff --git a/packages/orchestration/test/tasks/notice-defer.test.ts b/packages/orchestration/test/tasks/notice-defer.test.ts index 57338bf863..40bfc69bf0 100644 --- a/packages/orchestration/test/tasks/notice-defer.test.ts +++ b/packages/orchestration/test/tasks/notice-defer.test.ts @@ -124,19 +124,15 @@ describe("a notice whose block fails (FIX-2123)", () => { it("stays owed, taken by the request that failed, and the next copy takes it over and is heard once", async () => { const h = host(() => false, true); try { - // The request that filed the task sends its notice as it ends; that copy's block fails. await h.act("seed", { id: "t1" }); - const takenBy = async () => - ((await h.act("read", { id: "t1" })) as Record)["noticeTaken:1:completed"]?.by; - for (let i = 0; i < 200 && (await takenBy()) === undefined; i += 1) await new Promise((r) => setTimeout(r, 10)); - expect(await takenBy()).toEqual(expect.any(String)); + await expect(h.act("onTaskSettled", notice, true)).rejects.toThrow(/execution_error/); expect(h.heard).toEqual([]); + const failed = (await h.act("read", { id: "t1" })) as Record; + expect(failed["noticeOwed:1:completed"]).toMatchObject({ attempt: 1, ending: "completed" }); + expect(failed["noticeTaken:1:completed"]?.by).toEqual(expect.any(String)); - // The filer's pass sends it again: once the request that took it has ended, a copy takes it over. - for (let i = 0; i < 200 && h.heard.length === 0; i += 1) { - await h.act("onTaskSettled", notice, true); - if (h.heard.length === 0) await new Promise((r) => setTimeout(r, 10)); - } + // The filer's pass sends it again: the request that took it has ended, so this copy takes it over. + await h.act("onTaskSettled", notice, true); expect(h.heard).toEqual([expect.objectContaining({ taskId: "t1", ending: "completed" })]); const metadata = (await h.act("read", { id: "t1" })) as Record; expect(metadata["noticeOwed:1:completed"] ?? null).toBeNull(); From adbcf0b364b8fbbf199eae5d034274b7a4569f1f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 16:03:50 +0000 Subject: [PATCH 4/8] fix(orchestration): a parked filing request holds no notice; a live taker is never replaced (FIX-2123) Review 5630613272 on #3733: - A notice is held only while the request that filed its task is running, not while it is parked on a person or a wait. Core's RequestWaits gains an optional `running(requestIds)`, which the engine implements (in_progress, this session); `ended` folded suspended and running together. The request sends what it held as it parks too: the record still reads in_progress while its hooks run, so the filer's pass is owed shortly instead. - A request that runs again after its park notes the rows it filed when it resolves the board, so it sends what it held as it next parks or ends. Registered once per request and board; never on a ledger that owes no notices (a plan board). - A wait on a task whose notice another request took answers arrivesAsNotice whether that notice is under way or done. - A copy that finds the notice taken by a request that hasn't ended, running or parked, runs nothing and records itself as turned away, so the notice doesn't age out while it is delivered. - A filing's metadata can't carry notice records. Tests: a filer parked on a person hears its task during the park, and its resumed wait answers arrivesAsNotice; a request that runs again after its park sends the held notice as it ends; a second copy beside a running or parked taker runs nothing (red when the live-taker line goes); engine `running` tells running from parked. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A --- .changeset/a-notice-is-taken-once.md | 2 +- .changeset/request-waits-running.md | 6 ++ packages/core/src/types/request-host.ts | 9 ++ packages/engine/src/durability/wait-chain.ts | 16 +++ packages/engine/test/wait-chain.test.ts | 26 +++++ packages/orchestration/README.md | 25 ++--- .../src/tasks/collection/internal.ts | 5 +- .../src/tasks/collection/resource-backed.ts | 49 ++++++---- .../src/tasks/helpers/wait-on-task.ts | 17 ++-- .../src/tasks/notice/delivery.ts | 44 ++++++--- .../src/tasks/notice/task-notice.ts | 21 ++-- .../src/tasks/notice/task-settled-entry.ts | 40 +++++--- .../test/collection/notice-take.test.ts | 11 +++ .../test/tasks/notice-claim.test.ts | 92 +++++++++++++++++- .../test/tasks/notice-defer.test.ts | 97 ++++++++++++++++--- .../orchestration/test/tasks/wait-fixture.ts | 34 ++++++- .../test/conversation-board-wait.test.ts | 3 +- 17 files changed, 400 insertions(+), 97 deletions(-) create mode 100644 .changeset/request-waits-running.md diff --git a/.changeset/a-notice-is-taken-once.md b/.changeset/a-notice-is-taken-once.md index 17c6445692..6b7b8c6ec9 100644 --- a/.changeset/a-notice-is-taken-once.md +++ b/.changeset/a-notice-is-taken-once.md @@ -2,4 +2,4 @@ "@flow-state-dev/orchestration": patch --- -A task's ending reaches the session that filed it once, on the defaults too: a `getTask({ id, wait: true })` that reads a task already ended no longer races that task's `onTaskSettled`. A notice now waits until the request that filed its task has ended (that request sends it as it ends), and `taskSettledEntry` takes a notice in one conditional write before it runs your block. A wait that finds the notice already taken returns `{ ok: true, taskId, arrivesAsNotice: true, message }` instead of the task. Task collection refs gain an optional `takeNotice(id, notice, take)`, which both built-in backings implement; a ref without it admits notices as before (FIX-2123). +A task's ending reaches the session that filed it once, on the defaults too: a `getTask({ id, wait: true })` that reads a task already ended no longer races that task's `onTaskSettled`. A notice now waits while the request that filed its task is running (not while it is parked on a person or a wait), and that request sends it as it parks or ends. `taskSettledEntry` takes a notice in one conditional write before it runs your block, and a copy never takes a notice from a request that hasn't ended. A wait that finds the notice taken returns `{ ok: true, taskId, arrivesAsNotice: true, message }` instead of the task. Task collection refs gain an optional `takeNotice(id, notice, take)`, which both built-in backings implement; a ref without it admits notices as before. A filing's `metadata` can no longer carry notice records (FIX-2123). diff --git a/.changeset/request-waits-running.md b/.changeset/request-waits-running.md new file mode 100644 index 0000000000..1ed9c31a1a --- /dev/null +++ b/.changeset/request-waits-running.md @@ -0,0 +1,6 @@ +--- +"@flow-state-dev/core": patch +"@flow-state-dev/engine": patch +--- + +`RequestWaits` gains an optional `running(requestIds)`: whether each request of the running session is running its own work now (`in_progress`), as opposed to parked (`suspended`) or ended. The engine's request host implements it (FIX-2123). diff --git a/packages/core/src/types/request-host.ts b/packages/core/src/types/request-host.ts index b7c47af9e7..4e11a06550 100644 --- a/packages/core/src/types/request-host.ts +++ b/packages/core/src/types/request-host.ts @@ -192,6 +192,15 @@ export interface RequestWaits { * for a request that ended without reading the task. */ ended?(requestIds: readonly string[]): Promise>>; + /** + * Whether each request is running its own work now (`in_progress`): neither + * parked (`suspended`) nor ended. `false` for one parked + * on a person or a wait, for one that ended or whose process died, for a + * request of another session, and for one with no record. A board reads it + * to hold a task's notice while the request that filed the task still runs + * (FIX-2123), and not while that request is parked. + */ + running?(requestIds: readonly string[]): Promise>>; } /** diff --git a/packages/engine/src/durability/wait-chain.ts b/packages/engine/src/durability/wait-chain.ts index 0cbb3c6dd6..dd3532be41 100644 --- a/packages/engine/src/durability/wait-chain.ts +++ b/packages/engine/src/durability/wait-chain.ts @@ -250,10 +250,26 @@ export function createRequestWaits(inputs: { } } return answers; + }, + running: async (requestIds) => { + const answers: Record = {}; + for (const requestId of new Set(requestIds)) { + try { + const record = await inputs.readRequest(requestId); + answers[requestId] = record !== undefined && record.sessionId === inputs.sessionId && RUNNING.has(record.status); + } catch { + // A record that can't be read answers for itself only: not running, so nothing waits on it. + answers[requestId] = false; + } + } + return answers; } }; } +/** Request statuses of a request running its own work: started, and neither parked nor ended. */ +const RUNNING: ReadonlySet = new Set(["in_progress"]); + /** Request statuses after which a request runs no more of its own work. */ const WORK_OVER: ReadonlySet = new Set(["completed", "incomplete", "failed", "aborted"]); diff --git a/packages/engine/test/wait-chain.test.ts b/packages/engine/test/wait-chain.test.ts index e212683de3..193385a47e 100644 --- a/packages/engine/test/wait-chain.test.ts +++ b/packages/engine/test/wait-chain.test.ts @@ -181,6 +181,32 @@ describe("whether a request has stopped running its own work", () => { elsewhere: false }); }); + + it("tells running apart from parked: only a request in progress in this session is running", async () => { + const elsewhereRunning = { id: "elsewhereRunning", status: "in_progress", sessionId: "s_other" }; + const both = createRequestWaits({ + provider: { ...provider([]), loadSuspension: async () => null as never }, + deadlines: { defaultMs: 1, minMs: 1, maxMs: 1 }, + userId: "u", + requestId: "r", + own: undefined, + sessionId: "s", + readRequest: async (id) => { + if (id === "unreadable") throw new Error("store unavailable"); + return (id === "elsewhereRunning" ? elsewhereRunning : records[id]) as never; + } + }); + expect(await both.running!(["running", "parked", "done", "died", "redriven", "gone", "elsewhereRunning", "unreadable"])).toEqual({ + running: true, + parked: false, + done: false, + died: false, + redriven: false, + gone: false, + elsewhereRunning: false, + unreadable: false + }); + }); }); describe("the stamp a waited dispatch leaves on its child", () => { diff --git a/packages/orchestration/README.md b/packages/orchestration/README.md index 4f1bc72f49..4f5f53d3d7 100644 --- a/packages/orchestration/README.md +++ b/packages/orchestration/README.md @@ -507,12 +507,15 @@ the filer waited on (`getTask` with `wait`), nor a cancel the filer's own model task tool in its turn. The block runs once per ending; if it throws, the notice stays owed and is sent again. The entry queues on the session unless `concurrency` says otherwise. -A notice waits for the request that filed its task. While that request is still running (or -parked), the entry holds the notice back the way `defer` does (below), and the request sends it as -it ends. A wait in the filing request therefore reads the ending first. Before it runs `block`, -the entry takes the notice in one conditional write (the board's optional `takeNotice`, which both -built-in backings implement), so a wait that reads the task and the notice never both deliver the -same ending. A notice whose `block` threw stays owed, and the next copy takes it over. +A notice waits for the request that filed its task while that request is running, not while it is +parked on a person or a wait. The entry holds the notice back the way `defer` does (below), and the +request sends it as it parks or ends. A wait in the filing request therefore reads the ending +first. Before it runs `block`, the entry takes the notice in one conditional write (the board's +optional `takeNotice`, which both built-in backings implement), so a wait that reads the task and +the notice never both deliver the same ending. A notice whose `block` threw stays owed, and the +next copy takes it over; a copy that finds the notice taken by a request that hasn't ended, +running or parked, runs nothing. Holding needs a host that records which requests are running +(durable execution, `requestHost.waits.running`); anywhere else a notice is admitted at once. **Holding a notice back.** Give `taskSettledEntry` a `defer(ctx, notice)` when the session can be busy with something a notice must not run beside, such as a turn that is still under way. When it @@ -1006,11 +1009,11 @@ See [Per-generator binding](https://flow-state.dev/docs/skills/binding) for the until the task ends or the deadline passes, and returns the task as `getTask` always does. The deadline is milliseconds from the call, five minutes by default, within the host's `waitDeadlines` bounds. A task that already ended is returned at once, and your `onTaskSettled` -block isn't called for it: a task's notice waits until the request that filed it has ended, so a -wait in that request reads the ending first. If the notice is already being handled when the wait -reads the task (another request filed it, or its turn had ended), the wait returns -`{ ok: true, taskId, arrivesAsNotice: true, message }` instead of the task: the notice is how the -session hears that ending. At the deadline the task runs on and nothing is cancelled. If it ends before the +block isn't called for it: a task's notice waits while the request that filed it runs, so a wait +in that request reads the ending first. If the notice was taken by `onTaskSettled` before the wait +reads the task (the filing request was parked, or another request waits), the wait returns +`{ ok: true, taskId, arrivesAsNotice: true, message }` instead of the task, whether that notice is +still being handled or done: the notice is how the session hears that ending. At the deadline the task runs on and nothing is cancelled. If it ends before the waiting request does, that request's next `getTask` on it returns the ending. If the request ends without reading it, the ending goes to `onTaskSettled`, once. Waits nest: a task's own turn may wait too. A wait is refused when it would make a chain of more than five diff --git a/packages/orchestration/src/tasks/collection/internal.ts b/packages/orchestration/src/tasks/collection/internal.ts index fea690f4d8..7c3ef96e92 100644 --- a/packages/orchestration/src/tasks/collection/internal.ts +++ b/packages/orchestration/src/tasks/collection/internal.ts @@ -20,7 +20,7 @@ import { } from "../schema/task-status"; import { extractTaskItems } from "../items/extract-window"; import { applyEndingRecorder, endingOf, type TaskEnding, type TaskEndingRecorder } from "./ending"; -import { deliveredByWait, droppedHeldNotices, recordEnding as recordNoticeOwed, releasedHeldNotices, takenNoticePatch, type TaskNotice } from "../notice/task-notice"; +import { deliveredByWait, droppedHeldNotices, recordEnding as recordNoticeOwed, releasedHeldNotices, takenNoticePatch, withoutNoticeMarkers, type TaskNotice } from "../notice/task-notice"; import { assertDepsEditable, TaskFilingRefusedError } from "./filing"; import { hasStarted, lockedFields, type TaskEditField } from "./field-rules"; import { steersForAttempt, untakenSteers } from "../steers"; @@ -106,7 +106,8 @@ export function buildInitialTask( priority: init.priority, input: init.input, labels: init.labels, - metadata: init.metadata, + // A filing can't forge, clear or settle a notice (BP-031). + metadata: withoutNoticeMarkers(init.metadata), ...(init.followUpOf !== undefined ? { followUpOf: init.followUpOf } : {}), ...(init.filedByTask !== undefined ? { filedByTask: init.filedByTask } : {}), ...(init.session !== undefined ? { session: init.session } : {}), diff --git a/packages/orchestration/src/tasks/collection/resource-backed.ts b/packages/orchestration/src/tasks/collection/resource-backed.ts index b405f3658e..5e99034ff1 100644 --- a/packages/orchestration/src/tasks/collection/resource-backed.ts +++ b/packages/orchestration/src/tasks/collection/resource-backed.ts @@ -727,28 +727,33 @@ export async function createResourceBackedTaskCollection(); - /** * Note a row this request filed with itself as the filing request - * (FIX-2123), and, on the first, ask to be called as the request ends: a - * notice it turned away while it ran is sent then, past any backoff, rather - * than at the pass's next resend. Not at a park: the request hasn't ended. + * (FIX-2123), and ask to be called as the request stops running (it parks + * or ends): a notice turned away while it ran is sent then, past any + * backoff, rather than at the pass's next resend. The delivery registers + * once per request and board, so the call reads every row the request + * filed there, whichever resolution of the board filed it. */ function noteFiled(row: Task): void { - if (delivery === undefined || row.filer?.requestId === undefined || row.filer.requestId !== delivery.ownRequest) return; - const first = filedHere.size === 0; - filedHere.add(row.id); - if (!first) return; - delivery.hooks.whenRequestEnds?.(async () => { - for (const id of filedHere) { - await ref.refresh?.(id); - const row = lookupRow(id); - if (row === undefined || !owesUntakenNotice(row)) continue; - const sooner = resendNowPatch(row); - if (Object.keys(sooner).length > 0) await ref.patchMetadata(id, sooner); - await deliverAfter(lookupRow(id) ?? row); + const own = delivery?.ownRequest; + // A ledger that owes no notices (a plan board) has nothing to send, and + // its drain's parked rows are left untouched. + if (delivery === undefined || delivery.hooks.notices === false || own === undefined || row.filer?.requestId !== own) return; + delivery.hooks.whenRequestEnds?.(async (parked) => { + await adoptUnknownRows(mirror, options.collection); + for (const filed of listAll() as Task[]) { + if (filed.filer?.requestId !== own) continue; + await ref.refresh?.(filed.id); + const current = lookupRow(filed.id); + if (current === undefined || !owesUntakenNotice(current)) continue; + const sooner = resendNowPatch(current); + if (Object.keys(sooner).length > 0) await ref.patchMetadata(current.id, sooner); + // At a park the request still reads as running until its hooks have + // run, so a notice sent now would be turned away again: the filer's + // pass is owed shortly instead, and sends it once the park is recorded. + if (parked) await delivery.hooks.owePassAt(current, now() + NOTICE_DUE_MS); + else await deliverAfter(lookupRow(current.id) ?? current); } }); } @@ -849,6 +854,14 @@ export async function createResourceBackedTaskCollection row.filer?.requestId === delivery.ownRequest); + if (own !== undefined) noteFiled(own as Task); + } + const ref: TaskCollectionRef = { collectionId: options.collectionId, ...(options.partition !== undefined ? { partition: options.partition } : {}), diff --git a/packages/orchestration/src/tasks/helpers/wait-on-task.ts b/packages/orchestration/src/tasks/helpers/wait-on-task.ts index 7bba8c2d40..1736e7a5e0 100644 --- a/packages/orchestration/src/tasks/helpers/wait-on-task.ts +++ b/packages/orchestration/src/tasks/helpers/wait-on-task.ts @@ -69,7 +69,7 @@ import type { TaskCollectionRef } from "../collection/types"; import type { TaskClaimTicket } from "../claim-ticket"; import { resolveResourceCollection } from "../collection/resolve-resource-collection"; import { getOrCreateTaskCollection } from "../collection/get-or-create"; -import { deliveredByWait, holdsAnEnding, LAPSED_SLOT_CHECK_CAP_MS, LAPSED_SLOT_CHECK_MS, noticeDelivery } from "../notice/task-notice"; +import { deliveredByWait, holdsAnEnding, LAPSED_SLOT_CHECK_CAP_MS, LAPSED_SLOT_CHECK_MS, deliveredAsNotice } from "../notice/task-notice"; // The running request's own task, as the claim and the gate know it. Reached by // path, not through the task-board barrel, which imports this module. import { currentWorkerClaim } from "../../task-board/flow-policy-wiring"; @@ -507,9 +507,8 @@ async function ended(ctx: BlockContext, collection: TaskCollectionRef, taskId: s * and the filer hears the ending a second time. * * @returns Whether the filer's `onTaskSettled` took the notice first - * (FIX-2123), read off the row as stored after the drop: the notice is the - * delivery while it is owed or the request that took it still runs, and the - * wait says so. + * (FIX-2123), read off the row as stored after the drop: that notice is the + * delivery, whether it is under way or done, and the wait says so. */ async function deliverEndedBeforeWait(ctx: BlockContext, collection: TaskCollectionRef, task: Task): Promise { const filer = collection.filerOf?.(task) ?? task.filer?.sessionId; @@ -526,11 +525,7 @@ async function deliverEndedBeforeWait(ctx: BlockContext, collection: TaskCollect return false; } } - const delivery = noticeDelivery(row, ctx.request.identity.id); - if (delivery === undefined) return false; - if (delivery.owed) return true; - const ended = ctx.requestHost?.waits?.ended; - return ended !== undefined && (await ended([delivery.by]))[delivery.by] !== true; + return deliveredAsNotice(row, ctx.request.identity.id); } /** @@ -549,8 +544,8 @@ export function arrivesAsNoticeAnswer(task: Pick) taskId: task.id, arrivesAsNotice: true, message: - `Task "${task.title ?? task.goal}" (${task.id}) has ended. Its ending is arriving as a notice, ` + - `which you'll read on its own; don't wait on this task again.` + `Task "${task.title ?? task.goal}" (${task.id}) has ended. Its ending comes to this conversation as a notice, ` + + `on its own; don't wait on this task again.` }; } diff --git a/packages/orchestration/src/tasks/notice/delivery.ts b/packages/orchestration/src/tasks/notice/delivery.ts index e97967043f..dd13c13961 100644 --- a/packages/orchestration/src/tasks/notice/delivery.ts +++ b/packages/orchestration/src/tasks/notice/delivery.ts @@ -22,7 +22,7 @@ * undeliverable, recorded on the row, and handed to the board's * `onUndeliverable`. Nothing here fails the write that already committed. */ -import { dispatchThroughSeam, onRequestEnded } from "@flow-state-dev/core/types"; +import { dispatchThroughSeam, onRequestEnded, REQUEST_END_SEAM } from "@flow-state-dev/core/types"; import type { BlockContext, DispatchOutcome, SessionTarget } from "@flow-state-dev/core/types"; import type { Task } from "../schema/task"; import { clearNotice, NOTICE_DUE_MS, owedNotices, UNDELIVERABLE_PREFIX, type TaskNotice } from "./task-notice"; @@ -105,15 +105,25 @@ export interface DeliveryHooks { patchMetadata: (id: string, patch: Record) => Promise ): Promise; /** - * Run `send` once the running request has ended (FIX-2123), on the - * engine's request-end seam: not when it parks, which isn't its end. A - * failure is logged, never thrown; what `send` didn't send stays owed for - * the filer's pass. Nothing runs on a context with no seam. Optional, so - * hooks built by hand keep compiling: without it, the pass sends instead. + * Run `send` once the running request stops running (FIX-2123), on the + * engine's request-end seam: as it ends, or as it parks, since a parked + * request holds no notice; `send` is told which (`parked`). A failure is + * logged, never thrown; what `send` + * didn't send stays owed for the filer's pass. Nothing runs on a context + * with no seam, or on a host that doesn't record which requests are + * running. Optional, so hooks built by hand keep compiling: without it, + * the pass sends instead. */ - whenRequestEnds?(send: () => Promise): void; + whenRequestEnds?(send: (parked: boolean) => Promise): void; } +/** + * The boards a request has asked to send at its stop (FIX-2123), keyed by + * its request-end seam, which every resolution in the request shares and a + * resume replaces. Weak, so a request's entry goes with it. + */ +const sendsAtStop = new WeakMap>(); + /** The refusal that says the filer's session is gone: it can never be told. */ const FILER_GONE = "session-not-found"; @@ -171,17 +181,27 @@ export function createDelivery( whenRequestEnds }; - function whenRequestEnds(send: () => Promise): void { + function whenRequestEnds(send: (parked: boolean) => Promise): void { // Only where a notice can be held for this request: a host that records - // which requests have ended. Anywhere else nothing is held for it, and a + // which requests are running. Anywhere else nothing is held for it, and a // hook would be owed for nothing: a context built outside a request's // run (a test harness's) never runs its hooks, and a host's shutdown // waits out its drain budget for every hook still owed in the process. - if (ctx.requestHost?.waits?.ended === undefined) return; + if (ctx.requestHost?.waits?.running === undefined) return; + // Once per request and board: every resolution of the board in this + // request reaches the same request-end list. + const seam = (ctx as { readonly [REQUEST_END_SEAM]?: object })[REQUEST_END_SEAM]; + if (seam === undefined) return; + const boardKey = `${board.collectionId}\u0000${board.partition ?? ""}`; + const registered = sendsAtStop.get(seam) ?? new Set(); + if (registered.has(boardKey)) return; + registered.add(boardKey); + sendsAtStop.set(seam, registered); + // At a park too: a parked request holds no notice. The request that runs + // again after it registers anew (the board notes the rows it filed). onRequestEnded(ctx, async (ending) => { - if (ending.status === "suspended") return; try { - await send(); + await send(ending.status === "suspended"); } catch (error) { console.warn("[orchestration] a notice held for the request that filed its task wasn't sent as it ended: the filer's pass sends it", { boardId: board.collectionId, diff --git a/packages/orchestration/src/tasks/notice/task-notice.ts b/packages/orchestration/src/tasks/notice/task-notice.ts index 0aeb8007e7..ea0d78f4cf 100644 --- a/packages/orchestration/src/tasks/notice/task-notice.ts +++ b/packages/orchestration/src/tasks/notice/task-notice.ts @@ -42,7 +42,7 @@ * ({@link takenNoticePatch}). A wait's drop and that take are writes to the * same row, each conditional on the other not having happened, so exactly one * goes through: a wait leaves a taken marker alone ({@link deliveredByWait}) - * and answers that the ending arrives as a notice ({@link noticeDelivery}). A + * and answers that the ending arrives as a notice ({@link deliveredAsNotice}). A * taken marker is still owed, and the filer's pass still sends it again: a * copy finds it taken and runs nothing while the request that took it runs, * and takes it over once that request has ended without clearing it (its @@ -277,19 +277,16 @@ export function takenNoticePatch( } /** - * Who delivers the ending `row` holds as a notice (FIX-2123): the request, - * other than `requestId` (the one asking), that took the notice owed for that - * ending, and whether the notice is still owed. `undefined` when no other - * request took it. A wait answers that the ending arrives as a notice while - * the notice is owed, or that request still runs: the notice is the filer's - * one hearing of it. + * Whether the ending `row` holds reaches its filer as a notice (FIX-2123): a + * request other than `requestId` (the one asking) took the notice owed for + * that ending, whether it is delivering it now, has delivered it, or failed + * and leaves it to be sent again. A wait answers so, not with the ending: the + * notice is the filer's one hearing of it. */ -export function noticeDelivery(row: Task, requestId: string): { readonly by: string; readonly owed: boolean } | undefined { - if (row.status !== "completed" && row.status !== "errored" && row.status !== "cancelled") return undefined; +export function deliveredAsNotice(row: Task, requestId: string): boolean { + if (row.status !== "completed" && row.status !== "errored" && row.status !== "cancelled") return false; const by = takerOf(row, row.attempts, row.status); - if (by === undefined || by === requestId) return undefined; - const key = markerKey(row.attempts, row.status); - return { by, owed: isSendable(key, (row.metadata ?? {})[key]) }; + return by !== undefined && by !== requestId; } /** diff --git a/packages/orchestration/src/tasks/notice/task-settled-entry.ts b/packages/orchestration/src/tasks/notice/task-settled-entry.ts index cae6b6b2fa..4a33151325 100644 --- a/packages/orchestration/src/tasks/notice/task-settled-entry.ts +++ b/packages/orchestration/src/tasks/notice/task-settled-entry.ts @@ -20,10 +20,10 @@ * request that failed. Queued on the session by default, so a second copy of * one notice waits behind the first and finds it no longer owed. * - * While the request that filed the task still runs, its notice is turned away - * as `defer` turns one away (FIX-2123): a wait in that request reads the - * ending first and drops it, and the request sends what is still owed as it - * ends. + * While the request that filed the task is running (not while it is parked), + * its notice is turned away as `defer` turns one away (FIX-2123): a wait in + * that request reads the ending first and drops it, and the request sends + * what is still owed as it parks or ends. * * Woken with an empty input, by the wake a board's delivery owes when it * names this entry as its pass (the default): it pays what this session's @@ -92,6 +92,17 @@ async function hasEnded(ctx: BlockContext, requestId: string): Promise return (await ended([requestId]))[requestId] === true; } +/** + * Whether a request of this session is running its own work now, neither + * parked nor ended (FIX-2123), as the engine records it. A host that keeps no + * such record answers `false`: there no notice is held for it. + */ +async function isRunning(ctx: BlockContext, requestId: string): Promise { + const running = ctx.requestHost?.waits?.running; + if (running === undefined) return false; + return (await running([requestId]))[requestId] === true; +} + /** The running session's board a notice names, at this session's partition. */ async function boardOf(ctx: BlockContext, boardId: string) { return (await delegatingBoardsOf(ctx, boardId)).find((board) => board.ref.collectionId === boardId); @@ -133,24 +144,27 @@ export function taskSettledEntry(options: TaskSettledEntryOptions) { if (row === undefined || !isNoticeOwed(row, notice)) return { owed: false }; // Not now: left owed, for the pass to send again, with the time it was // turned away, so its age (which ends it as undeliverable) counts from - // then. Turned away while the request that filed the task runs too - // (FIX-2123): a wait in it reads the ending first, and that request - // sends what is still owed as it ends. + // then. Turned away while the request that filed the task is running + // too (FIX-2123), not while it is parked: a wait in it reads the ending + // first, and that request sends what is still owed as it parks or ends. const own = ctx.request.identity.id; const filing = row.filer?.requestId; const turnedAway = (options.defer !== undefined && (await options.defer(ctx, notice))) || - (filing !== undefined && filing !== own && !(await hasEnded(ctx, filing))); - if (turnedAway) { + (filing !== undefined && filing !== own && (await isRunning(ctx, filing))); + const notNow = async () => { await board!.ref.patchMetadata(row.id, deferredNoticePatch(row, notice, board!.ref.now())); return { owed: false }; - } + }; + if (turnedAway) return notNow(); // Taken in one conditional write before the block runs (FIX-2123), so a // wait's drop and this delivery can't both go through. Taken by a request - // still running: it is delivering it. Taken by one that ended without - // clearing it (its block failed): this copy takes it over. + // that hasn't ended, running or parked: it is delivering it, and this + // copy runs nothing (recorded as turned away, so the notice doesn't age + // out while it is delivered). Taken by one that ended without clearing + // it (its block failed): this copy takes it over. const taker = noticeTakenBy(row, notice); - if (taker !== undefined && taker !== own && !(await hasEnded(ctx, taker))) return { owed: false }; + if (taker !== undefined && taker !== own && !(await hasEnded(ctx, taker))) return notNow(); if (board!.ref.takeNotice !== undefined) { const took = await board!.ref.takeNotice(row.id, notice, { by: own, diff --git a/packages/orchestration/test/collection/notice-take.test.ts b/packages/orchestration/test/collection/notice-take.test.ts index 26949e1703..dfbadbc759 100644 --- a/packages/orchestration/test/collection/notice-take.test.ts +++ b/packages/orchestration/test/collection/notice-take.test.ts @@ -86,4 +86,15 @@ describe.each(BACKINGS)("taking a notice (%s)", (_name, makeBacking) => { await tasks.patchMetadata("t", { [KEY]: { attempt: 1, ending: "completed", due: 1000, held: true } }); expect(await tasks.takeNotice!("t", NOTICE, { by: "r2" })).toEqual({ outcome: "unchanged" }); }); + + it("files no notice record a filing's metadata names: a planted take can't keep a wait from dropping the notice", async () => { + const tasks = await makeBacking(); + await tasks.addTask({ + id: "t", + goal: "g", + assignee: "ops", + metadata: { note: "kept", "noticeTaken:1:completed": { by: "planted", at: 1 }, [KEY]: { attempt: 1, ending: "completed" } } + }); + expect((tasks.get("t") as Task).metadata).toEqual({ note: "kept" }); + }); }); diff --git a/packages/orchestration/test/tasks/notice-claim.test.ts b/packages/orchestration/test/tasks/notice-claim.test.ts index de6aba7a7a..17e1a6859f 100644 --- a/packages/orchestration/test/tasks/notice-claim.test.ts +++ b/packages/orchestration/test/tasks/notice-claim.test.ts @@ -26,7 +26,9 @@ import type { TaskSettledInput } from "../../src"; import { defineTaskCollection, type TaskCollectionRef, type TaskWorker } from "../../src/tasks"; import { taskBoard } from "../../src/task-board"; import { + askPerson, fileCall, + finishTask, finalAnswer, rows, runtimeFor, @@ -91,6 +93,26 @@ async function eventually(check: () => boolean, withinMs: number): Promise const settle = (ms: number) => new Promise((r) => setTimeout(r, ms)); +/** The person approves the turn parked on them, through the host's resume route. */ +async function approve(state: ReturnType, requestId: string): Promise { + const runtime = await state.getRuntime(); + const [suspension] = (await runtime.stores.suspensions.list({ sessionId: SESSION })).filter( + (s: { requestId: string; status: string }) => s.requestId === requestId && s.status === "pending" + ); + expect(suspension).toBeDefined(); + const router = await state.getRouter(); + const path = ["wait-board", "requests", requestId, "resume"]; + const sent = await router.POST( + new Request(`http://localhost/api/flows/${path.join("/")}`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ userId: USER, suspensionId: (suspension as { suspensionId: string }).suspensionId, action: "approve" }) + }), + { params: { path } } + ); + expect(sent.status).toBeLessThan(300); +} + describe.each(STORES)("a task's ending is heard once by its filer (%s)", { timeout: 30_000 }, (_name, stores) => { it("on the defaults, the filing turn's wait is the delivery: no onTaskSettled block runs beside it", async () => { // The turn action holds no line and the notice entry queues: the notice is @@ -132,6 +154,73 @@ describe.each(STORES)("a task's ending is heard once by its filer (%s)", { timeo } }); + it("a filing request parked on a person holds nothing: its task's notice is heard during the park, and the resumed wait on the task answers that it came as a notice", async () => { + const notices: TaskSettledInput[] = []; + const seen: GeneratorModelCallOptions[] = []; + let taskId = ""; + const { model } = stepModel( + [ + fileCall("f1"), + (opts) => { + taskId = waitedTaskId([opts], 0); + return askPerson("p1")(opts); + }, + waitAgain("w1", () => taskId), + finalAnswer + ], + seen + ); + const flow = waitFlow(model, { endsOnFiling: "already done", heard: notices }); + const state = runtimeFor(flow, await stores()); + try { + const turn = await startTurn(state); + await until(state, turn, "suspended", 10_000); + // While the person decides, the conversation hears the task that finished. + await eventually(() => notices.length === 1, 6_000); + expect(notices).toEqual([expect.objectContaining({ taskId, ending: "completed", output: "already done" })]); + + await approve(state, turn); + await until(state, turn, "completed", 10_000); + const answer = toolResults(seen[3]!.messages); + expect(answer).toContain('"arrivesAsNotice":true'); + expect(answer).toContain(taskId); + expect(answer).not.toContain("already done"); + await settle(500); + expect(notices).toHaveLength(1); + } finally { + await state.dispose(); + } + }); + + it("a request that runs again after its park sends the notice it held as it ends, not at the next resend", async () => { + const notices: TaskSettledInput[] = []; + let taskId = ""; + const { model } = stepModel([ + fileCall("f1"), + (opts) => { + taskId = waitedTaskId([opts], 0); + return askPerson("p1")(opts); + }, + // Back from the park, the turn runs while the task ends: its notice is held for it. + finishTask("x1", () => taskId, "found it"), + finalAnswer + ]); + const flow = waitFlow(model, { heard: notices }); + const state = runtimeFor(flow, await stores()); + try { + const turn = await startTurn(state); + await until(state, turn, "suspended", 10_000); + await approve(state, turn); + await until(state, turn, "completed", 10_000); + await eventually(() => notices.length > 0, 1_000); + expect(notices).toEqual([expect.objectContaining({ taskId, ending: "completed", output: "found it" })]); + await settle(2_500); + expect(notices).toHaveLength(1); + } finally { + await state.dispose(); + } + }); + it("a wait that finds the notice already running answers that the ending arrives as a notice, and the notice is heard once", async () => { // As in a Workforce session: the turn holds the line and the notice defers // on it, so the notice runs once the filing turn ends. A second turn @@ -165,7 +254,8 @@ describe.each(STORES)("a task's ending is heard once by its filer (%s)", { timeo await until(state, second, "completed", 10_000); const answer = toolResults(seen[3]!.messages); expect(answer).toContain(taskId); - expect(answer).toContain("arriving as a notice"); + expect(answer).toContain('"arrivesAsNotice":true'); + expect(answer).toContain("as a notice"); expect(answer).not.toContain("already done"); release(); diff --git a/packages/orchestration/test/tasks/notice-defer.test.ts b/packages/orchestration/test/tasks/notice-defer.test.ts index 40bfc69bf0..6a8d0b538d 100644 --- a/packages/orchestration/test/tasks/notice-defer.test.ts +++ b/packages/orchestration/test/tasks/notice-defer.test.ts @@ -4,6 +4,9 @@ * turned away (beside its marker), so its age (which ends it as undeliverable) * counts from then. * One it doesn't turn away runs and is acknowledged. + * + * And its take (FIX-2123): a notice whose block fails is taken over by the + * next copy, and one whose taker hasn't ended, running or parked, never is. */ import { DEFAULT_ORG_ID, defineFlow, handler } from "@flow-state-dev/core"; import type { BlockContext } from "@flow-state-dev/core/types"; @@ -22,7 +25,19 @@ async function boardOf(ctx: BlockContext): Promise { return getOrCreateTaskCollection({ ctx, backing: "resource", collectionId: BOARD, collection }); } -function host(defer: () => boolean, failFirst = false) { +/** How the flow's block handles a notice, besides recording it, and on what host. */ +interface HostOptions { + /** Throw on the first notice, before recording it. */ + readonly failFirst?: boolean; + /** After recording a notice: wait for this, holding the notice's request open. */ + readonly gate?: Promise; + /** After recording a notice: park its request on a person. */ + readonly park?: boolean; + /** A durable host, which records which requests are running and which have ended. */ + readonly durable?: boolean; +} + +function host(defer: () => boolean, options: HostOptions = {}) { const heard: TaskSettledInput[] = []; let failed = false; const seed = handler({ @@ -30,10 +45,17 @@ function host(defer: () => boolean, failFirst = false) { inputSchema: z.object({ id: z.string() }), outputSchema: z.any(), execute: async ({ id }, ctx) => { - const board = await boardOf(ctx); - await board.addTask({ id, goal: "Find the fact" }); - // The ending owes this session its notice, due now. - await board.patchMetadata(id, { "noticeOwed:1:completed": { attempt: 1, ending: "completed", due: Date.now() } }); + await (await boardOf(ctx)).addTask({ id, goal: "Find the fact" }); + return {}; + } + }); + const owe = handler({ + name: "owe", + inputSchema: z.object({ id: z.string() }), + outputSchema: z.any(), + execute: async ({ id }, ctx) => { + // The ending owes this session its notice, due now: written by a request that filed nothing. + await (await boardOf(ctx)).patchMetadata(id, { "noticeOwed:1:completed": { attempt: 1, ending: "completed", due: Date.now() } }); return {}; } }); @@ -46,7 +68,7 @@ function host(defer: () => boolean, failFirst = false) { const flow = defineFlow({ kind: "defer-host", resources: { [BOARD]: defineTaskCollection({ id: BOARD, scope: "session", delivery: {} }) }, - actions: { seed: { block: seed }, read: { block: read } }, + actions: { seed: { block: seed }, owe: { block: owe }, read: { block: read } }, internal: { actions: { onTaskSettled: taskSettledEntry({ @@ -54,12 +76,14 @@ function host(defer: () => boolean, failFirst = false) { name: "record-heard", inputSchema: z.any(), outputSchema: z.any(), - execute: (notice: TaskSettledInput) => { - if (failFirst && !failed) { + execute: async (notice: TaskSettledInput, ctx) => { + if (options.failFirst === true && !failed) { failed = true; throw new Error("the handling failed"); } heard.push(notice); + await options.gate; + if (options.park === true) await ctx.suspend!({ reason: "human_approval", message: "Tell the customer?" }); return {}; } }), @@ -68,8 +92,14 @@ function host(defer: () => boolean, failFirst = false) { } } })({ id: "defer-host" }); - const state = createFlowState({ flows: { "defer-host": flow }, stores: { default: { primary: inMemoryStores() } } }); + const state = createFlowState({ + flows: { "defer-host": flow }, + stores: { default: { primary: inMemoryStores() } }, + ...(options.durable === true ? { durable: true } : {}) + }); const act = async (actionName: string, input: unknown, internal = false) => { + // The router builds a durable host's request waits, which say what is running and what has ended. + if (options.durable === true) await state.getRouter(); const runtime = await state.getRuntime(); const result = await runAction({ orgId: DEFAULT_ORG_ID, @@ -85,7 +115,12 @@ function host(defer: () => boolean, failFirst = false) { if ((result as { error?: unknown }).error !== undefined) throw new Error(`${actionName}: ${JSON.stringify((result as { error?: unknown }).error)}`); return (result as { output?: unknown }).output; }; - return { heard, act, dispose: () => state.dispose() }; + /** File the task, then owe its notice from a request that filed nothing. */ + const owed = async (id: string) => { + await act("seed", { id }); + await act("owe", { id }); + }; + return { heard, act, owed, dispose: () => state.dispose() }; } const notice = { boardId: BOARD, taskId: "t1", attempt: 1, ending: "completed" as const, output: "fact" }; @@ -94,7 +129,7 @@ describe("a notice its session turns away (defer)", () => { it("runs nothing, stays owed, and records when it was turned away", async () => { const h = host(() => true); try { - await h.act("seed", { id: "t1" }); + await h.owed("t1"); const before = Date.now(); await h.act("onTaskSettled", notice, true); expect(h.heard).toEqual([]); @@ -109,7 +144,7 @@ describe("a notice its session turns away (defer)", () => { it("runs and is acknowledged when it isn't turned away", async () => { const h = host(() => false); try { - await h.act("seed", { id: "t1" }); + await h.owed("t1"); await h.act("onTaskSettled", notice, true); expect(h.heard).toEqual([expect.objectContaining({ taskId: "t1", ending: "completed" })]); const metadata = (await h.act("read", { id: "t1" })) as Record; @@ -122,9 +157,9 @@ describe("a notice its session turns away (defer)", () => { describe("a notice whose block fails (FIX-2123)", () => { it("stays owed, taken by the request that failed, and the next copy takes it over and is heard once", async () => { - const h = host(() => false, true); + const h = host(() => false, { failFirst: true, durable: true }); try { - await h.act("seed", { id: "t1" }); + await h.owed("t1"); await expect(h.act("onTaskSettled", notice, true)).rejects.toThrow(/execution_error/); expect(h.heard).toEqual([]); const failed = (await h.act("read", { id: "t1" })) as Record; @@ -144,3 +179,37 @@ describe("a notice whose block fails (FIX-2123)", () => { } }); }); + +describe("a notice taken by a request that hasn't ended (FIX-2123)", () => { + it.each([ + ["still running its block", false], + ["parked on a person in its block", true] + ] as const)("is not taken over by a second copy while that request is %s: the block runs once", async (_name, park) => { + let open!: () => void; + const gate = new Promise((resolve) => (open = resolve)); + const h = host(() => false, park ? { durable: true, park: true } : { durable: true, gate }); + try { + await h.owed("t1"); + const first = h.act("onTaskSettled", notice, true).catch(() => undefined); + if (park) await first; + for (let i = 0; i < 300 && h.heard.length === 0; i += 1) await new Promise((r) => setTimeout(r, 10)); + expect(h.heard).toHaveLength(1); + + // The filer's pass sends it again while the first copy is delivering it. + const before = Date.now(); + expect(await h.act("onTaskSettled", notice, true)).toEqual({ owed: false }); + expect(h.heard).toHaveLength(1); + const metadata = (await h.act("read", { id: "t1" })) as Record; + // Still owed, and recorded as turned away, so it doesn't age out while it is delivered. + expect(metadata["noticeOwed:1:completed"]).toMatchObject({ attempt: 1, ending: "completed" }); + expect(metadata["noticeDeferred:1:completed"]!.at).toBeGreaterThanOrEqual(before); + + open(); + await first; + expect(h.heard).toHaveLength(1); + } finally { + open(); + await h.dispose(); + } + }); +}); diff --git a/packages/orchestration/test/tasks/wait-fixture.ts b/packages/orchestration/test/tasks/wait-fixture.ts index e642b9bec2..39350206ec 100644 --- a/packages/orchestration/test/tasks/wait-fixture.ts +++ b/packages/orchestration/test/tasks/wait-fixture.ts @@ -76,6 +76,18 @@ export const fileCall = (id: string, question = "Is ACME's SOC 2 current?"): Ste finishReason: "tool-calls" }); +/** A step that parks the turn on a person, until they approve. */ +export const askPerson = (id: string): StepFn => (_options) => ({ + toolCalls: [{ toolCallId: id, toolName: "ask_person", args: {} }], + finishReason: "tool-calls" +}); + +/** A step in which the colleague finishes a task already filed, by id, while this turn runs. */ +export const finishTask = (callId: string, id: () => string, output: unknown): StepFn => (_options) => ({ + toolCalls: [{ toolCallId: callId, toolName: "finish_task", args: { id: id(), output } }], + finishReason: "tool-calls" +}); + /** A step that reads a task without waiting, as a plain `getTask({ id })` does. */ export const readTask = (callId: string, id: () => string): StepFn => (_options) => ({ toolCalls: [{ toolCallId: callId, toolName: "read_task", args: { id: id() } }], @@ -230,6 +242,26 @@ export function waitFlow( return waited.ok ? shown(waited.task) : { ok: false, taskId: input.id, error: `${waited.error}: ${waited.message}` }; } }); + const askPersonTool = handler({ + name: "ask_person", + inputSchema: z.object({}).passthrough(), + outputSchema: z.any(), + execute: async (_input, ctx) => ({ answer: await ctx.suspend!({ reason: "human_approval", message: "Go ahead?" }) }) + }); + const finishTaskTool = handler({ + name: "finish_task", + inputSchema: z.object({ id: z.string(), output: z.unknown() }), + outputSchema: z.any(), + execute: async (input, ctx) => { + const board = await boardOf(ctx); + await ctx.runOnce!(`fixture.finish:${toolCallOf(ctx)!.logicalId}`, async () => { + await board.claim("researcher-worker"); + await board.complete(input.id, input.output); + return true; + }); + return { ok: true }; + } + }); const fileTool = handler({ name: "file_task", inputSchema: z.object({ question: z.string() }), @@ -296,7 +328,7 @@ export function waitFlow( actions: { run: { block: sequencer({ name: "turn" }).step( - generator({ name: "agent", model, prompt: "p", tools: [wait, waitAgainTool, readTaskTool, fileTool] }) + generator({ name: "agent", model, prompt: "p", tools: [wait, waitAgainTool, readTaskTool, fileTool, askPersonTool, finishTaskTool] }) ), inputSchema: z.object({}).passthrough(), ...(options.noticeAfterTurn === true ? { concurrency: "hold" as const } : {}) diff --git a/packages/workforce/test/conversation-board-wait.test.ts b/packages/workforce/test/conversation-board-wait.test.ts index a1e3df55f2..c5f564d2e1 100644 --- a/packages/workforce/test/conversation-board-wait.test.ts +++ b/packages/workforce/test/conversation-board-wait.test.ts @@ -412,7 +412,8 @@ describe("a wait beside the notice turn of the task it waits on (FIX-2123)", { t const second = await host.post("alice", conv, "respond", { message: "check again" }); expect(await until(async () => (await statusOf(host, second)) === "completed", 10_000)).toBe(true); expect(secondAnswer).toContain(filed!); - expect(secondAnswer).toContain("arriving as a notice"); + expect(secondAnswer).toContain('"arrivesAsNotice":true'); + expect(secondAnswer).toContain("as a notice"); expect(secondAnswer).not.toContain("eng.tasker did"); openNotice(); From 697bcf995a1f3482d5d94688ffc1b709847bb62b Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 16:05:48 +0000 Subject: [PATCH 5/8] test(orchestration): name the person-park step awaitApproval, not a retired ask word (FIX-2123) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A --- packages/orchestration/test/tasks/notice-claim.test.ts | 6 +++--- packages/orchestration/test/tasks/wait-fixture.ts | 10 +++++----- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/packages/orchestration/test/tasks/notice-claim.test.ts b/packages/orchestration/test/tasks/notice-claim.test.ts index 17e1a6859f..59a1212410 100644 --- a/packages/orchestration/test/tasks/notice-claim.test.ts +++ b/packages/orchestration/test/tasks/notice-claim.test.ts @@ -26,7 +26,7 @@ import type { TaskSettledInput } from "../../src"; import { defineTaskCollection, type TaskCollectionRef, type TaskWorker } from "../../src/tasks"; import { taskBoard } from "../../src/task-board"; import { - askPerson, + awaitApproval, fileCall, finishTask, finalAnswer, @@ -163,7 +163,7 @@ describe.each(STORES)("a task's ending is heard once by its filer (%s)", { timeo fileCall("f1"), (opts) => { taskId = waitedTaskId([opts], 0); - return askPerson("p1")(opts); + return awaitApproval("p1")(opts); }, waitAgain("w1", () => taskId), finalAnswer @@ -199,7 +199,7 @@ describe.each(STORES)("a task's ending is heard once by its filer (%s)", { timeo fileCall("f1"), (opts) => { taskId = waitedTaskId([opts], 0); - return askPerson("p1")(opts); + return awaitApproval("p1")(opts); }, // Back from the park, the turn runs while the task ends: its notice is held for it. finishTask("x1", () => taskId, "found it"), diff --git a/packages/orchestration/test/tasks/wait-fixture.ts b/packages/orchestration/test/tasks/wait-fixture.ts index 39350206ec..eaf1989b2d 100644 --- a/packages/orchestration/test/tasks/wait-fixture.ts +++ b/packages/orchestration/test/tasks/wait-fixture.ts @@ -77,8 +77,8 @@ export const fileCall = (id: string, question = "Is ACME's SOC 2 current?"): Ste }); /** A step that parks the turn on a person, until they approve. */ -export const askPerson = (id: string): StepFn => (_options) => ({ - toolCalls: [{ toolCallId: id, toolName: "ask_person", args: {} }], +export const awaitApproval = (id: string): StepFn => (_options) => ({ + toolCalls: [{ toolCallId: id, toolName: "await_approval", args: {} }], finishReason: "tool-calls" }); @@ -242,8 +242,8 @@ export function waitFlow( return waited.ok ? shown(waited.task) : { ok: false, taskId: input.id, error: `${waited.error}: ${waited.message}` }; } }); - const askPersonTool = handler({ - name: "ask_person", + const approvalTool = handler({ + name: "await_approval", inputSchema: z.object({}).passthrough(), outputSchema: z.any(), execute: async (_input, ctx) => ({ answer: await ctx.suspend!({ reason: "human_approval", message: "Go ahead?" }) }) @@ -328,7 +328,7 @@ export function waitFlow( actions: { run: { block: sequencer({ name: "turn" }).step( - generator({ name: "agent", model, prompt: "p", tools: [wait, waitAgainTool, readTaskTool, fileTool, askPersonTool, finishTaskTool] }) + generator({ name: "agent", model, prompt: "p", tools: [wait, waitAgainTool, readTaskTool, fileTool, approvalTool, finishTaskTool] }) ), inputSchema: z.object({}).passthrough(), ...(options.noticeAfterTurn === true ? { concurrency: "hold" as const } : {}) From 4d9568b08c7ecf498c8e708d4f344cef45d454cf Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 16:10:47 +0000 Subject: [PATCH 6/8] docs(orchestration): a parked filing request holds no notice; arrivesAsNotice after the notice starts (FIX-2123) Correct "Hearing how a task ended" and "Waiting on a task": a notice waits only while the filing request is running and goes out during a park; any wait after an onTaskSettled request for the ending has started gets arrivesAsNotice, whether that request is still running or finished. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A --- apps/docs/docs/orchestration/task-board.md | 38 ++++++++++++---------- 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/apps/docs/docs/orchestration/task-board.md b/apps/docs/docs/orchestration/task-board.md index 487758acdd..29aa676534 100644 --- a/apps/docs/docs/orchestration/task-board.md +++ b/apps/docs/docs/orchestration/task-board.md @@ -796,18 +796,18 @@ When a task the board handed out ends, the session that filed it is told, once. | `question` | On `parked`: what the worker asked. Answer it with [`answerTask`](#when-a-task-stops-on-a-question) | | `cancelled` | On `errored` or `cancelled`: the ids of the dependents this ending [cancelled](#when-a-dependency-fails) that the same session filed. Those dependents send no notice of their own | -A notice is held until the request that filed the task has ended, including while that request is paused on a [`getTask` wait](#waiting-on-a-task), and it's sent as that request ends. A turn that files a task and never waits on it hears the ending once that turn ends. +While the request that filed the task is still running, its notice waits and goes out when that request parks or ends. While the request is parked, on a person's approval or input or on a [`getTask` wait](#waiting-on-a-task), the notice goes out right away. So a turn that files a task and never waits on it hears the ending once that turn ends or parks. Waiting for the request needs a host with durable execution; elsewhere a notice is sent as soon as the task ends. A task that fails and is retried sends nothing until its last attempt ends. These endings send no notice either: -- A task the filer [waited on](#waiting-on-a-task) with `getTask({ id, wait: true })`, either while it ended or after it ended but before an `onTaskSettled` request started handling its notice. The waiting call gets the result instead. Once a request has started handling the notice, the ending comes from the notice, and a wait while that request is in progress gets [`arrivesAsNotice`](#waiting-on-a-task). An ending after the wait's deadline goes to the waiting request's next `getTask` while it runs, and comes here only if that request ends without reading it. +- A task the filer [waited on](#waiting-on-a-task) with `getTask({ id, wait: true })`, while it ran or after it ended but before an `onTaskSettled` request for that ending had started. The waiting call gets the result instead. - A cancel the filer's own model made with its `updateTask` tool (`status: "cancelled"`), in its own turn. The tool's result already told it, and names the dependents the cancel took along. A cancel from anywhere else does send one: a person, an app calling the board's actions, or a cascade from a task another session filed. -Your block runs once per ending. If it throws, the notice stays owed and is sent again, so the same ending can reach the block a second time after a failure. The entry queues on the session by default, so two notices run one after the other; pass `concurrency` to `taskSettledEntry` to change that. +Your block runs once per ending. If it throws, the notice stays owed and is sent again, so the same ending can reach the block a second time after a failure. A second copy never runs while the first copy's request is still running or parked. The entry queues on the session by default, so two notices run one after the other; pass `concurrency` to `taskSettledEntry` to change that. -A notice that can't be sent when the task ends, because the process died first or the send failed, is sent about two seconds later, or as soon as the server is running again if it was down then. The [durability sweeper](../advanced/durable-execution.md#retention-and-cleanup) runs `onTaskSettled` in the filing session with an empty input, which sends whatever the session's boards still owe. You may see an extra `onTaskSettled` request with empty input in the session; your block isn't called for it. This needs a host with durable execution (`durable: true` on `createFlowState`) and a store that keeps wakes, which every built-in store does. Without them, a notice is sent once, when the task ends, and a lost one isn't sent again. +A notice that can't be sent when it falls due, because the process died first or the send failed, is sent about two seconds later, or as soon as the server is running again if it was down then. The [durability sweeper](../advanced/durable-execution.md#retention-and-cleanup) runs `onTaskSettled` in the filing session with an empty input, which sends whatever the session's boards still owe. You may see an extra `onTaskSettled` request with empty input in the session; your block isn't called for it. This needs a host with durable execution (`durable: true` on `createFlowState`) and a store that keeps wakes, which every built-in store does. Without them, a notice is sent once, when the task ends, and a lost one isn't sent again. If the filing session no longer exists, the notice can't be delivered. The task records that in its `metadata`, under `noticeUndeliverable::`, as `{ at, detail }`. A notice still owed a day after it fell due is given up the same way; the host's [`maxDebtAgeMs`](../advanced/durable-execution.md#retention-and-cleanup) sets that limit. @@ -847,29 +847,31 @@ getTask({ id: "task_8f2", wait: true }) You can wait in the same turn or in a later one. Your turn pauses until the task ends, then carries on with the task as `getTask` always returns it: its status, and its output once it -completed. Nothing holds a request open while it waits, so a server restart loses nothing. -A task that already ended is returned at once. If no `onTaskSettled` request has started -handling that ending's notice yet, the wait takes the ending and your block isn't called for it. -A wait after your block has handled the notice returns the task the same way. +completed. Nothing keeps a request open while it waits, so a server restart loses nothing. +A task that already ended is returned at once. If no `onTaskSettled` request for that ending +has started yet, the wait returns the ending, and no notice is sent for it. -If an `onTaskSettled` request has started handling the notice and hasn't finished, for example -a second turn waits on the task while that request is in progress, `getTask` returns this -instead of the task: +Once an `onTaskSettled` request for that ending has started, whether it is still running or +has finished, `getTask` returns this instead of the task: ```ts { ok: true, taskId: "task_8f2", arrivesAsNotice: true, message: "…" } ``` -`message` names the task, says its ending is arriving as a notice, and tells the model not to -wait on it again. -The session hears that ending once, from the notice. +`message` names the task, says its ending comes to the conversation as a notice, and tells the +model not to wait on it again. The session hears that ending once, from the notice. + +You'll see it when your turn parks between filing a task and waiting on it, or when a second +turn waits on a task the first one filed. Say a turn files a task, then waits for a person's +approval. The task finishes during that wait, and the conversation hears it from +`onTaskSettled`. After the approval, the same turn waits on the task and gets `arrivesAsNotice`. A wait has a deadline, five minutes unless you pass `deadline` in milliseconds. At the deadline your turn carries on with the task as it stands, usually still `in_progress`. Nothing is -cancelled: the task runs on, and you can wait on it again. If it ends before your request -does, even while your request waits on something else, your next `getTask` on it returns the -ending. If your request ends without reading it, you [hear about it](#hearing-how-a-task-ended) -once, the usual way. A worker you're +cancelled: the task runs on, and you can wait on it again. If it ends while your request is +running, your next `getTask` on it returns the ending. If it ends while your request is parked +on something else, or your request ends without reading it, you +[hear about it](#hearing-how-a-task-ended) once, the usual way. A worker you're waiting on can't stop to ask you a question; it answers with what it has, or fails. One wait per step, where a step is one model call and the tool calls it makes. From 0652a1502b81cd6ec808b8b82d189f51e84e3556 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 16:34:07 +0000 Subject: [PATCH 7/8] fix(orchestration): no board I/O at a park; the pass sends what a parked request held (FIX-2123) Review 5631310696 on #3733. The request-end hook ran at a park too, doing a list, a refresh per filed row and a pass wake. Engine hooks run before the record is written suspended, so that time widened a window in which a task-bound wait's resume is refused already-resolved and its resumeOwed marker cleared: the turn stalled until its deadline. The notices === false guard only hid it on plan boards. Now the hook does nothing at a park: a parked request holds no notice, so the filer's pass sends what it held at the notice's next send. The guard is gone; worker-plans-its-steps and the FIX-2082 waits pass without it, and still pass with 100 ms added to the hook after the park return. Unit test: whenRequestEnds runs nothing at a park and sends once the request ends (red without the park return); once per request and board; nothing on a host that doesn't record running requests. The core/engine changeset says a host with its own RequestWaits should add `running`. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A --- .changeset/a-notice-is-taken-once.md | 2 +- .changeset/request-waits-running.md | 2 +- packages/orchestration/README.md | 5 ++- .../src/tasks/collection/resource-backed.ts | 24 +++++------ .../src/tasks/notice/delivery.ts | 30 ++++++++------ .../src/tasks/notice/task-settled-entry.ts | 4 +- .../test/tasks/delivery-after-ending.test.ts | 40 +++++++++++++++++++ 7 files changed, 73 insertions(+), 34 deletions(-) diff --git a/.changeset/a-notice-is-taken-once.md b/.changeset/a-notice-is-taken-once.md index 6b7b8c6ec9..4eeff2e63f 100644 --- a/.changeset/a-notice-is-taken-once.md +++ b/.changeset/a-notice-is-taken-once.md @@ -2,4 +2,4 @@ "@flow-state-dev/orchestration": patch --- -A task's ending reaches the session that filed it once, on the defaults too: a `getTask({ id, wait: true })` that reads a task already ended no longer races that task's `onTaskSettled`. A notice now waits while the request that filed its task is running (not while it is parked on a person or a wait), and that request sends it as it parks or ends. `taskSettledEntry` takes a notice in one conditional write before it runs your block, and a copy never takes a notice from a request that hasn't ended. A wait that finds the notice taken returns `{ ok: true, taskId, arrivesAsNotice: true, message }` instead of the task. Task collection refs gain an optional `takeNotice(id, notice, take)`, which both built-in backings implement; a ref without it admits notices as before. A filing's `metadata` can no longer carry notice records (FIX-2123). +A task's ending reaches the session that filed it once, on the defaults too: a `getTask({ id, wait: true })` that reads a task already ended no longer races that task's `onTaskSettled`. A notice now waits while the request that filed its task is running (not while it is parked on a person or a wait); that request sends what it held as it ends, and once it parks the session's pass sends it. `taskSettledEntry` takes a notice in one conditional write before it runs your block, and a copy never takes a notice from a request that hasn't ended. A wait that finds the notice taken returns `{ ok: true, taskId, arrivesAsNotice: true, message }` instead of the task. Task collection refs gain an optional `takeNotice(id, notice, take)`, which both built-in backings implement; a ref without it admits notices as before. A filing's `metadata` can no longer carry notice records (FIX-2123). diff --git a/.changeset/request-waits-running.md b/.changeset/request-waits-running.md index 1ed9c31a1a..9bc6fa1fe4 100644 --- a/.changeset/request-waits-running.md +++ b/.changeset/request-waits-running.md @@ -3,4 +3,4 @@ "@flow-state-dev/engine": patch --- -`RequestWaits` gains an optional `running(requestIds)`: whether each request of the running session is running its own work now (`in_progress`), as opposed to parked (`suspended`) or ended. The engine's request host implements it (FIX-2123). +`RequestWaits` gains an optional `running(requestIds)`: whether each request of the running session is running its own work now (`in_progress`), as opposed to parked (`suspended`) or ended. The engine's request host implements it. A host that supplies its own `RequestWaits` should add `running` too: without it, a task board doesn't hold a task's notice while the request that filed the task runs (FIX-2123). diff --git a/packages/orchestration/README.md b/packages/orchestration/README.md index 4f5f53d3d7..66e1f20629 100644 --- a/packages/orchestration/README.md +++ b/packages/orchestration/README.md @@ -508,8 +508,9 @@ task tool in its turn. The block runs once per ending; if it throws, the notice sent again. The entry queues on the session unless `concurrency` says otherwise. A notice waits for the request that filed its task while that request is running, not while it is -parked on a person or a wait. The entry holds the notice back the way `defer` does (below), and the -request sends it as it parks or ends. A wait in the filing request therefore reads the ending +parked on a person or a wait. The entry holds the notice back the way `defer` does (below). The +request sends what it held as it ends; once it parks, the session's pass sends it at the notice's +next resend. A wait in the filing request therefore reads the ending first. Before it runs `block`, the entry takes the notice in one conditional write (the board's optional `takeNotice`, which both built-in backings implement), so a wait that reads the task and the notice never both deliver the same ending. A notice whose `block` threw stays owed, and the diff --git a/packages/orchestration/src/tasks/collection/resource-backed.ts b/packages/orchestration/src/tasks/collection/resource-backed.ts index 5e99034ff1..166585ee0b 100644 --- a/packages/orchestration/src/tasks/collection/resource-backed.ts +++ b/packages/orchestration/src/tasks/collection/resource-backed.ts @@ -729,18 +729,16 @@ export async function createResourceBackedTaskCollection { + if (delivery === undefined || own === undefined || row.filer?.requestId !== own) return; + delivery.hooks.whenRequestEnds?.(async () => { await adoptUnknownRows(mirror, options.collection); for (const filed of listAll() as Task[]) { if (filed.filer?.requestId !== own) continue; @@ -749,11 +747,7 @@ export async function createResourceBackedTaskCollection 0) await ref.patchMetadata(current.id, sooner); - // At a park the request still reads as running until its hooks have - // run, so a notice sent now would be turned away again: the filer's - // pass is owed shortly instead, and sends it once the park is recorded. - if (parked) await delivery.hooks.owePassAt(current, now() + NOTICE_DUE_MS); - else await deliverAfter(lookupRow(current.id) ?? current); + await deliverAfter(lookupRow(current.id) ?? current); } }); } @@ -856,7 +850,7 @@ export async function createResourceBackedTaskCollection row.filer?.requestId === delivery.ownRequest); if (own !== undefined) noteFiled(own as Task); diff --git a/packages/orchestration/src/tasks/notice/delivery.ts b/packages/orchestration/src/tasks/notice/delivery.ts index dd13c13961..8f38ca2144 100644 --- a/packages/orchestration/src/tasks/notice/delivery.ts +++ b/packages/orchestration/src/tasks/notice/delivery.ts @@ -105,16 +105,16 @@ export interface DeliveryHooks { patchMetadata: (id: string, patch: Record) => Promise ): Promise; /** - * Run `send` once the running request stops running (FIX-2123), on the - * engine's request-end seam: as it ends, or as it parks, since a parked - * request holds no notice; `send` is told which (`parked`). A failure is - * logged, never thrown; what `send` - * didn't send stays owed for the filer's pass. Nothing runs on a context - * with no seam, or on a host that doesn't record which requests are - * running. Optional, so hooks built by hand keep compiling: without it, - * the pass sends instead. + * Run `send` once the running request has ended (FIX-2123), on the + * engine's request-end seam. Nothing runs when it parks: a parked request + * holds no notice, so the filer's pass sends what it held, and a hook at a + * park would delay the request's park record. A failure is logged, never + * thrown; what `send` didn't send stays owed for the filer's pass. Nothing + * runs on a context with no seam, or on a host that doesn't record which + * requests are running. Optional, so hooks built by hand keep compiling: + * without it, the pass sends instead. */ - whenRequestEnds?(send: (parked: boolean) => Promise): void; + whenRequestEnds?(send: () => Promise): void; } /** @@ -181,7 +181,7 @@ export function createDelivery( whenRequestEnds }; - function whenRequestEnds(send: (parked: boolean) => Promise): void { + function whenRequestEnds(send: () => Promise): void { // Only where a notice can be held for this request: a host that records // which requests are running. Anywhere else nothing is held for it, and a // hook would be owed for nothing: a context built outside a request's @@ -197,11 +197,15 @@ export function createDelivery( if (registered.has(boardKey)) return; registered.add(boardKey); sendsAtStop.set(seam, registered); - // At a park too: a parked request holds no notice. The request that runs - // again after it registers anew (the board notes the rows it filed). onRequestEnded(ctx, async (ending) => { + // Nothing at a park: a hook runs there before the request is recorded + // parked, and any time it takes delays that record, which a wait's + // resume reads. A parked request holds no notice, so the filer's pass + // sends what it held at the notice's next send. The request that runs + // again after the park registers anew (the board notes the rows it filed). + if (ending.status === "suspended") return; try { - await send(ending.status === "suspended"); + await send(); } catch (error) { console.warn("[orchestration] a notice held for the request that filed its task wasn't sent as it ended: the filer's pass sends it", { boardId: board.collectionId, diff --git a/packages/orchestration/src/tasks/notice/task-settled-entry.ts b/packages/orchestration/src/tasks/notice/task-settled-entry.ts index 4a33151325..e03ccc6c94 100644 --- a/packages/orchestration/src/tasks/notice/task-settled-entry.ts +++ b/packages/orchestration/src/tasks/notice/task-settled-entry.ts @@ -23,7 +23,7 @@ * While the request that filed the task is running (not while it is parked), * its notice is turned away as `defer` turns one away (FIX-2123): a wait in * that request reads the ending first and drops it, and the request sends - * what is still owed as it parks or ends. + * what is still owed as it ends; at a park, the filer's pass sends it. * * Woken with an empty input, by the wake a board's delivery owes when it * names this entry as its pass (the default): it pays what this session's @@ -146,7 +146,7 @@ export function taskSettledEntry(options: TaskSettledEntryOptions) { // turned away, so its age (which ends it as undeliverable) counts from // then. Turned away while the request that filed the task is running // too (FIX-2123), not while it is parked: a wait in it reads the ending - // first, and that request sends what is still owed as it parks or ends. + // first, and that request sends what is still owed as it ends (at a park, the pass does). const own = ctx.request.identity.id; const filing = row.filer?.requestId; const turnedAway = diff --git a/packages/orchestration/test/tasks/delivery-after-ending.test.ts b/packages/orchestration/test/tasks/delivery-after-ending.test.ts index 69adafe2e0..08bee548ad 100644 --- a/packages/orchestration/test/tasks/delivery-after-ending.test.ts +++ b/packages/orchestration/test/tasks/delivery-after-ending.test.ts @@ -5,6 +5,7 @@ */ import { afterEach, describe, expect, it, vi } from "vitest"; import type { BlockContext } from "@flow-state-dev/core"; +import { REQUEST_END_SEAM } from "@flow-state-dev/core/types"; import { createDelivery } from "../../src/tasks/notice/delivery"; import type { Task } from "../../src/tasks/schema/task"; @@ -32,3 +33,42 @@ describe("afterEnding", () => { ); }); }); + +describe("whenRequestEnds (FIX-2123)", () => { + /** A request's context whose request-end seam keeps the hooks it is given, on a host that records what is running. */ + function requestContext() { + const hooks: Array<(ending: { status: string }) => Promise | unknown> = []; + const ctx = { + [REQUEST_END_SEAM]: { add: (hook: (ending: { status: string }) => unknown) => hooks.push(hook) }, + requestHost: { waits: { running: async () => ({}) } } + } as unknown as BlockContext; + return { ctx, hooks }; + } + + it("runs nothing at a park, where a hook would hold back the request's park record, and sends once the request ends", async () => { + const { ctx, hooks } = requestContext(); + const send = vi.fn(async () => {}); + createDelivery(ctx, {}, { collectionId: "desk" }).whenRequestEnds!(send); + expect(hooks).toHaveLength(1); + + await hooks[0]!({ status: "suspended" }); + expect(send).not.toHaveBeenCalled(); + await hooks[0]!({ status: "completed" }); + expect(send).toHaveBeenCalledTimes(1); + }); + + it("registers once per request and board, however often the board is resolved", () => { + const { ctx, hooks } = requestContext(); + createDelivery(ctx, {}, { collectionId: "desk" }).whenRequestEnds!(async () => {}); + createDelivery(ctx, {}, { collectionId: "desk" }).whenRequestEnds!(async () => {}); + createDelivery(ctx, {}, { collectionId: "other" }).whenRequestEnds!(async () => {}); + expect(hooks).toHaveLength(2); + }); + + it("registers nothing on a host that doesn't record which requests are running", () => { + const { ctx, hooks } = requestContext(); + (ctx as unknown as { requestHost: unknown }).requestHost = {}; + createDelivery(ctx, {}, { collectionId: "desk" }).whenRequestEnds!(async () => {}); + expect(hooks).toHaveLength(0); + }); +}); From 69831f2df3758ba91a7193b70bea62d9752b45fb Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 16:41:03 +0000 Subject: [PATCH 8/8] docs(orchestration): a parked request holds no notice; a deadline-passed wait's ending goes to the next getTask (FIX-2123) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A --- apps/docs/docs/orchestration/task-board.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/apps/docs/docs/orchestration/task-board.md b/apps/docs/docs/orchestration/task-board.md index 29aa676534..40b8ff4f10 100644 --- a/apps/docs/docs/orchestration/task-board.md +++ b/apps/docs/docs/orchestration/task-board.md @@ -796,11 +796,11 @@ When a task the board handed out ends, the session that filed it is told, once. | `question` | On `parked`: what the worker asked. Answer it with [`answerTask`](#when-a-task-stops-on-a-question) | | `cancelled` | On `errored` or `cancelled`: the ids of the dependents this ending [cancelled](#when-a-dependency-fails) that the same session filed. Those dependents send no notice of their own | -While the request that filed the task is still running, its notice waits and goes out when that request parks or ends. While the request is parked, on a person's approval or input or on a [`getTask` wait](#waiting-on-a-task), the notice goes out right away. So a turn that files a task and never waits on it hears the ending once that turn ends or parks. Waiting for the request needs a host with durable execution; elsewhere a notice is sent as soon as the task ends. +While the request that filed the task is still running, its notice is held. When that request ends, it sends what it held right away. When it parks, on a person's approval or input or on a [`getTask` wait](#waiting-on-a-task), it holds nothing more. A task that ends during the park sends its notice right away. A notice held while the request ran goes out the next time the board tries to send it again: seconds after the park, or up to 5 minutes later if it had already been held a while (those tries back off). A task whose `getTask` wait reached its deadline is the exception: its ending goes to the request's next `getTask`, even during a park. See [Waiting on a task](#waiting-on-a-task). So a turn that files a task and never waits on it hears the ending once that turn ends, or soon after it parks. Holding needs a host with durable execution; elsewhere a notice is sent as soon as the task ends. A task that fails and is retried sends nothing until its last attempt ends. These endings send no notice either: -- A task the filer [waited on](#waiting-on-a-task) with `getTask({ id, wait: true })`, while it ran or after it ended but before an `onTaskSettled` request for that ending had started. The waiting call gets the result instead. +- A task the filer [waited on](#waiting-on-a-task) with `getTask({ id, wait: true })`, either a wait that was open when the task ended, or one started after it ended but before an `onTaskSettled` request for that ending had started. The `getTask` call gets the result instead. A wait that reached its deadline first doesn't count: that ending goes to the request's next `getTask`, and is sent as a notice only if the request ends without reading it. - A cancel the filer's own model made with its `updateTask` tool (`status: "cancelled"`), in its own turn. The tool's result already told it, and names the dependents the cancel took along. A cancel from anywhere else does send one: a person, an app calling the board's actions, or a cascade from a task another session filed. @@ -869,8 +869,8 @@ approval. The task finishes during that wait, and the conversation hears it from A wait has a deadline, five minutes unless you pass `deadline` in milliseconds. At the deadline your turn carries on with the task as it stands, usually still `in_progress`. Nothing is cancelled: the task runs on, and you can wait on it again. If it ends while your request is -running, your next `getTask` on it returns the ending. If it ends while your request is parked -on something else, or your request ends without reading it, you +running, or while it is parked on something else, your request's next `getTask` on it returns +the ending. Only if your request ends without reading it do you [hear about it](#hearing-how-a-task-ended) once, the usual way. A worker you're waiting on can't stop to ask you a question; it answers with what it has, or fails. One wait per step, where a step is one model call and the tool calls it makes.