Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/a-notice-is-taken-once.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@flow-state-dev/orchestration": patch
---

A task's ending reaches the session that filed it once, on the defaults too: a `getTask({ id, wait: true })` that reads a task already ended no longer races that task's `onTaskSettled`. A notice now waits until the request that filed its task has ended (that request sends it as it ends), and `taskSettledEntry` takes a notice in one conditional write before it runs your block. A wait that finds the notice already taken returns `{ ok: true, taskId, arrivesAsNotice: true, message }` instead of the task. Task collection refs gain an optional `takeNotice(id, notice, take)`, which both built-in backings implement; a ref without it admits notices as before (FIX-2123).
22 changes: 18 additions & 4 deletions apps/docs/docs/orchestration/task-board.md
Original file line number Diff line number Diff line change
Expand Up @@ -796,9 +796,11 @@ When a task the board handed out ends, the session that filed it is told, once.
| `question` | On `parked`: what the worker asked. Answer it with [`answerTask`](#when-a-task-stops-on-a-question) |
| `cancelled` | On `errored` or `cancelled`: the ids of the dependents this ending [cancelled](#when-a-dependency-fails) that the same session filed. Those dependents send no notice of their own |

A notice is held until the request that filed the task has ended, including while that request is paused on a [`getTask` wait](#waiting-on-a-task), and it's sent as that request ends. A turn that files a task and never waits on it hears the ending once that turn ends.

A task that fails and is retried sends nothing until its last attempt ends. These endings send no notice either:

- A task the filer [waited on](#waiting-on-a-task) with `getTask({ id, wait: true })` while it ended. The waiting call gets the result instead. An ending after the wait's deadline goes to the waiting request's next `getTask` while it runs, and comes here only if that request ends without reading it.
- A task the filer [waited on](#waiting-on-a-task) with `getTask({ id, wait: true })`, either while it ended or after it ended but before an `onTaskSettled` request started handling its notice. The waiting call gets the result instead. Once a request has started handling the notice, the ending comes from the notice, and a wait while that request is in progress gets [`arrivesAsNotice`](#waiting-on-a-task). An ending after the wait's deadline goes to the waiting request's next `getTask` while it runs, and comes here only if that request ends without reading it.
- A cancel the filer's own model made with its `updateTask` tool (`status: "cancelled"`), in its own turn. The tool's result already told it, and names the dependents the cancel took along.

A cancel from anywhere else does send one: a person, an app calling the board's actions, or a cascade from a task another session filed.
Expand Down Expand Up @@ -846,9 +848,21 @@ getTask({ id: "task_8f2", wait: true })
You can wait in the same turn or in a later one. Your turn pauses until the task ends, then
carries on with the task as `getTask` always returns it: its status, and its output once it
completed. Nothing holds a request open while it waits, so a server restart loses nothing.
A task that already ended is returned at once, and no `onTaskSettled` follows for it, as long
as your `onTaskSettled` waits behind the turn that filed the task. A Workforce conversation or
task turn works that way. The request still runs, but your block isn't called.
A task that already ended is returned at once. If no `onTaskSettled` request has started
handling that ending's notice yet, the wait takes the ending and your block isn't called for it.
A wait after your block has handled the notice returns the task the same way.

If an `onTaskSettled` request has started handling the notice and hasn't finished, for example
a second turn waits on the task while that request is in progress, `getTask` returns this
instead of the task:

```ts
{ ok: true, taskId: "task_8f2", arrivesAsNotice: true, message: "…" }
```

`message` names the task, says its ending is arriving as a notice, and tells the model not to
wait on it again.
The session hears that ending once, from the notice.

A wait has a deadline, five minutes unless you pass `deadline` in milliseconds. At the deadline
your turn carries on with the task as it stands, usually still `in_progress`. Nothing is
Expand Down
17 changes: 13 additions & 4 deletions packages/orchestration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -507,6 +507,13 @@ the filer waited on (`getTask` with `wait`), nor a cancel the filer's own model
task tool in its turn. The block runs once per ending; if it throws, the notice stays owed and is
sent again. The entry queues on the session unless `concurrency` says otherwise.

A notice waits for the request that filed its task. While that request is still running (or
parked), the entry holds the notice back the way `defer` does (below), and the request sends it as
it ends. A wait in the filing request therefore reads the ending first. Before it runs `block`,
the entry takes the notice in one conditional write (the board's optional `takeNotice`, which both
built-in backings implement), so a wait that reads the task and the notice never both deliver the
same ending. A notice whose `block` threw stays owed, and the next copy takes it over.

**Holding a notice back.** Give `taskSettledEntry` a `defer(ctx, notice)` when the session can be
busy with something a notice must not run beside, such as a turn that is still under way. When it
answers `true`, the entry runs nothing and acknowledges nothing: the notice stays owed, and the
Expand Down Expand Up @@ -998,10 +1005,12 @@ See [Per-generator binding](https://flow-state.dev/docs/skills/binding) for the
`getTask({ id, wait: true, deadline? })` parks the calling turn on a task its session filed,
until the task ends or the deadline passes, and returns the task as `getTask` always does. The
deadline is milliseconds from the call, five minutes by default, within the host's
`waitDeadlines` bounds. A task that already ended is returned at once, and no `onTaskSettled`
follows for it when the flow's `onTaskSettled` waits behind the turn that filed the task, as
Workforce's `respond` and `work` turns make it: the request still runs, but your block isn't
called. At the deadline the task runs on and nothing is cancelled. If it ends before the
`waitDeadlines` bounds. A task that already ended is returned at once, and your `onTaskSettled`
block isn't called for it: a task's notice waits until the request that filed it has ended, so a
wait in that request reads the ending first. If the notice is already being handled when the wait
reads the task (another request filed it, or its turn had ended), the wait returns
`{ ok: true, taskId, arrivesAsNotice: true, message }` instead of the task: the notice is how the
session hears that ending. At the deadline the task runs on and nothing is cancelled. If it ends before the
waiting request does, that request's next `getTask` on it returns the ending. If
the request ends without reading it, the ending goes to `onTaskSettled`, once. Waits nest: a
task's own turn may wait too. A wait is refused when it would make a chain of more than five
Expand Down
5 changes: 5 additions & 0 deletions packages/orchestration/src/skills/task-tools-capability.ts
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ import { boardResolverOf } from "../task-board/board-resolver";
// The one task-turn test (FIX-1816 BR-5a), owned by the task-board.
import { isTaskTurn } from "../task-board/task-turn";
import {
arrivesAsNoticeAnswer,
canHoldWait,
deliverHeldEnding,
toolCallOf,
Expand Down Expand Up @@ -1487,6 +1488,8 @@ function buildTaskTools(resolve: TaskCollectionResolver, options: BuildTaskTools
.optional(),
}),
}),
/** A wait whose task's ending arrives as a notice, which the conversation hears on its own (FIX-2123). */
z.object({ ok: z.literal(true), taskId: z.string(), arrivesAsNotice: z.literal(true), message: z.string() }),
z.object({ ok: z.literal(false), error: z.string(), taskId: z.string().optional() }),
]),
execute: async (raw, ctx) => {
Expand Down Expand Up @@ -1541,6 +1544,8 @@ function buildTaskTools(resolve: TaskCollectionResolver, options: BuildTaskTools
};
}
if (!waited.ok) return { ok: false as const, taskId: t.id, error: `${waited.error}: ${waited.message}` };
// The filer's onTaskSettled took its ending first: that notice is the one hearing of it.
if (waited.arrivesAsNotice === true) return arrivesAsNoticeAnswer(waited.task);
t = waited.task;
} else {
// The ending held for this request, when its wait on the task lapsed:
Expand Down
3 changes: 3 additions & 0 deletions packages/orchestration/src/tasks/collection/get-or-create.ts
Original file line number Diff line number Diff line change
Expand Up @@ -217,10 +217,13 @@ export async function getOrCreateTaskCollection<TInput = unknown, TOutput = unkn
// (FIX-2078), bound to this request; and stamps the filer on each row it files.
const declaredDelivery = isDefinedTaskCollection(declared) ? declared.__taskCollection.delivery : undefined;
const sessionId = ctx.session?.identity?.id;
// The engine's identity of the running request (FIX-2123), never input (BP-031).
const requestId = ctx.request?.identity?.id;
const delivery =
declaredDelivery !== undefined && sessionId !== undefined
? {
ownSession: sessionId,
...(requestId !== undefined ? { ownRequest: requestId } : {}),
hooks: createDelivery(ctx, declaredDelivery, {
collectionId: options.collectionId,
...(partition !== undefined ? { partition } : {}),
Expand Down
24 changes: 22 additions & 2 deletions packages/orchestration/src/tasks/collection/internal.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import {
} from "../schema/task-status";
import { extractTaskItems } from "../items/extract-window";
import { applyEndingRecorder, endingOf, type TaskEnding, type TaskEndingRecorder } from "./ending";
import { deliveredByWait, droppedHeldNotices, recordEnding as recordNoticeOwed, releasedHeldNotices } from "../notice/task-notice";
import { deliveredByWait, droppedHeldNotices, recordEnding as recordNoticeOwed, releasedHeldNotices, takenNoticePatch, type TaskNotice } from "../notice/task-notice";
import { assertDepsEditable, TaskFilingRefusedError } from "./filing";
import { hasStarted, lockedFields, type TaskEditField } from "./field-rules";
import { steersForAttempt, untakenSteers } from "../steers";
Expand Down Expand Up @@ -120,7 +120,7 @@ export function buildInitialTask<TInput, TOutput>(
...(status === "cancelled" && init.causedBy !== undefined ? { causedBy: init.causedBy } : {}),
...(createdBy !== undefined ? { createdBy } : {}),
// Server-written by the backing (FIX-2078), never from `init`.
...(filer !== undefined ? { filer: { sessionId: filer.sessionId } } : {}),
...(filer !== undefined ? { filer: { sessionId: filer.sessionId, ...(filer.requestId !== undefined ? { requestId: filer.requestId } : {}) } } : {}),
createdAt: now,
updatedAt: now,
};
Expand Down Expand Up @@ -284,6 +284,8 @@ export function releaseWaitWrite(
* (`TaskCollectionRef.dropNoticeReadByWait`, FIX-2112): the wait's answer
* delivered the ending, so the notice the ending owes the filer is dropped.
*
* A notice already taken is left: its notice is the delivery (FIX-2123).
*
* @returns The patch to write, or `undefined` when the row owes no notice for its ending.
*/
export function noticeReadByWaitWrite(task: Task): { patch: Partial<Task> } | undefined {
Expand All @@ -292,6 +294,24 @@ export function noticeReadByWaitWrite(task: Task): { patch: Partial<Task> } | un
return { patch: { metadata: { ...(task.metadata ?? {}), ...dropped } } };
}

/**
* The one rule for taking a notice to deliver it
* (`TaskCollectionRef.takeNotice`, FIX-2123): decided on the row as stored, so
* a wait's drop and a take can't both go through.
*
* @returns The patch to write, or `undefined` when the row doesn't owe the
* notice, or another request holds it.
*/
export function noticeTakeWrite(
task: Task,
notice: Pick<TaskNotice, "attempt" | "ending">,
take: { readonly by: string; readonly at: number; readonly replacing?: string }
): { patch: Partial<Task> } | undefined {
const taken = takenNoticePatch(task, notice, take);
if (Object.keys(taken).length === 0) return undefined;
return { patch: { metadata: { ...(task.metadata ?? {}), ...taken } } };
}

/**
* The fields a filing may change until the task first starts (FIX-2080,
* ER-4; the brief's fields and the assignee since FIX-2081). Each field's rule
Expand Down
52 changes: 47 additions & 5 deletions packages/orchestration/src/tasks/collection/resource-backed.ts
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@ import {
markWaitWrite,
releaseWaitWrite,
noticeReadByWaitWrite,
noticeTakeWrite,
settingsWrite,
settingsChangeKind,
type TaskSettingsPatch,
Expand All @@ -147,7 +148,7 @@ import { generateId } from "../generate-id";
import { steerWrite, takeSteersPatch, untakenSteers } from "../steers";
import type { TaskEndingRecorder } from "./ending";
import type { DeliveryHooks } from "../notice/delivery";
import { NOTICE_DUE_MS, LAPSED_SLOT_CHECK_MS } from "../notice/task-notice";
import { NOTICE_DUE_MS, LAPSED_SLOT_CHECK_MS, owesUntakenNotice, resendNowPatch } from "../notice/task-notice";
import type { TaskChangeEvent, TaskChangeKind } from "./change-event";
import { createTaskChangeEmitter } from "./change-event";
import { assertDepsFileable, assertFilingSettings, withFollowUpPlacement, type TaskSettingsSupport } from "./filing";
Expand Down Expand Up @@ -273,7 +274,12 @@ export interface ResourceBackedOptions {
* the hooks run around every write that may end a row. Omitted on a ledger
* no board hands rows out from.
*/
delivery?: { ownSession: string; hooks: DeliveryHooks };
delivery?: {
ownSession: string;
/** The running request, by its engine identity (FIX-2123): stamped as the filing request of each row it files in its own session. */
ownRequest?: string;
hooks: DeliveryHooks;
};
}

/**
Expand Down Expand Up @@ -709,13 +715,42 @@ export async function createResourceBackedTaskCollection<TInput = unknown, TOutp

/**
* The filer a new row is stamped with (FIX-2078): the session filing it, or,
* for a follow-up, its root's filer, whoever typed it.
* for a follow-up, its root's filer, whoever typed it. With the request
* filing it when that is the filer's session (FIX-2123): the row's notice
* waits for that request to end.
*/
function filerFor(init: { followUpOf?: string }): Task["filer"] {
if (delivery === undefined) return undefined;
const root = init.followUpOf === undefined ? undefined : lookupRow(init.followUpOf);
const rootFiler = root === undefined ? undefined : delivery.hooks.filerOf(root);
return { sessionId: rootFiler ?? delivery.ownSession };
const sessionId = (root === undefined ? undefined : delivery.hooks.filerOf(root)) ?? delivery.ownSession;
const filing = sessionId === delivery.ownSession ? delivery.ownRequest : undefined;
return { sessionId, ...(filing !== undefined ? { requestId: filing } : {}) };
}

/** The rows this request filed whose notice waits for it to end (FIX-2123). */
const filedHere = new Set<string>();

/**
* Note a row this request filed with itself as the filing request
* (FIX-2123), and, on the first, ask to be called as the request ends: a
* notice it turned away while it ran is sent then, past any backoff, rather
* than at the pass's next resend. Not at a park: the request hasn't ended.
*/
function noteFiled(row: Task): void {
if (delivery === undefined || row.filer?.requestId === undefined || row.filer.requestId !== delivery.ownRequest) return;
const first = filedHere.size === 0;
filedHere.add(row.id);
if (!first) return;
delivery.hooks.whenRequestEnds?.(async () => {
for (const id of filedHere) {
await ref.refresh?.(id);
const row = lookupRow(id);
if (row === undefined || !owesUntakenNotice(row)) continue;
const sooner = resendNowPatch(row);
if (Object.keys(sooner).length > 0) await ref.patchMetadata(id, sooner);
await deliverAfter(lookupRow(id) ?? row);
}
});
}

/**
Expand Down Expand Up @@ -839,6 +874,7 @@ export async function createResourceBackedTaskCollection<TInput = unknown, TOutp
);
mirror.set(task.id, created);
emit("added", task);
noteFiled(task as Task);
if (await cancelIfDepEnded(task.id)) return lookupRow(task.id) as Task<TInput, TOutput>;
return task;
},
Expand All @@ -860,6 +896,7 @@ export async function createResourceBackedTaskCollection<TInput = unknown, TOutp
mirror.set(task.id, createdRef);
created.push(task);
emit("added", task);
noteFiled(task as Task);
}
// A dependency that ended meanwhile cancels its dependents, and a row
// that cancels can end another of the batch's, so the pass repeats
Expand Down Expand Up @@ -1401,6 +1438,11 @@ export async function createResourceBackedTaskCollection<TInput = unknown, TOutp
return patchRef(id, "metadata_changed", (task) => noticeReadByWaitWrite(task as Task)?.patch as Partial<Task<TInput, TOutput>> | undefined);
},

async takeNotice(id, notice, take) {
const at = now();
return patchRef(id, "metadata_changed", (task) => noticeTakeWrite(task as Task, notice, { ...take, at })?.patch as Partial<Task<TInput, TOutput>> | undefined);
},

async refresh(id) {
const taskRef = mirror.get(id);
if (taskRef !== undefined) await readCommitted(taskRef, () => undefined);
Expand Down
6 changes: 6 additions & 0 deletions packages/orchestration/src/tasks/collection/state-backed.ts
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ import {
markWaitWrite,
releaseWaitWrite,
noticeReadByWaitWrite,
noticeTakeWrite,
settingsWrite,
settingsChangeKind,
type EndingRecording,
Expand Down Expand Up @@ -1074,6 +1075,11 @@ export function createStateBackedTaskCollection<TInput = unknown, TOutput = unkn
return patchOne(id, "metadata_changed", (task) => noticeReadByWaitWrite(task as Task)?.patch as Partial<Task<TInput, TOutput>> | undefined);
},

async takeNotice(id, notice, take) {
const at = now();
return patchOne(id, "metadata_changed", (task) => noticeTakeWrite(task as Task, notice, { ...take, at })?.patch as Partial<Task<TInput, TOutput>> | undefined);
},

async payOwedCascades() {
return payOwedCascades(cascadeWrites, now());
},
Expand Down
20 changes: 19 additions & 1 deletion packages/orchestration/src/tasks/collection/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import type { Task, TaskRunLink, TaskStatus } from "../schema/task";
import type { TaskInit, TaskFilter } from "../schema/task-init";
import type { TaskSettingsPatch } from "./internal";
import type { SteerInput } from "../steers";
import type { NoticeEnding as TaskNoticeEnding } from "../notice/task-notice";

/**
* How a cancel was made (FIX-2081), as `Task.cancelledVia` records it:
Expand Down Expand Up @@ -1044,13 +1045,30 @@ export interface TaskCollectionRef<TInput = unknown, TOutput = unknown> {
/**
* The filer's wait found the row already ended and returned it (FIX-2112):
* drop the notice the ending owes the filer, which that answer delivered.
* `unchanged` when the row owes none for its ending. Server-owned, as the
* `unchanged` when the row owes none for its ending, or its notice is taken
* ({@link takeNotice}): that notice is the delivery. Server-owned, as the
* other wait writes are: a caller's `patchMetadata` can't clear a notice.
*
* Optional: both built-in backings implement it. On a ref without it the
* filer hears the ending again, as `onTaskSettled`.
*/
dropNoticeReadByWait?(id: string): Promise<TaskWriteOutcome>;
/**
* Take a notice the row owes, for the `onTaskSettled` request `take.by`
* that is about to deliver it (FIX-2123), in one conditional write: a wait's
* drop ({@link dropNoticeReadByWait}) and a take never both go through.
* `unchanged` when the row no longer owes it, or another request holds it,
* other than `take.replacing` (a taker the caller found has ended). The
* request that took it clears the marker once its block has run.
*
* Optional: both built-in backings implement it. On a ref without it a
* notice is admitted with no take, as before.
*/
takeNotice?(
id: string,
notice: { readonly attempt: number; readonly ending: TaskNoticeEnding },
take: { readonly by: string; readonly replacing?: string }
): Promise<TaskWriteOutcome>;
/**
* A touch of the board: pay every cascade owed past its due time
* (`Task.cascadeOwed`, FIX-2080 D2), so one a crash left owed is paid.
Expand Down
Loading
Loading