Skip to content
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/a-notice-is-taken-once.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@flow-state-dev/orchestration": patch
---

A task's ending reaches the session that filed it once, on the defaults too: a `getTask({ id, wait: true })` that reads a task already ended no longer races that task's `onTaskSettled`. A notice now waits while the request that filed its task is running (not while it is parked on a person or a wait), and that request sends it as it parks or ends. `taskSettledEntry` takes a notice in one conditional write before it runs your block, and a copy never takes a notice from a request that hasn't ended. A wait that finds the notice taken returns `{ ok: true, taskId, arrivesAsNotice: true, message }` instead of the task. Task collection refs gain an optional `takeNotice(id, notice, take)`, which both built-in backings implement; a ref without it admits notices as before. A filing's `metadata` can no longer carry notice records (FIX-2123).
6 changes: 6 additions & 0 deletions .changeset/request-waits-running.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
"@flow-state-dev/core": patch
"@flow-state-dev/engine": patch
---

`RequestWaits` gains an optional `running(requestIds)`: whether each request of the running session is running its own work now (`in_progress`), as opposed to parked (`suspended`) or ended. The engine's request host implements it (FIX-2123).
38 changes: 27 additions & 11 deletions apps/docs/docs/orchestration/task-board.md
Original file line number Diff line number Diff line change
Expand Up @@ -796,16 +796,18 @@ When a task the board handed out ends, the session that filed it is told, once.
| `question` | On `parked`: what the worker asked. Answer it with [`answerTask`](#when-a-task-stops-on-a-question) |
| `cancelled` | On `errored` or `cancelled`: the ids of the dependents this ending [cancelled](#when-a-dependency-fails) that the same session filed. Those dependents send no notice of their own |

While the request that filed the task is still running, its notice waits and goes out when that request parks or ends. While the request is parked, on a person's approval or input or on a [`getTask` wait](#waiting-on-a-task), the notice goes out right away. So a turn that files a task and never waits on it hears the ending once that turn ends or parks. Waiting for the request needs a host with durable execution; elsewhere a notice is sent as soon as the task ends.

A task that fails and is retried sends nothing until its last attempt ends. These endings send no notice either:

- A task the filer [waited on](#waiting-on-a-task) with `getTask({ id, wait: true })` while it ended. The waiting call gets the result instead. An ending after the wait's deadline goes to the waiting request's next `getTask` while it runs, and comes here only if that request ends without reading it.
- A task the filer [waited on](#waiting-on-a-task) with `getTask({ id, wait: true })`, while it ran or after it ended but before an `onTaskSettled` request for that ending had started. The waiting call gets the result instead.
- A cancel the filer's own model made with its `updateTask` tool (`status: "cancelled"`), in its own turn. The tool's result already told it, and names the dependents the cancel took along.

A cancel from anywhere else does send one: a person, an app calling the board's actions, or a cascade from a task another session filed.

Your block runs once per ending. If it throws, the notice stays owed and is sent again, so the same ending can reach the block a second time after a failure. The entry queues on the session by default, so two notices run one after the other; pass `concurrency` to `taskSettledEntry` to change that.
Your block runs once per ending. If it throws, the notice stays owed and is sent again, so the same ending can reach the block a second time after a failure. A second copy never runs while the first copy's request is still running or parked. The entry queues on the session by default, so two notices run one after the other; pass `concurrency` to `taskSettledEntry` to change that.

A notice that can't be sent when the task ends, because the process died first or the send failed, is sent about two seconds later, or as soon as the server is running again if it was down then. The [durability sweeper](../advanced/durable-execution.md#retention-and-cleanup) runs `onTaskSettled` in the filing session with an empty input, which sends whatever the session's boards still owe. You may see an extra `onTaskSettled` request with empty input in the session; your block isn't called for it. This needs a host with durable execution (`durable: true` on `createFlowState`) and a store that keeps wakes, which every built-in store does. Without them, a notice is sent once, when the task ends, and a lost one isn't sent again.
A notice that can't be sent when it falls due, because the process died first or the send failed, is sent about two seconds later, or as soon as the server is running again if it was down then. The [durability sweeper](../advanced/durable-execution.md#retention-and-cleanup) runs `onTaskSettled` in the filing session with an empty input, which sends whatever the session's boards still owe. You may see an extra `onTaskSettled` request with empty input in the session; your block isn't called for it. This needs a host with durable execution (`durable: true` on `createFlowState`) and a store that keeps wakes, which every built-in store does. Without them, a notice is sent once, when the task ends, and a lost one isn't sent again.

If the filing session no longer exists, the notice can't be delivered. The task records that in its `metadata`, under `noticeUndeliverable:<attempt>:<ending>`, as `{ at, detail }`. A notice still owed a day after it fell due is given up the same way; the host's [`maxDebtAgeMs`](../advanced/durable-execution.md#retention-and-cleanup) sets that limit.

Expand Down Expand Up @@ -845,17 +847,31 @@ getTask({ id: "task_8f2", wait: true })

You can wait in the same turn or in a later one. Your turn pauses until the task ends, then
carries on with the task as `getTask` always returns it: its status, and its output once it
completed. Nothing holds a request open while it waits, so a server restart loses nothing.
A task that already ended is returned at once, and no `onTaskSettled` follows for it, as long
as your `onTaskSettled` waits behind the turn that filed the task. A Workforce conversation or
task turn works that way. The request still runs, but your block isn't called.
completed. Nothing keeps a request open while it waits, so a server restart loses nothing.
A task that already ended is returned at once. If no `onTaskSettled` request for that ending
has started yet, the wait returns the ending, and no notice is sent for it.

Once an `onTaskSettled` request for that ending has started, whether it is still running or
has finished, `getTask` returns this instead of the task:

```ts
{ ok: true, taskId: "task_8f2", arrivesAsNotice: true, message: "…" }
```

`message` names the task, says its ending comes to the conversation as a notice, and tells the
model not to wait on it again. The session hears that ending once, from the notice.

You'll see it when your turn parks between filing a task and waiting on it, or when a second
turn waits on a task the first one filed. Say a turn files a task, then waits for a person's
approval. The task finishes during that wait, and the conversation hears it from
`onTaskSettled`. After the approval, the same turn waits on the task and gets `arrivesAsNotice`.

A wait has a deadline, five minutes unless you pass `deadline` in milliseconds. At the deadline
your turn carries on with the task as it stands, usually still `in_progress`. Nothing is
cancelled: the task runs on, and you can wait on it again. If it ends before your request
does, even while your request waits on something else, your next `getTask` on it returns the
ending. If your request ends without reading it, you [hear about it](#hearing-how-a-task-ended)
once, the usual way. A worker you're
cancelled: the task runs on, and you can wait on it again. If it ends while your request is
running, your next `getTask` on it returns the ending. If it ends while your request is parked
on something else, or your request ends without reading it, you
[hear about it](#hearing-how-a-task-ended) once, the usual way. A worker you're
waiting on can't stop to ask you a question; it answers with what it has, or fails. One wait
per step, where a step is one model call and the tool calls it makes.

Expand Down
9 changes: 9 additions & 0 deletions packages/core/src/types/request-host.ts
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,15 @@ export interface RequestWaits {
* for a request that ended without reading the task.
*/
ended?(requestIds: readonly string[]): Promise<Readonly<Record<string, boolean>>>;
/**
* Whether each request is running its own work now (`in_progress`): neither
* parked (`suspended`) nor ended. `false` for one parked
* on a person or a wait, for one that ended or whose process died, for a
* request of another session, and for one with no record. A board reads it
* to hold a task's notice while the request that filed the task still runs
* (FIX-2123), and not while that request is parked.
*/
running?(requestIds: readonly string[]): Promise<Readonly<Record<string, boolean>>>;
}

/**
Expand Down
16 changes: 16 additions & 0 deletions packages/engine/src/durability/wait-chain.ts
Original file line number Diff line number Diff line change
Expand Up @@ -250,10 +250,26 @@ export function createRequestWaits(inputs: {
}
}
return answers;
},
running: async (requestIds) => {
const answers: Record<string, boolean> = {};
for (const requestId of new Set(requestIds)) {
try {
const record = await inputs.readRequest(requestId);
answers[requestId] = record !== undefined && record.sessionId === inputs.sessionId && RUNNING.has(record.status);
} catch {
// A record that can't be read answers for itself only: not running, so nothing waits on it.
answers[requestId] = false;
}
}
return answers;
}
};
}

/** Request statuses of a request running its own work: started, and neither parked nor ended. */
const RUNNING: ReadonlySet<RequestRecord["status"]> = new Set(["in_progress"]);

/** Request statuses after which a request runs no more of its own work. */
const WORK_OVER: ReadonlySet<RequestRecord["status"]> = new Set(["completed", "incomplete", "failed", "aborted"]);

Expand Down
26 changes: 26 additions & 0 deletions packages/engine/test/wait-chain.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,32 @@ describe("whether a request has stopped running its own work", () => {
elsewhere: false
});
});

it("tells running apart from parked: only a request in progress in this session is running", async () => {
const elsewhereRunning = { id: "elsewhereRunning", status: "in_progress", sessionId: "s_other" };
const both = createRequestWaits({
provider: { ...provider([]), loadSuspension: async () => null as never },
deadlines: { defaultMs: 1, minMs: 1, maxMs: 1 },
userId: "u",
requestId: "r",
own: undefined,
sessionId: "s",
readRequest: async (id) => {
if (id === "unreadable") throw new Error("store unavailable");
return (id === "elsewhereRunning" ? elsewhereRunning : records[id]) as never;
}
});
expect(await both.running!(["running", "parked", "done", "died", "redriven", "gone", "elsewhereRunning", "unreadable"])).toEqual({
running: true,
parked: false,
done: false,
died: false,
redriven: false,
gone: false,
elsewhereRunning: false,
unreadable: false
});
});
});

describe("the stamp a waited dispatch leaves on its child", () => {
Expand Down
20 changes: 16 additions & 4 deletions packages/orchestration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -507,6 +507,16 @@ the filer waited on (`getTask` with `wait`), nor a cancel the filer's own model
task tool in its turn. The block runs once per ending; if it throws, the notice stays owed and is
sent again. The entry queues on the session unless `concurrency` says otherwise.

A notice waits for the request that filed its task while that request is running, not while it is
parked on a person or a wait. The entry holds the notice back the way `defer` does (below), and the
request sends it as it parks or ends. A wait in the filing request therefore reads the ending
first. Before it runs `block`, the entry takes the notice in one conditional write (the board's
optional `takeNotice`, which both built-in backings implement), so a wait that reads the task and
the notice never both deliver the same ending. A notice whose `block` threw stays owed, and the
next copy takes it over; a copy that finds the notice taken by a request that hasn't ended,
running or parked, runs nothing. Holding needs a host that records which requests are running
(durable execution, `requestHost.waits.running`); anywhere else a notice is admitted at once.

**Holding a notice back.** Give `taskSettledEntry` a `defer(ctx, notice)` when the session can be
busy with something a notice must not run beside, such as a turn that is still under way. When it
answers `true`, the entry runs nothing and acknowledges nothing: the notice stays owed, and the
Expand Down Expand Up @@ -998,10 +1008,12 @@ See [Per-generator binding](https://flow-state.dev/docs/skills/binding) for the
`getTask({ id, wait: true, deadline? })` parks the calling turn on a task its session filed,
until the task ends or the deadline passes, and returns the task as `getTask` always does. The
deadline is milliseconds from the call, five minutes by default, within the host's
`waitDeadlines` bounds. A task that already ended is returned at once, and no `onTaskSettled`
follows for it when the flow's `onTaskSettled` waits behind the turn that filed the task, as
Workforce's `respond` and `work` turns make it: the request still runs, but your block isn't
called. At the deadline the task runs on and nothing is cancelled. If it ends before the
`waitDeadlines` bounds. A task that already ended is returned at once, and your `onTaskSettled`
block isn't called for it: a task's notice waits while the request that filed it runs, so a wait
in that request reads the ending first. If the notice was taken by `onTaskSettled` before the wait
reads the task (the filing request was parked, or another request waits), the wait returns
`{ ok: true, taskId, arrivesAsNotice: true, message }` instead of the task, whether that notice is
still being handled or done: the notice is how the session hears that ending. At the deadline the task runs on and nothing is cancelled. If it ends before the
waiting request does, that request's next `getTask` on it returns the ending. If
the request ends without reading it, the ending goes to `onTaskSettled`, once. Waits nest: a
task's own turn may wait too. A wait is refused when it would make a chain of more than five
Expand Down
5 changes: 5 additions & 0 deletions packages/orchestration/src/skills/task-tools-capability.ts
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ import { boardResolverOf } from "../task-board/board-resolver";
// The one task-turn test (FIX-1816 BR-5a), owned by the task-board.
import { isTaskTurn } from "../task-board/task-turn";
import {
arrivesAsNoticeAnswer,
canHoldWait,
deliverHeldEnding,
toolCallOf,
Expand Down Expand Up @@ -1487,6 +1488,8 @@ function buildTaskTools(resolve: TaskCollectionResolver, options: BuildTaskTools
.optional(),
}),
}),
/** A wait whose task's ending arrives as a notice, which the conversation hears on its own (FIX-2123). */
z.object({ ok: z.literal(true), taskId: z.string(), arrivesAsNotice: z.literal(true), message: z.string() }),
z.object({ ok: z.literal(false), error: z.string(), taskId: z.string().optional() }),
]),
execute: async (raw, ctx) => {
Expand Down Expand Up @@ -1541,6 +1544,8 @@ function buildTaskTools(resolve: TaskCollectionResolver, options: BuildTaskTools
};
}
if (!waited.ok) return { ok: false as const, taskId: t.id, error: `${waited.error}: ${waited.message}` };
// The filer's onTaskSettled took its ending first: that notice is the one hearing of it.
if (waited.arrivesAsNotice === true) return arrivesAsNoticeAnswer(waited.task);
t = waited.task;
} else {
// The ending held for this request, when its wait on the task lapsed:
Expand Down
3 changes: 3 additions & 0 deletions packages/orchestration/src/tasks/collection/get-or-create.ts
Original file line number Diff line number Diff line change
Expand Up @@ -217,10 +217,13 @@ export async function getOrCreateTaskCollection<TInput = unknown, TOutput = unkn
// (FIX-2078), bound to this request; and stamps the filer on each row it files.
const declaredDelivery = isDefinedTaskCollection(declared) ? declared.__taskCollection.delivery : undefined;
const sessionId = ctx.session?.identity?.id;
// The engine's identity of the running request (FIX-2123), never input (BP-031).
const requestId = ctx.request?.identity?.id;
const delivery =
declaredDelivery !== undefined && sessionId !== undefined
? {
ownSession: sessionId,
...(requestId !== undefined ? { ownRequest: requestId } : {}),
hooks: createDelivery(ctx, declaredDelivery, {
collectionId: options.collectionId,
...(partition !== undefined ? { partition } : {}),
Expand Down
27 changes: 24 additions & 3 deletions packages/orchestration/src/tasks/collection/internal.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import {
} from "../schema/task-status";
import { extractTaskItems } from "../items/extract-window";
import { applyEndingRecorder, endingOf, type TaskEnding, type TaskEndingRecorder } from "./ending";
import { deliveredByWait, droppedHeldNotices, recordEnding as recordNoticeOwed, releasedHeldNotices } from "../notice/task-notice";
import { deliveredByWait, droppedHeldNotices, recordEnding as recordNoticeOwed, releasedHeldNotices, takenNoticePatch, withoutNoticeMarkers, type TaskNotice } from "../notice/task-notice";
import { assertDepsEditable, TaskFilingRefusedError } from "./filing";
import { hasStarted, lockedFields, type TaskEditField } from "./field-rules";
import { steersForAttempt, untakenSteers } from "../steers";
Expand Down Expand Up @@ -106,7 +106,8 @@ export function buildInitialTask<TInput, TOutput>(
priority: init.priority,
input: init.input,
labels: init.labels,
metadata: init.metadata,
// A filing can't forge, clear or settle a notice (BP-031).
metadata: withoutNoticeMarkers(init.metadata),
...(init.followUpOf !== undefined ? { followUpOf: init.followUpOf } : {}),
...(init.filedByTask !== undefined ? { filedByTask: init.filedByTask } : {}),
...(init.session !== undefined ? { session: init.session } : {}),
Expand All @@ -120,7 +121,7 @@ export function buildInitialTask<TInput, TOutput>(
...(status === "cancelled" && init.causedBy !== undefined ? { causedBy: init.causedBy } : {}),
...(createdBy !== undefined ? { createdBy } : {}),
// Server-written by the backing (FIX-2078), never from `init`.
...(filer !== undefined ? { filer: { sessionId: filer.sessionId } } : {}),
...(filer !== undefined ? { filer: { sessionId: filer.sessionId, ...(filer.requestId !== undefined ? { requestId: filer.requestId } : {}) } } : {}),
createdAt: now,
updatedAt: now,
};
Expand Down Expand Up @@ -284,6 +285,8 @@ export function releaseWaitWrite(
* (`TaskCollectionRef.dropNoticeReadByWait`, FIX-2112): the wait's answer
* delivered the ending, so the notice the ending owes the filer is dropped.
*
* A notice already taken is left: its notice is the delivery (FIX-2123).
*
* @returns The patch to write, or `undefined` when the row owes no notice for its ending.
*/
export function noticeReadByWaitWrite(task: Task): { patch: Partial<Task> } | undefined {
Expand All @@ -292,6 +295,24 @@ export function noticeReadByWaitWrite(task: Task): { patch: Partial<Task> } | un
return { patch: { metadata: { ...(task.metadata ?? {}), ...dropped } } };
}

/**
* The one rule for taking a notice to deliver it
* (`TaskCollectionRef.takeNotice`, FIX-2123): decided on the row as stored, so
* a wait's drop and a take can't both go through.
*
* @returns The patch to write, or `undefined` when the row doesn't owe the
* notice, or another request holds it.
*/
export function noticeTakeWrite(
task: Task,
notice: Pick<TaskNotice, "attempt" | "ending">,
take: { readonly by: string; readonly at: number; readonly replacing?: string }
): { patch: Partial<Task> } | undefined {
const taken = takenNoticePatch(task, notice, take);
if (Object.keys(taken).length === 0) return undefined;
return { patch: { metadata: { ...(task.metadata ?? {}), ...taken } } };
}

/**
* The fields a filing may change until the task first starts (FIX-2080,
* ER-4; the brief's fields and the assignee since FIX-2081). Each field's rule
Expand Down
Loading
Loading