Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/one-turn-of-a-task-at-a-time.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
"@flow-state-dev/orchestration": patch
"@flow-state-dev/workforce": patch
---

On a Workforce conversation's board, a task runs one turn at a time in its session: while one of its turns waits on a step it filed for itself in a session of its own, a notice's turn and a steer's turn wait until it answers, instead of planning beside it and stalling the task. `taskSettledEntry` takes `defer`, and `resendNowPatch` is exported (FIX-2113).
5 changes: 5 additions & 0 deletions .changeset/steer-turn-shows-its-steers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@flow-state-dev/workforce": patch
---

The turn a steer starts while a task waits on work it filed now shows the steer in the task's session as a line from its sender, so the turn that answers after that work ends reads it too (FIX-2113).
4 changes: 2 additions & 2 deletions apps/docs/docs/orchestration/task-board.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,14 +91,14 @@ Every rule is checked inside the write, so a task that starts between your read

- **Filed, not started yet.** The first attempt's brief carries every message, after the goal.
- **Running.** The worker takes it before its next model call, in the same turn. A coding run on a [harness manager](./harness-manager.md#steering-a-run) takes it at its next step.
- **Waiting on tasks it filed.** A turn starts now, with the message, while those tasks keep running.
- **Waiting on tasks it filed.** A turn starts with the message while those tasks keep running. On a Workforce conversation's board, a session runs one turn of a task at a time. If another of the task's turns is waiting on a step it filed for itself in a session of its own, the message's turn waits until that turn answers. So does the turn that a finished task wakes.
- **After the worker's last model call**, even while the task is being settled. The task gets one more turn, with the message, before it settles. Its output follows the usual rule: the latest checkpoint, or else that turn's answer.

The last case needs a turn to run. A Workforce conversation's board has one. For a board you wire yourself, name the internal entry that runs it in `taskLedgers({ endRules: { steerTurn: { entry } } })`. Without one, the task settles and the message stays on it, untaken: on such a board, a message that arrives after the last model call may not be acted on.

The filer's model sends a message with the `updateTask` tool, and a person with the board's `updateTask` action; the task records which. A message is sent alone, not with an edit or a hold, so a call never lands half of what it asked. A task takes at most 20 messages. Past that, cancel it and file it again with everything it needs, or file a follow-up once it ends.

`getTask` lists a task's messages as `steers`, oldest first, each with who sent it and whether its work has read it (`taken`). A message the work reads mid-run also shows in the task's session as a line from its sender.
`getTask` lists a task's messages as `steers`, oldest first, each with who sent it and whether its work has read it (`taken`). Unless it arrived before the task started, the message also shows in the task's session as a line from its sender, so later turns in that session read it.

A finished task takes no message, because its output never changes. To correct it, file a follow-up: it runs in the task's session, and `getTask` on the original shows the follow-up's output as `current`.

Expand Down
12 changes: 11 additions & 1 deletion packages/orchestration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -496,7 +496,7 @@ export default defineFlow({
```

**Hearing how a task ended.** The session that filed a row hears each of its endings once, as
a request to its flow's `onTaskSettled` internal entry. `taskSettledEntry({ block, concurrency? })`
a request to its flow's `onTaskSettled` internal entry. `taskSettledEntry({ block, concurrency?, defer? })`
builds that entry; `defineFlow` refuses a flow that holds a board that dispatches without one,
whichever flow the rows run on. `block` takes `taskSettledInputSchema` / `TaskSettledInput`:
`{ boardId, taskId, attempt, ending, output?, error?, question?, cancelled? }`, where `ending` is
Expand All @@ -507,6 +507,16 @@ the filer waited on (`getTask` with `wait`), nor a cancel the filer's own model
task tool in its turn. The block runs once per ending; if it throws, the notice stays owed and is
sent again. The entry queues on the session unless `concurrency` says otherwise.

**Holding a notice back.** Give `taskSettledEntry` a `defer(ctx, notice)` when the session can be
busy with something a notice must not run beside, such as a turn that is still under way. When it
answers `true`, the entry runs nothing and acknowledges nothing: the notice stays owed, and the
session's pass (below) sends it again. Each resend waits longer than the last (2 s, doubling, at most
5 minutes): that wait is the notice's *resend backoff*. A notice held back is not a failed delivery:
its age, which ends it as undeliverable past `maxDebtAgeMs`, counts from the last time it was held
back. Once the session is free, apply `resendNowPatch(row)` to each row that still owes a notice,
with `ref.patchMetadata(row.id, resendNowPatch(row))`, and run the pass: the patch drops the resend
backoff, so the pass sends the notice at once rather than at its next resend.

A notice that couldn't be sent when the row ended (the process died, the send failed) is sent
about two seconds later, or as soon as the server is running again if it was down then: on a host with durable execution and a store that keeps wakes, the
durability sweeper runs `onTaskSettled` in the filing session with an empty input, which sends
Expand Down
1 change: 1 addition & 0 deletions packages/orchestration/src/tasks/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ export {
type NoticeDecision,
NOTICE_DUE_MS,
noticeDueAt,
resendNowPatch,
cancelledDependents,
type NoticeRecording,
} from "./notice/task-notice";
Expand Down
6 changes: 4 additions & 2 deletions packages/orchestration/src/tasks/notice/pass.ts
Original file line number Diff line number Diff line change
Expand Up @@ -123,8 +123,10 @@ export async function payBoardDebts(
for (const notice of owedNotices(row, ref.collectionId, rows, options.filerOf)) {
const times = noticeTimes(row, notice);
if (times === undefined) continue;
if (tooOld(times.due)) {
const detail = `it was still owed ${Math.round((now - times.due) / 1000)} s past its due time`;
// Its age counts from its due time, or from the last time a busy session
// turned it away (FIX-2113), which is no failed delivery.
if (tooOld(times.since)) {
const detail = `it was still owed ${Math.round((now - times.since) / 1000)} s past its due time`;
log("a notice past the maximum debt age was settled undeliverable", { taskId: row.id, ending: notice.ending });
await ref.patchMetadata(row.id, { ...clearNotice(notice), ...undeliverableRecord(notice, detail, now) });
await options.onUndeliverable?.(ctx, row, notice, detail);
Expand Down
49 changes: 45 additions & 4 deletions packages/orchestration/src/tasks/notice/task-notice.ts
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,13 @@ interface OwedMarker {
* fast path and no pass until it is released. Absent on every other marker.
*/
readonly held?: true;
/**
* When a session last turned it away because it was busy (FIX-2113:
* `taskSettledEntry`'s `defer`). That is no failed delivery, so the
* notice's age, which ends it at the maximum debt age, counts from here.
* Absent on a marker never turned away.
*/
readonly deferredAt?: number;
}

function markerKey(attempt: number, ending: NoticeEnding): string {
Expand Down Expand Up @@ -248,6 +255,23 @@ export function releasedHeldNotices(row: Task, dueAt: number): Record<string, Ow
return patch;
}

/**
* The metadata patch that drops the resend records of every notice `row`
* still owes, so the filer's next pass sends each at once, past its backoff.
* For a session that turned its notices away while one of its turns was busy
* (`taskSettledEntry`'s `defer`) and is free now. An owed marker is left as
* it is.
*/
export function resendNowPatch(row: Task): Record<string, null> {
const patch: Record<string, null> = {};
for (const [key, value] of Object.entries(row.metadata ?? {})) {
if (!isSendable(key, value)) continue;
const resent = `${RESENT_PREFIX}${value.attempt}:${value.ending}`;
if ((row.metadata ?? {})[resent] != null) patch[resent] = null;
}
return patch;
}

/** Whether `row` holds an ending for the request whose wait on it lapsed (FIX-2082). */
export function holdsAnEnding(row: Task): boolean {
for (const [key, value] of Object.entries(row.metadata ?? {})) {
Expand Down Expand Up @@ -341,14 +365,31 @@ function nextSendAt(row: Task, marker: OwedMarker): number {
}

/**
* When `notice` first fell due, and when it is next sent (see `nextSendAt`).
* `undefined` when `row` no longer owes it. Its age counts from `due`.
* When `notice` first fell due, when it is next sent (see `nextSendAt`), and
* when its age counts from: its due time, or the last time a busy session
* turned it away, whichever is later. `undefined` when `row` no longer owes it.
*/
export function noticeTimes(row: Task, notice: Pick<TaskNotice, "attempt" | "ending">): { due: number; next: number } | undefined {
export function noticeTimes(
row: Task,
notice: Pick<TaskNotice, "attempt" | "ending">
): { due: number; next: number; since: number } | undefined {
const key = markerKey(notice.attempt, notice.ending);
const value = (row.metadata ?? {})[key];
if (!isSendable(key, value)) return undefined;
return { due: value.due ?? 0, next: nextSendAt(row, value) };
const due = value.due ?? 0;
return { due, next: nextSendAt(row, value), since: Math.max(due, value.deferredAt ?? 0) };
}

/**
* The metadata patch that records a busy session turning `notice` away at
* `now` (FIX-2113): the marker, as it is, with `deferredAt`. `{}` when `row`
* no longer owes it.
*/
export function deferredNoticePatch(row: Task, notice: Pick<TaskNotice, "attempt" | "ending">, now: number): Record<string, OwedMarker> {
const key = markerKey(notice.attempt, notice.ending);
const value = (row.metadata ?? {})[key];
if (!isSendable(key, value)) return {};
return { [key]: { ...value, deferredAt: now } };
}

/** The metadata patch that records one more resend of `notice` at `now`. */
Expand Down
18 changes: 17 additions & 1 deletion packages/orchestration/src/tasks/notice/task-settled-entry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
import { handler, sequencer } from "@flow-state-dev/core";
import type { BlockContext, BlockDefinition, ConcurrencyConfig } from "@flow-state-dev/core/types";
import { z } from "zod";
import { clearNotice, isNoticeOwed, type TaskNotice } from "./task-notice";
import { clearNotice, deferredNoticePatch, isNoticeOwed, type TaskNotice } from "./task-notice";
import { filerResolver, TASK_SETTLED_ENTRY } from "./delivery";
import { delegatingBoardsOf, payBoardDebts, settlePassWake, type BoardDebts } from "./pass";

Expand Down Expand Up @@ -62,6 +62,16 @@ export interface TaskSettledEntryOptions {
* one notice run one after the other and the second finds it acknowledged.
*/
readonly concurrency?: ConcurrencyConfig;
/**
* Whether this notice must wait: `true` runs nothing and acknowledges
* nothing, so the notice stays owed and the filer's pass sends it again
* (with its backoff, at most every 5 minutes). For a session that is busy
* with something the notice must not run beside. Being turned away is no
* failed delivery: the notice's age, which ends it as undeliverable past
* the host's maximum debt age, counts from the last time it was turned
* away. Default: never.
*/
readonly defer?: (ctx: BlockContext, notice: TaskSettledInput) => Promise<boolean>;
}

/** The running session's board a notice names, at this session's partition. */
Expand Down Expand Up @@ -103,6 +113,12 @@ export function taskSettledEntry(options: TaskSettledEntryOptions) {
const row = board?.ref.get(notice.taskId);
// A notice its row doesn't owe was delivered already, or never owed.
if (row === undefined || !isNoticeOwed(row, notice)) return { owed: false };
// Not now: left owed, for the pass to send again, with the time it was
// turned away, so its age (which ends it as undeliverable) counts from then.
if (options.defer !== undefined && (await options.defer(ctx, notice))) {
await board!.ref.patchMetadata(row.id, deferredNoticePatch(row, notice, board!.ref.now()));
return { owed: false };
}
return { owed: true, notice };
}
});
Expand Down
115 changes: 115 additions & 0 deletions packages/orchestration/test/tasks/notice-defer.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
/**
* `taskSettledEntry`'s `defer` (FIX-2113), on the real engine: a notice its
* session turns away runs nothing and stays owed, stamped with when it was
* turned away, so its age (which ends it as undeliverable) counts from then.
* One it doesn't turn away runs and is acknowledged.
*/
import { DEFAULT_ORG_ID, defineFlow, handler } from "@flow-state-dev/core";
import type { BlockContext } from "@flow-state-dev/core/types";
import { createFlowState, inMemoryStores, runAction } from "@flow-state-dev/engine";
import { describe, expect, it } from "vitest";
import { z } from "zod";
import { taskSettledEntry, type TaskSettledInput } from "../../src";
import { defineTaskCollection, getOrCreateTaskCollection, resolveResourceCollection, type TaskCollectionRef } from "../../src/tasks";

const BOARD = "defer-board";
const USER = "u_filer";
const SESSION = "s_filer";

async function boardOf(ctx: BlockContext): Promise<TaskCollectionRef> {
const collection = resolveResourceCollection(ctx, BOARD)!;
return getOrCreateTaskCollection({ ctx, backing: "resource", collectionId: BOARD, collection });
}

function host(defer: () => boolean) {
const heard: TaskSettledInput[] = [];
const seed = handler({
name: "seed",
inputSchema: z.object({ id: z.string() }),
outputSchema: z.any(),
execute: async ({ id }, ctx) => {
const board = await boardOf(ctx);
await board.addTask({ id, goal: "Find the fact" });
// The ending owes this session its notice, due now.
await board.patchMetadata(id, { "noticeOwed:1:completed": { attempt: 1, ending: "completed", due: Date.now() } });
return {};
}
});
const read = handler({
name: "read",
inputSchema: z.object({ id: z.string() }),
outputSchema: z.any(),
execute: async ({ id }, ctx) => (await boardOf(ctx)).get(id)?.metadata ?? null
});
const flow = defineFlow({
kind: "defer-host",
resources: { [BOARD]: defineTaskCollection({ id: BOARD, scope: "session", delivery: {} }) },
actions: { seed: { block: seed }, read: { block: read } },
internal: {
actions: {
onTaskSettled: taskSettledEntry({
block: handler({
name: "record-heard",
inputSchema: z.any(),
outputSchema: z.any(),
execute: (notice: TaskSettledInput) => {
heard.push(notice);
return {};
}
}),
defer: async () => defer()
})
}
}
})({ id: "defer-host" });
const state = createFlowState({ flows: { "defer-host": flow }, stores: { default: { primary: inMemoryStores() } } });
const act = async (actionName: string, input: unknown, internal = false) => {
const runtime = await state.getRuntime();
const result = await runAction({
orgId: DEFAULT_ORG_ID,
flow,
actionName,
input,
userId: USER,
sessionId: SESSION,
stores: runtime.stores,
runtimeConfig: runtime.runtimeConfig,
...(internal ? { source: "internal" } : {})
} as never);
if ((result as { error?: unknown }).error !== undefined) throw new Error(`${actionName}: ${JSON.stringify((result as { error?: unknown }).error)}`);
return (result as { output?: unknown }).output;
};
return { heard, act, dispose: () => state.dispose() };
}

const notice = { boardId: BOARD, taskId: "t1", attempt: 1, ending: "completed" as const, output: "fact" };

describe("a notice its session turns away (defer)", () => {
it("runs nothing, stays owed, and records when it was turned away", async () => {
const h = host(() => true);
try {
await h.act("seed", { id: "t1" });
const before = Date.now();
await h.act("onTaskSettled", notice, true);
expect(h.heard).toEqual([]);
const metadata = (await h.act("read", { id: "t1" })) as Record<string, { deferredAt?: number } | null>;
expect(metadata["noticeOwed:1:completed"]).toMatchObject({ attempt: 1, ending: "completed" });
expect(metadata["noticeOwed:1:completed"]!.deferredAt).toBeGreaterThanOrEqual(before);
} finally {
await h.dispose();
}
});

it("runs and is acknowledged when it isn't turned away", async () => {
const h = host(() => false);
try {
await h.act("seed", { id: "t1" });
await h.act("onTaskSettled", notice, true);
expect(h.heard).toEqual([expect.objectContaining({ taskId: "t1", ending: "completed" })]);
const metadata = (await h.act("read", { id: "t1" })) as Record<string, unknown>;
expect(metadata["noticeOwed:1:completed"] ?? null).toBeNull();
} finally {
await h.dispose();
}
});
});
30 changes: 30 additions & 0 deletions packages/orchestration/test/tasks/task-notice.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,18 @@ import {
cancelledDependents,
clearNotice,
decideNotice,
deferredNoticePatch,
deliveredByWait,
NOTICE_DUE_MS,
noticeDueAt,
isNoticeOwed,
noticeKey,
noticeTimes,
noticeText,
owedNotices,
recordEnding,
resendNowPatch,
resentRecord,
withoutNoticeMarkers,
type TaskNotice
} from "../../src/tasks/notice/task-notice";
Expand Down Expand Up @@ -108,6 +112,32 @@ describe("recordEnding: the row to write, with the notice owed", () => {
expect(noticeDueAt(row({ metadata: { "noticeOwed:1:completed": { attempt: 1, ending: "completed" } } }))).toBe(0);
});

it("drops an owed notice's resend backoff, so the next pass sends it at once (FIX-2113)", () => {
const written = recordEnding(row({ updatedAt: 10_000 }), { kind: "completed", output: 1 });
const notice = { attempt: 1, ending: "completed" } as const;
// Sent again three times: the next send backs off well past its due time.
const backedOff = { ...written, metadata: { ...written.metadata, ...resentRecord({ ...written, metadata: { ...written.metadata, "noticeResent:1:completed": { at: 20_000, n: 2 } } }, notice, 30_000) } };
expect(noticeDueAt(backedOff)).toBeGreaterThan(30_000);
const patch = resendNowPatch(backedOff);
expect(patch).toEqual({ "noticeResent:1:completed": null });
const now = { ...backedOff, metadata: { ...backedOff.metadata, ...patch } };
expect(noticeDueAt(now)).toBe(10_000 + NOTICE_DUE_MS);
// The owed marker itself stays; a row with no resend record needs nothing.
expect(isNoticeOwed(now, notice)).toBe(true);
expect(resendNowPatch(written)).toEqual({});
});

it("counts a turned-away notice's age from when it was turned away, not from its due time (FIX-2113)", () => {
const written = recordEnding(row({ updatedAt: 10_000 }), { kind: "completed", output: 1 });
const notice = { attempt: 1, ending: "completed" } as const;
expect(noticeTimes(written, notice)?.since).toBe(10_000 + NOTICE_DUE_MS);
const turnedAway = { ...written, metadata: { ...written.metadata, ...deferredNoticePatch(written, notice, 90_000) } };
expect(noticeTimes(turnedAway, notice)).toMatchObject({ due: 10_000 + NOTICE_DUE_MS, since: 90_000 });
// Still owed, and a row that owes nothing is left as it is.
expect(isNoticeOwed(turnedAway, notice)).toBe(true);
expect(deferredNoticePatch(row(), notice, 90_000)).toEqual({});
});

it("marks a park with its question (BR-25), and keeps it when the task completes later", () => {
const parked = recordEnding(row({ status: "parked" }), { kind: "parked", question: "Which region?", quiet: false });
const completed = recordEnding({ ...parked, status: "completed", attempts: 2, output: "eu" }, {
Expand Down
Loading
Loading