Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .changeset/task-ends-for-good.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
"@flow-state-dev/orchestration": patch
"@flow-state-dev/harness-manager": minor
---

A worker can now end its task on the current attempt without writing the ending itself: throw `TaskEndsForGoodError` (from `@flow-state-dev/orchestration/task-board`) naming the task it holds (`taskId`), and the board's error recorder ends that row under the worker's claim, `errored` with no retry by default or `cancelled` with `{ ending: "cancelled" }`. A marker naming any other row, such as one rethrown by a board nested inside the worker, is an ordinary failure.

Harness Manager's refused run uses it: the manager no longer fails or cancels its own row, and the board records the ending. The outcome is unchanged (`errored` on a board read from the claim, `cancelled` on the fixed board). `HarnessRunRefused` now extends `TaskEndsForGoodError`, and its constructor takes a second argument, `{ taskId, ending }` (FIX-2090).
9 changes: 9 additions & 0 deletions apps/docs/docs/orchestration/task-board.md
Original file line number Diff line number Diff line change
Expand Up @@ -704,6 +704,15 @@ waits.
- `maxTotalRetries` (default `50`) — how many failure retries the board may authorize in total, across every task. See [Bounding the retries](#bounding-the-retries).
- `maxIterations` — safety cap on how many times a single worker loops back to claim again, not a cap across the board. Default `10000`.

Some failures would come back the same on every attempt, such as a run refused because its task names nothing it may work on. Retrying those only spends the attempts. A worker that knows this throws `TaskEndsForGoodError`, exported from `@flow-state-dev/orchestration/task-board`, naming the task it is working on. The board ends that task on this attempt, with the error's message as its reason. It ends `errored` by default, or `cancelled` if you construct the error with `{ ending: "cancelled" }`. Dependents and filed work are cascaded as for any other ending. The error only counts for the task it names. If a board inside your worker rethrows one about its own task, your task treats it as an ordinary failure and keeps its retries.

```ts
import { TaskEndsForGoodError } from "@flow-state-dev/orchestration/task-board";

// in a worker: input.taskId is the task this worker is running
throw new TaskEndsForGoodError("the task names no project", { taskId: input.taskId });
```

`onError` reaches an [assignee that dispatches](#assignees-that-dispatch) too, where there is no board run left to fail. `"fail"` fails the dispatch run that worker is running in, so it reports `failed`. `"skip"` leaves it reporting `completed`, with the error on the task as usual. See [What `status` tells you](../server/background-work#what-status-tells-you).

A worker's result is not always the last word on its task. A coordinator can cancel the task while the worker runs. The worker can mark the task done itself partway through. The claim can expire and another worker can pick the task up. In each case the worker comes back with a result for a task that has already moved on.
Expand Down
57 changes: 21 additions & 36 deletions packages/harness-manager/src/manager.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ import type {
HarnessResolver,
HarnessSessionHook,
} from "@flow-state-dev/core/types";
import { taskWorkerInputSchema } from "@flow-state-dev/orchestration/task-board";
import { TaskEndsForGoodError, taskWorkerInputSchema } from "@flow-state-dev/orchestration/task-board";
import {
getOrCreateTaskCollection,
hasFrozenLedgerAssignee,
Expand Down Expand Up @@ -751,18 +751,12 @@ export class HarnessAttemptFailed extends Error {

/**
* The run's workspace refused it: its source answered `refused`, or the host
* would not provision what the source named.
*
* Not a failed attempt. The same source and host give the same answer next
* time, so a retry would spend every attempt on one refusal. The manager
* settles the row with this message as its reason, under the attempt's own
* claim, and the board's recorder then has nothing to re-queue: `cancelled`
* on the fixed board, `errored` on a board read from the claim, where the
* conversation that filed the row reports it.
* would not provision what the source named. The same answer next time, so
* the recorder ends the row on this attempt instead of retrying.
*/
export class HarnessRunRefused extends Error {
constructor(message: string) {
super(message);
export class HarnessRunRefused extends TaskEndsForGoodError {
constructor(message: string, options: { taskId: string; ending: "errored" | "cancelled" }) {
super(message, options);
this.name = "HarnessRunRefused";
}
}
Expand Down Expand Up @@ -884,6 +878,7 @@ function taskPayload(
`[harness-manager] task ${input.taskId} names only part of a feature payload: ` +
`${parsed.error.issues.map((i) => `${i.path.join(".")} ${i.message}`).join("; ")}. ` +
`Feature work names both an issue and a phase.`,
{ taskId: input.taskId, ending: "errored" },
);
}
return undefined;
Expand Down Expand Up @@ -936,6 +931,15 @@ function harnessCtxState(ctx: {
return ctx.sequencer?.state as ManagerState | undefined;
}

/** This run's refusal of the row it holds. Claim board: `errored` (a cancel there reaches nobody). Fixed board: `cancelled`. */
function runRefused(ctx: { sequencer?: { state?: unknown } | undefined }, message: string): HarnessRunRefused {
const state = harnessCtxState(ctx);
return new HarnessRunRefused(message, {
taskId: state?.taskId ?? "",
ending: state?.claim != null ? "errored" : "cancelled",
});
}

/** Read the manager's state, or throw rather than derive a wrong directory. */
function managerState(state: ManagerState | undefined): ManagerState {
if (state?.workspacePath == null) {
Expand Down Expand Up @@ -1707,7 +1711,7 @@ export function harnessManager(options: ManagerOptions): HarnessManager | TaskWo
} catch (cause) {
if (cause instanceof WorkspaceRefusedError) {
if (holding) await recordPlace(ctx, before!, "refused").catch(() => undefined);
throw new HarnessRunRefused(refusedMessage("host", cause));
throw runRefused(ctx, refusedMessage("host", cause));
}
if (cause instanceof HeldWorkMismatchError) await parkOnMismatch(ctx, cause, before);
throw cause;
Expand Down Expand Up @@ -1927,7 +1931,7 @@ export function harnessManager(options: ManagerOptions): HarnessManager | TaskWo
held: recorded?.held ?? null,
});
const answer = pinnedAnswer(await host.source(ctx), pinned);
if (answer.kind === "refused") throw new HarnessRunRefused(refusedMessage("source", answer));
if (answer.kind === "refused") throw runRefused(ctx, refusedMessage("source", answer));
const workspacePath = host.locate(answer, { place: placeFor(location) }).cwd;
await ctx.sequencer!.patchState({ workspacePath });

Expand Down Expand Up @@ -2124,7 +2128,7 @@ export function harnessManager(options: ManagerOptions): HarnessManager | TaskWo
state.branch,
);
const answer = pinnedAnswer(await host.source(ctx), state);
if (answer.kind === "refused") throw new HarnessRunRefused(refusedMessage("source", answer));
if (answer.kind === "refused") throw runRefused(ctx, refusedMessage("source", answer));
if (host.locate(answer, { place: placeFor(location) }).cwd !== state.workspacePath) {
throw new HarnessAttemptFailed(
`[harness-manager] the run's workspace source changed between opening the run and ` +
Expand Down Expand Up @@ -2583,27 +2587,8 @@ export function harnessManager(options: ManagerOptions): HarnessManager | TaskWo
{ outcome: "failed", reason },
Comment thread
jhoffner marked this conversation as resolved.
Resolved
);
}
// **A refused run is settled, not retried** (see `HarnessRunRefused`).
// Under this attempt's own claim, so a superseded attempt cannot cancel
// a row its replacement holds. Best effort, like the write above: the
// error being unwound is what the caller gets either way.
//
// On a board read from the claim the row fails for good instead: a
// cancelled row there reaches nobody, and an `errored` one is what the
// conversation that filed it reports. The fixed board keeps the cancel.
if (error instanceof HarnessRunRefused && state?.taskId != null && state.attempt != null) {
try {
const tasks = await runBoard(ctx as BlockContext);
const row = tasks.get(state.taskId);
if (row !== undefined) {
const claim = { ...ticketForClaim(tasks.collectionId, row, tasks.partition), attempt: state.attempt };
if (state.claim != null) await tasks.fail(state.taskId, reason, { claim, noRetry: true });
else await tasks.cancel(state.taskId, reason, { claim });
}
} catch {
// Left to the board's recorder, which re-queues it as before.
}
}
// A refusal's task row is ended by the recorder, not retried (see
// `HarnessRunRefused`); this handler updates only the run record above.
throw error;
},
});
Expand Down
11 changes: 11 additions & 0 deletions packages/harness-manager/test/claim-board-refused.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,17 @@ describe("a refused run on a board read from the claim", () => {
expect(row.error).toContain("the task names no project");
expect(refusing.seen).toHaveLength(0);
}, 60_000);

it("cancels a task that depends on the refused row, as any failed task's dependent is", async () => {
const lab = claimHost({ mode: "claim" });
await lab.seed({ ledgerId: CONVERSATIONS, partition: "conv-a", id: "t1", goal: "half a feature", input: { issue: "FIX-1" }, maxAttempts: 2 });
await lab.seed({ ledgerId: CONVERSATIONS, partition: "conv-a", id: "t2", goal: "after it", input: {}, deps: ["t1"] });

await lab.runAttempt(CONVERSATIONS, "t1", "conv-a");

expect(await lab.row(CONVERSATIONS, "t1", "conv-a")).toMatchObject({ status: "errored", attempts: 1 });
expect((await lab.row(CONVERSATIONS, "t2", "conv-a")).status).toBe("cancelled");
}, 60_000);
});

describe("a refused run on a fixed board", () => {
Expand Down
2 changes: 2 additions & 0 deletions packages/harness-manager/test/claim-host.ts
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,7 @@ export function claimHost(options: HostOptions) {
goal: z.string(),
input: z.unknown().optional(),
maxAttempts: z.number().optional(),
deps: z.array(z.string()).optional(),
}),
outputSchema: z.unknown(),
uses: [ledgerResources],
Expand All @@ -239,6 +240,7 @@ export function claimHost(options: HostOptions) {
goal: input.goal,
assignee: ASSIGNEE,
...(input.input !== undefined ? { input: input.input } : {}),
...(input.deps !== undefined ? { deps: input.deps } : {}),
maxAttempts: input.maxAttempts ?? 2,
});
}
Expand Down
9 changes: 9 additions & 0 deletions packages/orchestration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -410,6 +410,15 @@ reports `terminationReason: "retry-budget-exhausted"` alongside `counts.retries`
the limit in force. See the
[Task board guide](https://flow-state.dev/docs/orchestration/task-board).

**A failure no retry can change.** A worker that knows the next attempt would end the
same way throws `TaskEndsForGoodError` (from `@flow-state-dev/orchestration/task-board`)
instead of writing the ending itself, naming the task it holds (`{ taskId }`). The
board's error recorder then ends that row on this attempt, under the worker's claim,
with the error's message as its reason: `errored` with attempts left unspent (`fail`
with `noRetry`), or `cancelled` when constructed with `{ taskId, ending: "cancelled" }`.
A marker naming another row (one rethrown by a board nested inside the worker) and any
other error take the ordinary retry route.

**When the board cannot record a result.** Saving a result is a commit followed by a
change announcement, and the announcement can fail on its own. The board's two
recorders correlate every result write (`beginTaskWrite` / `didWriteLand`, above) and
Expand Down
68 changes: 55 additions & 13 deletions packages/orchestration/src/task-board/blocks/record-result.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@
* sequencer. Receives the caught error as input, reads
* `currentClaim` from worker state, calls `collection.fail`. Honors
* `onError`: `"skip"` swallows the error after writing the failure;
* `"fail"` rethrows so the parent forEach rejects.
* `"fail"` rethrows so the parent forEach rejects. A
* {@link TaskEndsForGoodError} ends the row on this attempt instead of
* re-queueing it: `fail` with `noRetry`, or `cancel`.
*
* Both write-backs are **advisory** (FIX-951): they pass `ifAllowed` and the
* worker's `claim`, so a result that arrives after the task was settled by
Expand Down Expand Up @@ -73,6 +75,7 @@ import {
type TaskWriteToken,
} from "../../tasks";
import {
advisoryCancel,
advisoryComplete,
advisoryFail,
} from "../../tasks/collection/advisory-write-back";
Expand Down Expand Up @@ -208,6 +211,27 @@ function readTaskQuietly(
}
}

/**
* Thrown when the next attempt would end the same way. The recorder ends the
* row on this attempt, under the worker's claim, so the worker writes no
* ending of its own. `ending` defaults to `errored`.
*
* It speaks only for `taskId`, the row its thrower holds: a recorder holding
* another row (a board nested inside the worker rethrows its worker's error on
* `onError: "fail"`) reads it as an ordinary failure.
*/
export class TaskEndsForGoodError extends Error {
readonly taskId: string;
readonly ending: "errored" | "cancelled";

constructor(message: string, options: { taskId: string; ending?: "errored" | "cancelled" }) {
super(message);
this.name = "TaskEndsForGoodError";
this.taskId = options.taskId;
this.ending = options.ending ?? "errored";
}
}

function messageOf(err: unknown): string {
return err instanceof Error ? err.message : String(err);
}
Expand Down Expand Up @@ -279,6 +303,8 @@ async function writeRethrowingIfNothingSaved(params: {
recorder: RecorderKind;
runId: RecorderFailureWiring["runId"];
attempt: (write: TaskWriteToken) => Promise<unknown>;
/** The row's failure is final, so a release ends it rather than re-queueing it. */
noRetry?: boolean;
}): Promise<RecorderFailureReport | undefined> {
const { ctx, collection, claim, recorder, attempt } = params;
const write = beginTaskWrite(readTaskQuietly(collection, claim.taskId));
Expand All @@ -304,7 +330,7 @@ async function writeRethrowingIfNothingSaved(params: {
landed === true ? "committed" : "undetermined";

if (verdict === "undetermined") {
await releaseRow(collection, claim, err);
await releaseRow(collection, claim, err, params.noRetry === true);
}

const runId = params.runId?.(ctx);
Expand All @@ -331,7 +357,8 @@ async function writeRethrowingIfNothingSaved(params: {
*
* One fenced write serves both halves of the uncertainty: if the original write
* did commit, the claim fence declines this one harmlessly; if it did not, this
* settles the row (or re-queues it, where retries remain).
* settles the row (or re-queues it, where retries remain). A row whose
* worker said its failure is final (`noRetry`) is not re-queued here either.
*
* Best-effort by design. Nothing is hidden by a failure here — the report still
* goes out and the run still fails — and letting a failed *recovery* displace
Expand All @@ -340,14 +367,15 @@ async function writeRethrowingIfNothingSaved(params: {
async function releaseRow(
collection: TaskCollectionRef,
claim: TaskClaimTicket,
err: unknown
err: unknown,
noRetry: boolean
): Promise<void> {
try {
await advisoryFail(
collection,
claim.taskId,
`task board could not confirm its own write: ${messageOf(err)}`,
{ ifAllowed: true, claim, refuseWhenParked: true }
{ ifAllowed: true, claim, refuseWhenParked: true, ...(noRetry ? { noRetry: true } : {}) }
);
} catch {
/* see above */
Expand Down Expand Up @@ -494,6 +522,11 @@ export function createRecordError(options: RecordErrorOptions) {
execute: async (error: unknown, ctx) => {
const claim = ctx.sequencer!.state.currentClaim;
const message = error instanceof Error ? error.message : String(error);
const forGood =
error instanceof TaskEndsForGoodError && claim !== undefined && error.taskId === claim.taskId
? error.ending
: undefined;
const recorded = forGood === "cancelled" ? ("cancelled" as const) : ("errored" as const);

// A report that could not be DELIVERED is fatal wherever it happens, and
// is never written to the row or swallowed by `onError`. The deferral
Expand Down Expand Up @@ -556,13 +589,22 @@ export function createRecordError(options: RecordErrorOptions) {
claim,
recorder: "fail",
runId: wiring.runId,
...(forGood !== undefined ? { noRetry: true } : {}),
attempt: (write) =>
advisoryFail(collection, claim.taskId, message, {
ifAllowed: true,
claim,
refuseWhenParked: true,
write,
}),
forGood === "cancelled"
? advisoryCancel(collection, claim.taskId, message, {
ifAllowed: true,
claim,
refuseWhenParked: true,
write,
})
: advisoryFail(collection, claim.taskId, message, {
ifAllowed: true,
claim,
refuseWhenParked: true,
write,
...(forGood === "errored" ? { noRetry: true } : {}),
}),
});
}
await ctx.sequencer!.patchState({ currentClaim: undefined, currentLedger: undefined });
Expand All @@ -586,13 +628,13 @@ export function createRecordError(options: RecordErrorOptions) {
// `onError`'s throw here instead would reject the fan-out and abandon
// every task that had not started, which is the defect this branch
// exists to remove.
return { recorded: "errored" as const, error: message };
return { recorded, error: message };
}

if (onError === "fail") {
throw error instanceof Error ? error : new Error(message);
}
return { recorded: "errored" as const, error: message };
return { recorded, error: message };
},
});
}
1 change: 1 addition & 0 deletions packages/orchestration/src/task-board/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,7 @@ export type { BuildWorkerStepOptions } from "./blocks/worker-step";
export {
createRecordSuccess,
createRecordError,
TaskEndsForGoodError,
} from "./blocks/record-result";
export type {
RecordSuccessOptions,
Expand Down
Loading
Loading