Skip to content
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
97f8a1c
FIX-2288: harnesses report spend as a run goes, keyed by run; the led…
claude Oct 28, 2026
20a34ac
FIX-2288: a task's run writes its spend onto its row; listTasks folds it
claude Oct 28, 2026
3664627
FIX-2288: docs, READMEs, changeset and the goal check for a task's spend
claude Oct 28, 2026
863f6f5
Merge #4013's head into the FIX-2288 branch
claude Oct 28, 2026
a6b4401
Merge #4013's README fix into the FIX-2288 branch
claude Oct 28, 2026
3b94e35
fix(FIX-2288): drop unused spend schema exports flagged by fallow
claude Oct 28, 2026
1c9eeba
fix(FIX-2288): address Codex review on task spend
claude Oct 28, 2026
506eaaf
fix(FIX-2288): address Codex's second review on task spend
claude Oct 28, 2026
b53003e
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude Oct 28, 2026
92c850c
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude Oct 28, 2026
3e56af1
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude Oct 29, 2026
0155ad9
fix(FIX-2288): keep a crashed run's harness tokens across its continu…
claude Oct 29, 2026
85cf7ce
fix(FIX-2288): keep each admission's spend apart, so a late ending ca…
claude Oct 29, 2026
2602b39
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude Oct 29, 2026
2393a27
test(FIX-2288): read the failed-result spend test's last report, as a…
claude Oct 29, 2026
a4b8552
chore(FIX-2288): drop the goal's unused LAB_USERS import
claude Oct 29, 2026
a79a2b5
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude Oct 29, 2026
8c632c2
fix(FIX-2288): keep a dead run's model calls its log never recorded
claude Oct 29, 2026
619af9b
chore: regenerate the Teammates lab's README inventory after main's m…
claude Oct 29, 2026
7380935
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude Oct 29, 2026
d9931c1
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude Oct 29, 2026
7a5d726
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude Oct 29, 2026
a0bc592
Merge #4013 (be92d6c5) into FIX-2288: logged harness runs, reported a…
claude Oct 29, 2026
1486bd7
FIX-2288: write a continuation's recovered spend at admission
claude Oct 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .changeset/task-spend.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
"@flow-state-dev/core": patch
"@flow-state-dev/engine": patch
"@flow-state-dev/claude-code": patch
"@flow-state-dev/codex": patch
"@flow-state-dev/cursor": patch
"@flow-state-dev/orchestration": patch
"@flow-state-dev/workforce": patch
---

A dispatched task records what its runs spent on its row (`spend`), every attempt counted, rising while a run works: the board writes it every 2 s and as the request ends, through the new `recordSpend` verb on `TaskCollectionRef`. `listTasks` from `taskToolActions` returns it folded (`taskSpendView`: harness, model, input and output tokens, `null` until reported). Harness runs report spend as they go, keyed by a `runId` with the model the run named, each report replacing the run's last; `readRequestSpend(ctx)` reads a running request's spend so far, and `requestSpendSources` moves from Workforce into core (FIX-2288).
25 changes: 25 additions & 0 deletions apps/docs/docs/orchestration/task-board.md
Original file line number Diff line number Diff line change
Expand Up @@ -677,6 +677,31 @@ The latest checkpoint is the task's output. When the task completes, a checkpoin

For a worker flow of your own, build the tool with `createReportProgress({ resolve })`, which takes the task list the turn's task is on, as [`createAskQuestion`](#a-tasks-questions) does. `reportProgress` is not an action: a screen doesn't report a worker's progress.

## What a task's run spent

A dispatched task's row carries what its runs spent: which harness and model ran it, and how many tokens it has used. A screen reads it from `listTasks` through [`taskToolActions`](#changing-tasks-from-outside-a-run), as `spend` on each row:

```ts
{ id: "t1", title: "Fix the login form", status: "in_progress",
run: { sessionId: "…", requestId: "…", attempt: 1 },
spend: { harness: "claude-code", model: "claude-sonnet-4-5",
inputTokens: 11_200, outputTokens: 1_240, unreported: false, live: true } }
```

| Field | What it holds |
|---|---|
| `harness` | The [coding agent](../tools/coding-agents.md) of the task's latest harness run, or `null` when none ran. |
| `model` | That run's model, as the harness reported it. A task with no harness run shows the model its worker last called. `null` until either is known. |
| `inputTokens` / `outputTokens` | Every request that worked the task, every attempt included: model calls and harness runs. `null` while nothing has reported tokens yet, never 0. |
| `unreported` | `true` while some harness run hasn't reported its tokens yet. Codex and Cursor report at the end of a turn, so a one-turn run shows its model from the start and its tokens only when the turn ends. |
| `live` | `true` while a request for the task is running. |

`spend` is `null` on a task that never ran in a request of its own.

The count rises while the run works. The request working the task writes what it has spent so far every two seconds while that changes, and writes the final figure when the request ends, however it ends. A failed or cancelled attempt keeps what it spent, and the next attempt adds to it. A worker's [usage](../workforce/worker-usage.md) is counted from the same figures, so a task's tokens and its worker's agree.

`spend` is on the action's rows only. The model's `listTasks` tool leaves it out, and nothing a model or a caller writes can change it. A task store of your own shows it once it implements `recordSpend`; without it, a task has no spend.

## Changing tasks from outside a run

Workers change tasks while a board drains. Sometimes a person needs to as well: cancel a task nobody needs, hold or release one, edit one before it starts, or answer its question. `taskToolActions` gives your flow the filer's four task tools, as actions any caller of the flow can run:
Expand Down
34 changes: 23 additions & 11 deletions apps/docs/docs/tools/coding-agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -459,49 +459,61 @@ and let the manager feed your harness its `onUsage`.

## What a run spent

When a run ends, however it ends, the harness block reports what it spent to the request it ran in: one `HarnessRunSpend` per run.
A harness block reports what its run has spent to the request it runs in, as the run goes: when it starts, each time the agent reports usage, and when the run ends, however it ends. Each report is a `HarnessRunSpend` carrying the whole run so far, and a later report for the same run replaces the earlier one, so a run counts once. A [task board](/docs/orchestration/task-board#what-a-tasks-run-spent) shows these reports on the task while it runs.

| Field | What it holds |
|---|---|
| `runId` | The harness's own id for the run. Reports with the same id replace each other. |
| `harness` | `claude-code`, `codex` or `cursor`. |
| `model` | The model the run uses: the one the agent named, else the one the run was started with. `null` when neither is known. |
| `inputTokens` / `outputTokens` | The tokens the agent reported, or `null` when it reported none, as for a run killed before its usage arrived. |
| `costUsd` | The cost the agent reported, in US dollars, or `null`. Never an estimate. |
| `billing` | `"api-key"`, `"subscription"`, or `null` when the harness can't tell. |

What each harness fills in:

| Harness | Tokens | `costUsd` | `billing` |
|---|---|---|---|
| Claude Code | From its result | The cost Claude Code reported | From Claude Code's own account info: `api-key` when it runs on a key, `subscription` when it is signed in to a plan, otherwise `null` |
| Codex | From its turns | Always `null` | `api-key` when you hand its client an `apiKey`, otherwise `null` |
| Cursor | Added up turn by turn, so a stopped run keeps the turns it finished | Always `null` | Always `null` |
| Harness | Tokens | `model` | `costUsd` | `billing` |
|---|---|---|---|---|
| Claude Code | Rising with each model message while it runs, then the whole query's count from its result | The one its first message names | The cost Claude Code reported | From Claude Code's own account info: `api-key` when it runs on a key, `subscription` when it is signed in to a plan, otherwise `null` |
| Codex | At the end of each turn, `null` until then | The model you started the run with; `null` when Codex picks its own default | Always `null` | `api-key` when you hand its client an `apiKey`, otherwise `null` |
| Cursor | Added up at the end of each turn, so a stopped run keeps the turns it finished; `null` until the first ends | The model you configured, or the one the run names | Always `null` | Always `null` |

A failed run reports the tokens it got before it failed. A run that failed before any usage arrived reports `null` tokens.

The report is a count kept apart from the request's model tokens: it never adds to `ctx.request.tokenUsage` or an action's [token budget](/docs/configuration/flow#token-budgets). Workforce records it per worker; see [Worker usage](/docs/workforce/worker-usage).

### Reporting from a harness of your own

Call `reportHarnessSpend` once per run, from a `finally`, with whatever the agent had sent by then:
Pick one `runId` per run, then call `reportHarnessSpend` when the run starts, each time the agent reports usage, and from a `finally`. Each call carries the run's total so far, never the change since the last one:

```ts
import { randomUUID } from "node:crypto";
import { reportHarnessSpend } from "@flow-state-dev/core/types";

const runId = `my-agent:${randomUUID()}`;
let usage: { input: number; output: number } | null = null;
try {
// ... run the agent, setting `usage` when it reports tokens
} finally {
const report = () =>
reportHarnessSpend(ctx, {
runId,
harness: "my-agent",
model: "my-model",
inputTokens: usage?.input ?? null,
outputTokens: usage?.output ?? null,
costUsd: null,
billing: null,
});

report(); // the model, before any tokens
try {
// ... run the agent; each time it reports usage, set `usage` to its total and call report()
} finally {
report();
}
```

It returns `false`, and counts nothing, when the context belongs to no request, such as a hand-built test context.
A report without a `runId` counts as one finished run of its own, so a harness that reports once, from its `finally`, still counts correctly; a task then shows its tokens only once the run ends.

`reportHarnessSpend` returns `false`, and counts nothing, when the context belongs to no request, such as a hand-built test context.

## What a run is doing

Expand Down
2 changes: 2 additions & 0 deletions apps/docs/docs/workforce/worker-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ When a request on that flow ends, however it ends, what it spent is added to the

Records belong to the user who made the request. One user never reads another's.

A [task's spend](../orchestration/task-board.md#what-a-tasks-run-spent) is counted from the same figures, per task rather than per worker and day. A task's tokens and what its worker's usage records for the same requests add up to the same numbers.

A worker whose flow declares `installation.session()` but leaves `installation.resources` out of `resources` has nowhere to write. In development, a request on it that spent tokens fails, naming the missing collection. In production, the request goes on and a warning is logged.

A record that fails to write is logged, and the request is not failed. Two requests that end at the same moment for the same worker both land.
Expand Down
105 changes: 105 additions & 0 deletions goals/shift-manager/a-task-shows-its-model-and-tokens/controls.mts
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
/**
* The goal's seven controls. Each is a scratch patch applied as the Lab under
* test loads the module (`goals/lib/module-patch.mjs`); nothing is written to
* the checkout. Every control must fail the assertions it names, or the check
* can't see what the control removes.
*/
import type { ModulePatch } from "../../lib/module-patch.mts";

export const CONTROL_NAMES = ["end-only", "zero-unreported", "last-attempt-only", "config-model", "from-traces", "add-result", "by-session"] as const;
export type ControlName = (typeof CONTROL_NAMES)[number];

/** One control: the assertions that must go red, and its patches. */
export interface Control {
name: ControlName;
assertions: string[];
patches: ModulePatch[];
does: string;
}

const ENGINE_CONTEXT = "packages/engine/src/context/createExecutionContext.ts";
const TASK_REQUEST = "packages/orchestration/src/task-board/task-request.ts";
const SPEND = "packages/orchestration/src/tasks/spend.ts";
const CLAUDE_CODE = "packages/claude-code/src/sdk/agent.ts";

export const CONTROLS: Record<ControlName, Control> = {
"end-only": {
name: "end-only",
assertions: ["b:rising"],
patches: [{ module: TASK_REQUEST, from: String.raw`export const SPEND_TICK_MS = 2_000;`, to: "export const SPEND_TICK_MS = 3_600_000;" }],
does: "no spend is written while a run works: the row moves only at a request's end",
},
"zero-unreported": {
name: "zero-unreported",
assertions: ["c:null"],
patches: [
Comment thread
jhoffner marked this conversation as resolved.
Resolved
{ module: SPEND, from: String.raw`inputTokens: reportedAny \? input : null,`, to: "inputTokens: input," },
{ module: SPEND, from: String.raw`outputTokens: reportedAny \? output : null,`, to: "outputTokens: output," },
],
does: "tokens nothing has reported read as 0",
},
"last-attempt-only": {
name: "last-attempt-only",
assertions: ["d:1500"],
patches: [{ module: SPEND, from: String.raw`for \(const entry of requests\) \{`, to: "for (const entry of requests.slice(-1)) {" }],
does: "a task counts only its latest request, as a run link names only the latest attempt",
},
"config-model": {
name: "config-model",
assertions: ["b:model"],
patches: [
{
module: CLAUDE_CODE,
from: String.raw`if \(typeof message\.model === "string" && message\.model !== ""\) spendModel = message\.model;`,
to: "void 0;",
},
],
does: "the model shown is the one the worker's harness is configured with, not the one the run named",
},
"from-traces": {
name: "from-traces",
assertions: ["e:same-numbers"],
patches: [
{
module: ENGINE_CONTEXT,
from: String.raw`onGeneratorModelResult: \(payload\) => requestSpendLedger\.addModelResult\(payload\),`,
to: "onGeneratorModelResult: (payload) => { if (isTraceObservabilityEnabled()) requestSpendLedger.addModelResult(payload); },",
},
],
does: "model calls are counted only where traces carry usage: never in production",
},
"add-result": {
name: "add-result",
assertions: ["b:exact", "f:equal"],
patches: [
{
module: SPEND,
from: String.raw`addRequestSpendSource\(all, source\) as TaskSpendSource\[\], was\.settled\)`,
to: "addRequestSpendSource(all, source) as TaskSpendSource[], [...was.settled, ...(was.live ?? [])])",
},
],
does: "an ending's spend is added on top of the running total the row already shows",
},
"by-session": {
name: "by-session",
assertions: ["g:own-only"],
patches: [
{
module: TASK_REQUEST,
// A ticket made for each row, as a writer that targets rows by worker would: the row's own fence can't tell.
from: String.raw`await recordSpend\.call\(record\.ref, record\.ticket\.taskId, \{ requestId, part, spend, writer: record\.writer, admittedAt: record\.admittedAt \}, \{ claim: record\.ticket \}\);`,
to: 'for (const t of record.ref.list().filter((t) => t.status === "in_progress" && t.assignee === record.ref.get(record.ticket.taskId)?.assignee)) await recordSpend.call(record.ref, t.id, { requestId, part, spend, writer: record.writer, admittedAt: record.admittedAt }, { claim: { ...record.ticket, taskId: t.id, createdAt: t.createdAt, incarnationId: t.incarnationId } });',
},
],
does: "a request's spend is written to every task its worker is running, not to the one it was admitted for",
},
};

/** Where the harness loads the Agent SDK from: pointed at the scripted one. */
export function sdkSeam(scriptedSdkUrl: string): ModulePatch {
return {
module: "packages/claude-code/src/sdk/sdk-client.ts",
from: String.raw`const SDK_MODULE = "@anthropic-ai/claude-agent-sdk";`,
to: `const SDK_MODULE = ${JSON.stringify(scriptedSdkUrl)};`,
};
}
Loading
Loading