Repository navigation
FIX-2288: a task's run reports its model, harness and live token count (stacked on FIX-2145) #4149
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
jhoffner
merged 24 commits into
claude/project-thread-w0un9y
from
claude/project-thread-ldiw1v
Oct 29, 2026
Merged
Changes from 17 commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
97f8a1c
FIX-2288: harnesses report spend as a run goes, keyed by run; the led…
claude 20a34ac
FIX-2288: a task's run writes its spend onto its row; listTasks folds it
claude 3664627
FIX-2288: docs, READMEs, changeset and the goal check for a task's spend
claude 863f6f5
Merge #4013's head into the FIX-2288 branch
claude a6b4401
Merge #4013's README fix into the FIX-2288 branch
claude 3b94e35
fix(FIX-2288): drop unused spend schema exports flagged by fallow
claude 1c9eeba
fix(FIX-2288): address Codex review on task spend
claude 506eaaf
fix(FIX-2288): address Codex's second review on task spend
claude b53003e
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude 92c850c
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude 3e56af1
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude 0155ad9
fix(FIX-2288): keep a crashed run's harness tokens across its continu…
claude 85cf7ce
fix(FIX-2288): keep each admission's spend apart, so a late ending ca…
claude 2602b39
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude 2393a27
test(FIX-2288): read the failed-result spend test's last report, as a…
claude a4b8552
chore(FIX-2288): drop the goal's unused LAB_USERS import
claude a79a2b5
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude 8c632c2
fix(FIX-2288): keep a dead run's model calls its log never recorded
claude 619af9b
chore: regenerate the Teammates lab's README inventory after main's m…
claude 7380935
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude d9931c1
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude 7a5d726
Merge remote-tracking branch 'origin/claude/project-thread-w0un9y' in…
claude a0bc592
Merge #4013 (be92d6c5) into FIX-2288: logged harness runs, reported a…
claude 1486bd7
FIX-2288: write a continuation's recovered spend at admission
claude File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| "@flow-state-dev/core": patch | ||
| "@flow-state-dev/engine": patch | ||
| "@flow-state-dev/claude-code": patch | ||
| "@flow-state-dev/codex": patch | ||
| "@flow-state-dev/cursor": patch | ||
| "@flow-state-dev/orchestration": patch | ||
| "@flow-state-dev/workforce": patch | ||
| --- | ||
|
|
||
| A dispatched task records what its runs spent on its row (`spend`), every attempt counted, rising while a run works: the board writes it every 2 s and as the request ends, through the new `recordSpend` verb on `TaskCollectionRef`. `listTasks` from `taskToolActions` returns it folded (`taskSpendView`: harness, model, input and output tokens, `null` until reported). Harness runs report spend as they go, keyed by a `runId` with the model the run named, each report replacing the run's last; `readRequestSpend(ctx)` reads a running request's spend so far, and `requestSpendSources` moves from Workforce into core (FIX-2288). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
105 changes: 105 additions & 0 deletions
105
goals/shift-manager/a-task-shows-its-model-and-tokens/controls.mts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,105 @@ | ||
| /** | ||
| * The goal's seven controls. Each is a scratch patch applied as the Lab under | ||
| * test loads the module (`goals/lib/module-patch.mjs`); nothing is written to | ||
| * the checkout. Every control must fail the assertions it names, or the check | ||
| * can't see what the control removes. | ||
| */ | ||
| import type { ModulePatch } from "../../lib/module-patch.mts"; | ||
|
|
||
| export const CONTROL_NAMES = ["end-only", "zero-unreported", "last-attempt-only", "config-model", "from-traces", "add-result", "by-session"] as const; | ||
| export type ControlName = (typeof CONTROL_NAMES)[number]; | ||
|
|
||
| /** One control: the assertions that must go red, and its patches. */ | ||
| export interface Control { | ||
| name: ControlName; | ||
| assertions: string[]; | ||
| patches: ModulePatch[]; | ||
| does: string; | ||
| } | ||
|
|
||
| const ENGINE_CONTEXT = "packages/engine/src/context/createExecutionContext.ts"; | ||
| const TASK_REQUEST = "packages/orchestration/src/task-board/task-request.ts"; | ||
| const SPEND = "packages/orchestration/src/tasks/spend.ts"; | ||
| const CLAUDE_CODE = "packages/claude-code/src/sdk/agent.ts"; | ||
|
|
||
| export const CONTROLS: Record<ControlName, Control> = { | ||
| "end-only": { | ||
| name: "end-only", | ||
| assertions: ["b:rising"], | ||
| patches: [{ module: TASK_REQUEST, from: String.raw`export const SPEND_TICK_MS = 2_000;`, to: "export const SPEND_TICK_MS = 3_600_000;" }], | ||
| does: "no spend is written while a run works: the row moves only at a request's end", | ||
| }, | ||
| "zero-unreported": { | ||
| name: "zero-unreported", | ||
| assertions: ["c:null"], | ||
| patches: [ | ||
| { module: SPEND, from: String.raw`inputTokens: reportedAny \? input : null,`, to: "inputTokens: input," }, | ||
| { module: SPEND, from: String.raw`outputTokens: reportedAny \? output : null,`, to: "outputTokens: output," }, | ||
| ], | ||
| does: "tokens nothing has reported read as 0", | ||
| }, | ||
| "last-attempt-only": { | ||
| name: "last-attempt-only", | ||
| assertions: ["d:1500"], | ||
| patches: [{ module: SPEND, from: String.raw`for \(const entry of requests\) \{`, to: "for (const entry of requests.slice(-1)) {" }], | ||
| does: "a task counts only its latest request, as a run link names only the latest attempt", | ||
| }, | ||
| "config-model": { | ||
| name: "config-model", | ||
| assertions: ["b:model"], | ||
| patches: [ | ||
| { | ||
| module: CLAUDE_CODE, | ||
| from: String.raw`if \(typeof message\.model === "string" && message\.model !== ""\) spendModel = message\.model;`, | ||
| to: "void 0;", | ||
| }, | ||
| ], | ||
| does: "the model shown is the one the worker's harness is configured with, not the one the run named", | ||
| }, | ||
| "from-traces": { | ||
| name: "from-traces", | ||
| assertions: ["e:same-numbers"], | ||
| patches: [ | ||
| { | ||
| module: ENGINE_CONTEXT, | ||
| from: String.raw`onGeneratorModelResult: \(payload\) => requestSpendLedger\.addModelResult\(payload\),`, | ||
| to: "onGeneratorModelResult: (payload) => { if (isTraceObservabilityEnabled()) requestSpendLedger.addModelResult(payload); },", | ||
| }, | ||
| ], | ||
| does: "model calls are counted only where traces carry usage: never in production", | ||
| }, | ||
| "add-result": { | ||
| name: "add-result", | ||
| assertions: ["b:exact", "f:equal"], | ||
| patches: [ | ||
| { | ||
| module: SPEND, | ||
| from: String.raw`addRequestSpendSource\(all, source\) as TaskSpendSource\[\], was\.settled\)`, | ||
| to: "addRequestSpendSource(all, source) as TaskSpendSource[], [...was.settled, ...(was.live ?? [])])", | ||
| }, | ||
| ], | ||
| does: "an ending's spend is added on top of the running total the row already shows", | ||
| }, | ||
| "by-session": { | ||
| name: "by-session", | ||
| assertions: ["g:own-only"], | ||
| patches: [ | ||
| { | ||
| module: TASK_REQUEST, | ||
| // A ticket made for each row, as a writer that targets rows by worker would: the row's own fence can't tell. | ||
| from: String.raw`await recordSpend\.call\(record\.ref, record\.ticket\.taskId, \{ requestId, part, spend, writer: record\.writer, admittedAt: record\.admittedAt \}, \{ claim: record\.ticket \}\);`, | ||
| to: 'for (const t of record.ref.list().filter((t) => t.status === "in_progress" && t.assignee === record.ref.get(record.ticket.taskId)?.assignee)) await recordSpend.call(record.ref, t.id, { requestId, part, spend, writer: record.writer, admittedAt: record.admittedAt }, { claim: { ...record.ticket, taskId: t.id, createdAt: t.createdAt, incarnationId: t.incarnationId } });', | ||
| }, | ||
| ], | ||
| does: "a request's spend is written to every task its worker is running, not to the one it was admitted for", | ||
| }, | ||
| }; | ||
|
|
||
| /** Where the harness loads the Agent SDK from: pointed at the scripted one. */ | ||
| export function sdkSeam(scriptedSdkUrl: string): ModulePatch { | ||
| return { | ||
| module: "packages/claude-code/src/sdk/sdk-client.ts", | ||
| from: String.raw`const SDK_MODULE = "@anthropic-ai/claude-agent-sdk";`, | ||
| to: `const SDK_MODULE = ${JSON.stringify(scriptedSdkUrl)};`, | ||
| }; | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.