diff --git a/.changeset/claude-code-usage-left.md b/.changeset/claude-code-usage-left.md new file mode 100644 index 0000000000..289867ae17 --- /dev/null +++ b/.changeset/claude-code-usage-left.md @@ -0,0 +1,9 @@ +--- +"@flow-state-dev/core": patch +"@flow-state-dev/claude-code": patch +"@flow-state-dev/codex": patch +"@flow-state-dev/harness-manager": minor +"@flow-state-dev/shift-manager": patch +--- + +See how much of Claude Code's subscription is used. Core's `HarnessUsageWindow` gains `level` (`ok`, `warning`, `reached`, or `null`) and its `usedPercent` may be `null` when a harness sends no percent; core adds `HarnessUsageHook` and `HarnessUsageReport`, the run-time twin of `HarnessSessionHook`. `claudeCodeAgent` takes `onUsage` and hands it each window a run's `rate_limit_event` reports, with the run's account; `readClaudeUsage` reads who is signed in for the usage query, and `probeClaudeAccount` takes a `signal` that stops and closes the probe, and marks a `none` it got no answer for with `unanswered: true`. `@flow-state-dev/harness-manager` feeds `onUsage` at both of its harness slots (`HarnessFeeds.onUsage`, now required, so code that builds `HarnessFeeds` itself must add it), writing into the run owner's usage record, and keeps a stored record's account and windows together across a sign-in change. Codex windows now carry `level: null`. Shift Manager's Usage view shows Claude Code's windows, saying "limit reached" or "under Claude Code's warning line" where there is no percent (FIX-2142). diff --git a/apps/docs/docs/orchestration/harness-manager.md b/apps/docs/docs/orchestration/harness-manager.md index e30bb163b5..d6e0e5aa66 100644 --- a/apps/docs/docs/orchestration/harness-manager.md +++ b/apps/docs/docs/orchestration/harness-manager.md @@ -64,7 +64,7 @@ To let a user type into a run's session, build the door with `harnessSessionDoor ## Choosing the harness -The harness is a slot. You pass a factory; the manager calls it once and hands it three things. +The harness is a slot. You pass a factory; the manager calls it once and hands it four things. ```ts import { harnessManager } from "@flow-state-dev/harness-manager"; @@ -77,8 +77,8 @@ const manager = harnessManager({ phase: implementPhase(), workspace: { root, sourceRepo, baseRef }, runTimeoutMs: 30 * 60_000, - harness: ({ cwd, resume, onSession }) => - claudeCodeAgent({ cwd, resume, onSession, detached: true, recordWork: true }), + harness: ({ cwd, resume, onSession, onUsage }) => + claudeCodeAgent({ cwd, resume, onSession, onUsage, detached: true, recordWork: true }), }); ``` @@ -97,11 +97,11 @@ Code needs `detached: true`: it keeps session state by default, and a dispatched row refuses a block that does. Codex keeps none, so it needs no equivalent. Each spells its sandbox its own way. None of it reaches the manager. -**What the manager hands down.** `cwd` says where this run works: the checkout the manager derived and provisioned. `resume` says which conversation this attempt continues, or `null` for a fresh one. `onSession` is what the harness calls when it names its session, so the manager can record it. +**What the manager hands down.** `cwd` says where this run works: the checkout the manager derived and provisioned. `resume` says which conversation this attempt continues, or `null` for a fresh one. `onSession` is what the harness calls when it names its session, so the manager can record it. `onUsage` is what the harness calls with each subscription usage window its run reports, so the manager can store it in the run owner's [usage record](#subscription-usage). A harness that reports no usage, such as Codex, has no use for it. **What the manager never does.** It does not read anything vendor-specific off the handle, and it does not tell the harness what model to use, what tools to allow, or how to sandbox itself. Those are yours, written inside the factory. -The three feeds are declared in `@flow-state-dev/core`. Any block that takes them and returns a conforming run handle is one this manager can drive. [Coding agents](/docs/tools/coding-agents) covers the contract in full. +`HarnessFeeds`, exported from `@flow-state-dev/harness-manager`, types the four feeds, and each signature is declared in `@flow-state-dev/core`. Any block that takes them and returns a conforming run handle is one this manager can drive. [Coding agents](/docs/tools/coding-agents) covers the contract in full. ## What a phase is @@ -271,7 +271,7 @@ harness, sorted by name: reason: null, // why there is no number, for the other three plan: "plus", windows: [ - { id: "codex:primary", label: null, minutes: 300, usedPercent: 42, resetsAt: 1798020000000, + { id: "codex:primary", label: null, minutes: 300, usedPercent: 42, level: null, resetsAt: 1798020000000, checkedAt: 1798012000000, origin: "read", ageMs: 65000, rolledOver: false }, ], checkedAt: 1798012000000, // when the stored reading was made, or null @@ -281,8 +281,33 @@ harness, sorted by name: ``` A window's `origin` says how it arrived: `"read"` when a reader fetched it on a -refresh. A harness that reports usage during its runs stores its windows through -`recordHarnessUsage` with an origin of its own. +refresh, `"run"` when a run reported it through `onUsage`. Each window carries +its own `checkedAt` and `ageMs`. The row's `checkedAt` and `ageMs` belong to the +last reading, so a harness that has only run windows and was never read has a +row `checkedAt` of `null`, and each window says how old it is. + +**Run windows land on their own.** Pass the slot's `onUsage` through to a +harness that reports usage, as the Claude Code example in +[Choosing the harness](#choosing-the-harness) does, and every window a +run reports is stored for the user the run belongs to. A line sent through the +[session door](#the-session-door) stores its run's windows for the user +talking. For Claude Code, register +[`readClaudeUsage`](/docs/tools/claude-code-sdk#reading-usage-left) as the +`claude-code` reader. It supplies the account and plan, and the windows come +from runs: + +```ts +import { readClaudeUsage } from "@flow-state-dev/claude-code/sdk"; + +const claudeEnv = harnessEnv({ pass: ["PATH", "HOME", "CLAUDE_CODE_OAUTH_TOKEN"] }); + +harnessUsageAction({ + readers: { + codex: (options) => readCodexUsage({ env, ...options }), + "claude-code": (options) => readClaudeUsage({ env: claudeEnv, ...options }), + }, +}); +``` **Without `refresh`, nothing is read.** The query returns what is stored. A harness with a reader and no reading yet is listed with `status: "unavailable"`, @@ -300,19 +325,27 @@ query runs, by the server's clock. A window whose `resetsAt` has passed reads `rolledOver: true`; its percent is from before the reset, so show the reset rather than the number. -**One current reading per user and harness.** A new reading replaces the -whole row, so after a sign-in change no window from the earlier account is left. -A reading older than the stored one is refused. The account is kept only as a -hash, with no email. There is no history. +**One current record per user and harness.** A reader's result and the +windows runs reported sit side by side: + +- A reading replaces the windows the last reading left and keeps the run windows it doesn't name. Where both name one window, the newer wins. +- A newer report for a window replaces the older one. An older one that arrives late is ignored, and so is a reading older than the stored one. +- A reading or run window from a different account clears every window and the plan the earlier account left. +- A reading with no number (`api-key`, `signed-out`) clears every window. A run that started before that reading doesn't bring them back. + +The account is kept only as a hash, with no email. There is no history. The readings live in `harnessUsageCollection` (`harness-usage/**`, user scope). A client can't read the collection directly. Read it through the `harnessUsage` query. On the server: - `readHarnessUsage(ctx, now, expected?)` returns the same rows the query does, without refreshing. -- `recordHarnessUsage(ctx, harness, reading, checkedAt)` stores a reading from a harness that reports usage during its own runs. It answers `"recorded"`, `"kept-stored"` (an `unavailable` reading over a stored one, which stays) or `"refused-older"`. +- `recordHarnessUsage(ctx, report)` stores one window a run reported, a `HarnessUsageReport`. The manager's and the session door's `onUsage` call it for you. +- `recordHarnessUsage(ctx, harness, reading, checkedAt)` stores a whole reading made at `checkedAt`. + +Both forms answer `"recorded"`, `"kept-stored"` (an `unavailable` reading over a stored one, which stays) or `"refused-older"` (what is stored is newer, or the run started before the stored no-number reading). -Both read the collection through the calling block's resources, under the key `HARNESS_USAGE`, and throw when it isn't installed there: give that block `resources: { [HARNESS_USAGE]: harnessUsageCollection }`. `harnessUsageAction` installs it on its own block. +All of these read the collection through the calling block's resources, under the key `HARNESS_USAGE`, and throw when it isn't installed there: give that block `resources: { [HARNESS_USAGE]: harnessUsageCollection }`. `harnessUsageAction` installs it on its own block. ## Pointing it away from your own code diff --git a/apps/docs/docs/tools/claude-code-sdk.md b/apps/docs/docs/tools/claude-code-sdk.md index b1fc915879..9bb9607722 100644 --- a/apps/docs/docs/tools/claude-code-sdk.md +++ b/apps/docs/docs/tools/claude-code-sdk.md @@ -64,9 +64,12 @@ When the host decides to run the agent (rather than letting a model choose), use the block directly: ```ts +import { harnessEnv } from "@flow-state-dev/core"; import { claudeCodeAgent } from "@flow-state-dev/claude-code/sdk"; +const env = harnessEnv({ pass: ["PATH", "HOME", "CLAUDE_CODE_OAUTH_TOKEN"] }); const agent = claudeCodeAgent({ + env, systemPrompt: { type: "preset", preset: "claude_code", @@ -781,11 +784,53 @@ Pass the same `env` you give `claudeCodeAgent`, so the probe sees the credential | `status` | When | Also carries | |----------|------|--------------| | `"account"` | Claude Code named where its credential comes from | `apiKeySource` and/or `tokenSource`, as the SDK reports them; a subscription sign-in names neither and comes back as `email` and `subscriptionType` | -| `"none"` | No credential, no answer within `timeoutMs` (default 10 000 ms), or Claude Code couldn't be asked | `reason`, a sentence you can show the user | +| `"none"` | No credential, no answer within `timeoutMs` (default 10 000 ms), or Claude Code couldn't be asked | `reason`, a sentence you can show the user, and `unanswered: true` when Claude Code gave no answer | | `"sdk-missing"` | `@anthropic-ai/claude-agent-sdk` couldn't be loaded | `reason`, with the load error | It never throws for any of these. It checks where a credential would come from, not that it works: a key that is set but rejected still reads as `"account"`, and fails on the run's first turn. +## Reading usage left + +A Claude subscription limits use per time window, such as five hours or a week, and a run fails once a window is used up. Claude Code reports where those limits stand while it runs, so the numbers come from runs. Pass `onUsage` and the block hands you each one: + +```ts +import { harnessEnv } from "@flow-state-dev/core"; +import { claudeCodeAgent } from "@flow-state-dev/claude-code/sdk"; + +const env = harnessEnv({ pass: ["PATH", "HOME", "CLAUDE_CODE_OAUTH_TOKEN"] }); +const agent = claudeCodeAgent({ + env, + onUsage: ({ window, observedAt }) => { + console.log(window.id, window.level, window.usedPercent, window.resetsAt && new Date(window.resetsAt)); + }, +}); +``` + +Claude Code sends an update whenever its limits change, and at least once per run on a subscription. Each update becomes one [usage window](./coding-agents.md#subscription-usage), for the limit that binds at that moment: + +| Field | Value | +|---|---| +| `id` | Claude Code's name for the window: `five_hour`, `seven_day`, or a weekly variant such as `seven_day_opus` | +| `minutes` | 300 for `five_hour`, 10 080 for the `seven_day` family, `null` for any other id | +| `level` | `"ok"` (under its warning line), `"warning"` (near the limit) or `"reached"` | +| `usedPercent` | 0 to 100, only when Claude Code sends one, which it does near a limit. Otherwise `null`, never 0 | +| `resetsAt` | When the window resets, in epoch ms | + +The report around it also carries `harness` (`"claude-code"`, exported as `CLAUDE_CODE_HARNESS`), `observedAt`, `runStartedAt`, and `accountId`, the signed-in email or `null`. A run on an API key reports nothing. `onUsage` is awaited, but a throw from it doesn't fail the run: a warning is logged and the run carries on. It works on in-session and detached blocks alike. `usageWindowOf(info)` maps one raw event payload to a window, or `null`, if you read the SDK stream yourself. + +To show who is signed in before any run, `readClaudeUsage({ env })` is a [usage reader](./coding-agents.md#subscription-usage). It never throws: + +| `status` | When | +|---|---| +| `windows` | Signed in, or running on an OAuth token. Carries `plan` and `accountId`, and **no windows**: those arrive from runs. | +| `api-key` | Claude Code runs on an API key, which has no subscription limit. | +| `signed-out` | Nobody is signed in. | +| `unavailable` | `@anthropic-ai/claude-agent-sdk` isn't installed, or Claude Code couldn't be asked or didn't answer within `timeoutMs` (10 seconds by default) or before `signal` aborted. The usage query keeps the last good reading and shows why the refresh failed. | + +Pass the same `env` as your runs. To keep run windows and readings together and serve them to a page or a worker, register `readClaudeUsage` as the `claude-code` reader of the [harness manager's usage query](/docs/orchestration/harness-manager#subscription-usage) and pass `onUsage` through from the manager's harness slot. + +**Where the numbers come from.** Only from Claude Code's own run events and its account answer. Nothing here reads a credentials file, calls a usage endpoint, or uses the SDK's experimental usage call. + ## Error handling | Situation | Behavior | diff --git a/apps/docs/docs/tools/coding-agents.md b/apps/docs/docs/tools/coding-agents.md index f711b33111..36dce28d6d 100644 --- a/apps/docs/docs/tools/coding-agents.md +++ b/apps/docs/docs/tools/coding-agents.md @@ -138,7 +138,14 @@ window is used, without a run. The shapes live in `@flow-state-dev/core`, so a page or a worker reads every agent's usage the same way: ```ts -import type { HarnessUsageReader, HarnessUsageReading, HarnessUsageWindow } from "@flow-state-dev/core/types"; +import type { + HarnessUsageHook, + HarnessUsageLevel, + HarnessUsageReader, + HarnessUsageReading, + HarnessUsageReport, + HarnessUsageWindow, +} from "@flow-state-dev/core/types"; import { harnessUsageWindowSchema } from "@flow-state-dev/core"; ``` @@ -153,15 +160,33 @@ A reading is one of four: | `unavailable` | `reason`. The read couldn't be made. | Each window has an `id` unique within the reading, the vendor's `label` for its -limit, its length in `minutes`, `usedPercent` from 0 to 100, and `resetsAt` in -epoch ms. `label`, `minutes` and `resetsAt` are `null` when the vendor didn't -say. The numbers are the vendor's own, unconverted. Only `windows` carries a -number: show the `reason` for the other three, never 0%. +limit, its length in `minutes`, `usedPercent` from 0 to 100, `level`, and +`resetsAt` in epoch ms. `label`, `minutes` and `resetsAt` are `null` when the +vendor didn't say. `level` (`HarnessUsageLevel`) is how close the window is to +its limit, when the vendor says: `"ok"` (under its warning line), `"warning"` +(near the limit) or `"reached"`, and `null` for a vendor that sends only a +percent, as Codex does. `usedPercent` is `null` when the vendor sent no +percent; show the `level` instead. `harnessUsageWindowSchema` reads a window +stored with no `level` as `level: null`. The numbers are the vendor's own: a +fraction is scaled to a percent and nothing else is derived. Only `windows` +carries a number: show the `reason` for the other three, never 0%. A reader returns those states rather than throwing. Codex ships one, -[`readCodexUsage`](./codex.md#reading-usage-left). To keep readings and serve -them with their age, hand your readers to the -[harness manager's usage query](/docs/orchestration/harness-manager#subscription-usage). +[`readCodexUsage`](./codex.md#reading-usage-left), and Claude Code ships +[`readClaudeUsage`](./claude-code-sdk.md#reading-usage-left). + +Some agents report usage only while they run. A harness block that does takes an +`onUsage` hook, a `HarnessUsageHook`: `(report, ctx) => void | Promise`. +Each `HarnessUsageReport` carries one window, plus `harness` (the name a usage +record is kept under), `observedAt` and `runStartedAt` in epoch ms, and the +`accountId` the run is signed in as, or `null`. A run reports the window that +applies to the whole account, not its own share. Claude Code is the agent that +reports this way: its reader names the account and plan, and its windows come +from [`onUsage`](./claude-code-sdk.md#reading-usage-left). + +To keep readings and run windows and serve them with their age, hand your readers +to the [harness manager's usage query](/docs/orchestration/harness-manager#subscription-usage) +and let the manager feed your harness its `onUsage`. ## The prompt is the input; everything else is configuration diff --git a/goals/devforce-lab/it-commits-from-the-seats-own-file/run.mts b/goals/devforce-lab/it-commits-from-the-seats-own-file/run.mts index 5a4784ef52..b58c2bf508 100644 --- a/goals/devforce-lab/it-commits-from-the-seats-own-file/run.mts +++ b/goals/devforce-lab/it-commits-from-the-seats-own-file/run.mts @@ -149,14 +149,14 @@ await runGoal(async (failures) => { // Claude Code as the lab's `claudeCodeHarness` sets it up — the same agent // a served Lab runs under `DEVFORCE_LAB_HARNESS=claude-code` — with the // prompt-recording tap in front of it. - harness: ({ cwd, resume, onSession }) => + harness: (feeds) => sequencer({ name: "devforce-lab-recorded-harness", inputSchema: harnessRunInputSchema, outputSchema: harnessRunHandleSchema, }) .tap(recordPrompt) - .step(claudeCodeHarness({ cwd, resume, onSession }) as never) as unknown as HarnessBlock, + .step(claudeCodeHarness(feeds) as never) as unknown as HarnessBlock, }); try { diff --git a/goals/shift-manager/it-hands-a-run-the-work-it-approved/run.mts b/goals/shift-manager/it-hands-a-run-the-work-it-approved/run.mts index 2c3d9ab70f..02c168d965 100644 --- a/goals/shift-manager/it-hands-a-run-the-work-it-approved/run.mts +++ b/goals/shift-manager/it-hands-a-run-the-work-it-approved/run.mts @@ -387,14 +387,14 @@ await runGoal(async (failures) => { }); const { lab, sourceRepo } = await open( "handed-b", - ({ cwd, resume, onSession }) => + (feeds) => sequencer({ name: "handed-recorded-harness", inputSchema: harnessRunInputSchema, outputSchema: harnessRunHandleSchema, }) .tap(recordPrompt) - .step(claudeCodeHarness({ cwd, resume, onSession }) as never) as unknown as HarnessBlock, + .step(claudeCodeHarness(feeds) as never) as unknown as HarnessBlock, fixture.realRun, { runTimeoutMs: REAL_RUN_TIMEOUT_MS }, ); diff --git a/goals/shift-manager/it-shows-claude-usage-left/controls.mts b/goals/shift-manager/it-shows-claude-usage-left/controls.mts new file mode 100644 index 0000000000..f3ad00ac81 --- /dev/null +++ b/goals/shift-manager/it-shows-claude-usage-left/controls.mts @@ -0,0 +1,83 @@ +/** + * The goal's four controls, and the one fault every run carries. Each is a + * scratch patch applied as the Lab under test loads the module + * (`goals/lib/module-patch.mjs`); nothing is written to the checkout. + */ +import type { ModulePatch } from "../../lib/module-patch.mts"; + +export const CONTROL_NAMES = ["no-scale", "absent-as-zero", "fan-out", "read-replaces-run"] as const; +export type ControlName = (typeof CONTROL_NAMES)[number]; + +/** One control: the assertions that must go red, and its patches. */ +export interface Control { + name: ControlName; + assertions: string[]; + patches: ModulePatch[]; + does: string; +} + +const CLAUDE_USAGE = "packages/claude-code/src/sdk/usage.ts"; +const USAGE = "packages/harness-manager/src/harness-usage.ts"; + +export const CONTROLS: Record = { + "no-scale": { + name: "no-scale", + assertions: ["b:91"], + patches: [{ module: CLAUDE_USAGE, from: String.raw`Math\.round\(utilization \* 1000\) / 10`, to: "utilization" }], + does: "Claude Code's fraction is stored as sent, not scaled to a percent", + }, + "absent-as-zero": { + name: "absent-as-zero", + assertions: ["a:no-number"], + patches: [ + { + module: CLAUDE_USAGE, + from: String.raw`const usedPercent = fraction \? Math\.round\(utilization \* 1000\) / 10 : null;`, + to: "const usedPercent = fraction ? Math.round(utilization * 1000) / 10 : 0;", + }, + ], + does: "an event with no percent is stored as 0%", + }, + "fan-out": { + name: "fan-out", + assertions: ["d:no-claude-window"], + patches: [{ module: USAGE, from: String.raw`scope: "user",`, to: 'scope: "org",' }], + does: "the usage record is one row for the whole org, so a run writes everyone's", + }, + "read-replaces-run": { + name: "read-replaces-run", + assertions: ["e:keeps-the-runs-window"], + patches: [ + { + module: USAGE, + from: String.raw`const kept = step === "rebind" \? \[\] : stored\.windows\.filter\(\(window\) => window\.origin === "run"\);`, + to: "const kept: typeof stored.windows = [];", + }, + ], + does: "a look replaces every window, the run's too", + }, +}; + +/** + * The write that throws, for leg f: a window named `goal_write_throws` fails + * in the one writer, as a store that is down would. Every run carries it, so + * every other window is written as shipped. + */ +export const FAULTS: ModulePatch[] = [ + { + module: USAGE, + from: String.raw`const record = await rowOf\(ctx, harness\);`, + to: + 'if (typeof harnessOrReport !== "string" && harnessOrReport.window.id === "goal_write_throws") throw new Error("goal: the usage write failed"); ' + + "const record = await rowOf(ctx, harness);", + }, +]; + +/** Where the harness loads the Agent SDK from: pointed at the scripted one. */ +export function sdkSeam(scriptedSdkUrl: string): ModulePatch { + return { + module: "packages/claude-code/src/sdk/sdk-client.ts", + from: String.raw`const SDK_MODULE = "@anthropic-ai/claude-agent-sdk";`, + to: `const SDK_MODULE = ${JSON.stringify(scriptedSdkUrl)};`, + }; +} diff --git a/goals/shift-manager/it-shows-claude-usage-left/goal.md b/goals/shift-manager/it-shows-claude-usage-left/goal.md new file mode 100644 index 0000000000..91d5a3a3cf --- /dev/null +++ b/goals/shift-manager/it-shows-claude-usage-left/goal.md @@ -0,0 +1,41 @@ +# shift-manager › it shows Claude usage left + +**Issue:** FIX-2142 + +**Outcome:** A person running Shift Manager on their own Claude subscription opens Usage and sees, for Claude Code, each window their runs reported: which limit, when it resets, and how close it is, as a percent when Claude Code sent one and in words when it didn't. A run only ever writes its own person's row, and a look at who is signed in never throws a run's window away or brings back an old account's. + +**Input:** DevTeam's tree and `openLab`, served by Shift Manager's command (`lab/fsdev.config.mts`), with the coder on `claudeCodeHarness`. A scripted chief of staff files one task per tag, and the coder's runs go to `lab/scripted-sdk.mjs` in place of the Agent SDK (`sdkSeam` in `controls.mts`): each run sends the `rate_limit_event`s its tag scripts, in the shape the pinned CLI (0.3.234) builds for each status, and answers `accountInfo()` with the scripted account. The `claude-code` usage reader asks the same seam who is signed in, or the real Agent SDK over an empty home (with and without an API key). Each leg runs as its own person. + +**Signal:** each failure tagged `:`. + +- **a** (event `allowed`, no percent): the query returns one `five_hour` window from a run, `level: "ok"`, `usedPercent: null` (**a:no-number**), its reset in ms. The view says "under Claude Code's warning line", a reset time, "from a run", and no `%`. +- **b** (`allowed_warning`, utilization 0.91): the query and the view say 91% (**b:91**). +- **c** (`rejected`): `level: "reached"`, no percent; the view says "limit reached". +- **d** (Alice's run, then Bob talks to his own finished coder): Alice's run leaves Bob's row empty (**d:no-claude-window**); Bob's talk writes his row at 50% and leaves Alice's. +- **e**: a look on the same sign-in keeps the run's window and adds the plan (**e:keeps-the-runs-window**); account B's run clears account A's window (**e:switch-clears**); the real probe on an empty home says `signed-out`; a minute later on an API key, `api-key` with no window. +- **f**: an older event from account A, written after account B's newer one, leaves B's (**f:newer-kept**); a run that started before a look found an API key brings no window back (**f:old-run**); a write that throws leaves the run completed and the row unchanged (**f:write-throws**). +- **g** (owed by a person): Jake's own Claude sign-in, a real run, then Usage beside Claude Code's `/status`. No CI has a sign-in, so until it is run the verdict is *goal met except leg g*. + +**Anti-game:** +- No assertion reads Shift Manager's modules or stubs the query. Rows are read through `harnessUsage` over the Lab's HTTP door, and the view in a browser. +- Runs go through the real chain: chief of staff, task board, harness manager, `claudeCodeAgent`, `onUsage`, the writer. Only the SDK module is replaced. +- The empty-home and API-key looks run the real Agent SDK probe. + +**Model:** n/a (scripted chief of staff and SDK). + +**Run:** `PLAYWRIGHT_BROWSERS_PATH=/opt/pw-browsers pnpm tsx goals/shift-manager/it-shows-claude-usage-left/run.mts` (about 100 s: leg e waits out the minute between looks). + +**Controls:** each patches a module as the Lab loads it (`controls.mts`). +- `GOAL_CONTROL=no-scale`: the fraction is stored as sent. Must fail **b:91**. +- `GOAL_CONTROL=absent-as-zero`: an event with no percent is stored as 0%. Must fail **a:no-number**. +- `GOAL_CONTROL=fan-out`: the usage record is one org-wide row. Must fail **d:no-claude-window**. +- `GOAL_CONTROL=read-replaces-run`: a look replaces every window, the run's too. Must fail **e:keeps-the-runs-window**. + +## Verdict log +| Date | Commit | Model | Verdict | Notes | +|------|--------|-------|---------|-------| +| 2026-10-23 | claude/project-thread-3x3z3h (pre-PR) | n/a | PASS (goal met except leg g) | 97 s. a: `five_hour`, 300 min, `ok`, no percent; view "under Claude Code's warning line", "from a run, a few seconds ago". b: 91 `warning`, "91% used". c: `reached`, "limit reached". d: Alice's run left Bob 0 windows; his talk wrote 50%. e: kept the run's window, plan max; account B cleared A; empty home `signed-out`; API key `api-key`, no window. f: B's window kept over A's late event; old run after the API-key look, 0 windows; the throwing write's run completed. g owed. | +| 2026-10-23 | same, `GOAL_CONTROL=no-scale` | n/a | FAIL (expected) | **b:91**: the query and the view say 0.91%; d's talk reads 0.5. | +| 2026-10-23 | same, `GOAL_CONTROL=absent-as-zero` | n/a | FAIL (expected) | **a:no-number**: "0% used"; c, e and f show 0% too. | +| 2026-10-23 | same, `GOAL_CONTROL=fan-out` | n/a | FAIL (expected) | **d:no-claude-window**: Alice's run wrote Bob's row; a and b read c's window from the shared row. | +| 2026-10-23 | same, `GOAL_CONTROL=read-replaces-run` | n/a | FAIL (expected) | **e:keeps-the-runs-window**: the look left 0 windows; the view's looks in a, b and c emptied their rows too. | diff --git a/goals/shift-manager/it-shows-claude-usage-left/lab/fsdev.config.mts b/goals/shift-manager/it-shows-claude-usage-left/lab/fsdev.config.mts new file mode 100644 index 0000000000..877f656b74 --- /dev/null +++ b/goals/shift-manager/it-shows-claude-usage-left/lab/fsdev.config.mts @@ -0,0 +1,102 @@ +/** + * DevTeam, served for this goal, with Claude Code as the coder: the DevForce + * lab's tree and `openLab`, with `claudeCodeHarness` in the coder's slot as + * the `fsd` profile puts it there, and the Claude Code usage reader as DevTeam + * registers it. + * + * What it adds over `packages/shift-manager/teams/devteam/fsdev.config.mts`: + * + * - **The Agent SDK is scripted** (`scripted-sdk.mjs`), loaded where the + * harness loads the real one: the check points `sdk-client.ts`'s module name + * at it. A run's `rate_limit_event`s and account come from the setup file. + * - **The reader's environment** comes from the setup file too: an empty home, + * or one with an API key, for the legs that read the real probe. + * - **The chief of staff's model is scripted**: a message carrying + * `[[usage:]]` files one task for the coder on Sandbox (a project with + * no repository), its goal carrying the tag. Its judgment is not under test. + * No judge, so a run is never retried. + * - **The store**, a SQLite file at `CLAUDE_USAGE_GOAL_STORE`. + * + * Nothing about the hook, the writer, the query or the view is set here. + */ +import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import { readClaudeUsage } from "@flow-state-dev/claude-code/sdk"; +import type { GeneratorModel, ModelResolver } from "@flow-state-dev/core/types"; +import { sqliteStores } from "@flow-state-dev/store-sqlite"; +import { readDeclaredRoster } from "@flow-state-dev/workforce/loader"; +import { claudeCodeHarness, CLAUDE_CODE_RUN_TIMEOUT_MS } from "../../../../packages/shift-manager/teams/devteam/harness.mts"; +import { LAB_CROWD, LAB_TREE, LAB_USERS, openLab } from "../../../../packages/shift-manager/teams/devteam/host.mts"; +import { CODER_FLOW } from "../../../../packages/shift-manager/teams/devteam/workforce/flows/workers/coder.mts"; +import { EM_FLOW } from "../../../../packages/shift-manager/teams/devteam/workforce/flows/workers/em.mts"; + +const STORE = process.env.CLAUDE_USAGE_GOAL_STORE; +const SETUP = process.env.CLAUDE_USAGE_GOAL_SETUP; +if (!STORE || !SETUP) throw new Error("CLAUDE_USAGE_GOAL_STORE and CLAUDE_USAGE_GOAL_SETUP name the store and the setup"); + +/** Sandbox: a project with no repository, so a run works on its files. */ +export const SANDBOX = { visibility: "shared", id: "sandbox" } as const; + +const roster = await readDeclaredRoster(LAB_TREE); +const workerFlowOf = (id: string) => roster.workers.find((w) => w.id === id)?.declared.flow; +const em = roster.workers.find((w) => w.declared.flow === EM_FLOW); +const coderWorkerId = ((em?.declared.delegates as string[] | undefined) ?? []).find((id) => workerFlowOf(id) === CODER_FLOW); +if (coderWorkerId === undefined) throw new Error("the DevTeam tree names no coder"); + +/** + * The chief of staff's turn: a tagged message files one task for the coder, + * once per tag; anything else is answered. Once, because the task's ending + * wakes the chief again with the same conversation in front of it. + */ +function scriptedChief(): ModelResolver { + const filed = new Set(); + const model: GeneratorModel = { + modelId: "goal/scripted-chief", + async generate() { + throw new Error("the owned tool loop calls generateStep"); + }, + async generateStep(options: { messages: unknown[] }) { + const last = options.messages.at(-1) as { role?: string; content?: unknown } | undefined; + if (last?.role === "tool") return { text: "Filed.", finishReason: "stop" }; + const text = typeof last?.content === "string" ? last.content : JSON.stringify(last?.content ?? ""); + const tag = /\[\[usage:([a-z0-9-]+)\]\]/.exec(text)?.[1]; + if (tag === undefined || filed.has(tag)) return { text: "Noted.", finishReason: "stop" }; + filed.add(tag); + const args = { goal: `Answer in one line. [[usage:${tag}]]`, assignee: coderWorkerId, input: { project: SANDBOX } }; + return { toolCalls: [{ toolCallId: `chief-${tag}`, toolName: "createTask", args }], finishReason: "tool-calls" }; + }, + }; + return Object.assign( + (_id: string, block?: string) => { + if (block === "coordinator-route-ask") throw new Error("no route model in this check"); + return model; + }, + { resolveId: (id: string) => id }, + ) as unknown as ModelResolver; +} + +/** How the reader's probe is pointed, as the check last wrote it. */ +type Setup = { reader: { home: string; apiKey?: string } }; + +const lab = await openLab({ + stores: sqliteStores({ filename: STORE }), + harness: claudeCodeHarness, + runTimeoutMs: CLAUDE_CODE_RUN_TIMEOUT_MS, + workspace: { root: join(STORE, "..", "runs"), remotes: { allow: [] } }, + coderWorkerId, + inventory: true, + devtool: true, + withoutJudge: true, + modelResolver: scriptedChief(), + projects: [{ id: SANDBOX.id, title: "Sandbox", members: [LAB_USERS.member.userId, ...LAB_CROWD.map((u) => u.userId)] }], + usage: { + "claude-code": () => { + const { reader } = JSON.parse(readFileSync(SETUP, "utf8")) as Setup; + return readClaudeUsage({ + env: { PATH: process.env.PATH ?? "", HOME: reader.home, ...(reader.apiKey === undefined ? {} : { ANTHROPIC_API_KEY: reader.apiKey }) }, + }); + }, + }, +}); + +export default lab.state; diff --git a/goals/shift-manager/it-shows-claude-usage-left/lab/scripted-sdk.mjs b/goals/shift-manager/it-shows-claude-usage-left/lab/scripted-sdk.mjs new file mode 100644 index 0000000000..ba303b3b6e --- /dev/null +++ b/goals/shift-manager/it-shows-claude-usage-left/lab/scripted-sdk.mjs @@ -0,0 +1,63 @@ +/** + * The Claude Code Agent SDK, scripted, loaded where `claudeCodeAgent` loads + * the real one (the goal points `sdk-client.ts`'s module name here). + * + * A run is a `query` with a string prompt. Its prompt carries a tag, + * `[[usage:]]`, and the setup file (`CLAUDE_USAGE_GOAL_SETUP`) says what + * that run sends: the `rate_limit_event`s, each as the pinned CLI builds one + * (FIX-2142's POC), and the account its `accountInfo()` names. Every run is + * logged (`RUN `) so the check can see it started. + * + * A query with no prompt is the account probe. With `probe.mode` `real`, it + * goes to the real SDK, so legs read the real `probeClaudeAccount` on an + * empty home and on an API key; with `scripted`, it answers the setup's + * account, standing in for a sign-in no check machine has. + */ +import { appendFileSync, existsSync, readFileSync } from "node:fs"; + +const SETUP = process.env.CLAUDE_USAGE_GOAL_SETUP; +const setup = () => JSON.parse(readFileSync(SETUP, "utf8")); +const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); + +export function query(args) { + const current = setup(); + if (typeof args.prompt !== "string") return probe(args, current.probe); + const tag = /\[\[usage:([a-z0-9-]+)\]\]/.exec(args.prompt)?.[1] ?? "untagged"; + const script = current.runs?.[tag] ?? { events: [] }; + appendFileSync(current.log, `RUN ${tag} ${Date.now()}\n`); + const sessionId = `sess-${tag}`; + const messages = (async function* () { + yield { type: "system", subtype: "init", session_id: sessionId }; + for (const event of script.events) { + // Held until the check says go, so a look can land while the run is going. + if (event.waitFor !== undefined) while (!existsSync(event.waitFor)) await sleep(100); + yield { type: "rate_limit_event", session_id: sessionId, uuid: `${tag}-${Date.now()}`, rate_limit_info: event.info }; + } + yield { type: "result", subtype: "success", is_error: false, result: `Answered ${tag}.`, session_id: sessionId }; + })(); + return Object.assign(messages, { + async accountInfo() { + if (script.accountDelayMs) await sleep(script.accountDelayMs); + return script.account ?? {}; + }, + close() {}, + }); +} + +/** The account probe's idle query: the real SDK's, or the setup's account. */ +function probe(args, how) { + if (how?.mode === "real") { + let real; + const handle = import("@anthropic-ai/claude-agent-sdk").then((sdk) => (real = sdk.query(args))); + return { + accountInfo: () => handle.then((q) => q.accountInfo()), + close: () => real?.close?.(), + [Symbol.asyncIterator]: () => ({ next: () => new Promise(() => {}) }), + }; + } + return { + accountInfo: async () => how?.account ?? {}, + close() {}, + [Symbol.asyncIterator]: () => ({ next: () => new Promise(() => {}) }), + }; +} diff --git a/goals/shift-manager/it-shows-claude-usage-left/run.mts b/goals/shift-manager/it-shows-claude-usage-left/run.mts new file mode 100644 index 0000000000..2d83f8e79d --- /dev/null +++ b/goals/shift-manager/it-shows-claude-usage-left/run.mts @@ -0,0 +1,340 @@ +/** + * Goal check for FIX-2142. See goal.md. + * + * Run: PLAYWRIGHT_BROWSERS_PATH=/opt/pw-browsers pnpm tsx goals/shift-manager/it-shows-claude-usage-left/run.mts + * Controls: GOAL_CONTROL=no-scale | absent-as-zero | fan-out | read-replaces-run + */ +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { REPO_ROOT, RUN_STAMP, goalTmpDir, runGoal } from "../../lib/index.mts"; +import { assertPatched, describePatches, modulePatchEnv } from "../../lib/module-patch.mts"; +import { launchChromium } from "../../lib/playwright.mts"; +import { buildShiftManagerPages, labRoutes, openShiftManager, personPage, startShiftManager, type LabUser } from "../../lib/shift-manager.mts"; +import { LAB_CROWD, LAB_USERS, PROJECTS_FLOW } from "../../../packages/shift-manager/teams/devteam/host.mts"; +import { CODER_FLOW } from "../../../packages/shift-manager/teams/devteam/workforce/flows/workers/coder.mts"; +import { LIST_TASKS_ACTION } from "../../../packages/shift-manager/src/lib/conversation-tasks.ts"; +import { CONTROL_NAMES, CONTROLS, FAULTS, sdkSeam, type Control } from "./controls.mts"; + +const SCRATCH = goalTmpDir("claude-usage"); +const say = (s: string) => console.error(`[claude-usage ${new Date().toISOString().slice(11, 19)}] ${s}`); +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +const controlName = process.env.GOAL_CONTROL ?? ""; +if (controlName !== "" && !(CONTROL_NAMES as readonly string[]).includes(controlName)) { + console.error(`unknown GOAL_CONTROL "${controlName}"; known: ${CONTROL_NAMES.join(", ")}`); + process.exit(2); +} +const control: Control | undefined = controlName === "" ? undefined : CONTROLS[controlName as keyof typeof CONTROLS]; + +/** One person per leg, so each leg reads a row only it wrote. */ +const PEOPLE = { + a: { label: "a", ...LAB_CROWD[0]! }, + b: { label: "b", ...LAB_CROWD[1]! }, + c: { label: "c", ...LAB_CROWD[2]! }, + alice: { label: "alice", ...LAB_USERS.owner }, + bob: { label: "bob", ...LAB_USERS.member }, + e: { label: "e", ...LAB_CROWD[3]! }, + eSwitch: { label: "e-switch", ...LAB_CROWD[4]! }, + eEmpty: { label: "e-empty", ...LAB_USERS.outsider }, + fOrder: { label: "f-order", ...LAB_CROWD[5]! }, + fOld: { label: "f-old", ...LAB_CROWD[6]! }, +} satisfies Record; + +type Window = { id: string; minutes: number | null; usedPercent: number | null; level: string | null; resetsAt: number | null; checkedAt: number; origin: string; ageMs: number; rolledOver: boolean }; +type Row = { harness: string; status: string; reason: string | null; plan: string | null; windows: Window[]; checkedAt: number | null; ageMs: number | null; refreshError: string | null }; + +/** A `rate_limit_event`'s `rate_limit_info`, as the pinned CLI builds each status (FIX-2142's POC). */ +const RESETS = Math.floor(Date.now() / 1000) + 2 * 3600; +const event = { + allowed: (window = "five_hour") => ({ status: "allowed", resetsAt: RESETS, rateLimitType: window }), + warning: (utilization: number, window = "five_hour") => ({ status: "allowed_warning", resetsAt: RESETS, rateLimitType: window, utilization }), + rejected: (window = "five_hour") => ({ status: "rejected", resetsAt: RESETS, rateLimitType: window }), + apiKey: () => ({ status: "allowed" }), +}; + +await runGoal(async () => { + const started = Date.now(); + const mark = join(SCRATCH, `patch-mark-${RUN_STAMP}`); + const scriptedSdk = pathToFileURL(fileURLToPath(new URL("./lab/scripted-sdk.mjs", import.meta.url))).href; + const patches = [sdkSeam(scriptedSdk), ...FAULTS, ...(control?.patches ?? [])]; + const patchEnv = modulePatchEnv(patches, REPO_ROOT, mark); + if (control !== undefined) say(`control ${control.name}: ${control.does}\n${describePatches(control.patches, REPO_ROOT)}`); + + // The homes the real probe reads: nobody signed in, and the same with an API key. + const emptyHome = mkdtempSync(join(SCRATCH, "claude-empty-")); + const setupFile = join(SCRATCH, "setup.json"); + const runLog = join(SCRATCH, "runs.log"); + writeFileSync(runLog, ""); + const runs: Record = {}; + let probe: unknown = { mode: "scripted", account: {} }; + let reader: { home: string; apiKey?: string } = { home: emptyHome }; + const writeSetup = () => writeFileSync(setupFile, JSON.stringify({ log: runLog, runs, probe, reader })); + /** What the run tagged `tag` sends, and the account it names. */ + const script = (tag: string, events: Array<{ info: unknown; waitFor?: string }>, account: Record = {}, accountDelayMs = 0) => { + runs[tag] = { events, account, accountDelayMs }; + writeSetup(); + }; + /** The look's account: a scripted sign-in, or the real probe on an empty home or an API key. */ + const signIn = (account: Record) => { + probe = { mode: "scripted", account }; + writeSetup(); + }; + const realProbe = (apiKey?: string) => { + probe = { mode: "real" }; + reader = { home: emptyHome, ...(apiKey === undefined ? {} : { apiKey }) }; + writeSetup(); + }; + const started_ = (tag: string) => readFileSync(runLog, "utf8").split("\n").filter((l) => l.startsWith(`RUN ${tag} `)).map((l) => Number(l.split(" ")[2])); + writeSetup(); + + say("building Shift Manager's pages"); + const pages = await buildShiftManagerPages(join(SCRATCH, "pages")); + const storeDir = join(SCRATCH, "store"); + mkdirSync(storeDir, { recursive: true }); + const served = await startShiftManager({ + scratch: SCRATCH, + label: "claude-usage", + config: fileURLToPath(new URL("./lab/fsdev.config.mts", import.meta.url)), + pages, + env: { CLAUDE_USAGE_GOAL_STORE: join(storeDir, "lab.sqlite"), CLAUDE_USAGE_GOAL_SETUP: setupFile, ...patchEnv }, + timeoutMs: 180_000, + }); + say(`Lab at ${served.origin}`); + + const failures: string[] = []; + const notes: string[] = []; + + /** The usage query, over the Lab's HTTP door, as `person`. */ + const query = async (person: LabUser, refresh: boolean): Promise => { + const done = await labRoutes(served.origin, person).act(PROJECTS_FLOW, `${PROJECTS_FLOW}-own-${person.userId}`, "harnessUsage", refresh ? { refresh: true } : {}); + if (done.status !== "completed") throw new Error(`harnessUsage as ${person.userId} ended ${done.status}: ${done.error ?? ""}`); + return (done.output as { harnesses: Row[] }).harnesses; + }; + const claudeRow = (rows: Row[]) => rows.find((row) => row.harness === "claude-code"); + + /** Ask the chief of staff, as `person`, for a task tagged `tag`, and wait for the coder's run to end. */ + const runTask = async (person: LabUser, tag: string, until: (task: Record) => boolean = ended) => { + const routes = labRoutes(served.origin, person); + const sessionId = `s-cos-${tag}-${RUN_STAMP}`; + const created = await routes.call("POST", "/coordinator/sessions", { userId: person.userId, sessionId, state: { workerId: "chief-of-staff" } }); + if (created.status !== 201) throw new Error(`${tag}: opening the chief of staff's session answered ${created.status} ${JSON.stringify(created.body)}`); + const said = await routes.act("coordinator", sessionId, "respond", { message: `Please take this on. [[usage:${tag}]]` }); + if (said.status !== "completed") throw new Error(`${tag}: the chief of staff's turn ended ${said.status}: ${said.error ?? ""}`); + for (const deadline = Date.now() + 120_000; Date.now() < deadline; await sleep(300)) { + const listed = await routes.act("coordinator", sessionId, LIST_TASKS_ACTION, {}); + const task = (listed.output as { tasks?: Array> } | undefined)?.tasks?.[0]; + if (task !== undefined && until(task)) return task; + } + throw new Error(`${tag}: the coder's task never ended`); + }; + /** File a task and return once its run has started, without waiting for it to end. */ + const startTask = (person: LabUser, tag: string) => runTask(person, tag, (task) => task.status !== "pending" || started_(tag).length > 0); + + const browser = await launchChromium(); + /** The Claude Code card on Usage, as `person` sees it. */ + const view = async (person: LabUser, owner: boolean) => { + const { context, page } = await personPage(browser, served.origin, person, owner); + try { + await openShiftManager(page, served.origin, "/usage"); + await page.locator("[data-testid=usage-harness][data-harness=claude-code]").waitFor({ timeout: 30_000 }); + return await page.$eval("[data-testid=usage-harness][data-harness=claude-code]", (card) => ({ + status: card.getAttribute("data-status"), + text: card.textContent ?? "", + windows: [...card.querySelectorAll("[data-testid=usage-window]")].map((w) => ({ + id: w.getAttribute("data-window-id"), + text: w.textContent ?? "", + percent: w.querySelector("[data-testid=usage-percent]")?.textContent ?? null, + level: w.querySelector("[data-testid=usage-level]")?.textContent ?? null, + reset: w.querySelector("[data-testid=usage-reset]")?.textContent ?? null, + age: w.querySelector("[data-testid=usage-window-age]")?.textContent ?? null, + })), + })); + } finally { + await context.close(); + } + }; + + try { + // ── Legs a, b, c: one run each, then the query and the view. ──────────── + // Token-only sign-ins (no account named), so the view's look keeps the run's window. + script("a", [{ info: event.allowed() }]); + script("b", [{ info: event.warning(0.91) }]); + script("c", [{ info: event.rejected() }]); + signIn({ tokenSource: "CLAUDE_CODE_OAUTH_TOKEN" }); + for (const leg of ["a", "b", "c"] as const) { + const task = await runTask(PEOPLE[leg], leg); + if (task.status !== "completed") failures.push(`${leg}:run — the run ended ${task.status}: ${task.error ?? ""}`); + } + + const a = claudeRow(await query(PEOPLE.a, false)); + const aWindow = a?.windows[0]; + if (a?.windows.length !== 1 || aWindow?.id !== "five_hour" || aWindow.level !== "ok" || aWindow.origin !== "run") failures.push(`a:window — the query returned ${JSON.stringify(a)}`); + if (aWindow !== undefined && aWindow.usedPercent !== null) failures.push(`a:no-number — an event with no percent reads ${aWindow.usedPercent}`); + if (aWindow !== undefined && aWindow.resetsAt !== RESETS * 1000) failures.push(`a:resets-in-ms — resetsAt ${aWindow.resetsAt}, not ${RESETS * 1000}`); + const aView = await view(PEOPLE.a, false); + const aShown = aView.windows[0]; + if (aShown === undefined || aShown.text.includes("%")) failures.push(`a:no-number — the view shows "${aShown?.text}"`); + else { + if (!aShown.level?.includes("under Claude Code's warning line")) failures.push(`a:view-level — the view says "${aShown.text}"`); + if (aShown.reset === null) failures.push(`a:view-reset — the view shows no reset: "${aShown.text}"`); + if (aShown.age === null || !aShown.age.includes("from a run")) failures.push(`a:view-source — the view doesn't say where the window came from: "${aShown.text}"`); + } + notes.push(`a: ${JSON.stringify(aWindow)}; view "${aShown?.text ?? aView.text}"`); + + const b = claudeRow(await query(PEOPLE.b, false))?.windows[0]; + if (b?.usedPercent !== 91 || b.level !== "warning") failures.push(`b:91 — the query returned ${JSON.stringify(b)}`); + const bShown = (await view(PEOPLE.b, false)).windows[0]; + if (bShown?.percent !== "91% used") failures.push(`b:91 — the view shows "${bShown?.text}"`); + notes.push(`b: ${b?.usedPercent} ${b?.level}; view "${bShown?.text}"`); + + const c = claudeRow(await query(PEOPLE.c, false))?.windows[0]; + if (c?.level !== "reached" || c.usedPercent !== null) failures.push(`c:reached — the query returned ${JSON.stringify(c)}`); + const cShown = (await view(PEOPLE.c, false)).windows[0]; + if (cShown?.level !== "limit reached" || cShown.reset === null || cShown.text.includes("%")) failures.push(`c:reached — the view shows "${cShown?.text}"`); + notes.push(`c: ${c?.level}; view "${cShown?.text}"`); + + // ── Leg d: only the run owner's row, on a run and on a talk. ──────────── + script("d-bob", [{ info: event.apiKey() }]); + script("d-alice", [{ info: event.allowed() }], { email: "alice@goal.test" }); + script("d-bob-talk", [{ info: event.warning(0.5) }], { email: "bob@goal.test" }); + const bobTask = await runTask(PEOPLE.bob, "d-bob"); + await runTask(PEOPLE.alice, "d-alice"); + const aliceRow = claudeRow(await query(PEOPLE.alice, false)); + if (aliceRow?.windows.length !== 1) failures.push(`d:owner-row — Alice's run left her row ${JSON.stringify(aliceRow)}`); + const bobBefore = claudeRow(await query(PEOPLE.bob, false)); + if ((bobBefore?.windows.length ?? 0) > 0) failures.push(`d:no-claude-window — Alice's run wrote Bob's row: ${JSON.stringify(bobBefore)}`); + const runSession = bobTask.run?.sessionId as string | undefined; + if (runSession === undefined) failures.push(`d:talk — Bob's finished task names no session: ${JSON.stringify(bobTask)}`); + else { + const talked = await labRoutes(served.origin, PEOPLE.bob).act(CODER_FLOW, runSession, "respond", { message: "How did it go? [[usage:d-bob-talk]]" }); + if (talked.status !== "completed") failures.push(`d:talk — Bob's talk ended ${talked.status}: ${talked.error ?? ""}`); + if (started_("d-bob-talk").length !== 1) failures.push(`d:talk — the talk reached the harness ${started_("d-bob-talk").length} times`); + const bobAfter = claudeRow(await query(PEOPLE.bob, false)); + const aliceAfter = claudeRow(await query(PEOPLE.alice, false)); + if (bobAfter?.windows[0]?.usedPercent !== 50) failures.push(`d:talk-writes-bob — after Bob's talk his row is ${JSON.stringify(bobAfter)}`); + if (JSON.stringify(aliceAfter?.windows.map((w) => [w.id, w.usedPercent, w.checkedAt])) !== JSON.stringify(aliceRow?.windows.map((w) => [w.id, w.usedPercent, w.checkedAt]))) { + failures.push(`d:talk-leaves-alice — Bob's talk changed Alice's row: ${JSON.stringify(aliceAfter)}`); + } + notes.push(`d: Alice ${JSON.stringify(aliceRow?.windows.map((w) => [w.id, w.level]))}; Bob before ${bobBefore?.windows.length ?? 0} window(s), after his talk ${JSON.stringify(bobAfter?.windows.map((w) => [w.id, w.usedPercent]))}`); + } + + // ── Leg e: a look on the same sign-in keeps the run's window; another account's run clears it. ── + script("e", [{ info: event.allowed() }], { email: "e@goal.test" }); + await runTask(PEOPLE.e, "e"); + signIn({ email: "e@goal.test", subscriptionType: "max" }); + const eLookedAt = Date.now(); + const eKept = claudeRow(await query(PEOPLE.e, true)); + if (eKept?.windows.length !== 1 || eKept.windows[0]!.origin !== "run") failures.push(`e:keeps-the-runs-window — after a look on the same sign-in the row is ${JSON.stringify(eKept)}`); + if (eKept?.plan !== "max") failures.push(`e:adds-the-plan — the look's plan is ${eKept?.plan}`); + notes.push(`e: after a look ${JSON.stringify(eKept?.windows.map((w) => [w.id, w.origin]))}, plan ${eKept?.plan}`); + + script("e-switch-a", [{ info: event.allowed() }, { info: event.warning(0.8, "seven_day") }], { email: "a@goal.test" }); + script("e-switch-b", [{ info: event.allowed() }], { email: "b@goal.test" }); + await runTask(PEOPLE.eSwitch, "e-switch-a"); + await runTask(PEOPLE.eSwitch, "e-switch-b"); + const switched = claudeRow(await query(PEOPLE.eSwitch, false)); + if (JSON.stringify(switched?.windows.map((w) => [w.id, w.usedPercent])) !== JSON.stringify([["five_hour", null]])) { + failures.push(`e:switch-clears — after account B's run the row still holds ${JSON.stringify(switched?.windows)}`); + } + notes.push(`e: account A then B, ${JSON.stringify(switched?.windows.map((w) => [w.id, w.usedPercent]))}`); + + realProbe(); + const empty = claudeRow(await query(PEOPLE.eEmpty, true)); + if (empty?.status !== "signed-out" || empty.windows.length > 0) failures.push(`e:signed-out — the real probe on an empty home gave ${JSON.stringify(empty)}`); + notes.push(`e: empty home "${empty?.status}" "${empty?.reason}"`); + + // ── Leg f: order, an old run, and a write that throws. ────────────────── + // Two runs at once, one person: account A's event is observed first and + // written last (its run's account answers slowly), after account B's. + const goB = join(SCRATCH, "f-b-go"); + script("f-a", [{ info: event.warning(0.8) }], { email: "a@goal.test" }, 4_000); + script("f-b", [{ info: event.allowed(), waitFor: goB }], { email: "b@goal.test" }); + const fa = runTask(PEOPLE.fOrder, "f-a"); + const fb = runTask(PEOPLE.fOrder, "f-b"); + for (const until = Date.now() + 60_000; (started_("f-a").length === 0 || started_("f-b").length === 0) && Date.now() < until; ) await sleep(100); + const [faStart] = started_("f-a"); + const [fbStart] = started_("f-b"); + await sleep(300); + writeFileSync(goB, ""); + await Promise.all([fa, fb]); + if (faStart === undefined || fbStart === undefined || Math.abs(fbStart - faStart) > 3_000) { + failures.push(`f:setup — the two runs didn't overlap (started ${faStart}, ${fbStart}), so the order isn't tested`); + } + const ordered = claudeRow(await query(PEOPLE.fOrder, false)); + if (ordered?.windows.length !== 1 || ordered.windows[0]!.level !== "ok" || ordered.windows[0]!.usedPercent !== null) { + failures.push(`f:newer-kept — account A's older event, written after account B's, left ${JSON.stringify(ordered?.windows)}`); + } + notes.push(`f: A started ${faStart}, B ${fbStart}; row ${JSON.stringify(ordered?.windows.map((w) => [w.id, w.level, w.usedPercent]))}`); + + // A run that started before a look found an API key. + const go = join(SCRATCH, "f-old-go"); + script("f-old", [{ info: event.warning(0.7), waitFor: go }], { email: "old@goal.test" }); + const old = startTask(PEOPLE.fOld, "f-old"); + for (const until = Date.now() + 60_000; started_("f-old").length === 0 && Date.now() < until; ) await sleep(200); + realProbe("sk-ant-goal-not-a-real-key"); + const keyed = claudeRow(await query(PEOPLE.fOld, true)); + writeFileSync(go, ""); + await old; + for (const until = Date.now() + 60_000; Date.now() < until; await sleep(300)) { + const listed = readFileSync(runLog, "utf8"); + if (listed.includes("RUN f-old")) break; + } + await sleep(2_000); + const afterOld = claudeRow(await query(PEOPLE.fOld, false)); + if (keyed?.status !== "api-key") failures.push(`f:setup — the look on an API key gave ${JSON.stringify(keyed)}`); + if (afterOld?.status !== "api-key" || afterOld.windows.length > 0) failures.push(`f:old-run — a run from before the look brought windows back: ${JSON.stringify(afterOld)}`); + notes.push(`f: old run after an API-key look, ${afterOld?.status} with ${afterOld?.windows.length} window(s)`); + + // A usage write that throws: the run still completes, and nothing is written. + script("f-throws", [{ info: event.allowed("goal_write_throws") }]); + const cBefore = claudeRow(await query(PEOPLE.c, false)); + const thrown = await runTask(PEOPLE.c, "f-throws"); + if (thrown.status !== "completed") failures.push(`f:write-throws — the run ended ${thrown.status}: ${thrown.error ?? ""}`); + const cAfter = claudeRow(await query(PEOPLE.c, false)); + if (JSON.stringify(cAfter?.windows) !== JSON.stringify(cBefore?.windows.map((w) => ({ ...w, ageMs: cAfter?.windows.find((x) => x.id === w.id)?.ageMs ?? w.ageMs })))) { + failures.push(`f:write-throws — the failed write changed the row: ${JSON.stringify(cAfter?.windows)}`); + } + notes.push(`f: the throwing write's run ${thrown.status}`); + + // ── Leg e, second half: a minute after the look, the same person moves to an API key. ── + const wait = eLookedAt + 61_000 - Date.now(); + if (wait > 0) { + say(`e: waiting ${Math.ceil(wait / 1000)} s for the minute to pass`); + await sleep(wait); + } + realProbe("sk-ant-goal-not-a-real-key"); + const eKey = claudeRow(await query(PEOPLE.e, true)); + if (eKey?.status !== "api-key" || eKey.windows.length > 0 || eKey.reason !== "Claude Code uses an API key: no subscription limit") { + failures.push(`e:api-key — after a move to an API key the row is ${JSON.stringify(eKey)}`); + } + notes.push(`e: then on an API key, "${eKey?.status}" "${eKey?.reason}"`); + } finally { + await browser.close(); + await served.stop(); + writeFileSync(join(SCRATCH, "lab.log"), served.log()); + } + + assertPatched(patches, REPO_ROOT, mark); + if (process.env.GOAL_KEEP_STORE === undefined) rmSync(storeDir, { recursive: true, force: true }); + for (const note of notes) say(note); + say("g: owed by a person — Jake's own Claude sign-in, a real run, then Usage beside Claude Code's /status"); + + if (control !== undefined) { + const red = control.assertions.filter((name) => failures.some((f) => f.startsWith(`${name} `))); + const missed = control.assertions.filter((name) => !red.includes(name)); + failures.unshift( + missed.length === 0 + ? `control ${control.name}: failed on ${red.join(", ")}, as it must` + : `control ${control.name}: did NOT fail on ${missed.join(", ")}, so the check can't see what the control removes`, + ); + } + return { + failures, + evidence: `legs a, b, c, d, e, f PASS in ${Math.round((Date.now() - started) / 1000)} s (leg g owed by a person: goal met except leg g)`, + }; +}); + +function ended(task: Record): boolean { + return ["completed", "errored", "failed", "cancelled"].includes(String(task.status)); +} diff --git a/goals/shift-manager/it-shows-codex-usage-left/controls.mts b/goals/shift-manager/it-shows-codex-usage-left/controls.mts index e7c7d92d2e..05ebeb2b7d 100644 --- a/goals/shift-manager/it-shows-codex-usage-left/controls.mts +++ b/goals/shift-manager/it-shows-codex-usage-left/controls.mts @@ -34,8 +34,8 @@ export const CONTROLS: Record = { patches: [ { module: USAGE, - from: String.raw`reading\.status === "windows" \? reading\.windows\.map\(\(window\) => \(\{ \.\.\.window, checkedAt, origin: "read" \}\)\) : \[\],`, - to: 'reading.status === "windows" ? reading.windows.map((window) => ({ ...window, checkedAt, origin: "read" })) : [{ id: "codex:primary", label: null, minutes: 300, usedPercent: 0, resetsAt: null, checkedAt, origin: "read" }],', + from: String.raw` windows: \[\],\n checkedAt,`, + to: ' windows: [{ id: "codex:primary", label: null, minutes: 300, usedPercent: 0, level: null, resetsAt: null, checkedAt, origin: "read" }],\n checkedAt,', }, ], does: "a harness with no number is stored as one window at 0%", @@ -46,7 +46,7 @@ export const CONTROLS: Record = { patches: [ { module: USAGE, - from: String.raw`if \(stored\.checkedAt !== null && checkedAt < stored\.checkedAt\) return \{ state: current, result: "refused-older" \};`, + from: String.raw`if \(stored\.checkedAt !== null && checkedAt < stored\.checkedAt\) return \{ state: \{\}, result: "refused-older" \};`, to: "", }, ], diff --git a/goals/shift-manager/it-shows-codex-usage-left/goal.md b/goals/shift-manager/it-shows-codex-usage-left/goal.md index eb8d3d753e..4309a4022d 100644 --- a/goals/shift-manager/it-shows-codex-usage-left/goal.md +++ b/goals/shift-manager/it-shows-codex-usage-left/goal.md @@ -43,3 +43,4 @@ The spec names *no vendor call* as where `ignore-age` fails leg e. Ignoring age | 2026-10-23 | same, `GOAL_CONTROL=last-write-wins` | n/a | FAIL (expected) | **a:leaves-the-stored-one**: the 5% refresh replaced the 77% reading stamped ahead of it. | | 2026-10-23 | same, `GOAL_CONTROL=throttle-ignores-failures` | n/a | FAIL (expected) | **g:one-child**: three looks started 2 Codex processes. | | 2026-10-23 | same, `GOAL_CONTROL=key-by-account` | n/a | FAIL (expected) | **h:no-earlier-percent**: two rows, `codex` (`api-key`) and `codex-` still holding the signed-in windows; a and d fail too, since the view's `codex` row is never written. | +| 2026-10-23 | claude/project-thread-3x3z3h (FIX-2142, pre-PR) | n/a | PASS (goal met except leg f) | 90 s, rerun over the reworked writer (run windows, `accountAt`); controls re-pointed at its new lines. All five controls fail on their assertions as before. | diff --git a/packages/claude-code/README.md b/packages/claude-code/README.md index fdf47a44aa..34c5389fba 100644 --- a/packages/claude-code/README.md +++ b/packages/claude-code/README.md @@ -537,6 +537,12 @@ Pass the same `env` you give `claudeCodeAgent`, so the probe sees the credential It never throws for any of these. It checks where a credential would come from, not that it works: a key that is set but rejected still reads as `"account"`, and fails on the run's first turn. +### Reading usage left + +On a subscription, Claude Code reports its limit windows while it runs. `claudeCodeAgent({ onUsage })` hands each one to your `HarnessUsageHook` as a `HarnessUsageReport` (`harness: "claude-code"`, the `window`, `observedAt`, `runStartedAt`, `accountId`), at least once per run. The window names the limit that binds (`five_hour` is 300 minutes, the `seven_day` family 10 080; other ids have `minutes: null`), its `resetsAt` and a `level` (`ok`, `warning`, `reached`). `usedPercent` is set only near a limit and is otherwise `null`. An API-key run reports nothing. A throw from `onUsage` is logged and the run carries on. It works on in-session blocks too. + +`readClaudeUsage({ env, timeoutMs?, signal? })` is a `HarnessUsageReader` for who is signed in: `windows` with `plan`, `accountId` and no windows (those come from runs), `api-key`, `signed-out`, or `unavailable` when the SDK isn't installed or Claude Code didn't answer (which keeps a stored reading). It never throws. `usageWindowOf(info)` maps one raw `rate_limit_event` payload to a window, and `CLAUDE_CODE_HARNESS` is `"claude-code"`. Nothing here reads a credentials file, calls a usage endpoint, or uses the SDK's experimental usage call. Guide: [reading usage left](https://flow-state.dev/docs/tools/claude-code-sdk#reading-usage-left). + ## Running tests ```bash diff --git a/packages/claude-code/src/sdk/account-probe.ts b/packages/claude-code/src/sdk/account-probe.ts index a42c2f7311..467b3bf944 100644 --- a/packages/claude-code/src/sdk/account-probe.ts +++ b/packages/claude-code/src/sdk/account-probe.ts @@ -25,8 +25,12 @@ export type ClaudeAccountProbe = * keychain or `~/.claude`) is reported by the last pair alone. */ | { status: "account"; apiKeySource?: string; tokenSource?: string; email?: string; subscriptionType?: string } - /** No credential source, or no answer in time. `reason` says which, for a message to a person. */ - | { status: "none"; reason: string } + /** + * No credential source, or no answer. `reason` says which, for a message to + * a person; `unanswered` is set when Claude Code timed out, was stopped, or + * failed, so nothing is known about the credential. + */ + | { status: "none"; reason: string; unanswered?: true } /** `@anthropic-ai/claude-agent-sdk` could not be loaded. `reason` carries the load error. */ | { status: "sdk-missing"; reason: string }; @@ -40,6 +44,8 @@ export interface ProbeClaudeAccountOptions { env: Record; /** How long to wait for Claude Code's answer before reading it as `none`. Default 10 000 ms. */ timeoutMs?: number; + /** Stops waiting and closes Claude Code when it aborts; the answer is then `none`, as on a timeout. */ + signal?: AbortSignal; /** How the SDK module is loaded. A seam for tests; the default imports the installed SDK. */ importSdk?: SdkImporter; } @@ -63,12 +69,17 @@ const DEFAULT_TIMEOUT_MS = 10_000; export async function probeClaudeAccount(options: ProbeClaudeAccountOptions): Promise { const timeoutMs = options.timeoutMs ?? DEFAULT_TIMEOUT_MS; const asked = await runIdleQuery>>( - { env: options.env, timeoutMs, ...(options.importSdk === undefined ? {} : { importSdk: options.importSdk }) }, + { + env: options.env, + timeoutMs, + ...(options.signal === undefined ? {} : { signal: options.signal }), + ...(options.importSdk === undefined ? {} : { importSdk: options.importSdk }), + }, (handle) => handle.accountInfo(), ); if (asked.status === "sdk-missing") return { status: "sdk-missing", reason: asked.reason }; - if (asked.status === "timeout") return { status: "none", reason: `Claude Code did not report an account within ${timeoutMs} ms` }; - if (asked.status === "error") return { status: "none", reason: `Claude Code could not be asked for its account: ${asked.reason}` }; + if (asked.status === "timeout") return { status: "none", reason: `Claude Code did not report an account within ${timeoutMs} ms`, unanswered: true }; + if (asked.status === "error") return { status: "none", reason: `Claude Code could not be asked for its account: ${asked.reason}`, unanswered: true }; const info = asked.value; const apiKeySource = typeof info.apiKeySource === "string" && info.apiKeySource !== "none" ? info.apiKeySource : undefined; const tokenSource = typeof info.tokenSource === "string" && info.tokenSource !== "none" ? info.tokenSource : undefined; diff --git a/packages/claude-code/src/sdk/agent.ts b/packages/claude-code/src/sdk/agent.ts index a70715d85b..0044bf6091 100644 --- a/packages/claude-code/src/sdk/agent.ts +++ b/packages/claude-code/src/sdk/agent.ts @@ -20,6 +20,7 @@ import type { HarnessCallbackContext, HarnessResolver, HarnessSessionHook, + HarnessUsageHook, } from "@flow-state-dev/core/types"; import type { UsesSlot } from "@flow-state-dev/core"; import { z } from "zod"; @@ -48,6 +49,7 @@ import { workRecorderResources, } from "./work-collections"; import { defaultResolveClaudeAgent } from "./sdk-client"; +import { CLAUDE_CODE_HARNESS, usageWindowOf } from "./usage"; import { trackProcess } from "./process-exit"; import { inProcessMcpServer, type ClaudeCodeMcpToolServer, type ClaudeCodeMcpTools } from "./mcp-tools"; import { @@ -548,6 +550,22 @@ export interface ClaudeCodeAgentOptions { * stream, so a slow hook slows the run and a throwing one fails it. */ onSession?: HarnessSessionHook; + /** + * Called with each subscription usage window Claude Code reports during the + * run (its `rate_limit_event`), so a host can record how much of the + * subscription is left. Default: unset, and the events are dropped. + * + * The report names `claude-code`, the window, when it was observed, when the + * run started, and the account the run is signed in as (its `accountInfo()` + * email, asked once at the first event; `null` when it names none). An event + * that names no window, as on an API key, is not reported. + * + * Awaited like `onSession`, so no write lands after the run ends, but **a + * throw is logged and swallowed**: a usage write never fails the run. Unlike + * `onSession`, it is allowed in-session too: it owns no part of the + * conversation, so it races nothing. + */ + onUsage?: HarnessUsageHook; /** Block name. Default `"claude-code-agent"`. */ name?: string; } @@ -765,6 +783,7 @@ export function claudeCodeAgent(options: ClaudeCodeAgentOptions = {}) { cwd: resolveCwd, resume: resolveResume, onSession, + onUsage, settingSources, env, sandbox, @@ -1010,6 +1029,9 @@ export function claudeCodeAgent(options: ClaudeCodeAgentOptions = {}) { // there, even on the rejection path that skips it) — `.current` on an // object isn't tracked that way. const inFlightRef: { current: Promise | null } = { current: null }; + // The account the run is signed in as, asked of the running query once, + // at its first usage event, so its windows are bound to it. + let runAccount: Promise | undefined; // The vendor loop, hoisted so the abort branch can wait on it (bounded) // after the race below has been lost to the signal. @@ -1060,10 +1082,11 @@ export function claudeCodeAgent(options: ClaudeCodeAgentOptions = {}) { // terminal `item.done`, or a recorder observation landing after // `stop()` and never flushed. loop = (async () => { - for await (const message of resolved.query({ + const running = resolved.query({ prompt: promptText, options: queryOptions, - })) { + }); + for await (const message of running) { // Capture the SDK session id from any message that carries it (the // `system` init message does, well before the terminal result), so an // aborted or failed run is still resumable. @@ -1116,6 +1139,34 @@ export function claudeCodeAgent(options: ClaudeCodeAgentOptions = {}) { if (ctx.signal.aborted) return; + // **A usage window, handed to the host and never failing the run.** + // Tracked like `onSession`, so it can't land after the run ends. + if (onUsage !== undefined && message.type === "rate_limit_event") { + const window = usageWindowOf(message.rate_limit_info); + if (window !== null) { + const observedAt = Date.now(); + runAccount ??= accountOf(running); + inFlightRef.current = runAccount + .then((accountId) => { + // Whose window this is isn't known: drop it rather than guess, and ask again on the next event. + if (accountId === undefined) { + runAccount = undefined; + return; + } + return onUsage({ harness: CLAUDE_CODE_HARNESS, window, observedAt, runStartedAt: dispatchedAt, accountId }, ctx); + }) + .catch((error: unknown) => { + console.warn(`[claude-code] recording a usage window failed: ${error instanceof Error ? error.message : String(error)}`); + }); + try { + await inFlightRef.current; + } finally { + inFlightRef.current = null; + } + if (ctx.signal.aborted) return; + } + } + // **Exactly the handed servers.** The init message lists every MCP // server the run connected to; one this block didn't hand it came // from somewhere the strict flags were meant to keep out. @@ -1356,6 +1407,35 @@ export function claudeCodeAgent(options: ClaudeCodeAgentOptions = {}) { * the resolver answers later is closed then, so a connection it was still * opening isn't left open. */ +/** How long a run waits for its own query to name its account. */ +const ACCOUNT_TIMEOUT_MS = 5_000; + +/** + * The email of the account a running query is signed in as, `null` when it + * names none (an OAuth token), or `undefined` when it can't be asked or + * doesn't answer in time, so a failed lookup never reads as an account change. + * Asks the query the run already holds, so no second process starts. + */ +async function accountOf(running: unknown): Promise { + const ask = (running as { accountInfo?: () => Promise<{ email?: unknown }> }).accountInfo; + if (typeof ask !== "function") return undefined; + let timer: ReturnType | undefined; + try { + const info = await Promise.race([ + ask.call(running), + new Promise((resolve) => { + timer = setTimeout(() => resolve(undefined), ACCOUNT_TIMEOUT_MS); + }), + ]); + if (info === undefined) return undefined; + return typeof info.email === "string" && info.email !== "" ? info.email : null; + } catch { + return undefined; + } finally { + clearTimeout(timer); + } +} + async function resolveMcpTools( resolve: (ctx: AgentCallbackContext) => ClaudeCodeMcpTools | Promise, ctx: AgentCallbackContext, diff --git a/packages/claude-code/src/sdk/idle-query.ts b/packages/claude-code/src/sdk/idle-query.ts index 2be4c8fe80..3b5f6b829c 100644 --- a/packages/claude-code/src/sdk/idle-query.ts +++ b/packages/claude-code/src/sdk/idle-query.ts @@ -26,6 +26,8 @@ export interface IdleQueryOptions { env: Record; /** How long to wait for the answer. */ timeoutMs: number; + /** Stops waiting and closes the query when it aborts, read as `timeout`. */ + signal?: AbortSignal; /** How the SDK module is loaded. A seam for tests; the default imports the installed SDK. */ importSdk?: SdkImporter; } @@ -57,10 +59,16 @@ export async function runIdleQuery void }, T>( } let timer: ReturnType | undefined; + let onAbort: (() => void) | undefined; let handle: H | undefined; try { + if (options.signal?.aborted) return { status: "timeout" }; handle = query({ prompt: noMessages(), options: { env: options.env } }); - const timedOut = new Promise((resolve) => (timer = setTimeout(() => resolve(TIMEOUT), options.timeoutMs))); + const timedOut = new Promise((resolve) => { + timer = setTimeout(() => resolve(TIMEOUT), options.timeoutMs); + onAbort = () => resolve(TIMEOUT); + options.signal?.addEventListener("abort", onAbort, { once: true }); + }); const value = await Promise.race([ask(handle), timedOut]); if (value === TIMEOUT) return { status: "timeout" }; return { status: "answer", value: value as T }; @@ -68,6 +76,7 @@ export async function runIdleQuery void }, T>( return { status: "error", reason: messageOf(error) }; } finally { clearTimeout(timer); + if (onAbort !== undefined) options.signal?.removeEventListener("abort", onAbort); release(); try { handle?.close?.(); diff --git a/packages/claude-code/src/sdk/index.ts b/packages/claude-code/src/sdk/index.ts index 53c4100a7b..6a3f8a7290 100644 --- a/packages/claude-code/src/sdk/index.ts +++ b/packages/claude-code/src/sdk/index.ts @@ -91,6 +91,7 @@ export { type UpsertableCollection, } from "./work-recorder"; export { ClaudeAgentSdkNotInstalledError, ClaudeAgentRunError } from "./errors"; +export { CLAUDE_CODE_HARNESS, readClaudeUsage, usageWindowOf, type ReadClaudeUsageOptions } from "./usage"; export { createWorkspaceAgentCapability, diff --git a/packages/claude-code/src/sdk/types.ts b/packages/claude-code/src/sdk/types.ts index 96d8f9eb59..3951775970 100644 --- a/packages/claude-code/src/sdk/types.ts +++ b/packages/claude-code/src/sdk/types.ts @@ -187,6 +187,12 @@ export type SdkMessageLike = }; }; } + | { + /** Sent when the subscription's limits change; read by `onUsage` (`./usage`), never translated to an item. */ + type: "rate_limit_event"; + session_id?: string; + rate_limit_info?: unknown; + } | { type: "result"; subtype?: SdkResultSubtype | string; diff --git a/packages/claude-code/src/sdk/usage.ts b/packages/claude-code/src/sdk/usage.ts new file mode 100644 index 0000000000..f53e5c95c9 --- /dev/null +++ b/packages/claude-code/src/sdk/usage.ts @@ -0,0 +1,98 @@ +/** + * Claude Code's subscription usage, in the neutral shape core declares + * (`HarnessUsageWindow`, `HarnessUsageReading`). + * + * Two sources, both sanctioned: + * + * - **A run's `rate_limit_event`.** Claude Code sends one when its limits + * change, at least once per run on a subscription. It names the window that + * binds (`rateLimitType`), its reset (`resetsAt`, epoch seconds) and a status. + * It carries a percent (`utilization`, a fraction) only near a limit. + * {@link usageWindowOf} maps one to a window; `claudeCodeAgent` hands it to + * its `onUsage` hook. + * - **The account, on demand.** {@link readClaudeUsage} asks + * `probeClaudeAccount` who is signed in. It reports the account and plan + * and no windows: no allowed source gives a percent without a run. + * + * Nothing here reads a credentials file, calls a usage endpoint, or uses the + * SDK's experimental usage call. + */ +import type { HarnessUsageLevel, HarnessUsageReading, HarnessUsageWindow } from "@flow-state-dev/core/types"; +import { probeClaudeAccount, type ClaudeAccountProbe, type ProbeClaudeAccountOptions } from "./account-probe"; + +/** The usage record's name for Claude Code. */ +export const CLAUDE_CODE_HARNESS = "claude-code"; + +/** Window lengths for the ids Claude Code sends. `overage` and anything new have none, and are labelled by id. */ +const WINDOW_MINUTES: Readonly> = { + five_hour: 300, + seven_day: 10_080, + seven_day_opus: 10_080, + seven_day_sonnet: 10_080, + seven_day_overage_included: 10_080, +}; + +const LEVELS: Readonly> = { + allowed: "ok", + allowed_warning: "warning", + rejected: "reached", +}; + +/** + * One `rate_limit_event`'s `rate_limit_info` as a usage window, or `null` + * when it names no window (an API-key run) or a status this package doesn't + * know. + * + * `utilization` is a fraction and is multiplied by 100. One above 1 is not a + * fraction, so it is dropped rather than shown as thousands of percent. + */ +export function usageWindowOf(info: unknown): HarnessUsageWindow | null { + if (typeof info !== "object" || info === null) return null; + const { rateLimitType, status, utilization, resetsAt } = info as Record; + if (typeof rateLimitType !== "string" || rateLimitType === "") return null; + const level = typeof status === "string" ? LEVELS[status] : undefined; + if (level === undefined) return null; + const fraction = typeof utilization === "number" && utilization >= 0 && utilization <= 1; + // To one decimal place, so a float's tail (0.29 * 100 = 28.999…) never shows. + const usedPercent = fraction ? Math.round(utilization * 1000) / 10 : null; + return { + id: rateLimitType, + label: null, + minutes: WINDOW_MINUTES[rateLimitType] ?? null, + usedPercent, + level, + resetsAt: typeof resetsAt === "number" && Number.isFinite(resetsAt) ? resetsAt * 1000 : null, + }; +} + +/** Options for {@link readClaudeUsage}: the probe's. */ +export type ReadClaudeUsageOptions = ProbeClaudeAccountOptions & { + /** How the account is asked for. A seam for tests; default {@link probeClaudeAccount}. */ + probe?: (options: ProbeClaudeAccountOptions) => Promise; +}; + +/** + * Read who Claude Code is signed in as, for the usage record: a + * `HarnessUsageReader` once its options are bound. + * + * An API key is `api-key`. A sign-in or an OAuth token is `windows`, with the + * email as the account id when there is one (the store keeps a hash) and the + * plan, and **no windows**: the windows come from runs. Nobody signed in is + * `signed-out`. An SDK that won't load, or a probe that got no answer, is + * `unavailable`, which leaves a stored reading in place. Never throws. + */ +export async function readClaudeUsage(options: ReadClaudeUsageOptions): Promise { + const { probe = probeClaudeAccount, ...probeOptions } = options; + let found: ClaudeAccountProbe; + try { + found = await probe(probeOptions); + } catch { + return { status: "unavailable", reason: "Claude Code's account could not be read" }; + } + if (found.status === "sdk-missing") return { status: "unavailable", reason: "Claude Code's Agent SDK is not installed" }; + // A probe that got no answer knows nothing: `unavailable` keeps the stored reading. + if (found.status === "none" && found.unanswered) return { status: "unavailable", reason: found.reason }; + if (found.status === "none") return { status: "signed-out", reason: "Claude Code is not signed in: not available automatically" }; + if (found.apiKeySource !== undefined) return { status: "api-key", reason: "Claude Code uses an API key: no subscription limit" }; + return { status: "windows", accountId: found.email ?? null, plan: found.subscriptionType ?? null, windows: [] }; +} diff --git a/packages/claude-code/test/sdk/account-probe.spec.ts b/packages/claude-code/test/sdk/account-probe.spec.ts index 113fa98c45..0bb856b552 100644 --- a/packages/claude-code/test/sdk/account-probe.spec.ts +++ b/packages/claude-code/test/sdk/account-probe.spec.ts @@ -95,7 +95,19 @@ describe("probeClaudeAccount, scripted", () => { it("says none when Claude Code does not answer within timeoutMs", async () => { const sdk = scriptedSdk(() => new Promise(() => {})); const result = await probeClaudeAccount({ env: {}, timeoutMs: 50, importSdk: sdk.importSdk }); - expect(result).toEqual({ status: "none", reason: "Claude Code did not report an account within 50 ms" }); + expect(result).toEqual({ status: "none", reason: "Claude Code did not report an account within 50 ms", unanswered: true }); + expect(sdk.seen.closed).toBe(true); + }); + + it("stops and closes Claude Code when its signal aborts, without waiting out timeoutMs", async () => { + // A usage refresh that gives up must not leave the probe's process running. + const sdk = scriptedSdk(() => new Promise(() => {})); + const stop = new AbortController(); + const started = Date.now(); + setTimeout(() => stop.abort(), 20); + const result = await probeClaudeAccount({ env: {}, timeoutMs: 60_000, signal: stop.signal, importSdk: sdk.importSdk }); + expect(result.status).toBe("none"); + expect(Date.now() - started).toBeLessThan(5_000); expect(sdk.seen.closed).toBe(true); }); @@ -106,6 +118,7 @@ describe("probeClaudeAccount, scripted", () => { expect(await probeClaudeAccount({ env: {}, importSdk: sdk.importSdk })).toEqual({ status: "none", reason: "Claude Code could not be asked for its account: spawn claude ENOENT", + unanswered: true, }); }); diff --git a/packages/claude-code/test/sdk/usage.spec.ts b/packages/claude-code/test/sdk/usage.spec.ts new file mode 100644 index 0000000000..f20f19feea --- /dev/null +++ b/packages/claude-code/test/sdk/usage.spec.ts @@ -0,0 +1,154 @@ +/** + * Claude Code's usage, two ways: a run's `rate_limit_event` handed to + * `onUsage` as one window, and `readClaudeUsage` mapping the account probe. + * The event shapes are the ones the pinned CLI builds (FIX-2142's POC): a + * percent only on a warning, as a fraction; `resetsAt` in epoch seconds; no + * window on an API key. + */ +import { describe, expect, it, vi } from "vitest"; +import { testBlock } from "@flow-state-dev/testing"; +import type { HarnessUsageReport } from "@flow-state-dev/core/types"; +import { claudeCodeAgent } from "../../src/sdk/agent"; +import { readClaudeUsage, usageWindowOf } from "../../src/sdk/usage"; +import type { SdkMessageLike } from "../../src/sdk/types"; + +const RESETS = 1_800_000_000; +const ALLOWED = { status: "allowed", resetsAt: RESETS, rateLimitType: "five_hour" }; +const WARNING = { status: "allowed_warning", resetsAt: RESETS, rateLimitType: "seven_day", utilization: 0.91 }; +const REJECTED = { status: "rejected", resetsAt: RESETS, rateLimitType: "five_hour" }; +const API_KEY = { status: "allowed" }; + +const RESULT_OK: SdkMessageLike = { type: "result", subtype: "success", result: "done", session_id: "sess_1" }; + +describe("usageWindowOf", () => { + it("reads a plain allowed event as ok with no percent, never 0", () => { + expect(usageWindowOf(ALLOWED)).toEqual({ id: "five_hour", label: null, minutes: 300, usedPercent: null, level: "ok", resetsAt: RESETS * 1000 }); + }); + + it("scales a warning's fraction to a percent", () => { + expect(usageWindowOf(WARNING)).toMatchObject({ id: "seven_day", minutes: 10_080, usedPercent: 91, level: "warning" }); + }); + + it("reads rejected as reached, and a window it has no length for by its id", () => { + expect(usageWindowOf(REJECTED)).toMatchObject({ level: "reached", usedPercent: null }); + expect(usageWindowOf({ status: "allowed", rateLimitType: "overage" })).toMatchObject({ id: "overage", minutes: null, resetsAt: null }); + }); + + it("reports nothing for an event that names no window, or a status it doesn't know", () => { + expect(usageWindowOf(API_KEY)).toBeNull(); + expect(usageWindowOf({ status: "throttled", rateLimitType: "five_hour" })).toBeNull(); + expect(usageWindowOf(undefined)).toBeNull(); + }); + + it("drops a utilization above 1 rather than show it as thousands of percent", () => { + expect(usageWindowOf({ ...WARNING, utilization: 91 })).toMatchObject({ usedPercent: null, level: "warning" }); + }); +}); + +describe("claudeCodeAgent — onUsage", () => { + /** A run that sends `events`, on a query that answers `accountInfo`. */ + function run(events: unknown[], options: { account?: () => Promise<{ email?: string }>; onUsage: (report: HarnessUsageReport) => unknown; detached?: boolean }) { + const accountInfo = vi.fn(options.account ?? (async () => ({ email: "alice@example.com", subscriptionType: "max" }))); + const block = claudeCodeAgent({ + resolveClaudeAgent: () => ({ + query: () => { + const iterable = (async function* () { + yield { type: "system", subtype: "init", session_id: "sess_1" } as SdkMessageLike; + for (const info of events) yield { type: "rate_limit_event", session_id: "sess_1", rate_limit_info: info } as SdkMessageLike; + yield RESULT_OK; + })(); + return Object.assign(iterable, { accountInfo }); + }, + }), + ...(options.detached === false ? {} : { detached: true }), + onUsage: async (report) => { + await options.onUsage(report); + }, + }); + return { block, accountInfo }; + } + + it("hands each window to the host with the run's account and start, asking the account once", async () => { + const reports: HarnessUsageReport[] = []; + const before = Date.now(); + const { block, accountInfo } = run([ALLOWED, WARNING, API_KEY], { onUsage: (r) => reports.push(r) }); + const { error } = await testBlock(block, { input: { prompt: "go" } }); + expect(error).toBeNull(); + expect(reports.map((r) => [r.harness, r.window.id, r.window.usedPercent, r.window.level, r.accountId])).toEqual([ + ["claude-code", "five_hour", null, "ok", "alice@example.com"], + ["claude-code", "seven_day", 91, "warning", "alice@example.com"], + ]); + expect(reports[0]!.runStartedAt).toBeGreaterThanOrEqual(before); + expect(reports[0]!.observedAt).toBeGreaterThanOrEqual(reports[0]!.runStartedAt); + expect(accountInfo).toHaveBeenCalledTimes(1); + }); + + it("reports no account for a token-only sign-in", async () => { + const reports: HarnessUsageReport[] = []; + const { block } = run([ALLOWED], { account: async () => ({}), onUsage: (r) => reports.push(r) }); + await testBlock(block, { input: { prompt: "go" } }); + expect(reports[0]!.accountId).toBeNull(); + }); + + it("drops a window whose account can't be read, rather than record it as a different account", async () => { + // A failed lookup read as "no account" would rebind the row and clear the real account's windows. + const reports: HarnessUsageReport[] = []; + let asked = 0; + const { block } = run([ALLOWED, WARNING], { + account: async () => { + asked += 1; + if (asked === 1) throw new Error("not ready"); + return { email: "alice@example.com" }; + }, + onUsage: (r) => reports.push(r), + }); + await testBlock(block, { input: { prompt: "go" } }); + expect(reports.map((r) => [r.window.level, r.accountId])).toEqual([["warning", "alice@example.com"]]); + }); + + it("finishes the run when the host's write throws", async () => { + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + const { block } = run([REJECTED], { + onUsage: () => { + throw new Error("store down"); + }, + }); + const { output, error } = await testBlock(block, { input: { prompt: "go" } }); + expect(error).toBeNull(); + expect((output as { status: string }).status).toBe("completed"); + expect(warn).toHaveBeenCalledWith(expect.stringContaining("store down")); + warn.mockRestore(); + }); + + it("is allowed in-session, where onSession is refused", () => { + expect(() => claudeCodeAgent({ onUsage: () => undefined })).not.toThrow(); + }); +}); + +describe("readClaudeUsage", () => { + const read = (found: Awaited[0]["probe"]>>>) => + readClaudeUsage({ env: {}, probe: async () => found }); + + it("maps a sign-in to its account and plan, with no windows", async () => { + expect(await read({ status: "account", email: "alice@example.com", subscriptionType: "max" })).toEqual({ + status: "windows", + accountId: "alice@example.com", + plan: "max", + windows: [], + }); + expect(await read({ status: "account", tokenSource: "CLAUDE_CODE_OAUTH_TOKEN" })).toEqual({ status: "windows", accountId: null, plan: null, windows: [] }); + }); + + it("says why there is no number, with no percent", async () => { + expect(await read({ status: "account", apiKeySource: "ANTHROPIC_API_KEY" })).toEqual({ status: "api-key", reason: "Claude Code uses an API key: no subscription limit" }); + expect(await read({ status: "none", reason: "x" })).toEqual({ status: "signed-out", reason: "Claude Code is not signed in: not available automatically" }); + expect((await read({ status: "sdk-missing", reason: "x" })).status).toBe("unavailable"); + }); + + it("says unavailable, not signed out, when the probe got no answer, so the stored reading stays", async () => { + expect(await read({ status: "none", reason: "Claude Code did not report an account within 10000 ms", unanswered: true })).toEqual({ + status: "unavailable", + reason: "Claude Code did not report an account within 10000 ms", + }); + }); +}); diff --git a/packages/codex/src/usage.ts b/packages/codex/src/usage.ts index 28ccb401a3..abdeee019a 100644 --- a/packages/codex/src/usage.ts +++ b/packages/codex/src/usage.ts @@ -173,6 +173,8 @@ function windowsOf(limits: RateLimitsResponse): HarnessUsageWindow[] { minutes: typeof window.windowDurationMins === "number" ? window.windowDurationMins : null, // The contract is 0 to 100; past a limit Codex can report more. usedPercent: Math.min(100, Math.max(0, window.usedPercent as number)), + // Codex always sends a percent, and no level. + level: null, // Codex sends seconds since the epoch. resetsAt: typeof window.resetsAt === "number" ? window.resetsAt * 1000 : null, }); diff --git a/packages/codex/test/usage.spec.ts b/packages/codex/test/usage.spec.ts index 7d87a56f8f..dbb019f1af 100644 --- a/packages/codex/test/usage.spec.ts +++ b/packages/codex/test/usage.spec.ts @@ -50,8 +50,8 @@ describe("readCodexUsage", () => { accountId: "acct_1", plan: "pro", windows: [ - { id: "codex:primary", label: null, minutes: 300, usedPercent: 42, resetsAt: 1_800_000_000_000 }, - { id: "codex:secondary", label: null, minutes: 10_080, usedPercent: 7.5, resetsAt: 1_800_500_000_000 }, + { id: "codex:primary", label: null, minutes: 300, usedPercent: 42, level: null, resetsAt: 1_800_000_000_000 }, + { id: "codex:secondary", label: null, minutes: 10_080, usedPercent: 7.5, level: null, resetsAt: 1_800_500_000_000 }, ], }); // The handshake was awaited: the fake refuses account/read before `initialized`. diff --git a/packages/core/README.md b/packages/core/README.md index 5746ecc7e0..757dfd8725 100644 --- a/packages/core/README.md +++ b/packages/core/README.md @@ -443,7 +443,7 @@ Set `client: { live: true }` to stream each mutation's projected `clientData` as - Types, from `@flow-state-dev/core/types`: `HarnessBlock`, `HarnessRunInput`, `HarnessRunHandle`, `HarnessRunEnvelope`, `HarnessSource`, `HarnessRunStatus`, `HarnessRunOutcome`, `HarnessRunUsage`, `HarnessRunCost`, `HarnessCostBasis`, `HarnessResolver`, `HarnessSessionHook`, `HarnessCallbackContext` - `harnessEnv({ pass })`, from the package root (options type `HarnessEnvOptions`): builds a harness's `env` from an allowlist of variable names. A harness's agent runs as a child process, and with no `env` it inherits the server's entire `process.env`. The result holds exactly the named variables that are set, read when `harnessEnv` is called. Nothing else is passed, `PATH` and `HOME` included. Values come from `process.env` by default; pass `env` to read from another source. Where there is no `process` (a browser or edge runtime) and no `env` is given, the result is empty rather than an error. - The run's plan record, from the package root: `OBSERVED_PLAN`, `observedPlanCollection`, `observedPlanItemStateSchema`, the type `ObservedOutcome` (with `observedOutcomeSchema`), the per-run list marker (`OBSERVED_PLAN_LIST`, `observedPlanListCollection`) and `replacePlanList`, which writes a whole-list plan as entries. One shape for a coding agent's own to-do list, which a harness that reports one declares and writes. A record of what the agent planned, never a work queue. To read one request's plan, `planRowsPrefix(requestId)` gives the key prefix its entries are under (pass `OBSERVED_PLAN_LIST` as the second argument for its markers), to hand a collection read as `topicPrefix`; `planKeySegment(value)` is how a request id or run name is written as one key segment. The keys, collections and both helpers are also exported from `@flow-state-dev/core/types`, for browser code. Guide: [the run's plan](https://flow-state.dev/docs/tools/coding-agents#the-runs-plan). -- Subscription usage: `harnessUsageWindowSchema` from the package root, and the types `HarnessUsageWindow`, `HarnessUsageReading` and `HarnessUsageReader` from `@flow-state-dev/core/types`. A reader `(options?: { signal }) => Promise` asks a harness how much of each subscription window is used, with no run. A reading is `windows` (each window's `id`, `label`, `minutes`, `usedPercent` 0 to 100 and `resetsAt` in epoch ms, plus `plan` and `accountId`) or one of `api-key`, `signed-out` and `unavailable`, each with a `reason` and no number. A reader returns those states rather than throwing. Guide: [subscription usage](https://flow-state.dev/docs/tools/coding-agents#subscription-usage). +- Subscription usage: `harnessUsageWindowSchema` from the package root, and the types `HarnessUsageWindow`, `HarnessUsageLevel`, `HarnessUsageReading`, `HarnessUsageReader`, `HarnessUsageReport` and `HarnessUsageHook` from `@flow-state-dev/core/types`. A reader `(options?: { signal }) => Promise` asks a harness how much of each subscription window is used, with no run. A reading is `windows` (each window's `id`, `label`, `minutes`, `usedPercent` 0 to 100 or `null` when the vendor sent none, `level` (`"ok"`, `"warning"`, `"reached"` or `null`) and `resetsAt` in epoch ms, plus `plan` and `accountId`) or one of `api-key`, `signed-out` and `unavailable`, each with a `reason` and no number. A reader returns those states rather than throwing. A harness that reports usage during a run calls a `HarnessUsageHook` `(report, ctx)` with one `HarnessUsageReport` per window: `{ harness, window, observedAt, runStartedAt, accountId }`. The schema reads a window with no `level` as `level: null`. Guide: [subscription usage](https://flow-state.dev/docs/tools/coding-agents#subscription-usage). - `harnessFailureKind(error)`, from the package root (type `HarnessFailureKind` from `@flow-state-dev/core/types`): `"unknown-conversation"` when a harness refused to continue the conversation a run asked to `resume`, `null` for any other failure. Follows `cause`, so it reads through wrapped errors. A harness package sets it on the error it throws. Two packages implement it: [`@flow-state-dev/claude-code`](../claude-code) (`claude-code/sdk`) and [`@flow-state-dev/codex`](../codex) (`codex/sdk`). [`@flow-state-dev/harness-manager`](../harness-manager) drives either. Full guide: [Coding agents](https://flow-state.dev/docs/tools/coding-agents). diff --git a/packages/core/src/types/harness.ts b/packages/core/src/types/harness.ts index b83f758c27..99acfdc60b 100644 --- a/packages/core/src/types/harness.ts +++ b/packages/core/src/types/harness.ts @@ -30,7 +30,8 @@ * - {@link HarnessFailureKind} / {@link harnessFailureKind} — the neutral * classification a harness puts on the error it fails with. * - {@link HarnessUsageWindow} / {@link harnessUsageWindowSchema}, - * {@link HarnessUsageReading} and {@link HarnessUsageReader} — a + * {@link HarnessUsageReading}, {@link HarnessUsageReader}, + * {@link HarnessUsageReport} and {@link HarnessUsageHook} — a * subscription's usage windows as a vendor reports them, and the on-demand * read a host registers per harness. * @@ -343,12 +344,14 @@ export function harnessFailureKind(error: unknown): HarnessFailureKind | null { /** * One window of a harness's subscription limit, as its vendor reported it: how - * much of it is used, and when it resets. + * much of it is used, how close to its limit it is, and when it resets. * * A vendor names a window by its length (Codex: 300 minutes, 10 080 minutes), * by its own word for it (Claude Code: `five_hour`), or both. `minutes` and - * `id` carry what was said, and nothing is derived: `usedPercent` is 0 to 100 - * exactly as sent. + * `id` carry what was said. `usedPercent` is 0 to 100; a vendor that sends a + * fraction has it scaled by 100, and nothing else is derived. A vendor that + * sends no percent leaves it `null`, never 0, and says how close the window + * is through `level` instead. */ export interface HarnessUsageWindow { /** The vendor's id for this window, unique within one reading. */ @@ -357,18 +360,28 @@ export interface HarnessUsageWindow { label: string | null; /** The window's length in minutes, or `null` when the vendor didn't say. */ minutes: number | null; - /** How much of the window is used, 0 to 100, as the vendor sent it. */ - usedPercent: number; + /** How much of the window is used, 0 to 100, or `null` when the vendor sent no percent. */ + usedPercent: number | null; + /** + * How close the window is to its limit, as the vendor said: `ok` (under its + * warning line), `warning` (near the limit) or `reached`. `null` when the + * vendor sends a percent and no level (Codex). + */ + level: HarnessUsageLevel | null; /** When the window resets, in epoch ms, or `null` when the vendor didn't say. */ resetsAt: number | null; } -/** Runtime validator for {@link HarnessUsageWindow}. */ +/** How close a {@link HarnessUsageWindow} is to its limit. */ +export type HarnessUsageLevel = "ok" | "warning" | "reached"; + +/** Runtime validator for {@link HarnessUsageWindow}. `level` defaults to `null`, so a window stored before it existed still parses. */ export const harnessUsageWindowSchema = z.object({ id: z.string(), label: z.string().nullable().default(null), minutes: z.number().nullable().default(null), - usedPercent: z.number().finite().min(0).max(100), + usedPercent: z.number().finite().min(0).max(100).nullable(), + level: z.enum(["ok", "warning", "reached"]).nullable().default(null), resetsAt: z.number().nullable().default(null), }); @@ -402,3 +415,33 @@ export type HarnessUsageReading = * aborts a read the caller stopped waiting for. */ export type HarnessUsageReader = (options?: { signal?: AbortSignal }) => Promise; + +/** + * One window a harness heard about during a run: what a {@link HarnessUsageHook} + * is handed. + * + * A run reports the pool it draws from, not its own share, and only the + * window its vendor named. The account and the run's start let a host keep + * one account's windows apart from another's: a run that started before a + * look found no subscription belongs to the old sign-in. + */ +export interface HarnessUsageReport { + /** The harness that heard it, by the name a usage record is kept under (`claude-code`). */ + harness: string; + window: HarnessUsageWindow; + /** When the vendor said it, in epoch ms. */ + observedAt: number; + /** When the run started, in epoch ms. */ + runStartedAt: number; + /** The vendor's id for the account the run is signed in as, or `null` when it named none. Never shown. */ + accountId: string | null; +} + +/** + * Called by a harness when its vendor reports a usage window during a run, so + * the host can record it. The twin of {@link HarnessSessionHook}. + * + * A harness awaits it, so a write can't land after the run ends, but a throw + * never fails the run: usage is a reading, not part of the work. + */ +export type HarnessUsageHook = (report: HarnessUsageReport, ctx: HarnessCallbackContext) => void | Promise; diff --git a/packages/core/src/types/index.ts b/packages/core/src/types/index.ts index 4a7e827b58..0e90c37784 100644 --- a/packages/core/src/types/index.ts +++ b/packages/core/src/types/index.ts @@ -467,7 +467,10 @@ export type { HarnessRunUsage, HarnessSessionHook, HarnessSource, + HarnessUsageHook, + HarnessUsageLevel, HarnessUsageReader, HarnessUsageReading, + HarnessUsageReport, HarnessUsageWindow, } from "./harness"; diff --git a/packages/harness-manager/README.md b/packages/harness-manager/README.md index 51454e5472..4390596264 100644 --- a/packages/harness-manager/README.md +++ b/packages/harness-manager/README.md @@ -26,9 +26,9 @@ const manager = harnessManager({ workspace: { root, sourceRepo, baseRef }, runTimeoutMs: 30 * 60_000, - // The slot. Called once; the manager hands down three feeds. - harness: ({ cwd, resume, onSession }) => - claudeCodeAgent({ cwd, resume, onSession, detached: true, recordWork: true }), + // The slot. Called once; the manager hands down four feeds. + harness: ({ cwd, resume, onSession, onUsage }) => + claudeCodeAgent({ cwd, resume, onSession, onUsage, detached: true, recordWork: true }), }); ``` @@ -185,13 +185,14 @@ picture. ## The slot -The manager calls your factory once, with three feeds: +The manager calls your factory once, with four feeds (`HarnessFeeds`): | Feed | What it does | |---|---| | `cwd` | Where this run works: the checkout the manager derived and provisioned. | | `resume` | Which session this attempt continues, or `null` for a fresh one. | | `onSession` | Called by the harness when it names its session, so the manager records it. | +| `onUsage` | Called by the harness with each subscription usage window its run reports, so the manager stores it in the run owner's [usage record](#subscription-usage). A harness that reports no usage ignores it. | They are the harness contract's own signatures, declared in `@flow-state-dev/core`. Each is handed the block's context, not the run's prompt. @@ -211,7 +212,7 @@ harness: ({ cwd, resume, onSession }) => cursorAgent({ cwd, resume, onSession, agent: { model: { id: "composer-2.5" } } }), ``` -Any block that takes the three feeds and returns a run handle conforming to `@flow-state-dev/core`'s harness contract is one this manager can drive. +Any block that takes the four feeds and returns a run handle conforming to `@flow-state-dev/core`'s harness contract is one this manager can drive. Claude Code needs `detached: true`. Leave it out and your flow fails to build, naming the entry. Codex and Cursor need no equivalent. @@ -353,13 +354,14 @@ const actions = { }; ``` -- **Input** `{ refresh?: boolean }`. **Output** `{ harnesses: HarnessUsageRow[] }`, sorted by harness: `status` (`windows`, `api-key`, `signed-out`, `unavailable`), `reason`, `plan`, `windows` (each with `checkedAt`, `origin`, `ageMs` and `rolledOver`; `origin` is `"read"` for a reader's refresh), `checkedAt`, `ageMs` and `refreshError`. No model call. +- **Input** `{ refresh?: boolean }`. **Output** `{ harnesses: HarnessUsageRow[] }`, sorted by harness: `status` (`windows`, `api-key`, `signed-out`, `unavailable`), `reason`, `plan`, `windows` (each with `checkedAt`, `origin`, `ageMs` and `rolledOver`; `origin` is `"read"` for a reader's refresh and `"run"` for a window a run reported), `checkedAt`, `ageMs` and `refreshError`. No model call. - **Without `refresh`, nothing is read.** A harness with a reader and no reading yet is listed as `unavailable`, `"Not checked yet"`. - **`refresh: true` runs the readers**, at most once a minute per user and harness (`USAGE_REFRESH_EVERY_MS`), counted from the last attempt whether or not it worked. Refreshes that arrive together share one call. A reader that throws or outlasts `timeoutMs` (default `USAGE_READ_TIMEOUT_MS`, 10 s) leaves the stored reading and sets `refreshError`. - **`ageMs` and `rolledOver` are worked out by the server's clock** when the query runs. `rolledOver` is true once a window's `resetsAt` has passed. -- **One current record per user and harness.** A reading replaces the whole record, so a sign-in change leaves no earlier account's windows. A reading older than the stored one is refused. The account is kept as a hash; no email is stored. No history. +- **Run windows land on their own.** Pass the slot's `onUsage` to the harness (`claudeCodeAgent` takes it) and each window a run reports is stored with `origin: "run"`; the session door stores a talk's windows for the user talking. For Claude Code, register `readClaudeUsage` from `@flow-state-dev/claude-code/sdk` as the `claude-code` reader: it names the account and plan, and the windows come from runs. +- **One current record per user and harness.** A reading replaces the last reading's windows and keeps run windows it doesn't name; on a shared window the newer wins. A newer report for a window replaces an older one, and an older one arriving late is ignored, as is a reading older than the stored one. A reading or run window from a different account clears the earlier account's windows and plan. A no-number reading (`api-key`, `signed-out`) clears every window, and a run that started before it doesn't bring them back. The account is kept as a hash; no email is stored. No history. -The record is `harnessUsageCollection` (`HARNESS_USAGE`, `harness-usage/**`, user scope, not client-readable, not writable by a model). On the server, `readHarnessUsage(ctx, now, expected?)` returns the query's rows without refreshing, and `recordHarnessUsage(ctx, harness, reading, checkedAt)` is the one writer, for a harness that reports usage during a run; it answers `"recorded"`, `"kept-stored"` or `"refused-older"`. Both throw when the calling block doesn't have the collection under `HARNESS_USAGE`. +The record is `harnessUsageCollection` (`HARNESS_USAGE`, `harness-usage/**`, user scope, not client-readable, not writable by a model). On the server, `readHarnessUsage(ctx, now, expected?)` returns the query's rows without refreshing, and `recordHarnessUsage` is the one writer, in two forms: `recordHarnessUsage(ctx, report)` for one window a run reported (the manager's and session door's `onUsage` call it), and `recordHarnessUsage(ctx, harness, reading, checkedAt)` for a whole reading. It answers `"recorded"`, `"kept-stored"` or `"refused-older"`. Both throw when the calling block doesn't have the collection under `HARNESS_USAGE`. ## The export surface, in two halves diff --git a/packages/harness-manager/src/harness-usage.ts b/packages/harness-manager/src/harness-usage.ts index 8699cbd56e..39cdf51701 100644 --- a/packages/harness-manager/src/harness-usage.ts +++ b/packages/harness-manager/src/harness-usage.ts @@ -20,12 +20,34 @@ * - **A failed read keeps the last good one.** An `unavailable` reading over a * stored reading leaves it and records why as `refreshError`. It replaces * only a row that never held one. - * - **One current record per harness.** The key is the harness alone, and a - * reading replaces the whole record, account included, so a sign-in change - * leaves no old account's percents behind. The account is kept only as - * `hashKeySegment(accountId)`; no email is stored. + * - **One current record per harness.** The key is the harness alone. The + * account is kept only as `hashKeySegment(accountId)`; no email is stored. + * - **Windows belong to one account at a time.** A reading or a run window + * from another account is a *rebind*: it replaces the account, clears the + * plan and every window the new account didn't send. No account (a + * token-only sign-in) counts as an account of its own, so a move either way + * between it and a named one is a rebind too. Every write that is taken + * advances `accountAt` (a look that got no answer excepted: it learned no + * account), and a rebind or a no-number reading observed before it is + * refused whole: a late event from the old account can't take the record + * back, and a late look can't clear a newer run's window. A run that started + * before a look confirmed no subscription stays refused after a newer run + * fills the record again. * - **Latest only.** No history: a window's next reading overwrites it. * + * ## Two forms of one write + * + * A *reading* (an on-demand read) replaces the record: its status, plan and + * its own windows, keeping the windows a run wrote that it doesn't name while + * the account is the same. A *run window* (a {@link HarnessUsageReport}, from + * a harness's `onUsage` during a run) upserts one window by id with + * `origin: "run"` and leaves the record's clock and plan alone. It + * is refused when that window holds a newer reading, and it lifts a + * no-number status (`api-key`, `signed-out`) only when the run started after + * that status was read: a run already going when a look found an API key + * belongs to the old sign-in. On a window id both forms write, the newer + * `checkedAt` wins. + * * ## Read when someone looks * * A registered reader runs only when a caller asks for `refresh`, at most once @@ -44,7 +66,7 @@ */ import { defineCapability, defineResourceCollection, handler } from "@flow-state-dev/core"; import { updateStateWith } from "@flow-state-dev/core/helpers"; -import type { BlockContext, HarnessUsageReader, HarnessUsageReading } from "@flow-state-dev/core/types"; +import type { BlockContext, HarnessUsageReader, HarnessUsageReading, HarnessUsageReport } from "@flow-state-dev/core/types"; import { harnessUsageWindowSchema } from "@flow-state-dev/core"; import { z } from "zod"; import { assertSafeSegment, hashKeySegment } from "./workspace"; @@ -60,12 +82,11 @@ export const USAGE_READ_TIMEOUT_MS = 10_000; /** * One stored window: the vendor's reading, when it was made, and how it - * arrived. `origin` is `"read"` for an on-demand read; a run-time writer adds - * its own value. + * arrived: `"read"` for an on-demand read, `"run"` for a run's `onUsage`. */ const storedUsageWindowSchema = harnessUsageWindowSchema.extend({ checkedAt: z.number(), - origin: z.string().default("read"), + origin: z.enum(["read", "run"]).default("read"), }); /** @@ -79,6 +100,18 @@ export const harnessUsageStateSchema = z.object({ reason: z.string().nullable().default("Not checked yet"), /** `hashKeySegment` of the vendor's account id, or `null`. Never the id or an email. */ account: z.string().nullable().default(null), + /** + * When `account` was last set or confirmed by a write, in epoch ms. An + * account change observed before it is refused. A row stored before it + * existed reads 0, so its first rebind wins. + */ + accountAt: z.number().default(0), + /** + * When a look last confirmed no subscription (`api-key` or `signed-out`), in + * epoch ms, or `null`. A run that started by then belongs to the old sign-in, + * even after a newer run has filled the record again. + */ + noSubscriptionAt: z.number().nullable().default(null), plan: z.string().nullable().default(null), windows: z.array(storedUsageWindowSchema).default([]), /** When the stored reading was made, in epoch ms. `null` until one was. */ @@ -107,51 +140,154 @@ export const harnessUsageCollection = defineResourceCollection({ llmWritable: false, }); -/** The collection, installed on the action that reads it. */ +/** + * The collection, installed on the action that reads it and on the manager + * and session door whose runs write it. + */ const harnessUsageCapability = defineCapability({ name: "harness-manager:usage", resources: { [HARNESS_USAGE]: harnessUsageCollection }, }); -/** What {@link recordHarnessUsage} did with a reading. */ +/** What {@link recordHarnessUsage} did with a reading or a run window. */ export type UsageWrite = "recorded" | "kept-stored" | "refused-older"; /** - * Record one reading of `harness`'s usage, made at `checkedAt` (epoch ms). + * Record one reading of `harness`'s usage, made at `checkedAt` (epoch ms), or + * one window a run heard about (`report`). * - * @returns `recorded` when the reading replaced the record; `kept-stored` - * when it was `unavailable` over a stored reading, which stays (its reason - * becomes `refreshError`); `refused-older` when the stored reading is newer. + * @returns `recorded` when it was written; `kept-stored` when a reading was + * `unavailable` over a stored reading, which stays (its reason becomes + * `refreshError`); `refused-older` when what is stored is newer, or when it + * is a run window from a run that started before the stored status was + * read. */ -export async function recordHarnessUsage( +export function recordHarnessUsage(ctx: BlockContext, report: HarnessUsageReport): Promise; +export function recordHarnessUsage( ctx: BlockContext, harness: string, reading: HarnessUsageReading, checkedAt: number, +): Promise; +export async function recordHarnessUsage( + ctx: BlockContext, + harnessOrReport: string | HarnessUsageReport, + reading?: HarnessUsageReading, + checkedAt?: number, ): Promise { + const harness = typeof harnessOrReport === "string" ? harnessOrReport : harnessOrReport.harness; const record = await rowOf(ctx, harness); const outcome = await updateStateWith, UsageWrite>(record, (current) => { const stored = harnessUsageStateSchema.parse(current ?? {}); - if (stored.checkedAt !== null && checkedAt < stored.checkedAt) return { state: current, result: "refused-older" }; - if (reading.status === "unavailable" && stored.checkedAt !== null && stored.status !== "unavailable") { - return { state: { ...current, refreshError: reading.reason }, result: "kept-stored" }; - } - const next: HarnessUsageState = { - ...stored, - status: reading.status, - reason: reading.status === "windows" ? null : reading.reason, - account: reading.status === "windows" && reading.accountId !== null ? hashKeySegment(reading.accountId) : null, - plan: reading.status === "windows" ? reading.plan : null, - windows: - reading.status === "windows" ? reading.windows.map((window) => ({ ...window, checkedAt, origin: "read" })) : [], - checkedAt, - refreshError: reading.status === "unavailable" ? reading.reason : null, - }; - return { state: next as unknown as Record, result: "recorded" }; + const merged = + typeof harnessOrReport === "string" ? applyReading(stored, reading!, checkedAt!) : applyRunWindow(stored, harnessOrReport); + if (merged.result !== "recorded" && merged.result !== "kept-stored") return { state: current, result: merged.result }; + return { state: { ...current, ...merged.state } as unknown as Record, result: merged.result }; }); return outcome ?? "refused-older"; } +/** What a merge made of the stored record: the next state, or why it was refused. */ +type Merged = { state: Partial; result: UsageWrite }; + +/** + * The account step both forms share. `incoming` is the hashed account the + * write came from, or `null` when it named none, which is an account of its + * own: a token-only sign-in names none. + * + * @returns `same` when the account doesn't change, `rebind` when it does and + * the write was observed after `accountAt`, `older` when it does and wasn't. + */ +function accountStep(stored: HarnessUsageState, incoming: string | null, observedAt: number): "same" | "rebind" | "older" { + if (incoming === stored.account) return "same"; + return observedAt > stored.accountAt ? "rebind" : "older"; +} + +/** A full reading over the stored record. Pure. */ +export function applyReading(stored: HarnessUsageState, reading: HarnessUsageReading, checkedAt: number): Merged { + if (stored.checkedAt !== null && checkedAt < stored.checkedAt) return { state: {}, result: "refused-older" }; + // A failed look keeps what is stored, a run's windows included before any look has landed. + const holdsSomething = stored.checkedAt !== null || stored.windows.length > 0; + if (reading.status === "unavailable" && holdsSomething && stored.status !== "unavailable") { + return { state: { refreshError: reading.reason }, result: "kept-stored" }; + } + if (reading.status !== "windows") { + // No number: every window goes, run-written ones too, and so does the + // account. Not over a write observed after this look was. + if (checkedAt < stored.accountAt) return { state: {}, result: "refused-older" }; + // A look that got no answer learned no account, so it sets no account clock. + const answered = reading.status !== "unavailable"; + return { + state: { + status: reading.status, + reason: reading.reason, + account: null, + accountAt: answered ? checkedAt : stored.accountAt, + ...(answered ? { noSubscriptionAt: checkedAt } : {}), + plan: null, + windows: [], + checkedAt, + refreshError: reading.status === "unavailable" ? reading.reason : null, + }, + result: "recorded", + }; + } + const incoming = reading.accountId === null ? null : hashKeySegment(reading.accountId); + const step = accountStep(stored, incoming, checkedAt); + if (step === "older") return { state: {}, result: "refused-older" }; + const own = reading.windows.map((window) => ({ ...window, checkedAt, origin: "read" as const })); + const named = new Set(own.map((window) => window.id)); + // A run's windows this reading doesn't name stay while the account does; on a + // shared id, the newer reading wins whichever wrote it. + const kept = step === "rebind" ? [] : stored.windows.filter((window) => window.origin === "run"); + const windows = [ + ...own.map((window) => { + const run = kept.find((k) => k.id === window.id); + return run !== undefined && run.checkedAt > window.checkedAt ? run : window; + }), + ...kept.filter((window) => !named.has(window.id)), + ]; + return { + state: { + status: "windows", + reason: null, + account: incoming, + accountAt: step === "rebind" ? checkedAt : Math.max(stored.accountAt, checkedAt), + plan: reading.plan, + windows, + checkedAt, + refreshError: null, + }, + result: "recorded", + }; +} + +/** One window a run heard about, over the stored record. Pure. */ +export function applyRunWindow(stored: HarnessUsageState, report: HarnessUsageReport): Merged { + // A run already going when a look found no subscription belongs to the old sign-in. + // A look that failed (`unavailable`) found nothing, so it doesn't hold a run back. + if (stored.noSubscriptionAt !== null && report.runStartedAt <= stored.noSubscriptionAt) { + return { state: {}, result: "refused-older" }; + } + const incoming = report.accountId === null ? null : hashKeySegment(report.accountId); + const step = accountStep(stored, incoming, report.observedAt); + if (step === "older") return { state: {}, result: "refused-older" }; + const others = step === "rebind" ? [] : stored.windows.filter((window) => window.id !== report.window.id); + const prior = step === "rebind" ? undefined : stored.windows.find((window) => window.id === report.window.id); + if (prior !== undefined && prior.checkedAt >= report.observedAt) return { state: {}, result: "refused-older" }; + return { + state: { + status: "windows", + reason: null, + account: incoming, + accountAt: step === "rebind" ? report.observedAt : Math.max(stored.accountAt, report.observedAt), + ...(step === "rebind" ? { plan: null } : {}), + windows: [...others, { ...report.window, checkedAt: report.observedAt, origin: "run" as const }], + }, + result: "recorded", + }; +} + /** One window as the query returns it: the stored reading, its age, and whether its reset has passed. */ export type HarnessUsageWindowRow = z.infer & { ageMs: number; rolledOver: boolean }; diff --git a/packages/harness-manager/src/manager.ts b/packages/harness-manager/src/manager.ts index acd493b93f..dfdbd387e4 100644 --- a/packages/harness-manager/src/manager.ts +++ b/packages/harness-manager/src/manager.ts @@ -47,6 +47,7 @@ import type { HarnessBlock, HarnessResolver, HarnessSessionHook, + HarnessUsageHook, } from "@flow-state-dev/core/types"; import { renderSteers, showTakenSteers } from "@flow-state-dev/orchestration"; import { TaskEndsForGoodError, taskWorkerInputSchema } from "@flow-state-dev/orchestration/task-board"; @@ -104,6 +105,7 @@ import { withHoldRetries, } from "./held"; import { askMarkerPath, readAskMarker } from "./ask"; +import { HARNESS_USAGE, harnessUsageCollection, recordHarnessUsage } from "./harness-usage"; import { readRunCommits, type RunCommits } from "./commits"; import { HARNESS_RUN_OWNER_KEY, @@ -570,6 +572,13 @@ export interface HarnessFeeds { * id onto the run row. **The sole writer of that field.** */ onSession: HarnessSessionHook; + /** + * Called by the harness with each subscription usage window its vendor + * reports during the run, writing it into the run owner's usage record + * (`harness-usage/**`, through `recordHarnessUsage`). A harness that reports + * no usage ignores it. + */ + onUsage: HarnessUsageHook; } /** @@ -1224,6 +1233,9 @@ function createManagerCapability(options: { // question a child session writes is the one the coordinator session // reads — one registration, not two storage slots that look alike. [INBOX]: inboxCollection, + // Where `onUsage` records the subscription windows a run reports, in the + // run owner's row. The usage query declares the same definition object. + [HARNESS_USAGE]: harnessUsageCollection, // Declared so the fence can read the LIVE claim off the board row. The // board declares the same definition object, so this is one registration // rather than a second storage slot that looks like the first. A board @@ -1371,7 +1383,7 @@ export function harnessManager(options: ManagerOptions): TaskWorker { // Merging the manager's entries LAST would prevent all three, and it is the // wrong fix: the host's declaration would simply not work, with nothing // anywhere saying why. This fails loudly, naming the key. - const RESERVED_ACCESSORS = new Set([RUNS, INBOX, ...(boardCollectionId !== undefined ? [boardCollectionId] : [])]); + const RESERVED_ACCESSORS = new Set([RUNS, INBOX, HARNESS_USAGE, ...(boardCollectionId !== undefined ? [boardCollectionId] : [])]); /** Refuse a capability set that claims one of the manager's accessors. */ const assertClaimsNothingReserved = ( @@ -1391,7 +1403,7 @@ export function harnessManager(options: ManagerOptions): TaskWorker { throw new Error( `[harness-manager] a capability on \`uses\` declares collection(s) ` + `${claimed.map((k) => `"${k}"`).join(", ")}, which the manager owns (${when}) — ` + - `"${RUNS}" is the run record, "${INBOX}" is the question inbox` + + `"${RUNS}" is the run record, "${INBOX}" is the question inbox, "${HARNESS_USAGE}" is the usage record` + (boardCollectionId !== undefined ? `, and "${boardCollectionId}" is the board ledger the attempt fence reads` : "") + @@ -2652,6 +2664,12 @@ export function harnessManager(options: ManagerOptions): TaskWorker { "the harness named its session", ); }, + // **Not fenced, unlike `onSession`.** A replaced attempt's window is still + // a true reading of the pool, and it lands in the run owner's own row, not + // on the run record a replacement holds. The harness swallows a throw. + onUsage: async (report, ctx) => { + await recordHarnessUsage(ctx, report); + }, }); /** diff --git a/packages/harness-manager/src/session-door.ts b/packages/harness-manager/src/session-door.ts index 5698a6ae66..6f20b75b00 100644 --- a/packages/harness-manager/src/session-door.ts +++ b/packages/harness-manager/src/session-door.ts @@ -28,6 +28,7 @@ import { import { z } from "zod"; import { resolveOwnership, type HarnessSlot } from "./manager"; import { readRunRow, RUNS, runRecordCollection, runTopic } from "./run-record"; +import { HARNESS_USAGE, harnessUsageCollection, recordHarnessUsage } from "./harness-usage"; import { acquireCheckout, releaseCheckout, runBoardSegment } from "./workspace"; /** What the door takes: the line. Shift Manager builds it without knowing the kind. */ @@ -148,7 +149,8 @@ export function harnessSessionDoor(options: HarnessSessionDoorOptions) { const name = options.name ?? "harness-manager"; const capability = defineCapability({ name: `harness-manager-session-door:${name}`, - resources: { [RUNS]: runRecordCollection }, + // The usage record a talk's harness writes its windows into, as a run's does. + resources: { [RUNS]: runRecordCollection, [HARNESS_USAGE]: harnessUsageCollection }, }) as unknown as DefinedCapability; const uses = [capability, ...(options.uses ?? [])]; const talkTimeoutMs = options.talkTimeoutMs ?? DEFAULT_TALK_TIMEOUT_MS; @@ -166,6 +168,10 @@ export function harnessSessionDoor(options: HarnessSessionDoorOptions) { cwd: (ctx) => stateOf(ctx as never).cwd ?? "", resume: (ctx) => stateOf(ctx as never).resume, onSession: () => undefined, + // The person talking is the principal, so the window lands in their row. + onUsage: async (report, ctx) => { + await recordHarnessUsage(ctx, report); + }, }); const decide = handler({ diff --git a/packages/harness-manager/test/harness-usage.spec.ts b/packages/harness-manager/test/harness-usage.spec.ts index 22ed5c2e43..3f08627059 100644 --- a/packages/harness-manager/test/harness-usage.spec.ts +++ b/packages/harness-manager/test/harness-usage.spec.ts @@ -7,14 +7,18 @@ import { describe, expect, it } from "vitest"; import { z } from "zod"; import { defineCapability, defineFlow, handler } from "@flow-state-dev/core"; -import type { BlockContext, HarnessUsageReader, HarnessUsageReading } from "@flow-state-dev/core/types"; +import type { BlockContext, HarnessUsageReader, HarnessUsageReading, HarnessUsageReport } from "@flow-state-dev/core/types"; import { createFlowState, inMemoryStores, runAction } from "@flow-state-dev/engine"; import { HARNESS_USAGE, + applyReading, + applyRunWindow, harnessUsageAction, harnessUsageCollection, + harnessUsageStateSchema, recordHarnessUsage, type HarnessUsageRow, + type HarnessUsageState, } from "../src/harness-usage"; const NOW = Date.now(); @@ -23,8 +27,8 @@ const SIGNED_IN: HarnessUsageReading = { accountId: "acct_1", plan: "pro", windows: [ - { id: "codex:primary", label: null, minutes: 300, usedPercent: 42, resetsAt: NOW + 3_600_000 }, - { id: "codex:secondary", label: null, minutes: 10_080, usedPercent: 7, resetsAt: NOW + 86_400_000 }, + { id: "codex:primary", label: null, minutes: 300, usedPercent: 42, level: null, resetsAt: NOW + 3_600_000 }, + { id: "codex:secondary", label: null, minutes: 10_080, usedPercent: 7, level: null, resetsAt: NOW + 86_400_000 }, ], }; @@ -39,6 +43,13 @@ function lab(readers: Record, timeoutMs?: number) { outcome: await recordHarnessUsage(ctx, input.harness, input.reading as HarnessUsageReading, input.checkedAt), }), }); + const runWrite = handler({ + name: "usage-run-write", + inputSchema: z.object({ report: z.unknown() }), + outputSchema: z.object({ outcome: z.string() }), + uses: [defineCapability({ name: "usage-run-write", resources: { [HARNESS_USAGE]: harnessUsageCollection } })], + execute: async (input, ctx: BlockContext) => ({ outcome: await recordHarnessUsage(ctx, input.report as HarnessUsageReport) }), + }); const raw = handler({ name: "usage-raw", inputSchema: z.object({}), @@ -52,6 +63,7 @@ function lab(readers: Record, timeoutMs?: number) { actions: { harnessUsage: harnessUsageAction({ readers, ...(timeoutMs === undefined ? {} : { timeoutMs }) }), write: { block: write }, + runWrite: { block: runWrite }, raw: { block: raw }, }, } as never)({ id: "usage-flow" } as never); @@ -210,3 +222,162 @@ describe("recordHarnessUsage — the one writer", () => { expect(JSON.stringify(row)).not.toContain("@"); }); }); + +/** A `five_hour` window as Claude Code reports it during a run. */ +function runReport(overrides: Partial & { usedPercent?: number | null; level?: "ok" | "warning" | "reached" } = {}): HarnessUsageReport { + const { usedPercent = null, level = "ok", ...rest } = overrides; + return { + harness: "claude-code", + window: { id: "five_hour", label: null, minutes: 300, usedPercent, level, resetsAt: NOW + 3_600_000 }, + observedAt: NOW, + runStartedAt: NOW - 1_000, + accountId: "alice@example.com", + ...rest, + }; +} + +const EMPTY: HarnessUsageState = harnessUsageStateSchema.parse({}); + +/** Apply a merge's result as the writer does. */ +function after(stored: HarnessUsageState, merged: { state: Partial }): HarnessUsageState { + return harnessUsageStateSchema.parse({ ...stored, ...merged.state }); +} + +describe("recordHarnessUsage — a run's one window", () => { + it("writes the run owner's row only, through the same collection the query reads", async () => { + const { act, usage } = lab({}); + expect(await act("runWrite", { report: runReport({ usedPercent: 91, level: "warning" }) }, "alice")).toEqual({ outcome: "recorded" }); + const [alice] = await usage(false, "alice"); + expect(alice).toMatchObject({ harness: "claude-code", status: "windows", checkedAt: null }); + expect(alice!.windows.map((w) => [w.id, w.usedPercent, w.level, w.origin])).toEqual([["five_hour", 91, "warning", "run"]]); + expect(await usage(false, "bob")).toEqual([]); + }); + + it("parses a row stored before level and accountAt existed", () => { + const legacy = harnessUsageStateSchema.parse({ + status: "windows", + account: "abc", + windows: [{ id: "codex:primary", label: null, minutes: 300, usedPercent: 42, resetsAt: null, checkedAt: 1, origin: "read" }], + checkedAt: 1, + }); + expect(legacy.accountAt).toBe(0); + expect(legacy.windows[0]!.level).toBeNull(); + }); +}); + +describe("applyRunWindow", () => { + it("upserts one window by id and keeps the others, leaving the record's clock and plan", () => { + const stored = after(EMPTY, applyReading(EMPTY, { status: "windows", accountId: "alice@example.com", plan: "max", windows: [] }, NOW - 5_000)); + const withWeek = after(stored, applyRunWindow(stored, runReport({ window: { id: "seven_day", label: null, minutes: 10_080, usedPercent: null, level: "ok", resetsAt: null } }))); + const both = after(withWeek, applyRunWindow(withWeek, runReport({ observedAt: NOW + 1 }))); + expect(both.windows.map((w) => w.id).sort()).toEqual(["five_hour", "seven_day"]); + expect(both).toMatchObject({ checkedAt: NOW - 5_000, plan: "max" }); + }); + + it("refuses an event older than the stored window", () => { + const stored = after(EMPTY, applyRunWindow(EMPTY, runReport({ usedPercent: 91, level: "warning", observedAt: NOW }))); + expect(applyRunWindow(stored, runReport({ observedAt: NOW - 1 })).result).toBe("refused-older"); + }); + + it("doesn't let a run that started before a look found an API key bring windows back", () => { + const keyed = after(EMPTY, applyReading(EMPTY, { status: "api-key", reason: "key" }, NOW)); + expect(applyRunWindow(keyed, runReport({ runStartedAt: NOW - 1, observedAt: NOW + 10 })).result).toBe("refused-older"); + expect(applyRunWindow(keyed, runReport({ runStartedAt: NOW + 1, observedAt: NOW + 10 })).result).toBe("recorded"); + }); + + it("lets a run that was already going fill a record a failed look left unavailable", () => { + // A failed look learned nothing about the sign-in, unlike one that found an API key. + const failed = after(EMPTY, applyReading(EMPTY, { status: "unavailable", reason: "no answer" }, NOW)); + const recovered = applyRunWindow(failed, runReport({ runStartedAt: NOW - 1, observedAt: NOW + 10 })); + expect(recovered.result).toBe("recorded"); + expect(after(failed, recovered)).toMatchObject({ status: "windows" }); + }); + + it("takes a named run's window observed before a failed look that landed first", () => { + // The run's account lookup can take seconds, so its event may land after a later look. + const failed = after(EMPTY, applyReading(EMPTY, { status: "unavailable", reason: "no answer" }, NOW)); + const late = applyRunWindow(failed, runReport({ accountId: "a@example.com", runStartedAt: NOW - 100, observedAt: NOW - 50 })); + expect(late.result).toBe("recorded"); + }); + + it("keeps holding back a run from before a sign-out after a newer run fills the record", () => { + const out = after(EMPTY, applyReading(EMPTY, { status: "signed-out", reason: "no sign-in" }, NOW)); + const fresh = after(out, applyRunWindow(out, runReport({ accountId: "b@example.com", runStartedAt: NOW + 100, observedAt: NOW + 200 }))); + const stale = applyRunWindow(fresh, runReport({ accountId: "a@example.com", runStartedAt: NOW - 100, observedAt: NOW + 300 })); + expect(stale.result).toBe("refused-older"); + expect(fresh.windows).toHaveLength(1); + }); + + it("rebinds to a new account whole: the old account's windows and plan go", () => { + const a = after(EMPTY, applyReading(EMPTY, { status: "windows", accountId: "a@example.com", plan: "max", windows: [] }, NOW - 10_000)); + const aRun = after(a, applyRunWindow(a, runReport({ accountId: "a@example.com", window: { id: "seven_day", label: null, minutes: 10_080, usedPercent: 80, level: "warning", resetsAt: null }, observedAt: NOW - 5_000 }))); + const b = after(aRun, applyRunWindow(aRun, runReport({ accountId: "b@example.com", observedAt: NOW }))); + expect(b.windows.map((w) => w.id)).toEqual(["five_hour"]); + expect(b).toMatchObject({ plan: null, accountAt: NOW }); + // Account A's event, retried after B's, is refused whole. + expect(applyRunWindow(b, runReport({ accountId: "a@example.com", observedAt: NOW - 1 })).result).toBe("refused-older"); + }); + + it("treats a move between no account and a named one, either way, as a rebind", () => { + const seven = { id: "seven_day", label: null, minutes: 10_080, usedPercent: null, level: "ok" as const, resetsAt: null }; + const anon = after(EMPTY, applyRunWindow(EMPTY, runReport({ accountId: null, window: seven }))); + const named = after(anon, applyRunWindow(anon, runReport({ accountId: "b@example.com", observedAt: NOW + 1 }))); + expect(named.windows.map((w) => w.id)).toEqual(["five_hour"]); + const tokenOnly = after(named, applyRunWindow(named, runReport({ accountId: null, observedAt: NOW + 2, window: seven }))); + expect(tokenOnly.account).toBeNull(); + expect(tokenOnly.windows.map((w) => w.id)).toEqual(["seven_day"]); + }); + + it("refuses a late event from another account once the stored one is confirmed after it (A, B late, A)", () => { + const a = after(EMPTY, applyRunWindow(EMPTY, runReport({ accountId: "a@example.com", observedAt: NOW }))); + // A look confirms A at NOW + 2; B's event, observed at NOW + 1, lands after it. + const confirmed = after(a, applyReading(a, { status: "windows", accountId: "a@example.com", plan: "max", windows: [] }, NOW + 2)); + expect(confirmed.accountAt).toBe(NOW + 2); + expect(applyRunWindow(confirmed, runReport({ accountId: "b@example.com", observedAt: NOW + 1 })).result).toBe("refused-older"); + }); +}); + +describe("applyReading over run windows", () => { + const stored = after(EMPTY, applyRunWindow(EMPTY, runReport({ usedPercent: 91, level: "warning" }))); + + it("keeps a run's window on the same account and adds the plan", () => { + const read = after(stored, applyReading(stored, { status: "windows", accountId: "alice@example.com", plan: "max", windows: [] }, NOW + 1_000)); + expect(read.plan).toBe("max"); + expect(read.windows.map((w) => [w.id, w.usedPercent, w.origin])).toEqual([["five_hour", 91, "run"]]); + }); + + it("keeps the newer of a run window and a read window on a shared id", () => { + const window = { id: "five_hour", label: null, minutes: 300, usedPercent: 50, level: null, resetsAt: null }; + const older = applyReading(stored, { status: "windows", accountId: "alice@example.com", plan: "max", windows: [window] }, NOW - 1); + // The record had no clock, so the reading is taken, but the run's newer window stays. + expect(after(stored, older).windows.map((w) => [w.usedPercent, w.origin])).toEqual([[91, "run"]]); + }); + + it("clears a run's windows on a look from a token-only sign-in after a named one", () => { + const named = after(EMPTY, applyRunWindow(EMPTY, runReport({ usedPercent: 91, level: "warning" }))); + const read = after(named, applyReading(named, { status: "windows", accountId: null, plan: null, windows: [] }, NOW + 1)); + expect(read.windows).toEqual([]); + expect(read.account).toBeNull(); + }); + + it("refuses a no-number look observed before a run's window it would clear", () => { + // The look saw an API key at NOW - 1; a run that started after it wrote at NOW; the look lands last. + expect(applyReading(stored, { status: "api-key", reason: "key" }, NOW - 1).result).toBe("refused-older"); + }); + + it("keeps a run's window when the first look fails, with the failure beside it", () => { + // No look has landed yet, so the record has no clock: the run's window is still what is known. + const merged = applyReading(stored, { status: "unavailable", reason: "Claude Code did not answer" }, NOW + 1); + expect(merged.result).toBe("kept-stored"); + const kept = after(stored, merged); + expect(kept.windows.map((w) => [w.usedPercent, w.origin])).toEqual([[91, "run"]]); + expect(kept.refreshError).toBe("Claude Code did not answer"); + }); + + it("clears every window on a move to an API key or another account", () => { + expect(after(stored, applyReading(stored, { status: "api-key", reason: "key" }, NOW + 1)).windows).toEqual([]); + const b = after(stored, applyReading(stored, { status: "windows", accountId: "b@example.com", plan: "pro", windows: [] }, NOW + 1)); + expect(b.windows).toEqual([]); + }); +}); + diff --git a/packages/shift-manager/README.md b/packages/shift-manager/README.md index f75fd52000..59ddae6259 100644 --- a/packages/shift-manager/README.md +++ b/packages/shift-manager/README.md @@ -229,7 +229,7 @@ A Lab with no Shift Coordinator needs nothing in its config. Every screen works, - **A project.** Your project coordinator, the project's workstreams and their progress, your own workstreams' boards, and its brief. See [A project](#a-project). - **A workstream.** On a project, your own workstream opens on its lead's conversation, and someone else's on its details and latest report. Its board has five columns: QUEUED, RUNNING, IN REVIEW, NEEDS YOU, DONE; a done task is one line, its id and title. - **A task.** One task's run, live, with Interrupt. See [A task](#a-task). -- **Usage** (`/usage`). How much of each coding agent's subscription you have used: one card per agent, with each limit window's percent and reset time, the plan, and where the numbers came from and how old they are. Opening Usage asks for a fresh reading; the Lab asks each agent at most once a minute. A window whose reset has passed shows "reset at …, not checked since" instead of a percent. An agent with no number (an API key, nobody signed in, a read that failed) says why, with no percent. Usage reads the `harnessUsage` query on whichever of the Lab's flows declares it (see [`harnessUsageAction`](../harness-manager/README.md#subscription-usage)); a Lab with none shows "No usage to show". A Lab opened with `openLab({ usage: { codex: reader } })` puts the query on its projects flow with those readers. The [`devteam`](#the-devteam-profile) profile registers `readCodexUsage` from `@flow-state-dev/codex`, which works only on your own machine. +- **Usage** (`/usage`). How much of each coding agent's subscription you have used: one card per agent, with each limit window's percent and reset time, the plan, and where the numbers came from and how old they are. Opening Usage asks for a fresh reading; the Lab asks each agent at most once a minute. A window whose reset has passed shows "reset at …, not checked since" instead of a percent. An agent with no number (an API key, nobody signed in, a read that failed) says why, with no percent. A window at its limit says "limit reached", and one the agent sent no percent for says how close it is instead: "under Claude Code's warning line" or "near its limit". A window a coding run reported says "from a run, 3 min ago". Usage reads the `harnessUsage` query on whichever of the Lab's flows declares it (see [`harnessUsageAction`](../harness-manager/README.md#subscription-usage)); a Lab with none shows "No usage to show". A Lab opened with `openLab({ usage: { codex: reader } })` puts the query on its projects flow with those readers. The [`devteam`](#the-devteam-profile) profile registers `readCodexUsage` from `@flow-state-dev/codex`, which works only on your own machine, and `readClaudeUsage` from `@flow-state-dev/claude-code/sdk` as the `claude-code` reader, so its Claude Code card names the plan and fills in windows as runs report them. - **Settings** (`/settings`). Your keys for the Lab's MCP servers: each server the Lab offers, *set* with when, or *not set*, a field to set or replace your key, and Clear. The page never shows a key back. A Lab opened with `openLab({ mcpServers })` serves the route the page writes through, `/api/flows/mcp-credentials`, which takes the caller from the Lab's own principal resolver; a Lab with none says it offers no servers. **Talking to a worker.** Start a line with `@` and a worker's name to send it to that worker's task in this workstream. If it has several, the composer asks which. If it has none, Send is off and says so. In a task, the composer sends to that task's run. From Inbox, the reply box sends to the worker that asked, if its worker flow takes messages. A worker whose worker flow takes no message gets a reply box that says so. diff --git a/packages/shift-manager/src/lib/usage.ts b/packages/shift-manager/src/lib/usage.ts index cd3a68daae..1b86413177 100644 --- a/packages/shift-manager/src/lib/usage.ts +++ b/packages/shift-manager/src/lib/usage.ts @@ -8,6 +8,7 @@ * "rolled over" is the Lab's, worked out by its clock, never this page's. */ import { useCallback, useEffect, useState } from "react"; +import type { HarnessUsageLevel } from "@flow-state-dev/core/types"; import { ActionRefused, runAction } from "./action"; import { newSessionId } from "./ids"; import { useLab } from "./lab-data"; @@ -22,9 +23,14 @@ export type UsageWindow = { id: string; label: string | null; minutes: number | null; - usedPercent: number; + /** `null` when the harness sent no percent: never shown as 0. */ + usedPercent: number | null; + /** How close to its limit, when the harness said: `ok`, `warning`, `reached`. `null` from Codex. */ + level: HarnessUsageLevel | null; resetsAt: number | null; checkedAt: number; + /** `read` when a look read it, `run` when a coding run reported it. */ + origin: string; ageMs: number; rolledOver: boolean; }; @@ -131,6 +137,18 @@ export function percentLabel(usedPercent: number): string { return `${usedPercent}% used`; } +/** + * What a window with no percent says instead, from its level: "limit + * reached", "near its limit", or "under Claude Code's warning line". Never a + * number the harness didn't send. + */ +export function levelLabel(level: UsageWindow["level"], harness: string): string { + if (level === "reached") return "limit reached"; + if (level === "warning") return "near its limit"; + if (level === "ok") return `under ${harnessName(harness)}'s warning line`; + return "no percent reported"; +} + /** A harness's name as the user reads it: `codex` is Codex, `claude-code` is Claude Code. */ export function harnessName(harness: string): string { return harness diff --git a/packages/shift-manager/src/surfaces/Usage.tsx b/packages/shift-manager/src/surfaces/Usage.tsx index d0e69cd49f..0d768720b9 100644 --- a/packages/shift-manager/src/surfaces/Usage.tsx +++ b/packages/shift-manager/src/surfaces/Usage.tsx @@ -5,19 +5,23 @@ * A number is never shown without where it came from and how old it is. A * window whose reset has passed shows when it reset instead of its old * percent, and a harness with no number (an API key, nobody signed in, a read - * that failed) says why, with no percent at all. + * that failed) says why, with no percent at all. A window the harness sent no + * percent for (Claude Code below its warning line, or at its limit) says how + * close it is instead, and a window a run reported says so, with its own age. */ import { EmptyState, Meta, ScreenTitle, SectionFailure } from "../components/ui"; import type { LoadedSnapshot } from "../lib/derive"; -import { ageLabel, harnessName, percentLabel, resetLabel, useHarnessUsage, windowName, type HarnessUsage, type UsageWindow } from "../lib/usage"; +import { ageLabel, harnessName, levelLabel, percentLabel, resetLabel, useHarnessUsage, windowName, type HarnessUsage, type UsageWindow } from "../lib/usage"; -function WindowLine({ window }: { window: UsageWindow }) { +function WindowLine({ window, harness }: { window: UsageWindow; harness: string }) { return (
  • {windowName(window)} @@ -27,15 +31,26 @@ function WindowLine({ window }: { window: UsageWindow }) { {window.resetsAt === null ? "reset" : `reset at ${resetLabel(window.resetsAt)}`}, not checked since ) : ( - - - {percentLabel(window.usedPercent)} - + + {window.level === "reached" || window.usedPercent === null ? ( + + {levelLabel(window.level, harness)} + + ) : ( + + {percentLabel(window.usedPercent)} + + )} {window.resetsAt === null ? null : ( resets at {resetLabel(window.resetsAt)} )} + {window.origin === "run" ? ( + + from a run, {ageLabel(window.ageMs)} + + ) : null} )}
  • @@ -54,7 +69,11 @@ function HarnessCard({ row }: { row: HarnessUsage }) { )} - {row.ageMs === null ? "not checked yet" : `from ${name}, checked ${ageLabel(row.ageMs)}`} + {row.ageMs !== null + ? `from ${name}, checked ${ageLabel(row.ageMs)}` + : row.windows.length > 0 + ? `from ${name}'s runs` + : "not checked yet"} {row.status === "windows" ? ( @@ -65,7 +84,7 @@ function HarnessCard({ row }: { row: HarnessUsage }) { ) : (
      {row.windows.map((window) => ( - + ))}
    ) diff --git a/packages/shift-manager/teams/devteam/fsd.fsdev.config.mts b/packages/shift-manager/teams/devteam/fsd.fsdev.config.mts index 4e198d72e7..4ac19f8af3 100644 --- a/packages/shift-manager/teams/devteam/fsd.fsdev.config.mts +++ b/packages/shift-manager/teams/devteam/fsd.fsdev.config.mts @@ -51,7 +51,8 @@ import { sqliteStores } from "@flow-state-dev/store-sqlite"; import { readDeclaredRoster } from "@flow-state-dev/workforce/loader"; import { CHECKOUT, CHECKOUTS } from "./fsd-checkouts.mts"; import { FSD_DOCUMENTS, FSD_MCP_SERVERS, FSD_PROJECTS, FSD_SKILLS } from "./fsd-projects.mts"; -import { errorReporterHarness, startHarness } from "./harness.mts"; +import { readClaudeUsage } from "@flow-state-dev/claude-code/sdk"; +import { claudeCodeEnv, errorReporterHarness, startHarness } from "./harness.mts"; import { LAB_TREE, openLab } from "./host.mts"; import { CODER_FLOW } from "./workforce/flows/workers/coder.mts"; import { EM_FLOW } from "./workforce/flows/workers/em.mts"; @@ -92,6 +93,11 @@ export const lab = await openLab({ // servers with the Lab user's key, and drafts with Claude Code from this // checkout's code, whichever coder the Lab runs. errorReporter: { harness: errorReporterHarness, cwd: CHECKOUT, tracker: { project: "shared/fsd" } }, + // Who Claude Code is signed in as, read when someone opens Usage; its + // windows come from the coder's runs. + usage: { + "claude-code": ({ signal } = {}) => readClaudeUsage({ env: claudeCodeEnv(), ...(signal === undefined ? {} : { signal }) }), + }, }); export default lab.state; diff --git a/packages/shift-manager/teams/devteam/fsdev.config.mts b/packages/shift-manager/teams/devteam/fsdev.config.mts index 1c2b499522..c4b5f944c7 100644 --- a/packages/shift-manager/teams/devteam/fsdev.config.mts +++ b/packages/shift-manager/teams/devteam/fsdev.config.mts @@ -41,9 +41,11 @@ * checkout's code. This team names no tracker, so reports are kept, not * filed; a draft needs a Claude Code account, and waits for one otherwise. * - * - **Codex usage left.** Usage shows how much of each Codex window this - * machine's Codex sign-in has used, read from the Codex CLI when someone - * looks, at most once a minute. Nothing is read until then. + * - **Usage left.** Usage shows how much of each Codex window this machine's + * Codex sign-in has used, read from the Codex CLI when someone looks, at + * most once a minute. Nothing is read until then. For Claude Code, a look + * reads who is signed in, and each window comes from a coding run, as + * Claude Code reports it. * * - **A store that survives a restart.** SQLite, at `DEVTEAM_STORE` or * `packages/shift-manager/.fsdev/devteam.sqlite` by default. Workers the chief of @@ -58,12 +60,13 @@ import { mkdirSync, readFileSync } from "node:fs"; import { dirname, join } from "node:path"; import { fileURLToPath } from "node:url"; +import { readClaudeUsage } from "@flow-state-dev/claude-code/sdk"; import { readCodexUsage } from "@flow-state-dev/codex"; import { harnessEnv } from "@flow-state-dev/core"; import { sqliteStores } from "@flow-state-dev/store-sqlite"; import { readDeclaredRoster } from "@flow-state-dev/workforce/loader"; import type { AskFeature } from "./ask.mts"; -import { errorReporterHarness, startHarness } from "./harness.mts"; +import { claudeCodeEnv, errorReporterHarness, startHarness } from "./harness.mts"; import { CHECKOUT } from "./fsd-checkouts.mts"; import { LAB_CROWD, LAB_TREE, LAB_USERS, openLab } from "./host.mts"; import { createBareRemote } from "./scratch-repo.mts"; @@ -194,6 +197,7 @@ const lab = await openLab({ usage: { codex: ({ signal } = {}) => readCodexUsage({ env: harnessEnv({ pass: CODEX_USAGE_PASS }), ...(signal === undefined ? {} : { signal }) }), + "claude-code": ({ signal } = {}) => readClaudeUsage({ env: claudeCodeEnv(), ...(signal === undefined ? {} : { signal }) }), }, }); diff --git a/packages/shift-manager/teams/devteam/harness-stub.mts b/packages/shift-manager/teams/devteam/harness-stub.mts index 8bbf23da7a..399aa191a9 100644 --- a/packages/shift-manager/teams/devteam/harness-stub.mts +++ b/packages/shift-manager/teams/devteam/harness-stub.mts @@ -25,6 +25,7 @@ import type { HarnessCallbackContext, HarnessRunOutcome, HarnessRunStatus, + HarnessUsageHook, } from "@flow-state-dev/core/types"; /** One run the stub was asked to perform, exactly as it arrived. */ @@ -113,6 +114,8 @@ export interface HarnessStub { cwd: (ctx: HarnessCallbackContext) => string | Promise; resume: (ctx: HarnessCallbackContext) => string | null | Promise; onSession: (sessionId: string, ctx: HarnessCallbackContext) => void | Promise; + /** Where a run's usage windows go. The stub reports none. */ + onUsage: HarnessUsageHook; /** The Lab's relayed MCP servers. The stub takes none, and says so when the project names some. */ mcpTools?: ProjectMcpToolsResolver; }) => HarnessBlock; diff --git a/packages/shift-manager/teams/devteam/harness.mts b/packages/shift-manager/teams/devteam/harness.mts index b7e0436d44..c34869d200 100644 --- a/packages/shift-manager/teams/devteam/harness.mts +++ b/packages/shift-manager/teams/devteam/harness.mts @@ -36,7 +36,8 @@ import { type ErrorReporterHarnessFeeds, type ProjectMcpToolsResolver, } from "@flow-state-dev/workforce"; -import type { HarnessBlock, HarnessCallbackContext } from "@flow-state-dev/core/types"; +import type { HarnessBlock } from "@flow-state-dev/core/types"; +import type { HarnessFeeds as ManagerHarnessFeeds } from "@flow-state-dev/harness-manager"; import { harnessStub, type StubRun } from "./harness-stub.mts"; import { commitAll, hasChanges } from "./scratch-repo.mts"; @@ -48,15 +49,13 @@ export const HARNESS_CHOICES = ["stub", "claude-code"] as const; export type HarnessChoice = (typeof HARNESS_CHOICES)[number]; /** - * The three feeds the harness contract hands a slot, and the Lab's fourth: - * the run's MCP servers, relayed through the Lab's own connection, when the - * Lab offers any (`openLab({ mcpServers })`). A harness that takes relayed - * tools hands them to its run; one that doesn't says so in the run's session. + * The feeds the harness manager hands a slot, and the Lab's own: the run's + * MCP servers, relayed through the Lab's own connection, when the Lab offers + * any (`openLab({ mcpServers })`). A harness that takes relayed tools hands + * them to its run; one that doesn't says so in the run's session. Claude Code + * hands `onUsage` the usage windows its runs report; the stub reports none. */ -export interface HarnessFeeds { - cwd: (ctx: HarnessCallbackContext) => string | Promise; - resume: (ctx: HarnessCallbackContext) => string | null | Promise; - onSession: (sessionId: string, ctx: HarnessCallbackContext) => void | Promise; +export interface HarnessFeeds extends ManagerHarnessFeeds { mcpTools?: ProjectMcpToolsResolver; } diff --git a/packages/shift-manager/teams/devteam/workforce/flows/workers/coder.mts b/packages/shift-manager/teams/devteam/workforce/flows/workers/coder.mts index 5b2021dc96..c5ba1820b6 100644 --- a/packages/shift-manager/teams/devteam/workforce/flows/workers/coder.mts +++ b/packages/shift-manager/teams/devteam/workforce/flows/workers/coder.mts @@ -51,11 +51,12 @@ import type { BlockDefinition } from "@flow-state-dev/core/types"; import { harnessManager, harnessSessionDoor, + type HarnessFeeds, type HarnessJudge, type PhaseSpec, type WorkspaceConfig, } from "@flow-state-dev/harness-manager"; -import type { HarnessBlock, HarnessCallbackContext } from "@flow-state-dev/core/types"; +import type { HarnessBlock } from "@flow-state-dev/core/types"; import { assertDispatchEntrySupported, taskWorkerInputSchema } from "@flow-state-dev/orchestration/task-board"; import type { Task, TaskWorker } from "@flow-state-dev/orchestration/tasks"; import { @@ -89,15 +90,11 @@ export interface CoderWorkerFlowOptions { /** * **The slot.** How this worker flow is pointed at a coding harness. * - * One expression, handed the three feeds the harness contract declares. The - * gate puts a scripted stub here; the honesty check puts a real coding agent + * One expression, handed the feeds the harness manager declares. The gate + * puts a scripted stub here; the honesty check puts a real coding agent * here. Nothing else differs between the two. Called once per manager. */ - harness: (feeds: { - cwd: (ctx: HarnessCallbackContext) => string | Promise; - resume: (ctx: HarnessCallbackContext) => string | null | Promise; - onSession: (sessionId: string, ctx: HarnessCallbackContext) => void | Promise; - }) => HarnessBlock; + harness: (feeds: HarnessFeeds) => HarnessBlock; /** * Where a feature task works: a fixed repository (`{ root, sourceRepo, * baseRef }`), or a workspace host whose source is `projectWorkspace()`, so diff --git a/packages/shift-manager/test/mcp-servers.test.ts b/packages/shift-manager/test/mcp-servers.test.ts index 2da1661a81..f5c2178df1 100644 --- a/packages/shift-manager/test/mcp-servers.test.ts +++ b/packages/shift-manager/test/mcp-servers.test.ts @@ -296,7 +296,7 @@ describe("the harness a Lab runs", () => { const { claudeCodeHarness } = await import("../teams/devteam/harness.mts"); process.env.FSD_TEST_LINEAR_SEED = SENTINEL; try { - const feeds = { cwd: () => "/w", resume: () => null, onSession: () => {} }; + const feeds = { cwd: () => "/w", resume: () => null, onSession: () => {}, onUsage: () => {} }; claudeCodeHarness(feeds); claudeCodeHarness({ ...feeds, mcpTools: Object.assign(async () => ({}), { named: async () => [] }) }); } finally { @@ -305,7 +305,7 @@ describe("the harness a Lab runs", () => { } const [without, withTools] = handed; expect(Object.keys(without!).sort()).toEqual( - ["allowedTools", "cwd", "detached", "env", "onSession", "permissionMode", "recordWork", "resume", "systemPrompt"].sort(), + ["allowedTools", "cwd", "detached", "env", "onSession", "onUsage", "permissionMode", "recordWork", "resume", "systemPrompt"].sort(), ); const { mcpTools, uses, ...rest } = withTools!; expect(rest).toEqual(without); diff --git a/packages/shift-manager/test/usage.test.ts b/packages/shift-manager/test/usage.test.ts index 3ad4a71c61..2e648958a9 100644 --- a/packages/shift-manager/test/usage.test.ts +++ b/packages/shift-manager/test/usage.test.ts @@ -6,7 +6,7 @@ import { describe, expect, it } from "vitest"; import { ClientHttpError } from "@flow-state-dev/client"; import type { LabClients } from "../src/lib/connection"; import { readerSessionId } from "../src/lib/reads"; -import { ageLabel, harnessName, percentLabel, runUsageQuery, toHarnessUsage, windowName } from "../src/lib/usage"; +import { ageLabel, harnessName, levelLabel, percentLabel, runUsageQuery, toHarnessUsage, windowName } from "../src/lib/usage"; describe("Usage labels", () => { it("names a window by its length, after its limit's name when it has one", () => { @@ -34,6 +34,13 @@ describe("Usage labels", () => { expect(percentLabel(42)).toBe("42% used"); }); + it("says how close a window with no percent is, never as a number", () => { + expect(levelLabel("ok", "claude-code")).toBe("under Claude Code's warning line"); + expect(levelLabel("warning", "claude-code")).toBe("near its limit"); + expect(levelLabel("reached", "claude-code")).toBe("limit reached"); + expect(levelLabel(null, "claude-code")).not.toMatch(/\d/); + }); + it("names harnesses for the user, and keeps only rows that are rows", () => { expect(harnessName("codex")).toBe("Codex"); expect(harnessName("claude-code")).toBe("Claude Code");