Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .changeset/claude-code-usage-left.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
"@flow-state-dev/core": patch
Comment thread
jhoffner marked this conversation as resolved.
Resolved
"@flow-state-dev/claude-code": patch
"@flow-state-dev/codex": patch
"@flow-state-dev/harness-manager": minor
"@flow-state-dev/shift-manager": patch
---

See how much of Claude Code's subscription is used. Core's `HarnessUsageWindow` gains `level` (`ok`, `warning`, `reached`, or `null`) and its `usedPercent` may be `null` when a harness sends no percent; core adds `HarnessUsageHook` and `HarnessUsageReport`, the run-time twin of `HarnessSessionHook`. `claudeCodeAgent` takes `onUsage` and hands it each window a run's `rate_limit_event` reports, with the run's account; `readClaudeUsage` reads who is signed in for the usage query, and `probeClaudeAccount` takes a `signal` that stops and closes the probe. `@flow-state-dev/harness-manager` feeds `onUsage` at both of its harness slots (`HarnessFeeds.onUsage`, now required, so code that builds `HarnessFeeds` itself must add it), writing into the run owner's usage record, and keeps a stored record's account and windows together across a sign-in change. Codex windows now carry `level: null`. Shift Manager's Usage view shows Claude Code's windows, saying "limit reached" or "under Claude Code's warning line" where there is no percent (FIX-2142).
61 changes: 47 additions & 14 deletions apps/docs/docs/orchestration/harness-manager.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ To let a user type into a run's session, build the door with `harnessSessionDoor

## Choosing the harness

The harness is a slot. You pass a factory; the manager calls it once and hands it three things.
The harness is a slot. You pass a factory; the manager calls it once and hands it four things.

```ts
import { harnessManager } from "@flow-state-dev/harness-manager";
Expand All @@ -77,8 +77,8 @@ const manager = harnessManager({
phase: implementPhase(),
workspace: { root, sourceRepo, baseRef },
runTimeoutMs: 30 * 60_000,
harness: ({ cwd, resume, onSession }) =>
claudeCodeAgent({ cwd, resume, onSession, detached: true, recordWork: true }),
harness: ({ cwd, resume, onSession, onUsage }) =>
claudeCodeAgent({ cwd, resume, onSession, onUsage, detached: true, recordWork: true }),
});
```

Expand All @@ -97,11 +97,11 @@ Code needs `detached: true`: it keeps session state by default, and a dispatched
row refuses a block that does. Codex keeps none, so it needs no equivalent. Each
spells its sandbox its own way. None of it reaches the manager.

**What the manager hands down.** `cwd` says where this run works: the checkout the manager derived and provisioned. `resume` says which conversation this attempt continues, or `null` for a fresh one. `onSession` is what the harness calls when it names its session, so the manager can record it.
**What the manager hands down.** `cwd` says where this run works: the checkout the manager derived and provisioned. `resume` says which conversation this attempt continues, or `null` for a fresh one. `onSession` is what the harness calls when it names its session, so the manager can record it. `onUsage` is what the harness calls with each subscription usage window its run reports, so the manager can store it in the run owner's [usage record](#subscription-usage). A harness that reports no usage, such as Codex, has no use for it.

**What the manager never does.** It does not read anything vendor-specific off the handle, and it does not tell the harness what model to use, what tools to allow, or how to sandbox itself. Those are yours, written inside the factory.

The three feeds are declared in `@flow-state-dev/core`. Any block that takes them and returns a conforming run handle is one this manager can drive. [Coding agents](/docs/tools/coding-agents) covers the contract in full.
`HarnessFeeds`, exported from `@flow-state-dev/harness-manager`, types the four feeds, and each signature is declared in `@flow-state-dev/core`. Any block that takes them and returns a conforming run handle is one this manager can drive. [Coding agents](/docs/tools/coding-agents) covers the contract in full.

## What a phase is

Expand Down Expand Up @@ -271,7 +271,7 @@ harness, sorted by name:
reason: null, // why there is no number, for the other three
plan: "plus",
windows: [
{ id: "codex:primary", label: null, minutes: 300, usedPercent: 42, resetsAt: 1798020000000,
{ id: "codex:primary", label: null, minutes: 300, usedPercent: 42, level: null, resetsAt: 1798020000000,
checkedAt: 1798012000000, origin: "read", ageMs: 65000, rolledOver: false },
],
checkedAt: 1798012000000, // when the stored reading was made, or null
Expand All @@ -281,8 +281,33 @@ harness, sorted by name:
```

A window's `origin` says how it arrived: `"read"` when a reader fetched it on a
refresh. A harness that reports usage during its runs stores its windows through
`recordHarnessUsage` with an origin of its own.
refresh, `"run"` when a run reported it through `onUsage`. Each window carries
its own `checkedAt` and `ageMs`. The row's `checkedAt` and `ageMs` belong to the
last reading, so a harness that has only run windows and was never read has a
row `checkedAt` of `null`, and each window says how old it is.

**Run windows land on their own.** Pass the slot's `onUsage` through to a
harness that reports usage, as the Claude Code example in
[Choosing the harness](#choosing-the-harness) does, and every window a
run reports is stored for the user the run belongs to. A line sent through the
[session door](#the-session-door) stores its run's windows for the user
talking. For Claude Code, register
[`readClaudeUsage`](/docs/tools/claude-code-sdk#reading-usage-left) as the
`claude-code` reader. It supplies the account and plan, and the windows come
from runs:

```ts
import { readClaudeUsage } from "@flow-state-dev/claude-code/sdk";

const claudeEnv = harnessEnv({ pass: ["PATH", "HOME", "CLAUDE_CODE_OAUTH_TOKEN"] });

harnessUsageAction({
readers: {
codex: (options) => readCodexUsage({ env, ...options }),
"claude-code": (options) => readClaudeUsage({ env: claudeEnv, ...options }),
},
});
```

**Without `refresh`, nothing is read.** The query returns what is stored.
A harness with a reader and no reading yet is listed with `status: "unavailable"`,
Expand All @@ -300,19 +325,27 @@ query runs, by the server's clock. A window whose `resetsAt` has passed reads
`rolledOver: true`; its percent is from before the reset, so show the reset
rather than the number.

**One current reading per user and harness.** A new reading replaces the
whole row, so after a sign-in change no window from the earlier account is left.
A reading older than the stored one is refused. The account is kept only as a
hash, with no email. There is no history.
**One current record per user and harness.** A reader's result and the
windows runs reported sit side by side:

- A reading replaces the windows the last reading left and keeps the run windows it doesn't name. Where both name one window, the newer wins.
- A newer report for a window replaces the older one. An older one that arrives late is ignored, and so is a reading older than the stored one.
- A reading or run window from a different account clears every window and the plan the earlier account left.
- A reading with no number (`api-key`, `signed-out`) clears every window. A run that started before that reading doesn't bring them back.

The account is kept only as a hash, with no email. There is no history.

The readings live in `harnessUsageCollection` (`harness-usage/**`, user scope).
A client can't read the collection directly. Read it through the `harnessUsage`
query. On the server:

- `readHarnessUsage(ctx, now, expected?)` returns the same rows the query does, without refreshing.
- `recordHarnessUsage(ctx, harness, reading, checkedAt)` stores a reading from a harness that reports usage during its own runs. It answers `"recorded"`, `"kept-stored"` (an `unavailable` reading over a stored one, which stays) or `"refused-older"`.
- `recordHarnessUsage(ctx, report)` stores one window a run reported, a `HarnessUsageReport`. The manager's and the session door's `onUsage` call it for you.
- `recordHarnessUsage(ctx, harness, reading, checkedAt)` stores a whole reading made at `checkedAt`.

Both forms answer `"recorded"`, `"kept-stored"` (an `unavailable` reading over a stored one, which stays) or `"refused-older"` (what is stored is newer, or the run started before the stored no-number reading).

Both read the collection through the calling block's resources, under the key `HARNESS_USAGE`, and throw when it isn't installed there: give that block `resources: { [HARNESS_USAGE]: harnessUsageCollection }`. `harnessUsageAction` installs it on its own block.
All of these read the collection through the calling block's resources, under the key `HARNESS_USAGE`, and throw when it isn't installed there: give that block `resources: { [HARNESS_USAGE]: harnessUsageCollection }`. `harnessUsageAction` installs it on its own block.

## Pointing it away from your own code

Expand Down
45 changes: 45 additions & 0 deletions apps/docs/docs/tools/claude-code-sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,12 @@ When the host decides to run the agent (rather than letting a model choose), use
the block directly:

```ts
import { harnessEnv } from "@flow-state-dev/core";
import { claudeCodeAgent } from "@flow-state-dev/claude-code/sdk";

const env = harnessEnv({ pass: ["PATH", "HOME", "CLAUDE_CODE_OAUTH_TOKEN"] });
const agent = claudeCodeAgent({
env,
systemPrompt: {
type: "preset",
preset: "claude_code",
Expand Down Expand Up @@ -786,6 +789,48 @@ Pass the same `env` you give `claudeCodeAgent`, so the probe sees the credential

It never throws for any of these. It checks where a credential would come from, not that it works: a key that is set but rejected still reads as `"account"`, and fails on the run's first turn.

## Reading usage left

A Claude subscription limits use per time window, such as five hours or a week, and a run fails once a window is used up. Claude Code reports where those limits stand while it runs, so the numbers come from runs. Pass `onUsage` and the block hands you each one:

```ts
import { harnessEnv } from "@flow-state-dev/core";
import { claudeCodeAgent } from "@flow-state-dev/claude-code/sdk";

const env = harnessEnv({ pass: ["PATH", "HOME", "CLAUDE_CODE_OAUTH_TOKEN"] });
const agent = claudeCodeAgent({
env,
onUsage: ({ window, observedAt }) => {
console.log(window.id, window.level, window.usedPercent, window.resetsAt && new Date(window.resetsAt));
},
});
```

Claude Code sends an update whenever its limits change, and at least once per run on a subscription. Each update becomes one [usage window](./coding-agents.md#subscription-usage), for the limit that binds at that moment:

| Field | Value |
|---|---|
| `id` | Claude Code's name for the window: `five_hour`, `seven_day`, or a weekly variant such as `seven_day_opus` |
| `minutes` | 300 for `five_hour`, 10 080 for the `seven_day` family, `null` for any other id |
| `level` | `"ok"` (under its warning line), `"warning"` (near the limit) or `"reached"` |
| `usedPercent` | 0 to 100, only when Claude Code sends one, which it does near a limit. Otherwise `null`, never 0 |
| `resetsAt` | When the window resets, in epoch ms |

The report around it also carries `harness` (`"claude-code"`, exported as `CLAUDE_CODE_HARNESS`), `observedAt`, `runStartedAt`, and `accountId`, the signed-in email or `null`. A run on an API key reports nothing. `onUsage` is awaited, but a throw from it doesn't fail the run: a warning is logged and the run carries on. It works on in-session and detached blocks alike. `usageWindowOf(info)` maps one raw event payload to a window, or `null`, if you read the SDK stream yourself.

To show who is signed in before any run, `readClaudeUsage({ env })` is a [usage reader](./coding-agents.md#subscription-usage). It never throws:

| `status` | When |
|---|---|
| `windows` | Signed in, or running on an OAuth token. Carries `plan` and `accountId`, and **no windows**: those arrive from runs. |
| `api-key` | Claude Code runs on an API key, which has no subscription limit. |
| `signed-out` | Nobody is signed in, or Claude Code couldn't be asked or didn't answer within `timeoutMs` (10 seconds by default) or before `signal` aborted. |
| `unavailable` | `@anthropic-ai/claude-agent-sdk` isn't installed. |

Pass the same `env` as your runs. To keep run windows and readings together and serve them to a page or a worker, register `readClaudeUsage` as the `claude-code` reader of the [harness manager's usage query](/docs/orchestration/harness-manager#subscription-usage) and pass `onUsage` through from the manager's harness slot.

**Where the numbers come from.** Only from Claude Code's own run events and its account answer. Nothing here reads a credentials file, calls a usage endpoint, or uses the SDK's experimental usage call.

## Error handling

| Situation | Behavior |
Expand Down
41 changes: 33 additions & 8 deletions apps/docs/docs/tools/coding-agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,14 @@ window is used, without a run. The shapes live in `@flow-state-dev/core`, so a
page or a worker reads every agent's usage the same way:

```ts
import type { HarnessUsageReader, HarnessUsageReading, HarnessUsageWindow } from "@flow-state-dev/core/types";
import type {
HarnessUsageHook,
HarnessUsageLevel,
HarnessUsageReader,
HarnessUsageReading,
HarnessUsageReport,
HarnessUsageWindow,
} from "@flow-state-dev/core/types";
import { harnessUsageWindowSchema } from "@flow-state-dev/core";
```

Expand All @@ -153,15 +160,33 @@ A reading is one of four:
| `unavailable` | `reason`. The read couldn't be made. |

Each window has an `id` unique within the reading, the vendor's `label` for its
limit, its length in `minutes`, `usedPercent` from 0 to 100, and `resetsAt` in
epoch ms. `label`, `minutes` and `resetsAt` are `null` when the vendor didn't
say. The numbers are the vendor's own, unconverted. Only `windows` carries a
number: show the `reason` for the other three, never 0%.
limit, its length in `minutes`, `usedPercent` from 0 to 100, `level`, and
`resetsAt` in epoch ms. `label`, `minutes` and `resetsAt` are `null` when the
vendor didn't say. `level` (`HarnessUsageLevel`) is how close the window is to
its limit, when the vendor says: `"ok"` (under its warning line), `"warning"`
(near the limit) or `"reached"`, and `null` for a vendor that sends only a
percent, as Codex does. `usedPercent` is `null` when the vendor sent no
percent; show the `level` instead. `harnessUsageWindowSchema` reads a window
stored with no `level` as `level: null`. The numbers are the vendor's own: a
fraction is scaled to a percent and nothing else is derived. Only `windows`
carries a number: show the `reason` for the other three, never 0%.

A reader returns those states rather than throwing. Codex ships one,
[`readCodexUsage`](./codex.md#reading-usage-left). To keep readings and serve
them with their age, hand your readers to the
[harness manager's usage query](/docs/orchestration/harness-manager#subscription-usage).
[`readCodexUsage`](./codex.md#reading-usage-left), and Claude Code ships
[`readClaudeUsage`](./claude-code-sdk.md#reading-usage-left).

Some agents report usage only while they run. A harness block that does takes an
`onUsage` hook, a `HarnessUsageHook`: `(report, ctx) => void | Promise<void>`.
Each `HarnessUsageReport` carries one window, plus `harness` (the name a usage
record is kept under), `observedAt` and `runStartedAt` in epoch ms, and the
`accountId` the run is signed in as, or `null`. A run reports the window that
applies to the whole account, not its own share. Claude Code is the agent that
reports this way: its reader names the account and plan, and its windows come
from [`onUsage`](./claude-code-sdk.md#reading-usage-left).

To keep readings and run windows and serve them with their age, hand your readers
to the [harness manager's usage query](/docs/orchestration/harness-manager#subscription-usage)
and let the manager feed your harness its `onUsage`.

## The prompt is the input; everything else is configuration

Expand Down
4 changes: 2 additions & 2 deletions goals/devforce-lab/it-commits-from-the-seats-own-file/run.mts
Original file line number Diff line number Diff line change
Expand Up @@ -149,14 +149,14 @@ await runGoal(async (failures) => {
// Claude Code as the lab's `claudeCodeHarness` sets it up — the same agent
// a served Lab runs under `DEVFORCE_LAB_HARNESS=claude-code` — with the
// prompt-recording tap in front of it.
harness: ({ cwd, resume, onSession }) =>
harness: (feeds) =>
sequencer({
name: "devforce-lab-recorded-harness",
inputSchema: harnessRunInputSchema,
outputSchema: harnessRunHandleSchema,
})
.tap(recordPrompt)
.step(claudeCodeHarness({ cwd, resume, onSession }) as never) as unknown as HarnessBlock,
.step(claudeCodeHarness(feeds) as never) as unknown as HarnessBlock,
});

try {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -387,14 +387,14 @@ await runGoal(async (failures) => {
});
const { lab, sourceRepo } = await open(
"handed-b",
({ cwd, resume, onSession }) =>
(feeds) =>
sequencer({
name: "handed-recorded-harness",
inputSchema: harnessRunInputSchema,
outputSchema: harnessRunHandleSchema,
})
.tap(recordPrompt)
.step(claudeCodeHarness({ cwd, resume, onSession }) as never) as unknown as HarnessBlock,
.step(claudeCodeHarness(feeds) as never) as unknown as HarnessBlock,
fixture.realRun,
{ runTimeoutMs: REAL_RUN_TIMEOUT_MS },
);
Expand Down
Loading
Loading