Skip to content

fix: harden rotation, persistence, and SSE failure handling (stress audit) - #617

Merged
ndycode merged 1 commit into
mainfrom
fix/stress-audit-hardening
Jun 19, 2026
Merged

ndycode merged 1 commit into
mainfrom
fix/stress-audit-hardening

Conversation

@ndycode

@ndycode ndycode commented Jun 19, 2026 •

Copy link
Copy Markdown
Owner

Fixes 7 silent-failure defects surfaced by a deep adversarial stress audit of main. Every fix ships with a regression test that fails on the old behavior and passes on the new.

Defects fixed

# Severity Summary
H1 high Unclamped retry-after could wedge an account unavailable for ~31 years; clamp to 7d (MAX_RATE_LIMIT_DELAY_MS), centralized in markRateLimitedWithReason.
H2 high Refresh lease release() unlinked unconditionally — a slow owner deleted a stolen lock. Added a per-owner nonce; release only unlinks when the on-disk nonce matches.
H3 high Routine saveToDisk clobbered a single-use refresh token another process had just rotated. Reconcile token material from disk (adopt strictly-newer expiresAt) before persisting.
H4 high A transient 30–60s 429 benched an account for the full 2h cap by folding the benign weekly window into the wait. Only exhausted windows count now.
H5 high getLiveQuotaWaitMs took a blind max of both quota windows, overstating wait ~6700× and inverting the recommendation. Applies the same exhausted-window filter the cache path uses (429 still honors all windows).
H6/H7 high Mid-stream error, and the unhandled response.failed/response.incomplete terminal events, returned raw SSE at HTTP 200 → recorded as account success and skipped rotation. Now synthesized to non-2xx; empty/no-error streams still pass through so the empty-response retry path is preserved.
M1 medium SSE parser required data: with a trailing space; now accepts data:.

Verification

  • Full suite: 5010 passed, 3 skipped, 0 failed (npm test).
  • tsc --noEmit, eslint, knip, and build all clean.
  • New/updated regression tests: forecast, accounts, preemptive-quota-scheduler, refresh-lease, response-handler, chaos/fault-injection, and a new h3-token-clobber test.
  • 4 existing tests that asserted the buggy behavior (SSE error → raw 200) were updated to assert the corrected failure routing.

Behavior-preserving where it matters: the 429 deferral still honors genuine rate-limit windows, commitRefreshedAuth still wins with its freshest token, and the empty-response retry path is untouched.

🤖 Generated with Claude Code

note: greptile review for oc-chatgpt-multi-auth. cite files like lib/foo.ts:123. confirm regression tests + windows concurrency/token redaction coverage.

Greptile Summary

this pr addresses 7 adversarially-surfaced silent-failure defects spanning rate-limit clamping, cross-process token clobber protection, refresh lease ownership, quota window filtering, and sse failure routing — each shipped with a targeted regression test.

  • H1/H3/H4/H5: clamps retry-after to 7d, reconciles rotated tokens before every disk persist, limits 429 deferral to genuinely exhausted windows, and filters getLiveQuotaWaitMs to only exhausted windows under non-429 conditions
  • H2: adds per-owner nonce to lock files so a slow lease holder cannot delete a lock stolen by a new owner after expiry
  • H6/H7/M1: synthesizes non-2xx responses for mid-stream error, response.failed, and response.incomplete events (which previously passed as HTTP 200), and accepts data: with no trailing space per the SSE spec

Confidence Score: 4/5

the core fixes are sound and well-tested for the primary account shapes; the two edge cases flagged (legacy RT-only accounts in H3, windows EACCES in ownsLock) are low-frequency paths that don't affect the rotation proxy's normal operation

all seven defects are addressed with clear logic and matching regression tests; the H3 reconcile silently skips refresh-token-only legacy accounts (no email/accountId), meaning the clobber fix doesn't protect that account shape, and the ownsLock function treats unreadable-file as 'i own it' which on windows with EACCES could trigger spurious safeUnlink retries — neither is a blocking regression on normal accounts but they leave residual gaps

lib/accounts.ts reconcileTokensFromDisk and lib/refresh-lease.ts ownsLock deserve a second look for the edge cases described in the inline comments

Important Files Changed

Filename Overview
lib/accounts.ts adds MAX_RATE_LIMIT_DELAY_MS clamp (H1) and reconcileTokensFromDisk (H3); H3 protection works for accounts with accountId/email but silently bypasses legacy refresh-token-only accounts since identity keys diverge after rotation
lib/refresh-lease.ts adds per-owner nonce to prevent slow-owner lock deletion (H2); ownsLock returns true when lock is unreadable which is safe for absent files but ambiguous on Windows EACCES/EPERM
lib/request/response-handler.ts SSE error/failed/incomplete → non-2xx (H6/H7); data: no-space fix (M1); response.incomplete is intentionally mapped to 502, discarding partial output
lib/forecast.ts getLiveQuotaWaitMs now filters to only exhausted windows under non-429 pressure (H5); consistent with cache path behavior for undefined usedPercent
lib/preemptive-quota-scheduler.ts 429 markRateLimited no longer folds benign quota windows into the deferral (H4); isRateLimitWindow helper correctly treats missing usedPercent as rate-limit conservative
test/h3-token-clobber.test.ts new regression test for H3 cross-process clobber; covers the accountId+email identity path but not the legacy refresh-token-only fallback path
test/refresh-lease.test.ts adds H2 regression test: slow owner A releases after B steals the lock, verifies B's nonce survives; solid coverage
test/response-handler.test.ts H6/H7/M1 regression tests are thorough; updated existing tests to assert corrected non-2xx behavior; control test verifies clean streams still pass

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[SSE Stream HTTP 200] --> B{parseSseStream}
    B -->|data: or data:value M1| C[parse JSON event]
    C --> D{event type}
    D -->|error| E[encounteredError=true]
    D -->|response.failed / response.incomplete H7| E
    D -->|response.done / response.completed| F[finalResponse captured]
    D -->|other delta events| G[accumulate state]
    E --> H{upstream status >= 400?}
    H -->|yes| I[return upstream status]
    H -->|no| J[return 502 Bad Gateway]
    F --> K[return 200 JSON]
    B -->|no terminal event, no error| L[return original SSE - preserve empty-retry path]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[SSE Stream HTTP 200] --> B{parseSseStream}
    B -->|data: or data:value M1| C[parse JSON event]
    C --> D{event type}
    D -->|error| E[encounteredError=true]
    D -->|response.failed / response.incomplete H7| E
    D -->|response.done / response.completed| F[finalResponse captured]
    D -->|other delta events| G[accumulate state]
    E --> H{upstream status >= 400?}
    H -->|yes| I[return upstream status]
    H -->|no| J[return 502 Bad Gateway]
    F --> K[return 200 JSON]
    B -->|no terminal event, no error| L[return original SSE - preserve empty-retry path]
Loading

Fix All in Codex

Prompt To Fix All With AI
Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
lib/accounts.ts:1300-1307
**H3 protection silently fails for refresh-token-only (legacy) accounts**

`getAccountIdentityKey` uses `accountId` → `email` → `refresh:hash(RT)` in priority order. when an account has no `accountId` and no `email`, the identity key is derived from the refresh token hash. after process 2 rotates that account's token, the in-memory snapshot holds the old RT → `refresh:hash(RT_old)`, while the disk copy has the new RT → `refresh:hash(RT_new)`. these keys never match, so the loop at line 1306 skips such accounts entirely — H3 clobber protection doesn't apply to them. modern accounts should always carry email, but legacy accounts without it would silently regress to the old behavior. `test/h3-token-clobber.test.ts` only exercises the `accountId`+`email` path, leaving this gap uncovered.

### Issue 2 of 3
lib/refresh-lease.ts:328-344
**ownsLock returns `true` when the lock file is unreadable — unsafe on Windows**

`readJson` silently catches all errors and returns `null` (line 117). on Windows, an `EACCES`/`EPERM` error (another process holds the file open) also produces `null`. `parseLeasePayload(null)` returns `null`, and `ownsLock` then returns `true` ("allow cleanup"). the slow owner then proceeds to call `safeUnlink`, which will retry up to 4 times. while `safeUnlink` will ultimately fail for a live locked file, those retries add latency and the TOCTOU window between the failed read and the unlink attempt means the guard may incorrectly attempt to evict a live lock. the repo convention for Windows EBUSY/EPERM paths is explicit retry-with-backoff, not silent failure→allow. worth at minimum logging a distinct warning when read fails vs. when the file is absent.

### Issue 3 of 3
lib/request/response-handler.ts:604-608
**`response.incomplete` yields 502 and discards all partial output**

`response.incomplete` (e.g. `max_output_tokens` hit) sets `encounteredError = true`, causing `parseSseStream` to early-exit with `finalResponse: null`. any partial text streamed before the incomplete event is discarded; the caller receives a 502. clients that handle truncated output gracefully (e.g. show partial code, trigger a continuation turn) lose that ability — they see only a 502 identical to a hard upstream failure. this is intentional per the PR description, but worth confirming: should `response.incomplete` really route identically to `response.failed`, or does the caller need a way to distinguish the two so partial output can at least be attempted?

Reviews (1): Last reviewed commit: "fix: harden rotation, persistence, and S..." | Re-trigger Greptile

Greptile also left 3 inline comments on this PR.

…udit)

Fixes 7 silent-failure defects found by a deep stress audit of main, each
with a regression test.

H1 — Unclamped retry-after wedges an account for years
  parseRetryAfterHeaderMs returned the raw header with no upper bound and the
  runtime-proxy 429 path passed it straight to the setter (only the legacy
  fetch path clamped). A bogus retry-after-ms (~31y) locked an account out,
  persisted, with no self-heal. Clamp centrally in markRateLimitedWithReason
  to MAX_RATE_LIMIT_DELAY_MS (7d), now shared via constants.ts.

H2 — Refresh lease release() had no ownership check
  A slow owner whose lease expired and was stolen would unlink the new owner's
  lock on release, collapsing cross-process mutual exclusion. Embed a per-owner
  nonce; release() only unlinks when the on-disk nonce still matches.

H3 — Cross-process save clobbered a freshly-rotated refresh token
  saveToDisk discarded the disk state and rewrote the whole in-memory pool,
  reverting a single-use refresh token another process had just rotated.
  Reconcile token material from disk (adopt strictly-newer expiresAt) before
  persisting.

H4 — Transient 429 benched an account for the full 2h cap
  markRateLimited folded the benign weekly secondary reset (~7d) into the
  primary window. Only treat genuinely-exhausted windows (not a healthy quota
  gauge) as rate-limit windows in markRateLimited and getDeferral.

H5 — Forecast over-stated wait and inverted the recommendation
  getLiveQuotaWaitMs took a blind max of both windows. Apply the same
  exhausted-window filter the quota-cache path already uses (429 still honors
  all windows).

H6/H7 — SSE failures returned as HTTP 200 success
  A mid-stream error, or the unhandled response.failed/response.incomplete
  terminal events, returned raw SSE at status 200 -> recorded as an account
  success and skipped rotation/retry. Treat them as terminal failures and
  synthesize a non-2xx; a no-error/empty stream still passes through so the
  empty-response retry path is preserved.

M1 — SSE parser required "data: " with a trailing space; accept "data:".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@coderabbitai

coderabbitai Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@ndycode, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 6 minutes and 14 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6fb67b0e-4db6-4ed7-bdd2-b76c992fa21f

📥 Commits

Reviewing files that changed from the base of the PR and between 07bab1c and 0b1218d.

📒 Files selected for processing (14)
  • lib/accounts.ts
  • lib/constants.ts
  • lib/forecast.ts
  • lib/preemptive-quota-scheduler.ts
  • lib/refresh-lease.ts
  • lib/request/fetch-helpers.ts
  • lib/request/response-handler.ts
  • test/accounts.test.ts
  • test/chaos/fault-injection.test.ts
  • test/forecast.test.ts
  • test/h3-token-clobber.test.ts
  • test/preemptive-quota-scheduler.test.ts
  • test/refresh-lease.test.ts
  • test/response-handler.test.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/stress-audit-hardening
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch fix/stress-audit-hardening

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@ndycode
ndycode merged commit 7d40645 into main Jun 19, 2026
1 of 2 checks passed
@ndycode
ndycode deleted the fix/stress-audit-hardening branch June 19, 2026 09:03
Comment thread lib/accounts.ts
Comment on lines +1300 to +1307
const key = getAccountIdentityKey({
accountId: account.accountId,
email: account.email,
refreshToken: account.refreshToken,
});
if (!key) continue;
const disk = diskByIdentity.get(key);
if (!disk) continue;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 H3 protection silently fails for refresh-token-only (legacy) accounts

getAccountIdentityKey uses accountId → email → refresh:hash(RT) in priority order. when an account has no accountId and no email, the identity key is derived from the refresh token hash. after process 2 rotates that account's token, the in-memory snapshot holds the old RT → refresh:hash(RT_old), while the disk copy has the new RT → refresh:hash(RT_new). these keys never match, so the loop at line 1306 skips such accounts entirely — H3 clobber protection doesn't apply to them. modern accounts should always carry email, but legacy accounts without it would silently regress to the old behavior. test/h3-token-clobber.test.ts only exercises the accountId+email path, leaving this gap uncovered.

Prompt To Fix With AI
This is a comment left during a code review.
Path: lib/accounts.ts
Line: 1300-1307

Comment:
**H3 protection silently fails for refresh-token-only (legacy) accounts**

`getAccountIdentityKey` uses `accountId` → `email` → `refresh:hash(RT)` in priority order. when an account has no `accountId` and no `email`, the identity key is derived from the refresh token hash. after process 2 rotates that account's token, the in-memory snapshot holds the old RT → `refresh:hash(RT_old)`, while the disk copy has the new RT → `refresh:hash(RT_new)`. these keys never match, so the loop at line 1306 skips such accounts entirely — H3 clobber protection doesn't apply to them. modern accounts should always carry email, but legacy accounts without it would silently regress to the old behavior. `test/h3-token-clobber.test.ts` only exercises the `accountId`+`email` path, leaving this gap uncovered.

How can I resolve this? If you propose a fix, please make it concise.

Fix in Codex

Comment thread lib/refresh-lease.ts
Comment on lines +328 to +344
{ lockPath },
);
}
}
},
};
}

private async ownsLock(
lockPath: string,
tokenHash: string,
nonce: string,
): Promise<boolean> {
const raw = await readJson(lockPath, this.fsOps);
const parsed = raw === null ? null : parseLeasePayload(raw);
if (!parsed) {
// Unreadable/absent lock: nothing of ours to protect, allow cleanup.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 ownsLock returns true when the lock file is unreadable — unsafe on Windows

readJson silently catches all errors and returns null (line 117). on Windows, an EACCES/EPERM error (another process holds the file open) also produces null. parseLeasePayload(null) returns null, and ownsLock then returns true ("allow cleanup"). the slow owner then proceeds to call safeUnlink, which will retry up to 4 times. while safeUnlink will ultimately fail for a live locked file, those retries add latency and the TOCTOU window between the failed read and the unlink attempt means the guard may incorrectly attempt to evict a live lock. the repo convention for Windows EBUSY/EPERM paths is explicit retry-with-backoff, not silent failure→allow. worth at minimum logging a distinct warning when read fails vs. when the file is absent.

Prompt To Fix With AI
This is a comment left during a code review.
Path: lib/refresh-lease.ts
Line: 328-344

Comment:
**ownsLock returns `true` when the lock file is unreadable — unsafe on Windows**

`readJson` silently catches all errors and returns `null` (line 117). on Windows, an `EACCES`/`EPERM` error (another process holds the file open) also produces `null`. `parseLeasePayload(null)` returns `null`, and `ownsLock` then returns `true` ("allow cleanup"). the slow owner then proceeds to call `safeUnlink`, which will retry up to 4 times. while `safeUnlink` will ultimately fail for a live locked file, those retries add latency and the TOCTOU window between the failed read and the unlink attempt means the guard may incorrectly attempt to evict a live lock. the repo convention for Windows EBUSY/EPERM paths is explicit retry-with-backoff, not silent failure→allow. worth at minimum logging a distinct warning when read fails vs. when the file is absent.

How can I resolve this? If you propose a fix, please make it concise.

Fix in Codex

Comment on lines +604 to +608
if (data.type === "response.failed" || data.type === "response.incomplete") {
log.warn("SSE terminal failure event received", { type: data.type });
state.encounteredError = true;
return;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 response.incomplete yields 502 and discards all partial output

response.incomplete (e.g. max_output_tokens hit) sets encounteredError = true, causing parseSseStream to early-exit with finalResponse: null. any partial text streamed before the incomplete event is discarded; the caller receives a 502. clients that handle truncated output gracefully (e.g. show partial code, trigger a continuation turn) lose that ability — they see only a 502 identical to a hard upstream failure. this is intentional per the PR description, but worth confirming: should response.incomplete really route identically to response.failed, or does the caller need a way to distinguish the two so partial output can at least be attempted?

is response.incomplete (max_output_tokens) intentionally mapped to the same 502 path as response.failed? clients that do graceful truncation handling would need a distinguishable status or body field to avoid treating a partial response as a hard error.

Prompt To Fix With AI
This is a comment left during a code review.
Path: lib/request/response-handler.ts
Line: 604-608

Comment:
**`response.incomplete` yields 502 and discards all partial output**

`response.incomplete` (e.g. `max_output_tokens` hit) sets `encounteredError = true`, causing `parseSseStream` to early-exit with `finalResponse: null`. any partial text streamed before the incomplete event is discarded; the caller receives a 502. clients that handle truncated output gracefully (e.g. show partial code, trigger a continuation turn) lose that ability — they see only a 502 identical to a hard upstream failure. this is intentional per the PR description, but worth confirming: should `response.incomplete` really route identically to `response.failed`, or does the caller need a way to distinguish the two so partial output can at least be attempted?

 is `response.incomplete` (max_output_tokens) intentionally mapped to the same 502 path as `response.failed`? clients that do graceful truncation handling would need a distinguishable status or body field to avoid treating a partial response as a hard error.

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Codex

ndycode added a commit that referenced this pull request Jun 19, 2026
…rce) (#619)

Follow-up to #617/#618, clearing the medium/low/info backlog from the deep
stress audit. Each fix ships with a regression test.

I1 — OAuth expires_in / expires accepted any number. A zero/negative value
minted an already-expired token, causing a tight refresh loop that burns the
single-use refresh token. Require z.number().int().positive() at the schema
boundary so a bogus value fails validation instead.

I2 — The free-text log scrubber (last line of defense) masked JWT/hex/sk-/
Bearer but not this app's own local bearer tokens (cma_local_<base64url>).
Added the pattern.

M3 — V1->V3 storage migration discarded migrateV1ToV3's per-account output:
normalizeAccountStorage rebuilt the account list from the raw V1 objects, so
the scalar rateLimitResetTime -> map rateLimitResetTimes conversion was lost
and a rate-limited account looked immediately available on upgrade (429 burst).
Build validAccounts from baseStorage.accounts.

L3 — The local-client-token store wrote temp + rename with no fsync, so a
crash/power-loss after rename could truncate it. fsync the temp file before
rename, mirroring the durable-write pattern in runtime/app-bind.ts.

H1-at-source — getQuotaNearExhaustionWaitMs returned an unclamped wait derived
from upstream reset headers (safe only because callers clamp). Clamp at source
to MAX_RATE_LIMIT_DELAY_MS as defense in depth.

Not changed: M2 (resolvePath sibling rejection) is a fail-closed security
boundary; M4/L1/L2/L4/L5 are minor perf/liveness items left as-is to avoid
churning hot paths on low-severity, lower-confidence findings.

Co-authored-by: ndycode <ndycode@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ndycode added a commit that referenced this pull request Jun 19, 2026
Bumps version to 2.3.3 and adds release notes for the security/durability
hardening landed in #617, #618, and #619 (rotation, persistence, and SSE
failure-handling fixes from a deep stress audit). No feature changes; routing,
account-selection, and the normal auth flow are unchanged.

- package.json / package-lock.json / .codex-plugin/plugin.json: 2.3.2 -> 2.3.3
- CHANGELOG.md: [2.3.3] section
- docs/releases/v2.3.3.md: full release notes
- AGENTS.md: package-version claim -> 2.3.3
- docs/README.md, README.md: v2.3.3 current stable, v2.3.2 demoted to prior

Co-authored-by: ndycode <ndycode@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant