Repository navigation
fix: harden rotation, persistence, and SSE failure handling (stress audit) - #617
Conversation
…udit) Fixes 7 silent-failure defects found by a deep stress audit of main, each with a regression test. H1 — Unclamped retry-after wedges an account for years parseRetryAfterHeaderMs returned the raw header with no upper bound and the runtime-proxy 429 path passed it straight to the setter (only the legacy fetch path clamped). A bogus retry-after-ms (~31y) locked an account out, persisted, with no self-heal. Clamp centrally in markRateLimitedWithReason to MAX_RATE_LIMIT_DELAY_MS (7d), now shared via constants.ts. H2 — Refresh lease release() had no ownership check A slow owner whose lease expired and was stolen would unlink the new owner's lock on release, collapsing cross-process mutual exclusion. Embed a per-owner nonce; release() only unlinks when the on-disk nonce still matches. H3 — Cross-process save clobbered a freshly-rotated refresh token saveToDisk discarded the disk state and rewrote the whole in-memory pool, reverting a single-use refresh token another process had just rotated. Reconcile token material from disk (adopt strictly-newer expiresAt) before persisting. H4 — Transient 429 benched an account for the full 2h cap markRateLimited folded the benign weekly secondary reset (~7d) into the primary window. Only treat genuinely-exhausted windows (not a healthy quota gauge) as rate-limit windows in markRateLimited and getDeferral. H5 — Forecast over-stated wait and inverted the recommendation getLiveQuotaWaitMs took a blind max of both windows. Apply the same exhausted-window filter the quota-cache path already uses (429 still honors all windows). H6/H7 — SSE failures returned as HTTP 200 success A mid-stream error, or the unhandled response.failed/response.incomplete terminal events, returned raw SSE at status 200 -> recorded as an account success and skipped rotation/retry. Treat them as terminal failures and synthesize a non-2xx; a no-error/empty stream still passes through so the empty-response retry path is preserved. M1 — SSE parser required "data: " with a trailing space; accept "data:". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
Warning Review limit reached
More reviews will be available in 6 minutes and 14 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (14)
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
| const key = getAccountIdentityKey({ | ||
| accountId: account.accountId, | ||
| email: account.email, | ||
| refreshToken: account.refreshToken, | ||
| }); | ||
| if (!key) continue; | ||
| const disk = diskByIdentity.get(key); | ||
| if (!disk) continue; |
There was a problem hiding this comment.
H3 protection silently fails for refresh-token-only (legacy) accounts
getAccountIdentityKey uses accountId → email → refresh:hash(RT) in priority order. when an account has no accountId and no email, the identity key is derived from the refresh token hash. after process 2 rotates that account's token, the in-memory snapshot holds the old RT → refresh:hash(RT_old), while the disk copy has the new RT → refresh:hash(RT_new). these keys never match, so the loop at line 1306 skips such accounts entirely — H3 clobber protection doesn't apply to them. modern accounts should always carry email, but legacy accounts without it would silently regress to the old behavior. test/h3-token-clobber.test.ts only exercises the accountId+email path, leaving this gap uncovered.
Prompt To Fix With AI
This is a comment left during a code review.
Path: lib/accounts.ts
Line: 1300-1307
Comment:
**H3 protection silently fails for refresh-token-only (legacy) accounts**
`getAccountIdentityKey` uses `accountId` → `email` → `refresh:hash(RT)` in priority order. when an account has no `accountId` and no `email`, the identity key is derived from the refresh token hash. after process 2 rotates that account's token, the in-memory snapshot holds the old RT → `refresh:hash(RT_old)`, while the disk copy has the new RT → `refresh:hash(RT_new)`. these keys never match, so the loop at line 1306 skips such accounts entirely — H3 clobber protection doesn't apply to them. modern accounts should always carry email, but legacy accounts without it would silently regress to the old behavior. `test/h3-token-clobber.test.ts` only exercises the `accountId`+`email` path, leaving this gap uncovered.
How can I resolve this? If you propose a fix, please make it concise.| { lockPath }, | ||
| ); | ||
| } | ||
| } | ||
| }, | ||
| }; | ||
| } | ||
|
|
||
| private async ownsLock( | ||
| lockPath: string, | ||
| tokenHash: string, | ||
| nonce: string, | ||
| ): Promise<boolean> { | ||
| const raw = await readJson(lockPath, this.fsOps); | ||
| const parsed = raw === null ? null : parseLeasePayload(raw); | ||
| if (!parsed) { | ||
| // Unreadable/absent lock: nothing of ours to protect, allow cleanup. |
There was a problem hiding this comment.
ownsLock returns
true when the lock file is unreadable — unsafe on Windows
readJson silently catches all errors and returns null (line 117). on Windows, an EACCES/EPERM error (another process holds the file open) also produces null. parseLeasePayload(null) returns null, and ownsLock then returns true ("allow cleanup"). the slow owner then proceeds to call safeUnlink, which will retry up to 4 times. while safeUnlink will ultimately fail for a live locked file, those retries add latency and the TOCTOU window between the failed read and the unlink attempt means the guard may incorrectly attempt to evict a live lock. the repo convention for Windows EBUSY/EPERM paths is explicit retry-with-backoff, not silent failure→allow. worth at minimum logging a distinct warning when read fails vs. when the file is absent.
Prompt To Fix With AI
This is a comment left during a code review.
Path: lib/refresh-lease.ts
Line: 328-344
Comment:
**ownsLock returns `true` when the lock file is unreadable — unsafe on Windows**
`readJson` silently catches all errors and returns `null` (line 117). on Windows, an `EACCES`/`EPERM` error (another process holds the file open) also produces `null`. `parseLeasePayload(null)` returns `null`, and `ownsLock` then returns `true` ("allow cleanup"). the slow owner then proceeds to call `safeUnlink`, which will retry up to 4 times. while `safeUnlink` will ultimately fail for a live locked file, those retries add latency and the TOCTOU window between the failed read and the unlink attempt means the guard may incorrectly attempt to evict a live lock. the repo convention for Windows EBUSY/EPERM paths is explicit retry-with-backoff, not silent failure→allow. worth at minimum logging a distinct warning when read fails vs. when the file is absent.
How can I resolve this? If you propose a fix, please make it concise.| if (data.type === "response.failed" || data.type === "response.incomplete") { | ||
| log.warn("SSE terminal failure event received", { type: data.type }); | ||
| state.encounteredError = true; | ||
| return; | ||
| } |
There was a problem hiding this comment.
response.incomplete yields 502 and discards all partial output
response.incomplete (e.g. max_output_tokens hit) sets encounteredError = true, causing parseSseStream to early-exit with finalResponse: null. any partial text streamed before the incomplete event is discarded; the caller receives a 502. clients that handle truncated output gracefully (e.g. show partial code, trigger a continuation turn) lose that ability — they see only a 502 identical to a hard upstream failure. this is intentional per the PR description, but worth confirming: should response.incomplete really route identically to response.failed, or does the caller need a way to distinguish the two so partial output can at least be attempted?
is response.incomplete (max_output_tokens) intentionally mapped to the same 502 path as response.failed? clients that do graceful truncation handling would need a distinguishable status or body field to avoid treating a partial response as a hard error.
Prompt To Fix With AI
This is a comment left during a code review.
Path: lib/request/response-handler.ts
Line: 604-608
Comment:
**`response.incomplete` yields 502 and discards all partial output**
`response.incomplete` (e.g. `max_output_tokens` hit) sets `encounteredError = true`, causing `parseSseStream` to early-exit with `finalResponse: null`. any partial text streamed before the incomplete event is discarded; the caller receives a 502. clients that handle truncated output gracefully (e.g. show partial code, trigger a continuation turn) lose that ability — they see only a 502 identical to a hard upstream failure. this is intentional per the PR description, but worth confirming: should `response.incomplete` really route identically to `response.failed`, or does the caller need a way to distinguish the two so partial output can at least be attempted?
is `response.incomplete` (max_output_tokens) intentionally mapped to the same 502 path as `response.failed`? clients that do graceful truncation handling would need a distinguishable status or body field to avoid treating a partial response as a hard error.
How can I resolve this? If you propose a fix, please make it concise.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
…rce) (#619) Follow-up to #617/#618, clearing the medium/low/info backlog from the deep stress audit. Each fix ships with a regression test. I1 — OAuth expires_in / expires accepted any number. A zero/negative value minted an already-expired token, causing a tight refresh loop that burns the single-use refresh token. Require z.number().int().positive() at the schema boundary so a bogus value fails validation instead. I2 — The free-text log scrubber (last line of defense) masked JWT/hex/sk-/ Bearer but not this app's own local bearer tokens (cma_local_<base64url>). Added the pattern. M3 — V1->V3 storage migration discarded migrateV1ToV3's per-account output: normalizeAccountStorage rebuilt the account list from the raw V1 objects, so the scalar rateLimitResetTime -> map rateLimitResetTimes conversion was lost and a rate-limited account looked immediately available on upgrade (429 burst). Build validAccounts from baseStorage.accounts. L3 — The local-client-token store wrote temp + rename with no fsync, so a crash/power-loss after rename could truncate it. fsync the temp file before rename, mirroring the durable-write pattern in runtime/app-bind.ts. H1-at-source — getQuotaNearExhaustionWaitMs returned an unclamped wait derived from upstream reset headers (safe only because callers clamp). Clamp at source to MAX_RATE_LIMIT_DELAY_MS as defense in depth. Not changed: M2 (resolvePath sibling rejection) is a fail-closed security boundary; M4/L1/L2/L4/L5 are minor perf/liveness items left as-is to avoid churning hot paths on low-severity, lower-confidence findings. Co-authored-by: ndycode <ndycode@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps version to 2.3.3 and adds release notes for the security/durability hardening landed in #617, #618, and #619 (rotation, persistence, and SSE failure-handling fixes from a deep stress audit). No feature changes; routing, account-selection, and the normal auth flow are unchanged. - package.json / package-lock.json / .codex-plugin/plugin.json: 2.3.2 -> 2.3.3 - CHANGELOG.md: [2.3.3] section - docs/releases/v2.3.3.md: full release notes - AGENTS.md: package-version claim -> 2.3.3 - docs/README.md, README.md: v2.3.3 current stable, v2.3.2 demoted to prior Co-authored-by: ndycode <ndycode@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes 7 silent-failure defects surfaced by a deep adversarial stress audit of
main. Every fix ships with a regression test that fails on the old behavior and passes on the new.Defects fixed
retry-aftercould wedge an account unavailable for ~31 years; clamp to 7d (MAX_RATE_LIMIT_DELAY_MS), centralized inmarkRateLimitedWithReason.release()unlinked unconditionally — a slow owner deleted a stolen lock. Added a per-owner nonce; release only unlinks when the on-disk nonce matches.saveToDiskclobbered a single-use refresh token another process had just rotated. Reconcile token material from disk (adopt strictly-newerexpiresAt) before persisting.getLiveQuotaWaitMstook a blindmaxof both quota windows, overstating wait ~6700× and inverting the recommendation. Applies the same exhausted-window filter the cache path uses (429 still honors all windows).error, and the unhandledresponse.failed/response.incompleteterminal events, returned raw SSE at HTTP 200 → recorded as account success and skipped rotation. Now synthesized to non-2xx; empty/no-error streams still pass through so the empty-response retry path is preserved.data:with a trailing space; now acceptsdata:.Verification
npm test).tsc --noEmit,eslint,knip, andbuildall clean.forecast,accounts,preemptive-quota-scheduler,refresh-lease,response-handler,chaos/fault-injection, and a newh3-token-clobbertest.Behavior-preserving where it matters: the 429 deferral still honors genuine rate-limit windows,
commitRefreshedAuthstill wins with its freshest token, and the empty-response retry path is untouched.🤖 Generated with Claude Code
note: greptile review for oc-chatgpt-multi-auth. cite files like
lib/foo.ts:123. confirm regression tests + windows concurrency/token redaction coverage.Greptile Summary
this pr addresses 7 adversarially-surfaced silent-failure defects spanning rate-limit clamping, cross-process token clobber protection, refresh lease ownership, quota window filtering, and sse failure routing — each shipped with a targeted regression test.
retry-afterto 7d, reconciles rotated tokens before every disk persist, limits 429 deferral to genuinely exhausted windows, and filtersgetLiveQuotaWaitMsto only exhausted windows under non-429 conditionserror,response.failed, andresponse.incompleteevents (which previously passed as HTTP 200), and acceptsdata:with no trailing space per the SSE specConfidence Score: 4/5
the core fixes are sound and well-tested for the primary account shapes; the two edge cases flagged (legacy RT-only accounts in H3, windows EACCES in ownsLock) are low-frequency paths that don't affect the rotation proxy's normal operation
all seven defects are addressed with clear logic and matching regression tests; the H3 reconcile silently skips refresh-token-only legacy accounts (no email/accountId), meaning the clobber fix doesn't protect that account shape, and the ownsLock function treats unreadable-file as 'i own it' which on windows with EACCES could trigger spurious safeUnlink retries — neither is a blocking regression on normal accounts but they leave residual gaps
lib/accounts.ts reconcileTokensFromDisk and lib/refresh-lease.ts ownsLock deserve a second look for the edge cases described in the inline comments
Important Files Changed
Flowchart
%%{init: {'theme': 'neutral'}}%% flowchart TD A[SSE Stream HTTP 200] --> B{parseSseStream} B -->|data: or data:value M1| C[parse JSON event] C --> D{event type} D -->|error| E[encounteredError=true] D -->|response.failed / response.incomplete H7| E D -->|response.done / response.completed| F[finalResponse captured] D -->|other delta events| G[accumulate state] E --> H{upstream status >= 400?} H -->|yes| I[return upstream status] H -->|no| J[return 502 Bad Gateway] F --> K[return 200 JSON] B -->|no terminal event, no error| L[return original SSE - preserve empty-retry path]%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% flowchart TD A[SSE Stream HTTP 200] --> B{parseSseStream} B -->|data: or data:value M1| C[parse JSON event] C --> D{event type} D -->|error| E[encounteredError=true] D -->|response.failed / response.incomplete H7| E D -->|response.done / response.completed| F[finalResponse captured] D -->|other delta events| G[accumulate state] E --> H{upstream status >= 400?} H -->|yes| I[return upstream status] H -->|no| J[return 502 Bad Gateway] F --> K[return 200 JSON] B -->|no terminal event, no error| L[return original SSE - preserve empty-retry path]Prompt To Fix All With AI
Reviews (1): Last reviewed commit: "fix: harden rotation, persistence, and S..." | Re-trigger Greptile