Skip to content

fix(engine): scope check-interrupted sweep to the caller's tenant (FIX-1569) - #2205

Merged
jhoffner merged 4 commits into
mainfrom
fix/fix-1569
Sep 24, 2026
Merged

jhoffner merged 4 commits into
mainfrom
fix/fix-1569

Conversation

@jhoffner

@jhoffner jhoffner commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

The problem

This is a cross-tenant integrity fix. The recovery check that the DevTool runs on every session-list refresh (POST /api/flows/users/:userId/check-interrupted) looks for in-flight requests that have stopped sending heartbeats, marks them interrupted, and removes them from the active list. It picked which requests to touch by user and organization only. It never looked at the tenant.

So in a multi-tenant deployment where one user id exists under two tenants, a caller on tenant A could:

  • read tenant B's request and session ids for that user, and
  • force tenant B's live requests into interrupted, which stops B's work, not only exposes it.

Retry and continue already refuse a request from another tenant. This route didn't.

What this does

The check now considers only requests in the caller's own tenant, before it reports or writes anything. A request in another tenant is left completely alone: it isn't returned, isn't marked interrupted, and stays registered as running. A caller with no tenant header reaches only requests that have no tenant, which is the rule retry and continue already follow.

What's asked of you

Nothing here is yours to decide. This is a bug on the direct route, so this PR is the review gate.

Parts worth reviewing closely

  • sweepAdmits in packages/engine/src/routes/recovery-routes.ts. The tenant check runs first on every path. That includes the framework-default path, which used to return undefined (only the path userId restricted it). The function now always returns a predicate. Check that no path skips the tenant clause.
  • Tenant threading in packages/engine/src/routes/http-handlers.ts. One line: tenantId goes into the check-interrupted context, in the same place and shape fix(engine): list a seat's sessions and runs to the caller its own resolver accepts (FIX-1566) #2192 uses for the active-requests listing. fix(engine): list a seat's sessions and runs to the caller its own resolver accepts (FIX-1566) #2192 has since merged, and main merged into this branch without conflicts.
  • The write path. detectInterruptedRequests filters the stale entries through ownedBy before it reads any request record or writes anything, so an entry the predicate rejects is never touched. The tests assert this on the store (record still in_progress, entry still registered), not only on the response.

Links: Linear FIX-1569 · epic FIX-1528 · related #2192 (FIX-1566, the same class of leak on the listings) · merges after human review

How to review this — altitude, what's in scope, what's deliberately unsettled

How to review this. This is a bug fix on the direct route: there is no spec and no
spec PR, because the hard part was the diagnosis and it happened in code. This PR is the
only gate
, so unlike a spec-backed change, the approach is in scope. Nothing here was
signed off upstream.

Most valuable here: the diagnosis chain. Does the repro actually reproduce the reported
failure, is the named cause the real one, does the fix address that cause rather than the
symptom, and does the regression test fail without the fix? Then the second path (the legacy
shape, the null boundary, the concurrent case, the cancel path).

In scope to challenge: where the guard was placed, any behaviour change a user could
notice, and whether this should have been a spec'd feature instead of a fix.

Engineering calls — decided along the way, no product sign-off needed
  • The guard is inside sweepAdmits, not in detectInterruptedRequests or route-auth. It sits with the other admission clauses, which scopes the fix to this one route. The periodic and startup sweeps are host-wide by design and should not be tenant-scoped. If this is wrong, a future caller of detectInterruptedRequests has to remember to pass a tenant-aware ownedBy.
  • The entry's tenantId is judged, not the request record's. The entry is what the sweep admits on, and the host writes both from the same dispatch envelope. If they could ever diverge, the record would need its own check. I found no path where they do.
  • Route-auth's user subject is unchanged. It sets no ownerOrgId, so route-auth skips its org/tenant comparison on user-addressed routes, and each handler has to scope its own rows. That's a sharp edge for the next user-addressed route that writes. It isn't reworked here, to keep this minimal. It's worth a follow-up if another user route starts mutating.
  • Docs updated. docs/architecture/dispatched-work.md says that the client-triggered sweep admits only entries in the caller's tenant (tenantMatches), that a caller with no tenant header sweeps only entries with no tenant, and that the startup and periodic sweeps still cover every tenant. The user row in docs/architecture/authentication.md says the guard compares no organization or tenant for user-addressed routes, so each handler scopes its own rows. One sentence was added to the recovery example in both the client API docs page (apps/docs/docs/api/client.md) and packages/client/README.md.
Everything else — verification, files

Verification (BP-003). Run with FSDEV_DEFAULT_MODEL and FSDEV_INTENT_* unset.

  • Red on main. With the three new tests in place and packages/engine/src reverted to origin/main, all three fail:
    • default resolver: expected [ 'acme_req', 'globex_req' ] to not include 'globex_req'
    • host resolver, same user and org: expected [ 'acme_req', 'globex_req' ] to not include 'globex_req'
    • no tenant header: expected [ 'untenanted_req', 'acme_req' ] to deeply equal [ 'untenanted_req' ]
  • Fence-break. Only the tenantMatches line was commented out, with the tenant still threaded. Full engine suite: 3 failed | 2725 passed | 4 skipped. The failures are exactly the three new tests.
  • Green. Full engine suite: 215 files passed; 2728 passed | 4 skipped. After merging main (with fix(engine): list a seat's sessions and runs to the caller its own resolver accepts (FIX-1566) #2192): 216 files passed; 2740 passed | 4 skipped. pnpm --filter @flow-state-dev/engine typecheck exits 0.

What the tests pin (packages/engine/test/tenant-route-isolation.test.ts). Two tenants, the same userId and org, and one stale in-flight request in each. A caller on tenant A runs check-interrupted:

  • B's id is not in the response
  • B's record is still in_progress
  • B's active entry is still registered
  • A's own stale entry is swept (interrupted, deregistered)

This runs twice: once on the framework default resolver (no principal) and once with a host resolver that returns the same user and org for both tenants (the principal branch). A third case covers a caller with no tenant header: it sweeps the row with no tenant and leaves tenant A's row alone.

Files

  • packages/engine/src/routes/recovery-routes.ts: tenant clause first in sweepAdmits, which now always returns a predicate.
  • packages/engine/src/routes/http-handlers.ts: passes tenantId into the check-interrupted context.
  • packages/engine/test/tenant-route-isolation.test.ts: three regression tests.
  • .changeset/check-interrupted-tenant-scope.md: engine patch.
  • docs/architecture/dispatched-work.md, docs/architecture/authentication.md, apps/docs/docs/api/client.md, packages/client/README.md: the tenant boundary.

🤖 Generated with Claude Code

https://claude.ai/code/session_018e4MkWPYX62KRow9TSn75D

…X-1569)

The user-addressed check-interrupted sweep admitted active-request entries by
user and organization only. listStale spans every tenant, so a caller on one
tenant who shares a user id with another tenant saw that tenant's request and
session ids and forced its live requests to `interrupted`.

Thread the request tenant into the handler and require
`tenantMatches(entry.tenantId, ctx.tenantId)` in `sweepAdmits` before any
other clause, on every path, so an entry from another tenant is never
returned, marked interrupted, or deregistered. Same rule retry/continue
already apply to a single record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018e4MkWPYX62KRow9TSn75D
@vercel

vercel Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
flow-state-dev-docs Ignored Ignored Sep 24, 2026 10:07pm UTC
flow-state-dev-kitchen-sink Ignored Ignored Sep 24, 2026 10:07pm UTC
flow-state-dev-ui Ignored Ignored Preview Sep 24, 2026 10:07pm UTC

Request Review

@linear-code

linear-code Bot commented Sep 24, 2026

Copy link
Copy Markdown

FIX-1569

@cursor

cursor Bot commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_fb98318d-9d66-47bb-8bbc-e2bd44910f12)

@jhoffner
jhoffner marked this pull request as ready for review September 24, 2026 21:56

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 538cd1ab1a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-24T22:00:56.621948Z 538cd1a Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

- dispatched-work.md: the client-triggered sweep admits only entries whose
  tenant matches the request's (`tenantMatches`); a caller with no tenant
  header sweeps only untenanted entries; the startup and periodic sweeps
  pass no caller filter and still cover every tenant.
- authentication.md: the `user` subject row says the guard compares no
  organization or tenant for user-addressed routes and each handler scopes
  its own rows.
- apps/docs client API page and client README: the recovery sweep only
  touches the caller's tenant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018e4MkWPYX62KRow9TSn75D
…X-1569)

docs-editor: "tenant header" was used nowhere else in the docs; say the
tenant id and its default header, matching the tenancy pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018e4MkWPYX62KRow9TSn75D
@jhoffner
jhoffner merged commit 0f51b98 into main Sep 24, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants