Skip to content

fix(bullmq): queue workers honour hold and defer (FIX-1836) - #2990

Merged
jhoffner merged 3 commits into
mainfrom
fix/FIX-1836-bullmq-hold-defer
Oct 10, 2026
Merged

jhoffner merged 3 commits into
mainfrom
fix/FIX-1836-bullmq-hold-defer

Conversation

@jhoffner

@jhoffner jhoffner commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

The problem

On a BullMQ deployment, the hold and defer policies from FIX-1834 did nothing once work went to a queue worker. A job under either policy ran as allow. So a task's notice could start a second reply in a conversation while the first reply was still streaming. On a single server this was already fixed. On a queue it was not.

What this does

A queue worker now applies hold and defer the same way the single-server path does:

  • A reply (hold) runs as soon as a worker takes it. From the moment it is accepted until it ends, the conversation reads busy. Another reply still starts at once beside it.
  • A notice (defer) waits until no reply is running or waiting on the conversation, then runs. Notices on one conversation run one at a time. While it waits, the notice goes back on the queue as a delayed job, so it never holds a worker slot. It yields to newer replies for 30 seconds. After that, it lines up behind the replies already running.
  • A crashed worker never gives back the reply's place. The place's lease runs out (10 s by default), and then the notice runs.
  • The ownership check and the cap work as on one server. A caller who doesn't own the conversation is refused before it can mark the conversation busy or use up the cap. The cap allows 32 waiting notices per conversation, counted in the process that accepted them, and each one counts until its job ends.

All of this uses the engine's existing lease backend and arbiter. The new planDeferWait drives both the in-process wait and the worker's requeue schedule, so there is one definition of the defer wait.

What's asked of you

Nothing here is yours to decide. This carries out your 2026-10-09 call to plan a follow-up rather than leave a known gap.

Parts worth reviewing closely

  • packages/bullmq/src/worker.ts → takeTurn / claimWhenFree: the three ways a job reaches its turn (now, when-free, behind), and the retake-when-missing path, which now also covers a hold place.
  • packages/engine/src/transports/host/createInboundTransportHost.ts: the downgrade of hold/defer to allow is gone when the backend is shared. leaseTurn is set from the resolved decision, never from the caller (BP-031).
  • packages/engine/src/transports/concurrency/arbiter.ts: handOff(jobEnded). A handed-off defer leaves the per-key count only when its job's finished settles.

Linear: FIX-1836 (epic FIX-1786). Bug route, so there is no spec. This PR merges.

How to review this: implementation PR, so correctness and second paths

Is this correct, and does it match the direction? Look hardest at correctness, second paths (BP-035), auth and routing from caller-controllable input (BP-031), legacy-shape tolerance (BP-030), and concurrency and null boundaries. Don't reopen the hold and defer semantics. FIX-1834 settled them, and this PR only carries them across the queue.

Engineering calls: decided along the way, no product sign-off needed
  • The cap counts a queued notice until its job ends, not until it starts. The accepting process can't see when a worker starts the job. If the bridge never delivers the job's terminal (a lost pub/sub message), that notice keeps one of the 32 slots on that conversation until the process restarts. If this is wrong, a long-lived web process can slowly lose cap headroom on a busy conversation. Moving the count into Redis would fix that and needs a new lease-backend call.
  • A waiting notice is not woken when the key empties. It checks again on the backoff, at most every 2 s. The engine's own wait over a shared backend behaves the same way. The cost is up to about 2 s of extra latency before a notice starts.
  • Backward compatible on the job shape (BP-030). leaseTurn == null means wait in line, as before. A worker from the previous release treats a hold job as queue (a reply can wait for the one before it and time out after 30 s) and runs a defer job at once. The bullmq README tells you to roll out workers first.
  • FlowWorkerDeps.deferPatienceMs is added so the processor matches an arbiter built with its own patience, and so the tests can use a short one. It is not exposed through bullmqWorker options.
Verification: each Done-when line, red then green

Red was produced by running the new tests against the pre-change code: the old worker.ts from HEAD~ for the worker cases, and the old host for the engine cases.

Done-when Test Red Green
Reply and notice on BullMQ workers run one at a time, notice after reply bullmq/test/queue-arbitration.redis.test.ts → "runs the notice after the reply, waits for a reply that starts while it waits, and runs notices one at a time" (1 web + 2 worker processes, real Redis) notices overlapped the replies; replies serialized pass
Same ownership-first ordering and per-conversation cap engine/test/transports/external-hold-defer.test.ts (cap refuses past N with nothing enqueued and frees when a job ends; a stranger can neither use the cap nor mark the conversation busy) 3 failed with the downgrade in place; the 2 ownership cases failed when admission was moved ahead of ownership pass
A crashed reply's worker doesn't strand the notice queue-arbitration.redis.test.ts → "runs the notice once the lease of a crashed reply's worker runs out" (the reply ignores abort and its worker loses Redis; the notice starts after leaseMs and before leaseMs + 3 s, while the reply is still running) the notice ran immediately pass
Worker wait details bullmq/test/lease-backend.redis.test.ts → defer suite (no place taken while held; out of patience it lines up ahead of newer replies with no budget; cancelled while waiting it never runs) 3 failed pass
README documents it packages/bullmq/README.md → Concurrency across workers — —

Runs:

  • REDIS_URL=redis://127.0.0.1:6379 CI=1 pnpm --filter @flow-state-dev/bullmq exec vitest run: 10 files, 80 tests passed. The two Redis suites were repeated 3 times, 19/19 each time.
  • pnpm --filter @flow-state-dev/engine exec vitest run: 3647 passed and 6 skipped. One timing test ("stays well under a quadratic blowup") failed once while the machine was under load (load average about 12). It passes in isolation, and this change doesn't touch it.
  • turbo typecheck --concurrency=2: 74/74 tasks passed.

No goal check applies. This is transport plumbing with no model in the path, and the real-Redis end-to-end cases above are the real-path proof.

Files
  • engine: dispatcher.ts (leaseTurn), concurrency/lease-backend.ts (planDeferWait, DEFER_PATIENCE_MS, planQueueWait({ budgetMs })), concurrency/arbiter.ts (uses both planners, handOff(jobEnded)), host/createInboundTransportHost.ts, exports, README, flowstate/types.ts doc
  • bullmq: worker.ts, dispatcher.ts, types.ts, README
  • docs: apps/docs/docs/advanced/concurrency-policies.md (removed the "runs as allow on a queue" bullet and added the queue behaviour), docs/architecture/inbound-transports.md
  • .changeset/bullmq-hold-defer.md: patch for bullmq and engine

🤖 Generated with Claude Code

https://claude.ai/code/session_01XMg2dz8sMEdyzCQTRsaMTV


Generated by Claude Code


Note

High Risk
Changes distributed concurrency arbitration, lease handoff, and BullMQ job processing—areas where ordering bugs or cap leaks can cause duplicate runs or stuck sessions.

Overview
BullMQ workers now enforce hold and defer across the deployment instead of treating queued jobs as allow. The host stops downgrading those policies when a shared leaseBackend is in use, and jobs carry DispatchEnvelope.leaseTurn (now for hold, when-free for defer) through the dispatcher into job data.

The worker’s turn logic was extended: hold runs immediately on its place; defer claims the key when empty via delayed requeues driven by shared planDeferWait / DEFER_PATIENCE_MS, then may line up behind with unbounded planQueueWait({ budgetMs: Infinity }). The arbiter’s handOff(jobEnded) keeps handed-off defer requests in the per-key cap until the job finishes; ownership is still checked before admission.

Docs and changesets describe queue behaviour, the 32-defer cap counting through job end, and worker-before-enqueuer rollout (older workers mis-handle leaseTurn).

Reviewed by Cursor Bugbot for commit b1f5575. Configure here.

A request handed to a BullMQ worker under `hold` or `defer` used to run as
`allow`, so a task's notice could start a second reply mid-reply.

The host now stops downgrading them when the arbiter's backend is shared,
and the job carries `leaseTurn`: a `hold` place has its turn at once; a
`defer` job carries no place and its worker claims the key once it is free,
requeueing (no slot held) on the engine's new `planDeferWait` schedule and
lining up behind the key's places once its patience is spent. The arbiter's
in-process defer wait uses the same `planDeferWait`. Ownership is still
checked before any place is taken, and the dispatching process counts a
handed-off defer against its per-key cap until the job ends
(`handOff(jobEnded)`). A crashed reply's place expires with its lease and
the notice then runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMg2dz8sMEdyzCQTRsaMTV
@linear-code

linear-code Bot commented Oct 9, 2026

Copy link
Copy Markdown

FIX-1836

@vercel

vercel Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
flow-state-dev-docs Ignored Ignored Oct 10, 2026 12:11am UTC
flow-state-dev-kitchen-sink Ignored Ignored Oct 10, 2026 12:11am UTC
flow-state-dev-ui Ignored Ignored Preview Oct 10, 2026 12:11am UTC

Request Review

@cursor

cursor Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 14753314-64c3-4a41-baec-fd3b0bb5ab24)

Describe what a waiting defer does to queue/reject requests rather than
how its worker checks; show planDeferWait's two return shapes; drop
place/lease internals before they are introduced; and make the cap and
requeue wording match the code: a queued defer counts until its job ends
(running included), and the requeue delay is at most two seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMg2dz8sMEdyzCQTRsaMTV
@cursor

cursor Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 257776aa-5a28-4065-8744-b222d0bee40f)

@jhoffner
jhoffner marked this pull request as ready for review October 9, 2026 20:52
@cursor

cursor Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 990eca5f-d1d6-4fa0-a43d-38e3fb8fc446)

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Snob — already tight. No POC.

Read packages/bullmq/src/worker.ts (claimWhenFree, takeTurn), packages/bullmq/src/types.ts, packages/bullmq/src/dispatcher.ts, packages/engine/src/transports/dispatcher.ts, packages/engine/src/transports/concurrency/lease-backend.ts (planDeferWait, planQueueWait), packages/engine/src/transports/concurrency/arbiter.ts (deferredAdmission, pollForTurn, handOff), and packages/engine/src/transports/host/createInboundTransportHost.ts.

A rewrite would not shrink the contract:

  • planDeferWait is already the one patience decision the arbiter and the worker both call. Folding it into planQueueWait ties: the terminals differ (line-up vs a timeout error), and the in-memory wake still needs patienceLeftMs.
  • claimWhenFree's two takes are different contracts. ifEmpty's heldBy means try later; a line-up's heldBy is an error. A shared flag saves about ten lines and hides that.
  • { kind: "behind" } is the bit a delayed job has to remember. leaseTurn == null is already the 30s queue budget. A second field is the same size.
  • The now path's updateData matches the queue success path. Sharing that write saves a handful of lines and wraps the wait in a negation.
  • The host's hold/defer ternary is the policy mapping. A helper is larger. handOff(jobEnded) is one callback; only defer uses the promise.

Head 44b3685180cc764ac42a65464ffe54061518e0a7.

Open in Web View Automation 

Sent by Cursor Automation: Code Snob

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplify / approach review (FIX-1836)

Verdict: The direction is right — extend the FIX-1634 spine (leasePlace + shared planQueueWait) rather than a second concurrency path. Pulling planDeferWait into lease-backend and refactoring the arbiter onto it is the main simplification win in this diff; handOff(handle.finished) for handed-off defers is the minimal fix for process-local maxDeferredPerKey when the worker claims the key. I don't see a materially smaller design that keeps enqueue-side cap, ownership-before-admit, and parity with in-process defer without moving counting into Redis.

Alternatives considered (why this isn't over-built)

Alternative Why it's not clearly better
Policy string on the job (concurrency: "defer") instead of leaseTurn Worker still needs runtime phases (when-free → now → behind); host would still map policy → envelope.
Defer cap only in the worker / Redis New distributed counter + policy in the lease backend; fights "backend orders places, arbiter owns policy".
Third-party mutex/semaphore Doesn't replace ordered lines, ifEmpty claim, job-state reconciliation, or BullMQ delayed requeue.
Keep downgrading hold/defer to allow on external dispatch Correctly removed — that was the bug.

Where complexity is real vs. cosmetic

Inherent: Two executors (in-process arbiter vs BullMQ worker) must implement the same decisions with different waits (whenFree + sleep vs moveToDelayed). Sharing planDeferWait / planQueueWait is the right ceiling unless you introduce a pluggable "sleep until next defer step" helper (probably not worth it for ~35 lines × 2).

Cosmetic / follow-ups (non-blocking):

  • Name the worker-only turn union (JobLeaseTurn = LeaseTurn | { kind: "behind" }) so DispatchEnvelope stays host→worker contract only.
  • Flatten claimWhenFree → takeTurn recursion after line-up (single "wait with place" path).
  • Pin planDeferWait in lease-backend.test.ts the way queue wait is pinned (arbiter + worker tests cover it indirectly today).
  • Export an UNBOUNDED_QUEUE_WAIT (or reuse arbiter's constant) instead of raw Infinity in the worker's behind branch.
  • One-line doc on why external defer counts until jobEnded but in-process defer counts until run starts — easy to misread as a bug.

Performance (only if you expect hot keys)

Defer polling is timer-driven (good), but each failed claim is updateData + moveToDelayed + Redis take/reconcile — fine for notice-style volume; worth watching if many defers pile on one key. In-process gets whenFree wake; workers accept up to ~2s extra latency after the key frees. A Redis "key emptied" wake for placeless defers would be a future optimization, not a simplify.

Tests / evidence

external-hold-defer.test.ts + Redis arbitration/processor coverage match existing FIX-1634 patterns. I ran pnpm --filter @flow-state-dev/engine test (includes the new transport tests) — green.

Approval: Ship as-is; optional nits above are polish, not a different architecture.

Open in Web View Automation 

Sent by Cursor Automation: Review For Best Approach

Comment thread packages/bullmq/src/worker.ts Resolved
Comment thread packages/bullmq/src/types.ts Resolved

jhoffner commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

Test (1/3) and Packed release installs and imports on 44b3685 failed before any test ran. Both times, the runner couldn't start the Redis service container: Docker Hub refused the unauthenticated redis:7 pull (toomanyrequests). That isn't caused by this PR. The one re-run I allow myself failed the same way, and #2987 is hitting the same failure.

No fix exists on main yet. My proposed patch is to pull the same official image from the ECR Public mirror, which allows anonymous pulls, in .github/workflows/ci.yml at lines 98 and 173:

-        image: redis:7
+        image: public.ecr.aws/docker/library/redis:7

It's waiting on a maintainer's go-ahead. This PR's own code already passed every check on 452b702; the only commit since then changed docs.


Generated by Claude Code

@jhoffner

jhoffner commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-09T23:47:28.286104Z 44b3685 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 44b3685180

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/bullmq/src/worker.ts Resolved
Comment thread packages/bullmq/src/worker.ts Resolved
… (FIX-1836)

A defer that claimed a free key was saved as `leaseTurn: now`, so on a
retry after its worker crashed and another run took the key, it retook a
place at the back and ran beside that run. It is now saved as `claimed`:
while it keeps its place it runs at once (it was first by construction);
having lost it, it goes back to `when-free` and claims the key again. A
`hold` (`now`) still runs at once.

A place taken but not recorded on the job's data (a failed updateData) was
known to no one and blocked its key until its lease ran out. Every take on
the worker path (claim, line-up, retake) now records the place in the same
step and gives it back if the record fails.

Also: name the job-side union `JobLeaseTurn`; pin `planDeferWait`'s
patience boundary and clamp, and `planQueueWait`'s unbounded budget; widen
the crash case's lower bound to the last renewal (a quarter-lease early).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMg2dz8sMEdyzCQTRsaMTV
@cursor

cursor Bot commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: c5f3e11e-4ab6-4c9b-9f6d-632ec4af4832)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants