Repository navigation
fix(bullmq): queue workers honour hold and defer (FIX-1836) - #2990
Conversation
A request handed to a BullMQ worker under `hold` or `defer` used to run as `allow`, so a task's notice could start a second reply mid-reply. The host now stops downgrading them when the arbiter's backend is shared, and the job carries `leaseTurn`: a `hold` place has its turn at once; a `defer` job carries no place and its worker claims the key once it is free, requeueing (no slot held) on the engine's new `planDeferWait` schedule and lining up behind the key's places once its patience is spent. The arbiter's in-process defer wait uses the same `planDeferWait`. Ownership is still checked before any place is taken, and the dispatching process counts a handed-off defer against its per-key cap until the job ends (`handOff(jobEnded)`). A crashed reply's place expires with its lease and the notice then runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XMg2dz8sMEdyzCQTRsaMTV
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 14753314-64c3-4a41-baec-fd3b0bb5ab24) |
Describe what a waiting defer does to queue/reject requests rather than how its worker checks; show planDeferWait's two return shapes; drop place/lease internals before they are introduced; and make the cap and requeue wording match the code: a queued defer counts until its job ends (running included), and the requeue delay is at most two seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XMg2dz8sMEdyzCQTRsaMTV
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 257776aa-5a28-4065-8744-b222d0bee40f) |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 990eca5f-d1d6-4fa0-a43d-38e3fb8fc446) |
There was a problem hiding this comment.
Code Snob — already tight. No POC.
Read packages/bullmq/src/worker.ts (claimWhenFree, takeTurn), packages/bullmq/src/types.ts, packages/bullmq/src/dispatcher.ts, packages/engine/src/transports/dispatcher.ts, packages/engine/src/transports/concurrency/lease-backend.ts (planDeferWait, planQueueWait), packages/engine/src/transports/concurrency/arbiter.ts (deferredAdmission, pollForTurn, handOff), and packages/engine/src/transports/host/createInboundTransportHost.ts.
A rewrite would not shrink the contract:
planDeferWaitis already the one patience decision the arbiter and the worker both call. Folding it intoplanQueueWaitties: the terminals differ (line-upvs a timeout error), and the in-memory wake still needspatienceLeftMs.claimWhenFree's twotakes are different contracts.ifEmpty'sheldBymeans try later; a line-up'sheldByis an error. A shared flag saves about ten lines and hides that.{ kind: "behind" }is the bit a delayed job has to remember.leaseTurn == nullis already the 30s queue budget. A second field is the same size.- The
nowpath'supdateDatamatches the queue success path. Sharing that write saves a handful of lines and wraps the wait in a negation. - The host's hold/defer ternary is the policy mapping. A helper is larger.
handOff(jobEnded)is one callback; onlydeferuses the promise.
Head 44b3685180cc764ac42a65464ffe54061518e0a7.
Sent by Cursor Automation: Code Snob
There was a problem hiding this comment.
Simplify / approach review (FIX-1836)
Verdict: The direction is right — extend the FIX-1634 spine (leasePlace + shared planQueueWait) rather than a second concurrency path. Pulling planDeferWait into lease-backend and refactoring the arbiter onto it is the main simplification win in this diff; handOff(handle.finished) for handed-off defers is the minimal fix for process-local maxDeferredPerKey when the worker claims the key. I don't see a materially smaller design that keeps enqueue-side cap, ownership-before-admit, and parity with in-process defer without moving counting into Redis.
Alternatives considered (why this isn't over-built)
| Alternative | Why it's not clearly better |
|---|---|
Policy string on the job (concurrency: "defer") instead of leaseTurn |
Worker still needs runtime phases (when-free → now → behind); host would still map policy → envelope. |
| Defer cap only in the worker / Redis | New distributed counter + policy in the lease backend; fights "backend orders places, arbiter owns policy". |
| Third-party mutex/semaphore | Doesn't replace ordered lines, ifEmpty claim, job-state reconciliation, or BullMQ delayed requeue. |
Keep downgrading hold/defer to allow on external dispatch |
Correctly removed — that was the bug. |
Where complexity is real vs. cosmetic
Inherent: Two executors (in-process arbiter vs BullMQ worker) must implement the same decisions with different waits (whenFree + sleep vs moveToDelayed). Sharing planDeferWait / planQueueWait is the right ceiling unless you introduce a pluggable "sleep until next defer step" helper (probably not worth it for ~35 lines × 2).
Cosmetic / follow-ups (non-blocking):
- Name the worker-only turn union (
JobLeaseTurn = LeaseTurn | { kind: "behind" }) soDispatchEnvelopestays host→worker contract only. - Flatten
claimWhenFree→takeTurnrecursion afterline-up(single "wait with place" path). - Pin
planDeferWaitinlease-backend.test.tsthe way queue wait is pinned (arbiter + worker tests cover it indirectly today). - Export an
UNBOUNDED_QUEUE_WAIT(or reuse arbiter's constant) instead of rawInfinityin the worker'sbehindbranch. - One-line doc on why external defer counts until
jobEndedbut in-process defer counts untilrunstarts — easy to misread as a bug.
Performance (only if you expect hot keys)
Defer polling is timer-driven (good), but each failed claim is updateData + moveToDelayed + Redis take/reconcile — fine for notice-style volume; worth watching if many defers pile on one key. In-process gets whenFree wake; workers accept up to ~2s extra latency after the key frees. A Redis "key emptied" wake for placeless defers would be a future optimization, not a simplify.
Tests / evidence
external-hold-defer.test.ts + Redis arbitration/processor coverage match existing FIX-1634 patterns. I ran pnpm --filter @flow-state-dev/engine test (includes the new transport tests) — green.
Approval: Ship as-is; optional nits above are polish, not a different architecture.
Sent by Cursor Automation: Review For Best Approach
|
No fix exists on main yet. My proposed patch is to pull the same official image from the ECR Public mirror, which allows anonymous pulls, in - image: redis:7
+ image: public.ecr.aws/docker/library/redis:7It's waiting on a maintainer's go-ahead. This PR's own code already passed every check on 452b702; the only commit since then changed docs. Generated by Claude Code |
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 44b3685180
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… (FIX-1836) A defer that claimed a free key was saved as `leaseTurn: now`, so on a retry after its worker crashed and another run took the key, it retook a place at the back and ran beside that run. It is now saved as `claimed`: while it keeps its place it runs at once (it was first by construction); having lost it, it goes back to `when-free` and claims the key again. A `hold` (`now`) still runs at once. A place taken but not recorded on the job's data (a failed updateData) was known to no one and blocked its key until its lease ran out. Every take on the worker path (claim, line-up, retake) now records the place in the same step and gives it back if the record fails. Also: name the job-side union `JobLeaseTurn`; pin `planDeferWait`'s patience boundary and clamp, and `planQueueWait`'s unbounded budget; widen the crash case's lower bound to the last renewal (a quarter-lease early). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XMg2dz8sMEdyzCQTRsaMTV
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: c5f3e11e-4ab6-4c9b-9f6d-632ec4af4832) |


The problem
On a BullMQ deployment, the
holdanddeferpolicies from FIX-1834 did nothing once work went to a queue worker. A job under either policy ran asallow. So a task's notice could start a second reply in a conversation while the first reply was still streaming. On a single server this was already fixed. On a queue it was not.What this does
A queue worker now applies
holdanddeferthe same way the single-server path does:hold) runs as soon as a worker takes it. From the moment it is accepted until it ends, the conversation reads busy. Another reply still starts at once beside it.defer) waits until no reply is running or waiting on the conversation, then runs. Notices on one conversation run one at a time. While it waits, the notice goes back on the queue as a delayed job, so it never holds a worker slot. It yields to newer replies for 30 seconds. After that, it lines up behind the replies already running.All of this uses the engine's existing lease backend and arbiter. The new
planDeferWaitdrives both the in-process wait and the worker's requeue schedule, so there is one definition of thedeferwait.What's asked of you
Nothing here is yours to decide. This carries out your 2026-10-09 call to plan a follow-up rather than leave a known gap.
Parts worth reviewing closely
packages/bullmq/src/worker.ts→takeTurn/claimWhenFree: the three ways a job reaches its turn (now,when-free,behind), and the retake-when-missing path, which now also covers aholdplace.packages/engine/src/transports/host/createInboundTransportHost.ts: the downgrade ofhold/defertoallowis gone when the backend is shared.leaseTurnis set from the resolved decision, never from the caller (BP-031).packages/engine/src/transports/concurrency/arbiter.ts:handOff(jobEnded). A handed-offdeferleaves the per-key count only when its job'sfinishedsettles.Linear: FIX-1836 (epic FIX-1786). Bug route, so there is no spec. This PR merges.
How to review this: implementation PR, so correctness and second paths
Is this correct, and does it match the direction? Look hardest at correctness, second paths (BP-035), auth and routing from caller-controllable input (BP-031), legacy-shape tolerance (BP-030), and concurrency and null boundaries. Don't reopen the
holdanddefersemantics. FIX-1834 settled them, and this PR only carries them across the queue.Engineering calls: decided along the way, no product sign-off needed
leaseTurn == nullmeans wait in line, as before. A worker from the previous release treats aholdjob asqueue(a reply can wait for the one before it and time out after 30 s) and runs adeferjob at once. The bullmq README tells you to roll out workers first.FlowWorkerDeps.deferPatienceMsis added so the processor matches an arbiter built with its own patience, and so the tests can use a short one. It is not exposed throughbullmqWorkeroptions.Verification: each Done-when line, red then green
Red was produced by running the new tests against the pre-change code: the old
worker.tsfromHEAD~for the worker cases, and the old host for the engine cases.bullmq/test/queue-arbitration.redis.test.ts→ "runs the notice after the reply, waits for a reply that starts while it waits, and runs notices one at a time" (1 web + 2 worker processes, real Redis)engine/test/transports/external-hold-defer.test.ts(cap refuses past N with nothing enqueued and frees when a job ends; a stranger can neither use the cap nor mark the conversation busy)queue-arbitration.redis.test.ts→ "runs the notice once the lease of a crashed reply's worker runs out" (the reply ignores abort and its worker loses Redis; the notice starts afterleaseMsand beforeleaseMs + 3 s, while the reply is still running)bullmq/test/lease-backend.redis.test.ts→ defer suite (no place taken while held; out of patience it lines up ahead of newer replies with no budget; cancelled while waiting it never runs)packages/bullmq/README.md→ Concurrency across workersRuns:
REDIS_URL=redis://127.0.0.1:6379 CI=1 pnpm --filter @flow-state-dev/bullmq exec vitest run: 10 files, 80 tests passed. The two Redis suites were repeated 3 times, 19/19 each time.pnpm --filter @flow-state-dev/engine exec vitest run: 3647 passed and 6 skipped. One timing test ("stays well under a quadratic blowup") failed once while the machine was under load (load average about 12). It passes in isolation, and this change doesn't touch it.turbo typecheck --concurrency=2: 74/74 tasks passed.No goal check applies. This is transport plumbing with no model in the path, and the real-Redis end-to-end cases above are the real-path proof.
Files
dispatcher.ts(leaseTurn),concurrency/lease-backend.ts(planDeferWait,DEFER_PATIENCE_MS,planQueueWait({ budgetMs })),concurrency/arbiter.ts(uses both planners,handOff(jobEnded)),host/createInboundTransportHost.ts, exports, README,flowstate/types.tsdocworker.ts,dispatcher.ts,types.ts, READMEapps/docs/docs/advanced/concurrency-policies.md(removed the "runs asallowon a queue" bullet and added the queue behaviour),docs/architecture/inbound-transports.md.changeset/bullmq-hold-defer.md: patch for bullmq and engine🤖 Generated with Claude Code
https://claude.ai/code/session_01XMg2dz8sMEdyzCQTRsaMTV
Generated by Claude Code
Note
High Risk
Changes distributed concurrency arbitration, lease handoff, and BullMQ job processing—areas where ordering bugs or cap leaks can cause duplicate runs or stuck sessions.
Overview
BullMQ workers now enforce
holdanddeferacross the deployment instead of treating queued jobs asallow. The host stops downgrading those policies when a sharedleaseBackendis in use, and jobs carryDispatchEnvelope.leaseTurn(nowforhold,when-freefordefer) through the dispatcher into job data.The worker’s turn logic was extended:
holdruns immediately on its place;deferclaims the key when empty via delayed requeues driven by sharedplanDeferWait/DEFER_PATIENCE_MS, then may line upbehindwith unboundedplanQueueWait({ budgetMs: Infinity }). The arbiter’shandOff(jobEnded)keeps handed-offdeferrequests in the per-key cap until the job finishes; ownership is still checked before admission.Docs and changesets describe queue behaviour, the 32-defer cap counting through job end, and worker-before-enqueuer rollout (older workers mis-handle
leaseTurn).Reviewed by Cursor Bugbot for commit b1f5575. Configure here.