Repository navigation
Fix/cluster stuck in close - #2217
Merged
Merged
Conversation
… node resolution When the status changes while connect() is resolving the startup nodes, connect() rejects before its "close" listeners are attached and with an empty pool, so nothing moves the cluster out of "close" or "disconnecting". Handle that case explicitly so the cluster reconnects or emits "end", and pending commands are rejected.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Reviewed by Cursor Bugbot for commit ea1445b. Configure here.
Pipelines, transactions and auto-pipelined commands issued before the slots are known wait in `_readyDelayedCallbacks`. When connect() was aborted during startup node resolution, the close path flushed the offline queue but left these callbacks pending forever. Reject them with the same error as the regular close listener. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
|
@Duwab Thanks for the PR, I'll review it and get back to you |
Contributor
Author
|
Thanks for the quick review and merge @PavelPashov 🙏 Feel free to ping me if anything comes up! |
github-actions Bot
pushed a commit
that referenced
this pull request
Oct 12, 2026
# [6.1.0-beta.1](v6.0.0...v6.1.0-beta.1) (2026-10-12) ### Bug Fixes * accept field arrays in hmget/hmgetBuffer types ([#2167](#2167)) ([c0cd66c](c0cd66c)) * align auto-pipeline callbacks for dynamic commands ([#2180](#2180)) ([7479cab](7479cab)) * clean up Sentinel failover subscriptions after quit ([#2198](#2198)) ([727edfa](727edfa)) * cluster stuck in close ([#2217](#2217)) ([a5fad1d](a5fad1d)) * **cluster:** bucket autopipelined MSET objects by their first key ([#2207](#2207)) ([b4e3237](b4e3237)) * **cluster:** hash autopipelined Buffer keys by their bytes ([#2208](#2208)) ([a8378bd](a8378bd)), closes [Command#getSlot](https://github.com/Command/issues/getSlot) * **cluster:** match Buffer shard channels when unsubscribing ([#2203](#2203)) ([d997d01](d997d01)) * **cluster:** support zero-argument sunsubscribe with sharded subscribers ([#2137](#2137)) ([13ff050](13ff050)), closes [#2133](#2133) * **cluster:** vendor cluster-key-slot to fix empty hash tag slot bug ([#2172](#2172)) ([b479b76](b479b76)) * decode integer replies up to MAX_SAFE_INTEGER exactly ([#2164](#2164)) ([28940b5](28940b5)) * decode RESP3 attribute replies instead of failing the connection ([#2213](#2213)) ([6a8cb3e](6a8cb3e)), closes [#decodeNestedType](https://github.com/redis/ioredis/issues/decodeNestedType) [#decodeMapAsObject](https://github.com/redis/ioredis/issues/decodeMapAsObject) [#decodeMapKey](https://github.com/redis/ioredis/issues/decodeMapKey) * decode RESP3 doubles with Number ([#2150](#2150)) ([6fb5a10](6fb5a10)) * handle rejections from auto-resubscribe and select on reconnect ([#2187](#2187)) ([de2fbdf](de2fbdf)) * inherit connectTimeout for cluster subscribers ([#2220](#2220)) ([db57c2d](db57c2d)) * keep String.prototype in fast mode after defining VerbatimString ([#2206](#2206)) ([90e1261](90e1261)) * keep subscriber mode while other channel kinds remain subscribed ([#2196](#2196)) ([eb97beb](eb97beb)) * keep the exec error out of its own previousErrors ([#2218](#2218)) ([2c602cf](2c602cf)) * preserve empty keys and match patterns in scan streams ([#2199](#2199)) ([b169c3a](b169c3a)) * reject unfulfilled commands dropped on reconnect ([#2194](#2194)) ([d95d05a](d95d05a)), closes [#2193](#2193) * reset retry attempts when connect() is called after giving up ([#2204](#2204)) ([b597303](b597303)), closes [#1463](#1463) * **types:** accept documented TLS profile options ([#2195](#2195)) ([56a7fb8](56a7fb8)), closes [#1626](#1626) * **types:** add event overloads to Cluster interface ([#2201](#2201)) ([3996eaf](3996eaf)), closes [#2191](#2191) * **types:** add smessage and smessageBuffer event overloads ([#2191](#2191)) ([ee48f4c](ee48f4c)), closes [#2190](#2190) * **types:** allow number for zrange stop argument ([#2163](#2163)) ([932f4c2](932f4c2)) ### Features * bless commands ([#2209](#2209)) ([9181ee0](9181ee0)) * **cluster:** allow configuring node role for Pub/Sub subscriber ([#2176](#2176)) ([6b9130c](6b9130c)) * smart client handoffs ([#2165](#2165)) ([d8ae3cb](d8ae3cb))
github-actions Bot
pushed a commit
that referenced
this pull request
Oct 12, 2026
# [6.1.0-beta.1](v6.0.0...v6.1.0-beta.1) (2026-10-12) ### Bug Fixes * accept field arrays in hmget/hmgetBuffer types ([#2167](#2167)) ([c0cd66c](c0cd66c)) * align auto-pipeline callbacks for dynamic commands ([#2180](#2180)) ([7479cab](7479cab)) * clean up Sentinel failover subscriptions after quit ([#2198](#2198)) ([727edfa](727edfa)) * cluster stuck in close ([#2217](#2217)) ([a5fad1d](a5fad1d)) * **cluster:** bucket autopipelined MSET objects by their first key ([#2207](#2207)) ([b4e3237](b4e3237)) * **cluster:** hash autopipelined Buffer keys by their bytes ([#2208](#2208)) ([a8378bd](a8378bd)), closes [Command#getSlot](https://github.com/Command/issues/getSlot) * **cluster:** match Buffer shard channels when unsubscribing ([#2203](#2203)) ([d997d01](d997d01)) * **cluster:** support zero-argument sunsubscribe with sharded subscribers ([#2137](#2137)) ([13ff050](13ff050)), closes [#2133](#2133) * **cluster:** vendor cluster-key-slot to fix empty hash tag slot bug ([#2172](#2172)) ([b479b76](b479b76)) * decode integer replies up to MAX_SAFE_INTEGER exactly ([#2164](#2164)) ([28940b5](28940b5)) * decode RESP3 attribute replies instead of failing the connection ([#2213](#2213)) ([6a8cb3e](6a8cb3e)), closes [#decodeNestedType](https://github.com/redis/ioredis/issues/decodeNestedType) [#decodeMapAsObject](https://github.com/redis/ioredis/issues/decodeMapAsObject) [#decodeMapKey](https://github.com/redis/ioredis/issues/decodeMapKey) * decode RESP3 doubles with Number ([#2150](#2150)) ([6fb5a10](6fb5a10)) * handle rejections from auto-resubscribe and select on reconnect ([#2187](#2187)) ([de2fbdf](de2fbdf)) * inherit connectTimeout for cluster subscribers ([#2220](#2220)) ([db57c2d](db57c2d)) * keep String.prototype in fast mode after defining VerbatimString ([#2206](#2206)) ([90e1261](90e1261)) * keep subscriber mode while other channel kinds remain subscribed ([#2196](#2196)) ([eb97beb](eb97beb)) * keep the exec error out of its own previousErrors ([#2218](#2218)) ([2c602cf](2c602cf)) * preserve empty keys and match patterns in scan streams ([#2199](#2199)) ([b169c3a](b169c3a)) * reject unfulfilled commands dropped on reconnect ([#2194](#2194)) ([d95d05a](d95d05a)), closes [#2193](#2193) * reset retry attempts when connect() is called after giving up ([#2204](#2204)) ([b597303](b597303)), closes [#1463](#1463) * **types:** accept documented TLS profile options ([#2195](#2195)) ([56a7fb8](56a7fb8)), closes [#1626](#1626) * **types:** add event overloads to Cluster interface ([#2201](#2201)) ([3996eaf](3996eaf)), closes [#2191](#2191) * **types:** add smessage and smessageBuffer event overloads ([#2191](#2191)) ([ee48f4c](ee48f4c)), closes [#2190](#2190) * **types:** allow number for zrange stop argument ([#2163](#2163)) ([932f4c2](932f4c2)) ### Features * bless commands ([#2209](#2209)) ([9181ee0](9181ee0)) * **cluster:** allow configuring node role for Pub/Sub subscriber ([#2176](#2176)) ([6b9130c](6b9130c)) * smart client handoffs ([#2165](#2165)) ([d8ae3cb](d8ae3cb))
github-actions Bot
pushed a commit
that referenced
this pull request
Oct 12, 2026
# [6.1.0-beta.1](v6.0.0...v6.1.0-beta.1) (2026-10-12) ### Bug Fixes * accept field arrays in hmget/hmgetBuffer types ([#2167](#2167)) ([c0cd66c](c0cd66c)) * align auto-pipeline callbacks for dynamic commands ([#2180](#2180)) ([7479cab](7479cab)) * clean up Sentinel failover subscriptions after quit ([#2198](#2198)) ([727edfa](727edfa)) * cluster stuck in close ([#2217](#2217)) ([a5fad1d](a5fad1d)) * **cluster:** bucket autopipelined MSET objects by their first key ([#2207](#2207)) ([b4e3237](b4e3237)) * **cluster:** hash autopipelined Buffer keys by their bytes ([#2208](#2208)) ([a8378bd](a8378bd)), closes [Command#getSlot](https://github.com/Command/issues/getSlot) * **cluster:** match Buffer shard channels when unsubscribing ([#2203](#2203)) ([d997d01](d997d01)) * **cluster:** support zero-argument sunsubscribe with sharded subscribers ([#2137](#2137)) ([13ff050](13ff050)), closes [#2133](#2133) * **cluster:** vendor cluster-key-slot to fix empty hash tag slot bug ([#2172](#2172)) ([b479b76](b479b76)) * decode integer replies up to MAX_SAFE_INTEGER exactly ([#2164](#2164)) ([28940b5](28940b5)) * decode RESP3 attribute replies instead of failing the connection ([#2213](#2213)) ([6a8cb3e](6a8cb3e)), closes [#decodeNestedType](https://github.com/redis/ioredis/issues/decodeNestedType) [#decodeMapAsObject](https://github.com/redis/ioredis/issues/decodeMapAsObject) [#decodeMapKey](https://github.com/redis/ioredis/issues/decodeMapKey) * decode RESP3 doubles with Number ([#2150](#2150)) ([6fb5a10](6fb5a10)) * handle rejections from auto-resubscribe and select on reconnect ([#2187](#2187)) ([de2fbdf](de2fbdf)) * inherit connectTimeout for cluster subscribers ([#2220](#2220)) ([db57c2d](db57c2d)) * keep String.prototype in fast mode after defining VerbatimString ([#2206](#2206)) ([90e1261](90e1261)) * keep subscriber mode while other channel kinds remain subscribed ([#2196](#2196)) ([eb97beb](eb97beb)) * keep the exec error out of its own previousErrors ([#2218](#2218)) ([2c602cf](2c602cf)) * preserve empty keys and match patterns in scan streams ([#2199](#2199)) ([b169c3a](b169c3a)) * reject unfulfilled commands dropped on reconnect ([#2194](#2194)) ([d95d05a](d95d05a)), closes [#2193](#2193) * reset retry attempts when connect() is called after giving up ([#2204](#2204)) ([b597303](b597303)), closes [#1463](#1463) * **types:** accept documented TLS profile options ([#2195](#2195)) ([56a7fb8](56a7fb8)), closes [#1626](#1626) * **types:** add event overloads to Cluster interface ([#2201](#2201)) ([3996eaf](3996eaf)), closes [#2191](#2191) * **types:** add smessage and smessageBuffer event overloads ([#2191](#2191)) ([ee48f4c](ee48f4c)), closes [#2190](#2190) * **types:** allow number for zrange stop argument ([#2163](#2163)) ([932f4c2](932f4c2)) ### Features * bless commands ([#2209](#2209)) ([9181ee0](9181ee0)) * **cluster:** allow configuring node role for Pub/Sub subscriber ([#2176](#2176)) ([6b9130c](6b9130c)) * smart client handoffs ([#2165](#2165)) ([d8ae3cb](d8ae3cb))
|
🎉 This PR is included in version 6.1.0-beta.1 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
PavelPashov
pushed a commit
that referenced
this pull request
Oct 20, 2026
…2237) disconnect() empties the connection pool before the node connections have closed. Each node's later "end" then finds the pool empty and emits "drain", and every drain set the cluster status to "close". The first one closes and ends the cluster, the rest move the ended cluster back to "close", where sendCommand() queues commands instead of rejecting them, so they never settle. Ignore a drain once the cluster has ended. A drain in any other status still closes the cluster, so quit(), reconnects and the aborted-connect path from #2217 are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

This pull request fixes a
Clusterthat can stay incloseordisconnectingforever when its status changes whileconnect()is resolving the startup node hostnames.In that state, nothing reconnects or ends the client, and queued commands never settle.
Context
Full details of the investigation are in this gist. In short:
ioredis@4.28.5, but the affected code is the same inioredis@5.11.1and onmain.A connection failure on some nodes of a Redis Cluster was followed by an ever-growing command queue, which led to an OOM.
→ that is what
test/functional/cluster/lateNodeEnd.tsdoes in this PR.fix(cluster): handle connection errors by reconnection #762: similar fix (adds
handleCloseEvent()to the lookup-failure branch)fix: Improve cluster connection pool logic when disconnecting #1864: similar problem
Changes
Add a test that reproduces the stuck connection: failing run before the fix.
Fix it with a new private
handleAbortedConnect()method. It is only called in the branch whereconnect()already gives up withConnection is aborted; nothing else inconnect()changes. It runs the step that the missingcloselistener would have run, and only in the two states where nothing else ever will:close: the lateendof a removed node made the pool emitdrainbeforeconnect()had registered its listener. →handleCloseEvent(): reconnect throughclusterRetryStrategy, or end. (covered bylateNodeEnd.ts)disconnectingwith an empty pool:disconnect()orquit()was called during the lookup, and no node is left to emitdrain. → setclose, then the samehandleCloseEvent(), which ends the client and rejects the offline queue. (covered bydnsLookup.ts)Every other status (
reconnecting,end,disconnectingwith nodes left) is left as is. This is the sameclose+handleCloseEvent()pair that the lookup-failure branch just below already uses (fix(cluster): handle connection errors by reconnection #762).Add tests in
dnsLookup.tsfordisconnect()andquit()called while the startup nodes are being resolved.lateNodeEndtest before the fixAs the issue may seem a bit theoretical, the
lateNodeEnd.tstest gives a concrete example of how the cluster can get stuck.See how it behaves before the fix (results in failure) :
sequenceDiagram participant N1 as Redis node 1 <br/>(:30001) participant C as client <br/>= new ioredis.Cluster() participant N2 as Redis node 2 <br/>(:30002) Note over C: 1. connect() → dnsLookup("redis.test") resolves at once<br/>cluster.get("foo") waits in the offline queue C->>N1: 2. CLUSTER SLOTS N1-->>C: 3. 0-8000 → :30001, 8001-16383 → :30002 C->>N2: 4. connect (pool = node 1 + node 2) C->>N1: 5. CLUSTER INFO (ready check) N1-->>C: 6. cluster_state:fail Note over C: 7. disconnect(true) → pool.reset([])<br/>both nodes leave the pool at once, status "disconnecting" C--)N1: 8. socket.end() → TCP FIN C--)N2: 9. socket.end() → TCP FIN N2--)C: 10. TCP FIN (Node.js auto-ends the socket) Note over C: 11. node 2 emits "end" → pool empty → "drain"<br/>status "close" → retry scheduled in ~100 ms Note over N1: 12. no TCP FIN back (allowHalfOpen)<br/>socket stays half-closed rect rgba(255, 80, 80, 0.12) Note over C: 13. ~100 ms: connect() again, status "connecting"<br/>dnsLookup("redis.test") still pending Note over C: 14. ~200 ms: disconnectTimeout elapses C--xN1: 15. socket.destroy() Note over C: 16. node 1 emits "end" → pool empty → "drain" again<br/>status "close", but connect() has no "close" listener yet Note over C: 17. dnsLookup resolves → status is "close", not "connecting"<br/>connect() rejects "Connection is aborted" end Note over N1,N2: 18. stuck in "close": no reconnectTimeout, no "end"<br/>get("foo") never settleslateNodeEndtest after the fixconnect()now callshandleAbortedConnect()when it aborts, and that method checks the current client status.→ In this particular case, the connection is already closed (
this.status === "close"), so it callsthis.handleCloseEvent().See how it behaves after the fix :
sequenceDiagram participant N1 as Redis node 1 <br/>(:30001) participant C as client <br/>= new ioredis.Cluster() participant N2 as Redis node 2 <br/>(:30002) Note over N1,N2: 1–12. same as without the fix: ready check fails, disconnect(true),<br/>node 2 closes at once, node 1 stays half-closed rect rgba(255, 80, 80, 0.12) Note over C: 13. ~100 ms: connect() again, status "connecting"<br/>dnsLookup("redis.test") still pending Note over C: 14. ~200 ms: disconnectTimeout elapses C--xN1: 15. socket.destroy() Note over C: 16. node 1 emits "end" → pool empty → "drain" again<br/>status "close", but connect() has no "close" listener yet Note over C: 17. dnsLookup resolves → status is "close", not "connecting"<br/>connect() rejects "Connection is aborted" end rect rgba(80, 200, 120, 0.15) Note over C: 18. handleAbortedConnect() → handleCloseEvent()<br/>status "reconnecting" → retry scheduled in ~100 ms end Note over C: 19. ~300 ms: connect() again → dnsLookup resolves at once C->>N1: 20. CLUSTER SLOTS (new connection) N1-->>C: 21. 0-8000 → :30001, 8001-16383 → :30002 C->>N2: 22. connect (pool = node 1 + node 2) C->>N2: 23. CLUSTER INFO (ready check) N2-->>C: 24. cluster_state:ok Note over C: 25. status "ready" → offline queue is flushed C->>N2: 26. GET foo (slot 12182) N2-->>C: 27. "bar" Note over N1,N2: 28. get("foo") resolves with "bar"Note
Medium Risk
Touches core Cluster connection lifecycle during races between DNS lookup, pool drain, and disconnect; limited to the aborted-connect branch with new regression tests.
Overview
Fixes a Cluster client that could stay stuck in
closeordisconnectingwhenconnect()was still resolving startup hostnames and the status changed beforeconnect()registered itscloselisteners—so nothing retried, nothing emittedend, and offline-queued commands never settled.When DNS resolution finishes and the status is no longer
connecting,connect()still rejects withConnection is aborted, but it now callshandleAbortedConnect(). That method transitionsdisconnectingwith an empty pool toclose, then forcloseruns the sameinvokeReadyDelayedCallbacks+handleCloseEvent()path the lookup-failure branch already used—schedulingclusterRetryStrategyor ending and flushing the queue.Adds
lateNodeEnd(late nodeendduring reconnect lookup) anddnsLookuptests fordisconnect()/quit()during pending startup DNS resolution.Reviewed by Cursor Bugbot for commit 3b3c509. Bugbot is set up for automated code reviews on this repo. Configure here.