Skip to content

Fix/cluster stuck in close - #2217

Merged
PavelPashov merged 8 commits into
redis:mainfrom
Duwab:fix/cluster-stuck-in-close
Oct 7, 2026
Merged

PavelPashov merged 8 commits into
redis:mainfrom
Duwab:fix/cluster-stuck-in-close

Conversation

@Duwab

@Duwab Duwab commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

This pull request fixes a Cluster that can stay in close or disconnecting forever when its status changes while connect() is resolving the startup node hostnames.

In that state, nothing reconnects or ends the client, and queued commands never settle.

Context

Full details of the investigation are in this gist. In short:

Changes

  1. Add a test that reproduces the stuck connection: failing run before the fix.

  2. Fix it with a new private handleAbortedConnect() method. It is only called in the branch where connect() already gives up with Connection is aborted; nothing else in connect() changes. It runs the step that the missing close listener would have run, and only in the two states where nothing else ever will:

    1. close: the late end of a removed node made the pool emit drain before connect() had registered its listener. → handleCloseEvent(): reconnect through clusterRetryStrategy, or end. (covered by lateNodeEnd.ts)
    2. disconnecting with an empty pool: disconnect() or quit() was called during the lookup, and no node is left to emit drain. → set close, then the same handleCloseEvent(), which ends the client and rejects the offline queue. (covered by dnsLookup.ts)

    Every other status (reconnecting, end, disconnecting with nodes left) is left as is. This is the same close + handleCloseEvent() pair that the lookup-failure branch just below already uses (fix(cluster): handle connection errors by reconnection #762).

  3. Add tests in dnsLookup.ts for disconnect() and quit() called while the startup nodes are being resolved.

lateNodeEnd test before the fix

As the issue may seem a bit theoretical, the lateNodeEnd.ts test gives a concrete example of how the cluster can get stuck.

See how it behaves before the fix (results in failure) :

sequenceDiagram
    participant N1 as Redis node 1 <br/>(:30001)
    participant C as client <br/>= new ioredis.Cluster()
    participant N2 as Redis node 2 <br/>(:30002)

    Note over C: 1. connect() → dnsLookup("redis.test") resolves at once<br/>cluster.get("foo") waits in the offline queue
    C->>N1: 2. CLUSTER SLOTS
    N1-->>C: 3. 0-8000 → :30001, 8001-16383 → :30002
    C->>N2: 4. connect (pool = node 1 + node 2)
    C->>N1: 5. CLUSTER INFO (ready check)
    N1-->>C: 6. cluster_state:fail
    Note over C: 7. disconnect(true) → pool.reset([])<br/>both nodes leave the pool at once, status "disconnecting"
    C--)N1: 8. socket.end() → TCP FIN
    C--)N2: 9. socket.end() → TCP FIN
    N2--)C: 10. TCP FIN (Node.js auto-ends the socket)
    Note over C: 11. node 2 emits "end" → pool empty → "drain"<br/>status "close" → retry scheduled in ~100 ms
    Note over N1: 12. no TCP FIN back (allowHalfOpen)<br/>socket stays half-closed

    rect rgba(255, 80, 80, 0.12)
    Note over C: 13. ~100 ms: connect() again, status "connecting"<br/>dnsLookup("redis.test") still pending
    Note over C: 14. ~200 ms: disconnectTimeout elapses
    C--xN1: 15. socket.destroy()
    Note over C: 16. node 1 emits "end" → pool empty → "drain" again<br/>status "close", but connect() has no "close" listener yet
    Note over C: 17. dnsLookup resolves → status is "close", not "connecting"<br/>connect() rejects "Connection is aborted"
    end

    Note over N1,N2: 18. stuck in "close": no reconnectTimeout, no "end"<br/>get("foo") never settles
Loading

lateNodeEnd test after the fix

connect() now calls handleAbortedConnect() when it aborts, and that method checks the current client status.

→ In this particular case, the connection is already closed (this.status === "close"), so it calls this.handleCloseEvent().

See how it behaves after the fix :

sequenceDiagram
    participant N1 as Redis node 1 <br/>(:30001)
    participant C as client <br/>= new ioredis.Cluster()
    participant N2 as Redis node 2 <br/>(:30002)

    Note over N1,N2: 1–12. same as without the fix: ready check fails, disconnect(true),<br/>node 2 closes at once, node 1 stays half-closed

    rect rgba(255, 80, 80, 0.12)
    Note over C: 13. ~100 ms: connect() again, status "connecting"<br/>dnsLookup("redis.test") still pending
    Note over C: 14. ~200 ms: disconnectTimeout elapses
    C--xN1: 15. socket.destroy()
    Note over C: 16. node 1 emits "end" → pool empty → "drain" again<br/>status "close", but connect() has no "close" listener yet
    Note over C: 17. dnsLookup resolves → status is "close", not "connecting"<br/>connect() rejects "Connection is aborted"
    end

    rect rgba(80, 200, 120, 0.15)
    Note over C: 18. handleAbortedConnect() → handleCloseEvent()<br/>status "reconnecting" → retry scheduled in ~100 ms
    end

    Note over C: 19. ~300 ms: connect() again → dnsLookup resolves at once
    C->>N1: 20. CLUSTER SLOTS (new connection)
    N1-->>C: 21. 0-8000 → :30001, 8001-16383 → :30002
    C->>N2: 22. connect (pool = node 1 + node 2)
    C->>N2: 23. CLUSTER INFO (ready check)
    N2-->>C: 24. cluster_state:ok
    Note over C: 25. status "ready" → offline queue is flushed
    C->>N2: 26. GET foo (slot 12182)
    N2-->>C: 27. "bar"
    Note over N1,N2: 28. get("foo") resolves with "bar"
Loading

Note

Medium Risk
Touches core Cluster connection lifecycle during races between DNS lookup, pool drain, and disconnect; limited to the aborted-connect branch with new regression tests.

Overview
Fixes a Cluster client that could stay stuck in close or disconnecting when connect() was still resolving startup hostnames and the status changed before connect() registered its close listeners—so nothing retried, nothing emitted end, and offline-queued commands never settled.

When DNS resolution finishes and the status is no longer connecting, connect() still rejects with Connection is aborted, but it now calls handleAbortedConnect(). That method transitions disconnecting with an empty pool to close, then for close runs the same invokeReadyDelayedCallbacks + handleCloseEvent() path the lookup-failure branch already used—scheduling clusterRetryStrategy or ending and flushing the queue.

Adds lateNodeEnd (late node end during reconnect lookup) and dnsLookup tests for disconnect() / quit() during pending startup DNS resolution.

Reviewed by Cursor Bugbot for commit 3b3c509. Bugbot is set up for automated code reviews on this repo. Configure here.

Duwab added 7 commits October 2, 2026 11:51
… node resolution

When the status changes while connect() is resolving the startup nodes,
connect() rejects before its "close" listeners are attached and with an
empty pool, so nothing moves the cluster out of "close" or
"disconnecting". Handle that case explicitly so the cluster reconnects or
emits "end", and pending commands are rejected.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit ea1445b. Configure here.

Comment thread lib/cluster/index.ts Resolved
Pipelines, transactions and auto-pipelined commands issued before the
slots are known wait in `_readyDelayedCallbacks`. When connect() was
aborted during startup node resolution, the close path flushed the
offline queue but left these callbacks pending forever. Reject them with
the same error as the regular close listener.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@PavelPashov

Copy link
Copy Markdown
Contributor

@Duwab Thanks for the PR, I'll review it and get back to you

@PavelPashov PavelPashov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@PavelPashov
PavelPashov merged commit a5fad1d into redis:main Oct 7, 2026
22 checks passed
@Duwab

Duwab commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the quick review and merge @PavelPashov 🙏

Feel free to ping me if anything comes up!

github-actions Bot pushed a commit that referenced this pull request Oct 12, 2026
# [6.1.0-beta.1](v6.0.0...v6.1.0-beta.1) (2026-10-12)

### Bug Fixes

* accept field arrays in hmget/hmgetBuffer types ([#2167](#2167)) ([c0cd66c](c0cd66c))
* align auto-pipeline callbacks for dynamic commands ([#2180](#2180)) ([7479cab](7479cab))
* clean up Sentinel failover subscriptions after quit ([#2198](#2198)) ([727edfa](727edfa))
* cluster stuck in close ([#2217](#2217)) ([a5fad1d](a5fad1d))
* **cluster:** bucket autopipelined MSET objects by their first key ([#2207](#2207)) ([b4e3237](b4e3237))
* **cluster:** hash autopipelined Buffer keys by their bytes ([#2208](#2208)) ([a8378bd](a8378bd)), closes [Command#getSlot](https://github.com/Command/issues/getSlot)
* **cluster:** match Buffer shard channels when unsubscribing ([#2203](#2203)) ([d997d01](d997d01))
* **cluster:** support zero-argument sunsubscribe with sharded subscribers ([#2137](#2137)) ([13ff050](13ff050)), closes [#2133](#2133)
* **cluster:** vendor cluster-key-slot to fix empty hash tag slot bug ([#2172](#2172)) ([b479b76](b479b76))
* decode integer replies up to MAX_SAFE_INTEGER exactly ([#2164](#2164)) ([28940b5](28940b5))
* decode RESP3 attribute replies instead of failing the connection ([#2213](#2213)) ([6a8cb3e](6a8cb3e)), closes [#decodeNestedType](https://github.com/redis/ioredis/issues/decodeNestedType) [#decodeMapAsObject](https://github.com/redis/ioredis/issues/decodeMapAsObject) [#decodeMapKey](https://github.com/redis/ioredis/issues/decodeMapKey)
* decode RESP3 doubles with Number ([#2150](#2150)) ([6fb5a10](6fb5a10))
* handle rejections from auto-resubscribe and select on reconnect ([#2187](#2187)) ([de2fbdf](de2fbdf))
* inherit connectTimeout for cluster subscribers ([#2220](#2220)) ([db57c2d](db57c2d))
* keep String.prototype in fast mode after defining VerbatimString ([#2206](#2206)) ([90e1261](90e1261))
* keep subscriber mode while other channel kinds remain subscribed ([#2196](#2196)) ([eb97beb](eb97beb))
* keep the exec error out of its own previousErrors ([#2218](#2218)) ([2c602cf](2c602cf))
* preserve empty keys and match patterns in scan streams ([#2199](#2199)) ([b169c3a](b169c3a))
* reject unfulfilled commands dropped on reconnect ([#2194](#2194)) ([d95d05a](d95d05a)), closes [#2193](#2193)
* reset retry attempts when connect() is called after giving up ([#2204](#2204)) ([b597303](b597303)), closes [#1463](#1463)
* **types:** accept documented TLS profile options ([#2195](#2195)) ([56a7fb8](56a7fb8)), closes [#1626](#1626)
* **types:** add event overloads to Cluster interface ([#2201](#2201)) ([3996eaf](3996eaf)), closes [#2191](#2191)
* **types:** add smessage and smessageBuffer event overloads ([#2191](#2191)) ([ee48f4c](ee48f4c)), closes [#2190](#2190)
* **types:** allow number for zrange stop argument ([#2163](#2163)) ([932f4c2](932f4c2))

### Features

* bless commands ([#2209](#2209)) ([9181ee0](9181ee0))
* **cluster:** allow configuring node role for Pub/Sub subscriber ([#2176](#2176)) ([6b9130c](6b9130c))
* smart client handoffs ([#2165](#2165)) ([d8ae3cb](d8ae3cb))
github-actions Bot pushed a commit that referenced this pull request Oct 12, 2026
# [6.1.0-beta.1](v6.0.0...v6.1.0-beta.1) (2026-10-12)

### Bug Fixes

* accept field arrays in hmget/hmgetBuffer types ([#2167](#2167)) ([c0cd66c](c0cd66c))
* align auto-pipeline callbacks for dynamic commands ([#2180](#2180)) ([7479cab](7479cab))
* clean up Sentinel failover subscriptions after quit ([#2198](#2198)) ([727edfa](727edfa))
* cluster stuck in close ([#2217](#2217)) ([a5fad1d](a5fad1d))
* **cluster:** bucket autopipelined MSET objects by their first key ([#2207](#2207)) ([b4e3237](b4e3237))
* **cluster:** hash autopipelined Buffer keys by their bytes ([#2208](#2208)) ([a8378bd](a8378bd)), closes [Command#getSlot](https://github.com/Command/issues/getSlot)
* **cluster:** match Buffer shard channels when unsubscribing ([#2203](#2203)) ([d997d01](d997d01))
* **cluster:** support zero-argument sunsubscribe with sharded subscribers ([#2137](#2137)) ([13ff050](13ff050)), closes [#2133](#2133)
* **cluster:** vendor cluster-key-slot to fix empty hash tag slot bug ([#2172](#2172)) ([b479b76](b479b76))
* decode integer replies up to MAX_SAFE_INTEGER exactly ([#2164](#2164)) ([28940b5](28940b5))
* decode RESP3 attribute replies instead of failing the connection ([#2213](#2213)) ([6a8cb3e](6a8cb3e)), closes [#decodeNestedType](https://github.com/redis/ioredis/issues/decodeNestedType) [#decodeMapAsObject](https://github.com/redis/ioredis/issues/decodeMapAsObject) [#decodeMapKey](https://github.com/redis/ioredis/issues/decodeMapKey)
* decode RESP3 doubles with Number ([#2150](#2150)) ([6fb5a10](6fb5a10))
* handle rejections from auto-resubscribe and select on reconnect ([#2187](#2187)) ([de2fbdf](de2fbdf))
* inherit connectTimeout for cluster subscribers ([#2220](#2220)) ([db57c2d](db57c2d))
* keep String.prototype in fast mode after defining VerbatimString ([#2206](#2206)) ([90e1261](90e1261))
* keep subscriber mode while other channel kinds remain subscribed ([#2196](#2196)) ([eb97beb](eb97beb))
* keep the exec error out of its own previousErrors ([#2218](#2218)) ([2c602cf](2c602cf))
* preserve empty keys and match patterns in scan streams ([#2199](#2199)) ([b169c3a](b169c3a))
* reject unfulfilled commands dropped on reconnect ([#2194](#2194)) ([d95d05a](d95d05a)), closes [#2193](#2193)
* reset retry attempts when connect() is called after giving up ([#2204](#2204)) ([b597303](b597303)), closes [#1463](#1463)
* **types:** accept documented TLS profile options ([#2195](#2195)) ([56a7fb8](56a7fb8)), closes [#1626](#1626)
* **types:** add event overloads to Cluster interface ([#2201](#2201)) ([3996eaf](3996eaf)), closes [#2191](#2191)
* **types:** add smessage and smessageBuffer event overloads ([#2191](#2191)) ([ee48f4c](ee48f4c)), closes [#2190](#2190)
* **types:** allow number for zrange stop argument ([#2163](#2163)) ([932f4c2](932f4c2))

### Features

* bless commands ([#2209](#2209)) ([9181ee0](9181ee0))
* **cluster:** allow configuring node role for Pub/Sub subscriber ([#2176](#2176)) ([6b9130c](6b9130c))
* smart client handoffs ([#2165](#2165)) ([d8ae3cb](d8ae3cb))
github-actions Bot pushed a commit that referenced this pull request Oct 12, 2026
# [6.1.0-beta.1](v6.0.0...v6.1.0-beta.1) (2026-10-12)

### Bug Fixes

* accept field arrays in hmget/hmgetBuffer types ([#2167](#2167)) ([c0cd66c](c0cd66c))
* align auto-pipeline callbacks for dynamic commands ([#2180](#2180)) ([7479cab](7479cab))
* clean up Sentinel failover subscriptions after quit ([#2198](#2198)) ([727edfa](727edfa))
* cluster stuck in close ([#2217](#2217)) ([a5fad1d](a5fad1d))
* **cluster:** bucket autopipelined MSET objects by their first key ([#2207](#2207)) ([b4e3237](b4e3237))
* **cluster:** hash autopipelined Buffer keys by their bytes ([#2208](#2208)) ([a8378bd](a8378bd)), closes [Command#getSlot](https://github.com/Command/issues/getSlot)
* **cluster:** match Buffer shard channels when unsubscribing ([#2203](#2203)) ([d997d01](d997d01))
* **cluster:** support zero-argument sunsubscribe with sharded subscribers ([#2137](#2137)) ([13ff050](13ff050)), closes [#2133](#2133)
* **cluster:** vendor cluster-key-slot to fix empty hash tag slot bug ([#2172](#2172)) ([b479b76](b479b76))
* decode integer replies up to MAX_SAFE_INTEGER exactly ([#2164](#2164)) ([28940b5](28940b5))
* decode RESP3 attribute replies instead of failing the connection ([#2213](#2213)) ([6a8cb3e](6a8cb3e)), closes [#decodeNestedType](https://github.com/redis/ioredis/issues/decodeNestedType) [#decodeMapAsObject](https://github.com/redis/ioredis/issues/decodeMapAsObject) [#decodeMapKey](https://github.com/redis/ioredis/issues/decodeMapKey)
* decode RESP3 doubles with Number ([#2150](#2150)) ([6fb5a10](6fb5a10))
* handle rejections from auto-resubscribe and select on reconnect ([#2187](#2187)) ([de2fbdf](de2fbdf))
* inherit connectTimeout for cluster subscribers ([#2220](#2220)) ([db57c2d](db57c2d))
* keep String.prototype in fast mode after defining VerbatimString ([#2206](#2206)) ([90e1261](90e1261))
* keep subscriber mode while other channel kinds remain subscribed ([#2196](#2196)) ([eb97beb](eb97beb))
* keep the exec error out of its own previousErrors ([#2218](#2218)) ([2c602cf](2c602cf))
* preserve empty keys and match patterns in scan streams ([#2199](#2199)) ([b169c3a](b169c3a))
* reject unfulfilled commands dropped on reconnect ([#2194](#2194)) ([d95d05a](d95d05a)), closes [#2193](#2193)
* reset retry attempts when connect() is called after giving up ([#2204](#2204)) ([b597303](b597303)), closes [#1463](#1463)
* **types:** accept documented TLS profile options ([#2195](#2195)) ([56a7fb8](56a7fb8)), closes [#1626](#1626)
* **types:** add event overloads to Cluster interface ([#2201](#2201)) ([3996eaf](3996eaf)), closes [#2191](#2191)
* **types:** add smessage and smessageBuffer event overloads ([#2191](#2191)) ([ee48f4c](ee48f4c)), closes [#2190](#2190)
* **types:** allow number for zrange stop argument ([#2163](#2163)) ([932f4c2](932f4c2))

### Features

* bless commands ([#2209](#2209)) ([9181ee0](9181ee0))
* **cluster:** allow configuring node role for Pub/Sub subscriber ([#2176](#2176)) ([6b9130c](6b9130c))
* smart client handoffs ([#2165](#2165)) ([d8ae3cb](d8ae3cb))
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 6.1.0-beta.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

PavelPashov pushed a commit that referenced this pull request Oct 20, 2026
…2237)

disconnect() empties the connection pool before the node connections
have closed. Each node's later "end" then finds the pool empty and emits
"drain", and every drain set the cluster status to "close". The first
one closes and ends the cluster, the rest move the ended cluster back to
"close", where sendCommand() queues commands instead of rejecting them,
so they never settle.

Ignore a drain once the cluster has ended. A drain in any other status
still closes the cluster, so quit(), reconnects and the aborted-connect
path from #2217 are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants