Skip to content

fix(cluster): hash autopipelined Buffer keys by their bytes - #2208

Merged
PavelPashov merged 1 commit into
redis:mainfrom
breken-ai:fix/autopipeline-buffer-key-slot
Sep 30, 2026
Merged

PavelPashov merged 1 commit into
redis:mainfrom
breken-ai:fix/autopipeline-buffer-key-slot

Conversation

@breken-ai

@breken-ai breken-ai commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

This pull request fixes auto-pipelining in cluster mode for commands whose key is a binary Buffer.

Bug

executeWithAutoPipelining builds the slot for the auto pipeline with a template string, `${prefix}${key}`. That decodes a Buffer key as UTF-8, so bytes that are not valid UTF-8 become U+FFFD, and the slot is computed for the decoded string instead of the bytes that go on the wire. Command#getSlot and Pipeline#exec hash the raw bytes, so the two disagree.

Different binary keys can then land in the same auto pipeline even though their real slots belong to different nodes, and Pipeline#exec rejects the whole batch:

Error: All keys in the pipeline should belong to the same slots allocation group

Repro with enableAutoPipelining: true on a 3-node cluster (slots 0-5000, 5001-9999, 10000-16383):

await Promise.all([
  cluster.get(Buffer.from([0xff])), // slot 7920
  cluster.get(Buffer.from([0xfe])), // slot 3793
]);
// both reject: each key decodes to "�" (slot 13381), so they share one pipeline

Without auto-pipelining the same calls work. This affects binary keys such as serialized IDs or hashes stored as raw bytes.

Fix

When the first key is a Buffer, compute the slot from Buffer.concat([Buffer.from(prefix), key]), the same bytes Command sends after applying keyPrefix. String and number keys still use the existing string path.

Tests

Added should bucket binary Buffer keys by their raw bytes to test/functional/cluster/autopipelining.ts (mock servers, no Redis needed). It fails on main with the error above and passes with the fix. test/unit plus test/functional/cluster pass (591 passing), and tsc --noEmit, eslint and prettier are clean.

This is a small, isolated bug fix with a focused test, so I opened it directly as CONTRIBUTING allows.


Note

Low Risk
Small, isolated change to cluster auto-pipeline slot routing for Buffer keys, with a focused regression test.

Overview
Fixes cluster auto-pipelining when commands use binary Buffer keys. Slot grouping used `${prefix}${key}`, which turns invalid UTF-8 bytes into the same replacement character and can put keys that belong on different nodes into one pipeline; Pipeline#exec then rejects the batch.

executeWithAutoPipelining now hashes Buffer keys from Buffer.concat([Buffer.from(prefix), key]), matching how Command applies keyPrefix before calculateSlot. String and number keys are unchanged.

Adds a functional test that Promise.all two get calls with Buffer.from([0xff]) and Buffer.from([0xfe]) under enableAutoPipelining, with mock handlers on the correct nodes.

Reviewed by Cursor Bugbot for commit ca8d279. Bugbot is set up for automated code reviews on this repo. Configure here.

Auto-pipelining built the cluster slot key with a template string, which
decodes Buffer keys as UTF-8. Binary keys that are not valid UTF-8 then
hash to the wrong slot, and different keys can collapse into the same
auto pipeline even though their real slots live on different nodes. The
batch then failed with "All keys in the pipeline should belong to the
same slots allocation group". Hash Buffer keys by their raw bytes, as
Command#getSlot already does.

@PavelPashov PavelPashov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@breken-ai looks good, thank you for the contribution

@PavelPashov
PavelPashov merged commit a8378bd into redis:main Sep 30, 2026
22 checks passed
github-actions Bot pushed a commit that referenced this pull request Oct 12, 2026
# [6.1.0-beta.1](v6.0.0...v6.1.0-beta.1) (2026-10-12)

### Bug Fixes

* accept field arrays in hmget/hmgetBuffer types ([#2167](#2167)) ([c0cd66c](c0cd66c))
* align auto-pipeline callbacks for dynamic commands ([#2180](#2180)) ([7479cab](7479cab))
* clean up Sentinel failover subscriptions after quit ([#2198](#2198)) ([727edfa](727edfa))
* cluster stuck in close ([#2217](#2217)) ([a5fad1d](a5fad1d))
* **cluster:** bucket autopipelined MSET objects by their first key ([#2207](#2207)) ([b4e3237](b4e3237))
* **cluster:** hash autopipelined Buffer keys by their bytes ([#2208](#2208)) ([a8378bd](a8378bd)), closes [Command#getSlot](https://github.com/Command/issues/getSlot)
* **cluster:** match Buffer shard channels when unsubscribing ([#2203](#2203)) ([d997d01](d997d01))
* **cluster:** support zero-argument sunsubscribe with sharded subscribers ([#2137](#2137)) ([13ff050](13ff050)), closes [#2133](#2133)
* **cluster:** vendor cluster-key-slot to fix empty hash tag slot bug ([#2172](#2172)) ([b479b76](b479b76))
* decode integer replies up to MAX_SAFE_INTEGER exactly ([#2164](#2164)) ([28940b5](28940b5))
* decode RESP3 attribute replies instead of failing the connection ([#2213](#2213)) ([6a8cb3e](6a8cb3e)), closes [#decodeNestedType](https://github.com/redis/ioredis/issues/decodeNestedType) [#decodeMapAsObject](https://github.com/redis/ioredis/issues/decodeMapAsObject) [#decodeMapKey](https://github.com/redis/ioredis/issues/decodeMapKey)
* decode RESP3 doubles with Number ([#2150](#2150)) ([6fb5a10](6fb5a10))
* handle rejections from auto-resubscribe and select on reconnect ([#2187](#2187)) ([de2fbdf](de2fbdf))
* inherit connectTimeout for cluster subscribers ([#2220](#2220)) ([db57c2d](db57c2d))
* keep String.prototype in fast mode after defining VerbatimString ([#2206](#2206)) ([90e1261](90e1261))
* keep subscriber mode while other channel kinds remain subscribed ([#2196](#2196)) ([eb97beb](eb97beb))
* keep the exec error out of its own previousErrors ([#2218](#2218)) ([2c602cf](2c602cf))
* preserve empty keys and match patterns in scan streams ([#2199](#2199)) ([b169c3a](b169c3a))
* reject unfulfilled commands dropped on reconnect ([#2194](#2194)) ([d95d05a](d95d05a)), closes [#2193](#2193)
* reset retry attempts when connect() is called after giving up ([#2204](#2204)) ([b597303](b597303)), closes [#1463](#1463)
* **types:** accept documented TLS profile options ([#2195](#2195)) ([56a7fb8](56a7fb8)), closes [#1626](#1626)
* **types:** add event overloads to Cluster interface ([#2201](#2201)) ([3996eaf](3996eaf)), closes [#2191](#2191)
* **types:** add smessage and smessageBuffer event overloads ([#2191](#2191)) ([ee48f4c](ee48f4c)), closes [#2190](#2190)
* **types:** allow number for zrange stop argument ([#2163](#2163)) ([932f4c2](932f4c2))

### Features

* bless commands ([#2209](#2209)) ([9181ee0](9181ee0))
* **cluster:** allow configuring node role for Pub/Sub subscriber ([#2176](#2176)) ([6b9130c](6b9130c))
* smart client handoffs ([#2165](#2165)) ([d8ae3cb](d8ae3cb))
github-actions Bot pushed a commit that referenced this pull request Oct 12, 2026
# [6.1.0-beta.1](v6.0.0...v6.1.0-beta.1) (2026-10-12)

### Bug Fixes

* accept field arrays in hmget/hmgetBuffer types ([#2167](#2167)) ([c0cd66c](c0cd66c))
* align auto-pipeline callbacks for dynamic commands ([#2180](#2180)) ([7479cab](7479cab))
* clean up Sentinel failover subscriptions after quit ([#2198](#2198)) ([727edfa](727edfa))
* cluster stuck in close ([#2217](#2217)) ([a5fad1d](a5fad1d))
* **cluster:** bucket autopipelined MSET objects by their first key ([#2207](#2207)) ([b4e3237](b4e3237))
* **cluster:** hash autopipelined Buffer keys by their bytes ([#2208](#2208)) ([a8378bd](a8378bd)), closes [Command#getSlot](https://github.com/Command/issues/getSlot)
* **cluster:** match Buffer shard channels when unsubscribing ([#2203](#2203)) ([d997d01](d997d01))
* **cluster:** support zero-argument sunsubscribe with sharded subscribers ([#2137](#2137)) ([13ff050](13ff050)), closes [#2133](#2133)
* **cluster:** vendor cluster-key-slot to fix empty hash tag slot bug ([#2172](#2172)) ([b479b76](b479b76))
* decode integer replies up to MAX_SAFE_INTEGER exactly ([#2164](#2164)) ([28940b5](28940b5))
* decode RESP3 attribute replies instead of failing the connection ([#2213](#2213)) ([6a8cb3e](6a8cb3e)), closes [#decodeNestedType](https://github.com/redis/ioredis/issues/decodeNestedType) [#decodeMapAsObject](https://github.com/redis/ioredis/issues/decodeMapAsObject) [#decodeMapKey](https://github.com/redis/ioredis/issues/decodeMapKey)
* decode RESP3 doubles with Number ([#2150](#2150)) ([6fb5a10](6fb5a10))
* handle rejections from auto-resubscribe and select on reconnect ([#2187](#2187)) ([de2fbdf](de2fbdf))
* inherit connectTimeout for cluster subscribers ([#2220](#2220)) ([db57c2d](db57c2d))
* keep String.prototype in fast mode after defining VerbatimString ([#2206](#2206)) ([90e1261](90e1261))
* keep subscriber mode while other channel kinds remain subscribed ([#2196](#2196)) ([eb97beb](eb97beb))
* keep the exec error out of its own previousErrors ([#2218](#2218)) ([2c602cf](2c602cf))
* preserve empty keys and match patterns in scan streams ([#2199](#2199)) ([b169c3a](b169c3a))
* reject unfulfilled commands dropped on reconnect ([#2194](#2194)) ([d95d05a](d95d05a)), closes [#2193](#2193)
* reset retry attempts when connect() is called after giving up ([#2204](#2204)) ([b597303](b597303)), closes [#1463](#1463)
* **types:** accept documented TLS profile options ([#2195](#2195)) ([56a7fb8](56a7fb8)), closes [#1626](#1626)
* **types:** add event overloads to Cluster interface ([#2201](#2201)) ([3996eaf](3996eaf)), closes [#2191](#2191)
* **types:** add smessage and smessageBuffer event overloads ([#2191](#2191)) ([ee48f4c](ee48f4c)), closes [#2190](#2190)
* **types:** allow number for zrange stop argument ([#2163](#2163)) ([932f4c2](932f4c2))

### Features

* bless commands ([#2209](#2209)) ([9181ee0](9181ee0))
* **cluster:** allow configuring node role for Pub/Sub subscriber ([#2176](#2176)) ([6b9130c](6b9130c))
* smart client handoffs ([#2165](#2165)) ([d8ae3cb](d8ae3cb))
github-actions Bot pushed a commit that referenced this pull request Oct 12, 2026
# [6.1.0-beta.1](v6.0.0...v6.1.0-beta.1) (2026-10-12)

### Bug Fixes

* accept field arrays in hmget/hmgetBuffer types ([#2167](#2167)) ([c0cd66c](c0cd66c))
* align auto-pipeline callbacks for dynamic commands ([#2180](#2180)) ([7479cab](7479cab))
* clean up Sentinel failover subscriptions after quit ([#2198](#2198)) ([727edfa](727edfa))
* cluster stuck in close ([#2217](#2217)) ([a5fad1d](a5fad1d))
* **cluster:** bucket autopipelined MSET objects by their first key ([#2207](#2207)) ([b4e3237](b4e3237))
* **cluster:** hash autopipelined Buffer keys by their bytes ([#2208](#2208)) ([a8378bd](a8378bd)), closes [Command#getSlot](https://github.com/Command/issues/getSlot)
* **cluster:** match Buffer shard channels when unsubscribing ([#2203](#2203)) ([d997d01](d997d01))
* **cluster:** support zero-argument sunsubscribe with sharded subscribers ([#2137](#2137)) ([13ff050](13ff050)), closes [#2133](#2133)
* **cluster:** vendor cluster-key-slot to fix empty hash tag slot bug ([#2172](#2172)) ([b479b76](b479b76))
* decode integer replies up to MAX_SAFE_INTEGER exactly ([#2164](#2164)) ([28940b5](28940b5))
* decode RESP3 attribute replies instead of failing the connection ([#2213](#2213)) ([6a8cb3e](6a8cb3e)), closes [#decodeNestedType](https://github.com/redis/ioredis/issues/decodeNestedType) [#decodeMapAsObject](https://github.com/redis/ioredis/issues/decodeMapAsObject) [#decodeMapKey](https://github.com/redis/ioredis/issues/decodeMapKey)
* decode RESP3 doubles with Number ([#2150](#2150)) ([6fb5a10](6fb5a10))
* handle rejections from auto-resubscribe and select on reconnect ([#2187](#2187)) ([de2fbdf](de2fbdf))
* inherit connectTimeout for cluster subscribers ([#2220](#2220)) ([db57c2d](db57c2d))
* keep String.prototype in fast mode after defining VerbatimString ([#2206](#2206)) ([90e1261](90e1261))
* keep subscriber mode while other channel kinds remain subscribed ([#2196](#2196)) ([eb97beb](eb97beb))
* keep the exec error out of its own previousErrors ([#2218](#2218)) ([2c602cf](2c602cf))
* preserve empty keys and match patterns in scan streams ([#2199](#2199)) ([b169c3a](b169c3a))
* reject unfulfilled commands dropped on reconnect ([#2194](#2194)) ([d95d05a](d95d05a)), closes [#2193](#2193)
* reset retry attempts when connect() is called after giving up ([#2204](#2204)) ([b597303](b597303)), closes [#1463](#1463)
* **types:** accept documented TLS profile options ([#2195](#2195)) ([56a7fb8](56a7fb8)), closes [#1626](#1626)
* **types:** add event overloads to Cluster interface ([#2201](#2201)) ([3996eaf](3996eaf)), closes [#2191](#2191)
* **types:** add smessage and smessageBuffer event overloads ([#2191](#2191)) ([ee48f4c](ee48f4c)), closes [#2190](#2190)
* **types:** allow number for zrange stop argument ([#2163](#2163)) ([932f4c2](932f4c2))

### Features

* bless commands ([#2209](#2209)) ([9181ee0](9181ee0))
* **cluster:** allow configuring node role for Pub/Sub subscriber ([#2176](#2176)) ([6b9130c](6b9130c))
* smart client handoffs ([#2165](#2165)) ([d8ae3cb](d8ae3cb))
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 6.1.0-beta.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

JacobLey added a commit to JacobLey/iovalkey that referenced this pull request Oct 22, 2026
Auto-pipelining built the slot key with a template string, which
decodes a Buffer key as UTF-8. Bytes that are not valid UTF-8 become
U+FFFD, so different binary keys could share an auto pipeline even when
their slots are served by different nodes, and Pipeline#exec rejected
the batch. Hash Buffer keys from their bytes (with keyPrefix prepended),
as Command does.

Ported from redis/ioredis#2208.

Signed-off-by: Jacob Ley <jacobtley@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants