Skip to content

[FEATURE]: need to use consistent hashing for KV events in backend engines #2924

@PeaBrane

Description

@PeaBrane

Feature request

The router may not update the radix tree properly in some cases, if the hashing algorothm used by the backend engines are not consistent across processes/nodes.

For example, we can set PYTHONHASHSEED = 0, or --prefix-caching-algo sha256 for vllm. Need to document this or showcase in examples.

Similarly, verify/document for sglang + trtllm + other engines.

Describe the problem you're encountering

N/A

Describe alternatives you've tried

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions