feat: Adding nixl read() multimodal support for vLLM backend #4271
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Alexandre Milesi <[email protected]>
Signed-off-by: Alexandre Milesi <[email protected]>
Signed-off-by: Alexandre Milesi <[email protected]>
Signed-off-by: Alexandre Milesi <[email protected]>
Signed-off-by: Alexandre Milesi <[email protected]>
Signed-off-by: Alexandre Milesi <[email protected]>
Signed-off-by: Alexandre Milesi <[email protected]>
Signed-off-by: Alexandre Milesi <[email protected]>
Signed-off-by: Alexandre Milesi <[email protected]>
Signed-off-by: Alexandre Milesi <[email protected]>
Signed-off-by: Alexandre Milesi <[email protected]>
Signed-off-by: Alexandre Milesi <[email protected]>
Signed-off-by: Krishnan Prashanth <[email protected]>
Signed-off-by: Krishnan Prashanth <[email protected]>
…h/vllm-nixl-read Signed-off-by: KrishnanPrash <[email protected]>
Signed-off-by: Krishnan Prashanth <[email protected]>
KrishnanPrash
commented
Nov 13, 2025
Comment on lines
+127
to
+166
| async def _read_decoded_image_via_nixl( | ||
| self, decoded_meta: Dict[str, Any] | ||
| ) -> PIL.Image.Image: | ||
| """Read decoded image via NIXL RDMA and convert to PIL.Image.""" | ||
| # Lazy-init connector | ||
| if self._connector is None: | ||
| self._connector = connect.Connector() | ||
| await self._connector.initialize() | ||
| logger.info("NIXL connector initialized for decoded media") | ||
|
|
||
| # Extract fields | ||
| meta_str = decoded_meta["nixl_metadata"] | ||
| desc = decoded_meta["nixl_descriptor"] | ||
| shape = decoded_meta["shape"] | ||
|
|
||
| # Create tensor to receive RDMA data | ||
| tensor = torch.empty(shape, dtype=torch.uint8) | ||
|
|
||
| # Build RdmaMetadata from frontend-provided descriptor | ||
| # Frontend sends compressed metadata (matches Python nixl_connect) | ||
| rdma_meta = RdmaMetadata( | ||
| descriptors=[ | ||
| SerializedDescriptor( | ||
| device="cpu" | ||
| if desc.get("mem_type") == "Dram" | ||
| else f"cuda:{desc.get('device_id', 0)}", | ||
| ptr=desc["addr"], | ||
| size=desc["size"], | ||
| ) | ||
| ], | ||
| nixl_metadata=meta_str, | ||
| notification_key=f"img-{shape}", | ||
| operation_kind=int(OperationKind.READ), | ||
| ) | ||
|
|
||
| # RDMA read | ||
| read_op = await self._connector.begin_read( | ||
| rdma_meta, connect.Descriptor(tensor) | ||
| ) | ||
| await read_op.wait_for_completion() |
Contributor
Author
There was a problem hiding this comment.
Not a NIXL expert, so please let me know if I can be doing anything here better.
KrishnanPrash
commented
Nov 13, 2025
Comment on lines
125
to
129
| // Compress metadata before base64 encoding (matches Python nixl_connect behavior) | ||
| // Backend expects: b64:<base64_of_compressed_bytes> | ||
| let mut encoder = ZlibEncoder::new(Vec::new(), Compression::new(6)); | ||
| encoder.write_all(&nixl_md)?; | ||
| let compressed = encoder.finish()?; |
Contributor
Author
There was a problem hiding this comment.
Once again, welcome any suggestions on correct nixl usage.
Contributor
Author
|
Open Question for Testing: Ideally, we would like to test both test cases:
Based on my conversation with @nv-tusharma, IIUC they suggested creating a separate workflow outside |
KrishnanPrash
changed the title
rmccorm4
reviewed
Nov 14, 2025
indrajit96
reviewed
Nov 14, 2025
Signed-off-by: Ayush Agarwal <[email protected]>
Signed-off-by: Krishnan Prashanth <[email protected]>
…h/vllm-nixl-read Signed-off-by: KrishnanPrash <[email protected]>
Signed-off-by: Krishnan Prashanth <[email protected]>
whoisj
reviewed
Nov 25, 2025
|
|
||
| # Build RdmaMetadata from frontend-provided descriptor | ||
| # Frontend sends compressed metadata (matches Python nixl_connect) | ||
| rdma_meta = RdmaMetadata( |
Collaborator
There was a problem hiding this comment.
does this work, have you tested it?
the "normal flow" is to create a passive operation (ReadableOp or WritableOp) and use their .metadata property to get the set of SerializedDescriptors and not manually compose this.
given that this is using an active operation (ReadOp) it should be taking in the metadata to perform the read, not sending the metadata.
Collaborator
There was a problem hiding this comment.
usually metadata comes from the secondary connection, which in turn got it from its ReadableOperation.
|
|
||
| # Extract fields | ||
| meta_str = decoded_meta["nixl_metadata"] | ||
| desc = decoded_meta["nixl_descriptor"] |
| # Frontend sends compressed metadata (matches Python nixl_connect) | ||
| rdma_meta = RdmaMetadata( | ||
| descriptors=[ | ||
| SerializedDescriptor( |
Collaborator
There was a problem hiding this comment.
why not pass desc to a Descriptor and then serialize the descriptor to get the metadata?
Base automatically changed from
alexandrem/frontend-image-decoding-nixl
to
main
November 25, 2025 19:15
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview:
With #3988, we have functional image decoding in the frontend for any b64 or http urls passed with the inference request. This PR builds on top of #3988, and implements the nixl
read()portion of the image decoding workflow for the backend.Details:
Look at
handlers.pyfor the additions to theDECODEDworkflow.