chore: set default device to cpu on Multimodal models

[yechank-nvidia]

This PR changes the default device type to be cpu on the main-process so that we are avoiding the interference for worker process GPU job.

Also,

1. Change use_fast=True for Qwen2/2.5-VL models
2. Change the test keywords accordingly

Summary by CodeRabbit

• Bug Fixes

  • Changed the default device for multimodal input processing and media loading from GPU to CPU, improving compatibility for users without GPU access.
  • Enabled faster tokenization for certain models.
  • Adjusted expected keywords in multimodal model tests to better reflect current output descriptions.

• Tests

  • Updated test expectations for multimodal models to align with revised output keywords.

[yechank-nvidia]

/bot run

[tensorrt-cicd]

PR_Github #11771 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #11771 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #8720 completed with status: 'FAILURE'

[yechank-nvidia]

/bot run

[tensorrt-cicd]

PR_Github #11795 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #11795 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #8739 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

[brb-nv]

Non-blocking comment: Is this change not needed for Gemma3, @yechank-nvidia?

[yechank-nvidia]

Hi @brb-nv, double-checked on the H100, seeing one keyword difference output, so changing again. Thx for pointing out.

[yechank-nvidia]

/bot run

[tensorrt-cicd]

PR_Github #11882 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #11882 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #8808 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

[yechank-nvidia]

/bot run

[tensorrt-cicd]

PR_Github #12491 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #12491 [ run ] completed with state FAILURE

[yechank-nvidia]

/bot run

[coderabbitai]

Walkthrough

The changes update device defaults from "cuda" to "cpu" for multimodal input processing and model inference across utility functions, example scripts, and model input processors. Test data for multimodal model outputs is also revised, updating expected keyword lists for various models and modalities. No new features or control flow modifications are introduced.

Changes

File(s)	Change Summary
examples/llm-api/quickstart_multimodal.py	Changed device for multimodal input/model inference from "cuda" to "cpu" in main function.
tensorrt_llm/_torch/models/modeling_mistral.py	Removed explicit "cuda" device placement in Mistral3InputProcessor; device attribute and .to() removed.
tensorrt_llm/_torch/models/modeling_qwen2vl.py	Enabled fast tokenizer, removed device attribute and explicit device placement in input processor base.
tensorrt_llm/inputs/utils.py	Changed default device parameter from "cuda" to "cpu" for all media loading/processing functions.
tests/integration/defs/test_e2e.py	Updated expected keyword lists in multimodal model tests for images and videos.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant ExampleScript
    participant Utils
    participant ModelInputProcessor
    participant Model

    User->>ExampleScript: Run multimodal example
    ExampleScript->>Utils: Load image/video/audio (device="cpu")
    Utils-->>ExampleScript: Return processed tensor(s) on CPU
    ExampleScript->>ModelInputProcessor: Preprocess inputs (no explicit device)
    ModelInputProcessor-->>ExampleScript: Return processed inputs
    ExampleScript->>Model: Run inference (inputs on CPU)
    Model-->>ExampleScript: Output results

Loading

Estimated code review effort

2 (~15 minutes)

Possibly related PRs

• [fix] Fix flaky mistral E2E test #6230: Updates multimodal test keyword expectations, with some overlap in test data adjustments for the mistral-small-3.1-24b-instruct model.

Suggested reviewers

• brb-nv
• liji-nv
• venkywonka
• symphonylyh

Poem

A bunny hopped from CUDA to CPU land,
With images and videos now close at hand.
The keywords it checked, in tests so neat,
Ensuring descriptions were accurate and sweet.
Fast tokenizers now join the crew—
Multimodal magic, all reviewed by you! 🐰✨

---

📜 Recent review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 3e1a0fb and 85255e0.

📒 Files selected for processing (5)

• examples/llm-api/quickstart_multimodal.py (1 hunks)
• tensorrt_llm/_torch/models/modeling_mistral.py (0 hunks)
• tensorrt_llm/_torch/models/modeling_qwen2vl.py (3 hunks)
• tensorrt_llm/inputs/utils.py (5 hunks)
• tests/integration/defs/test_e2e.py (2 hunks)

💤 Files with no reviewable changes (1)

• tensorrt_llm/_torch/models/modeling_mistral.py

🔇 Additional comments (15)

tensorrt_llm/inputs/utils.py (5)

48-48: LGTM: Default device change aligns with PR objectives.

The change from "cuda" to "cpu" as the default device parameter supports the goal of avoiding GPU interference with worker processes.

---

70-70: LGTM: Consistent device default change.

The async version maintains consistency with the synchronous load_image function.

---

95-95: LGTM: Video loading follows the same device pattern.

Consistent with the image loading functions' device default change.

---

144-144: LGTM: Async video loading maintains consistency.

Follows the same pattern as other modified functions.

---

483-483: LGTM: Main multimodal input loader updated consistently.

The default_multimodal_input_loader function now defaults to CPU, maintaining consistency with the individual media loading functions.

examples/llm-api/quickstart_multimodal.py (1)

141-141: LGTM: Example script updated to match new device defaults.

The device change from "cuda" to "cpu" is consistent with the updated default_multimodal_input_loader function and supports the PR's objective of using CPU by default.

tensorrt_llm/_torch/models/modeling_qwen2vl.py (3)

37-37: LGTM: Fast tokenizer enabled as mentioned in PR objectives.

The change to use_fast = True aligns with the PR objectives which specifically mention updating the use_fast parameter for Qwen2 and Qwen2.5-VL models.

---

227-227: LGTM: Removed explicit device placement for rotary embeddings.

The removal of device-specific tensor operations supports the CPU-first approach mentioned in the PR objectives. The tensor will use the default device or be moved to the appropriate device when needed downstream.

---

345-345: LGTM: Removed device-specific operations from preprocessing.

Consistent with the broader pattern of removing explicit CUDA device assignments from input processing components.

tests/integration/defs/test_e2e.py (6)

1997-2002: Test data updated for llava-v1.6-mistral-7b model.

The expected keywords have been updated to reflect changes in model outputs after the device placement modification from GPU to CPU. The new keywords maintain semantic coherence with the test images.

---

2007-2010: Test data updated for qwen2-vl-7b-instruct model.

The keyword changes appear logical and maintain appropriate descriptions for the test scenarios (ocean/stormy scenes, natural landscapes, traffic conditions).

---

2018-2021: Test data updated for qwen2.5-vl-7b-instruct model.

Multiple keyword lists have been updated for both image and video modalities. The changes maintain thematic consistency with the test content.

Also applies to: 2024-2024

---

2029-2031: Test data updated for mistral-small-3.1-24b-instruct model.

The updated keywords continue to appropriately describe the test images with terms like "dramatic", "seascape", "scenic", and "landscape".

---

2038-2038: Test data updated for gemma-3-27b-it model.

Single keyword change from generic "traffic" to more specific "Changi" (location reference), which suggests more precise location identification in the model output.

---

1980-2041: Manual verification required: No evidence of automated validation found

I did not find any scripts or test run artifacts that confirm the newly updated keyword sets. Please manually execute the multimodal end-to-end tests and compare the actual model outputs on CPU against these expected keywords to prevent false failures.

• File: tests/integration/defs/test_e2e.py (Lines 1980–2041)

✨ Finishing Touches

• 📝 Generate Docstrings

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[yechank-nvidia]

/bot run

[tensorrt-cicd]

PR_Github #12548 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #12548 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #9330 completed with status: 'FAILURE'

[yechank-nvidia]

/bot run

[tensorrt-cicd]

PR_Github #12571 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #12571 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #9350 completed with status: 'SUCCESS'