Skip to content

[vllm] Implement int8 rollout using the flash_rl tool on NPU#5174

Open
nuerxiati wants to merge 1 commit intoverl-project:mainfrom
nuerxiati:int8_rollout
Open

[vllm] Implement int8 rollout using the flash_rl tool on NPU#5174
nuerxiati wants to merge 1 commit intoverl-project:mainfrom
nuerxiati:int8_rollout

Conversation

@nuerxiati
Copy link
Contributor

What does this PR do?

On Ascend device using the Flash-RL tool to modify the inference backend and perform online quantization of weights and activation values ​​by comparing INT8 and BF16 models. Uses the Qwen3-30B INT8 as an example to demonstrate how to run the end-to-end functionality on the NPU.

Checklist Before Starting

  • Search for similar PRs. Paste at least one query link here: ...
  • Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
    • {modules} include fsdp, megatron, veomni, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data, cfg, reward
    • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
    • {type} is in feat, fix, refactor, chore, test
    • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
    • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

Throughput bf16: 167, int8: 195, increased by 16.8%; reward bf16: -0.1012, int8: -0.1204, decreased by 0.96%;

Precision:
image

Performence:
image

int8 100steps rewards:
image

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

@nuerxiati nuerxiati changed the title add int8 rollout [vllm] Implement int8 rollout using the flash_rl tool on NPU Feb 2, 2026
@nuerxiati nuerxiati force-pushed the int8_rollout branch 4 times, most recently from 610a327 to 478f6f5 Compare February 3, 2026 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant