Skip to content

Add deterministic RL training experiment with vLLM #1975

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tianyu-l merged 17 commits into from
Nov 7, 2025
Merged

Add deterministic RL training experiment with vLLM #1975

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
5fc60f8
Add deterministic RL training experiment with vLLM
bwasti Oct 31, 2025
2823b41
Add missing weight conversion files and fix imports
teja-rao Nov 1, 2025
aea9af1
Address review comments
teja-rao Nov 1, 2025
62b20b6
Remove redundant round-trip weight loading
teja-rao Nov 1, 2025
3c18115
Optimize vLLM weight reloading using collective_rpc
teja-rao Nov 1, 2025
7be6112
Add comprehensive gradient checking and debugging
bwasti Nov 4, 2025
602e92e
Fix missing gradients for FFN layers by adding SiluAndMul backward pass
bwasti Nov 4, 2025
d53740a
Remove gradient debugging statements
bwasti Nov 4, 2025
53b56a5
Update readme
bwasti Nov 4, 2025
90bc444
Cleanup readme and remove unneeded flashv3 backward
bwasti Nov 4, 2025
ae4ebd3
lint
bwasti Nov 4, 2025
5c00c9a
More cleanup
bwasti Nov 4, 2025
1d41a08
Remove extra function
bwasti Nov 5, 2025
44c9946
add better loss controls and a real dataset
bwasti Nov 5, 2025
3268821
Update torchtitan/experiments/deterministic_vllm_rl/models/qwen3/mode…
bwasti Nov 7, 2025
24f4cca
Update README.md
bwasti Nov 7, 2025
e502c65
Improve reference log prob computation and increase max tokens
teja-rao Nov 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update torchtitan/experiments/deterministic_vllm_rl/models/qwen3/mode… 
…l_vllm_compat.py

Co-authored-by: Chien-Chin Huang <[email protected]>
  • Loading branch information
@bwasti @fegin
bwasti and fegin authored Nov 7, 2025
commit 3268821d40b0161024d17006df043012f0a6e016
2 changes: 1 addition & 1 deletion torchtitan/experiments/deterministic_vllm_rl/models/qwen3/model_vllm_compat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -348,7 +348,7 @@ def get_attention_masks(
input_batch: torch.Tensor,
tokenizer: BaseTokenizer,
extra_inputs: dict[str, torch.Tensor] | None = None,
) -> AttentionMasksType:
) -> AttentionMasksType | None:
# vLLM compat mode: no flex attention masks
return None

Expand Down