[rollout] feat: deprecate all rollout sharding manager#3285
Merged
vermouth1992 merged 11 commits intoverl-project:mainfrom Sep 3, 2025
Merged
[rollout] feat: deprecate all rollout sharding manager#3285vermouth1992 merged 11 commits intoverl-project:mainfrom
vermouth1992 merged 11 commits intoverl-project:mainfrom
Conversation
wuxibin89
requested review from
PeterSH6,
SwordFaith,
chenhaiq and
zhaochenyang20
as code owners
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request refactors the rollout mechanism by deprecating the various sharding managers and introducing a unified trainer_mode and rollout_mode in the hybrid workers. This is a significant architectural improvement that simplifies the codebase. The changes are consistent across both FSDP and Megatron workers. I've found two critical issues: one is the use of os._exit() which can cause unsafe process termination, and the other is an incorrect path in the rollout registry that would lead to an ImportError.
wuxibin89
force-pushed
the
wuxibin/refactor_rollout
branch
from
63c8324 to
8d2922a
Compare
wuxibin89
requested review from
eric-haibin-lin,
tongyx361 and
vermouth1992
as code owners
Collaborator
Author
|
/gemini review |
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a significant and well-executed refactoring to deprecate the rollout sharding managers, replacing them with a cleaner trainer_mode and rollout_mode abstraction in the hybrid workers. The changes are consistently applied across the codebase, including updates to recipes and tests, which simplifies the architecture. My review focuses on two key areas for improvement: enhancing the robustness of the async worker's error handling to prevent service disruption, and addressing code duplication to improve maintainability.
wuxibin89
force-pushed
the
wuxibin/refactor_rollout
branch
from
71f5ca1 to
31c2a64
Compare
vermouth1992
approved these changes
Sep 3, 2025
yellowbee686
pushed a commit
to yellowbee686/verl
that referenced
this pull request
Sep 4, 2025
…3285) ### What does this PR do? Deprecate all rollout sharding manager and replaced by `trainer_mode` and `rollout_mode` in hybrid worker.
cczitong123
pushed a commit
to cczitong123/verl
that referenced
this pull request
Sep 5, 2025
…3285) ### What does this PR do? Deprecate all rollout sharding manager and replaced by `trainer_mode` and `rollout_mode` in hybrid worker.
DDVD233
pushed a commit
to DDVD233/mirl
that referenced
this pull request
Sep 5, 2025
…3285) ### What does this PR do? Deprecate all rollout sharding manager and replaced by `trainer_mode` and `rollout_mode` in hybrid worker.
7 tasks
7 tasks
WncFht
pushed a commit
to WncFht/verl
that referenced
this pull request
Oct 10, 2025
…3285) ### What does this PR do? Deprecate all rollout sharding manager and replaced by `trainer_mode` and `rollout_mode` in hybrid worker.
7 tasks
masoudhashemi
pushed a commit
to masoudhashemi/verl
that referenced
this pull request
Oct 19, 2025
…3285) ### What does this PR do? Deprecate all rollout sharding manager and replaced by `trainer_mode` and `rollout_mode` in hybrid worker.
wuxibin89
added a commit
that referenced
this pull request
Oct 22, 2025
… checkpoint (#3861) ### What does this PR do? Fix a bug introduce in #3285. Currently the initialization steps are: 1. **build actor**: init model and optimizer, then offload to cpu 2. **build rollout**: init model and kv cache 3. **switch to `trainer_mode`**: discard rollout weight and kv cache, load actor model and optimizer to gpu 4. **load_checkpoint**: if checkpoint exists, load model and optimizer from checkpoint, then offload to cpu 5. **switch to `rollout_mode`**: load actor model to gpu, resume rollout weight and sync weight, then offload actor model to cpu and resume kv cache 6. **generate_sequences** The bug is step 4: if checkpoint doesn't exists, then this step is skipped, leading to actor and optimizer both remain in gpu(from step 3), which may cause step 5 cuda oom.
sunnweiwei
pushed a commit
to sunnweiwei/verl
that referenced
this pull request
Oct 23, 2025
… checkpoint (verl-project#3861) ### What does this PR do? Fix a bug introduce in verl-project#3285. Currently the initialization steps are: 1. **build actor**: init model and optimizer, then offload to cpu 2. **build rollout**: init model and kv cache 3. **switch to `trainer_mode`**: discard rollout weight and kv cache, load actor model and optimizer to gpu 4. **load_checkpoint**: if checkpoint exists, load model and optimizer from checkpoint, then offload to cpu 5. **switch to `rollout_mode`**: load actor model to gpu, resume rollout weight and sync weight, then offload actor model to cpu and resume kv cache 6. **generate_sequences** The bug is step 4: if checkpoint doesn't exists, then this step is skipped, leading to actor and optimizer both remain in gpu(from step 3), which may cause step 5 cuda oom.
techkang
pushed a commit
to techkang/verl
that referenced
this pull request
Oct 31, 2025
…3285) ### What does this PR do? Deprecate all rollout sharding manager and replaced by `trainer_mode` and `rollout_mode` in hybrid worker.
wangboxiong320
pushed a commit
to wangboxiong320/verl
that referenced
this pull request
Nov 1, 2025
… checkpoint (verl-project#3861) ### What does this PR do? Fix a bug introduce in verl-project#3285. Currently the initialization steps are: 1. **build actor**: init model and optimizer, then offload to cpu 2. **build rollout**: init model and kv cache 3. **switch to `trainer_mode`**: discard rollout weight and kv cache, load actor model and optimizer to gpu 4. **load_checkpoint**: if checkpoint exists, load model and optimizer from checkpoint, then offload to cpu 5. **switch to `rollout_mode`**: load actor model to gpu, resume rollout weight and sync weight, then offload actor model to cpu and resume kv cache 6. **generate_sequences** The bug is step 4: if checkpoint doesn't exists, then this step is skipped, leading to actor and optimizer both remain in gpu(from step 3), which may cause step 5 cuda oom.
NenoL2001
pushed a commit
to NenoL2001/verl
that referenced
this pull request
Nov 3, 2025
… checkpoint (verl-project#3861) ### What does this PR do? Fix a bug introduce in verl-project#3285. Currently the initialization steps are: 1. **build actor**: init model and optimizer, then offload to cpu 2. **build rollout**: init model and kv cache 3. **switch to `trainer_mode`**: discard rollout weight and kv cache, load actor model and optimizer to gpu 4. **load_checkpoint**: if checkpoint exists, load model and optimizer from checkpoint, then offload to cpu 5. **switch to `rollout_mode`**: load actor model to gpu, resume rollout weight and sync weight, then offload actor model to cpu and resume kv cache 6. **generate_sequences** The bug is step 4: if checkpoint doesn't exists, then this step is skipped, leading to actor and optimizer both remain in gpu(from step 3), which may cause step 5 cuda oom.
AlexJJ009
pushed a commit
to AlexJJ009/verl
that referenced
this pull request
Nov 5, 2025
… checkpoint (verl-project#3861) ### What does this PR do? Fix a bug introduce in verl-project#3285. Currently the initialization steps are: 1. **build actor**: init model and optimizer, then offload to cpu 2. **build rollout**: init model and kv cache 3. **switch to `trainer_mode`**: discard rollout weight and kv cache, load actor model and optimizer to gpu 4. **load_checkpoint**: if checkpoint exists, load model and optimizer from checkpoint, then offload to cpu 5. **switch to `rollout_mode`**: load actor model to gpu, resume rollout weight and sync weight, then offload actor model to cpu and resume kv cache 6. **generate_sequences** The bug is step 4: if checkpoint doesn't exists, then this step is skipped, leading to actor and optimizer both remain in gpu(from step 3), which may cause step 5 cuda oom.
chenjiaoAngel
added a commit
to chenjiaoAngel/verl
that referenced
this pull request
Nov 14, 2025
…3285) ### What does this PR do? Deprecate all rollout sharding manager and replaced by `trainer_mode` and `rollout_mode` in hybrid worker.
chenjiaoAngel
added a commit
to chenjiaoAngel/verl
that referenced
this pull request
Nov 14, 2025
… checkpoint (verl-project#3861) ### What does this PR do? Fix a bug introduce in verl-project#3285. Currently the initialization steps are: 1. **build actor**: init model and optimizer, then offload to cpu 2. **build rollout**: init model and kv cache 3. **switch to `trainer_mode`**: discard rollout weight and kv cache, load actor model and optimizer to gpu 4. **load_checkpoint**: if checkpoint exists, load model and optimizer from checkpoint, then offload to cpu 5. **switch to `rollout_mode`**: load actor model to gpu, resume rollout weight and sync weight, then offload actor model to cpu and resume kv cache 6. **generate_sequences** The bug is step 4: if checkpoint doesn't exists, then this step is skipped, leading to actor and optimizer both remain in gpu(from step 3), which may cause step 5 cuda oom.
chenhaiq
pushed a commit
to The-Hierophant/verl-1
that referenced
this pull request
Nov 18, 2025
… checkpoint (verl-project#3861) ### What does this PR do? Fix a bug introduce in verl-project#3285. Currently the initialization steps are: 1. **build actor**: init model and optimizer, then offload to cpu 2. **build rollout**: init model and kv cache 3. **switch to `trainer_mode`**: discard rollout weight and kv cache, load actor model and optimizer to gpu 4. **load_checkpoint**: if checkpoint exists, load model and optimizer from checkpoint, then offload to cpu 5. **switch to `rollout_mode`**: load actor model to gpu, resume rollout weight and sync weight, then offload actor model to cpu and resume kv cache 6. **generate_sequences** The bug is step 4: if checkpoint doesn't exists, then this step is skipped, leading to actor and optimizer both remain in gpu(from step 3), which may cause step 5 cuda oom.
NenoL2001
pushed a commit
to NenoL2001/verl
that referenced
this pull request
Nov 26, 2025
… checkpoint (verl-project#3861) ### What does this PR do? Fix a bug introduce in verl-project#3285. Currently the initialization steps are: 1. **build actor**: init model and optimizer, then offload to cpu 2. **build rollout**: init model and kv cache 3. **switch to `trainer_mode`**: discard rollout weight and kv cache, load actor model and optimizer to gpu 4. **load_checkpoint**: if checkpoint exists, load model and optimizer from checkpoint, then offload to cpu 5. **switch to `rollout_mode`**: load actor model to gpu, resume rollout weight and sync weight, then offload actor model to cpu and resume kv cache 6. **generate_sequences** The bug is step 4: if checkpoint doesn't exists, then this step is skipped, leading to actor and optimizer both remain in gpu(from step 3), which may cause step 5 cuda oom.
paolo328
added a commit
to paolo328/Verl
that referenced
this pull request
Nov 27, 2025
… checkpoint (#3861) ### What does this PR do? Fix a bug introduce in verl-project/verl#3285. Currently the initialization steps are: 1. **build actor**: init model and optimizer, then offload to cpu 2. **build rollout**: init model and kv cache 3. **switch to `trainer_mode`**: discard rollout weight and kv cache, load actor model and optimizer to gpu 4. **load_checkpoint**: if checkpoint exists, load model and optimizer from checkpoint, then offload to cpu 5. **switch to `rollout_mode`**: load actor model to gpu, resume rollout weight and sync weight, then offload actor model to cpu and resume kv cache 6. **generate_sequences** The bug is step 4: if checkpoint doesn't exists, then this step is skipped, leading to actor and optimizer both remain in gpu(from step 3), which may cause step 5 cuda oom.
TimurTaepov
pushed a commit
to giorgossideris/verl
that referenced
this pull request
Dec 20, 2025
…3285) ### What does this PR do? Deprecate all rollout sharding manager and replaced by `trainer_mode` and `rollout_mode` in hybrid worker.
TimurTaepov
pushed a commit
to giorgossideris/verl
that referenced
this pull request
Dec 20, 2025
… checkpoint (verl-project#3861) ### What does this PR do? Fix a bug introduce in verl-project#3285. Currently the initialization steps are: 1. **build actor**: init model and optimizer, then offload to cpu 2. **build rollout**: init model and kv cache 3. **switch to `trainer_mode`**: discard rollout weight and kv cache, load actor model and optimizer to gpu 4. **load_checkpoint**: if checkpoint exists, load model and optimizer from checkpoint, then offload to cpu 5. **switch to `rollout_mode`**: load actor model to gpu, resume rollout weight and sync weight, then offload actor model to cpu and resume kv cache 6. **generate_sequences** The bug is step 4: if checkpoint doesn't exists, then this step is skipped, leading to actor and optimizer both remain in gpu(from step 3), which may cause step 5 cuda oom.
vyomakesh0728
added a commit
to vyomakesh0728/verl
that referenced
this pull request
Jan 22, 2026
…3285) ### What does this PR do? Deprecate all rollout sharding manager and replaced by `trainer_mode` and `rollout_mode` in hybrid worker.
vyomakesh0728
added a commit
to vyomakesh0728/verl
that referenced
this pull request
Jan 22, 2026
… checkpoint (verl-project#3861) ### What does this PR do? Fix a bug introduce in verl-project#3285. Currently the initialization steps are: 1. **build actor**: init model and optimizer, then offload to cpu 2. **build rollout**: init model and kv cache 3. **switch to `trainer_mode`**: discard rollout weight and kv cache, load actor model and optimizer to gpu 4. **load_checkpoint**: if checkpoint exists, load model and optimizer from checkpoint, then offload to cpu 5. **switch to `rollout_mode`**: load actor model to gpu, resume rollout weight and sync weight, then offload actor model to cpu and resume kv cache 6. **generate_sequences** The bug is step 4: if checkpoint doesn't exists, then this step is skipped, leading to actor and optimizer both remain in gpu(from step 3), which may cause step 5 cuda oom.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Deprecate all rollout sharding manager and replaced by
trainer_modeandrollout_modein hybrid worker.