cuda: enable checkpoint support for paused tasks #2517
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rst0git
force-pushed
the
2024-11-12-cuda-checkpointing-paused-tasks
branch
6 times, most recently
from
03b51f9 to
8f544eb
Compare
jesus-ramos
reviewed
Nov 12, 2024
rst0git
force-pushed
the
2024-11-12-cuda-checkpointing-paused-tasks
branch
from
8f544eb to
030bc9a
Compare
Member
|
Is there any real use-case for that? |
rst0git
force-pushed
the
2024-11-12-cuda-checkpointing-paused-tasks
branch
from
030bc9a to
58de16a
Compare
Member
Author
The use-case is similar to #2514 -- the CUDA tasks may be in a "locked" or "checkpointed" state before |
If a CUDA process is already in a "locked" or "checkpointed" state during criu dump, the CUDA plugin currently fails with an error because it attempts an unnecessary "lock" action using the cuda-checkpoint tool. This patch extends the CUDA plugin to handle such cases by first verifying the initial state of the CUDA processes and skipping unnecessary "lock" and "checkpoint" actions when a process has been locked or checkpointed before CRIU is invoked. In particular, CUDA tasks may already be in a "locked" or "checkpointed" state to ensure consistent checkpoint/restore for distributed workloads, such as model training, where multiple containers run across different cluster nodes. Another use case for this functionality is optimizing resource utilization, where CUDA tasks with low-priority are preempted immediately to release GPU resources needed by high-priority tasks, and the paused workloads are later resumed or migrated to another node. Signed-off-by: Radostin Stoyanov <[email protected]>
Signed-off-by: Radostin Stoyanov <[email protected]>
rst0git
force-pushed
the
2024-11-12-cuda-checkpointing-paused-tasks
branch
from
58de16a to
20a7cfa
Compare
Member
Author
|
@avagin I've updated the commit message with a brief description of the use-cases for this functionality. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If a CUDA process is already in a "locked" or "checkpointed" state during
criu dump, the CUDA plugin fails with an error because it attempts an unnecessary "lock" action using thecuda-checkpointtool. This pull request extends the CUDA plugin to handle such case by first checking the original state of the CUDA process and skipping unnecessary "lock" and "checkpoint" actions if the process was already locked or checkpointed before CRIU was invoked.