Skip to content

[examples] Add dynamic context parallel example#3892

Draft
ilml wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
ilml:codex/dynamic-cp-example
Draft

[examples] Add dynamic context parallel example#3892
ilml wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
ilml:codex/dynamic-cp-example

Conversation

@ilml
Copy link
Copy Markdown

@ilml ilml commented May 19, 2026

Summary

  • Add a minimal long-context example for Megatron-Core dynamic context parallelism (hybrid_context_parallel).
  • Show the Bridge config knobs, HybridCPDataLoaderWrapper, and a custom forward step that consumes per-sample local_cp_size.
  • Include a --dry-run scheduler preview and a tiny packed toy dataset for a 2-GPU smoke run.

Testing

  • uv run python -m py_compile examples/long_context/dynamic_context_parallel.py
  • RUFF_CACHE_DIR=/tmp/megatron_bridge_ruff_cache uv run ruff check examples/long_context/dynamic_context_parallel.py
  • RUFF_CACHE_DIR=/tmp/megatron_bridge_ruff_cache uv run ruff format --check examples/long_context/dynamic_context_parallel.py
  • RUFF_CACHE_DIR=/tmp/megatron_bridge_ruff_cache uv run pre-commit run --all-files
  • tmux window 0:2, Docker container 7af67331b54b: uv run python -m torch.distributed.run --master_port=29513 --nproc_per_node=2 examples/long_context/dynamic_context_parallel.py --eval-iters 1 (__EXIT_CODE:0__)

Signed-off-by: ilml <tolong@nvidia.com>
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented May 19, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ilml ilml marked this pull request as draft May 19, 2026 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant