Add airgap packaging for Nemotron Customizer#194
Open
rapaul-nv wants to merge 5 commits into
Open
Conversation
- Add deploy-scoped airgap tooling for Nemotron Customizer steps under src/nemotron/steps. - Build a portable submitter image plus deduplicated task images for selected workflow targets. - Expand step dependencies and map selected steps to task image families through a single airgap.yaml. - Discover small task-image Python dependency gaps and bake pinned repo overlays required by step configs. - Models, datasets, checkpoints, and customer data to be kept in external persistent storage by user - Add resumable build state, image manifests with checksums, Dockerfiles, SFT overlay configs, README guidance, and focused tests. Signed-off-by: Rakesh Paul <rapaul@nvidia.com>
- Rename airgap artifacts to use launcher and execution image terminology - Update runner stages, manifests, README, and config keys to match the new naming - Keep execution image generation scoped to selected Nemotron Customizer steps - Preserve external handling for models, datasets, checkpoints, and customer storage paths - Refresh SFT Megatron Bridge airgap overlay configs - Update tests for launcher/execution image behavior and staged runner flow Signed-off-by: Rakesh Paul <rapaul@nvidia.com>
- Install git and CA certificates in the launcher image before uv sync - Capture only docker inspect stdout while suppressing stderr during platform checks - Keep the airgap runner platform probe compatible with subprocess stderr handling Signed-off-by: Rakesh Paul <rapaul@nvidia.com>
rapaul-nv
force-pushed
the
rapaul/airgap-support
branch
from
c7dedb0 to
4c02460
Compare
- Move generic step commands and backends from commands/step to commands/steps - Register only `nemotron steps`; remove the singular `nemotron step` alias - Expose `steps list`, `steps show`, `steps run`, and `steps translation` - Update imports, tests, docs, skills, and config examples to the plural CLI - Add coverage for plural command registration and singular alias rejection Signed-off-by: Rakesh Paul <rapaul@nvidia.com>
Signed-off-by: Rakesh Paul <rapaul@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a lightweight airgap packaging flow for Nemotron Customizer, scoped to
src/nemotron/steps.It introduces a deploy-side runner under
deploy/nemotron-customizer/airgap/that can package the selected workflow steps into:Models, datasets, checkpoints, and customer data are intentionally not baked into the images. They remain external assets expected to live in customer-managed persistent storage and are referenced through runtime configs.
Changes
deploy/nemotron-customizer/airgapairgap.yamlto describe selectable workflow stages and execution image groupsValidation
uv run pytest tests/deploy/test_airgap_runner.pyResult:
17 passed