Skip to content

HyperPod Slurm observability improvement #857

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shimomut merged 30 commits into from
Sep 16, 2025
Merged

HyperPod Slurm observability improvement #857

Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
57f9678
Starting development wth manual installation
shimomut Aug 19, 2025
83d3eb1
Adding install_efa_exporter
shimomut Aug 19, 2025
8456e8b
Rename container
shimomut Aug 19, 2025
38a0035
Adding node exporter
shimomut Aug 19, 2025
261fb0c
Adding otel-collector
shimomut Aug 20, 2025
49ee0a1
Added installation for head node
shimomut Aug 20, 2025
a64e61a
organize make targets
shimomut Aug 21, 2025
554f5fc
Starting to use Python as the main observability installation script
shimomut Aug 22, 2025
b26ed22
Create OTEL config from template
shimomut Aug 22, 2025
d318aa9
Adding Python script to stop all
shimomut Aug 22, 2025
813403b
Adding initial version of dashboard jsons
shimomut Aug 22, 2025
13a0e84
Pass REGION variable from main script
shimomut Aug 22, 2025
bb75ac1
Get region from cluster ARN automatically
shimomut Aug 22, 2025
8b4008f
Updating lifecycle_script.sh to use install_observability.py
shimomut Aug 22, 2025
b4d8b36
Deleting legacy observability scripts
shimomut Aug 22, 2025
1c7406c
Feed additional arguments when advanced mode
shimomut Aug 22, 2025
15df85e
Typo
shimomut Aug 22, 2025
6e02701
Use https://github.com/SckyzO/slurm_exporter.git instead of https://g…
shimomut Aug 26, 2025
665c198
Adding Dashboard Custom Resource for Slurm Observability template (#833)
Madhubalasri-B Aug 28, 2025
a8dfd03
Fixing parameters for template (#843)
Madhubalasri-B Sep 10, 2025
5b337db
Deleting dashboard files because now CF template contains them.
shimomut Sep 10, 2025
efe6e2a
Updating FSX and EFA dashboard metrics (#848)
Madhubalasri-B Sep 10, 2025
d3394cd
Updating EFA, Node and FSx metrics for slurm (#853)
Madhubalasri-B Sep 15, 2025
fa9c472
Add region name suffix to the S3 bucket
shimomut Sep 15, 2025
aaf5132
Updating Node Dashboard JSON (#854)
Madhubalasri-B Sep 16, 2025
a30b8bd
Deleting unnecessary files for publishing
shimomut Sep 16, 2025
ca220a1
Updating comment lines in config.py
shimomut Sep 16, 2025
ade4bf9
Updating Node Dashboard JSON
Madhubalasri-B Sep 16, 2025
1ed1b1d
Updating Node Dashboard JSON (#855)
Madhubalasri-B Sep 16, 2025
e512aea
Updating Node dashboard for instances (#856)
Madhubalasri-B Sep 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Rename container
  • Loading branch information
@shimomut
shimomut committed Aug 19, 2025
commit 8456e8b1ce3ed3c0278e6dceb699308a5dfbd573
4 changes: 2 additions & 2 deletions ...s/5.sagemaker-hyperpod/LifecycleScripts/base-config/observability/install_efa_exporter.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
REGION=us-west-2

# Define the container name
CONTAINER_NAME="efa-node-exporter"
CONTAINER_NAME="efa-exporter"

ECR_ACCOUNT_ID=602401143452

@nghtm nghtm Sep 16, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hard coded account ID

@shimomut shimomut Sep 16, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is EKS add-on's ECR repo - officially documented here:
https://docs.aws.amazon.com/eks/latest/userguide/add-ons-images.html

I can add comment to clarify this point.

VERSION=1.0.0
Expand Down Expand Up @@ -56,7 +56,7 @@ if sudo docker run -d --restart always \
-v "/:/host:ro,rslave" \
$IMAGE \
--path.rootfs=/host; then
echo "Successfully started EFA Node Exporter on node"
echo "Successfully started EFA Exporter on node"
exit 0
else
echo "Failed to run Docker container"
Expand Down