HyperPod Slurm observability improvement by shimomut · Pull Request #857 · awslabs/awsome-distributed-ai

shimomut · 2025-09-16T05:10:52Z

Description of changes:

Updating the CloudFormation template for the HyperPod Slurm observability stack. Now it automatically register Prometheus as the data-source for Grafana, and installs pre-configured dashboard to Grafana.
Updating lifecycle script to install metric exporters and OTEL collector in each node - more scalable architecture similar to the HyperPod EKS Observability.
Added "ObservabilityConfig" class in config.py. It has Prometheus Remote Write URL and advanced flag.

For more details - https://catalog.workshops.aws/sagemaker-hyperpod/en-US/09-observability

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…ithub.com/vpenso/prometheus-slurm-exporter.git

* Adding Dashboard Custom Resource for Slurm Observability template * Fixing parameters for template

* Fixing parameters for template * Updating observability custom resource lambda * Updating json file for exporter metrics

* Updating EFA, Node and FSx metrics for slurm * Updating OTEL Config, Adding cfn params for observability template, updating dashboard metrics * Updating zip for dashboard

* Updating Node Dashboard JSON * Updating Node Dashboard JSON

nghtm

Concerns about several hard coded resources, adjustments made to docker installation in LCS

nghtm · 2025-09-16T14:30:47Z

+class ObservabilityConfig:
+
+    # Prometheus remote write URL
+    prometheus_remote_write_url = "https://aps-workspaces.us-west-2.amazonaws.com/workspaces/ws-abcd1234-abcd-1234-ab12-1234abcd1234/api/v1/remote_write"


Why is this hardcoded to an AMP URL?!

This is a placeholder URL (used dummy ID with 1234, abcd, etc). Customer needs to replace with their URL. I can add additional comment to explain that.

nghtm · 2025-09-16T14:31:44Z

+
+            from config import ObservabilityConfig
+
+            ExecuteBashScript("./utils/install_docker.sh").run()


Docker should be installed on all SMHP nodes, regardless of if customer enables observability or not.

"install_docker.sh" is called when "enable_docker_enroot_pyxis" is True or "enable_observability" is True, not always. (This PR is not changing this condition.)

nghtm · 2025-09-16T14:35:36Z

@@ -0,0 +1,47 @@
+# Clocks


Do we need to create and maintain a new file here, or can we just use:

https://github.com/aws-samples/awsome-distributed-training/blob/main/4.validation_and_observability/4.prometheus-grafana/dcgm-metrics.csv

Wanted to put these two configuration files (basic and advanced) side by side in a single place, and download together with lifecycle script itself, just to avoid additional HTTP download. We can delete this file once the full production version of solution comes. But please let me know if this is a blocker.

nghtm · 2025-09-16T14:35:58Z

-IMAGE="nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_VERSION}"
-METRICS_CSV_URL="https://raw.githubusercontent.com/aws-samples/awsome-distributed-training/refs/heads/main/4.validation_and_observability/4.prometheus-grafana/dcgm-metrics.csv"
-METRICS_CSV_PATH="/tmp/dcgm-metrics.csv"
+ECR_ACCOUNT_ID=602401143452


hard coded account ID?

This is EKS add-on's ECR repo - officially documented here:
https://docs.aws.amazon.com/eks/latest/userguide/add-ons-images.html

I can add comment to clarify this point.

nghtm · 2025-09-16T14:36:59Z

+# Define the container name
+CONTAINER_NAME="efa-exporter"
+
+ECR_ACCOUNT_ID=602401143452


Hard coded account ID

This is EKS add-on's ECR repo - officially documented here:
https://docs.aws.amazon.com/eks/latest/userguide/add-ons-images.html

I can add comment to clarify this point.

nghtm · 2025-09-16T14:37:16Z

+# Define the container name
+CONTAINER_NAME="node-exporter"
+
+ECR_ACCOUNT_ID=602401143452


Hard coded account ID

This is EKS add-on's ECR repo - officially documented here:
https://docs.aws.amazon.com/eks/latest/userguide/add-ons-images.html

I can add comment to clarify this point.

nghtm · 2025-09-16T14:37:31Z

-IMAGE="public.ecr.aws/hpc-cloud/efa-node-exporter:latest"
+CONTAINER_NAME="otel-collector"
+
+ECR_ACCOUNT_ID=602401143452


Hard coded account ID

This is EKS add-on's ECR repo - officially documented here:
https://docs.aws.amazon.com/eks/latest/userguide/add-ons-images.html

I can add comment to clarify this point.

nghtm

Approving as long as comments are resolved. Requesting future PRs for LCS be opened against new LCS repo being maintained by Mark Yu's team

Description of changes: - Updating the CloudFormation template for the HyperPod Slurm observability stack. Now it automatically register Prometheus as the data-source for Grafana, and installs pre-configured dashboard to Grafana. - Updating lifecycle script to install metric exporters and OTEL collector in each node - more scalable architecture similar to the HyperPod EKS Observability. - Added "ObservabilityConfig" class in config.py. It has Prometheus Remote Write URL and advanced flag. For more details - https://catalog.workshops.aws/sagemaker-hyperpod/en-US/09-observability --------- Co-authored-by: Madhubalasri-B <madbal@amazon.com>

* Removed inclusion of nsight * Boilerplate settings * Added nsys profile for BioNemo * Updated twith comments from PR Signed-off-by: Ankur Srivastava <awsankur@amazon.com> * Fix/readme - fix borked logo (#356) * Change readme to refer to recent test cases & assets * Remove borked ascii logo * Deleting SMP/SMDDP test-cases based on PM's input (#617) * HyperPod Slurm observability improvement (#857) Description of changes: - Updating the CloudFormation template for the HyperPod Slurm observability stack. Now it automatically register Prometheus as the data-source for Grafana, and installs pre-configured dashboard to Grafana. - Updating lifecycle script to install metric exporters and OTEL collector in each node - more scalable architecture similar to the HyperPod EKS Observability. - Added "ObservabilityConfig" class in config.py. It has Prometheus Remote Write URL and advanced flag. For more details - https://catalog.workshops.aws/sagemaker-hyperpod/en-US/09-observability --------- Co-authored-by: Madhubalasri-B <madbal@amazon.com> * Add optional Training Plan support for HyperPod instance groups * Standardize instance_groups to list objects with optional training plan ARN * chore: revert unrelated regressions from bad rebase --------- Signed-off-by: Ankur Srivastava <awsankur@amazon.com> Co-authored-by: Pierre-Yves Aquilanti <pierreya@amazon.com> Co-authored-by: Verdi March <marcverd@amazon.com> Co-authored-by: Ankur Srivastava <awsankur@amazon.com> Co-authored-by: Pierre-Yves <62559210+perifaws@users.noreply.github.com> Co-authored-by: Tomonori Shimomura <59209764+shimomut@users.noreply.github.com> Co-authored-by: Madhubalasri-B <madbal@amazon.com> Co-authored-by: newabdosheham <ubuntu@L007313.mbzu.ae>

shimomut and others added 30 commits August 19, 2025 23:02

Starting development wth manual installation

57f9678

Adding install_efa_exporter

83d3eb1

Rename container

8456e8b

Adding node exporter

38a0035

Adding otel-collector

261fb0c

Added installation for head node

49ee0a1

organize make targets

a64e61a

Starting to use Python as the main observability installation script

554f5fc

Create OTEL config from template

b26ed22

Adding Python script to stop all

d318aa9

Adding initial version of dashboard jsons

813403b

Pass REGION variable from main script

13a0e84

Get region from cluster ARN automatically

bb75ac1

Updating lifecycle_script.sh to use install_observability.py

8b4008f

Deleting legacy observability scripts

b4d8b36

Feed additional arguments when advanced mode

1c7406c

Typo

15df85e

Use https://github.com/SckyzO/slurm_exporter.git instead of https://g…

6e02701

…ithub.com/vpenso/prometheus-slurm-exporter.git

Adding Dashboard Custom Resource for Slurm Observability template (#833)

665c198

* Adding Dashboard Custom Resource for Slurm Observability template * Fixing parameters for template

Fixing parameters for template (#843)

a8dfd03

* Fixing parameters for template * Updating observability custom resource lambda * Updating json file for exporter metrics

Deleting dashboard files because now CF template contains them.

5b337db

Updating FSX and EFA dashboard metrics (#848)

efe6e2a

Updating EFA, Node and FSx metrics for slurm (#853)

d3394cd

* Updating EFA, Node and FSx metrics for slurm * Updating OTEL Config, Adding cfn params for observability template, updating dashboard metrics * Updating zip for dashboard

Add region name suffix to the S3 bucket

fa9c472

Updating Node Dashboard JSON (#854)

aaf5132

Deleting unnecessary files for publishing

a30b8bd

Updating comment lines in config.py

ca220a1

Updating Node Dashboard JSON

ade4bf9

Updating Node Dashboard JSON (#855)

1ed1b1d

* Updating Node Dashboard JSON * Updating Node Dashboard JSON

Updating Node dashboard for instances (#856)

e512aea

shimomut requested a review from nghtm September 16, 2025 05:11

nghtm reviewed Sep 16, 2025

View reviewed changes

nghtm approved these changes Sep 16, 2025

View reviewed changes

shimomut merged commit 27a8033 into main Sep 16, 2025
4 checks passed

shimomut deleted the hpslurm-observability-20250819 branch September 16, 2025 18:17


		from config import ObservabilityConfig

		ExecuteBashScript("./utils/install_docker.sh").run()

Uh oh!

Conversation

shimomut commented Sep 16, 2025

Uh oh!

nghtm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shimomut Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shimomut Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shimomut Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shimomut Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shimomut Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nghtm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shimomut Sep 16, 2025 •

edited

Loading

shimomut Sep 16, 2025 •

edited

Loading

shimomut Sep 16, 2025 •

edited

Loading

shimomut Sep 16, 2025 •

edited

Loading

shimomut Sep 16, 2025 •

edited

Loading