feat(fault-injection): Add Python client library for fault injection API #4048
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- FaultInjectionClient class with high-level API - GPU fault injection and XID error methods - Network partition injection (NetworkPolicy and ChaosMesh) - Context managers for automatic cleanup - Kubernetes integration for pod discovery Provides clean interface for fault injection in test scripts.
nnshah1
reviewed
Nov 19, 2025
| # DCGM Infrastructure Management | ||
| # ======================================================================== | ||
|
|
||
| def deploy_dcgm( |
Contributor
There was a problem hiding this comment.
can we turn this into a check instead of a deploy? IIUC dcgm should already be in the cluster / naemspace
nnshah1
reviewed
Nov 19, 2025
| # Fault Recovery | ||
| # ======================================================================== | ||
|
|
||
| def recover_fault(self, fault_id: str) -> dict[str, Any]: |
Contributor
There was a problem hiding this comment.
not sure what recover means here -
nnshah1
reviewed
Nov 19, 2025
Contributor
nnshah1
left a comment
nnshah1
left a comment
There was a problem hiding this comment.
basic question: what can we do from the client without an api server and agent? the agent is needed to write into the kmesg log is that right? Do we need the api server or could we run all the operations from the test runner (just thinking) -
We could define fault types and move the logic to the common fault types folder ...
nnshah1
approved these changes
Nov 24, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview:
This PR adds a Python client library providing high-level API for fault injection. Handles GPU faults, network partitions, Kubernetes integration, and automatic cleanup via context managers.
Details:
client/fault_injection_client.py(798 lines): Main client libraryCore components:
GPUFault,NetworkPartition,NetworkMode,FaultSeverityFaultInfo,MetricsFaultInjectionClient: Main client with comprehensive APIGPU fault injection:
inject_gpu_fault(): Generic GPU fault injectioninject_xid_error(): XID-specific injection (79, 48, 94, 95, 43, 74)Network fault injection:
inject_network_partition(): Main network fault methodNETWORKPOLICY(complete blocking),CHAOS_MESH(packet loss, delay)FRONTEND_WORKER,WORKER_NATS,WORKER_WORKER,CUSTOMFault management:
get_fault_status(): Query current statusdelete_fault(): Remove fault and cleanupwait_for_recovery(): Wait for healthy stateContext managers:
gpu_fault(): Auto-cleanup GPU faultsnetwork_partition(): Auto-cleanup network faultsWhere should the reviewer start?
inject_network_partition()- Main entry pointKey patterns:
Related Issues: