coredump with RTX 4090 #33
Description
OS:
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
Kernel:
Linux 5.15.0-136-generic
Driver:
nvidia-smi
Wed Nov 19 10:49:58 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.153.02 Driver Version: 570.153.02 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 On | 00000000:67:00.0 Off | Off |
| 30% 34C P8 22W / 450W | 1MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 4090 On | 00000000:68:00.0 Off | Off |
| 30% 36C P8 29W / 450W | 1MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA GeForce RTX 4090 On | 00000000:6C:00.0 Off | Off |
| 30% 35C P8 31W / 450W | 1MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA GeForce RTX 4090 On | 00000000:6D:00.0 Off | Off |
| 30% 35C P8 26W / 450W | 1MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 4 NVIDIA GeForce RTX 4090 On | 00000000:E5:00.0 Off | Off |
| 30% 39C P8 29W / 450W | 1MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 5 NVIDIA GeForce RTX 4090 On | 00000000:E6:00.0 Off | Off |
| 31% 36C P8 27W / 450W | 1MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 6 NVIDIA GeForce RTX 4090 On | 00000000:E7:00.0 Off | Off |
| 30% 35C P8 23W / 450W | 1MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 7 NVIDIA GeForce RTX 4090 On | 00000000:E8:00.0 Off | Off |
| 30% 36C P8 21W / 450W | 1MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
coredump with command python legged_gym/scripts/train.py --task g1_ground --run_name test_g1 --headless
(cuda-gdb) target cudacore core_1763432930_i-57e8ydfp6v26xx7w2h682507e_3391243.nvcudmp
Opening GPU coredump: core_1763432930_i-57e8ydfp6v26xx7w2h682507e_3391243.nvcudmp
CUDA Exception: Warp Illegal Address
The exception was triggered at PC 0x7fdcf8dc3920
[Current focus set to CUDA kernel 0, grid 25714637, block (46,1,0), thread (0,1,0), device 0, sm 93, warp 1, lane 0]
#0 0x00007fdcf8dc3960 in artiPropagateVelocityTGS2<<<(64,2,1),(32,2,1)>>> ()
(cuda-gdb) bt
#0 0x00007fdcf8dc3960 in artiPropagateVelocityTGS2<<<(64,2,1),(32,2,1)>>> ()
(cuda-gdb)