Let CUDA and ROCm read different loss result #2156
Conversation
|
Warning: Unknown label
Please add the new label to .github/pytorch-probot.yml |
meta-cla
bot
added
the
CLA Signed
|
Warning: Unknown label
Please add the new label to .github/pytorch-probot.yml |
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.12.0) (oldest at bottom): * #2156 * __->__ #2155 This assumes that the local built version has the same parent folder as torchtitan. Also fixes some pyrefly errors for moe.py
fegin
changed the title
| # after 5 steps. Leave fore AMD people to fix this if this is | ||
| # something bother users. | ||
| LOSS_FILE="tests/assets/losses/llama3_rocm.txt" | ||
| STEPS=5 |
There was a problem hiding this comment.
why setting it to 5 if you are already diverging the loss file?
There was a problem hiding this comment.
It's in the comment of the yaml and the summary of this PR.
The loss results of FSDP and HSDP start to diverge after 5th step when running with ROCm, we also need to adjust this. But this is more an unknown issue that AMD people may want to figure out the root cause or confirm that this is an expected behavior.
There was a problem hiding this comment.
oh I see, I missed that part.
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.12.0) (oldest at bottom): * __->__ #2157 CUDA and ROCm have different loss results. So we need to read from different loss result files. The loss results of FSDP and HSDP start to diverge after 5th step when running with ROCm, we also need to adjust this. But this is more an unknown issue that AMD people may want to figure out the root cause or confirm that this is an expected behavior. **This PR is a reland PR of #2156 due to some landing issue of the previous PR.
2 participants
Stack from ghstack (oldest at bottom):