[SPARK-32645][INFRA] Upload unit-tests.log as an artifact #29454
Conversation
| name: test-results-${{ matrix.modules }}-${{ matrix.comment }}-${{ matrix.java }}-${{ matrix.hadoop }}-${{ matrix.hive }} | ||
| path: "**/target/test-reports/*.xml" | ||
| - name: Upload unit tests log files | ||
| if: failure() |
There was a problem hiding this comment.
Zipped logfiles are ~300 MB if we collect all. Jenkins always uploads but GitHub Actions uploads failed tests only for now.
I can't find the clear answer of the size limit from my googling but seems like there's a limit internally. Should be best to save the resources.
There was a problem hiding this comment.
Logs for failed tests only should be good enough from my experience.
There was a problem hiding this comment.
With if: failure(), only the failed test logs are supposed to be uploaded, right?
This comment has been minimized.
This comment has been minimized.
|
cc @Ngone51 and @tgravescs FYI. This is being tested at https://github.com/apache/spark/actions/runs/213000777 |
Ngone51
left a comment
Ngone51
left a comment
There was a problem hiding this comment.
Thank you @HyukjinKwon for add this!!
| name: test-results-${{ matrix.modules }}-${{ matrix.comment }}-${{ matrix.java }}-${{ matrix.hadoop }}-${{ matrix.hive }} | ||
| path: "**/target/test-reports/*.xml" | ||
| - name: Upload unit tests log files | ||
| if: failure() |
There was a problem hiding this comment.
Logs for failed tests only should be good enough from my experience.
|
Should be ready for a look or possibly to be merged. |
|
Test build #127523 has finished for PR 29454 at commit
|
|
cc @dongjoon-hyun, @gengliangwang, @viirya FYI |
HyukjinKwon
force-pushed
the
SPARK-32645
branch
from
bd7ff11 to
876d0a4
Compare
HyukjinKwon
force-pushed
the
SPARK-32645
branch
from
876d0a4 to
d787cd1
Compare
|
I've downloaded the log from: https://github.com/apache/spark/actions/runs/213000777 and it looks good! LGTM |
|
Test build #127565 has finished for PR 29454 at commit
|
viirya
left a comment
viirya
left a comment
There was a problem hiding this comment.
Looks good. Do we know how long the artifacts will be kept?
|
GitHub keeps 90 days (https://docs.github.com/en/actions/configuring-and-managing-workflows/managing-a-workflow-run#viewing-logs-to-diagnose-failures) but I think there's another internal limit .. given my random googling. |
| if: failure() | ||
| uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: unit-tests-log-${{ matrix.modules }}-${{ matrix.comment }}-${{ matrix.java }}-${{ matrix.hadoop }}-${{ matrix.hive }} |
There was a problem hiding this comment.
nit though, adding a prefix jdk for matrix.java will make the log name clearer just like hadoop3.2 and hive2.3?
There was a problem hiding this comment.
Yup. I plan to have a proper name instead of combination of the matrix to address such issue clearly. I'll make a separate PR to address the issue of all such names in this whole file later.
|
Nice feature! |
|
Thanks guys. Merged to master. |
### What changes were proposed in this pull request? This PR proposes to upload `target/unit-tests.log` into the artifact so it will be able to download here:  ### Why are the changes needed? Jenkins has this feature. It should be best to have the same dev functionalities with it. Also, note that this was pointed out apache#29225 (comment). ### Does this PR introduce _any_ user-facing change? No, dev-only ### How was this patch tested? https://github.com/apache/spark/actions/runs/213000777 should demonstrate it Closes apache#29454 from HyukjinKwon/SPARK-32645. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
### What changes were proposed in this pull request? This PR proposes to upload `target/unit-tests.log` into the artifact so it will be able to download here:  ### Why are the changes needed? Jenkins has this feature. It should be best to have the same dev functionalities with it. Also, note that this was pointed out apache#29225 (comment). ### Does this PR introduce _any_ user-facing change? No, dev-only ### How was this patch tested? https://github.com/apache/spark/actions/runs/213000777 should demonstrate it Closes apache#29454 from HyukjinKwon/SPARK-32645. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
### What changes were proposed in this pull request? This PR proposes to upload `target/unit-tests.log` into the artifact so it will be able to download here:  ### Why are the changes needed? Jenkins has this feature. It should be best to have the same dev functionalities with it. Also, note that this was pointed out apache#29225 (comment). ### Does this PR introduce _any_ user-facing change? No, dev-only ### How was this patch tested? https://github.com/apache/spark/actions/runs/213000777 should demonstrate it Closes apache#29454 from HyukjinKwon/SPARK-32645. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
6 participants
What changes were proposed in this pull request?
This PR proposes to upload
target/unit-tests.loginto the artifact so it will be able to download here:Why are the changes needed?
Jenkins has this feature. It should be best to have the same dev functionalities with it.
Also, note that this was pointed out #29225 (comment).
Does this PR introduce any user-facing change?
No, dev-only
How was this patch tested?
https://github.com/apache/spark/actions/runs/213000777 should demonstrate it