Skip to content

[SPARK-32249][INFRA][3.0] Run Github Actions builds in branch-3.0 #29460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 18 commits into from
Closed

[SPARK-32249][INFRA][3.0] Run Github Actions builds in branch-3.0 #29460

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
9d99de0
[SPARK-32245][INFRA] Run Spark tests in Github Actions
HyukjinKwon Jul 11, 2020
dc96207
[SPARK-32245][INFRA][FOLLOWUP] Reenable Github Actions on commit
dongjoon-hyun Jul 12, 2020
d4726dd
[SPARK-32292][SPARK-32252][INFRA] Run the relevant tests only in GitH…
HyukjinKwon Jul 13, 2020
3845400
[SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Ac…
HyukjinKwon Jul 15, 2020
4083d5e
[SPARK-32408][BUILD] Enable crossPaths back to prevent side effects
HyukjinKwon Jul 24, 2020
5624db3
[SPARK-32303][PYTHON][TESTS] Remove leftover from editable mode insta…
HyukjinKwon Jul 14, 2020
b8fc422
[SPARK-32363][PYTHON][BUILD] Fix flakiness in pip package testing in …
HyukjinKwon Jul 21, 2020
130a9d0
[SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)a…
HyukjinKwon Jul 25, 2020
3f1acea
[SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6…
HyukjinKwon Jul 25, 2020
78b5833
[SPARK-32491][INFRA] Do not install SparkR in test-only mode in testi…
HyukjinKwon Jul 30, 2020
32a5fee
[SPARK-32493][INFRA] Manually install R instead of using setup-r in G…
HyukjinKwon Jul 30, 2020
5c23116
[SPARK-32496][INFRA] Include GitHub Action file as the changes in tes…
HyukjinKwon Jul 30, 2020
a278c5e
[SPARK-32497][INFRA] Installs qpdf package for CRAN check in GitHub A…
HyukjinKwon Jul 30, 2020
11d170f
[SPARK-32357][INFRA] Publish failed and succeeded test reports in Git…
HyukjinKwon Aug 14, 2020
2fc2b82
[SPARK-32606][SPARK-32605][INFRA] Remove the forks of action-surefire…
HyukjinKwon Aug 17, 2020
2366f37
[SPARK-32248][BUILD] Recover Java 11 build in Github Actions
dongjoon-hyun Jul 30, 2020
fbb4ac8
[MINOR][INFRA] Rename master.yml to build_and_test.yml
HyukjinKwon Aug 18, 2020
d029dba
[SPARK-32645][INFRA] Upload unit-tests.log as an artifact
HyukjinKwon Aug 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[SPARK-32408][BUILD] Enable crossPaths back to prevent side effects 
### What changes were proposed in this pull request?

This PR proposes to enable `corssPaths` back for now to match with the build as it was.
It still indeterministically doesn't run JUnit tests given my observation, and this PR basically reverts the partial fix from #29057.

See also #29205 for the full context.

### Why are the changes needed?

To prevent the side effects from crossPaths such as SPARK_PREPEND_CLASSES or tests that run conditionally if the test classes are present in PySpark.

### Does this PR introduce _any_ user-facing change?

No, dev-only.

### How was this patch tested?

Manually tested:

```bash
build/sbt -Phadoop-2.7 -Phive -Phive-2.3 -Phive-thriftserver -DskipTests clean test:package
./python/run-tests --python-executable=python3 --testname="pyspark.sql.tests.test_dataframe QueryExecutionListenerTests"
```

Closes #29218 from HyukjinKwon/SPARK-32408-1.

Authored-by: HyukjinKwon <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
@HyukjinKwon
HyukjinKwon committed Aug 19, 2020
commit 4083d5e59d6c8f655fbc4dc06ef940ce69978f3f
2 changes: 0 additions & 2 deletions project/SparkBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1029,8 +1029,6 @@ object TestSettings {
// Show full stack trace and duration in test cases.
testOptions in Test += Tests.Argument("-oDF"),
testOptions in Test += Tests.Argument(TestFrameworks.JUnit, "-v", "-a"),
// Required to detect Junit tests for each project, see also https://github.com/sbt/junit-interface/issues/35
crossPaths := false,
// Enable Junit testing.
libraryDependencies += "com.novocode" % "junit-interface" % "0.11" % "test",
// `parallelExecutionInTest` controls whether test suites belonging to the same SBT project
Expand Down