Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
9d99de0
[SPARK-32245][INFRA] Run Spark tests in Github Actions
HyukjinKwon Jul 11, 2020
dc96207
[SPARK-32245][INFRA][FOLLOWUP] Reenable Github Actions on commit
dongjoon-hyun Jul 12, 2020
d4726dd
[SPARK-32292][SPARK-32252][INFRA] Run the relevant tests only in GitH…
HyukjinKwon Jul 13, 2020
3845400
[SPARK-32316][TESTS][INFRA] Test PySpark with Python 3.8 in Github Ac…
HyukjinKwon Jul 15, 2020
4083d5e
[SPARK-32408][BUILD] Enable crossPaths back to prevent side effects
HyukjinKwon Jul 24, 2020
5624db3
[SPARK-32303][PYTHON][TESTS] Remove leftover from editable mode insta…
HyukjinKwon Jul 14, 2020
b8fc422
[SPARK-32363][PYTHON][BUILD] Fix flakiness in pip package testing in …
HyukjinKwon Jul 21, 2020
130a9d0
[SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)a…
HyukjinKwon Jul 25, 2020
3f1acea
[SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6…
HyukjinKwon Jul 25, 2020
78b5833
[SPARK-32491][INFRA] Do not install SparkR in test-only mode in testi…
HyukjinKwon Jul 30, 2020
32a5fee
[SPARK-32493][INFRA] Manually install R instead of using setup-r in G…
HyukjinKwon Jul 30, 2020
5c23116
[SPARK-32496][INFRA] Include GitHub Action file as the changes in tes…
HyukjinKwon Jul 30, 2020
a278c5e
[SPARK-32497][INFRA] Installs qpdf package for CRAN check in GitHub A…
HyukjinKwon Jul 30, 2020
11d170f
[SPARK-32357][INFRA] Publish failed and succeeded test reports in Git…
HyukjinKwon Aug 14, 2020
2fc2b82
[SPARK-32606][SPARK-32605][INFRA] Remove the forks of action-surefire…
HyukjinKwon Aug 17, 2020
2366f37
[SPARK-32248][BUILD] Recover Java 11 build in Github Actions
dongjoon-hyun Jul 30, 2020
fbb4ac8
[MINOR][INFRA] Rename master.yml to build_and_test.yml
HyukjinKwon Aug 18, 2020
d029dba
[SPARK-32645][INFRA] Upload unit-tests.log as an artifact
HyukjinKwon Aug 19, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[SPARK-32497][INFRA] Installs qpdf package for CRAN check in GitHub A…
…ctions

### What changes were proposed in this pull request?

CRAN check fails due to the size of the generated PDF docs as below:

```
...
 WARNING
‘qpdf’ is needed for checks on size reduction of PDFs
...
Status: 1 WARNING, 1 NOTE
See
  ‘/home/runner/work/spark/spark/R/SparkR.Rcheck/00check.log’
for details.
```

This PR proposes to install `qpdf` in GitHub Actions.

Note that I cannot reproduce in my local with the same R version so I am not documenting it for now.

Also, while I am here, I piggyback to install SparkR when the module includes `sparkr`. it is rather a followup of SPARK-32491.

### Why are the changes needed?

To fix SparkR CRAN check failure.

### Does this PR introduce _any_ user-facing change?

No, dev-only.

### How was this patch tested?

GitHub Actions will test it out.

Closes #29306 from HyukjinKwon/SPARK-32497.

Authored-by: HyukjinKwon <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
  • Loading branch information
HyukjinKwon committed Aug 19, 2020
commit a278c5eac87a0cbc577f002575e8c931bf3b8f52
3 changes: 2 additions & 1 deletion .github/workflows/master.yml
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,8 @@ jobs:
- name: Install R packages
if: contains(matrix.modules, 'sparkr')
run: |
sudo apt-get install -y libcurl4-openssl-dev
# qpdf is required to reduce the size of PDFs to make CRAN check pass. See SPARK-32497.
sudo apt-get install -y libcurl4-openssl-dev qpdf
sudo Rscript -e "install.packages(c('knitr', 'rmarkdown', 'testthat', 'devtools', 'e1071', 'survival', 'arrow', 'roxygen2'), repos='https://cloud.r-project.org/')"
# Show installed packages in R.
sudo Rscript -e 'pkg_list <- as.data.frame(installed.packages()[, c(1,3:4)]); pkg_list[is.na(pkg_list$Priority), 1:2, drop = FALSE]'
Expand Down
16 changes: 9 additions & 7 deletions dev/run-tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -609,8 +609,14 @@ def main():

# Install SparkR
should_only_test_modules = opts.modules is not None
if not should_only_test_modules:
test_modules = []
if should_only_test_modules:
str_test_modules = [m.strip() for m in opts.modules.split(",")]
test_modules = [m for m in modules.all_modules if m.name in str_test_modules]

if not should_only_test_modules or modules.sparkr in test_modules:
# If tests modules are specified, we will not run R linter.
# SparkR needs the manual SparkR installation.
if which("R"):
run_cmd([os.path.join(SPARK_HOME, "R", "install-dev.sh")])
else:
Expand Down Expand Up @@ -641,15 +647,11 @@ def main():
"and Hive profile", hive_version, "under environment", test_env)
extra_profiles = get_hadoop_profiles(hadoop_version) + get_hive_profiles(hive_version)

changed_modules = None
test_modules = None
changed_files = None
changed_modules = []
changed_files = []
included_tags = []
excluded_tags = []
if should_only_test_modules:
str_test_modules = [m.strip() for m in opts.modules.split(",")]
test_modules = [m for m in modules.all_modules if m.name in str_test_modules]

# If we're running the tests in Github Actions, attempt to detect and test
# only the affected modules.
if test_env == "github_actions":
Expand Down