Skip to content

[SPARK-32497][INFRA] Installs qpdf package for CRAN check in GitHub Actions #29306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 4 commits into from
Closed

[SPARK-32497][INFRA] Installs qpdf package for CRAN check in GitHub Actions #29306

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7722de6
Installs qpdf package for CRAN check in GitHub Actions
HyukjinKwon Jul 30, 2020
21d6bfc
Tweak run-tests.py script
HyukjinKwon Jul 30, 2020
772f710
Show qpdf still is the cause
HyukjinKwon Jul 30, 2020
8ff181e
Revert "Show qpdf still is the cause"
HyukjinKwon Jul 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/master.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,8 @@ jobs:
- name: Install R packages
if: contains(matrix.modules, 'sparkr')
run: |
sudo apt-get install -y libcurl4-openssl-dev
# qpdf is required to reduce the size of PDFs to make CRAN check pass. See SPARK-32497.
sudo apt-get install -y libcurl4-openssl-dev qpdf
sudo Rscript -e "install.packages(c('knitr', 'rmarkdown', 'testthat', 'devtools', 'e1071', 'survival', 'arrow', 'roxygen2'), repos='https://cloud.r-project.org/')"
# Show installed packages in R.
sudo Rscript -e 'pkg_list <- as.data.frame(installed.packages()[, c(1,3:4)]); pkg_list[is.na(pkg_list$Priority), 1:2, drop = FALSE]'
Expand Down
16 changes: 9 additions & 7 deletions dev/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -610,8 +610,14 @@ def main():

# Install SparkR
should_only_test_modules = opts.modules is not None
if not should_only_test_modules:
test_modules = []
if should_only_test_modules:
str_test_modules = [m.strip() for m in opts.modules.split(",")]
test_modules = [m for m in modules.all_modules if m.name in str_test_modules]

if not should_only_test_modules or modules.sparkr in test_modules:
Copy link
Member Author

@HyukjinKwon HyukjinKwon Jul 30, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I piggyback this change together. We don't need to install SparkR when --modules is specified except that it includes sparkr.

# If tests modules are specified, we will not run R linter.
# SparkR needs the manual SparkR installation.
if which("R"):
run_cmd([os.path.join(SPARK_HOME, "R", "install-dev.sh")])
else:
Expand Down Expand Up @@ -642,15 +648,11 @@ def main():
"and Hive profile", hive_version, "under environment", test_env)
extra_profiles = get_hadoop_profiles(hadoop_version) + get_hive_profiles(hive_version)

changed_modules = None
test_modules = None
changed_files = None
changed_modules = []
changed_files = []
included_tags = []
excluded_tags = []
if should_only_test_modules:
str_test_modules = [m.strip() for m in opts.modules.split(",")]
test_modules = [m for m in modules.all_modules if m.name in str_test_modules]

# If we're running the tests in Github Actions, attempt to detect and test
# only the affected modules.
if test_env == "github_actions":
Expand Down