Skip to content

[SPARK-8505][SparkR] Add settings to kick lint-r from ./dev/run-test.py #7883

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yu-iskw wants to merge 6 commits into from
Closed

[SPARK-8505][SparkR] Add settings to kick lint-r from ./dev/run-test.py #7883

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d654fc8
[SPARK-8505][SparkR] Add settings to kick `lint-r` from `./dev/run-te…
yu-iskw Aug 3, 2015
1b7fa13
Modify the place to checking R style
yu-iskw Aug 6, 2015
c981f83
Revert how to install lintr
yu-iskw Aug 6, 2015
fad3c95
Fix evaluationg the condition about the number of lint results
yu-iskw Aug 6, 2015
4c8da4d
Add a key to check the R style block
yu-iskw Aug 6, 2015
54365fc
Modify `dev/run-tests-codes.sh`
yu-iskw Aug 6, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions dev/lint-r
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,14 @@ if ! type "Rscript" > /dev/null; then
fi

`which Rscript` --vanilla "$SPARK_ROOT_DIR/dev/lint-r.R" "$SPARK_ROOT_DIR" | tee "$LINT_R_REPORT_FILE_NAME"

NUM_LINES=`wc -l "$LINT_R_REPORT_FILE_NAME"`
if [ $NUM_LINES -eq 0 ]; then
lint_status=0
echo "lintr checks passed."
else
lint_status=1
echo "lintr checks failed."
fi

exit "$lint_status"
17 changes: 10 additions & 7 deletions dev/lint-r.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,19 +17,22 @@

argv <- commandArgs(TRUE)
SPARK_ROOT_DIR <- as.character(argv[1])
LOCAL_LIB_LOC <- file.path(SPARK_ROOT_DIR, "R", "lib")

# Installs lintr from Github.
# Checks if SparkR is installed in a local directory.
if (! library(SparkR, lib.loc = LOCAL_LIB_LOC, logical.return = TRUE)) {
stop("You should install SparkR in a local directory with `R/install-dev.sh`.")
}

# Installs lintr from Github in a local directory.
# NOTE: The CRAN's version is too old to adapt to our rules.
if ("lintr" %in% row.names(installed.packages()) == FALSE) {
devtools::install_github("jimhester/lintr")
installed_packages <- row.names(installed.packages(lib.loc = LOCAL_LIB_LOC))
if ("lintr" %in% installed_packages == FALSE) {
devtools::with_lib(LOCAL_LIB_LOC, devtools::install_github("jimhester/lintr"))
}

library(lintr)
library(methods)
library(testthat)
if (! library(SparkR, lib.loc = file.path(SPARK_ROOT_DIR, "R", "lib"), logical.return = TRUE)) {
stop("You should install SparkR in a local directory with `R/install-dev.sh`.")
}

path.to.package <- file.path(SPARK_ROOT_DIR, "R", "pkg")
lint_package(path.to.package, cache = FALSE)
4 changes: 4 additions & 0 deletions dev/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -385,6 +385,10 @@ def run_sparkr_tests():

if which("R"):
run_cmd([os.path.join(SPARK_HOME, "R", "install-dev.sh")])
# R style check should be executed after `install-dev.sh`.
# Since warnings about `no visible global function definition` appear
# without the installation. SEE ALSO: SPARK-9121.
run_cmd([os.path.join(SPARK_HOME, "dev", "lint-r")])
Copy link
Contributor

@JoshRosen JoshRosen Aug 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we place the lint check in run_sparkr_tests then this may result in us running those lint checks after running other tests, which will make it take longer to discover R style failures in scenarios where non-R tests had to be run. As a result, I wonder if we should place this linting check near line 488, by the existing run_python_style_checks() call.

Copy link
Contributor

@shivaram shivaram Aug 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah thats a valid point, but the R lint tests have this problem where they don't resolve internal / private functions unless the corresponding package is included (i.e. SparkR in this case) . This is SPARK-9121 that is described in the comment.

FWIW I think we can do this right after the Maven build finishes as the SparkR package would have been built at that point -- so this will be before running Scala unit tests at least.

run_cmd([os.path.join(SPARK_HOME, "R", "run-tests.sh")])
else:
print("Ignoring SparkR tests as R was not found in PATH")
Expand Down