Skip to content

[WIP][SPARK-7018][Build]: Refactor dev/run-tests-jenkins into Python #7401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
brennonyork wants to merge 34 commits into from
Closed

[WIP][SPARK-7018][Build]: Refactor dev/run-tests-jenkins into Python #7401

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
31b51de
initial cut of refactored run-tests-jenkins script into python
Jul 13, 2015
f2a1dc6
fixed pep8 issues
Jul 14, 2015
d202c42
fixed list bug when finding log files
Jul 14, 2015
3ae1d49
correctly escape newlines for long strings
Jul 14, 2015
2a3d67d
fixed numerous bugs, updated to use urllib2 over curl, removed 'find'…
Jul 15, 2015
d83c535
fixed typos, style, and linting issues
Jul 15, 2015
3acac90
fixed indenting with list
Jul 15, 2015
395928d
fixed typo
Jul 15, 2015
30f90bc
fixed merge conflicts
Jul 15, 2015
4765dc8
added variables to remove indents, fixed ' : ' => ': ' in maps
Jul 15, 2015
e90fdeb
moved if statement for urllib2 into the try block
Jul 15, 2015
64c13c1
updated token passing for github oauth
Jul 15, 2015
85aaf76
added subprocess_check_call() as backport from python 2.7, moved subp…
Jul 15, 2015
f756098
removed unnecessary escapes
Jul 15, 2015
39f6a1f
changed test process to only return the returncode
Jul 15, 2015
7b11665
fixed serialization error between int and string, created an error re…
Jul 15, 2015
cf110bc
fixed reference error and pep8 bugs
Jul 15, 2015
c9e5c5f
commented out tests and fixed test result output
Jul 16, 2015
5272381
printing of the pr results
Jul 16, 2015
528279a
fixed strip issue with PR results, from strip => rstrip
Jul 16, 2015
6a9ed32
run-tests-jenkins now properly runs the tests, removed license from f…
Jul 16, 2015
22f9be8
reverted license
Jul 16, 2015
5d63bc5
removed mima to fail binary compatibility checks
Jul 16, 2015
33e3143
reverted mima excludes
Jul 16, 2015
d82567a
set accumulator test suite to fail
Jul 16, 2015
2117c64
reverted accumulator test in favor of graphx test to only trigger 'gr…
Jul 16, 2015
831376e
fixed scala style check
Jul 16, 2015
d76f559
fixed scala style checks
Jul 16, 2015
05d9ef4
reverted scala test failure, added pyspark failure
Jul 16, 2015
0ad51b4
updated to add --absolute-names with tar command for archiving logs
Jul 27, 2015
a218baf
reverting python test
Aug 4, 2015
2853b60
fixed merge conflicts
Oct 16, 2015
fe397fb
updated to account for PR #7883
Oct 16, 2015
206930e
removed 'send_archived_logs' function
Oct 16, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixed reference error and pep8 bugs
  • Loading branch information
Brennon York committed Jul 15, 2015
commit cf110bcfacebd593162f90b1585205a0901349f8
9 changes: 5 additions & 4 deletions dev/run-tests-jenkins.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,10 +102,10 @@ def send_archived_logs():
print_err(" > scp_output: %s" % scp_stdout)
else:
print(" > Send successful.")
else:
print_err(" > No log files found.")

rm_r(log_archive)
else:
print_err(" > No log files found.")


def pr_message(build_display_name,
Expand Down Expand Up @@ -137,7 +137,8 @@ def run_pr_checks(pr_tests, ghprb_actual_commit, sha1):
pr_results = list()

for pr_test in pr_tests:
pr_results.append(run_cmd(['bash', os.path.join(SPARK_HOME, 'dev', 'tests', pr_test + '.sh'),
test_name = pr_test + '.sh'
pr_results.append(run_cmd(['bash', os.path.join(SPARK_HOME, 'dev', 'tests', test_name),
ghprb_actual_commit, sha1],
return_output=True).strip())
# Ensure, after each test, that we're back on the current PR
Expand All @@ -157,7 +158,7 @@ def run_tests(tests_timeout):
os.path.join(SPARK_HOME, 'dev', 'run-tests')]).wait()

failure_note_by_errcode = {
1: 'executing the `dev/run-tests` script', # error to denote run-tests script failures
1: 'executing the `dev/run-tests` script', # error to denote run-tests script failures
ERROR_CODES["BLOCK_GENERAL"]: 'some tests',
ERROR_CODES["BLOCK_RAT"]: 'RAT tests',
ERROR_CODES["BLOCK_SCALA_STYLE"]: 'Scala style tests',
Expand Down
2 changes: 1 addition & 1 deletion dev/sparktestsupport/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,5 @@
"BLOCK_SPARK_UNIT_TESTS": 17,
"BLOCK_PYSPARK_UNIT_TESTS": 18,
"BLOCK_SPARKR_UNIT_TESTS": 19,
"BLOCK_TIMEOUT": 124
"BLOCK_TIMEOUT": 124
}
1 change: 1 addition & 0 deletions dev/sparktestsupport/shellutils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ def subprocess_check_output(*popenargs, **kwargs):
cmd = popenargs[0]
raise subprocess.CalledProcessError(retcode, cmd, output=output)
return output

# backported from subprocess module in Python 2.7
def subprocess_check_call(*popenargs, **kwargs):
retcode = call(*popenargs, **kwargs)
Expand Down