Skip to content

[SPARK-7017][Build][Project Infra]: Refactor dev/run-tests into Python #5694

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
brennonyork wants to merge 57 commits into from
Closed

[SPARK-7017][Build][Project Infra]: Refactor dev/run-tests into Python #5694

Changes from 1 commit
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
6126c4f
refactored run-tests into python
Apr 24, 2015
3c53a1a
uncomment the scala tests :)
Apr 24, 2015
639f1e9
updated with pep8 rules, fixed minor bugs, added run-tests file in ba…
Apr 26, 2015
2cb413b
upcased global variables, changes various calling methods from check_…
Apr 27, 2015
ec03bf3
added namedtuple for java version to add readability
Apr 27, 2015
07210a9
minor doc string change for java version with namedtuple update
Apr 27, 2015
26e18e8
removed unnecessary wait()
Apr 27, 2015
c095fa6
removed another wait() call
Apr 27, 2015
83e80ef
attempt at better python output when called from bash
Apr 27, 2015
b0b2604
comment out import to see if build fails and returns properly
Apr 28, 2015
803143a
removed license file for SparkContext
Apr 28, 2015
a5bd445
reverted license, changed test in shuffle to fail
Apr 28, 2015
7613558
updated to return the proper env variable for return codes
Apr 29, 2015
b37328c
fixed typo and added default return is no error block was found in th…
Apr 29, 2015
56d3cb9
changed test back and commented out import to break compile
Apr 29, 2015
e4a96cc
removed the import error and added license error, fixed the way run-t…
Apr 29, 2015
76335fb
reverted rat license issue for sparkconf
Apr 29, 2015
2386785
Merge remote-tracking branch 'upstream/master' into SPARK-7017
Apr 29, 2015
983f2a2
comment out import to fail build test
Apr 29, 2015
f041d8a
added space from commented import to now test build breaking
Apr 29, 2015
d825aa4
revert build break, add mima break
Apr 29, 2015
9a592ec
reverted mima exclude issue, added pyspark test failure
Apr 30, 2015
1dada6b
reverted pyspark test failure
Apr 30, 2015
afeb093
updated to make sparkR test fail
Apr 30, 2015
b1ca593
reverted the sparkR test
May 1, 2015
703f095
fixed merge conflicts
May 11, 2015
f950010
removed building hive-0.12.0 per SPARK-6908
May 11, 2015
6d0a052
incorporated merge conflicts with SPARK-7249
May 19, 2015
f9deba1
python to python2 and removed newline
May 19, 2015
b1248dc
exec python rather than running python and exiting with return code
May 21, 2015
d90ab2d
fixed merge conflicts, ensured that for regular builds both core and …
May 26, 2015
0629de8
updated to refactor and remove various small bugs, removed pep8 compl…
Jun 5, 2015
8afbe93
made error codes a global
Jun 5, 2015
1f607b1
finalizing revisions to modular tests
Jun 9, 2015
2fcdfc0
testing targte branch dump on jenkins
Jun 10, 2015
1ecca26
fixed merge conflicts
Jun 10, 2015
db7ae6f
reverted SPARK_HOME from start of command
Jun 10, 2015
eb684b6
fixed sbt_test_goals reference error
Jun 10, 2015
2898717
added a change to streaming test to check if it only runs streaming t…
Jun 10, 2015
7d2f5e2
updated python tests to remove unused variable
Jun 10, 2015
60b3d51
prepend rather than append onto PATH
Jun 10, 2015
705d12e
changed example to comply with pep3113 supporting python3
Jun 10, 2015
03fdd7b
fixed the tuple () wraps around example lambda
Jun 10, 2015
b7c72b9
reverting streaming context
Jun 10, 2015
ec1ae78
minor name changes, bug fixes
Jun 14, 2015
aa03d9e
added documentation builds as a top level test component, altered hig…
Jun 15, 2015
0379833
minor doc addition to print the changed modules
Jun 15, 2015
fb85a41
fixed minor set bug
Jun 15, 2015
c42cf9a
unpack set operations with splat (*)
Jun 15, 2015
767a668
fixed path joining issues, ensured docs actually build on doc changes
Jun 16, 2015
2dff136
fixed pep8 whitespace errors
Jun 16, 2015
22edb78
add check if jekyll isn't installed on the path
Jun 16, 2015
05d435b
added check for jekyll install
Jun 16, 2015
8135518
removed the test check for documentation changes until jenkins can ge…
Jun 16, 2015
f9fbe54
reverted doc test change
Jun 16, 2015
3922a85
removed necessary passed in variable
Jun 16, 2015
154ed73
updated finding java binary if JAVA_HOME not set
Jun 16, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
reverted SPARK_HOME from start of command
  • Loading branch information
Brennon York committed Jun 10, 2015
commit db7ae6f74a46b71488ba81249dd5198bf4bd6606
6 changes: 2 additions & 4 deletions dev/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,6 @@ def run_cmd(cmd):
if not isinstance(cmd, list):
cmd = cmd.split()
try:
# prepend SPARK_HOME onto the first element of the command
cmd[0] = os.path.join(SPARK_HOME, *filter(lambda x: x, cmd[0].split(os.path.sep)))
subprocess.check_call(cmd)
except subprocess.CalledProcessError as e:
exit_from_command_with_retcode(e.cmd, e.returncode)
Expand Down Expand Up @@ -293,7 +291,7 @@ def determine_test_modules(test_env):

if test_env == "amplab_jenkins":
target_branch = os.environ.get("ghprbTargetBranch")
Copy link
Contributor

@JoshRosen JoshRosen Jun 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ghprbTargetBranch should always be defined in Jenkins, right? By using get() here, I think we'll get confusing error reporting if ghprbTargetBranch isn't set.

print "target_branch at", target_branch

run_cmd(['git', 'fetch', 'origin', str(target_branch+':'+target_branch)])

raw_output = subprocess.check_output(['git', 'diff', '--name-only', target_branch])
Expand Down Expand Up @@ -449,7 +447,7 @@ def main():
print "ensure the $HOME environment variable is set properly."
sys.exit(1)

#os.chdir(SPARK_HOME)
os.chdir(SPARK_HOME)

rm_r(os.path.join(SPARK_HOME, "work"))
rm_r(os.path.join(USER_HOME, ".ivy2/local/org.apache.spark"))
Expand Down