Skip to content

SPARK-9545, SPARK-9547: Use Maven in PRB if title contains "[test-maven]" #7878

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pwendell wants to merge 16 commits into from
Closed

SPARK-9545, SPARK-9547: Use Maven in PRB if title contains "[test-maven]" #7878

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
6184a00
SPARK-9545: Use Maven in prb if title contains "[maven-test]"
pwendell Aug 2, 2015
a15f53c
Wording change to test-maven
pwendell Aug 2, 2015
e83b49f
Adding support for SPARK-9547
pwendell Aug 2, 2015
5d15d31
Adding randomized zinc ports
pwendell Aug 2, 2015
7f1ff75
Fixing up python lint issues
pwendell Aug 2, 2015
c77f951
Small style fix
pwendell Aug 2, 2015
69dd9af
Push Zinc support down into exec_maven
pwendell Aug 2, 2015
0d3944a
Small fixes
pwendell Aug 2, 2015
05d7720
Bug fix to kill command
pwendell Aug 3, 2015
3d36d0a
Temporarily disabling mima
pwendell Aug 3, 2015
2e9edcc
Increasing timeout
pwendell Aug 3, 2015
acf1e4f
Merge remote-tracking branch 'apache/master' into maven-tests
pwendell Aug 4, 2015
a938de3
Only do compatibility tests in sbt
pwendell Aug 4, 2015
603f891
removing unncessary spacing changes
pwendell Aug 4, 2015
87d4d19
Whitepsace issues
pwendell Aug 4, 2015
cf58c49
Merge remote-tracking branch 'apache/master' into maven-tests
pwendell Aug 27, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions dev/run-tests-jenkins
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,8 +164,9 @@ pr_message=""
current_pr_head="`git rev-parse HEAD`"

echo "HEAD: `git rev-parse HEAD`"
echo "GHPRB: $ghprbActualCommit"
echo "SHA1: $sha1"
echo "\$ghprbActualCommit: $ghprbActualCommit"
echo "\$sha1: $sha1"
echo "\$ghprbPullTitle: $ghprbPullTitle"

# Run pull request tests
for t in "${PR_TESTS[@]}"; do
Expand All @@ -189,6 +190,19 @@ done
{
# Marks this build is a pull request build.
export AMP_JENKINS_PRB=true
if [[ $ghprbPullTitle == *"test-maven"* ]]; then
export AMPLAB_JENKINS_BUILD_TOOL="maven"
fi
if [[ $ghprbPullTitle == *"test-hadoop1.0"* ]]; then
export AMPLAB_JENKINS_BUILD_PROFILE="hadoop1.0"
elif [[ $ghprbPullTitle == *"test-hadoop2.0"* ]]; then
export AMPLAB_JENKINS_BUILD_PROFILE="hadoop2.0"
elif [[ $ghprbPullTitle == *"test-hadoop2.2"* ]]; then
export AMPLAB_JENKINS_BUILD_PROFILE="hadoop2.2"
elif [[ $ghprbPullTitle == *"test-hadoop2.3"* ]]; then
export AMPLAB_JENKINS_BUILD_PROFILE="hadoop2.3"
fi

timeout "${TESTS_TIMEOUT}" ./dev/run-tests
test_result="$?"

Expand Down
32 changes: 27 additions & 5 deletions dev/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import itertools
from optparse import OptionParser
import os
import random
import re
import sys
import subprocess
Expand Down Expand Up @@ -231,7 +232,7 @@ def exec_maven(mvn_args=()):
"""Will call Maven in the current directory with the list of mvn_args passed
in and returns the subprocess for any further processing"""

run_cmd([os.path.join(SPARK_HOME, "build", "mvn")] + mvn_args)
run_cmd([os.path.join(SPARK_HOME, "build", "mvn"), "--force"] + mvn_args)


def exec_sbt(sbt_args=()):
Expand Down Expand Up @@ -283,16 +284,35 @@ def get_hadoop_profiles(hadoop_version):
sys.exit(int(os.environ.get("CURRENT_BLOCK", 255)))


def get_zinc_port():
"""
Get a randomized port on which to start Zinc
"""

return random.randrange(3030, 4030)


def kill_zinc_on_port(zinc_port):
"""
Kill the Zinc process running on the given port, if one exists.
"""

cmd = ("/usr/sbin/lsof -P |grep %s | grep LISTEN "
"| awk '{ print $2; }' | xargs kill") % zinc_port
# TODO: Not sure what happens here if no process exists
Copy link
Contributor

@JoshRosen JoshRosen Aug 2, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a non-zero exit code, run_cmd will print an error message and will call sys.exit(), so we may want to ensure that this always returns a zero exit code. If this is a concern, you could always just perform the underlying subprocess call yourself.

run_cmd(cmd)


def build_spark_maven(hadoop_version):
# Enable all of the profiles for the build:
build_profiles = get_hadoop_profiles(hadoop_version) + modules.root.build_profile_flags
mvn_goals = ["clean", "package", "-DskipTests"]
zinc_port = get_zinc_port()
mvn_goals = ["clean", "package", "-DskipTests", "-DzincPort=%s" % zinc_port]
profiles_and_goals = build_profiles + mvn_goals

print("[info] Building Spark (w/Hive 0.13.1) using Maven with these arguments: ",
" ".join(profiles_and_goals))

exec_maven(profiles_and_goals)
kill_zinc_on_port(zinc_port)


def build_spark_sbt(hadoop_version):
Expand Down Expand Up @@ -331,13 +351,15 @@ def detect_binary_inop_with_mima():


def run_scala_tests_maven(test_profiles):
mvn_test_goals = ["test", "--fail-at-end"]
zinc_port = get_zinc_port()
mvn_test_goals = ["test", "--fail-at-end", "-DzincPort=%s" % zinc_port]
profiles_and_goals = test_profiles + mvn_test_goals

print("[info] Running Spark tests using Maven with these arguments: ",
" ".join(profiles_and_goals))

exec_maven(profiles_and_goals)
kill_zinc_on_port(zinc_port)


def run_scala_tests_sbt(test_modules, test_profiles):
Expand Down