Skip to content

SPARK-9545, SPARK-9547: Use Maven in PRB if title contains "[test-maven]" #7878

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pwendell wants to merge 16 commits into from
Closed

SPARK-9545, SPARK-9547: Use Maven in PRB if title contains "[test-maven]" #7878

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
6184a00
SPARK-9545: Use Maven in prb if title contains "[maven-test]"
pwendell Aug 2, 2015
a15f53c
Wording change to test-maven
pwendell Aug 2, 2015
e83b49f
Adding support for SPARK-9547
pwendell Aug 2, 2015
5d15d31
Adding randomized zinc ports
pwendell Aug 2, 2015
7f1ff75
Fixing up python lint issues
pwendell Aug 2, 2015
c77f951
Small style fix
pwendell Aug 2, 2015
69dd9af
Push Zinc support down into exec_maven
pwendell Aug 2, 2015
0d3944a
Small fixes
pwendell Aug 2, 2015
05d7720
Bug fix to kill command
pwendell Aug 3, 2015
3d36d0a
Temporarily disabling mima
pwendell Aug 3, 2015
2e9edcc
Increasing timeout
pwendell Aug 3, 2015
acf1e4f
Merge remote-tracking branch 'apache/master' into maven-tests
pwendell Aug 4, 2015
a938de3
Only do compatibility tests in sbt
pwendell Aug 4, 2015
603f891
removing unncessary spacing changes
pwendell Aug 4, 2015
87d4d19
Whitepsace issues
pwendell Aug 4, 2015
cf58c49
Merge remote-tracking branch 'apache/master' into maven-tests
pwendell Aug 27, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Push Zinc support down into exec_maven
  • Loading branch information
@pwendell
pwendell committed Aug 2, 2015
commit 69dd9afe387c1db24542df566ae39ccd3ccd90a8
14 changes: 7 additions & 7 deletions dev/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,12 @@ def exec_maven(mvn_args=()):
"""Will call Maven in the current directory with the list of mvn_args passed
in and returns the subprocess for any further processing"""

run_cmd([os.path.join(SPARK_HOME, "build", "mvn"), "--force"] + mvn_args)
zinc_port = get_zinc_port()
os.environ["ZINC_PORT"] = "%s" % zinc_port
zinc_flag = "-DzincPort=%s" % zinc_port
flags = [os.path.join(SPARK_HOME, "build", "mvn"), "--force", zinc_flag]
run_cmd(flags + mvn_args)
kill_zinc_on_port(zinc_port)


def exec_sbt(sbt_args=()):
Expand Down Expand Up @@ -288,15 +293,13 @@ def get_zinc_port():
"""
Get a randomized port on which to start Zinc
"""

return random.randrange(3030, 4030)


def kill_zinc_on_port(zinc_port):
"""
Kill the Zinc process running on the given port, if one exists.
"""

cmd = ("/usr/sbin/lsof -P |grep %s | grep LISTEN "
"| awk '{ print $2; }' | xargs kill") % zinc_port
# TODO: Not sure what happens here if no process exists
Copy link
Contributor

@JoshRosen JoshRosen Aug 2, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a non-zero exit code, run_cmd will print an error message and will call sys.exit(), so we may want to ensure that this always returns a zero exit code. If this is a concern, you could always just perform the underlying subprocess call yourself.

Expand All @@ -306,7 +309,6 @@ def kill_zinc_on_port(zinc_port):
def build_spark_maven(hadoop_version):
# Enable all of the profiles for the build:
build_profiles = get_hadoop_profiles(hadoop_version) + modules.root.build_profile_flags
zinc_port = get_zinc_port()
mvn_goals = ["clean", "package", "-DskipTests", "-DzincPort=%s" % zinc_port]
profiles_and_goals = build_profiles + mvn_goals
print("[info] Building Spark (w/Hive 0.13.1) using Maven with these arguments: ",
Expand Down Expand Up @@ -351,15 +353,13 @@ def detect_binary_inop_with_mima():


def run_scala_tests_maven(test_profiles):
zinc_port = get_zinc_port()
mvn_test_goals = ["test", "--fail-at-end", "-DzincPort=%s" % zinc_port]
mvn_test_goals = ["test", "--fail-at-end"]
profiles_and_goals = test_profiles + mvn_test_goals

print("[info] Running Spark tests using Maven with these arguments: ",
" ".join(profiles_and_goals))

exec_maven(profiles_and_goals)
kill_zinc_on_port(zinc_port)


def run_scala_tests_sbt(test_modules, test_profiles):
Expand Down