Skip to content

[SPARK-10359][PROJECT-INFRA] Backport dev/test-dependencies script to branch-1.6 #10680

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JoshRosen wants to merge 3 commits into from
Closed

[SPARK-10359][PROJECT-INFRA] Backport dev/test-dependencies script to branch-1.6 #10680

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f97d4d5
[SPARK-10359] Enumerate dependencies in a file and diff against it fo…
JoshRosen Dec 30, 2015
e1ab044
[SPARK-10359][PROJECT-INFRA] Multiple fixes to dev/test-dependencies.…
JoshRosen Jan 1, 2016
f8b4fdb
[SPARK-10359][PROJECT-INFRA] Use more random number in dev/test-depen…
JoshRosen Jan 4, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
[SPARK-10359][PROJECT-INFRA] Use more random number in dev/test-depen… 
…dencies.sh; fix version switching

This patch aims to fix another potential source of flakiness in the `dev/test-dependencies.sh` script.

pwendell's original patch and my version used `$(date +%s | tail -c6)` to generate a suffix to use when installing temporary Spark versions into the local Maven cache, but this value only changes once per second and thus is highly collision-prone when concurrent builds launch on AMPLab Jenkins. In order to reduce the potential for conflicts, this patch updates the script to call Python's random number generator instead.

I also fixed a bug in how we captured the original project version; the bug was causing the exit handler code to fail.

Author: Josh Rosen <[email protected]>

Closes #10558 from JoshRosen/build-dep-tests-round-3.
  • Loading branch information
@JoshRosen
JoshRosen committed Jan 10, 2016
commit f8b4fdb91870e0702bd2735f9a5a58dea4a63fff
16 changes: 13 additions & 3 deletions dev/test-dependencies.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,19 @@ HADOOP_PROFILES=(
# the old version. We need to do this because the `dependency:build-classpath` task needs to
# resolve Spark's internal submodule dependencies.

# See http://stackoverflow.com/a/3545363 for an explanation of this one-liner:
OLD_VERSION=$($MVN help:evaluate -Dexpression=project.version|grep -Ev '(^\[|Download\w+:)')
TEMP_VERSION="spark-$(date +%s | tail -c6)"
# From http://stackoverflow.com/a/26514030
set +e
OLD_VERSION=$($MVN -q \
-Dexec.executable="echo" \
-Dexec.args='${project.version}' \
--non-recursive \
org.codehaus.mojo:exec-maven-plugin:1.3.1:exec)
if [ $? != 0 ]; then
echo -e "Error while getting version string from Maven:\n$OLD_VERSION"
exit 1
fi
set -e
TEMP_VERSION="spark-$(python -S -c "import random; print(random.randrange(100000, 999999))")"

function reset_version {
# Delete the temporary POMs that we wrote to the local Maven repo:
Expand Down