Skip to content

[SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests #10461

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JoshRosen wants to merge 37 commits into from
Closed

[SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests #10461

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
9cf442d
Adding build test module
pwendell Aug 28, 2015
20ae791
Adding new deps file without spark
pwendell Aug 31, 2015
5670b6a
Moving build tests earlier
pwendell Aug 31, 2015
5b2910b
Adding license header
pwendell Aug 31, 2015
0c8153d
Adding RAT exclude
pwendell Aug 31, 2015
f4243b9
Merge remote-tracking branch 'apache/master' into dependency-audits
pwendell Nov 3, 2015
17970f7
Ignore spark itself in deps file
pwendell Nov 3, 2015
16adab6
Style fix
pwendell Nov 3, 2015
8b973bf
Adding test block
pwendell Nov 3, 2015
d181e3d
Only include compile dependencies when generating list
pwendell Nov 3, 2015
84cab2c
Adding Hadoop 1 profile
pwendell Nov 3, 2015
5c8ccf6
Adding new hadoop versions
pwendell Nov 3, 2015
fcea19f
Bug fix
pwendell Nov 3, 2015
2f4d3e5
Adding new hadoop deps
pwendell Nov 3, 2015
37230f0
More RAT excludes
pwendell Nov 3, 2015
1d91634
Testing new dependencies
pwendell Nov 3, 2015
1e3ff47
Merge remote-tracking branch 'origin/master' into SPARK-10359
JoshRosen Dec 23, 2015
b013514
Refactor to avoid compile in install.
JoshRosen Dec 23, 2015
8f7007e
Don't generate version backup POMs.
JoshRosen Dec 23, 2015
258e4f7
Use bash exit traps to ensure proper cleanup
JoshRosen Dec 23, 2015
e8af274
Whoops, missed another -DgenerateBackupPoms=false
JoshRosen Dec 23, 2015
8507ffd
Support multiple Hadoop profiles.
JoshRosen Dec 23, 2015
2711252
Roll back test dependency change.
JoshRosen Dec 23, 2015
4ad3ae4
Fix RAT excludes (CLI RAT doesn't seem to support full path pattern m…
JoshRosen Dec 23, 2015
7c86eb7
dev/test-dependencies.sh is part of the build module.
JoshRosen Dec 23, 2015
87918d8
Use runtime scope (per Sean's comments)
JoshRosen Dec 23, 2015
15a0e68
Remove old dependency diffing code.
JoshRosen Dec 23, 2015
5acdd9a
Remove accidentally-added files
JoshRosen Dec 23, 2015
63e4109
Update pom.xml
JoshRosen Dec 30, 2015
8a9e4a7
Update pom.xml
JoshRosen Dec 30, 2015
2e78d57
Move build tests earlier in script
JoshRosen Dec 30, 2015
51f9752
whitespace
JoshRosen Dec 30, 2015
9932c80
Dummy build change.
JoshRosen Dec 30, 2015
1454f5f
Changes to test module system
JoshRosen Dec 30, 2015
329bdf2
Use `git diff` for nicer diff formatting.
JoshRosen Dec 30, 2015
367c8ed
Minor change in exit trap
JoshRosen Dec 30, 2015
cc99de6
Roll back test dependency change.
JoshRosen Dec 30, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use runtime scope (per Sean's comments)
  • Loading branch information
@JoshRosen
JoshRosen committed Dec 23, 2015
commit 87918d8df6ce2608f18f70704ba6b8f900e8ca5d
15 changes: 0 additions & 15 deletions dev/deps/spark-deps-hadoop-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@ geronimo-annotation_1.0_spec-1.1.1.jar
geronimo-jaspic_1.0_spec-1.0.jar
geronimo-jta_1.1_spec-1.1.1.jar
groovy-all-2.1.6.jar
guava-14.0.1.jar
guice-3.0.jar
guice-servlet-3.0.jar
hadoop-annotations-2.3.0.jar
Expand Down Expand Up @@ -92,11 +91,8 @@ jackson-xc-1.9.13.jar
janino-2.7.8.jar
jansi-1.4.jar
java-xmlbuilder-1.0.jar
javax.activation-1.1.0.v201105071233.jar
javax.inject-1.jar
javax.mail.glassfish-1.4.1.v201005082020.jar
javax.servlet-3.0.0.v201112011016.jar
javax.transaction-1.1.1.v201105210645.jar
javolution-5.5.1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
Expand All @@ -110,18 +106,7 @@ jets3t-0.9.3.jar
jettison-1.1.jar
jetty-6.1.26.jar
jetty-all-7.6.0.v20120127.jar
jetty-continuation-8.1.14.v20131031.jar
jetty-http-8.1.14.v20131031.jar
jetty-io-8.1.14.v20131031.jar
jetty-jndi-8.1.14.v20131031.jar
jetty-plus-8.1.14.v20131031.jar
jetty-security-8.1.14.v20131031.jar
jetty-server-8.1.14.v20131031.jar
jetty-servlet-8.1.14.v20131031.jar
jetty-util-6.1.26.jar
jetty-util-8.1.14.v20131031.jar
jetty-webapp-8.1.14.v20131031.jar
jetty-xml-8.1.14.v20131031.jar
jline-2.10.5.jar
jline-2.12.jar
joda-time-2.9.jar
Expand Down
15 changes: 0 additions & 15 deletions dev/deps/spark-deps-hadoop-2.4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@ geronimo-annotation_1.0_spec-1.1.1.jar
geronimo-jaspic_1.0_spec-1.0.jar
geronimo-jta_1.1_spec-1.1.1.jar
groovy-all-2.1.6.jar
guava-14.0.1.jar
guice-3.0.jar
guice-servlet-3.0.jar
hadoop-annotations-2.4.0.jar
Expand Down Expand Up @@ -92,11 +91,8 @@ jackson-xc-1.9.13.jar
janino-2.7.8.jar
jansi-1.4.jar
java-xmlbuilder-1.0.jar
javax.activation-1.1.0.v201105071233.jar
javax.inject-1.jar
javax.mail.glassfish-1.4.1.v201005082020.jar
javax.servlet-3.0.0.v201112011016.jar
javax.transaction-1.1.1.v201105210645.jar
javolution-5.5.1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
Expand All @@ -111,18 +107,7 @@ jets3t-0.9.3.jar
jettison-1.1.jar
jetty-6.1.26.jar
jetty-all-7.6.0.v20120127.jar
jetty-continuation-8.1.14.v20131031.jar
jetty-http-8.1.14.v20131031.jar
jetty-io-8.1.14.v20131031.jar
jetty-jndi-8.1.14.v20131031.jar
jetty-plus-8.1.14.v20131031.jar
jetty-security-8.1.14.v20131031.jar
jetty-server-8.1.14.v20131031.jar
jetty-servlet-8.1.14.v20131031.jar
jetty-util-6.1.26.jar
jetty-util-8.1.14.v20131031.jar
jetty-webapp-8.1.14.v20131031.jar
jetty-xml-8.1.14.v20131031.jar
jline-2.10.5.jar
jline-2.12.jar
joda-time-2.9.jar
Expand Down
4 changes: 2 additions & 2 deletions dev/test-dependencies.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ $MVN -q versions:set -DnewVersion=$TEMP_VERSION -DgenerateBackupPoms=false > /de
# Generate manifests for each Hadoop profile:
for HADOOP_PROFILE in "${HADOOP_PROFILES[@]}"; do
echo "Performing Maven install for $HADOOP_PROFILE"
$MVN $HADOOP2_MODULE_PROFILES -P$HADOOP_PROFILE jar:jar install:install \
$MVN $HADOOP2_MODULE_PROFILES -P$HADOOP_PROFILE jar:jar install:install -q \
-pl '!assembly' \
-pl '!examples' \
-pl '!external/flume-assembly' \
Expand Down Expand Up @@ -89,7 +89,7 @@ for HADOOP_PROFILE in "${HADOOP_PROFILES[@]}"; do
set -e
if [ "$dep_diff" != "" ]; then
echo "Spark's published dependencies DO NOT MATCH the manifest file (dev/spark-deps)."
echo "To update the manifest file, run './dev/test-dependencies --replace-manifest'."
echo "To update the manifest file, run './dev/test-dependencies.sh --replace-manifest'."
echo "$dep_diff"
rm -rf dev/pr-deps
exit 1
Expand Down
5 changes: 3 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2123,8 +2123,9 @@
<goal>build-classpath</goal>
</goals>
<configuration>
<!-- Only count compile-scoped dependencies when generating classpath -->
<includeScope>compile</includeScope>
<!-- This includes dependencies with 'runtime' and 'compile' scopes;
see the docs for includeScope for more details -->
<includeScope>runtime</includeScope>
</configuration>
</execution>
</executions>
Expand Down