Skip to content

[SPARK-10359] Enumerate Spark's dependencies in a file and diff against it for new pull requests #8531

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pwendell wants to merge 16 commits into from
Closed

[SPARK-10359] Enumerate Spark's dependencies in a file and diff against it for new pull requests #8531

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
9cf442d
Adding build test module
pwendell Aug 28, 2015
20ae791
Adding new deps file without spark
pwendell Aug 31, 2015
5670b6a
Moving build tests earlier
pwendell Aug 31, 2015
5b2910b
Adding license header
pwendell Aug 31, 2015
0c8153d
Adding RAT exclude
pwendell Aug 31, 2015
f4243b9
Merge remote-tracking branch 'apache/master' into dependency-audits
pwendell Nov 3, 2015
17970f7
Ignore spark itself in deps file
pwendell Nov 3, 2015
16adab6
Style fix
pwendell Nov 3, 2015
8b973bf
Adding test block
pwendell Nov 3, 2015
d181e3d
Only include compile dependencies when generating list
pwendell Nov 3, 2015
84cab2c
Adding Hadoop 1 profile
pwendell Nov 3, 2015
5c8ccf6
Adding new hadoop versions
pwendell Nov 3, 2015
fcea19f
Bug fix
pwendell Nov 3, 2015
2f4d3e5
Adding new hadoop deps
pwendell Nov 3, 2015
37230f0
More RAT excludes
pwendell Nov 3, 2015
1d91634
Testing new dependencies
pwendell Nov 3, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Only include compile dependencies when generating list
  • Loading branch information
@pwendell
pwendell committed Nov 3, 2015
commit d181e3d15c1ca436084a05f76eb18cde4ac1a209
2 changes: 1 addition & 1 deletion dev/test-dependencies.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ $MVN install -q \
-DskipTests

echo "Generating dependency manifest"
$MVN dependency:build-classpath \
$MVN dependency:build-classpath -pl assembly \
| grep "Building Spark Project Assembly" -A 5 \
| tail -n 1 | tr ":" "\n" | rev | cut -d "/" -f 1 | rev | sort \
| grep -v spark > dev/pr-deps
Expand Down
16 changes: 16 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2052,6 +2052,22 @@
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.2</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>default-cli</id>
<goals>
<goal>build-classpath</goal>
</goals>
<configuration>
<!-- Only count compile-scoped dependencies when generating classpath -->
Copy link
Member

@srowen srowen Nov 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need runtime as well?

Copy link
Contributor

@JoshRosen JoshRosen Dec 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I think you're right. According to the maven-dependency-plugin docs (https://maven.apache.org/plugins/maven-dependency-plugin/list-mojo.html#includeScope):

Scope to include. An Empty string indicates all scopes (default). The scopes being interpreted are the scopes as Maven sees them, not as specified in the pom. In summary:

  • runtime scope gives runtime and compile dependencies,
  • compile scope gives compile, provided, and system dependencies,
  • test (default) scope gives all dependencies,
  • provided scope just gives provided dependencies,
  • system scope just gives system dependencies.

Based on this language, it seems like we want to be using runtime here so that we include both compile and runtime scope dependencies but do not exclude provided ones.

<includeScope>compile</includeScope>
</configuration>
</execution>
</executions>
</plugin>
<!-- This plugin's configuration is used to store Eclipse m2e settings only. -->
<!-- It has no influence on the Maven build itself. -->
<plugin>
Expand Down