Skip to content

[SPARK-13294] [PROJECT INFRA] Remove MiMa's dependency on spark-class / Spark assembly #11178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JoshRosen wants to merge 25 commits into from
Closed

[SPARK-13294] [PROJECT INFRA] Remove MiMa's dependency on spark-class / Spark assembly #11178

Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
73b1550
[BUILD] Remove assembly/assembly step from dev/run-tests.
JoshRosen Feb 12, 2016
b6f1ce8
Also remove from Maven build step.
JoshRosen Feb 12, 2016
5528c48
Set SPARK_PREPEND_CLASSES in dev/mima.
JoshRosen Feb 12, 2016
bef62eb
Use 1 instead of true
JoshRosen Feb 12, 2016
76a365e
More attempts towards fixing MiMa.
JoshRosen Feb 12, 2016
9fc0f7a
Fix heap sizing.
JoshRosen Feb 13, 2016
31854eb
Try using classutil to find Spark classes.
JoshRosen Feb 13, 2016
906d8c8
Fix PySpark tests by setting SPARK_DIST_CLASSPATH
JoshRosen Feb 17, 2016
1f995a4
Merge remote-tracking branch 'origin/master' into remove-assembly-in-…
JoshRosen Mar 1, 2016
902b1b7
Update to fix classpaths in MiMA.
JoshRosen Mar 1, 2016
a73118e
Update to reflect #11426
JoshRosen Mar 1, 2016
64330d6
Second shot at fixing MiMa
JoshRosen Mar 2, 2016
9a122ad
Merge remote-tracking branch 'origin/master' into remove-assembly-in-…
JoshRosen Mar 10, 2016
db8e532
Update AbstractCommandBuilder to reflect recent relocations
JoshRosen Mar 10, 2016
cd7eb04
Just grab fullClasspath from assembly in MiMa, since we only run MiMA…
JoshRosen Mar 10, 2016
4756a1e
Remove now-unused o.a.s.tools code from SparkClassCommandBuilder
JoshRosen Mar 10, 2016
ae6b002
Fix help command of ./python/run-tests
JoshRosen Mar 10, 2016
dae4725
Bump to working version of sbt-dependency-graph to aid debugging
JoshRosen Mar 10, 2016
373fd52
Roll back non-MiMa-related changes (they'll go in later).
JoshRosen Mar 11, 2016
8ec1cc4
Revert "Bump to working version of sbt-dependency-graph to aid debugg…
JoshRosen Mar 11, 2016
5d32e74
Continue to build assembly for tests, but skip it before MiMa.
JoshRosen Mar 11, 2016
97b5d78
Revert attempts at improving MiMa code to minimize diff (will add bac…
JoshRosen Mar 11, 2016
f9e2b42
Turns out that we do need transitive deps.
JoshRosen Mar 11, 2016
4070c0d
Merge remote-tracking branch 'origin/master' into remove-assembly-in-…
JoshRosen Mar 11, 2016
86cd513
We can't use .tree.tpe due to Scala 2.10.
JoshRosen Mar 11, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Also remove from Maven build step.
  • Loading branch information
@JoshRosen
JoshRosen committed Feb 12, 2016
commit b6f1ce89dd4e65885abca9d22665aa518eebf10b
2 changes: 1 addition & 1 deletion dev/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,7 @@ def get_hadoop_profiles(hadoop_version):
def build_spark_maven(hadoop_version):
# Enable all of the profiles for the build:
build_profiles = get_hadoop_profiles(hadoop_version) + modules.root.build_profile_flags
mvn_goals = ["clean", "package", "-DskipTests"]
mvn_goals = ["clean", "package", "-DskipTests", "-pl", "!assembly"]
profiles_and_goals = build_profiles + mvn_goals

print("[info] Building Spark (w/Hive 1.2.1) using Maven with these arguments: ",
Expand Down