Skip to content

[SPARK-4048] Enhance and extend hadoop-provided profile. #2982

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vanzin wants to merge 28 commits into from
Closed

[SPARK-4048] Enhance and extend hadoop-provided profile. #2982

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
284dda6
Rework the "hadoop-provided" profile, add new ones.
Oct 20, 2014
1adf91c
Re-enable maven-install-plugin for a few projects.
Oct 21, 2014
2f95f0d
Propagate classpath to child processes during testing.
Oct 23, 2014
417d90e
Introduce "SPARK_DIST_CLASSPATH".
Oct 23, 2014
4d67469
Propagate SPARK_DIST_CLASSPATH on Yarn.
Oct 27, 2014
d928d62
Redirect child stderr to parent's log.
Oct 28, 2014
9e4e001
Remove duplicate hive profile.
Nov 4, 2014
f7b3bbe
Add snappy to hadoop-provided list.
Nov 4, 2014
1fc4d0b
Update dependencies for hive-thriftserver.
Nov 5, 2014
5c54a25
Fix HiveThriftServer2Suite with *-provided profiles.
Nov 6, 2014
82a54b9
Remove unused profile.
Nov 12, 2014
d1399ed
Restore jetty dependency.
Nov 12, 2014
1be73d4
Restore flume-provided profile.
Nov 15, 2014
7820d58
Fix CliSuite with provided profiles.
Nov 17, 2014
e3ab2da
Fix hive-thriftserver profile.
Nov 17, 2014
115fde5
Simplify a comment (and make it consistent with another pom).
Nov 17, 2014
9640503
Cleanup child process log message.
Nov 17, 2014
8b00b6a
Merge branch 'master' into SPARK-4048
Nov 19, 2014
f24e9e7
Merge branch 'master' into SPARK-4048
Nov 19, 2014
322f882
Fix merge fail.
Nov 20, 2014
7377e7b
Merge branch 'master' into SPARK-4048
Dec 9, 2014
83099fc
Merge branch 'master' into SPARK-4048
Dec 24, 2014
52f366d
Merge branch 'master' into SPARK-4048
Jan 6, 2015
371ebee
Review feedback.
Jan 6, 2015
9ef79a3
Alternative way to propagate test classpath to child processes.
Jan 8, 2015
4e38f4e
Merge branch 'master' into SPARK-4048
Jan 8, 2015
eb228c0
Fix borked merge.
Jan 8, 2015
82eb688
Add a comment.
Jan 8, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Alternative way to propagate test classpath to child processes.
  • Loading branch information
Marcelo Vanzin committed Jan 8, 2015
commit 9ef79a3257ef54848491bf90b68a0ad887a78199
7 changes: 0 additions & 7 deletions core/src/main/scala/org/apache/spark/util/Utils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -991,13 +991,6 @@ private[spark] object Utils extends Logging {
environment.put(key, value)
}

// When testing, expose the parent class path to the child. This is processed by
// compute-classpath.{cmd,sh} and makes all needed jars available to child processes
// when the assembly is built with the "*-provided" profiles enabled.
if (sys.props.contains("spark.testing")) {
environment.put("SPARK_DIST_CLASSPATH", sys.props("java.class.path"))
}

val process = builder.start()
new Thread("read stderr for " + command(0)) {
override def run() {
Expand Down
4 changes: 3 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1144,13 +1144,15 @@
<filereports>SparkTestSuite.txt</filereports>
<argLine>-ea -Xmx3g -XX:MaxPermSize=${MaxPermGen} -XX:ReservedCodeCacheSize=${CodeCacheSize}</argLine>
<stderr/>
<environmentVariables>
<SPARK_DIST_CLASSPATH>${test_classpath}</SPARK_DIST_CLASSPATH>
Copy link
Contributor

@pwendell pwendell Jan 8, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a comment so that it's clear what is going on? Something like:

<!-- Setting SPARK_DIST_CLASSPATH is a simple way to make sure any child processes launched
      by the tests have access to the correct test-time classpath -->

</environmentVariables>
<systemProperties>
<java.awt.headless>true</java.awt.headless>
<spark.test.home>${spark.test.home}</spark.test.home>
<spark.testing>1</spark.testing>
<spark.ui.enabled>false</spark.ui.enabled>
<spark.ui.showConsoleProgress>false</spark.ui.showConsoleProgress>
<spark.executor.extraClassPath>${test_classpath}</spark.executor.extraClassPath>
<spark.driver.allowMultipleContexts>true</spark.driver.allowMultipleContexts>
</systemProperties>
</configuration>
Expand Down