Skip to content

SPARK-1230: [WIP] Enable SparkContext.addJars() to load classes not in CLASSPATH #119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pwendell wants to merge 15 commits into from
Closed

SPARK-1230: [WIP] Enable SparkContext.addJars() to load classes not in CLASSPATH #119

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
8854dab
Print out the class that was not found
Dec 13, 2013
ae1c199
Fix bug where classes in jars added via addJar() are not available to…
Dec 13, 2013
5701bf4
Merge branch 'master' into debug-class-not-found
Dec 27, 2013
d3df241
Move classLoader initialization to SparkContext
Dec 29, 2013
5c24b0b
Pass the current class loader when creating Akka threadpool
Dec 29, 2013
cbabe80
CR from pwendell
Jan 12, 2014
b0cc61d
Merge branch 'master' into debug-class-not-found
Jan 17, 2014
97a19b5
CR: Rename ExecutorURLClassLoader -> SparkURLClassLoader
Jan 17, 2014
9757c6f
Use SparkURLClassLoader for driver only if spark.driver.add-dynamic-j…
Jan 17, 2014
3e52b0c
Document new config option and effect on addJar
Jan 17, 2014
178083d
Merge remote-tracking branch 'apache/master' into addJars
pwendell Mar 11, 2014
b132d7b
Minor changes
pwendell Mar 11, 2014
95b24f2
Adding unit tests.
pwendell Mar 11, 2014
fbcb4a0
Review feedback
pwendell Mar 12, 2014
9637d21
Style fixes
pwendell Mar 12, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
CR from pwendell
  • Loading branch information
Evan Chan committed Jan 12, 2014
commit cbabe809baf1f02c7c9097734b84a3c94ce364db
9 changes: 5 additions & 4 deletions core/src/main/scala/org/apache/spark/SparkContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,10 +102,11 @@ class SparkContext(

val isLocal = (master == "local" || master.startsWith("local["))

// Create a classLoader for use by the driver so that jars added via addJar are available to the driver
// Do this before all other initialization so that any thread pools created for this SparkContext
// uses the class loader
private[spark] val classLoader = new ExecutorURLClassLoader(Array.empty[URL], this.getClass.getClassLoader)
// Create a classLoader for use by the driver so that jars added via addJar are available to the
// driver. Do this before all other initialization so that any thread pools created for this
// SparkContext uses the class loader.
private[spark] val classLoader = new ExecutorURLClassLoader(Array.empty[URL],
this.getClass.getClassLoader)
Thread.currentThread.setContextClassLoader(classLoader)

// Create the Spark execution environment (cache, map output tracker, etc)
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/util/AkkaUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ private[spark] object AkkaUtils {
*/
def createActorSystem(name: String, host: String, port: Int,
indestructible: Boolean = false,
classLoader: ClassLoader = this.getClass.getClassLoader)
classLoader: ClassLoader = Thread.currentThread.getContextClassLoader)
: (ActorSystem, Int) = {

val akkaThreads = System.getProperty("spark.akka.threads", "4").toInt
Expand Down