Skip to content

SPARK-1230: [WIP] Enable SparkContext.addJars() to load classes not in CLASSPATH #119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pwendell wants to merge 15 commits into from
Closed

SPARK-1230: [WIP] Enable SparkContext.addJars() to load classes not in CLASSPATH #119

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
8854dab
Print out the class that was not found
Dec 13, 2013
ae1c199
Fix bug where classes in jars added via addJar() are not available to…
Dec 13, 2013
5701bf4
Merge branch 'master' into debug-class-not-found
Dec 27, 2013
d3df241
Move classLoader initialization to SparkContext
Dec 29, 2013
5c24b0b
Pass the current class loader when creating Akka threadpool
Dec 29, 2013
cbabe80
CR from pwendell
Jan 12, 2014
b0cc61d
Merge branch 'master' into debug-class-not-found
Jan 17, 2014
97a19b5
CR: Rename ExecutorURLClassLoader -> SparkURLClassLoader
Jan 17, 2014
9757c6f
Use SparkURLClassLoader for driver only if spark.driver.add-dynamic-j…
Jan 17, 2014
3e52b0c
Document new config option and effect on addJar
Jan 17, 2014
178083d
Merge remote-tracking branch 'apache/master' into addJars
pwendell Mar 11, 2014
b132d7b
Minor changes
pwendell Mar 11, 2014
95b24f2
Adding unit tests.
pwendell Mar 11, 2014
fbcb4a0
Review feedback
pwendell Mar 12, 2014
9637d21
Style fixes
pwendell Mar 12, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Style fixes
  • Loading branch information
@pwendell
pwendell committed Mar 12, 2014
commit 9637d2110b4566e6f5db1ee67feb8c8ce3837e7a
9 changes: 5 additions & 4 deletions core/src/main/scala/org/apache/spark/SparkContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -782,17 +782,18 @@ class SparkContext(
path
}

// Add jar to driver class loader so it is available for driver, even if it is not on the classpath
// Add jar to driver class loader so it is available for driver,
// even if it is not on the classpath
uri.getScheme match {
case null | "file" | "local" =>
// Assume file exists on current (driver) node as well. Unlike executors, driver doesn't need to
// download the jar since it's local.
// Assume file exists on current (driver) node as well. Unlike executors, driver
// doesn't need to download the jar since it's local.
addUrlToDriverLoader(new URL("file:" + uri.getPath))
case "http" | "https" | "ftp" =>
// Should be handled by the URLClassLoader, pass along entire URL
addUrlToDriverLoader(new URL(path))
case other =>
logWarning("This URI scheme for URI " + path + " is not supported by the driver class loader")
logWarning(s"This URI scheme for URI $path is not supported by the driver class loader")
Copy link
Contributor

@mateiz mateiz Mar 12, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a poor way to handle this case. Why don't we create a ClassLoader from HDFS or whatever the other URI scheme is? In general if some URI scheme is supported on the workers, it should also be supported on the driver.

Copy link
Contributor

@mateiz mateiz Mar 12, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other option BTW would be to throw an exception, since the user explicitly enabled loading JARs in the driver. In general logging a warning and hoping the user will notice it is not great.

Copy link
Contributor

@velvia velvia Apr 2, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes true. At the minimum will replace with an exception.

}
}
if (key != null) {
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ private[spark] class TaskResultGetter(sparkEnv: SparkEnv, scheduler: TaskSchedul
} catch {
case cnf: ClassNotFoundException =>
val loader = Thread.currentThread.getContextClassLoader
taskSetManager.abort("ClassNotFound [" + cnf.getMessage + "] with classloader: " + loader)
taskSetManager.abort(s"ClassNotFound [${cnf.getMessage}] with classloader: " + loader)
case ex: Throwable =>
taskSetManager.abort("Exception while deserializing and fetching task: %s".format(ex))
}
Expand Down