Skip to content

[SPARK-5794] [SQL] fix add jar #4586

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
adrian-wang wants to merge 7 commits into from
Closed

[SPARK-5794] [SQL] fix add jar #4586

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
95a40da
support env argus in add jar, and set add jar ret to 0
adrian-wang Mar 3, 2015
1898309
fix classnotfound
adrian-wang Mar 5, 2015
0810e71
remove variable substitution
adrian-wang Apr 2, 2015
9957d87
use sessionstate classloader in makeRDDforTable
adrian-wang Apr 8, 2015
32c4fb8
fix style and add a test
adrian-wang Apr 8, 2015
6c707e8
restrict hive version for test
adrian-wang Apr 8, 2015
efdd602
move jar to another place
adrian-wang Apr 9, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix classnotfound
  • Loading branch information
@adrian-wang
adrian-wang committed Apr 2, 2015
commit 1898309aa300436341acb1c5036ed2e820b58b72
11 changes: 11 additions & 0 deletions core/src/main/scala/org/apache/spark/SparkContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,17 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
}
}

private var classLoader = Thread.currentThread().getContextClassLoader();
if (classLoader == null) {
classLoader = this.getClass.getClassLoader()
}

def getClassLoader: ClassLoader = classLoader

def setClassLoader(cl: ClassLoader) = {
classLoader = cl
}

/**
* Create a SparkContext that loads settings from system properties (for instance, when
* launching with ./bin/spark-submit).
Expand Down
4 changes: 3 additions & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ import org.apache.spark.broadcast.Broadcast
import org.apache.spark.rdd.{EmptyRDD, HadoopRDD, RDD, UnionRDD}
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.types.DateUtils
import org.apache.spark.util.Utils

/**
* A trait for subclasses that handle table scans.
Expand Down Expand Up @@ -76,7 +77,8 @@ class HadoopTableReader(
override def makeRDDForTable(hiveTable: HiveTable): RDD[Row] =
makeRDDForTable(
hiveTable,
relation.tableDesc.getDeserializerClass.asInstanceOf[Class[Deserializer]],
Class.forName(relation.tableDesc.getSerdeClassName, true, sc.sparkContext.getClassLoader)
Copy link
Contributor

@chenghao-intel chenghao-intel Mar 5, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After looking the code http://www.grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hive/hive-exec/0.13.1-cdh5.3.0/org/apache/hadoop/hive/ql/session/SessionState.java#61

Probably we don't need to create the new class loader by ourselves, we can reuse the SessionState code instead.

Copy link
Contributor

@jeanlyn jeanlyn Mar 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we also do this in makeRDDForPartitionedTable?

.asInstanceOf[Class[Deserializer]],
filterOpt = None)

/**
Expand Down
17 changes: 16 additions & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/commands.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

package org.apache.spark.sql.hive.execution

import org.apache.commons.lang.StringUtils
import org.apache.hadoop.hive.ql.exec.Utilities
import org.apache.hadoop.hive.ql.parse.VariableSubstitution

import org.apache.spark.sql.AnalysisException
Expand Down Expand Up @@ -83,7 +85,20 @@ case class AddJar(path: String) extends RunnableCommand {
val substituted =
new VariableSubstitution().substitute(sqlContext.asInstanceOf[HiveContext].hiveconf, path)
hiveContext.sparkContext.addJar(substituted)
Seq(Row(0))
try {
val loader = Thread.currentThread().getContextClassLoader()
val newLoader = Utilities.addToClassPath(loader, StringUtils.split(substituted, ","))
Thread.currentThread().setContextClassLoader(newLoader)
sqlContext.sparkContext.setClassLoader(newLoader)
logInfo("Added " + substituted + " to class path")
Seq(Row(0))
} catch {
case e: Throwable =>
logError("Unable to register " + substituted + "\nException: "
+ e.getMessage() + "\n"
+ org.apache.hadoop.util.StringUtils.stringifyException(e), e)
Seq(Row(1))
}
}
}

Expand Down