Skip to content

[SPARK-28704][SQL][TEST] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+ #30451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AngersZhuuuu wants to merge 9 commits into from
Closed

[SPARK-28704][SQL][TEST] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+ #30451

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
d553765
Update HiveExternalCatalogVersionsSuite.scala
AngersZhuuuu Nov 19, 2020
b9dd1e4
Update HiveExternalCatalogVersionsSuite.scala
AngersZhuuuu Nov 20, 2020
9400d9a
Merge branch 'master' into SPARK-28704
AngersZhuuuu Nov 21, 2020
67222ca
Update package.scala
AngersZhuuuu Nov 21, 2020
053974a
with extra Deps
AngersZhuuuu Nov 21, 2020
668db8b
Update package.scala
AngersZhuuuu Nov 21, 2020
b853f5e
test
AngersZhuuuu Nov 21, 2020
c4543ef
hive version change with jdk version
AngersZhuuuu Nov 22, 2020
568150a
Update IsolatedClientLoader.scala
AngersZhuuuu Nov 22, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
test
  • Loading branch information
@AngersZhuuuu
AngersZhuuuu committed Nov 21, 2020
commit b853f5ee653d7afd00bce7644e07d121dc93abf8
18 changes: 15 additions & 3 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ private[hive] object IsolatedClientLoader extends Logging {
val hiveArtifacts = version.extraDeps ++
Seq("hive-metastore", "hive-exec", "hive-common", "hive-serde")
.map(a => s"org.apache.hive:$a:${version.fullVersion}") ++
Seq("com.google.guava:guava:14.0.1") ++ hadoopJarNames
Seq("com.google.guava:guava:14.0.1")

val extraExclusions = if (hadoopVersion.startsWith("3")) {
// this introduced from lower version of Hive could conflict with jars in Hadoop 3.2+, so
Expand All @@ -131,15 +131,27 @@ private[hive] object IsolatedClientLoader extends Logging {
Seq.empty
}

val classpath = quietly {
val hiveJarsClasspath = quietly {
SparkSubmitUtils.resolveMavenCoordinates(
hiveArtifacts.mkString(","),
SparkSubmitUtils.buildIvySettings(
Some(remoteRepos),
ivyPath),
exclusions = version.exclusions ++ extraExclusions)
}
val allFiles = classpath.split(",").map(new File(_)).toSet

val hadoopJarsClasspath = quietly {
SparkSubmitUtils.resolveMavenCoordinates(
hadoopJarNames.mkString(","),
SparkSubmitUtils.buildIvySettings(
Some(remoteRepos),
ivyPath),
exclusions = version.exclusions ++ extraExclusions)
}

val allFiles =
(hiveJarsClasspath.split(",") ++ hadoopJarsClasspath.split(","))
.map(new File(_)).toSet

// TODO: Remove copy logic.
val tempDir = Utils.createTempDir(namePrefix = s"hive-${version}")
Expand Down
1 change: 0 additions & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,6 @@ package object client {
// Since HIVE-14496, Hive materialized view need calcite-core.
// For spark, only VersionsSuite currently creates a hive materialized view for testing.
case object v2_3 extends HiveVersion("2.3.7",
extraDeps = Seq("org.apache.zookeeper:zookeeper:3.4.6"),
exclusions = Seq("org.apache.calcite:calcite-druid",
"org.apache.calcite.avatica:avatica",
"org.apache.curator:*",
Expand Down