Skip to content

[SPARK-23510][SQL] Support Hive 2.2 and Hive 2.3 metastore #20671

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gatorsmile wants to merge 5 commits into from
Closed

[SPARK-23510][SQL] Support Hive 2.2 and Hive 2.3 metastore #20671

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5b1fc01
Support Hive 2.2 and Hive 2.3
wangyum Feb 24, 2018
48343bc
Revert HIVE_STATS_RETRIES_WAIT
wangyum Feb 24, 2018
4db3dc9
Update HiveClientVersions.scala and HiveVersionSuite.scala
wangyum Feb 25, 2018
0fdf29b
fix.
gatorsmile Feb 25, 2018
2e12c10
add one more end-to-end test
gatorsmile Feb 25, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update HiveClientVersions.scala and HiveVersionSuite.scala
  • Loading branch information
@wangyum
wangyum committed Feb 25, 2018
commit 4db3dc9a1a4f6f83289f21a61d0ef51da3a3e232
8 changes: 2 additions & 6 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -880,9 +880,7 @@ private[client] class Shim_v0_14 extends Shim_v0_13 {

}

private[client] class Shim_v1_0 extends Shim_v0_14 {

}
private[client] class Shim_v1_0 extends Shim_v0_14

private[client] class Shim_v1_1 extends Shim_v1_0 {

Expand Down Expand Up @@ -1147,9 +1145,7 @@ private[client] class Shim_v2_1 extends Shim_v2_0 {
}
}

private[client] class Shim_v2_2 extends Shim_v2_1 {

}
private[client] class Shim_v2_2 extends Shim_v2_1

private[client] class Shim_v2_3 extends Shim_v2_2 {

Expand Down
3 changes: 2 additions & 1 deletion sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientVersions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,5 +22,6 @@ import scala.collection.immutable.IndexedSeq
import org.apache.spark.SparkFunSuite

private[client] trait HiveClientVersions {
protected val versions = IndexedSeq("0.12", "0.13", "0.14", "1.0", "1.1", "1.2", "2.0", "2.1")
protected val versions =
IndexedSeq("0.12", "0.13", "0.14", "1.0", "1.1", "1.2", "2.0", "2.1", "2.2", "2.3")
}
2 changes: 1 addition & 1 deletion sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveVersionSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ private[client] abstract class HiveVersionSuite(version: String) extends SparkFu
// Hive changed the default of datanucleus.schema.autoCreateAll from true to false and
// hive.metastore.schema.verification from false to true since 2.0
// For details, see the JIRA HIVE-6113 and HIVE-12463
if (version == "2.0" || version == "2.1") {
if (version == "2.0" || version == "2.1" || version == "2.2" || version == "2.3") {
hadoopConf.set("datanucleus.schema.autoCreateAll", "true")
hadoopConf.set("hive.metastore.schema.verification", "false")
}
Expand Down
2 changes: 1 addition & 1 deletion sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ class VersionsSuite extends SparkFunSuite with Logging {
// Hive changed the default of datanucleus.schema.autoCreateAll from true to false and
// hive.metastore.schema.verification from false to true since 2.0
// For details, see the JIRA HIVE-6113 and HIVE-12463
if (version.split("\\.").head.toInt > 1) {
if (version == "2.0" || version == "2.1" || version == "2.2" || version == "2.3") {
hadoopConf.set("datanucleus.schema.autoCreateAll", "true")
hadoopConf.set("hive.metastore.schema.verification", "false")
}
Expand Down