Skip to content

[SPARK-20594][SQL]The staging directory should be a child directory starts with "." to avoid being deleted if we set hive.exec.stagingdir under the table directory. #17858

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zuotingbing wants to merge 9 commits into from
Closed

[SPARK-20594][SQL]The staging directory should be a child directory starts with "." to avoid being deleted if we set hive.exec.stagingdir under the table directory. #17858

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
c154f3a
[SPARK-20594]The staging directory should be appended with ".hive-sta…
May 4, 2017
de938ed
add a test case for SPARK-20594
May 8, 2017
6b22d3e
[SPARK-20594]The staging directory should be a child directory starts…
May 9, 2017
6b1b153
should add stripPrefix(File.separator)
May 9, 2017
2a542e4
Fix scala style checks
May 9, 2017
9f41436
Fix code review
May 10, 2017
bf1b4ec
Fix code review
May 11, 2017
4e1b6a0
Fix test case
May 11, 2017
639d63a
simplify the test case
May 12, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ case class InsertIntoHiveTable(
val externalCatalog = sparkSession.sharedState.externalCatalog
val hiveVersion = externalCatalog.asInstanceOf[HiveExternalCatalog].client.version
val hadoopConf = sessionState.newHadoopConf()
val stagingDir = hadoopConf.get("hive.exec.stagingdir", ".hive-staging")
val stagingDir = hadoopConf.get("hive.exec.stagingdir", ".hive-staging") + "/.hive-staging"
Copy link
Member

@gatorsmile gatorsmile May 8, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about?

    // SPARK-20594 After Hive 1.1, Hive will create the staging directory under the table directory,
    // and when moving staging directory to table directory, Hive will still empty the table
    // directory, but will exclude the staging directory, which start with "." or "_".
    val stagingDir =
      new Path(hadoopConf.get("hive.exec.stagingdir", ".hive-staging"), ".hive-staging").toString

val scratchDir = hadoopConf.get("hive.exec.scratchdir", "/tmp/hive")

val hiveQlTable = HiveClientImpl.toHiveTable(table)
Expand Down