Skip to content

[SPARK-29786][SQL] Fix MetaException when dropping a partition not exists on HDFS #26422

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Deegue wants to merge 14 commits into from
Closed

[SPARK-29786][SQL] Fix MetaException when dropping a partition not exists on HDFS #26422

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
90233e4
Merge pull request #1 from apache/master
Deegue Oct 22, 2019
7a246d5
Merge pull request #2 from apache/master
Deegue Oct 28, 2019
783211f
Merge pull request #3 from apache/master
Deegue Nov 5, 2019
355496e
Fix drop partition MetaException:File does not exist.
Deegue Nov 7, 2019
74d5984
Add notes.
Deegue Nov 14, 2019
84b1da8
1584531350
Deegue Mar 18, 2020
248b6cf
UT failed #119927, getDataLocation->getPartitionPath
Deegue Mar 20, 2020
c20c390
UT failed #120076, add HIDDEN_FILES_PATH_FILTER
Deegue Mar 20, 2020
9d75854
optimize
Deegue Mar 20, 2020
5a5ff74
UT failed #120087. Hive-0.13 doesn't have `Partition.getPartitionPath…
Deegue Mar 20, 2020
f4b3793
UT failed #120093, purge and ignoreIfNotExists should be false.
Deegue Mar 20, 2020
4d1c35e
isEmptyPath->isExistPath
Deegue Mar 27, 2020
7cf7c6b
remove redundant code
Deegue Mar 27, 2020
f8b96de
restore '!'
Deegue May 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
UT failed #120076, add HIDDEN_FILES_PATH_FILTER
  • Loading branch information
@Deegue
Deegue committed Mar 20, 2020
commit c20c390acd00f2f3adc219ba373fc5f619c2299a
11 changes: 9 additions & 2 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ import scala.collection.JavaConverters._
import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer

import org.apache.hadoop.fs.Path
import org.apache.hadoop.fs.{Path, PathFilter}
import org.apache.hadoop.hive.common.{FileUtils, StatsSetupConst}
import org.apache.hadoop.hive.conf.HiveConf
import org.apache.hadoop.hive.conf.HiveConf.ConfVars
Expand Down Expand Up @@ -616,11 +616,18 @@ private[hive] class HiveClientImpl(
shim.createPartitions(client, db, table, parts, ignoreIfExists)
}

val HIDDEN_FILES_PATH_FILTER: PathFilter = new PathFilter() {
override def accept(p: Path): Boolean = {
val name = p.getName
!name.startsWith("_") && !name.startsWith(".")
}
}

@throws[Exception]
private def isEmptyPath(dirPath: Path): Boolean = {
val inpFs = dirPath.getFileSystem(conf)
if (inpFs.exists(dirPath)) {
val fStats = inpFs.listStatus(dirPath, FileUtils.HIDDEN_FILES_PATH_FILTER)
val fStats = inpFs.listStatus(dirPath, HIDDEN_FILES_PATH_FILTER)
if (fStats.nonEmpty) {
return false
}
Expand Down