Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Review comments fix
  • Loading branch information
Udbhav30 committed Aug 24, 2020
commit a2df53b48db372ed8f9a303cd8c0c499e33f3adf
7 changes: 4 additions & 3 deletions core/src/main/scala/org/apache/spark/util/Utils.scala
Original file line number Diff line number Diff line change
Expand Up @@ -270,20 +270,21 @@ private[spark] object Utils extends Logging {
}

/**
* Move data to trash on truncate table given
* 'spark.sql.truncate.trash.enabled' is true
* Move data to trash if 'spark.sql.truncate.trash.enabled' is true
*/
def moveToTrashIfEnabled(
fs: FileSystem,
partitionPath: Path,
isTrashEnabled: Boolean,
hadoopConf: Configuration): Unit = {
hadoopConf: Configuration): Boolean = {
if (isTrashEnabled) {
logDebug(s"will move data ${partitionPath.toString} to trash")
val isSuccess = Trash.moveToAppropriateTrash(fs, partitionPath, hadoopConf)
if (!isSuccess) {
logWarning(s"Failed to move data ${partitionPath.toString} to trash")
return fs.delete(partitionPath, true)
}
isSuccess
} else {
fs.delete(partitionPath, true)
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2729,6 +2729,7 @@ object SQLConf {
"fs.trash.interval, and in default, the server side configuration value takes " +
"precedence over the client-side one. Note that if fs.trash.interval is non-positive, " +
"this will be a no-op and log a warning message.")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a question. Is the following still true?

Note that if fs.trash.interval is non-positive, this will be a no-op and log a warning message.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, this is still true.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. This is because when fs.trash.interval is not positive, Hadoop side will consider trash as disabled and will not delete the data. See here. Currently this just logs a warning but we could consider another flag to hard delete the data instead.

.version("3.1.0")
.booleanConf
.createWithDefault(false)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3157,7 +3157,7 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
}
}

test("SPARK-32481 Donot move data to trash on truncate table if disabled") {
test("SPARK-32481 Do not move data to trash on truncate table if disabled") {
withTable("tab1") {
withSQLConf(SQLConf.TRUNCATE_TRASH_ENABLED.key -> "false") {
sql("CREATE TABLE tab1 (col INT) USING parquet")
Expand Down