Skip to content

[SPARK-23815][Core]Spark writer dynamic partition overwrite mode may fail to write output on multi level partition #20931

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
fangshil wants to merge 4 commits into from
Closed

[SPARK-23815][Core]Spark writer dynamic partition overwrite mode may fail to write output on multi level partition #20931

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
da63c17
Spark writer dynamic partition overwrite mode may fail to write outpu…
Mar 28, 2018
686a404
[SPARK-23815]follow up. add comment
Apr 7, 2018
a9342cc
[SPARK-23815]follow up. improve comment
Apr 7, 2018
08b9601
[SPARK-23815]follow up. further improve comment
Apr 7, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
[SPARK-23815]follow up. further improve comment
  • Loading branch information
Fangshi Li committed Apr 7, 2018
commit 08b9601ee9a0850f8fea0af7c9027bdef2016857
4 changes: 2 additions & 2 deletions core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,8 +190,8 @@ class HadoopMapReduceCommitProtocol(
// According to the official hadoop FileSystem API spec, delete op should assume
// the destination is no longer present regardless of return value, thus we do not
// need to double check if finalPartPath exists before rename.
// Also, in our case, delete should return false only when finalPartPath does not
// exist. When this happens, we need to take action only if parent of finalPartPath
// Also in our case, based on the spec, delete returns false only when finalPartPath
// does not exist. When this happens, we need to take action if parent of finalPartPath
// also does not exist(e.g. the scenario described on SPARK-23815), because
// FileSystem API spec on rename op says the rename dest(finalPartPath) must have
// a parent that exists, otherwise we may get unexpected result on the rename.
Expand Down