Skip to content

[SPARK-25299] Implement default version of the API for shuffle writes #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ifilonenko merged 17 commits into from
Mar 27, 2019
Merged

[SPARK-25299] Implement default version of the API for shuffle writes #6

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
7160ce3
musings without unit tests
ifilonenko Mar 14, 2019
460f0ea
first pass of comments
ifilonenko Mar 20, 2019
96d1774
resolve some comments
ifilonenko Mar 22, 2019
64fb327
small style choice with java.io.
ifilonenko Mar 22, 2019
996e903
resolved more comments
ifilonenko Mar 22, 2019
3b9d33c
initial unit tests of outputstream writing
ifilonenko Mar 23, 2019
1f1c159
slight update
ifilonenko Mar 23, 2019
0737515
added tests and resolved comments
ifilonenko Mar 23, 2019
1ded83d
initial comments resolved
ifilonenko Mar 25, 2019
3353155
trying to change API to return FileChannel instead of WriteableByteCh…
ifilonenko Mar 25, 2019
7a79bd9
small comments to resolve
ifilonenko Mar 25, 2019
9e3f05c
fix to issue of closing channel
ifilonenko Mar 25, 2019
9f6230b
Various changes to the API and state assumptions in writers. (#8)
mccheah Mar 26, 2019
14df750
small comment addiiton
ifilonenko Mar 26, 2019
8cf80f7
resolving comments
ifilonenko Mar 26, 2019
46a0174
resolving comments
ifilonenko Mar 26, 2019
1325903
added null check
ifilonenko Mar 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
added null check
  • Loading branch information
@ifilonenko
ifilonenko committed Mar 26, 2019
commit 1325903dc34cd7c92dbdd20df2a95a665a914eae
12 changes: 7 additions & 5 deletions core/src/main/java/org/apache/spark/shuffle/sort/io/DefaultShuffleMapOutputWriter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,11 +93,13 @@ public void abort(Throwable error) {
} catch (Exception e) {
log.error("Unable to close appropriate underlying file stream", e);
}
if (!outputTempFile.delete() && outputTempFile.exists()) {
log.warn("Failed to delete temporary shuffle file at {}", outputTempFile.getAbsolutePath());
}
if (!outputFile.delete() && outputFile.exists()) {
log.warn("Failed to delete output shuffle file at {}", outputFile.getAbsolutePath());
if (outputTempFile != null) {
if (!outputTempFile.delete() && outputTempFile.exists()) {
Copy link

@mccheah mccheah Mar 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switch the ordering of the statements in this boolean condition.

log.warn("Failed to delete temporary shuffle file at {}", outputTempFile.getAbsolutePath());
}
if (!outputFile.delete() && outputFile.exists()) {
Copy link

@mccheah mccheah Mar 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I don't think we want to delete here - a concurrent attempt shouldn't have its results removed if this attempt fails. I think this lines up with the behavior from the original shuffle writer.

log.warn("Failed to delete output shuffle file at {}", outputFile.getAbsolutePath());
}
}
}

Expand Down