Skip to content
Closed
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix foreach
  • Loading branch information
jose-torres committed Mar 3, 2018
commit 4588616bea544deb0165986404ccc2918620b8d6
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,9 @@ object DataWritingSparkTask extends Logging {
try {
dataWriter = writeTask.createDataWriter(
context.partitionId(), context.attemptNumber(), currentEpoch)
iter.foreach(dataWriter.write)
while (iter.hasNext) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason to change foreach to a while loop?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC (/cc @tdas) foreach can be problematic in tight loops, because it introduces a lambda that isn't always optimized away.

dataWriter.write(iter.next())
}
logInfo(s"Writer for partition ${context.partitionId()} is committing.")
val msg = dataWriter.commit()
logInfo(s"Writer for partition ${context.partitionId()} committed.")
Expand All @@ -199,7 +201,8 @@ object DataWritingSparkTask extends Logging {
// Continuous shutdown always involves an interrupt. Just finish the task.
}
})(catchBlock = {
// If there is an error, abort this writer
// If there is an error, abort this writer. We enter this callback in the middle of
// rethrowing an exception, so runContinuous will stop executing at this point.
logError(s"Writer for partition ${context.partitionId()} is aborting.")
if (dataWriter != null) dataWriter.abort()
logError(s"Writer for partition ${context.partitionId()} aborted.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add comment that the exception will be rethrown.

Expand Down