Skip to content

[SPARK-16575][CORE] partition calculation mismatch with sc.binaryFiles #15327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
fidato13 wants to merge 8 commits into from
Closed

[SPARK-16575][CORE] partition calculation mismatch with sc.binaryFiles #15327

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
823aba1
[SPARK-16575] [spark core] partition calculation mismatch with sc.bin…
fidato13 Oct 2, 2016
73ac673
Merge branch 'master' of https://github.com/apache/spark into SPARK-1…
fidato13 Oct 13, 2016
83a5a7a
[SPARK-16575][CORE] ADD cost of opening a file calcuation for BinaryF…
fidato13 Oct 13, 2016
8b2d27b
Merge branch 'master' of https://github.com/apache/spark into SPARK-1…
fidato13 Oct 13, 2016
7b57840
Fix for failing test case.
fidato13 Oct 16, 2016
4ebf501
Merge branch 'master' of https://github.com/apache/spark into SPARK-1…
fidato13 Oct 16, 2016
53d9d27
targets with meta-annotations - SPARK-12527, for Spark Context in Bin…
fidato13 Nov 3, 2016
18a8554
Merge branch 'master' of https://github.com/apache/spark into SPARK-1…
fidato13 Nov 3, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
[SPARK-16575] [spark core] partition calculation mismatch with sc.bin… 
…aryFiles
  • Loading branch information
@fidato13
fidato13 committed Oct 2, 2016
commit 823aba185c562d08474c1aca846bf8de467beee4
5 changes: 3 additions & 2 deletions core/src/main/scala/org/apache/spark/input/PortableDataStream.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,9 @@ private[spark] abstract class StreamFileInputFormat[T]
* which is set through setMaxSplitSize
*/
def setMinPartitions(context: JobContext, minPartitions: Int) {
val totalLen = listStatus(context).asScala.filterNot(_.isDirectory).map(_.getLen).sum
val maxSplitSize = math.ceil(totalLen / math.max(minPartitions, 1.0)).toLong
val files = listStatus(context).asScala
val totalLen = files.filterNot(_.isDirectory).map(_.getLen).sum
val maxSplitSize = math.ceil(totalLen / math.max(minPartitions, files.size)).toLong
super.setMaxSplitSize(maxSplitSize)
}

Expand Down