Skip to content

[SPARK-22465][Core] Add a safety-check to RDD defaultPartitioner #20002

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sujithjay wants to merge 12 commits into from
Closed

[SPARK-22465][Core] Add a safety-check to RDD defaultPartitioner #20002

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
176270b
[SPARK-22465][Core][WIP] Add a safety-check to RDD defaultPartitioner
Dec 16, 2017
be391a7
Merge remote-tracking branch 'origin-apache/master' into SPARK-22465
Dec 16, 2017
4b2dcac
Merge remote-tracking branch 'origin-apache/master' into SPARK-22465
Dec 17, 2017
ca6aa08
[SPARK-22465][Core][WIP] Fix Scala style checks
Dec 20, 2017
961e384
[SPARK-22465][Core][WIP] Changes after code review
Dec 23, 2017
8b35452
[SPARK-22465][Core][WIP] Scala style checks
Dec 23, 2017
4729d80
[SPARK-22465][Core][WIP] More Scala style checks
Dec 23, 2017
7d88e6c
Merge branch 'master' of github.com:apache/spark into SPARK-22465
Dec 23, 2017
6623227
Merge remote-tracking branch 'origin-apache/master' into SPARK-22465
Dec 23, 2017
3dd1ad8
[SPARK-22465][Core][WIP] Import scala.language.existentials to fix co…
Dec 23, 2017
62b17e9
[SPARK-22465][Core][WIP] Remove import scala.language.existentials; e…
Dec 24, 2017
3b08951
[SPARK-22465][Core][WIP] Add test for 9 vs 11 numPartitions case
Dec 24, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
[SPARK-22465][Core][WIP] Add test for 9 vs 11 numPartitions case
  • Loading branch information
sujithjay committed Dec 24, 2017
commit 3b089518e66bc4facf7bc07db1d12663dd567393
6 changes: 6 additions & 0 deletions core/src/test/scala/org/apache/spark/PartitioningSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,16 +268,22 @@ class PartitioningSuite extends SparkFunSuite with SharedSparkContext with Priva
val rdd3 = sc
.parallelize(Array((1, 6), (7, 8), (3, 10), (5, 12), (13, 14)))
.partitionBy(new HashPartitioner(100))
val rdd4 = sc
.parallelize(Array((1, 2), (2, 3), (2, 4), (3, 4)))
.partitionBy(new HashPartitioner(9))
val rdd5 = sc.parallelize((1 to 10).map(x => (x, x)), 11)

val partitioner1 = Partitioner.defaultPartitioner(rdd1, rdd2)
val partitioner2 = Partitioner.defaultPartitioner(rdd2, rdd3)
val partitioner3 = Partitioner.defaultPartitioner(rdd3, rdd1)
val partitioner4 = Partitioner.defaultPartitioner(rdd1, rdd2, rdd3)
val partitioner5 = Partitioner.defaultPartitioner(rdd4, rdd5)

assert(partitioner1.numPartitions == rdd1.getNumPartitions)
assert(partitioner2.numPartitions == rdd3.getNumPartitions)
assert(partitioner3.numPartitions == rdd3.getNumPartitions)
assert(partitioner4.numPartitions == rdd3.getNumPartitions)
Copy link
Contributor

@mridulm mridulm Dec 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a testcase such that numPartitions 9 vs 11 is not treated as an order of magnitude jump (to prevent future changes which end up breaking this).

assert(partitioner5.numPartitions == rdd4.getNumPartitions)

}
}
Expand Down