Skip to content

[SPARK-22465][Core] Add a safety-check to RDD defaultPartitioner #20002

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sujithjay wants to merge 12 commits into from
+78 −3
Closed

[SPARK-22465][Core] Add a safety-check to RDD defaultPartitioner #20002

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
176270b
[SPARK-22465][Core][WIP] Add a safety-check to RDD defaultPartitioner
Dec 16, 2017
be391a7
Merge remote-tracking branch 'origin-apache/master' into SPARK-22465
Dec 16, 2017
4b2dcac
Merge remote-tracking branch 'origin-apache/master' into SPARK-22465
Dec 17, 2017
ca6aa08
[SPARK-22465][Core][WIP] Fix Scala style checks
Dec 20, 2017
961e384
[SPARK-22465][Core][WIP] Changes after code review
Dec 23, 2017
8b35452
[SPARK-22465][Core][WIP] Scala style checks
Dec 23, 2017
4729d80
[SPARK-22465][Core][WIP] More Scala style checks
Dec 23, 2017
7d88e6c
Merge branch 'master' of github.com:apache/spark into SPARK-22465
Dec 23, 2017
6623227
Merge remote-tracking branch 'origin-apache/master' into SPARK-22465
Dec 23, 2017
3dd1ad8
[SPARK-22465][Core][WIP] Import scala.language.existentials to fix co…
Dec 23, 2017
62b17e9
[SPARK-22465][Core][WIP] Remove import scala.language.existentials; e…
Dec 24, 2017
3b08951
[SPARK-22465][Core][WIP] Add test for 9 vs 11 numPartitions case
Dec 24, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[SPARK-22465][Core][WIP] Scala style checks
  • Loading branch information
sujithjay committed Dec 23, 2017
commit 8b3545265b534e511ac947071e416360184d740e
3 changes: 2 additions & 1 deletion core/src/main/scala/org/apache/spark/Partitioner.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,8 @@ object Partitioner {
* less than and within a single order of magnitude of the max number of upstream partitions;
* otherwise, returns false
*/
private def isEligiblePartitioner(hasMaxPartitioner: Option[RDD[_]], rdds: Seq[RDD[_]]): Boolean = {
private def isEligiblePartitioner(hasMaxPartitioner: Option[RDD[_]],
rdds: Seq[RDD[_]]): Boolean = {
if(hasMaxPartitioner.isEmpty){
return false
}
Expand Down