Skip to content

[SPARK-22465][Core] Add a safety-check to RDD defaultPartitioner #20002

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sujithjay wants to merge 12 commits into from
Closed

[SPARK-22465][Core] Add a safety-check to RDD defaultPartitioner #20002

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
176270b
[SPARK-22465][Core][WIP] Add a safety-check to RDD defaultPartitioner
Dec 16, 2017
be391a7
Merge remote-tracking branch 'origin-apache/master' into SPARK-22465
Dec 16, 2017
4b2dcac
Merge remote-tracking branch 'origin-apache/master' into SPARK-22465
Dec 17, 2017
ca6aa08
[SPARK-22465][Core][WIP] Fix Scala style checks
Dec 20, 2017
961e384
[SPARK-22465][Core][WIP] Changes after code review
Dec 23, 2017
8b35452
[SPARK-22465][Core][WIP] Scala style checks
Dec 23, 2017
4729d80
[SPARK-22465][Core][WIP] More Scala style checks
Dec 23, 2017
7d88e6c
Merge branch 'master' of github.com:apache/spark into SPARK-22465
Dec 23, 2017
6623227
Merge remote-tracking branch 'origin-apache/master' into SPARK-22465
Dec 23, 2017
3dd1ad8
[SPARK-22465][Core][WIP] Import scala.language.existentials to fix co…
Dec 23, 2017
62b17e9
[SPARK-22465][Core][WIP] Remove import scala.language.existentials; e…
Dec 24, 2017
3b08951
[SPARK-22465][Core][WIP] Add test for 9 vs 11 numPartitions case
Dec 24, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[SPARK-22465][Core][WIP] Import scala.language.existentials to fix co… 
…mpiler warnings
  • Loading branch information
sujithjay committed Dec 23, 2017
commit 3dd1ad8e25b7c23b58d33cc422570f4cb133fd4b
1 change: 1 addition & 0 deletions core/src/main/scala/org/apache/spark/Partitioner.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import java.io.{IOException, ObjectInputStream, ObjectOutputStream}

import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer
import scala.language.existentials
Copy link
Contributor

@mridulm mridulm Dec 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, why was this required ?

Copy link
Author

@sujithjay sujithjay Dec 23, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this import, there was a compiler warning:

Warning:(66, 29) inferred existential type Option[org.apache.spark.rdd.RDD[_$2]]( forSome { type _$2 }), which cannot be expressed by wildcards,  should be enabled
by making the implicit value scala.language.existentials visible.
This can be achieved by adding the import clause 'import scala.language.existentials'
or by setting the compiler option -language:existentials.
See the Scaladoc for value scala.language.existentials for a discussion
why the feature should be explicitly enabled.

The build on Jenkins failed because of this warning.

Copy link
Contributor

@mridulm mridulm Dec 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we explicitly set the type, is it still required ? For example, with val hasMaxPartitioner: Option[RDD[_]] = ... ?

import scala.math.log10
import scala.reflect.ClassTag
import scala.util.hashing.byteswap32
Expand Down