Skip to content

[SPARK-7913][Core]Make AppendOnlyMap use the same growth strategy of OpenHashSet and consistent exception message #6879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zsxwing wants to merge 2 commits into from
Closed

[SPARK-7913][Core]Make AppendOnlyMap use the same growth strategy of OpenHashSet and consistent exception message #6879

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dd4385b
Make AppendOnlyMap use the same growth strategy of OpenHashSet and co…
zsxwing Jun 18, 2015
912c0ad
Fix the doc
zsxwing Jun 18, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 4 additions & 6 deletions core/src/main/scala/org/apache/spark/util/collection/AppendOnlyMap.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ import org.apache.spark.annotation.DeveloperApi
* size, which is guaranteed to explore all spaces for each key (see
* http://en.wikipedia.org/wiki/Quadratic_probing).
*
* The map can support up to `536870912 (2 ^ 29)` elements.
* The map can support up to `375809638 (0.7 * 2 ^ 29)` elements.
*
* TODO: Cache the hash values of each key? java.util.HashMap does that.
*/
Expand Down Expand Up @@ -199,11 +199,8 @@ class AppendOnlyMap[K, V](initialCapacity: Int = 64)

/** Increase table size by 1, rehashing if necessary */
private def incrementSize() {
if (curSize == MAXIMUM_CAPACITY) {
throw new IllegalStateException(s"Can't put more that ${MAXIMUM_CAPACITY} elements")
}
curSize += 1
if (curSize > growThreshold && capacity < MAXIMUM_CAPACITY) {
if (curSize > growThreshold) {
growTable()
}
}
Expand All @@ -216,7 +213,8 @@ class AppendOnlyMap[K, V](initialCapacity: Int = 64)
/** Double the table's size and re-hash everything */
protected def growTable() {
// capacity < MAXIMUM_CAPACITY (2 ^ 29) so capacity * 2 won't overflow
val newCapacity = (capacity * 2).min(MAXIMUM_CAPACITY)
val newCapacity = capacity * 2
require(newCapacity <= MAXIMUM_CAPACITY, s"Can't contain more than ${growThreshold} elements")
Copy link
Member

@srowen srowen Jun 18, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose this doesn't have the same overflow issue if the max is 2^29, OK. Yes nice to try to make these a little more consistent.

val newData = new Array[AnyRef](2 * newCapacity)
val newMask = newCapacity - 1
// Insert all our old values into the new array. Note that because our old keys are
Expand Down