Skip to content

[SPARK-7913][Core]Make AppendOnlyMap use the same growth strategy of OpenHashSet and consistent exception message #6879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zsxwing wants to merge 2 commits into from
Closed

[SPARK-7913][Core]Make AppendOnlyMap use the same growth strategy of OpenHashSet and consistent exception message #6879

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dd4385b
Make AppendOnlyMap use the same growth strategy of OpenHashSet and co…
zsxwing Jun 18, 2015
912c0ad
Fix the doc
zsxwing Jun 18, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Make AppendOnlyMap use the same growth strategy of OpenHashSet and co… 
…nsistent exception message
  • Loading branch information
@zsxwing
zsxwing committed Jun 18, 2015
commit dd4385b4652dc353574dea360f2c7bc65e413ac1
8 changes: 3 additions & 5 deletions core/src/main/scala/org/apache/spark/util/collection/AppendOnlyMap.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,11 +199,8 @@ class AppendOnlyMap[K, V](initialCapacity: Int = 64)

/** Increase table size by 1, rehashing if necessary */
private def incrementSize() {
if (curSize == MAXIMUM_CAPACITY) {
throw new IllegalStateException(s"Can't put more that ${MAXIMUM_CAPACITY} elements")
}
curSize += 1
if (curSize > growThreshold && capacity < MAXIMUM_CAPACITY) {
if (curSize > growThreshold) {
growTable()
}
}
Expand All @@ -216,7 +213,8 @@ class AppendOnlyMap[K, V](initialCapacity: Int = 64)
/** Double the table's size and re-hash everything */
protected def growTable() {
// capacity < MAXIMUM_CAPACITY (2 ^ 29) so capacity * 2 won't overflow
val newCapacity = (capacity * 2).min(MAXIMUM_CAPACITY)
val newCapacity = capacity * 2
require(newCapacity <= MAXIMUM_CAPACITY, s"Can't contain more than ${growThreshold} elements")
Copy link
Member

@srowen srowen Jun 18, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose this doesn't have the same overflow issue if the max is 2^29, OK. Yes nice to try to make these a little more consistent.

val newData = new Array[AnyRef](2 * newCapacity)
val newMask = newCapacity - 1
// Insert all our old values into the new array. Note that because our old keys are
Expand Down