Skip to content

[SPARK-25542][Core][Test] Move flaky test in OpenHashMapSuite to OpenHashSetSuite and make it against OpenHashSet #22569

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 3 commits into from
Closed

[SPARK-25542][Core][Test] Move flaky test in OpenHashMapSuite to OpenHashSetSuite and make it against OpenHashSet #22569

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
39a77d6
Move 12M capability test from OpenHashMapSuite to OpenHashSetSuite.
viirya Sep 27, 2018
dca4e5c
Add one more check.
viirya Sep 27, 2018
22bfa80
Change for comment.
viirya Sep 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 0 additions & 10 deletions core/src/test/scala/org/apache/spark/util/collection/OpenHashMapSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,16 +185,6 @@ class OpenHashMapSuite extends SparkFunSuite with Matchers {
assert(map.contains(null))
}

test("support for more than 12M items") {
val cnt = 12000000 // 12M
val map = new OpenHashMap[Int, Int](cnt)
for (i <- 0 until cnt) {
map(i) = 1
}
val numInvalidValues = map.iterator.count(_._2 == 0)
assertResult(0)(numInvalidValues)
}

test("distinguish between the 0/0.0/0L and null") {
val specializedMap1 = new OpenHashMap[String, Long]
specializedMap1("a") = null.asInstanceOf[Long]
Expand Down
13 changes: 13 additions & 0 deletions core/src/test/scala/org/apache/spark/util/collection/OpenHashSetSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,4 +255,17 @@ class OpenHashSetSuite extends SparkFunSuite with Matchers {
val set = new OpenHashSet[Long](0)
assert(set.size === 0)
}

test("support for more than 12M items") {
val cnt = 12000000 // 12M
val set = new OpenHashSet[Int](cnt)
for (i <- 0 until cnt) {
set.add(i)
assert(set.contains(i))

val pos1 = set.getPos(i)
val pos2 = set.addWithoutResize(i) & OpenHashSet.POSITION_MASK
assert(pos1 == pos2)
}
Copy link
Member

@kiszk kiszk Sep 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Is it better to add the following to check each value after adding all, too?

for (i <- 0 until cnt) {
  assert(set.contains(i))
}

Copy link
Member Author

@viirya viirya Sep 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to add the check, we can also add it inside the loop. Another loop seems unnecessary to me.

Copy link
Member

@kiszk kiszk Sep 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My intention was to verify whether small values (e.g. 0, 1, 2, 3) are still valid after several resizings.

Copy link
Member Author

@viirya viirya Sep 28, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I think that is not what this wants test and there are other tests to make sure it works. If you insist I can add it too.

Copy link
Member

@kiszk kiszk Sep 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original test performed a check by using map.iterator.count(). But, this is not a strong preference.

Copy link
Member Author

@viirya viirya Sep 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, the original test performed the count check to see how many values are invalid. They are invalid values because the index is wrong due to wrong position mask in OpenHashSet.

This rewritten test tests directly the index of OpenHashSet.

}
}