Skip to content

[SPARK-19527][Core] Approximate Size of Intersection of Bloom Filters #16864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Bcpoole wants to merge 13 commits into from
Closed

[SPARK-19527][Core] Approximate Size of Intersection of Bloom Filters #16864

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
7a3ad46
Swamidass & Baldi approx. items in intersection of two Bloom filters.…
Bcpoole Feb 9, 2017
b9680c5
Changed createUnionBloomFilter & approxItemsInIntersection to be inst…
Bcpoole Feb 9, 2017
501ad7e
Updated abstract class to reflect changes in previous commit
Bcpoole Feb 9, 2017
c2a775d
renamed ‘createUnionBloomFilter’ to ‘createUnion’ as per @rxin sugges…
Bcpoole Feb 9, 2017
40a3954
renamed to to createUnion to union
Bcpoole Feb 9, 2017
ea554a6
Added some more javadoc
Bcpoole Feb 9, 2017
9cd7165
Readded ‘-‘ to start of approxItems() formula that somehow got removed
Bcpoole Feb 9, 2017
0ac7bf1
Handling of expected size being converted by BitArray concur
Bcpoole Feb 9, 2017
0a3910e
Added more description (formula + explanation) of approxItems and app…
Bcpoole Feb 10, 2017
8ec6092
Changed @seealso to @see. Added a note about approximation divergence
Bcpoole Feb 10, 2017
5e085c0
Wrote test for approxItems() being infinity when full
Bcpoole Feb 10, 2017
d1921f8
Added a non-full Bloom filter to testApproxItems
Bcpoole Feb 10, 2017
2cbf490
Added explanation as to why can’t run approxItems() directly on the i…
Bcpoole Feb 11, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Readded ‘-‘ to start of approxItems() formula that somehow got removed
  • Loading branch information
@Bcpoole
Bcpoole committed Feb 9, 2017
commit 9cd716506daaf3508a984713bc8a267e4d1bfcb2
2 changes: 1 addition & 1 deletion common/sketch/src/main/java/org/apache/spark/util/sketch/BloomFilterImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ public BloomFilter mergeInPlace(BloomFilter other) throws IncompatibleMergeExcep
@Override
public double approxItems() {
double m = bitSize();
return (m / numHashFunctions) * Math.log(1 - (bits.cardinality() / m));
return (-m / numHashFunctions) * Math.log(1 - (bits.cardinality() / m));
Copy link
Contributor

@mengxr mengxr Feb 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Math is deprecated. Use math.
  • Please add a test when bits is full. This should return Double.Infinity.

Copy link
Author

@Bcpoole Bcpoole Feb 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Math is deprecated. Use math.

Assume you were thinking of Scala?

}

@Override
Expand Down