Skip to content

[SPARK-19527][Core] Approximate Size of Intersection of Bloom Filters #16864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Bcpoole wants to merge 13 commits into from
Closed

[SPARK-19527][Core] Approximate Size of Intersection of Bloom Filters #16864

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
7a3ad46
Swamidass & Baldi approx. items in intersection of two Bloom filters.…
Bcpoole Feb 9, 2017
b9680c5
Changed createUnionBloomFilter & approxItemsInIntersection to be inst…
Bcpoole Feb 9, 2017
501ad7e
Updated abstract class to reflect changes in previous commit
Bcpoole Feb 9, 2017
c2a775d
renamed ‘createUnionBloomFilter’ to ‘createUnion’ as per @rxin sugges…
Bcpoole Feb 9, 2017
40a3954
renamed to to createUnion to union
Bcpoole Feb 9, 2017
ea554a6
Added some more javadoc
Bcpoole Feb 9, 2017
9cd7165
Readded ‘-‘ to start of approxItems() formula that somehow got removed
Bcpoole Feb 9, 2017
0ac7bf1
Handling of expected size being converted by BitArray concur
Bcpoole Feb 9, 2017
0a3910e
Added more description (formula + explanation) of approxItems and app…
Bcpoole Feb 10, 2017
8ec6092
Changed @seealso to @see. Added a note about approximation divergence
Bcpoole Feb 10, 2017
5e085c0
Wrote test for approxItems() being infinity when full
Bcpoole Feb 10, 2017
d1921f8
Added a non-full Bloom filter to testApproxItems
Bcpoole Feb 10, 2017
2cbf490
Added explanation as to why can’t run approxItems() directly on the i…
Bcpoole Feb 11, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Added some more javadoc
  • Loading branch information
@Bcpoole
Bcpoole committed Feb 9, 2017
commit ea554a632f46d66503a1f78624498cfdba3fc3e6
6 changes: 5 additions & 1 deletion common/sketch/src/main/java/org/apache/spark/util/sketch/BloomFilter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,12 +157,16 @@ int getVersionNumber() {
* Unlike mergeInplace, this will not cause a mutation.
* Callers must ensure the bloom filters are appropriately sized to avoid saturating them.
*
* @throws IncompatibleUnionException if either are null, different classes, or different size or number of hash functions
* @param other The bloom filter to union this bloom filter with.
* @throws IncompatibleUnionException if {@code isCompatible(other) == false}
*/
public abstract BloomFilterImpl union(BloomFilter other) throws IncompatibleUnionException;

/**
* Swamidass & Baldi (2007) approximation for number of items in the intersection of two Bloom filters
Copy link
Contributor

@mengxr mengxr Feb 10, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Same here. Document the method first and then mention the reference.
  • How is it different from intersecting two bloom filters and then estimate the number of items? Union might lead to larger approximation error. Okay, I got why. Please also document it.

*
* @param other The bloom filter to intersect this bloom filter with.
* @throws IncompatibleUnionException if {@code isCompatible(other) == false}
*/
public abstract double approxItemsInIntersection(BloomFilter that) throws IncompatibleUnionException;

Expand Down