Skip to content

[Spark-18408][ML] API Improvements for LSH #15874

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Yunni wants to merge 24 commits into from
Closed

[Spark-18408][ML] API Improvements for LSH #15874

Changes from 3 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
559c099
[SPARK-18334] MinHash should use binary hash distance
Nov 7, 2016
517a97b
Remove misleading documentation as requested
Yunni Nov 8, 2016
b546dbd
Add warning for multi-probe in MinHash
Nov 8, 2016
a3cd928
Merge branch 'SPARK-18334-yunn-minhash-bug' of https://github.com/Yun…
Nov 8, 2016
c8243c7
(1) Fix documentation as CR suggested (2) Fix typo in unit test
Nov 9, 2016
6aac8b3
Fix typo in unit test
Nov 9, 2016
9870743
[SPARK-18408] API Improvements for LSH
Nov 14, 2016
0e9250b
(1) Fix description for numHashFunctions (2) Make numEntries in MinHa…
Nov 14, 2016
adbbefe
Add assertion for hashFunction in BucketedRandomProjectionLSHSuite
Nov 14, 2016
c115ed3
Revert AND-amplification for a future PR
Nov 14, 2016
033ae5d
Code Review Comments
Nov 15, 2016
c597f4c
Add unit tests to run on Jenkins.
Nov 16, 2016
d759875
Add unit tests to run on Jenkins.
Nov 16, 2016
596eb06
CR comments
Nov 17, 2016
00d08bf
Merge branch 'master' of https://github.com/apache/spark into SPARK-1…
Nov 17, 2016
3d0810f
Update comments
Nov 17, 2016
257ef19
Add scaladoc for approximately min-wise independence
Yunni Nov 18, 2016
2c264b7
Change documentation reference
Yunni Nov 18, 2016
36ca278
Removing modulo numEntries
Nov 19, 2016
4508393
Merge branch 'SPARK-18408-yunn-api-improvements' of https://github.co…
Nov 19, 2016
939e9d5
Code Review Comments
Nov 22, 2016
8b9403d
Minimize the test cases by directly using artificial models
Nov 22, 2016
f0ebcb7
Code review comments
Nov 22, 2016
e198080
Merge branch 'master' of https://github.com/apache/spark into SPARK-1…
Yunni Nov 28, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion mllib/src/main/scala/org/apache/spark/ml/feature/MinHashLSH.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,15 @@ import org.apache.spark.sql.types.StructType
* :: Experimental ::
*
* Model produced by [[MinHashLSH]], where multiple hash functions are stored. Each hash function is
* picked from a hash function for a specific set `S` with cardinality equal to `numEntries`:
* picked from a hash family for a specific set `S` with cardinality equal to `numEntries`:
* `h_i(x) = ((x \cdot a_i + b_i) \mod prime) \mod numEntries`
Copy link
Contributor

@sethah sethah Nov 21, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remove the numEntries part here if we have removed it from the code. I still haven't gotten around to properly digging into this. For now, I'd like to change the second sentence to: "Each hash function is picked from the following family of hash functions, where a_i and b_i are randomly chosen integers less than prime:"

And I prefer this paper: "http://www.combinatorics.org/ojs/index.php/eljc/article/download/v7i1r26/pdf" as a reference because it is concise and easier to parse. That said, since it's a direct download link we could maybe not put the link in the doc, and just list the reference.

Copy link
Contributor Author

@Yunni Yunni Nov 22, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

*
* This hash family is approximately min-wise independent according to the reference.
*
* Reference:
* [[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.121.8215&rep=rep1&type=pdf Min-wise
* independent permutations]]
*
* @param randCoefficients Pairs of random coefficients. Each pair is used by one hash function.
*/
@Experimental
Expand Down