Skip to content

[SPARK-5992][ML] Locality Sensitive Hashing #15148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Yunni wants to merge 46 commits into from
Closed

[SPARK-5992][ML] Locality Sensitive Hashing #15148

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
1bbd48c
First Commit of LSH function implementation. Implement basic Estimato…
Yunni Sep 13, 2016
ca46d82
Implementation of Approximate Nearest Neighbors. Add distCol as anoth…
Yunni Sep 13, 2016
c693f5b
Implement approxSimilarityJoin(). Remove modelDataset and distCol as …
Yunni Sep 15, 2016
c9ee0f9
Add test utility method to check LSH property. Tested on random proje…
Yunni Sep 19, 2016
fc838e0
Add testing utility for approximate nearest neighbor. Run the testing…
Yunni Sep 19, 2016
aa138e8
Add testing utility for approximate similarity join. Run the testing …
Yunni Sep 19, 2016
bbcbcf0
Code review comments. A new unit test of k nearest neighbor for large k
Sep 19, 2016
d389159
Code review comments. A new unit test of k nearest neighbor for large k
Sep 19, 2016
19d012a
(1) Refactor hashDistCol for nearest neighbor search. (2) Add scalado…
Sep 19, 2016
269c8c9
Code Review comments: (1) Rewrite hashDistance (2) Move the lsh packa…
Yunni Sep 20, 2016
9065f7d
Add comment to clarify the implementation of RandomProjection
Yunni Sep 20, 2016
d22dff4
Implementation of MinHash with unit tests
Yunni Sep 26, 2016
7e6d938
Add options for Probing Single/Multiple bucket(s) in approxNearestNei…
Yunni Sep 26, 2016
0fad3ef
Allow users to transform datasets themselves before doing approxNeare…
Yunni Sep 26, 2016
0080b87
Generalize Input types to Vector. For MinHash, use Sparse Vectors to …
Yunni Sep 28, 2016
a1c344b
Code Review Comments
Yunni Sep 28, 2016
396ad60
Bug fixed. Typo of distCol
Yunni Sep 28, 2016
b79ebbd
Fix Jenkins Build. Explicitly annotate type of modelDataset
Yunni Sep 28, 2016
7936315
Move all code to org.apache.spark.ml.feature
Yunni Sep 28, 2016
f805658
Tune threshold for approxNearestNeighbors unit tests
Yunni Sep 28, 2016
8f04ee8
Fix import ordering
Yunni Sep 28, 2016
f82f3fe
Add scaladoc for overloaded methods
Yunni Sep 28, 2016
ccd98f7
Code review comments
Yunni Oct 4, 2016
69efc84
Move private[ml] to MinHash constructor
Yunni Oct 4, 2016
eced98d
Detailed doc on bucketLength. Move private[ml] to Model constructor
Yunni Oct 4, 2016
3487bcc
Tune threshold for MinHash
Oct 4, 2016
df19886
Code review comments
Oct 5, 2016
efe323c
Code Review Comments
Yunni Oct 10, 2016
142d8e9
Revert unrelated changes
Yunni Oct 10, 2016
40d1f1b
Code review comments for MinHash: (1) Compute unionSize based on setS…
Oct 10, 2016
2c95e5c
Code review comments
Yunni Oct 11, 2016
fb120af
SignRandomProjection: LSH Classes for cosine distance metrics
Oct 11, 2016
19f6d89
Change hashFunctions to Arrays
Oct 11, 2016
1b63173
BitSampling: LSH Class for Hamming Distance
Oct 12, 2016
126d980
Merge remote-tracking branch 'upstream/master' into SPARK-5992-yunn-lsh
Yunni Oct 13, 2016
a35e261
Move distinct() before calculating the distance to improve running time
Yunni Oct 13, 2016
66d553a
For similarity join, expose leftCol and rightCol as parameters
Yunni Oct 17, 2016
cad4ecb
Code Review comments: (1) Save BitSampling and SignRandomProjection f…
Yunni Oct 22, 2016
e14f73e
(1) Reset all random seed != 0 (2) Add docstring about the output sch…
Yunni Oct 23, 2016
1c4b9fb
(1) Add readers/writers (2) Change unit tests thresholds to more rebo…
Oct 27, 2016
20a9ebf
Change a few Since annotations
Oct 27, 2016
9bb3fd6
Code Review Comments: (1) Remove all Since in LSH (2) Add doc on hash…
Oct 27, 2016
9a3704c
Organize the scaladoc
Oct 27, 2016
6cda936
Remove default values for outputCol
Oct 28, 2016
97e1238
Remove default values for outputCol
Oct 28, 2016
3570845
Add more scaladoc
Oct 28, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Code Review comments: (1) Rewrite hashDistance (2) Move the lsh packa… 
…ge to be under feature
  • Loading branch information
@Yunni
Yunni committed Sep 20, 2016
commit 269c8c91dfbc20d84a4e2e658a910b5adc68314c
5 changes: 3 additions & 2 deletions ...n/scala/org/apache/spark/ml/lsh/LSH.scala → ...org/apache/spark/ml/feature/lsh/LSH.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.ml.lsh
package org.apache.spark.ml.feature.lsh

import scala.util.Random

Expand Down Expand Up @@ -95,7 +95,8 @@ abstract class LSHModel[KeyType, T <: LSHModel[KeyType, T]] private[ml]
* @return The distance between hash vectors x and y in double
*/
protected[ml] def hashDistance(x: Vector, y: Vector): Double = {
(x.asBreeze - y.asBreeze).toArray.map(math.abs).min
// Since it's generated by hashing, it will be a pair of dense vectors.
x.toDense.values.zip(y.toDense.values).map(x => math.abs(x._1 - x._2)).min
Copy link

@karlhigley karlhigley Sep 25, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default, this is computing the Manhattan distance between hash values, which probably works as a proxy for the distance between hash buckets when using LSH based on p-stable distributions and any other approach that produces vectors of integers/doubles as hash signatures (e.g. MinHash).

However, the default won't work for approaches that produce vectors of booleans as hash signatures (e.g. sign random projection for cosine distance). It could be overridden to compute Hamming distance in that case, though.

Copy link
Contributor Author

@Yunni Yunni Sep 26, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I am planning to override it for BitSampling (LSH for Hamming distance)

Copy link
Member

@jkbradley jkbradley Sep 26, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's algorithm-specific, I'd recommend making it abstract here so it's more future bug-proof.

Copy link
Contributor Author

@Yunni Yunni Sep 28, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

}

/**
Expand Down
2 changes: 1 addition & 1 deletion ...pache/spark/ml/lsh/RandomProjection.scala → ...ark/ml/feature/lsh/RandomProjection.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.ml.lsh
package org.apache.spark.ml.feature.lsh

import scala.util.Random

Expand Down
2 changes: 1 addition & 1 deletion ...ala/org/apache/spark/ml/lsh/LSHTest.scala → ...apache/spark/ml/feature/lsh/LSHTest.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.ml.lsh
package org.apache.spark.ml.feature.lsh

import org.apache.spark.ml.linalg.Vector
import org.apache.spark.sql.Dataset
Expand Down
2 changes: 1 addition & 1 deletion .../spark/ml/lsh/RandomProjectionSuite.scala → ...l/feature/lsh/RandomProjectionSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.ml.lsh
package org.apache.spark.ml.feature.lsh

import breeze.numerics.{cos, sin}
import breeze.numerics.constants.Pi
Expand Down