Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
85d22c3
Locality Sensitive Hashing (LSH) Python API.
yanboliang Nov 4, 2016
cdeca1c
Fix typos.
yanboliang Nov 4, 2016
66d308b
Merge branch 'spark-18080' of https://github.com/yanboliang/spark int…
Jan 25, 2017
d62a2d0
Merge branch 'master' of https://github.com/apache/spark into spark-1…
Jan 26, 2017
dafc4d1
Changes to fix LSH Python API
Jan 26, 2017
ac1f4f7
Merge branch 'spark-18080' of https://github.com/Yunni/spark into spa…
Yunni Jan 26, 2017
3a21f26
Fix examples and class definition
Yunni Jan 26, 2017
65dab3e
Add python examples and updated the user guide
Jan 26, 2017
3d3bcf0
Fix lint issues
Jan 26, 2017
69dccde
Fix python doc issues
Jan 26, 2017
e7542d0
Fix 'Definition list ends without a blank line'
Jan 26, 2017
5cfc9c5
Fix python unit tests
Jan 26, 2017
ccabbf4
Merge branch 'master' of https://github.com/apache/spark into spark-1…
Feb 7, 2017
2508a2f
Code Review Comments
Feb 8, 2017
2dd6aad
Merge branch 'master' of https://github.com/apache/spark into spark-1…
Feb 8, 2017
8e5468f
Add printing messages for the LSH Scala/Java/Python exmaples
Feb 8, 2017
6e85e1a
(1) Rename 'keys''values' to 'features''hashes' (2) Printing the ids …
Feb 8, 2017
4bc670c
Fix jenkins build
Feb 9, 2017
b45ec0a
Fix failed jenkins test
Feb 9, 2017
1b70b91
Fix Jenkins test
Feb 9, 2017
b1da01e
Code Review Comments for the LSH examples
Feb 10, 2017
8f1d708
Add alias for similarity join examples
Feb 10, 2017
49edc93
Merge branch 'master' of https://github.com/apache/spark into spark-1…
Feb 14, 2017
c64d50b
Code Review Comments
Feb 14, 2017
5d55752
Code Review Comments: Some minor fixes
Feb 14, 2017
d849c3a
Code Review Comment
Feb 15, 2017
36fd9bc
Merge branch 'master' of https://github.com/apache/spark into spark-1…
Feb 15, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Locality Sensitive Hashing (LSH) Python API.
  • Loading branch information
yanboliang committed Nov 4, 2016
commit 85d22c37d3fe0b907f2eaf892729d087f9efb76c
26 changes: 13 additions & 13 deletions mllib/src/main/scala/org/apache/spark/ml/feature/LSH.scala
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,9 @@ private[ml] trait LSHParams extends HasInputCol with HasOutputCol {
* higher the dimension is, the lower the false negative rate.
* @group param
*/
final val outputDim: IntParam = new IntParam(this, "outputDim", "output dimension, where" +
"increasing dimensionality lowers the false negative rate, and decreasing dimensionality" +
" improves the running performance", ParamValidators.gt(0))
final val outputDim: IntParam = new IntParam(this, "outputDim", "The output dimension, where" +
" increasing dimensionality lowers the false negative rate, and decreasing dimensionality" +
" improves the running performance.", ParamValidators.gt(0))

/** @group getParam */
final def getOutputDim: Int = $(outputDim)
Expand Down Expand Up @@ -109,11 +109,11 @@ private[ml] abstract class LSHModel[T <: LSHModel[T]]
* - Single Probing: Fast, return at most k elements (Probing only one buckets)
* - Multiple Probing: Slow, return exact k elements (Probing multiple buckets close to the key)
*
* @param dataset the dataset to search for nearest neighbors of the key
* @param key Feature vector representing the item to search for
* @param numNearestNeighbors The maximum number of nearest neighbors
* @param singleProbing True for using Single Probing; false for multiple probing
* @param distCol Output column for storing the distance between each result row and the key
* @param dataset The dataset to search for nearest neighbors of the key.
* @param key Feature vector representing the item to search for.
* @param numNearestNeighbors The maximum number of nearest neighbors.
* @param singleProbing True for using Single Probing; false for multiple probing.
* @param distCol Output column for storing the distance between each result row and the key.
* @return A dataset containing at most k items closest to the key. A distCol is added to show
* the distance between each row and the key.
*/
Expand Down Expand Up @@ -215,12 +215,12 @@ private[ml] abstract class LSHModel[T <: LSHModel[T]]
* [[outputCol]] exists, it will use the [[outputCol]]. This allows caching of the transformed
* data when necessary.
*
* @param datasetA One of the datasets to join
* @param datasetB Another dataset to join
* @param threshold The threshold for the distance of row pairs
* @param distCol Output column for storing the distance between each result row and the key
* @param datasetA One of the datasets to join.
* @param datasetB Another dataset to join.
* @param threshold The threshold for the distance of row pairs.
* @param distCol Output column for storing the distance between each result row and the key.
* @return A joined dataset containing pairs of rows. The original rows are in columns
* "datasetA" and "datasetB", and a distCol is added to show the distance of each pair
* "datasetA" and "datasetB", and a distCol is added to show the distance of each pair.
*/
def approxSimilarityJoin(
datasetA: Dataset[_],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ import org.apache.spark.sql.types.StructType
*
* Model produced by [[MinHash]], where multiple hash functions are stored. Each hash function is
* a perfect hash function:
* `h_i(x) = (x * k_i mod prime) mod numEntries`
* where `k_i` is the i-th coefficient, and both `x` and `k_i` are from `Z_prime^*`
* `h_i(x) = (x * k_i \mod prime) \mod numEntries`
* where `k_i` is the i-th coefficient, and both `x` and `k_i` are from `Z_{prime^*}`
*
* Reference:
* [[https://en.wikipedia.org/wiki/Perfect_hash_function Wikipedia on Perfect Hash Function]]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ private[ml] trait RandomProjectionParams extends Params {
*
* Model produced by [[RandomProjection]], where multiple random vectors are stored. The vectors
* are normalized to be unit vectors and each vector is used in a hash function:
* `h_i(x) = floor(r_i.dot(x) / bucketLength)`
* `h_i(x) = floor(r_i * x / bucketLength)`
* where `r_i` is the i-th random unit vector. The number of buckets will be `(max L2 norm of input
* vectors) / bucketLength`.
*
Expand Down
Loading