Skip to content

[Spark-18080][ML][PYTHON] Python API & Examples for Locality Sensitive Hashing #16715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Yunni wants to merge 27 commits into from
Closed

[Spark-18080][ML][PYTHON] Python API & Examples for Locality Sensitive Hashing #16715

Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
85d22c3
Locality Sensitive Hashing (LSH) Python API.
yanboliang Nov 4, 2016
cdeca1c
Fix typos.
yanboliang Nov 4, 2016
66d308b
Merge branch 'spark-18080' of https://github.com/yanboliang/spark int…
Jan 25, 2017
d62a2d0
Merge branch 'master' of https://github.com/apache/spark into spark-1…
Jan 26, 2017
dafc4d1
Changes to fix LSH Python API
Jan 26, 2017
ac1f4f7
Merge branch 'spark-18080' of https://github.com/Yunni/spark into spa…
Yunni Jan 26, 2017
3a21f26
Fix examples and class definition
Yunni Jan 26, 2017
65dab3e
Add python examples and updated the user guide
Jan 26, 2017
3d3bcf0
Fix lint issues
Jan 26, 2017
69dccde
Fix python doc issues
Jan 26, 2017
e7542d0
Fix 'Definition list ends without a blank line'
Jan 26, 2017
5cfc9c5
Fix python unit tests
Jan 26, 2017
ccabbf4
Merge branch 'master' of https://github.com/apache/spark into spark-1…
Feb 7, 2017
2508a2f
Code Review Comments
Feb 8, 2017
2dd6aad
Merge branch 'master' of https://github.com/apache/spark into spark-1…
Feb 8, 2017
8e5468f
Add printing messages for the LSH Scala/Java/Python exmaples
Feb 8, 2017
6e85e1a
(1) Rename 'keys''values' to 'features''hashes' (2) Printing the ids …
Feb 8, 2017
4bc670c
Fix jenkins build
Feb 9, 2017
b45ec0a
Fix failed jenkins test
Feb 9, 2017
1b70b91
Fix Jenkins test
Feb 9, 2017
b1da01e
Code Review Comments for the LSH examples
Feb 10, 2017
8f1d708
Add alias for similarity join examples
Feb 10, 2017
49edc93
Merge branch 'master' of https://github.com/apache/spark into spark-1…
Feb 14, 2017
c64d50b
Code Review Comments
Feb 14, 2017
5d55752
Code Review Comments: Some minor fixes
Feb 14, 2017
d849c3a
Code Review Comment
Feb 15, 2017
36fd9bc
Merge branch 'master' of https://github.com/apache/spark into spark-1…
Feb 15, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix Jenkins test
  • Loading branch information
Yun Ni committed Feb 9, 2017
commit 1b70b919edea26321f21220f11d520d4f4f98ede
2 changes: 1 addition & 1 deletion python/pyspark/ml/feature.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -974,7 +974,7 @@ class MinHashLSH(JavaEstimator, LSHParams, HasInputCol, HasOutputCol, HasSeed,
... (4, Vectors.sparse(6, [2, 3, 5], [1.0, 1.0, 1.0]),),
... (5, Vectors.sparse(6, [1, 2, 4], [1.0, 1.0, 1.0]),)]
>>> df2 = spark.createDataFrame(data2, ["id", "features"])
>>> key = Vectors.sparse(6, [1], [1.0])
>>> key = Vectors.sparse(6, [1, 2], [1.0, 1.0])
>>> model.approxNearestNeighbors(df2, key, 1).collect()
[Row(id=5, features=SparseVector(6, {1: 1.0, 2: 1.0, 4: 1.0}), hashes=[DenseVector([-163892...
>>> model.approxSimilarityJoin(df, df2, 0.6).select("datasetA.id",
Copy link
Contributor

@sethah sethah Feb 9, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above, except distcol could be "jaccardSimilarity" or something.

Expand Down