Skip to content

[SPARK-18081][ML][DOCS] Add user guide for Locality Sensitive Hashing(LSH) #15795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Yunni wants to merge 12 commits into from
Closed

[SPARK-18081][ML][DOCS] Add user guide for Locality Sensitive Hashing(LSH) #15795

Changes from 2 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
8c7971b
[SPARK-18081] Add user guide to LSH
Yunni Nov 7, 2016
84877ee
Code Review Comments: (1) Fix indention for Java Examples (2) Rephras…
Nov 7, 2016
19404c4
Merge branch 'master' of https://github.com/Yunni/spark into SPARK-18…
Nov 8, 2016
6654f8b
Improve documenatation about multi-probe nearest neighbor
Nov 8, 2016
40a0caa
Changes in docs for the API Improvements.
Yunni Nov 28, 2016
a78d920
Merge branch 'master' of https://github.com/apache/spark into SPARK-1…
Yunni Nov 29, 2016
7e60b76
Changes of the examples after API improvements
Yunni Nov 29, 2016
b9f716d
Changes of the examples after API improvements
Yunni Nov 29, 2016
19653d1
Merge branch 'SPARK-18081-lsh-guide' of https://github.com/Yunni/spar…
Yunni Dec 2, 2016
a048194
minor updates to lsh docs
jkbradley Dec 3, 2016
7922117
Merge pull request #1 from jkbradley/Yunni-SPARK-18081-lsh-guide
Yunni Dec 3, 2016
7c09f9a
Capitalize first letter of each word in title
Yunni Dec 3, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 13 additions & 38 deletions examples/src/main/java/org/apache/spark/examples/ml/JavaMinHashLSHExample.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,8 @@

import org.apache.spark.ml.feature.MinHashLSH;
import org.apache.spark.ml.feature.MinHashLSHModel;
import org.apache.spark.ml.linalg.Vector;
import org.apache.spark.ml.linalg.Vectors;
import org.apache.spark.ml.linalg.VectorUDT;
import org.apache.spark.ml.linalg.Vectors;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.RowFactory;
Expand All @@ -45,49 +44,25 @@ public static void main(String[] args) {
.getOrCreate();

// $example on$
List<Row> dataA = Arrays.asList(
RowFactory.create(0, Vectors.sparse(6, new int[]{0, 1, 2}, new double[]{1.0, 1.0, 1.0})),
RowFactory.create(1, Vectors.sparse(6, new int[]{2, 3, 4}, new double[]{1.0, 1.0, 1.0})),
RowFactory.create(2, Vectors.sparse(6, new int[]{0, 2, 4}, new double[]{1.0, 1.0, 1.0}))
);

List<Row> dataB = Arrays.asList(
RowFactory.create(3, Vectors.sparse(6, new int[]{1, 3, 5}, new double[]{1.0, 1.0, 1.0})),
RowFactory.create(4, Vectors.sparse(6, new int[]{2, 3, 5}, new double[]{1.0, 1.0, 1.0})),
RowFactory.create(5, Vectors.sparse(6, new int[]{1, 2, 4}, new double[]{1.0, 1.0, 1.0}))
List<Row> data = Arrays.asList(
RowFactory.create(0, Vectors.sparse(6, new int[]{0, 1, 2}, new double[]{1.0, 1.0, 1.0})),
RowFactory.create(1, Vectors.sparse(6, new int[]{2, 3, 4}, new double[]{1.0, 1.0, 1.0})),
RowFactory.create(2, Vectors.sparse(6, new int[]{0, 2, 4}, new double[]{1.0, 1.0, 1.0}))
);

StructType schema = new StructType(new StructField[]{
new StructField("id", DataTypes.IntegerType, false, Metadata.empty()),
new StructField("keys", new VectorUDT(), false, Metadata.empty())
new StructField("id", DataTypes.IntegerType, false, Metadata.empty()),
new StructField("keys", new VectorUDT(), false, Metadata.empty())
});
Dataset<Row> dfA = spark.createDataFrame(dataA, schema);
Dataset<Row> dfB = spark.createDataFrame(dataB, schema);

Vector key = Vectors.sparse(6, new int[]{1, 3}, new double[]{1.0, 1.0, 1.0});
Dataset<Row> dataFrame = spark.createDataFrame(data, schema);

MinHashLSH mh = new MinHashLSH()
.setNumHashTables(3)
.setInputCol("keys")
.setOutputCol("values");

MinHashLSHModel model = mh.fit(dfA);

// Feature Transformation
model.transform(dfA).show();
// Cache the transformed columns
Dataset<Row> transformedA = model.transform(dfA).cache();
Dataset<Row> transformedB = model.transform(dfB).cache();

// Approximate similarity join
model.approxSimilarityJoin(dfA, dfB, 0.6).show();
model.approxSimilarityJoin(transformedA, transformedB, 0.6).show();
// Self Join
model.approxSimilarityJoin(dfA, dfA, 0.6).filter("datasetA.id < datasetB.id").show();
.setNumHashTables(1)
.setInputCol("keys")
.setOutputCol("values");

// Approximate nearest neighbor search
model.approxNearestNeighbors(dfA, key, 2).show();
model.approxNearestNeighbors(transformedA, key, 2).show();
MinHashLSHModel model = mh.fit(dataFrame);
model.transform(dataFrame).show();
// $example off$

spark.stop();
Expand Down