Skip to content

Commit e232720

Browse files
hhbyyhmengxr
hhbyyh
authored and
mengxr
committed
[SPARK-11689][ML] Add user guide and example code for LDA under spark.ml
jira: https://issues.apache.org/jira/browse/SPARK-11689 Add simple user guide for LDA under spark.ml and example code under examples/. Use include_example to include example code in the user guide markdown. Check SPARK-11606 for instructions. Original PR is reverted due to document build error. apache#9722 mengxr feynmanliang yinxusen Sorry for the troubling. Author: Yuhao Yang <hhbyyh@gmail.com> Closes apache#9974 from hhbyyh/ldaMLExample.
1 parent a8ceec5 commit e232720

File tree

5 files changed

+208
-1
lines changed

5 files changed

+208
-1
lines changed

‎docs/ml-clustering.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
---
2+
layout: global
3+
title: Clustering - ML
4+
displayTitle: <a href="ml-guide.html">ML</a> - Clustering
5+
---
6+
7+
In this section, we introduce the pipeline API for [clustering in mllib](mllib-clustering.html).
8+
9+
## Latent Dirichlet allocation (LDA)
10+
11+
`LDA` is implemented as an `Estimator` that supports both `EMLDAOptimizer` and `OnlineLDAOptimizer`,
12+
and generates a `LDAModel` as the base models. Expert users may cast a `LDAModel` generated by
13+
`EMLDAOptimizer` to a `DistributedLDAModel` if needed.
14+
15+
<div class="codetabs">
16+
17+
<div data-lang="scala" markdown="1">
18+
19+
Refer to the [Scala API docs](api/scala/index.html#org.apache.spark.ml.clustering.LDA) for more details.
20+
21+
{% include_example scala/org/apache/spark/examples/ml/LDAExample.scala %}
22+
</div>
23+
24+
<div data-lang="java" markdown="1">
25+
26+
Refer to the [Java API docs](api/java/org/apache/spark/ml/clustering/LDA.html) for more details.
27+
28+
{% include_example java/org/apache/spark/examples/ml/JavaLDAExample.java %}
29+
</div>
30+
31+
</div>

‎docs/ml-guide.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ Also, some algorithms have additional capabilities in the `spark.ml` API; e.g.,
4040
provide class probabilities, and linear models provide model summaries.
4141

4242
* [Feature extraction, transformation, and selection](ml-features.html)
43+
* [Clustering](ml-clustering.html)
4344
* [Decision Trees for classification and regression](ml-decision-tree.html)
4445
* [Ensembles](ml-ensembles.html)
4546
* [Linear methods with elastic net regularization](ml-linear-methods.html)
@@ -950,4 +951,4 @@ model.transform(test)
950951
{% endhighlight %}
951952
</div>
952953

953-
</div>
954+
</div>

‎docs/mllib-guide.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ We list major functionality from both below, with links to detailed guides.
6969
concepts. It also contains sections on using algorithms within the Pipelines API, for example:
7070

7171
* [Feature extraction, transformation, and selection](ml-features.html)
72+
* [Clustering](ml-clustering.html)
7273
* [Decision trees for classification and regression](ml-decision-tree.html)
7374
* [Ensembles](ml-ensembles.html)
7475
* [Linear methods with elastic net regularization](ml-linear-methods.html)

‎examples/src/main/java/org/apache/spark/examples/ml/JavaLDAExample.java‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
package org.apache.spark.examples.ml;
19+
// $example on$
20+
import java.util.regex.Pattern;
21+
22+
import org.apache.spark.SparkConf;
23+
import org.apache.spark.api.java.JavaRDD;
24+
import org.apache.spark.api.java.JavaSparkContext;
25+
import org.apache.spark.api.java.function.Function;
26+
import org.apache.spark.ml.clustering.LDA;
27+
import org.apache.spark.ml.clustering.LDAModel;
28+
import org.apache.spark.mllib.linalg.Vector;
29+
import org.apache.spark.mllib.linalg.VectorUDT;
30+
import org.apache.spark.mllib.linalg.Vectors;
31+
import org.apache.spark.sql.DataFrame;
32+
import org.apache.spark.sql.Row;
33+
import org.apache.spark.sql.SQLContext;
34+
import org.apache.spark.sql.catalyst.expressions.GenericRow;
35+
import org.apache.spark.sql.types.Metadata;
36+
import org.apache.spark.sql.types.StructField;
37+
import org.apache.spark.sql.types.StructType;
38+
// $example off$
39+
40+
/**
41+
* An example demonstrating LDA
42+
* Run with
43+
* <pre>
44+
* bin/run-example ml.JavaLDAExample
45+
* </pre>
46+
*/
47+
public class JavaLDAExample {
48+
49+
// $example on$
50+
private static class ParseVector implements Function<String, Row> {
51+
private static final Pattern separator = Pattern.compile(" ");
52+
53+
@Override
54+
public Row call(String line) {
55+
String[] tok = separator.split(line);
56+
double[] point = new double[tok.length];
57+
for (int i = 0; i < tok.length; ++i) {
58+
point[i] = Double.parseDouble(tok[i]);
59+
}
60+
Vector[] points = {Vectors.dense(point)};
61+
return new GenericRow(points);
62+
}
63+
}
64+
65+
public static void main(String[] args) {
66+
67+
String inputFile = "data/mllib/sample_lda_data.txt";
68+
69+
// Parses the arguments
70+
SparkConf conf = new SparkConf().setAppName("JavaLDAExample");
71+
JavaSparkContext jsc = new JavaSparkContext(conf);
72+
SQLContext sqlContext = new SQLContext(jsc);
73+
74+
// Loads data
75+
JavaRDD<Row> points = jsc.textFile(inputFile).map(new ParseVector());
76+
StructField[] fields = {new StructField("features", new VectorUDT(), false, Metadata.empty())};
77+
StructType schema = new StructType(fields);
78+
DataFrame dataset = sqlContext.createDataFrame(points, schema);
79+
80+
// Trains a LDA model
81+
LDA lda = new LDA()
82+
.setK(10)
83+
.setMaxIter(10);
84+
LDAModel model = lda.fit(dataset);
85+
86+
System.out.println(model.logLikelihood(dataset));
87+
System.out.println(model.logPerplexity(dataset));
88+
89+
// Shows the result
90+
DataFrame topics = model.describeTopics(3);
91+
topics.show(false);
92+
model.transform(dataset).show(false);
93+
94+
jsc.stop();
95+
}
96+
// $example off$
97+
}

‎examples/src/main/scala/org/apache/spark/examples/ml/LDAExample.scala‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
package org.apache.spark.examples.ml
19+
20+
// scalastyle:off println
21+
import org.apache.spark.{SparkContext, SparkConf}
22+
import org.apache.spark.mllib.linalg.{VectorUDT, Vectors}
23+
// $example on$
24+
import org.apache.spark.ml.clustering.LDA
25+
import org.apache.spark.sql.{Row, SQLContext}
26+
import org.apache.spark.sql.types.{StructField, StructType}
27+
// $example off$
28+
29+
/**
30+
* An example demonstrating a LDA of ML pipeline.
31+
* Run with
32+
* {{{
33+
* bin/run-example ml.LDAExample
34+
* }}}
35+
*/
36+
object LDAExample {
37+
38+
final val FEATURES_COL = "features"
39+
40+
def main(args: Array[String]): Unit = {
41+
42+
val input = "data/mllib/sample_lda_data.txt"
43+
// Creates a Spark context and a SQL context
44+
val conf = new SparkConf().setAppName(s"${this.getClass.getSimpleName}")
45+
val sc = new SparkContext(conf)
46+
val sqlContext = new SQLContext(sc)
47+
48+
// $example on$
49+
// Loads data
50+
val rowRDD = sc.textFile(input).filter(_.nonEmpty)
51+
.map(_.split(" ").map(_.toDouble)).map(Vectors.dense).map(Row(_))
52+
val schema = StructType(Array(StructField(FEATURES_COL, new VectorUDT, false)))
53+
val dataset = sqlContext.createDataFrame(rowRDD, schema)
54+
55+
// Trains a LDA model
56+
val lda = new LDA()
57+
.setK(10)
58+
.setMaxIter(10)
59+
.setFeaturesCol(FEATURES_COL)
60+
val model = lda.fit(dataset)
61+
val transformed = model.transform(dataset)
62+
63+
val ll = model.logLikelihood(dataset)
64+
val lp = model.logPerplexity(dataset)
65+
66+
// describeTopics
67+
val topics = model.describeTopics(3)
68+
69+
// Shows the result
70+
topics.show(false)
71+
transformed.show(false)
72+
73+
// $example off$
74+
sc.stop()
75+
}
76+
}
77+
// scalastyle:on println

0 commit comments

Comments
 (0)