Skip to content

[SPARK-6259][MLlib] Python API for LDA #6791

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yu-iskw wants to merge 15 commits into from
Closed

[SPARK-6259][MLlib] Python API for LDA #6791

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
25ef2ac
Resolve conflicts with rebasing
yu-iskw Jul 2, 2015
68f0653
Support some parameters for `ALS.train()` in Python
yu-iskw Jun 12, 2015
77fd1b7
Not use LabeledPoint
yu-iskw Jun 19, 2015
8117e18
Fix the validation problems by `lint-scala`
yu-iskw Jun 23, 2015
39514ec
Modify how to cast the input data
yu-iskw Jun 23, 2015
2278829
Fix the indentation
yu-iskw Jun 23, 2015
73412c3
Fix the typo
yu-iskw Jun 23, 2015
57ac03d
Remove the unnecessary import in Python unit testing
yu-iskw Jun 23, 2015
98f645a
Remove the interface for `describeTopics`. Because it is not implemen…
yu-iskw Jun 23, 2015
faa9764
Add some comments for the LDA paramters
yu-iskw Jun 23, 2015
9f8bed8
Simplify casting
yu-iskw Jul 2, 2015
083e226
Add the comment about the supported values and the default value of `…
yu-iskw Jul 2, 2015
d7a332a
Remove the doc comment about the optimizer's default value
yu-iskw Jul 9, 2015
28bd165
Change the place of testing code
yu-iskw Jul 14, 2015
6855f59
LDA inherits object
yu-iskw Jul 14, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Not use LabeledPoint
  • Loading branch information
@yu-iskw
yu-iskw committed Jul 15, 2015
commit 77fd1b7b1c4960df79d0a597c0eae0010c35b666
13 changes: 8 additions & 5 deletions mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -506,7 +506,7 @@ private[python] class PythonMLLibAPI extends Serializable {
* Java stub for Python mllib LDA.run()
*/
def trainLDAModel(
data: JavaRDD[LabeledPoint],
data: JavaRDD[java.util.List[Any]],
k: Int,
maxIterations: Int,
docConcentration: Double,
Expand All @@ -524,11 +524,14 @@ private[python] class PythonMLLibAPI extends Serializable {

if (seed != null) algo.setSeed(seed)

try {
algo.run(data.rdd.map(x => (x.label.toLong, x.features)))
} finally {
data.rdd.unpersist(blocking = false)
val documents = data.rdd.map(_.asScala.toArray).map { r =>
r(0).getClass.getSimpleName match {
Copy link
Contributor

@davies davies Jun 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

r(0) match {
case i: java.lang.Integer => i.toLong
case l: java.lang.Long => l
}

case "Integer" => (r(0).asInstanceOf[java.lang.Integer].toLong, r(1).asInstanceOf[Vector])
case "Long" => (r(0).asInstanceOf[java.lang.Long].toLong, r(1).asInstanceOf[Vector])
case _ => throw new IllegalArgumentException("input values contains invalid type value.")
}
}
algo.run(documents)
}


Expand Down
4 changes: 2 additions & 2 deletions python/pyspark/mllib/clustering.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -593,8 +593,8 @@ class LDAModel(JavaModelWrapper):
>>> from collections import namedtuple
Copy link
Member

@jkbradley jkbradley Jun 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not needed

>>> from numpy.testing import assert_almost_equal
>>> data = [
... LabeledPoint(1, [0.0, 1.0]),
... LabeledPoint(2, [1.0, 0.0]),
... [1, Vectors.dense([0.0, 1.0])],
... [2, SparseVector(2, {0: 1.0})],
... ]
>>> rdd = sc.parallelize(data)
>>> model = LDA.train(rdd, k=2)
Expand Down