Skip to content

[MLlib] [SPARK-2510]Word2Vec: Distributed Representation of Words #1719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Ishiihara wants to merge 15 commits into from
Closed

[MLlib] [SPARK-2510]Word2Vec: Distributed Representation of Words #1719

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
8d6befe
initial commit
Aug 1, 2014
0aafb1b
Add comments, minor fixes
Aug 1, 2014
e4a04d3
minor fix
Aug 1, 2014
57dc50d
code formatting
Aug 1, 2014
2e92b59
modify according to feedback
Aug 2, 2014
720b5a3
Add test for Word2Vec algorithm, minor fixes
Aug 2, 2014
6bcc8be
add multiple iteration support
Aug 3, 2014
7efbb6f
use broadcast version of vocab in aggregate
Aug 3, 2014
1a8fb41
use weighted sum in combOp
Aug 3, 2014
e93e726
use treeAggregate instead of aggregate
Aug 4, 2014
384c771
remove minCount and window from constructor
mengxr Aug 4, 2014
c14da41
fix styles
mengxr Aug 4, 2014
26a948d
Merge pull request #1 from mengxr/Ishiihara-master
Ishiihara Aug 4, 2014
e248441
minor style change
Aug 4, 2014
2ba9483
minor fix for Word2Vec test
Aug 4, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
use treeAggregate instead of aggregate
  • Loading branch information
Liquan Pei committed Aug 4, 2014
commit e93e7263d74879379257e6fff40d5efc8417f2ce
7 changes: 4 additions & 3 deletions mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import org.apache.spark.SparkContext._
import org.apache.spark.mllib.linalg.{Vector, Vectors}
import org.apache.spark.HashPartitioner
import org.apache.spark.storage.StorageLevel
import org.apache.spark.mllib.rdd.RDDFunctions._
/**
* Entry in vocabulary
*/
Expand Down Expand Up @@ -111,9 +112,9 @@ class Word2Vec(
}

private def learnVocabPerPartition(words:RDD[String]) {

}

private def createExpTable(): Array[Double] = {
val expTable = new Array[Double](EXP_TABLE_SIZE)
var i = 0
Expand Down Expand Up @@ -254,7 +255,7 @@ class Word2Vec(
val (aggSyn0, aggSyn1, _, _) =
// TODO: broadcast temp instead of serializing it directly
// or initialize the model in each executor
newSentences.aggregate((syn0Global.clone(), syn1Global.clone(), 0, 0))(
newSentences.treeAggregate((syn0Global.clone(), syn1Global.clone(), 0, 0))(
seqOp = (c, v) => (c, v) match {
case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
var lwc = lastWordCount
Expand Down