[SPARK-9883][MLlib] Distance to each kmean cluster given a point

[BertrandDechoux]

First step done by @tex0l ;)

[mengxr]

does it compile?

[BertrandDechoux]

Indeed not. The diff I received must have been a work in progress. This is now fixed.

[tex0l]

Indeed it was... Sorry about that.

2015-09-21 14:01 GMT-07:00 Bertrand Dechoux [email protected]:

In
mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala
#8849 (comment):

@@ -77,6 +77,22 @@ class KMeansModel @SInCE("1.1.0") (@SInCE("1.0.0") val clusterCenters: Array[Vec
def predict(points: JavaRDD[Vector]): JavaRDD[java.lang.Integer] =
predict(points.rdd).toJavaRDD().asInstanceOf[JavaRDD[java.lang.Integer]]

• /** */
• def distanceToCenters(point: Vector): (Int, Double) = {

Indeed not. The diff I received must have been a work in progress. This is
now fixed.

—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/8849/files#r40025294.

Timothée Rebours
13, rue Georges Bizet
78380 BOUGIVAL

[mengxr]

ok to test

[BertrandDechoux]

ERROR: Timeout after 15 minutes
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/apache/spark.git

Somehow I am glad that it does not only happen to my projects.

The patch is still basic and need a few changes

• cluster index instead of cluster location?
• Since annotation (1.5.0?)
  but the tests shouldn't fail.

[SparkQA]

Test build #42773 has finished for PR 8849 at commit 7a746d1.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[BertrandDechoux]

ERROR: Timeout after 15 minutes
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/apache/spark.git

[SparkQA]

Test build #42775 has finished for PR 8849 at commit beed882.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[BertrandDechoux]

I think I still need to change the method signature.

Predict on a RDD is the following

def predict(points: RDD[Vector]): RDD[Int]

Distances on a RDD is the following

def distanceToCenters(points: RDD[Vector]): RDD[(Vector, Iterable[(VectorWithNorm, Double)])]

I am more confortable with having the input point in the output but predict does no work like that.
If you say the predict template should be followed, I will change the method signature.
But having to rely on the order to match them with the results is not natural for me.

The second aspect is that I may want to output cluster indices instead of their locations.

I would be glad to have your point of view @mengxr.

[BertrandDechoux]

ERROR: Timeout after 15 minutes
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/apache/spark.git

[SparkQA]

Test build #42856 has finished for PR 8849 at commit a193011.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #42854 has finished for PR 8849 at commit d97d8f8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[BertrandDechoux]

ERROR: Timeout after 15 minutes
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/apache/spark.git

[BertrandDechoux]

• Both methods now return the cluster indices.
• I have added a toList call to make sure there is no laziness involved but that's maybe not necessary.
• Still need a feedback for the return type of

def distanceToCenters(points: RDD[Vector]): RDD[(Vector, Iterable[(VectorWithNorm, Double)])]

[yu-iskw]

Jenkins, test this please

[SparkQA]

Test build #44797 has finished for PR 8849 at commit 159608a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yu-iskw]

@BertrandDechoux thank you for the update. Sounds interesting, but I don't think we should calculate distances between a point and all centers. Personally, I think we should calculate the distance between a point and the closest center.

And I'm wondering if the method should return the pair of cluster index and distance or only distance.

@mengxr @jkbradley what do you think?

[BertrandDechoux]

In a perfect world, each point belongs to a specific cluster and the number of clusters is easy to find. In reality, it is less so. Knowing the distance is a way to appreciate the closeness of a point with regard to a cluster.

K-means can be thought as a special mixture model. When using a mixture model, the impact of each 'cluster' with regard to a specific point is an important information. I think the same holds true for K-means.

But, in the end, it does depend in which context and how you are using K-means indeed.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

I think this should just be closed. I don't think there's enough value in adding this API now.