[SPARK-8140] [MLlib] Remove empty model check in StreamingLinearAlgorithm

[MechCoder]

1. Prevent creating a map of data to find numFeatures
2. If model is empty, then initialize with a zero vector of numFeature

[srowen]

I see your intent here, but if the vectors are very large, it might actually be faster to get its size remotely and return only the size to the driver. But yes it entails a small distributed operation. I'm not sure which is better.

[MechCoder]

There is also a good possibility that input is very large. Is accessing the first element of a RDD slower than doing a distributed operation across the entire RDD?

[srowen]

It would only evaluate one element in this case. first() accesses the first partition and first element of its iterator, so map should only be applied once.

[MechCoder]

I see, I had thought, input.map(_.features.size) is evaluated first and then from that the first element is extracted. Should I revert this?

[srowen]

You could double-check with a very quick/small benchmark to see if it is in fact evaluated that way and if there is much difference at all. I suspect both are OK and don't know which is better.

[SparkQA]

Test build #34361 has finished for PR 6684 at commit e043b36.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #34362 has finished for PR 6684 at commit 1fa8501.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #34363 has finished for PR 6684 at commit d5d6178.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MechCoder]

@srowen I removed the None check and restored the model.isEmpty check.

[SparkQA]

Test build #34364 has finished for PR 6684 at commit 4f214fc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #34369 has finished for PR 6684 at commit 50ba5e2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #34370 has finished for PR 6684 at commit 7fbf5f9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

I'm OK with this. CC @mengxr and/or @jkbradley for a second look, so I'll leave it open a little while.

[MechCoder]

Also if input.first().features.size is better than input.map(_.features.size).first()

[jkbradley]

Sorry for the slow response! I think it looks OK, though I wonder if the incomplete match causes a compilation warning.

[srowen]

Hm, good point. That results in a warning actually, but no runtime problem since the model must exist here. Really this whole construct can be removed now; I should have seen that. Well we can follow this up with that additional change to further simplify and remove a warning.