[SPARK-10188] [Pyspark] Pyspark CrossValidator with RMSE selects incorrect model

[noel-smith]

• Added isLargerBetter() method to Pyspark Evaluator to match the Scala version.
• JavaEvaluator delegates isLargerBetter() to underlying Scala object.
• Added check for isLargerBetter() in CrossValidator to determine whether to use argmin or argmax.
• Added test cases for where smaller is better (RMSE) and larger is better (R-Squared).

(This contribution is my original work and that I license the work to the project under Sparks' open source license)

[jkbradley]

ok to test

[jkbradley]

Thanks for the PR! I'll take a look now.

[jkbradley]

Can you please copy the doc from Scala here? (no need to copy to child classes since the "inherit_doc" tag will handle that)

[jkbradley]

Looks good except for those 2 items.

[SparkQA]

Test build #1695 has finished for PR 8399 at commit 7794cf7.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #1699 has finished for PR 8399 at commit 7794cf7.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

Ping @mengxr In case I can't check this soon, it would be great to get this into 1.5 if there is an RC3.

[SparkQA]

Test build #1700 has finished for PR 8399 at commit bada453.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[noel-smith]

That would be great - I've just messaged him. If there are any other changes you need to get this into 1.5 I'll get them in ASAP today.

[jkbradley]

No, this LGTM. I'll merge this with branch-1.5 and master now. Thanks very much!