[SPARK-8711] [ML] Add additional methods to PySpark ML tree models #7095

MechCoder · 2015-06-29T19:45:49Z

Add numNodes and depth to treeModels, add treeWeights to ensemble Models.
Add repr to all models.

AmplabJenkins · 2015-06-29T19:47:14Z

Merged build triggered.

AmplabJenkins · 2015-06-29T19:47:20Z

Merged build started.

MechCoder · 2015-06-29T19:48:38Z

@mengxr Should we add wrapper around the trees methos in ensembles as well? I'm not sure how we would manipulate java objects in python.

AmplabJenkins · 2015-06-29T19:52:12Z

Merged build triggered.

AmplabJenkins · 2015-06-29T19:52:20Z

Merged build started.

SparkQA · 2015-06-29T19:55:19Z

Test build #36023 has started for PR 7095 at commit 849aabe.

AmplabJenkins · 2015-06-29T20:02:32Z

Merged build finished. Test FAILed.

SparkQA · 2015-06-29T20:06:46Z

Test build #36023 has finished for PR 7095 at commit 849aabe.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(DecisionTreeModel):
- class RandomForestClassificationModel(treeEnsembleModels):
- class GBTClassificationModel(treeEnsembleModels):
- class DecisionTreeModel(JavaModel):
- class treeEnsembleModels(JavaModel):
- class DecisionTreeRegressionModel(DecisionTreeModel):
- class RandomForestRegressionModel(treeEnsembleModels):
- class GBTRegressionModel(treeEnsembleModels):

AmplabJenkins · 2015-06-29T20:07:09Z

Merged build finished. Test PASSed.

feynmanliang · 2015-06-29T23:47:06Z

python/pyspark/ml/regression.py

Floating point equality is causing this test to fail. Try numpy.allclose

MechCoder · 2015-06-30T04:59:33Z

@feynmanliang Thanks. I have addressed your comments.

AmplabJenkins · 2015-06-30T05:02:11Z

Merged build triggered.

AmplabJenkins · 2015-06-30T05:02:17Z

Merged build started.

SparkQA · 2015-06-30T05:05:31Z

Test build #36090 has started for PR 7095 at commit 2fde31a.

SparkQA · 2015-06-30T05:16:27Z

Test build #36090 has finished for PR 7095 at commit 2fde31a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(DecisionTreeModel):
- class RandomForestClassificationModel(TreeEnsembleModels):
- class GBTClassificationModel(TreeEnsembleModels):
- class DecisionTreeModel(JavaModel):
- class TreeEnsembleModels(JavaModel):
- class DecisionTreeRegressionModel(DecisionTreeModel):
- class RandomForestRegressionModel(TreeEnsembleModels):
- class GBTRegressionModel(TreeEnsembleModels):

AmplabJenkins · 2015-06-30T05:16:50Z

Merged build finished. Test PASSed.

JoshRosen · 2015-06-30T05:19:43Z

Uh oh: looks like this is actually failing some tests:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/anaconda/envs/py3k/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/home/anaconda/envs/py3k/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "./python/run-tests.py", line 166, in process_queue
    run_individual_python_test(test_goal, python_exec)
  File "./python/run-tests.py", line 74, in run_individual_python_test
    if not re.match('[0-9]+', line):
  File "/home/anaconda/envs/py3k/lib/python3.4/re.py", line 160, in match
    return _compile(pattern, flags).match(string)
TypeError: can't use a string pattern on a bytes-like object

Finished test(python3.4): pyspark.ml.recommendation (15s)
Finished test(python3.4): pyspark.ml.feature (18s)
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/anaconda/envs/py3k/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/home/anaconda/envs/py3k/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "./python/run-tests.py", line 166, in process_queue
    run_individual_python_test(test_goal, python_exec)
  File "./python/run-tests.py", line 74, in run_individual_python_test
    if not re.match('[0-9]+', line):
  File "/home/anaconda/envs/py3k/lib/python3.4/re.py", line 160, in match
    return _compile(pattern, flags).match(string)
TypeError: can't use a string pattern on a bytes-like object

This is the consequence of not having good test coverage on the test script... I'll fix this shortly.

JoshRosen · 2015-06-30T05:41:22Z

I've opened #7112 to hotfix the build issue which masked the test failure here.

JoshRosen · 2015-06-30T06:09:44Z

Jenkins, retest this please.

AmplabJenkins · 2015-06-30T06:12:12Z

Merged build triggered.

AmplabJenkins · 2015-06-30T06:12:18Z

Merged build started.

SparkQA · 2015-06-30T06:16:00Z

Test build #36096 has started for PR 7095 at commit 2fde31a.

SparkQA · 2015-06-30T06:26:44Z

Test build #36096 has finished for PR 7095 at commit 2fde31a.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(DecisionTreeModel):
- class RandomForestClassificationModel(TreeEnsembleModels):
- class GBTClassificationModel(TreeEnsembleModels):
- class DecisionTreeModel(JavaModel):
- class TreeEnsembleModels(JavaModel):
- class DecisionTreeRegressionModel(DecisionTreeModel):
- class RandomForestRegressionModel(TreeEnsembleModels):
- class GBTRegressionModel(TreeEnsembleModels):

AmplabJenkins · 2015-06-30T06:26:50Z

Merged build finished. Test FAILed.

JoshRosen · 2015-06-30T06:29:14Z

Now that the test reporting is fixed, we can see the failure:

**********************************************************************
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/ml/regression.py", line 175, in __main__.DecisionTreeRegressor
Failed example:
    model.depth
Expected:
    2
Got:
    1
**********************************************************************
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/ml/regression.py", line 177, in __main__.DecisionTreeRegressor
Failed example:
    model.numNodes
Expected:
    1
Got:
    3
**********************************************************************
   2 of  10 in __main__.DecisionTreeRegressor
***Test Failed*** 2 failures.

JoshRosen · 2015-06-30T06:30:18Z

By the way, check out --help on python/run-tests if you haven't already; there's some neat options for controlling which suites, python versions, and degree of parallelism to use, which might be a big productivity boost for you given all of the Python work that you've been doing lately.

AmplabJenkins · 2015-06-30T12:52:11Z

Merged build triggered.

MechCoder · 2015-07-06T19:41:48Z

I renamed pyTreeWeights to javaTreeWeights.

AmplabJenkins · 2015-07-06T19:43:11Z

Merged build triggered.

AmplabJenkins · 2015-07-06T19:43:21Z

Merged build started.

SparkQA · 2015-07-06T19:44:37Z

Test build #36598 has started for PR 7095 at commit 38a0860.

SparkQA · 2015-07-06T20:19:00Z

Test build #36598 has finished for PR 7095 at commit 38a0860.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(DecisionTreeModel):
- class RandomForestClassificationModel(TreeEnsembleModels):
- class GBTClassificationModel(TreeEnsembleModels):
- class DecisionTreeModel(JavaModel):
- class TreeEnsembleModels(JavaModel):
- class DecisionTreeRegressionModel(DecisionTreeModel):
- class RandomForestRegressionModel(TreeEnsembleModels):
- class GBTRegressionModel(TreeEnsembleModels):

AmplabJenkins · 2015-07-06T20:19:27Z

Merged build finished. Test PASSed.

mengxr · 2015-07-06T23:24:31Z

mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala

Make it package private. Java users do not really need it.

mengxr · 2015-07-06T23:25:15Z

LGTM except that the utility method should be package private.

MechCoder · 2015-07-07T04:49:44Z

done. I could not make it private[python] . It was giving some error.

AmplabJenkins · 2015-07-07T04:53:11Z

Merged build triggered.

AmplabJenkins · 2015-07-07T04:53:19Z

Merged build started.

AmplabJenkins · 2015-07-07T05:08:20Z

Merged build finished. Test FAILed.

MechCoder · 2015-07-07T06:48:03Z

jenkins retest this please

AmplabJenkins · 2015-07-07T06:48:12Z

Merged build triggered.

AmplabJenkins · 2015-07-07T06:48:21Z

Merged build started.

SparkQA · 2015-07-07T06:49:32Z

Test build #36652 has started for PR 7095 at commit 23b08be.

SparkQA · 2015-07-07T06:59:35Z

Test build #36652 has finished for PR 7095 at commit 23b08be.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(DecisionTreeModel):
- class RandomForestClassificationModel(TreeEnsembleModels):
- class GBTClassificationModel(TreeEnsembleModels):
- class DecisionTreeModel(JavaModel):
- class TreeEnsembleModels(JavaModel):
- class DecisionTreeRegressionModel(DecisionTreeModel):
- class RandomForestRegressionModel(TreeEnsembleModels):
- class GBTRegressionModel(TreeEnsembleModels):

AmplabJenkins · 2015-07-07T06:59:41Z

Merged build finished. Test FAILed.

MechCoder · 2015-07-07T13:29:21Z

jenkins retest this please

mengxr · 2015-07-07T15:05:27Z

test this please

AmplabJenkins · 2015-07-07T15:08:12Z

Merged build triggered.

AmplabJenkins · 2015-07-07T15:08:18Z

Merged build started.

SparkQA · 2015-07-07T15:08:26Z

Test build #36690 has started for PR 7095 at commit 23b08be.

SparkQA · 2015-07-07T15:42:24Z

Test build #36690 has finished for PR 7095 at commit 23b08be.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(DecisionTreeModel):
- class RandomForestClassificationModel(TreeEnsembleModels):
- class GBTClassificationModel(TreeEnsembleModels):
- class DecisionTreeModel(JavaModel):
- class TreeEnsembleModels(JavaModel):
- class DecisionTreeRegressionModel(DecisionTreeModel):
- class RandomForestRegressionModel(TreeEnsembleModels):
- class GBTRegressionModel(TreeEnsembleModels):

AmplabJenkins · 2015-07-07T15:42:54Z

Merged build finished. Test PASSed.

mengxr · 2015-07-07T15:58:57Z

Merged into master. Thanks!

MechCoder force-pushed the missing_methods_tree branch from ce343b9 to 849aabe Compare June 29, 2015 19:47

feynmanliang reviewed Jun 29, 2015
View reviewed changes

python/pyspark/ml/regression.py Outdated

Copy link

Contributor

feynmanliang Jun 29, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Floating point equality is causing this test to fail. Try numpy.allclose

MechCoder force-pushed the missing_methods_tree branch from 2fde31a to a6a27ca Compare June 30, 2015 12:52

rename pyTreeWeights to javaTreeWeights

38a0860

mengxr reviewed Jul 6, 2015
View reviewed changes

mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala Outdated

Copy link

Contributor

mengxr Jul 6, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make it package private. Java users do not really need it.

private [spark]

23b08be

asfgit closed this in 1dbc4a1 Jul 7, 2015

MechCoder deleted the missing_methods_tree branch July 7, 2015 16:23

[SPARK-8711] [ML] Add additional methods to PySpark ML tree models #7095

[SPARK-8711] [ML] Add additional methods to PySpark ML tree models #7095

Uh oh!

Conversation

MechCoder commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

MechCoder commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

SparkQA commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

SparkQA commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

feynmanliang Jun 29, 2015

Choose a reason for hiding this comment

Uh oh!

MechCoder commented Jun 30, 2015

Uh oh!

AmplabJenkins commented Jun 30, 2015

Uh oh!

AmplabJenkins commented Jun 30, 2015

Uh oh!

SparkQA commented Jun 30, 2015

Uh oh!

SparkQA commented Jun 30, 2015

Uh oh!

AmplabJenkins commented Jun 30, 2015

Uh oh!

JoshRosen commented Jun 30, 2015

Uh oh!

JoshRosen commented Jun 30, 2015

Uh oh!

JoshRosen commented Jun 30, 2015

Uh oh!

AmplabJenkins commented Jun 30, 2015

Uh oh!

AmplabJenkins commented Jun 30, 2015

Uh oh!

SparkQA commented Jun 30, 2015

Uh oh!

SparkQA commented Jun 30, 2015

Uh oh!

AmplabJenkins commented Jun 30, 2015

Uh oh!

JoshRosen commented Jun 30, 2015

Uh oh!

JoshRosen commented Jun 30, 2015

Uh oh!

AmplabJenkins commented Jun 30, 2015

Uh oh!

MechCoder commented Jul 6, 2015

Uh oh!

AmplabJenkins commented Jul 6, 2015

Uh oh!

AmplabJenkins commented Jul 6, 2015

Uh oh!

SparkQA commented Jul 6, 2015

Uh oh!

SparkQA commented Jul 6, 2015

Uh oh!

AmplabJenkins commented Jul 6, 2015

Uh oh!

mengxr Jul 6, 2015

Choose a reason for hiding this comment

Uh oh!

mengxr commented Jul 6, 2015

Uh oh!

MechCoder commented Jul 7, 2015

Uh oh!