-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-15750][MLLib][PYSPARK] Constructing FPGrowth fails when no numPartitions specified in pyspark #13493
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Test build #59923 has finished for PR 13493 at commit
|
|
lgtm cc: @MLnick |
|
Shall we change the logic in the Python wrapper and set |
|
Test build #3236 has finished for PR 13493 at commit
|
|
Pinging! Is this still active? |
|
@jkbradley I will update the PR soon |
|
@mengxr @jkbradley I think we should keep as less logic in python side as possible. So I didn't update the PR, please help review. Thanks |
|
Test build #65071 has finished for PR 13493 at commit
|
|
This looks pretty reasonable to me, the only thing is (and its relatively minor) testing the default param might be better as a tests.py test rather than as a doc test since the doctests are part of the documentation and we might not need this in the docs along with the current example. |
|
Same here, looks fine except for @holdenk 's comment. Thanks! |
|
Ping --- let me know if you'd like someone to take it over. Thanks! |
|
Sorry, I am in vacation. Will update it next week. |
|
@zjffdu Enjoy vacation and no worries - I don't think this is blocking anything and we can circle back in a week :) |
|
Test build #66589 has finished for PR 13493 at commit
|
|
PR is updated, @holdenk @jkbradley |
|
ping @jkbradley @holdenk |
|
Test build #81076 has finished for PR 13493 at commit
|
|
ping @zjffdu Looks reasonable fix, But pls resolve the conflicts! Thanks! |
…Partitions specified in pyspark
python/pyspark/mllib/tests.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line is useless.
|
Test build #82315 has finished for PR 13493 at commit
|
|
Test build #82319 has finished for PR 13493 at commit
|
|
retest this please |
|
Test build #82330 has finished for PR 13493 at commit
|
|
The bug is confirmed. The fix looks pretty reasonable to me. ping @jkbradley . |
|
LGTM pending fresh tests |
|
Test build #4167 has finished for PR 13493 at commit
|
|
Thanks @jkbradley The failed tests seems unrelated. |
|
LGTM retest this please |
|
Test build #90147 has finished for PR 13493 at commit
|
|
LGTM! |
|
Merging with master |
What changes were proposed in this pull request?
Change FPGrowth from private to private[spark]. If no numPartitions is specified, then default value -1 is used. But -1 is only valid in the construction function of FPGrowth, but not in setNumPartitions. So I make this change and use the constructor directly rather than using set method.
How was this patch tested?
Unit test is added