Skip to content

[SPARK-19791] [ML] Add doc and example for fpgrowth #17130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hhbyyh wants to merge 22 commits into from
Closed

[SPARK-19791] [ML] Add doc and example for fpgrowth #17130

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
fdce240
fpm doc
YY-OnCall Mar 1, 2017
ca12877
change transform to filter
YY-OnCall Mar 2, 2017
4223d94
Merge remote-tracking branch 'upstream/master' into fpmdoc
YY-OnCall Mar 11, 2017
9ce0093
merge and move
YY-OnCall Mar 10, 2017
fa4c734
merge conflict
YY-OnCall Mar 13, 2017
0a5dbb2
Merge remote-tracking branch 'upstream/master' into fpmdoc
YY-OnCall Mar 14, 2017
de1bfc8
comments and doc refine
YY-OnCall Mar 15, 2017
d4828b7
Merge remote-tracking branch 'upstream/master' into fpmdoc
YY-OnCall Mar 21, 2017
9fef280
adapt to itemsCol
YY-OnCall Mar 21, 2017
9e908d0
resolve conflict
YY-OnCall Mar 23, 2017
16f845c
Merge remote-tracking branch 'upstream/master' into fpmdoc
YY-OnCall Mar 25, 2017
2f0ef8e
Merge remote-tracking branch 'upstream/master' into fpmdoc
YY-OnCall Mar 27, 2017
c957ba5
add python example
YY-OnCall Mar 27, 2017
8d0ccb1
Merge remote-tracking branch 'upstream/master' into fpmdoc
YY-OnCall Mar 30, 2017
e9b090a
docs update
YY-OnCall Mar 31, 2017
99530f1
resolve conflict
YY-OnCall Apr 11, 2017
0fb5a87
refine python example
YY-OnCall Apr 11, 2017
170c31e
Merge remote-tracking branch 'upstream/master' into fpmdoc
YY-OnCall Apr 19, 2017
2b1efb3
add R example
YY-OnCall Apr 19, 2017
45139cd
Merge remote-tracking branch 'upstream/master' into fpmdoc
YY-OnCall Apr 19, 2017
af0b755
Merge remote-tracking branch 'upstream/master' into fpmdoc
YY-OnCall Apr 28, 2017
ea3b973
remove code change per comments
YY-OnCall Apr 28, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
merge and move
  • Loading branch information
@YY-OnCall
YY-OnCall committed Mar 10, 2017
commit 9ce00930ea18c7bb8fe0cc59b98f6ece34d20311
4 changes: 2 additions & 2 deletions mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,10 +241,10 @@ class FPGrowthModel private[ml] (
if (items != null) {
val itemset = items.toSet
brRules.value.filter(_._1.forall(itemset.contains))
.flatMap(_._2.filter(!itemset.contains(_)))
.flatMap(_._2.filter(!itemset.contains(_))).distinct
Copy link
Member

@felixcheung felixcheung Apr 25, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we aware of code changes here

Copy link
Member

@felixcheung felixcheung Apr 25, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so we don't need to handle null item?

Copy link
Contributor Author

@hhbyyh hhbyyh Apr 25, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @felixcheung , can you be more specific about the null item that concerns you? Thanks.

Copy link
Member

@felixcheung felixcheung Apr 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, 2 things - first just calling out while the PR says doc changes there is this one code change here.
second, before this code was checking items != null do we need not consider that now?

Copy link
Contributor Author

@hhbyyh hhbyyh Apr 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

items != null is already checked at two lines above.
Please refer to the comments in the PR for the history of the code change. I can update title to include the code change.

Copy link
Member

@felixcheung felixcheung Apr 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's update the PR/JIRA if code change is required for the doc change.
otherwise, let's leave code change as a separate PR?

Copy link
Contributor Author

@hhbyyh hhbyyh Apr 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess that's the right way. I will revert the code change.

} else {
Seq.empty
}.distinct }, dt)
}}, dt)
dataset.withColumn($(predictionCol), predictUDF(col($(featuresCol))))
}

Expand Down