Skip to content

[SPARK-9612][ML][FOLLOWUP] fix GBT support weights if subsamplingRate<1 #27070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zhengruifeng wants to merge 4 commits into from
Closed

[SPARK-9612][ML][FOLLOWUP] fix GBT support weights if subsamplingRate<1 #27070

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
dac8f69
init
zhengruifeng Jan 2, 2020
a41bcba
update testsuite
zhengruifeng Jan 2, 2020
e3d9200
update doc
zhengruifeng Jan 3, 2020
53750d4
destory nodeToFeaturesBc
zhengruifeng Jan 3, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update doc
  • Loading branch information
@zhengruifeng
zhengruifeng committed Jan 3, 2020
commit e3d9200add19e30417775395465b7da38ac2dee3
8 changes: 5 additions & 3 deletions mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,11 @@ private[spark] object RandomForest extends Logging with Serializable {
}

/**
* Train a random forest.
* Train a random forest with metadata. This method is mainly for GBT, in which metadata can
* be reused among trees.
*
* @param input Training data: RDD of `Instance`
* @param metadata Learning and dataset metadata for DecisionTree.
* @return an unweighted set of trees
*/
def runWithMetadata(
Expand Down Expand Up @@ -590,7 +592,7 @@ private[spark] object RandomForest extends Logging with Serializable {

// transform nodeStatsAggregators array to (nodeIndex, nodeAggregateStats) pairs,
// which can be combined with other partition using `reduceByKey`
nodeStatsAggregators.view.zipWithIndex.map(_.swap).iterator
nodeStatsAggregators.iterator.zipWithIndex.map(_.swap)
Copy link
Contributor Author

@zhengruifeng zhengruifeng Jan 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IDEA editor always shows warnings on the two lines, change them to avoid warnings.

}
} else {
input.mapPartitions { points =>
Expand All @@ -608,7 +610,7 @@ private[spark] object RandomForest extends Logging with Serializable {

// transform nodeStatsAggregators array to (nodeIndex, nodeAggregateStats) pairs,
// which can be combined with other partition using `reduceByKey`
nodeStatsAggregators.view.zipWithIndex.map(_.swap).iterator
nodeStatsAggregators.iterator.zipWithIndex.map(_.swap)
}
}

Expand Down