Skip to content

MLI-1 Decision Trees #79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
manishamde wants to merge 51 commits into from
Closed

MLI-1 Decision Trees #79

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
cd53eae
skeletal framework
manishamde Nov 28, 2013
92cedce
basic building blocks for intermediate RDD calculation. untested.
manishamde Dec 2, 2013
8bca1e2
additional code for creating intermediate RDD
manishamde Dec 9, 2013
0012a77
basic stump working
manishamde Dec 10, 2013
03f534c
some more tests
manishamde Dec 10, 2013
dad0afc
decison stump functionality working
manishamde Dec 15, 2013
4798aae
added gain stats class
manishamde Dec 15, 2013
80e8c66
working version of multi-level split calculation
manishamde Dec 16, 2013
b0eb866
added logic to handle leaf nodes
manishamde Dec 16, 2013
98ec8d5
tree building and prediction logic
manishamde Dec 22, 2013
02c595c
added command line parsing
manishamde Dec 22, 2013
733d6dd
fixed tests
manishamde Dec 22, 2013
154aa77
enums for configurations
manishamde Dec 23, 2013
b0e3e76
adding enum for feature type
manishamde Jan 12, 2014
c8f6d60
adding enum for feature type
manishamde Jan 12, 2014
e23c2e5
added regression support
manishamde Jan 19, 2014
53108ed
fixing index for highest bin
manishamde Jan 20, 2014
6df35b9
regression predict logic
manishamde Jan 21, 2014
dbb7ac1
categorical feature support
manishamde Jan 23, 2014
d504eb1
more tests for categorical features
manishamde Jan 23, 2014
6b7de78
minor refactoring and tests
manishamde Jan 26, 2014
b09dc98
minor refactoring
manishamde Jan 26, 2014
c0e522b
updated predict and split threshold logic
manishamde Jan 27, 2014
f067d68
minor cleanup
manishamde Jan 27, 2014
5841c28
unit tests for categorical features
manishamde Jan 27, 2014
0dd7659
basic doc
manishamde Jan 27, 2014
dd0c0d7
minor: some docs
manishamde Jan 27, 2014
9372779
code style: max line lenght <= 100
manishamde Feb 17, 2014
84f85d6
code documentation
manishamde Feb 28, 2014
d3023b3
adding more docs for nested methods
manishamde Mar 6, 2014
63e786b
added multiple train methods for java compatability
manishamde Mar 6, 2014
cd2c2b4
fixing code style based on feedback
manishamde Mar 7, 2014
eb8fcbe
minor code style updates
manishamde Mar 7, 2014
794ff4d
minor improvements to docs and style
manishamde Mar 10, 2014
d1ef4f6
more documentation
manishamde Mar 10, 2014
ad1fc21
incorporated mengxr's code style suggestions
manishamde Mar 11, 2014
62c2562
fixing comment indentation
manishamde Mar 11, 2014
6068356
ensuring num bins is always greater than max number of categories
manishamde Mar 12, 2014
2116360
removing dummy bin calculation for categorical variables
manishamde Mar 12, 2014
632818f
removing threshold for classification predict method
manishamde Mar 13, 2014
ff363a7
binary search for bins and while loop for categorical feature bins
manishamde Mar 17, 2014
4576b64
documentation and for to while loop conversion
manishamde Mar 23, 2014
24500c5
minor style updates
mengxr Mar 23, 2014
c487e6a
Merge pull request #1 from mengxr/dtree
manishamde Mar 23, 2014
f963ef5
making methods private
manishamde Mar 23, 2014
201702f
making some more methods private
manishamde Mar 23, 2014
62dc723
updating javadoc and converting helper methods to package private to …
manishamde Mar 24, 2014
e1dd86f
implementing code style suggestions
manishamde Mar 25, 2014
f536ae9
another pass on code style
mengxr Mar 31, 2014
7d54b4f
Merge pull request #4 from mengxr/dtree
manishamde Mar 31, 2014
1e8c704
remove numBins field in the Strategy class
manishamde Apr 1, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
updated predict and split threshold logic 
Signed-off-by: Manish Amde <[email protected]>
  • Loading branch information
@manishamde
manishamde committed Feb 28, 2014
commit c0e522b7d1f5e27c81d682e5c8c97543fb4242be
9 changes: 5 additions & 4 deletions mllib/src/main/scala/org/apache/spark/mllib/tree/DecisionTree.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ object DecisionTree extends Serializable with Logging {
val lowThreshold = bin.lowSplit.threshold
val highThreshold = bin.highSplit.threshold
val features = labeledPoint.features
if ((lowThreshold <= features(featureIndex)) & (highThreshold > features(featureIndex))) {
if ((lowThreshold < features(featureIndex)) & (highThreshold >= features(featureIndex))) {
return binIndex
}
}
Expand Down Expand Up @@ -400,7 +400,8 @@ object DecisionTree extends Serializable with Logging {
}
}

val predict = leftCount / (leftCount + rightCount)
//val predict = leftCount / (leftCount + rightCount)
val predict = (left1Count + right1Count) / (leftCount + rightCount)

new InformationGainStats(gain,impurity,leftImpurity,rightImpurity,predict)
Copy link
Contributor

@mengxr mengxr Mar 6, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put a space between arguments

}
Expand Down Expand Up @@ -672,8 +673,8 @@ object DecisionTree extends Serializable with Logging {

//Find all bins
for (featureIndex <- 0 until numFeatures){
val isFeatureContinous = strategy.categoricalFeaturesInfo.get(featureIndex).isEmpty
if (isFeatureContinous) { //bins for categorical variables are already assigned
val isFeatureContinuous = strategy.categoricalFeaturesInfo.get(featureIndex).isEmpty
if (isFeatureContinuous) { //bins for categorical variables are already assigned
bins(featureIndex)(0)
= new Bin(new DummyLowSplit(featureIndex, Continuous),splits(featureIndex)(0),Continuous,Double.MinValue)
for (index <- 1 until numBins - 1){
Expand Down
1 change: 0 additions & 1 deletion mllib/src/main/scala/org/apache/spark/mllib/tree/DecisionTreeRunner.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,6 @@ object DecisionTreeRunner extends Logging {
//TODO: Make these generic MLTable metrics
def meanSquaredError(tree : DecisionTreeModel, data : RDD[LabeledPoint]) : Double = {
val meanSumOfSquares = data.map(y => (tree.predict(y.features) - y.label)*(tree.predict(y.features) - y.label)).mean()
println("meanSumOfSquares = " + meanSumOfSquares)
meanSumOfSquares
}

Expand Down
2 changes: 1 addition & 1 deletion mllib/src/main/scala/org/apache/spark/mllib/tree/model/DecisionTreeModel.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ class DecisionTreeModel(val topNode : Node, val algo : Algo) extends Serializabl
def predict(features : Array[Double]) = {
algo match {
case Classification => {
if (topNode.predictIfLeaf(features) >= 0.5) 0.0 else 1.0
if (topNode.predictIfLeaf(features) < 0.5) 0.0 else 1.0
Copy link
Contributor

@mengxr mengxr Mar 10, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question was asked by @srowen : It is easy to support multi-class?

Also, why 0.5 is used here as the threshold?

Copy link
Contributor Author

@manishamde manishamde Mar 11, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0.5 was just a threshold used for verification. I will make it configurable and for now return a double value between 0 and 1 similar to other classification algorithms in mllib. This will make it easier for performing ROC/AUC calculations.

}
case Regression => {
topNode.predictIfLeaf(features)
Expand Down
7 changes: 4 additions & 3 deletions mllib/src/main/scala/org/apache/spark/mllib/tree/model/InformationGainStats.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,10 @@ class InformationGainStats(val gain : Double,
//val rightSamples : Long
val predict : Double) extends Serializable {

override def toString =
"gain = " + gain + ", impurity = " + impurity + ", left impurity = "
+ leftImpurity + ", right impurity = " + rightImpurity + ", predict = " + predict
override def toString = {
"gain = %f, impurity = %f, left impurity = %f, right impurity = %f, predict = %f"
.format(gain, impurity, leftImpurity, rightImpurity, predict)
}


}
1 change: 1 addition & 0 deletions mllib/src/main/scala/org/apache/spark/mllib/tree/model/Node.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ class Node ( val id : Int,
def build(nodes : Array[Node]) : Unit = {

logDebug("building node " + id + " at level " + (scala.math.log(id + 1)/scala.math.log(2)).toInt )
logDebug("id = " + id + ", split = " + split)
logDebug("stats = " + stats)
logDebug("predict = " + predict)
if (!isLeaf) {
Expand Down
8 changes: 4 additions & 4 deletions mllib/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ class DecisionTreeSuite extends FunSuite with BeforeAndAfterAll {
assert(0==bestSplits(0)._2.gain)
assert(0==bestSplits(0)._2.leftImpurity)
assert(0==bestSplits(0)._2.rightImpurity)
assert(0.01==bestSplits(0)._2.predict)
println(bestSplits(0)._2.predict)
}

test("stump with fixed label 1 for Gini"){
Expand All @@ -181,7 +181,7 @@ class DecisionTreeSuite extends FunSuite with BeforeAndAfterAll {
assert(0==bestSplits(0)._2.gain)
assert(0==bestSplits(0)._2.leftImpurity)
assert(0==bestSplits(0)._2.rightImpurity)
assert(0.01==bestSplits(0)._2.predict)
assert(1==bestSplits(0)._2.predict)

}

Expand All @@ -207,7 +207,7 @@ class DecisionTreeSuite extends FunSuite with BeforeAndAfterAll {
assert(0==bestSplits(0)._2.gain)
assert(0==bestSplits(0)._2.leftImpurity)
assert(0==bestSplits(0)._2.rightImpurity)
assert(0.01==bestSplits(0)._2.predict)
assert(0==bestSplits(0)._2.predict)
}

test("stump with fixed label 1 for Entropy"){
Expand All @@ -231,7 +231,7 @@ class DecisionTreeSuite extends FunSuite with BeforeAndAfterAll {
assert(0==bestSplits(0)._2.gain)
assert(0==bestSplits(0)._2.leftImpurity)
assert(0==bestSplits(0)._2.rightImpurity)
assert(0.01==bestSplits(0)._2.predict)
assert(1==bestSplits(0)._2.predict)
}


Expand Down