Skip to content
Closed
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
cd53eae
skeletal framework
manishamde Nov 28, 2013
92cedce
basic building blocks for intermediate RDD calculation. untested.
manishamde Dec 2, 2013
8bca1e2
additional code for creating intermediate RDD
manishamde Dec 9, 2013
0012a77
basic stump working
manishamde Dec 10, 2013
03f534c
some more tests
manishamde Dec 10, 2013
dad0afc
decison stump functionality working
manishamde Dec 15, 2013
4798aae
added gain stats class
manishamde Dec 15, 2013
80e8c66
working version of multi-level split calculation
manishamde Dec 16, 2013
b0eb866
added logic to handle leaf nodes
manishamde Dec 16, 2013
98ec8d5
tree building and prediction logic
manishamde Dec 22, 2013
02c595c
added command line parsing
manishamde Dec 22, 2013
733d6dd
fixed tests
manishamde Dec 22, 2013
154aa77
enums for configurations
manishamde Dec 23, 2013
b0e3e76
adding enum for feature type
manishamde Jan 12, 2014
c8f6d60
adding enum for feature type
manishamde Jan 12, 2014
e23c2e5
added regression support
manishamde Jan 19, 2014
53108ed
fixing index for highest bin
manishamde Jan 20, 2014
6df35b9
regression predict logic
manishamde Jan 21, 2014
dbb7ac1
categorical feature support
manishamde Jan 23, 2014
d504eb1
more tests for categorical features
manishamde Jan 23, 2014
6b7de78
minor refactoring and tests
manishamde Jan 26, 2014
b09dc98
minor refactoring
manishamde Jan 26, 2014
c0e522b
updated predict and split threshold logic
manishamde Jan 27, 2014
f067d68
minor cleanup
manishamde Jan 27, 2014
5841c28
unit tests for categorical features
manishamde Jan 27, 2014
0dd7659
basic doc
manishamde Jan 27, 2014
dd0c0d7
minor: some docs
manishamde Jan 27, 2014
9372779
code style: max line lenght <= 100
manishamde Feb 17, 2014
84f85d6
code documentation
manishamde Feb 28, 2014
d3023b3
adding more docs for nested methods
manishamde Mar 6, 2014
63e786b
added multiple train methods for java compatability
manishamde Mar 6, 2014
cd2c2b4
fixing code style based on feedback
manishamde Mar 7, 2014
eb8fcbe
minor code style updates
manishamde Mar 7, 2014
794ff4d
minor improvements to docs and style
manishamde Mar 10, 2014
d1ef4f6
more documentation
manishamde Mar 10, 2014
ad1fc21
incorporated mengxr's code style suggestions
manishamde Mar 11, 2014
62c2562
fixing comment indentation
manishamde Mar 11, 2014
6068356
ensuring num bins is always greater than max number of categories
manishamde Mar 12, 2014
2116360
removing dummy bin calculation for categorical variables
manishamde Mar 12, 2014
632818f
removing threshold for classification predict method
manishamde Mar 13, 2014
ff363a7
binary search for bins and while loop for categorical feature bins
manishamde Mar 17, 2014
4576b64
documentation and for to while loop conversion
manishamde Mar 23, 2014
24500c5
minor style updates
mengxr Mar 23, 2014
c487e6a
Merge pull request #1 from mengxr/dtree
manishamde Mar 23, 2014
f963ef5
making methods private
manishamde Mar 23, 2014
201702f
making some more methods private
manishamde Mar 23, 2014
62dc723
updating javadoc and converting helper methods to package private to …
manishamde Mar 24, 2014
e1dd86f
implementing code style suggestions
manishamde Mar 25, 2014
f536ae9
another pass on code style
mengxr Mar 31, 2014
7d54b4f
Merge pull request #4 from mengxr/dtree
manishamde Mar 31, 2014
1e8c704
remove numBins field in the Strategy class
manishamde Apr 1, 2014
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
adding more docs for nested methods
  • Loading branch information
manishamde committed Mar 6, 2014
commit d3023b37f8954f8d79b2f0b0d081d9a4eb51415b
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,20 @@ object DecisionTree extends Serializable with Logging {
arr
}

/**
Performs a sequential aggregation over a partition for classification.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use JavaDoc style.


for p bins, k features, l nodes (level = log2(l)) storage is of the form:
b111_left_count,b111_right_count, .... , ..
.. bpk1_left_count, bpk1_right_count, .... , ..
.. bpkl_left_count, bpkl_right_count

@param agg Array[Double] storing aggregate calculation of size
2*numSplits*numFeatures*numNodes for classification
@param arr Array[Double] of size 1+(numFeatures*numNodes)
@return Array[Double] storing aggregate calculation of size 2*numSplits*numFeatures*numNodes
for classification
*/
def classificationBinSeqOp(arr: Array[Double], agg: Array[Double]) {
for (node <- 0 until numNodes) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use "nodeIndex" or simply "i" instead of "node"

val validSignalIndex = 1 + numFeatures * node
Expand All @@ -326,6 +340,20 @@ object DecisionTree extends Serializable with Logging {
}
}

/**
Performs a sequential aggregation over a partition for regression.

for p bins, k features, l nodes (level = log2(l)) storage is of the form:
b111_count,b111_sum, b111_sum_squares .... , ..
.. bpk1_count, bpk1_sum, bpk1_sum_squares, .... , ..
.. bpkl_count, bpkl_sum, bpkl_sum_squares

@param agg Array[Double] storing aggregate calculation of size
3*numSplits*numFeatures*numNodes for classification
@param arr Array[Double] of size 1+(numFeatures*numNodes)
@return Array[Double] storing aggregate calculation of size 3*numSplits*numFeatures*numNodes
for regression
*/
def regressionBinSeqOp(arr: Array[Double], agg: Array[Double]) {
for (node <- 0 until numNodes) {
val validSignalIndex = 1 + numFeatures * node
Expand Down Expand Up @@ -354,11 +382,11 @@ object DecisionTree extends Serializable with Logging {
.. bpk1_left_count, bpk1_right_count, .... , ..
.. bpkl_left_count, bpkl_right_count

@param agg Array[Double] storing aggregate calculation of size
2*numSplits*numFeatures*numNodes for classification
@param agg Array[Double] storing aggregate calculation of size 2*numSplits*numFeatures*numNodes for classification
and 3*numSplits*numFeatures*numNodes for regression
@param arr Array[Double] of size 1+(numFeatures*numNodes)
@return Array[Double] storing aggregate calculation of size 2*numSplits*numFeatures*numNodes
for classification
@return Array[Double] storing aggregate calculation of size 2*numSplits*numFeatures*numNodes for classification
and 3*numSplits*numFeatures*numNodes for regression
*/
def binSeqOp(agg : Array[Double], arr: Array[Double]) : Array[Double] = {
strategy.algo match {
Expand Down Expand Up @@ -411,7 +439,15 @@ object DecisionTree extends Serializable with Logging {
logDebug("binAggregates.length = " + binAggregates.length)
//binAggregates.foreach(x => logDebug(x))


/**
* Calculates the information gain for all splits
* @param leftNodeAgg left node aggregates
* @param featureIndex feature index
* @param splitIndex split index
* @param rightNodeAgg right node aggregate
* @param topImpurity impurity of the parent node
* @return information gain and statistics for all splits
*/
def calculateGainForSplit(leftNodeAgg: Array[Array[Double]],
featureIndex: Int,
splitIndex: Int,
Expand Down