more notes

Author: tdpetrou

Introduction to Statistical Learning/Chapter 8.ipynb

@@ -83,7 +83,23 @@
     "### Tree Pruning\n",
     "It is possible to build a decision tree so specific (one with so many branches) that each observation can be predicted exactly. This would be complete memorization, ie overfitting, of the data. Because we want to have the tree work with unseen data, we can prune the tree.\n",
     "\n",
-    "One strategy would be some have some threshold for stopping a branch from splitting - it must have decreased RSS by a certain amount."
+    "One strategy would be some have some threshold for stopping a branch from splitting - it must have decreased RSS by a certain amount. Since this might miss a good split deeper in the tree, pruning is preferred.\n",
+    "\n",
+    "Pruning works by.... \n",
+    "1. growing a very large tree and stopping only when a minimum number of observations are left in each branch.\n",
+    "2. At each stage during the growing process add a penalty term $\\alpha|T|$ to RSS where |T| is the number of terminal nodes.\n",
+    "3. This will give a function that maps $\\alpha$ to a particular subtree. So $\\alpha = 0$ would map to the original huge tree and for example $\\alpha = 5$ could map to a tree that with only half of the terminal nodes.\n",
+    "\n",
+    "Choose $\\alpha$ through cross validation by...\n",
+    "1. Splitting training data into K folds\n",
+    "2. Grow a large tree and apply the penalty term exactly as above (map each $\\alpha$ to a particular subtree.)\n",
+    "3. evaluate each $\\alpha$ (subtree) on the left-out fold\n",
+    "4. Average all the $\\alpha$ (subtrees) for each iteration of the K-folds\n",
+    "\n",
+    "Then use this $\\alpha$ to choose the tree from above.\n",
+    "\n",
+    "### Classification Trees\n",
+    "Predict at each node, the most commonly occurring class."
    ]
   },
   {