Decision Trees & Ensembles

A decision tree makes predictions the way a person might work through a flowchart. Ask a yes/no question about the input, branch based on the answer, ask another question, and repeat until you reach a final prediction at a leaf of the tree.

For a loan approval model, a tree might first ask whether annual income is above $50,000, then, depending on the answer, ask a different follow-up question, like whether credit history is longer than 5 years, branching deeper until it reaches a leaf that says approve or deny. Crucially, the tree learns which questions to ask and in what order directly from training data, rather than a human hand-designing the flowchart.

At each node, the tree considers every possible yes/no question it could ask and picks the one that makes the resulting two groups as pure as possible, meaning each group is dominated by a single class. The most common way to measure impurity is the Gini impurity.

A single decision tree, grown deep enough to fit its training data very precisely, tends to overfit: it memorizes quirks of the specific training examples rather than learning patterns that generalize. Overfitting gets its own full treatment in a later module.

The fix that dominates practice today is to train many trees and combine their predictions, an approach called an ensemble. There are two main families. Bagging (short for bootstrap aggregating) trains many trees independently, each on a random subset of the data and features, then averages their votes. Random forests are the classic bagging example. Boosting instead trains trees one after another, each one specifically focused on correcting the mistakes of the trees before it, which is how gradient-boosted trees work. Ensembles of trees remain one of the strongest, most widely used techniques for structured, tabular data, often outperforming far more complex approaches on exactly that kind of data.