Decision Trees and Forests
Level 4 · Classic models
How a decision tree asks yes or no questions, and why many trees beat one. Random forests, gradient boosting and XGBoost, the leaders on tables.
Course contentsLesson 10 of 18
Level 1 · How a machine learns
Level 2 · Getting the data ready
Level 3 · Kinds of learning
Level 4 · Classic models
Level 5 · Neural networks
Level 6 · Better models, real tools
After this lesson you can
- Read a decision tree from top to bottom
- Tell a random forest from gradient boosting
- Say why boosted trees win on table data
13 mincovers 5 glossary entries
Worth reading first: Classification vs Regression
From one tree of questions to hundreds
A decision tree reaches an answer by asking a chain of yes or no questions. Is it raining? Is it colder than 5 degrees? Each answer leads to the next question, until it lands on an answer. Training picks the questions that split the data most cleanly. You can read the whole tree and see exactly why it decided.
One tree has a weak spot. Let it grow deep and it makes a branch for almost every row, so it overfits. Keep it shallow and it is too rough. The fix that works best is not a better tree but many trees, with their answers joined. That is called ensemble learning.
A random forest grows hundreds of trees side by side. Each one learns from a random slice of the rows and looks at a random mix of features, so each makes different mistakes. For a new example, every tree votes and the most common answer wins. The method was set out in 2001.
The mistakes of one tree are cancelled out by the others. If 240 of 300 trees say a review is fake, the forest is far surer than any one tree. Forests are hard to overfit, need little tuning and need no feature scaling, which makes them a strong first try.
Gradient boosting builds its trees one after another instead. The first tree makes rough guesses. Each new tree is trained on the errors the earlier trees left behind, and the answers are added up. Hundreds of small trees in a row, each fixing the last, often beat a forest of the same size.
XGBoost is a free library that does gradient boosting fast, with guards against overfitting built in. It came out in 2014. Of 29 winning entries in Kaggle contests in 2015, 17 used it. On table data such as sales, loans or sign-ups, it is still often the model to beat.
People reach for deep learning on every problem. On rows and columns, boosted trees are often more accurate, faster to train and easier to explain. Deep learning wins on photos, sound and text, where the raw input has no neat columns. Pick by the shape of the data.
The three side by side
| Model | How the trees work together | Strength |
|---|---|---|
| Decision tree | One tree on its own | Easy to read and explain |
| Random forest | Many trees side by side, then a vote | Hard to overfit, little tuning |
| Gradient boosting | Trees in a row, each fixing the last | Often the best on tables |
Which one to pick
- 1
Pick one decision tree
When people must read and trust every rule, like a loan choice you must explain.
- 2
Pick a random forest
When you want a strong first model fast, with little tuning.
- 3
Pick gradient boosting
When you need the best score on table data and have time to tune it.
Worked example
Lucy predicts meals
Lucy's restaurants predict how many meals each site sells per day. One decision tree misses by 38 meals on average.
A random forest of 300 trees cuts the miss to 24. Gradient boosting with 500 small trees gets it to 19, so that is the one she uses.
The words for this lesson5
Check yourself
How does gradient boosting build its trees?
Finished this lesson?
Marking it complete also marks its 5 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.