Skip to content
BENIBIX Learn

Decision Trees and Forests

Level 4 · Classic models

How a decision tree asks yes or no questions, and why many trees beat one. Random forests, gradient boosting and XGBoost, the leaders on tables.

Course contentsLesson 10 of 18

After this lesson you can

  • Read a decision tree from top to bottom
  • Tell a random forest from gradient boosting
  • Say why boosted trees win on table data

13 mincovers 5 glossary entries

Worth reading first: Classification vs Regression

From one tree of questions to hundreds

A decision tree reaches an answer by asking a chain of yes or no questions. Is it raining? Is it colder than 5 degrees? Each answer leads to the next question, until it lands on an answer. Training picks the questions that split the data most cleanly. You can read the whole tree and see exactly why it decided.

One tree has a weak spot. Let it grow deep and it makes a branch for almost every row, so it overfits. Keep it shallow and it is too rough. The fix that works best is not a better tree but many trees, with their answers joined. That is called ensemble learning.

A random forest grows hundreds of trees side by side. Each one learns from a random slice of the rows and looks at a random mix of features, so each makes different mistakes. For a new example, every tree votes and the most common answer wins. The method was set out in 2001.

The mistakes of one tree are cancelled out by the others. If 240 of 300 trees say a review is fake, the forest is far surer than any one tree. Forests are hard to overfit, need little tuning and need no feature scaling, which makes them a strong first try.

Gradient boosting builds its trees one after another instead. The first tree makes rough guesses. Each new tree is trained on the errors the earlier trees left behind, and the answers are added up. Hundreds of small trees in a row, each fixing the last, often beat a forest of the same size.

XGBoost is a free library that does gradient boosting fast, with guards against overfitting built in. It came out in 2014. Of 29 winning entries in Kaggle contests in 2015, 17 used it. On table data such as sales, loans or sign-ups, it is still often the model to beat.

People reach for deep learning on every problem. On rows and columns, boosted trees are often more accurate, faster to train and easier to explain. Deep learning wins on photos, sound and text, where the raw input has no neat columns. Pick by the shape of the data.

The three side by side

ModelHow the trees work togetherStrength
Decision treeOne tree on its ownEasy to read and explain
Random forestMany trees side by side, then a voteHard to overfit, little tuning
Gradient boostingTrees in a row, each fixing the lastOften the best on tables

Which one to pick

  1. 1

    Pick one decision tree

    When people must read and trust every rule, like a loan choice you must explain.

  2. 2

    Pick a random forest

    When you want a strong first model fast, with little tuning.

  3. 3

    Pick gradient boosting

    When you need the best score on table data and have time to tune it.

Forest trees never see each other. Boosting trees each learn from the one before.

Worked example

Lucy predicts meals

Lucy's restaurants predict how many meals each site sells per day. One decision tree misses by 38 meals on average.

A random forest of 300 trees cuts the miss to 24. Gradient boosting with 500 small trees gets it to 19, so that is the one she uses.

The words for this lesson5

Check yourself

How does gradient boosting build its trees?

Finished this lesson?

Marking it complete also marks its 5 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.