# Decision Trees and Forests

> How a decision tree asks yes or no questions, and why many trees beat one. Random forests, gradient boosting and XGBoost, the leaders on tables.

**Course:** [Machine Learning](https://learn.benibix.com/learn/ai-machine-learning)  
**Time:** 13 min

**After this lesson you can:**

- Read a decision tree from top to bottom
- Tell a random forest from gradient boosting
- Say why boosted trees win on table data

## From one tree of questions to hundreds

A decision tree reaches an answer by asking a chain of yes or no questions. Is it raining? Is it colder than 5 degrees? Each answer leads to the next question, until it lands on an answer. Training picks the questions that split the data most cleanly. You can read the whole tree and see exactly why it decided.

One tree has a weak spot. Let it grow deep and it makes a branch for almost every row, so it overfits. Keep it shallow and it is too rough. The fix that works best is not a better tree but many trees, with their answers joined. That is called ensemble learning.

A random forest grows hundreds of trees side by side. Each one learns from a random slice of the rows and looks at a random mix of features, so each makes different mistakes. For a new example, every tree votes and the most common answer wins. The method was set out in 2001.

The mistakes of one tree are cancelled out by the others. If 240 of 300 trees say a review is fake, the forest is far surer than any one tree. Forests are hard to overfit, need little tuning and need no feature scaling, which makes them a strong first try.

Gradient boosting builds its trees one after another instead. The first tree makes rough guesses. Each new tree is trained on the errors the earlier trees left behind, and the answers are added up. Hundreds of small trees in a row, each fixing the last, often beat a forest of the same size.

XGBoost is a free library that does gradient boosting fast, with guards against overfitting built in. It came out in 2014. Of 29 winning entries in Kaggle contests in 2015, 17 used it. On table data such as sales, loans or sign-ups, it is still often the model to beat.

People reach for deep learning on every problem. On rows and columns, boosted trees are often more accurate, faster to train and easier to explain. Deep learning wins on photos, sound and text, where the raw input has no neat columns. Pick by the shape of the data.

## The three side by side

| Model | How the trees work together | Strength |
| --- | --- | --- |
| Decision tree | One tree on its own | Easy to read and explain |
| Random forest | Many trees side by side, then a vote | Hard to overfit, little tuning |
| Gradient boosting | Trees in a row, each fixing the last | Often the best on tables |

## Which one to pick

1. **Pick one decision tree.** When people must read and trust every rule, like a loan choice you must explain.
2. **Pick a random forest.** When you want a strong first model fast, with little tuning.
3. **Pick gradient boosting.** When you need the best score on table data and have time to tune it.

![Two panels. Random forest: three trees side by side vote fake, fake and real, and fake wins 2 to 1. Gradient boosting: trees in a row, where each one fixes the errors of the one before, and their answers are added up for the final answer.](https://learn.benibix.com/diagrams/ml-forest-boosting.svg)

### Example: Lucy predicts meals

Lucy's restaurants predict how many meals each site sells per day. One decision tree misses by 38 meals on average.

A random forest of 300 trees cuts the miss to 24. Gradient boosting with 500 small trees gets it to 19, so that is the one she uses.

**Words in this lesson:** [Decision Tree](https://learn.benibix.com/term/ai/decision-tree), [Ensemble Learning](https://learn.benibix.com/term/ai/ensemble-learning), [Random Forest](https://learn.benibix.com/term/ai/random-forest), [Gradient Boosting](https://learn.benibix.com/term/ai/gradient-boosting), [XGBoost](https://learn.benibix.com/term/ai/xgboost)

> **The one thing to remember.** A forest grows its trees side by side and lets them vote. Boosting grows them in a row, each one fixing the last.

**Question:** How does gradient boosting build its trees?

**Answer:** One after another, each fixing the errors before it. Each new tree learns from the errors the earlier ones left, and the answers are added up. A random forest is the one that grows them side by side.

---

Source: https://learn.benibix.com/learn/ai-machine-learning/decision-trees-and-forests
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
