# Holdout Group

> People who do not see a campaign, on purpose. They show what it really added.

**Field:** Marketing & SEO  
**Category:** Attribution & Testing  
**Also known as:** Holdout, Holdout Test

## Think of it like

It is like keeping a few plants away from the plant food. Then you can see what the food really did.

## Definition

A holdout group is a set of people you keep from seeing a campaign on purpose. Everyone else gets it as normal. Later you compare how the two groups behaved. The difference shows what the campaign added. The people must be picked at random, or the two groups will differ for other reasons.

## Example

An app sends a welcome series to 90% of new users and holds back 10%. After a month, 31% of the emailed group pay, against 28% of the holdout.

## Why it matters

Without one, you cannot tell if a campaign changed a thing. It may just have taken the credit.

## Common mistake

Making the holdout too small. With only a few hundred people, the gap can be pure luck.

## Three principles

- **Picked at random.** Both groups must start alike.
- **Kept out.** They see none of the campaign.
- **Big enough.** Small groups give noisy answers.

## Related terms

- [Incrementality](https://learn.benibix.com/term/marketing/incrementality)
- [A/B Test](https://learn.benibix.com/term/marketing/ab-test)
- [Geo Lift Test](https://learn.benibix.com/term/marketing/geo-lift-test)
- [Drip Campaign](https://learn.benibix.com/term/marketing/drip-campaign)

![Two bars side by side. The group that saw the ads reached 1,200 sales and the held-out group reached 1,000. The top 200 of the first bar is shaded and labelled lift.](https://learn.benibix.com/diagrams/incrementality.svg)

---

Source: https://learn.benibix.com/term/marketing/holdout-group
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-20
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
