# A/B Test

> Two versions, shown to two random groups. You keep the one that does better.

**Field:** Marketing & SEO  
**Category:** Attribution & Testing  
**Also known as:** A/B Testing, Split Test, Split Testing

## Think of it like

It is like baking two cakes with one change. Friends taste both, and you keep the one they like.

## Definition

An A/B test shows two versions of something to two random groups at the same time. Version A is the current one, and version B has one change. You then compare which one gets more of the result you want, such as sign-ups. Enough people must take part, or the winner may be luck. A statistics check tells you if the gap is real.

## Example

A shop tests a green Buy button against its blue one. Over two weeks and 40,000 visits, green gets 4.1% of visitors to buy, against 3.8%.

## Why it matters

It swaps opinions for evidence, one change at a time.

## Common mistake

Stopping the test the moment one side pulls ahead. Early leads often vanish, so set the run length first.

## Three principles

- **One change.** So you know what worked.
- **Random groups.** Split the traffic fairly.
- **Enough people.** Small tests crown lucky winners.

## Related terms

- [Multivariate Test](https://learn.benibix.com/term/marketing/multivariate-test)
- [A/A Test](https://learn.benibix.com/term/marketing/a-a-test)
- [Conversion Rate Optimization (CRO)](https://learn.benibix.com/term/marketing/conversion-rate-optimization)
- [Holdout Group](https://learn.benibix.com/term/marketing/holdout-group)
- [Landing Page](https://learn.benibix.com/term/marketing/landing-page)
- [Statistical Significance](https://learn.benibix.com/term/general/statistical-significance)
- [P-Value](https://learn.benibix.com/term/general/p-value)

## Told apart from

- [A/B Test vs Multivariate Test](https://learn.benibix.com/compare/ab-test-vs-multivariate-test): An A/B test checks one change at a time. A multivariate test checks every mix of several changes, so it needs far more visitors.
- [Preference Testing vs A/B Testing](https://learn.benibix.com/compare/preference-testing-vs-ab-test): People think a preference test is an A/B test, and it is not. A preference test asks people which design they like, and why. An A/B test shows each design to real visitors on a live site and counts what they do. One measures what people say. The other measures what they do, and the two can disagree.

## Taught in

- [Opportunity Solution Trees](https://learn.benibix.com/learn/business-product-management/opportunity-solution-tree): Product Management, 7/13
- [Email Open and Click Rates](https://learn.benibix.com/learn/marketing-email/email-open-and-click-rates): Email Marketing, 11/11

![Three panels of split traffic. In an A/A test both arms show the same page. In an A/B test the arms show a blue and a green button. In a multivariate test four arms show every mix of two headlines and two images.](https://learn.benibix.com/diagrams/ab-test.svg)

---

Source: https://learn.benibix.com/term/marketing/ab-test
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-20
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
