Skip to content
BENIBIX Learn

Statistical Significance

General🔢 Math & StatisticsStatistics

Also known as Statistically Significant, Significance Level, Alpha Level

A result that clears a bar set before the test. Chance alone would rarely give it.

Think of it like

It is like the bar in a high jump. Clear the bar and the jump counts.

Example

A shop tests two buttons on 1 million visitors. The new one wins by a tenth of a percentage point, with p = 0.01. The result is significant, but the gain is too small to matter.

Why it matters

News stories often use the word as if it meant big. Knowing the difference stops you from trusting a result too much.

Definition

A result is called significant when its p-value falls below a line set before the test. That line is called the significance level. The most common one is 0.05. Clearing it means the result would be rare if there were no real effect. It does not mean the effect is big or important. With lots of data, even a tiny effect can clear it.
The dashed line is the 0.05 bar. A result past it counts as significant.

Common mistake

Setting the bar after seeing the data. Picking the line to suit the result makes the test worthless.

Key takeaways

  1. 1
    Bar set first. Choose the level before the test.
  2. 2
    0.05 is common. Some fields use far stricter bars.
  3. 3
    Not the same as big. Tiny effects can still clear it.

Learn it properly