Skip to content
BENIBIX Learn

Hypothesis Test and P-Value

Level 5 · Testing ideas

How a test asks whether a result could be luck. What a p-value really means, and why a result that passes the test may still not matter.

Course contentsLesson 9 of 10

After this lesson you can

  • Walk through the steps of a hypothesis test
  • Say what a p-value does and does not mean
  • Tell significant from important

12 mincovers 3 glossary entries

Worth reading first: Correlation vs Causation

What does a p-value really mean?

A hypothesis test checks whether data shows a real effect or could just be chance. It starts by supposing there is no effect, called the null hypothesis. The p-value is the chance of seeing data at least this odd if that were true. If it falls below a bar set in advance, often 0.05, the result passes. It is then said to have statistical significance.

Take a coin you suspect is unfair. You flip it 100 times and get 60 heads. If the coin were fair, how often would you see 60 or more? About 3 times in 100. That small chance makes it hard to believe the coin is fair.

A p-value of 0.03 does not mean a 3 percent chance that the coin is fair. It means that a fair coin would give a result this lopsided about 3 percent of the time. It is a statement about the data, given no effect, not about the idea itself.

The 0.05 bar means that, when there is truly no effect, about 1 test in 20 will still clear it by luck. Run 20 tests on things that do nothing, and on average one will look like a discovery. That is why a single surprising result needs to be repeated.

Statistical Significance is not the same as importance. With a big enough sample, a tiny effect can be significant. A diet that cuts weight by 100 grams over a year might clear the bar in a study of 50,000 people. It is real, and almost useless.

Not clearing the bar does not prove there is no effect either. A small study may simply be too small to see a real one. The fair conclusion is that this study could not tell.

The common mistake is to run many tests and report only the ones that cleared the bar. That makes chance look like real findings. Good studies say in advance what they will test, and report every result.

The curve is what luck alone would give. The thin tail past your result is the p-value.

Worked example

Twenty jelly bean colours

A team tests whether jelly beans cause spots. Overall, they find nothing. Then they test 20 colours one by one, and green clears the 0.05 bar.

A headline says green jelly beans cause spots. With 20 tests, one passing by luck is just what you would expect.

The words for this lesson3

Check yourself

A study reports p = 0.04. What does that mean?

Finished this lesson?

Marking it complete also marks its 3 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.