Skip to content
BENIBIX Learn

P-Hacking

General🔬 ResearchResearch Quality

Also known as Data Dredging, Fishing for Significance, Significance Chasing

Doing test after test, then showing just the one that came out the way you hoped.

Think of it like

It is like a dart game. You draw the target after you throw.

Example

A team finds no effect of a diet. They drop some people, split by age, then by sex, and at last find one group where p is 0.04. The paper reports only that group.

Why it matters

It is one reason many published findings fail to repeat. Writing down the plan before the data comes in guards against it.

Definition

P-hacking is trying many ways to analyse data until one gives a p-value under 0.05, then reporting only that one. Each extra try raises the chance of a fluke. Run 20 tests on pure noise, and on average one will cross the line by luck. The result looks real, but it is likely to fail when someone repeats the study.
Right: of 20 tests on random data, one dips under 0.05, and only that one is reported.

Common mistake

Thinking it needs cheating. Most of it comes from honest people who try one more test and do not count the tries.

Key takeaways

  1. 1
    Many tries. Each test adds a chance of a fluke.
  2. 2
    One reported. The misses stay hidden.
  3. 3
    Plan first. Write down the analysis before you look.