Skip to content
BENIBIX Learn

Needle in a Haystack Test

AI📏 Evaluation & BenchmarksBenchmarks and Leaderboards

Also known as Needle in a Haystack, NIAH, Needle-in-a-Haystack Benchmark

A test that hides one fact in a long text and asks the model to find it.

Think of it like

It is like hiding one odd sentence in a long book. Then you ask if the reader spotted it.

Example

Lucy hides the line 'The secret code is 7429' in a 300-page report. The model finds it near the start but misses it in the middle.

Why it matters

Models now take very long inputs. This test checks if they can really use every part of them.

Definition

A needle in a haystack test checks if a model can find one fact hidden in a very long text. A short line, the needle, is placed somewhere in a long pile of other text. The model is then asked a question that only that line answers. The test is run many times, moving the needle from start to end and making the text longer. The results form a grid that shows where the model misses. Passing it shows the model can find a fact, not that it understands the whole text.
Each cell is one run. The crosses show where, and at what length, the model lost the needle.

Common mistake

Thinking a perfect score means the model reasons well over long texts. Finding one line is far easier than linking many.

Key takeaways

  1. 1
    Hide one fact. Then ask about it.
  2. 2
    Move it around. Start, middle and end.
  3. 3
    Grow the pile. Try longer and longer texts.