Skip to content
BENIBIX Learn

Small Files Problem

Also known as Small File Problem, Too Many Small Files

A lake table with too many tiny files. Each read is slow.

Think of it like

It is like a book torn into single pages. Reading it means picking up a thousand sheets.

Example

A stream writes a new file every 10 seconds. After 30 days the table holds about 260,000 files, and a simple count takes minutes.

Why it matters

Slow reads cost money on every query. It is one of the most common reasons a lake gets slow.

Definition

The small files problem is when a table is split into far too many tiny files. Each file costs time to open and to list, however small it is. A query that reads a million small files spends most of its time on that. It often comes from streaming jobs that write a little every few seconds. The fix is to merge the files into fewer large ones, which is called compaction.
The left box is the small files problem. Every query has to open each tiny file.

Common mistake

Adding more compute to make the query faster. The time goes on opening files, so merge them first.

Key takeaways

  1. 1
    Many tiny files. Each one costs time to open.
  2. 2
    Often from streams. Small writes, very often.
  3. 3
    Fixed by merging. Compaction makes fewer, larger files.