# Small Files Problem

> A lake table with too many tiny files. Each read is slow.

**Field:** Data Engineering  
**Category:** Storage & Warehousing  
**Also known as:** Small File Problem, Too Many Small Files

## Think of it like

It is like a book torn into single pages. Reading it means picking up a thousand sheets.

## Definition

The small files problem is when a table is split into far too many tiny files. Each file costs time to open and to list, however small it is. A query that reads a million small files spends most of its time on that. It often comes from streaming jobs that write a little every few seconds. The fix is to merge the files into fewer large ones, which is called compaction.

## Example

A stream writes a new file every 10 seconds. After 30 days the table holds about 260,000 files, and a simple count takes minutes.

## Why it matters

Slow reads cost money on every query. It is one of the most common reasons a lake gets slow.

## Common mistake

Adding more compute to make the query faster. The time goes on opening files, so merge them first.

## Three principles

- **Many tiny files.** Each one costs time to open.
- **Often from streams.** Small writes, very often.
- **Fixed by merging.** Compaction makes fewer, larger files.

## Related terms

- [File Compaction](https://learn.benibix.com/term/data/file-compaction)
- [Data Lake](https://learn.benibix.com/term/data/data-lake)
- [Apache Parquet](https://learn.benibix.com/term/data/apache-parquet)
- [Stream Processing](https://learn.benibix.com/term/data/stream-processing)
- [Open Table Format](https://learn.benibix.com/term/data/open-table-format)
- [Data Partitioning](https://learn.benibix.com/term/data/data-partitioning)

![Left, a box of thirty tiny grey files that a stream adds one every few seconds, and every query opens all of them. An arrow marked compact leads right to a box holding three large highlighted files with the same rows and nothing lost, so a query opens three files.](https://learn.benibix.com/diagrams/dat3-compaction.svg)

---

Source: https://learn.benibix.com/term/data/small-files-problem
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-16
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
