Skip to content
BENIBIX Learn

Apache Arrow

Also known as Apache Arrow Format, PyArrow, Arrow Columnar Format

A shared way to hold tables in memory. Tools can swap them fast.

Think of it like

It is like shops that all use the same boxes. Goods move from shop to shop without being unpacked.

Example

A Polars script hands a table to DuckDB. Both speak Arrow, so the 10 million rows move across without being converted.

Why it matters

Converting data between tools can take longer than the work itself. A shared layout removes that cost.

Definition

Apache Arrow is an open standard for holding table data in memory, column by column. It works the same in Python, Java, Rust and more. When two tools both use Arrow, they can pass data at little or no cost. There is no slow step to convert it first. Parquet is for storing data on disk, and Arrow is the layout in memory.
Arrow keeps data in columns, like the right side. It does this in memory, not on disk.

Common mistake

Thinking Arrow is a file format like Parquet. Arrow is for memory. Parquet is for disk.

Key takeaways

  1. 1
    In memory. Not mainly a file on disk.
  2. 2
    Columnar. Each column kept together.
  3. 3
    Shared. Many languages, one layout.

Compare