pandas vs Polars
People think Polars is just a faster pandas, and it is more than that. Both hold a table in memory on one machine. pandas runs each step as soon as you write it, mostly on one core. Polars can plan the whole job first, read only what the answer needs, and use every core. The code looks alike, but the habits differ.
pandas
A free tool in Python for tables.
pandas is a free Python library for working with tables as DataFrames. It reads CSV, Excel, JSON, Parquet and SQL tables. It can clean, join, group and reshape data in a few lines. It keeps all the data in memory on one computer. That makes it fast for small and medium data, but it struggles once the data is bigger than memory.
Polars
A fast tool for tables. It plans the whole job first.
Polars is a free DataFrame library written in Rust, with a Python API. It stores data by column and uses all the cores of the computer at once. Its lazy mode plans the whole query before it runs, so it can skip work and read less. It can also stream data that is bigger than memory. It is often many times faster than pandas.
In one line
- pandas
- A free tool in Python for tables.
- Polars
- A fast tool for tables. It plans the whole job first.