Apache Parquet vs Apache Avro
People think Parquet and Avro do the same job. Both are open file formats with a schema. Parquet stores data by column, so reports that read a few columns are fast. Avro stores whole rows, so it suits sending one record at a time, as in event streams.
Apache Parquet
A file type that keeps data by column. Most data lakes use it.
Apache Parquet is an open file format that stores data by column. A file is split into row groups, and each group stores its columns apart. Each column chunk keeps small notes, such as its smallest and largest value. Readers use those notes to skip chunks they do not need. It is the most common file format in data lakes.
Apache Avro
A small file type that keeps whole rows. The plan for the data rides along.
Apache Avro is an open file format that stores data row by row in a small binary form. Its schema is written in JSON and travels with the data. A reader uses the schema to decode each record. Fields can be added or removed over time, with rules that keep old and new readers working. It is popular for messages in event streams.
In one line
- Apache Parquet
- A file type that keeps data by column. Most data lakes use it.
- Apache Avro
- A small file type that keeps whole rows. The plan for the data rides along.