Columnar File Formats (Parquet and ORC)

How Parquet and ORC store a table column by column inside large row groups or stripes, and how their encodings, compression and per-chunk statistics let analytic queries read a small fraction of the bytes.

Overview

A row format writes each record whole, one after another: all of row 1, then all of row 2. A columnar format splits a table into columns and writes the values of each column together: every order_id, then every amount, then every country. Apache Parquet and Apache ORC are two widely used open columnar file formats for data lakes and warehouses.

The reason is the shape of analytic queries. A report such as 'total sales by country last month' touches 3 columns of a 100-column table, but it touches millions of rows. A columnar file lets the reader fetch only those 3 columns. Values of one column also look alike, so they compress far better than mixed rows. And each file stores small summaries (such as the minimum and maximum value of each column in each chunk), so a reader can skip whole chunks that cannot match a filter.

The price is paid on the write side. A writer must buffer many rows before it can lay them out by column, and a file is written once and not updated in place. That is why streams and change events usually arrive in a row format and are later compacted into Parquet or ORC.

What you will be able to do

  • Explain why a columnar scan reads less data than a row scan, with numbers.
  • Draw the inside of a Parquet file (row groups, column chunks, pages, footer) and an ORC file (stripes, indexes, tail).
  • Describe dictionary, run-length and bit-packing encodings, and where compression applies.
  • Show how min/max statistics and predicate pushdown skip data, and why sort order decides how much.
  • Choose file and row-group sizes, and say when Avro or JSON is the better format.

Key concepts

  • Row vs columnar layout
  • Row groups and stripes
  • Column chunks and pages
  • Footer metadata
  • Dictionary and run-length encoding
  • Min/max statistics
  • Predicate pushdown
  • Repetition and definition levels
  • Small-files problem
  • Avro for row-at-a-time data

Interview questions

  • Why are columnar formats faster for analytics?
  • Walk me through the structure of a Parquet file.
  • How does predicate pushdown work, and when does it fail to help?
  • Parquet or ORC: how do they differ?
  • Explain dictionary encoding and when it stops working.
  • Your data lake has millions of small Parquet files. What do you do?
  • How do columnar formats store nested data such as a list inside a record?
  • When would you not use Parquet?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio