Apache ORC

Apache ORC is a columnar file format from the Hive ecosystem. This page walks through its physical layout, how its layered statistics let engines skip data, how it encodes columns, and how it compares with Parquet and Avro.

1. What ORC Is

Apache ORC (Optimized Row Columnar) is an open-source columnar file format for analytic data. It was created in 2013 within the Apache Hive project, by Hortonworks working with Facebook, to replace Hive's older RCFile format, and became a top-level Apache project in 2015. Hive, Spark, Trino, Presto and Impala can all read it, and Apache Iceberg supports it as a data file format alongside Parquet and Avro.

Like Parquet, ORC stores each column's values together instead of storing whole rows together, which is why it compresses well and lets queries read only the columns they need. If that idea is new, read Row vs Columnar Storage first.

2. Why It Matters

Analytic queries typically scan many rows but touch few columns, and filter heavily. ORC is built for exactly that: column pruning skips columns, built-in indexes skip row groups, and type-aware encodings shrink the bytes that remain. For a Hive-centric warehouse it was historically the default and best-supported format, including Hive's ACID transactional tables, which are implemented on ORC. Knowing ORC also helps you reason about Parquet, since the two share most ideas and differ in the details.

3. How It Works

An ORC file is a sequence of stripes followed by a file footer and a postscript. Stripes are large (the default target is 64 MB) so that reads are long sequential I/O. Each stripe holds three parts:

• Index data: for every column, an entry per row group (10,000 rows by default) with min, max, count and has-null statistics, plus positions so a reader can seek into the middle of a column stream. Bloom filters can optionally be added per column.
• Row data: each column split into one or more streams, for example a PRESENT stream for nulls, a DATA stream for values and a LENGTH stream for variable-width types.
• Stripe footer: the directory of streams and the encoding of each column.

The file footer stores the schema, the list of stripes with their offsets and row counts, and file-level statistics per column. Stripe-level statistics live in the file metadata section. Because the postscript sits at the very end and its length is stored in the final byte, a reader opens the file by reading the tail, then decides which stripes and streams to fetch.

This three-level statistics hierarchy (file, stripe, row group) is what makes predicate pushdown effective: the engine passes the query's filter to the ORC reader, which eliminates whole stripes or row groups whose ranges cannot match. Pushdown works best when data is sorted or clustered on the filter column, because then each row group covers a narrow range.

Within a stream ORC applies type-specific encodings first: run-length and delta encoding with bit-packing for integers, dictionary encoding for strings with few distinct values, a bit-per-row PRESENT stream for nulls. A general codec (zlib by default, or Snappy, LZ4, Zstandard) then compresses the result in chunks so that seeks remain possible.

4. Worked Example in PyArrow

PyArrow can write and read ORC directly. The example writes a sorted table, reads only two columns, and uses a filter so the reader can skip stripes.

import numpy as np, pyarrow as pa, pyarrow.orc as orc, pyarrow.dataset as ds

n = 1_000_000
rng = np.random.default_rng(0)
table = pa.table({
    "order_id": np.arange(n),
    "country":  rng.choice(["US", "DE", "JP", "BR"], n),
    "amount":   rng.gamma(2.0, 150.0, n).round(2),
}).sort_by("amount")                     # clustering improves min/max skipping

orc.write_table(table, "orders.orc", compression="zstd", stripe_size=8 * 1024 * 1024)

f = orc.ORCFile("orders.orc")
print("stripes:", f.nstripes, "rows:", f.nrows, "schema:", f.schema)

# Column pruning: only two columns are decoded
small = orc.read_table("orders.orc", columns=["country", "amount"])
print(small.num_columns)

# Dataset API with a filter expression
dset = ds.dataset("orders.orc", format="orc")
big = dset.to_table(columns=["order_id", "amount"], filter=ds.field("amount") > 900)
print("rows > 900:", big.num_rows)

In Spark the equivalent is df.write.format("orc").option("compression","zstd").save(path); with spark.sql.orc.filterPushdown enabled (the default in recent versions) filters are pushed to the reader. In Hive, CREATE TABLE ... STORED AS ORC TBLPROPERTIES ("orc.compress"="ZSTD", "orc.bloom.filter.columns"="customer_id") configures compression and Bloom filters.

5. Tuning ORC Files

• Stripe size. Larger stripes mean longer sequential reads and better compression but more memory when writing. Keep the default 64 MB unless writers run out of memory; on object stores, align stripes with the split size your engine uses.
• Row index stride. The 10,000-row default balances index size against skipping precision. A smaller stride skips more finely for selective point lookups at the cost of a larger index.
• Codec. zlib gives good ratios; Zstandard usually matches or beats it with faster decompression; Snappy and LZ4 favour speed. Measure on your own data.
• Dictionary threshold. Writers fall back from dictionary to direct encoding when the ratio of distinct values to rows is high, because a huge dictionary saves nothing.
• Sort order. The cheapest optimisation of all: write data sorted on the column most queries filter by, and the indexes start doing real work.

Inspect a file before tuning: orc-tools meta file.orc (from the Java ORC tools) or orc-metadata (from the C++ library) prints stripes, encodings and statistics, so you can see whether ranges are narrow enough to skip and whether dictionaries were used.

6. ORC vs Parquet vs Avro

In practice ORC and Parquet perform similarly; the choice is usually decided by ecosystem support. Parquet has broader support outside Hadoop, so new lakes usually default to it, while ORC remains common where Hive is central.

7. Pitfalls

• Small files. Many tiny ORC files mean tiny stripes, more footers to read and little compression benefit. Compact them.
• Unsorted data. Random order makes min/max ranges wide, so pushdown skips nothing. Sort or cluster on common filter columns.
• Schema evolution by position. Older Hive setups matched ORC columns by position; renames and reorders can misalign data. Prefer name-based resolution where your engine supports it.
• Bloom filters everywhere. They cost space and write time; add them only for equality filters on high-cardinality columns.
• Mixed engine versions. Very old readers may not understand newer encodings or codecs such as Zstandard.

8. Related Topics

• Row vs Columnar Storage — the underlying idea
• Apache Avro — the row-oriented counterpart
• Apache Parquet — the closest alternative
• Apache Iceberg — a table format that can store ORC files
• Storage Formats — section overview

9. Further Reading

• Apache ORC specification, orc.apache.org (file format v1 specification).
• Huai et al. (2014), Major Technical Advancements in Apache Hive, SIGMOD.
• Apache Arrow documentation, pyarrow.orc module.
• Apache Spark documentation, "ORC Files" data source guide.