How data is laid out in files shapes the cost and speed of everything built on it. This section explains row and columnar formats and when to use each.
A storage format decides how records are turned into bytes on disk or in object storage. That choice drives file size, how fast queries run, whether schemas can evolve, and which tools can read the data. Formats split into two families. Row-oriented formats (CSV, JSON lines, Avro) store each record contiguously, which suits writing one event at a time and reading whole records. Columnar formats (Parquet, ORC) store each column contiguously, which suits analytics that scan a few columns across millions of rows and compress well because similar values sit together.
File formats are the bottom of the analytical stack. Table formats such as Iceberg and Delta Lake add transactions, snapshots and schema management on top of collections of Parquet (or ORC, or Avro) files, and engines read through them.
Read Row vs Columnar Storage first: it explains the physical layout difference that everything else follows from. Then read about the formats themselves. Avro is the usual choice for event streams and row-wise landing zones because of its schema evolution rules. Parquet is the default columnar format for lakes and lakehouses; ORC is its close relative from the Hive ecosystem. A typical pipeline lands Avro or JSON from Kafka, then compacts into Parquet under a table format for querying.
| Format | Layout | Schema | Compression | Typical use |
|---|---|---|---|---|
| CSV | Row, text | None (header only) | Poor, not splittable when gzipped | Exchange with spreadsheets and legacy systems |
| JSON lines | Row, text | Self-describing, untyped | Poor | APIs, logs, semi-structured landing |
| Avro | Row, binary | Embedded, evolvable | Good, splittable blocks | Kafka, CDC, row landing zones |
| Parquet | Columnar, binary | Embedded in footer | Excellent, per column | Lakehouse tables, analytics, ML features |
| ORC | Columnar, binary | Embedded | Excellent, built-in indexes | Hive and Trino warehouses |
Rule of thumb: row formats for writing and transporting events, columnar formats for storing data you will query. Avoid CSV for anything internal; it has no types, ambiguous quoting and no schema.
Whatever the format, a few practices matter more than the choice itself. File size: aim for files in the range of roughly a hundred megabytes to a gigabyte for analytical data; thousands of tiny files overwhelm planners and object-store listing. Compression codec: Snappy and LZ4 favour speed, Zstandard offers a strong balance of ratio and speed, and gzip compresses well but is slower. Partitioning: split by a column queries usually filter on (often a date), but not so finely that each partition holds only a few rows. Schema discipline: decide how columns are added and renamed before the first consumer appears, because changing a format later means rewriting data.