Avro is a compact, schema-based binary format designed for data whose shape changes over time. This page explains how schemas and the binary encoding work, how reader and writer schemas are reconciled, and how Avro is used with Kafka and a schema registry.
Apache Avro is a data serialization system that came out of the Hadoop project. It defines three things: a schema language (written in JSON) that describes the shape of a record, a compact binary encoding for records that follow a schema, and an object container file format that stores many records together with the schema that wrote them.
The key design choice is that Avro binary data contains no field names and no type tags. A record of {"id": 42, "name": "Ada"} is encoded as a variable-length integer followed by a length-prefixed string, nothing else. That makes it small and fast, but it also means you cannot decode the bytes without the schema that wrote them. Avro solves this by always keeping the writer's schema close to the data: in the file header, or by reference through a schema registry.
Data that lives longer than the code that produced it will eventually be read by programs that expect a different shape. Avro is built around that fact. Its schema resolution rules let a reader with a newer (or older) schema decode data written with a different version, as long as the changes follow compatibility rules. This is why Avro is the most common format for event streams on Kafka and for row-oriented landing zones in data lakes: producers and consumers can be upgraded independently.
Avro is row-oriented, so it suits write-heavy and record-at-a-time workloads: append-only logs, message payloads, and RPC. For analytical scans that read a few columns out of many, a columnar format such as Parquet is usually a better fit; it is common to land data as Avro and compact it into Parquet later.
A schema is a JSON document. Primitive types are null, boolean, int, long, float, double, bytes and string. Complex types are record, enum, array, map, union and fixed. Logical types layer meaning on top: date is an int of days since the epoch, timestamp-millis is a long, decimal is bytes or fixed with a precision and scale.
Encoding is straightforward. Integers and longs use zig-zag variable-length encoding, so small magnitudes take one byte. Strings and bytes are a length followed by the raw bytes. A record is just its fields concatenated in schema order. A union is written as the index of the chosen branch followed by the value, which is how optional fields work: ["null","string"] writes 0 for null or 1 followed by the string.
The container file in the diagram starts with a header holding the magic bytes, metadata (including the full schema and the compression codec such as deflate, snappy or zstandard), and a random 16-byte sync marker. Records are grouped into blocks; each block is optionally compressed and ends with the sync marker. Because the marker is known, a distributed engine can start reading in the middle of a file, scan forward to the next marker, and process blocks in parallel. That is what makes Avro files splittable even when compressed.
When a reader decodes data it holds two schemas: the writer schema (what the bytes actually contain) and its own reader schema (what the application wants). Avro walks both in parallel. Fields are matched by name (aliases can map renamed fields). A field in the writer but not in the reader is skipped. A field in the reader but not in the writer takes the reader's default; if no default exists, resolution fails. Some type promotions are allowed, for example int to long, float or double, and string to bytes.
From these rules come the practical compatibility modes used by schema registries:
Renaming a field, changing a type incompatibly, or adding a required field with no default are breaking changes. The usual remedy is a new field plus a migration period, or a new topic.
The script below defines version 1 of a schema, writes a container file with the fastavro library, then reads it back with a version 2 reader schema that adds an optional email field. It runs with pip install fastavro.
from fastavro import writer, reader, parse_schema
v1 = parse_schema({
"type": "record", "name": "Customer", "namespace": "shop",
"fields": [
{"name": "id", "type": "long"},
{"name": "name", "type": "string"},
],
})
v2 = parse_schema({
"type": "record", "name": "Customer", "namespace": "shop",
"fields": [
{"name": "id", "type": "long"},
{"name": "name", "type": "string"},
{"name": "email", "type": ["null", "string"], "default": None},
],
})
rows = [{"id": 1, "name": "Ada"}, {"id": 2, "name": "Grace"}]
with open("customers.avro", "wb") as f:
writer(f, v1, rows, codec="deflate")
with open("customers.avro", "rb") as f:
for rec in reader(f, reader_schema=v2):
print(rec)
# {'id': 1, 'name': 'Ada', 'email': None}
# {'id': 2, 'name': 'Grace', 'email': None}The file header stores v1, so the reader knows exactly what the bytes contain and fills email from the default. If v2 had declared email as a plain string with no default, the read would fail, which is precisely the kind of change a registry in backward mode would reject.
Embedding a full schema in every Kafka message would waste far more bytes than the payload itself. The common pattern (popularised by the Confluent Schema Registry, and also offered by Apicurio and AWS Glue Schema Registry) is to store schemas centrally. The producer's serializer registers the schema under a subject (by default <topic>-value), receives a numeric id, and writes a small header followed by the Avro bytes. The consumer reads the id, fetches the schema once, caches it, and decodes with its own reader schema.
The registry also enforces the compatibility mode per subject, so an incompatible change is rejected at deploy time rather than discovered as a consumer crash at 3 a.m.
# pip install "confluent-kafka[avro]"
from confluent_kafka import Producer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer
from confluent_kafka.serialization import SerializationContext, MessageField
schema_str = open("customer_v2.avsc").read()
sr = SchemaRegistryClient({"url": "http://localhost:8081"})
ser = AvroSerializer(sr, schema_str)
p = Producer({"bootstrap.servers": "localhost:9092"})
value = {"id": 3, "name": "Linus", "email": None}
p.produce("customers",
value=ser(value, SerializationContext("customers", MessageField.VALUE)))
p.flush()| Format | Layout | Schema | Evolution | Best for |
|---|---|---|---|---|
| Avro | Row, binary | JSON schema travels with data or by registry id | Rich reader/writer resolution by field name | Kafka events, row landing zones, Hadoop-era pipelines |
| Protocol Buffers | Row, binary | .proto compiled to code | By field number; add optional fields, never reuse numbers | gRPC services, typed APIs |
| JSON | Row, text | Optional (JSON Schema) | Ad hoc; consumers must be tolerant | Human-readable APIs, low volume |
| Parquet | Columnar, binary | Embedded in footer | Add columns; engine-dependent | Analytical scans, lakehouse tables |
| ORC | Columnar, binary | Embedded | Similar to Parquet | Hive-centric warehouses |
A rough rule: use Avro or Protobuf on the wire, and Parquet (often under a table format such as Iceberg or Delta) at rest for analytics.
["null", "string"] with default: null.decimal as raw bytes or timestamps as plain longs. Check that every consumer library supports the logical types you use.