Lance

Lance is an open-source columnar data format optimized for machine-learning and AI workloads. Built by LanceDB, Lance is positioned as a successor to Parquet for ML use cases — cheap random access, native vector embeddings, automatic versioning, and zero-copy reads into PyTorch / NumPy / Arrow without a deserialization step. LanceDB, the embedded vector database, is built directly on the Lance format.

1. How It Is Organised

A Lance dataset is a directory of immutable fragments plus a series of versioned manifests. Each fragment holds a range of rows, stored column by column in Lance's own file format, which uses encodings designed so that any single row can be located without decoding a large page. The manifest for each version lists the fragments, the schema and any indexes. Appends add fragments; deletes write small deletion files rather than rewriting data; adding a column writes new column files for existing fragments. Readers open a specific manifest, so they always see a consistent version.

2. Key Features:

Random-Access Columnar. Unlike Parquet, which is optimized for full-column scans, Lance is built for fast point lookups by row ID — the dominant pattern in ML training (random shuffled batches) and vector retrieval.
Native Vector Type. First-class fixed-size list (vector) columns with built-in ANN indexing — IVF, IVF-PQ, HNSW — stored alongside the data.
Zero-Copy to Arrow. Tables map directly into Arrow buffers; no row-group parsing, no allocation per batch. Drop-in to PyTorch DataLoader and JAX.
Versioning & Time Travel. Every table mutation produces a new version; rollback and reproducible-training queries are first-class.
Schema Evolution. Add or remove columns without rewriting data files.
Cloud Storage Native. S3, GCS, Azure Blob; range reads optimized for object storage.

3. Lance vs. Parquet:

Lance — Random access by row, vectors as a data type, ML-pipeline-shaped. Newer, less ecosystem support than Parquet today.
Parquet — Column-scan and predicate-pushdown for analytics, the lingua franca of the lakehouse. Sequential reads beat random reads dominantly.
The two coexist: Parquet for analytical / BI tables, Lance for ML feature stores and embedding indexes.

4. Use Cases:

Embedding feature stores feeding RAG retrieval and recommendation systems.
ML training datasets where shuffle reads dominate — Parquet’s row-group model is a poor fit.
Reproducible model training — pin a dataset version for the duration of a training run.
Vector search via LanceDB — lightweight, embedded, no separate vector-DB server.

5. Vector Search

Lance can build approximate nearest neighbour indexes on vector columns. The common IVF_PQ index first clusters vectors into partitions (IVF), then compresses each vector into a short product-quantisation code (PQ). At query time Lance probes the partitions nearest to the query, computes approximate distances on the compressed codes, and optionally re-ranks the best candidates with the original vectors. Because the index lives inside the dataset, the same files serve as the training dataset, the feature store and the search index.

6. Python Example

import lance, numpy as np, pyarrow as pa   # pip install pylance

n, dim = 10_000, 128
vectors = np.random.rand(n, dim).astype("float32")
table = pa.table({
    "id": pa.array(range(n)),
    "label": pa.array(["cat" if i % 2 else "dog" for i in range(n)]),
    "vector": pa.FixedSizeListArray.from_arrays(pa.array(vectors.ravel()), dim),
})
lance.write_dataset(table, "animals.lance")              # version 1

ds = lance.dataset("animals.lance")
print(ds.take([5, 500, 5000], columns=["id", "label"]))   # random access by row

ds.create_index("vector", index_type="IVF_PQ",
                num_partitions=64, num_sub_vectors=16)
hits = ds.to_table(nearest={"column": "vector", "q": vectors[0], "k": 5},
                   filter="label = 'dog'", columns=["id", "label"])
print(hits)

old = lance.dataset("animals.lance", version=1)           # time travel

The take call is the operation Lance is optimised for: fetching scattered rows, as a shuffled data loader or a retrieval step does, without reading whole row groups. LanceDB adds a database API (tables, full-text search, hybrid search) on top of the same format.

7. When to Use It

Workload	Lance	Parquet / Iceberg / Delta
Full scans and aggregations for BI	Works, not its focus	Best fit, broad engine support
Random row access, shuffled training reads	Best fit	Slow: must decode row groups
Embeddings with ANN search	Built in	Needs a separate vector database
Images, audio and large blobs next to metadata	Designed for it	Possible but awkward
Multi-engine SQL (Trino, Snowflake, Athena)	Limited	Strong

8. Pitfalls

Many small appends create many fragments; run compaction (ds.optimize.compact_files()) periodically.
Old versions keep their files until cleaned up; schedule version cleanup to control storage.
Indexes are not automatically updated for newly appended rows in every case; new rows may be searched by brute force until the index is optimised.
Ecosystem support is narrower than Parquet's, so keep analytical copies in Parquet-based formats if other teams need them.