FoundationDB

FoundationDB (FDB) is an open-source, distributed, ordered key-value store with strictly serializable multi-key ACID transactions. Originally built by FoundationDB Inc., acquired by Apple in 2015 and open-sourced in 2018, it is deliberately minimal: it stores byte-string keys and values in sorted order and gives you transactions over them. Higher-level data models (documents, records, tables, queues, indexes) are built on top as "layers". This page explains the architecture, the transaction model, the Python example below, and when FDB is a good fit.

1. Overview

Most key-value stores force a trade-off: scale out and give up multi-key transactions, or keep transactions and stay on one machine. FoundationDB was designed to avoid that trade-off. A cluster spreads ordered key ranges across many storage servers, automatically splits and moves ranges as they grow, and still lets any transaction read and write any set of keys with serializable isolation. The project is known for its deterministic simulation testing: the whole cluster, including network and disk faults, runs inside a single-threaded simulator so that bugs found in testing can be replayed exactly.

Key properties:

• Ordered keys. Keys are sorted bytewise, so range reads (get_range) are efficient and prefix scans are the basic query primitive.
• ACID, strictly serializable transactions across any keys in the cluster, using optimistic concurrency control and MVCC.
• Limits by design. Keys up to 10 KB, values up to 100 KB, and a transaction should finish within 5 seconds and stay under 10 MB of writes. Large objects are chunked across keys.
• Layers. The Tuple layer encodes typed, order-preserving keys; the Directory layer maps paths to short prefixes; the Record Layer (Java) adds schemas and indexes; the Document Layer once offered a MongoDB-compatible API.

2. How It Works

The cluster is split into a write path and a read path. On the write path, a client first obtains a read version from a GRV (get-read-version) proxy. All reads in the transaction see a consistent snapshot at that version, served directly by storage servers, which keep a short MVCC history (about five seconds). Writes are not sent immediately; the client buffers them. At commit time the client sends its read-conflict ranges and write set to a commit proxy. The sequencer hands out a commit version, resolvers check whether any key the transaction read was modified by another transaction committed after its read version, and if not, the mutations are made durable on the transaction logs and acknowledged. Storage servers pull mutations from the logs asynchronously.

Because conflict checking is optimistic, transactions never hold locks. A conflicting transaction simply fails at commit with error 1020 (not_committed) and is retried. The language bindings wrap this in a retry loop: in Python, a function decorated with @fdb.transactional is re-run automatically on retryable errors, which is why such functions must be idempotent apart from their database effects.

3. Functions to insert, retrieve all, and retrieve a specific key

import foundationdb
import json

foundationdb.api_version(630)
db = foundationdb.open()

# Helper: Convert integer ID to bytes
def id_to_key(student_id: int) -> bytes:
    return str(student_id).encode()

# Insert multiple students
@foundationdb.transactional
def insert_students(tr):
    students = {
        1001: {"first": "Alice",   "last": "Smith",   "email": "alice@example.com",   "city": "Seattle",  "state": "WA"},
        1002: {"first": "Bob",     "last": "Johnson", "email": "bob@example.com",     "city": "Portland", "state": "OR"},
        1003: {"first": "Charlie", "last": "Lee",     "email": "charlie@example.com", "city": "San Jose", "state": "CA"}
    }
    for student_id, info in students.items():
        key = id_to_key(student_id)
        tr[key] = json.dumps(info).encode()

# Read all students
@foundationdb.transactional
def read_all_students(tr):
    print("All student records:")
    for key, value in tr.get_range(b'', b'\xff'):
        student_id = key.decode()
        info = json.loads(value.decode())
        print(f"ID: {student_id}, Name: {info['first']} {info['last']}, Email: {info['email']}, City: {info['city']}, State: {info['state']}")

# Retrieve a specific student by ID
@foundationdb.transactional
def get_student_by_id(tr, student_id: int):
    key = id_to_key(student_id)
    value = tr.get(key).wait()
    if value:
        info = json.loads(value.decode())
        print(f"Record for ID {student_id}: {info['first']} {info['last']}, {info['email']}, {info['city']}, {info['state']}")
    else:
        print(f"No record found for ID {student_id}")

# Main entry
def main():
    insert_students(db)
    read_all_students(db)
    get_student_by_id(db, 1002)  # Retrieve Bob's record

if __name__ == '__main__':
    main()

4. Architecture

The diagram shows the main roles. A small set of coordinators stores the cluster's configuration using Paxos and elects a cluster controller, which in turn recruits the other processes. Commit and GRV proxies are the client-facing entry points, the sequencer (master) assigns monotonically increasing versions, resolvers detect read-write conflicts, transaction logs make commits durable, and storage servers own shards of the key space and serve reads. Data distribution moves and splits shards in the background, and each shard is replicated according to the configured redundancy mode (single, double, triple). If any role in the write path fails, the cluster performs a recovery that recruits a new generation of the transaction system, typically within seconds.

5. Data Modelling Notes

The example on this page stores JSON under a bare student-ID key, which works but is not idiomatic. Real applications use the Tuple layer and a subspace so that keys sort correctly and different entity types do not collide:

import fdb, json
fdb.api_version(710)
db = fdb.open()

students = fdb.Subspace(('students',))
by_state = fdb.Subspace(('idx', 'state'))

@fdb.transactional
def put_student(tr, sid, info):
    old = tr[students.pack((sid,))]
    if old.present():                      # maintain the index atomically
        prev = json.loads(old)
        del tr[by_state.pack((prev['state'], sid))]
    tr[students.pack((sid,))] = json.dumps(info).encode()
    tr[by_state.pack((info['state'], sid))] = b''

@fdb.transactional
def students_in_state(tr, state):
    r = by_state.range((state,))
    return [by_state.unpack(k)[1] for k, _ in tr.get_range(r.start, r.stop)]

Because the record and its index entry are written in the same transaction, the index can never drift from the data. This is the core FoundationDB pattern: secondary indexes, counters (using atomic add mutations to avoid conflicts), and queues are all just extra keys maintained transactionally. Note that the official Python package is imported as fdb; the code above on this page uses an alias name, so adapt the import to your installation.

6. When to Use It

FoundationDB is used as the metadata and transaction backbone of systems such as Snowflake's metadata store and Apple's CloudKit (via the Record Layer). It suits teams that want strong guarantees and are willing to design their own key layout.

7. Pitfalls

• Hot keys. Many transactions updating one key (a global counter) conflict constantly; use atomic mutations or shard the counter.
• Long transactions fail with transaction_too_old after about five seconds; page through large ranges in multiple transactions.
• Side effects in retried functions. Anything non-idempotent inside a @transactional function (sending email, incrementing a Python variable) may run more than once.
• API version pinning. Always call fdb.api_version() with the version your client library and cluster support.

8. Related Topics

• LMDB — an embedded, single-machine ordered key-value store.
• Key-value stores overview.

Featured Videos