Key-Value Stores

1. Overview

A key-value store maps opaque keys to opaque values and exposes a small API: get, put, delete, and in ordered stores, range scans by key prefix. That narrow interface is what makes them fast and composable; many larger databases are built on top of one. The eight stores in this section fall into two groups. Embedded stores (LevelDB, RocksDB, BadgerDB, LMDB, bbolt) are libraries linked into a single process with no server or network hop. Networked stores (Redis, Memcached, FoundationDB) run as servers shared by many clients, ranging from a pure cache to a distributed transactional database.

2. Families and storage engines

Embedded stores differ mainly in their on-disk structure. LSM-tree engines (LevelDB, RocksDB, BadgerDB) append writes to a log and memtable and merge sorted files in the background, giving high write throughput at the cost of compaction work and some read amplification; BadgerDB separates large values from keys to reduce that cost. B+tree engines (LMDB, bbolt) update pages copy-on-write in a single file, which gives very fast reads, simple crash safety, and one writer at a time. Among servers, Memcached is a volatile cache with no persistence; Redis keeps data in memory with optional persistence and rich data structures; FoundationDB distributes an ordered key space across a cluster with serializable transactions.

3. When to use which

4. Example: the same operations in two stores

# Redis: networked, with TTLs and atomic counters
redis-cli SET session:42 '{"user":"kevin"}' EX 3600
redis-cli INCR rate:ip:203.0.113.7
redis-cli GET session:42

# LMDB: embedded, transactional, ordered keys
import lmdb
env = lmdb.open("/tmp/kv-demo", map_size=10 * 2**20)
with env.begin(write=True) as txn:
    txn.put(b"user:001", b"alice")
    txn.put(b"user:002", b"bob")
with env.begin() as txn:
    cur = txn.cursor()
    cur.set_range(b"user:")          # ordered prefix scan
    for k, v in cur:
        if not k.startswith(b"user:"):
            break
        print(k, v)

The Redis example shows what networked stores add: expiry and atomic server-side operations shared by many clients. The LMDB example shows what embedded ordered stores add: transactions and range scans with no network hop.

5. Common pitfalls

• Treating a cache (Memcached, or Redis without persistence) as the system of record.
• Designing keys without a prefix scheme, which makes range scans and cleanup impossible.
• Ignoring LSM compaction settings and running out of disk or I/O during compaction bursts.
• Sharing one LMDB or bbolt file across processes on network file systems, where locking is unreliable.

See also wide-column stores.

Featured Videos