BadgerDB

BadgerDB is an embedded, persistent key-value store written in pure Go by the Dgraph team. Designed in 2017 to address the problem that LevelDB- and RocksDB-cgo bindings were the only practical KV options for Go programs at the time, BadgerDB is a native Go LSM implementation with a key innovation: it stores values separately from keys (WiscKey design) when values are above a threshold, dramatically reducing write amplification.

1. Key Features:

Pure Go. No CGO, no C library dependency — deploys as a single Go binary.
WiscKey Value Separation. Keys + small values in the LSM-tree; large values in a separate value-log. Compaction touches only keys, slashing write amplification.
Concurrent Writers. Unlike LevelDB’s single-writer model, BadgerDB supports concurrent transactional writes via SSI (serializable snapshot isolation).
Transactions. First-class ACID transactions with snapshot isolation.
Streaming Backups. Backup / Load APIs for hot-incremental backup.
TTL. Per-key time-to-live with automatic expiry during compaction.
Encryption at Rest. AES-128/256 encryption of SSTables and value logs.

2. BadgerDB vs. LevelDB / RocksDB:

Language. BadgerDB — pure Go. LevelDB / RocksDB — C++ with bindings.
Concurrent writes. BadgerDB — yes. LevelDB — no. RocksDB — via TransactionDB.
Write amplification. BadgerDB — lower (WiscKey). RocksDB — tunable per workload.
Ecosystem. RocksDB has wider production deployment; BadgerDB is the go-to for Go-native embedding.

3. Use Cases:

Storage layer for Dgraph — the canonical user.
Go applications wanting an embedded KV without CGO build complexity.
Single-binary services that ship persistence inside the executable.
Workloads with large values (> 1 KiB) where WiscKey’s value separation pays off.

4. Architecture: LSM tree with key-value separation

BadgerDB is an embeddable key-value store written in Go by Dgraph Labs, and it is the storage engine underneath the Dgraph graph database. It follows the log-structured merge (LSM) tree design used by LevelDB and RocksDB, with one important change taken from the WiscKey paper: values above a configurable threshold are written to a separate append-only value log, and the LSM tree stores only the key and a small pointer into that log. Small values are still stored inline in the tree.

Because the LSM tree holds keys and pointers rather than full values, it is much smaller, so more of it fits in memory and compaction rewrites far less data. This reduces write amplification, which matters on SSDs. The cost is that a read of a large value needs one extra random read into the value log, and deleted or overwritten values must be reclaimed by a separate value-log garbage collection.

5. Read path

A read checks the in-memory tables first, then each level of SSTables from newest to oldest, using bloom filters and block indexes to skip tables that cannot contain the key. When the entry found is a value pointer, Badger performs a single read from the value log. Iteration over key ranges can be configured to fetch only keys (PrefetchValues = false), which is very fast because it touches only the compact LSM tree.

6. Worked example: Go

package main

import (
    "fmt"
    "log"

    badger "github.com/dgraph-io/badger/v4"
)

func main() {
    db, err := badger.Open(badger.DefaultOptions("/tmp/badger"))
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    // read-write transaction
    err = db.Update(func(txn *badger.Txn) error {
        e := badger.NewEntry([]byte("user:42"), []byte(`{"name":"Ana"}`))
        return txn.SetEntry(e)
    })

    // read-only transaction
    err = db.View(func(txn *badger.Txn) error {
        item, err := txn.Get([]byte("user:42"))
        if err != nil {
            return err
        }
        return item.Value(func(v []byte) error {
            fmt.Println(string(v))
            return nil
        })
    })

    // prefix scan over keys only
    db.View(func(txn *badger.Txn) error {
        opts := badger.DefaultIteratorOptions
        opts.PrefetchValues = false
        it := txn.NewIterator(opts)
        defer it.Close()
        for it.Seek([]byte("user:")); it.ValidForPrefix([]byte("user:")); it.Next() {
            fmt.Println(string(it.Item().Key()))
        }
        return nil
    })

    // reclaim space in the value log (call periodically)
    _ = db.RunValueLogGC(0.5)
}

Transactions are serializable snapshot isolation (SSI): Update retries are up to the caller when a conflict returns badger.ErrConflict. Entries can carry a TTL with WithTTL, which is handy for caches and sessions.

7. When to use it

Choose BadgerDB when you are writing a Go service that needs a fast, persistent, embedded store with transactions and you want to avoid cgo (RocksDB bindings require it). Prefer bbolt or LMDB when the workload is read-heavy with modest writes and you value a single-file B+ tree with simple operational behaviour. Prefer RocksDB when you need its very large tuning surface, column families, or bindings for many languages. Prefer Redis or Memcached when data must be shared over the network by many processes rather than embedded in one.

8. Pitfalls

Value-log GC is manual. Space from overwritten or deleted large values is only reclaimed by calling RunValueLogGC, typically on a ticker; forgetting it makes disk usage grow indefinitely.
Single process. A directory can be opened by one process at a time (a lock file enforces this); it is not a server.
Memory tuning. Default table and block caches are sized for servers; on small devices reduce memtable size, number of memtables and cache sizes.
Long transactions. Read-write transactions hold pending writes in memory and fail with ErrTxnTooBig if too large; batch with WriteBatch for bulk loads.