Replication

Replication keeps copies of the same data on several machines so a system survives failures, serves more reads and sits closer to users. This page explains the three replication topologies, quorum reads and writes, replication lag, failover and conflict resolution, with a runnable quorum simulation.


1. What Replication Is

Three Replication Topologies Leader-Follower Multi-Leader Leaderless Leader reads + writes Follower 1 reads Follower 2 reads one writer, async or sync log Leader A DC east Leader B DC west Follower Follower writes in both, must resolve conflicts Client writes to W of N R1 R2 R3 quorum W + R > N

Replication means keeping a copy of the same data on several machines that are connected by a network. It is the most basic tool for building data systems that survive failures, and nearly every database, message log and object store you will use relies on it. The three diagrams above show the three ways replicas can be arranged; the rest of this page explains how each one works and what it costs.

A replica is a node that stores a full copy of a dataset (or of one partition of it). The hard part is not copying data once, it is keeping every copy consistent while writes keep arriving, the network delays or drops messages, and nodes crash and come back.

2. Why It Matters

There are three reasons to replicate, and they pull in slightly different directions:

The cost is coordination. Every design below picks a point on a trade-off between how fresh a read is guaranteed to be, how long a write takes, and what happens when the network partitions. The CAP and PACELC page formalises that trade-off and Consistency Models names the guarantees.

3. How It Works

Leader-follower (single leader)

One node, the leader, accepts all writes. It appends each change to a replication log (PostgreSQL's write-ahead log, MySQL's binlog) and streams the log to followers, which apply the changes in the same order. Clients may read from any node. Because there is a single writer, there are no write conflicts: the log defines one total order.

Replication can be synchronous (the leader waits for a follower to acknowledge before confirming the write) or asynchronous (the leader confirms immediately). Synchronous replication guarantees the follower is up to date but makes every write wait on the slowest synchronous follower; asynchronous replication is fast but can lose recently confirmed writes if the leader dies before shipping them. Most systems use a mix: one synchronous follower and the rest asynchronous ("semi-synchronous").

Failover is the dangerous moment. The system must detect that the leader is dead (usually with a timeout), choose a new leader (often the most up-to-date follower, ideally via a consensus protocol such as Raft), and redirect clients. Too short a timeout causes needless failovers; two nodes both believing they are leader ("split brain") can corrupt data, which is why fencing tokens and consensus are used.

Multi-leader

Several nodes accept writes, typically one leader per data centre, and each forwards its changes to the others. Writes are fast locally and each region keeps working if the inter-region link fails. The price is write conflicts: two leaders can modify the same row concurrently. Conflicts must be resolved by last-write-wins (simple, silently drops data), by merging values, by application-specific logic, or by data types designed to merge automatically (CRDTs).

Leaderless

Popularised by Amazon's Dynamo paper and used by Cassandra and Riak, leaderless replication lets the client (or a coordinator node acting for it) send each write to all N replicas and treat it as successful once W acknowledge. Reads query R replicas and take the value with the newest version.

Quorum Overlap: N = 3, W = 2, R = 2 Replica 1 v2 Replica 2 v2 Replica 3 v1 (stale) write set (W = 2) read set (R = 2) Replica 2 is in both sets, so the read sees v2 and returns the newest version. If W + R <= N the sets can miss each other and a read may return stale data.

The quorum condition is W + R > N: every read set must overlap every write set in at least one node, so at least one replica in the read has the latest write. Common settings for N = 3 are W = 2, R = 2. Stale replicas are repaired by read repair (the coordinator writes the newest value back to stale nodes it read from) and by background anti-entropy that compares data using Merkle trees. Quorums are not linearizable on their own: concurrent writes, sloppy quorums and partial failures can still produce anomalies.

4. Worked Example: Quorum Simulation

The script below simulates leaderless replication with N = 3. Each replica stores (version, value); a write succeeds when W replicas accept it, and a read returns the highest version among R replicas and repairs any stale ones. Change W and R to watch stale reads appear when W + R <= N.

import random

N, W, R = 3, 2, 2
replicas = [{"version": 0, "value": None} for _ in range(N)]

def write(value, version, down=()):
    acks = 0
    for i in random.sample(range(N), N):          # contact replicas in random order
        if i in down:
            continue
        replicas[i] = {"version": version, "value": value}
        acks += 1
        if acks == W:                                 # stop once quorum reached
            return True
    return False

def read(down=()):
    alive = [i for i in random.sample(range(N), N) if i not in down][:R]
    if len(alive) < R:
        raise RuntimeError("read quorum not available")
    newest = max((replicas[i] for i in alive), key=lambda r: r["version"])
    for i in alive:                                   # read repair
        if replicas[i]["version"] < newest["version"]:
            replicas[i] = dict(newest)
    return newest["value"]

random.seed(7)
stale = 0
for v in range(1, 1001):
    write(f"v{v}", v)
    if read() != f"v{v}":
        stale += 1
print(f"W={W} R={R} N={N}: stale reads = {stale} / 1000")
# With W=2,R=2 this prints 0. Try W=1,R=1 and stale reads appear.

The next diagram shows the most common anomaly in leader-follower systems with asynchronous replication: a user writes, then immediately reads from a follower that has not yet applied the change.

Replication Lag: Read-Your-Writes Violation Client Leader Follower 1. write x=5 3. log shipped (lag) 2. read x returns old x=3 applies x=5 too late Fix: route a user's reads to the leader for a while after they write, or track log positions.

Three guarantees address lag: read-your-writes (a user always sees their own updates), monotonic reads (a user never sees time go backwards, typically by pinning them to one replica), and consistent prefix reads (causally related writes are seen in order).

5. Comparison of Approaches

ApproachWrite pathConflictsFailure behaviourExamples
Single leader, asyncLeader only; fastNoneFailover may lose last writesPostgreSQL streaming, MySQL
Single leader, sync / consensusLeader + majority ackNoneNo acknowledged write lostetcd, CockroachDB ranges, Kafka with acks=all
Multi-leaderAny leader; local latencyYes, must resolveRegions independentMulti-region MySQL setups, CouchDB
Leaderless quorumAny W of N replicasConcurrent writes possibleTolerates N - W down for writesCassandra, Riak, DynamoDB-style

Rule of thumb: start with single-leader replication; move to consensus-based replication when you need automatic failover without losing acknowledged writes; reach for multi-leader or leaderless only when you need writes to stay available across regions and can accept conflict handling.

6. Pitfalls

8. Further Reading