Partitioning splits data so that many machines can share the work. This page explains range and hash schemes, how systems rebalance when nodes join, how requests find the right shard, and how to deal with hot keys.
Partitioning splits one logical dataset into smaller pieces so that each piece can be stored, scanned or served independently. When the pieces live on different machines the technique is usually called sharding, and each piece a shard. The two words are often used interchangeably; databases such as PostgreSQL also use "partitioning" for splitting a table into pieces on a single server, which helps pruning and maintenance but not capacity.
Partitioning is distinct from replication, which copies the same data to several machines for availability and read throughput. Real systems use both: each partition is replicated, and the replicas of different partitions are spread across nodes.
A single machine has finite disk, memory, CPU and network. Once a dataset or write rate outgrows the largest affordable machine, the only way forward is to spread data across many. Done well, partitioning gives near-linear scaling: double the nodes, double the throughput. Done badly, it produces one overloaded node (a hot spot) while the rest sit idle, plus queries that must contact every node.
The same ideas appear in analytics: Hive-style directory partitions (dt=2026-10-01/), Kafka topic partitions, Spark shuffle partitions and BigQuery or Snowflake pruning all rely on routing work to a subset of the data.
The central design decision is the partition key: the attribute that decides where each record lives. It should have many distinct values, spread load evenly over time, and match the most common access path, so that the queries you run most often touch one partition. Changing the key later usually means rewriting the entire dataset, so it deserves more thought than almost any other schema choice.
Range partitioning assigns contiguous key ranges to partitions, like volumes of an encyclopedia. Range scans touch few partitions and data stays sorted, which is why HBase, Bigtable, CockroachDB and TiDB use it. The danger is skew: if keys are timestamps, every new write lands in the newest range.
Hash partitioning applies a hash function to the key and maps the result to a partition. A good hash spreads keys uniformly, so load is even, but adjacent keys scatter and range queries must ask every partition. Cassandra, DynamoDB and Kafka (for keyed messages) use hashing. A popular compromise is a compound key: hash the first part (say user_id) to choose the partition, and sort by the second part (timestamp) within it, so "one user's recent events" is still a single-partition range read.
Rebalancing happens when nodes are added or removed. Naive hash(key) mod N is a trap: changing N moves almost every key. Three better strategies:
Request routing then needs a map from partition to node. Either clients ask any node and get forwarded (Cassandra's gossip), a routing tier holds the map (MongoDB's mongos), or the map lives in a coordination service such as ZooKeeper or etcd.
Even a perfect hash cannot help when one key is hot: a celebrity account, a viral product, the NULL join key. Every request for that key lands on the same shard. Mitigations: append a random suffix ("salt") to hot keys to spread writes over several partitions and merge on read; cache hot reads in front of the store; or split the entity itself (for example, separate counters per shard summed periodically). In Spark the equivalent problem is a skewed join, handled by salting or by adaptive query execution's skew-join splitting.
The script compares modulo hashing with a consistent-hash ring when growing from 4 to 5 nodes, counting how many keys move.
import hashlib, bisect
def h(s): # stable 64-bit hash (Python's hash() is salted per process)
return int(hashlib.md5(s.encode()).hexdigest()[:16], 16)
keys = [f"user:{i}" for i in range(100_000)]
def mod_assign(n):
return {k: h(k) % n for k in keys}
class Ring:
def __init__(self, nodes, vnodes=100):
self.points = sorted((h(f"{n}#{v}"), n) for n in nodes for v in range(vnodes))
self.hashes = [p for p, _ in self.points]
def node(self, key):
i = bisect.bisect(self.hashes, h(key)) % len(self.points)
return self.points[i][1]
a, b = mod_assign(4), mod_assign(5)
print("mod N moved: ", sum(a[k] != b[k] for k in keys) / len(keys))
r4, r5 = Ring(["n1","n2","n3","n4"]), Ring(["n1","n2","n3","n4","n5"])
print("ring moved: ", sum(r4.node(k) != r5.node(k) for k in keys) / len(keys))Modulo hashing moves about 80% of keys (a key stays only if its hash gives the same remainder for 4 and 5). The ring moves close to 20%, the theoretical minimum of 1/5, because only the keys now owned by n5 relocate.
In SQL, PostgreSQL declarative partitioning shows the single-server form:
CREATE TABLE events (id bigint, ts timestamptz, payload jsonb)
PARTITION BY RANGE (ts);
CREATE TABLE events_2026_10 PARTITION OF events
FOR VALUES FROM ('2026-10-01') TO ('2026-11-01');
-- WHERE ts >= '2026-10-05' scans only events_2026_10 (partition pruning)| Strategy | Range queries | Load balance | Rebalancing cost | Examples |
|---|---|---|---|---|
| Range | Efficient | Skew-prone (sequential keys) | Split/merge ranges | HBase, Bigtable, CockroachDB |
| Hash mod N | Scatter-gather | Even | Very high (most keys move) | Simple app-level sharding |
| Fixed partitions + hash | Scatter-gather | Even | Move whole partitions | Kafka, Elasticsearch |
| Consistent hashing | Scatter-gather | Even with vnodes | About 1/N of keys | Cassandra, Dynamo-style stores |
| Compound (hash + sort) | Within one hash key | Even across hash keys | Depends on outer scheme | DynamoDB, Cassandra clustering keys |