Wide-Column Stores

Partitioned, column-family databases built for write-heavy scale: Cassandra, ScyllaDB and HBase.

Apache Cassandra

Apache Cassandra is the canonical wide-column / Dynamo-style distributed database. Originally developed at Facebook for Inbox Search, open-sourced in 2008…

Apache HBase

Apache HBase is an open-source clone of Google’s Bigtable, built on top of HDFS as the storage layer. Started in 2008 as part of the Hadoop ecosystem, HBase…

ScyllaDB

ScyllaDB is a wire-compatible, drop-in-compatible rewrite of Apache Cassandra in C++ on the Seastar shared-nothing async framework. Founded in 2015 by KVM…

1. Overview

Wide-column stores organise data into tables whose rows are grouped into partitions by a partition key and sorted inside each partition by clustering columns. Each row may hold a different set of columns, so very sparse data costs nothing for missing values. The design traces back to Google Bigtable (column families on a distributed file system) and Amazon Dynamo (consistent hashing and tunable replication). The result is a database family that scales writes almost linearly by adding nodes, at the price of designing tables around the queries you intend to run rather than around normalised entities.

2. Data model and write path

Wide-Column Data Model Keyspace / tablenamespace Partition keyhash -> node Clustering colssort within partition Rowsparse columns Memtablewrites in RAM SSTablesimmutable files A partition is the unit of distribution and of efficient reads; queries should name the full partition key. Writes land in a commit log and memtable, then flush to sorted SSTables that compaction merges later.

All three systems use a log-structured merge (LSM) storage engine. A write is appended to a commit log for durability and inserted into an in-memory memtable; it is acknowledged without any read-before-write. When the memtable fills, it is flushed as an immutable sorted file (an SSTable in Cassandra and ScyllaDB, an HFile in HBase). Background compaction merges files and drops overwritten or deleted values. Reads consult the memtable and the relevant files, helped by Bloom filters and partition indexes. This is why writes are cheap and reads of a single partition are fast, while scans across partitions are expensive.

3. Cassandra vs ScyllaDB vs HBase

CassandraScyllaDBHBase
ArchitectureMasterless ring, JavaMasterless ring, C++ shard-per-coreMaster + RegionServers on HDFS
ConsistencyTunable per query (ONE, QUORUM, ALL)Tunable, Cassandra-compatibleStrong per row
Query APICQLCQL and DynamoDB-compatible APIJava client, Get/Put/Scan
StrengthMulti-datacenter replication, mature ecosystemLow tail latency, fewer nodes for same loadHadoop integration, consistent range scans
Watch out forGC pauses, tombstonesSmaller ecosystemZooKeeper/HDFS operational weight

4. When to choose a wide-column store

Choose one when the workload is write-heavy, the access pattern is known in advance, and data volume or availability requirements exceed what a single relational primary can handle: time series, event logs, messaging inboxes, IoT telemetry, user activity feeds. Avoid it when you need ad-hoc joins, multi-row transactions or flexible analytics; a relational database or a lakehouse engine is a better fit there.

Apache Cassandrawide columnthe Dynamo-style reference implementation with CQL and tunable consistency. ScyllaDBwide columna C++ rewrite of Cassandra on the Seastar shard-per-core framework. Apache HBasewide columnthe Bigtable clone on HDFS with strongly consistent rows.

5. Common pitfalls

See also key-value stores.