NoSQL Databases

Why NoSQL appeared

NoSQL ("not only SQL") is a collective name for databases that do not use the relational model of tables joined by keys, and usually not SQL as their main language. The name groups very different products; what they share is what they are not.

They became popular in the late 2000s, when large web companies hit limits of a single relational server:

• Scale-out. A relational database traditionally grows by buying a bigger server (vertical scaling). Systems with hundreds of millions of users needed to spread data and load over many ordinary servers (horizontal scaling), and joins and transactions across servers are hard to make fast.
• Flexible schemas. Product catalogs, user profiles and event data change shape often, and different records have different fields. Changing a table definition for every new field felt slow.
• Specific access patterns. Some workloads ask one kind of question over and over: "get the session with this key", "store this sensor reading", "who are the friends of friends of this person?". A database built for exactly that question can be much faster than a general-purpose one.

The managed cloud services for each family are listed in Managed Databases; this page explains how the families work and what they trade away.

The four families

Key-value stores

A key-value store keeps each value under a unique key and does essentially two things: store a value for a key, and return the value for a key. The database does not look inside the value. It works like a hash table or a Python dictionary, shared over the network, and lookups take about the same time however many keys there are.

Redis (and compatible forks such as Valkey) is the best-known example. It keeps data in memory, optionally saving it to disk, and is used for caching, sessions, shopping carts, rate limits and counters. Amazon DynamoDB and Azure Table Storage are key-value services at cloud scale. A session in Redis (not run):

SET session:8f3a "customer=3;cart=1,8" EX 1800   # expires after 30 minutes
GET session:8f3a
INCR page_views:product:5

Use a key-value store when you always know the key. Avoid it when you need to search by the contents of the values.

Document databases

A document database stores each record as a self-contained document, usually JSON (JavaScript Object Notation) or a binary form of it. A document can contain nested objects and lists, and documents in the same collection may have different fields. Unlike a key-value store, the database understands the structure: it can index and query fields inside the documents. MongoDB, Azure Cosmos DB for NoSQL, Google Cloud Firestore and Couchbase are document databases.

Order 104 from the sample shop spans four tables (orders, customers, order_items, products). As a document it is one record, with the items embedded (not run):

{
  "_id": 104,
  "order_date": "2026-07-01",
  "status": "shipped",
  "customer": {
    "customer_id": 3,
    "name": "Mei Chen",
    "city": "Amsterdam"
  },
  "items": [
    { "product_id": 8, "name": "Noise-cancelling headset", "quantity": 1, "unit_price": 129 },
    { "product_id": 1, "name": "USB-C cable 2 m", "quantity": 3, "unit_price": 9 }
  ]
}

Reading the whole order is now one lookup, with no joins. The price is duplication: Mei's name and city are copied into every order she places (a deliberate denormalization), and changing them everywhere is the application's job. Document databases suit data that is read and written as a unit: product catalogs, user profiles, content management, orders, mobile app data.

Wide-column stores

A wide-column store organizes data in rows grouped into partitions by a partition key; within a partition, rows are kept sorted by further key columns, and rows do not all need the same columns. Apache Cassandra is the best-known example; Apache HBase, ScyllaDB and Google Cloud Bigtable belong to the same family. They are built for very high write volumes spread over many servers: sensor readings, event logs, messages, time series.

You design a table per query. In Cassandra's CQL (Cassandra Query Language, SQL-like but without joins), a table that answers "the orders of one customer, newest first" looks like this (not run):

CREATE TABLE orders_by_customer (
  customer_id int,
  order_date  date,
  order_id    int,
  status      text,
  total       int,
  PRIMARY KEY ((customer_id), order_date, order_id)
) WITH CLUSTERING ORDER BY (order_date DESC, order_id DESC);

customer_id is the partition key: all orders of one customer live together on the same servers, so the query reads one partition. A question this table was not designed for, such as "all orders on 1 July", would need a second table holding the same data under a different key.

Graph databases

A graph database stores nodes (things, such as customers and products) and relationships between them (ORDERED, CONTAINS, FOLLOWS) as first-class data, both with properties. Following a relationship is a direct step from node to node instead of a join, so questions that hop across many relationships stay fast: recommendations, social networks, fraud rings, network topologies, access rights. Neo4j is the best-known example; Amazon Neptune and Azure Cosmos DB for Apache Gremlin are cloud options.

Neo4j's query language, Cypher, draws the pattern with arrows. The products Mei has ever ordered (not run):

MATCH (c:Customer {name: 'Mei Chen'})-[:ORDERED]->(:Order)-[:CONTAINS]->(p:Product)
RETURN DISTINCT p.name
ORDER BY p.name

For the shop's simple structure SQL works just as well. The difference shows with deep or variable-length paths such as "people within three steps of this person", which in SQL become recursive queries over large tables.

What NoSQL trades away

NoSQL databases are not "relational databases without the annoying parts". They get their speed and scale by giving up things a relational database gives you for free:

• Joins and ad hoc queries. Most NoSQL stores cannot join, or only in a limited way. You decide in advance which questions the data must answer and shape it for them; a new question may mean restructuring or copying the data.
• Transactions. Updates to one key or one document are atomic, but transactions over many records are limited or slower. Several products added them over time (MongoDB has supported multi-document ACID transactions since version 4.0, on sharded clusters since 4.2), yet they are not the normal way of working.
• Constraints. Foreign keys, UNIQUE and CHECK constraints are mostly absent; keeping the data valid becomes the application's job.
• Immediate consistency. Many distributed NoSQL databases return possibly outdated data by default, in exchange for speed and availability (next section).

Schema-on-write and schema-on-read

A relational database works schema-on-write: the table definition is checked when data is written, so a row with a missing required column or a text in a number column is refused. Many NoSQL databases work schema-on-read: they accept any document, and the code that reads it decides what the fields mean and what to do when one is missing. "Schema-less" is a misleading word: the schema still exists, but it lives in the application code, spread over every program that reads the data. Most document databases can enforce a schema when you ask them to (MongoDB with JSON Schema validation, for example).

Consistency and the CAP theorem

Distributed databases keep copies of each piece of data on several servers (replicas). When a write arrives, the replicas cannot all change at the same instant. A system with eventual consistency accepts the write on one or some replicas, copies it to the others in the background, and promises only that, if no new writes arrive, all replicas will eventually return the same value. Until then a read may return an older value. For a "like" counter or a product review that is fine; for an account balance it is not.

The CAP theorem

The CAP theorem (proposed by Eric Brewer in 2000, proven by Seth Gilbert and Nancy Lynch in 2002) is about three properties of a distributed data store:

• Consistency: every read returns the most recent write, as if there were a single copy. (This is not the C of ACID, which is about constraints.)
• Availability: every request to a working node gets a non-error answer.
• Partition tolerance: the system keeps working when the network between nodes loses messages, splitting them into groups that cannot reach each other.

Network partitions happen in any real distributed system, so they are not optional. The theorem says that during a partition the system must choose: either refuse some requests to stay consistent (a CP choice), or answer every request, accepting that some answers are out of date (an AP choice).

Two common misreadings to avoid:

• "Pick any two of three." When the network is healthy, a system can be both consistent and available. The choice only applies while a partition lasts, and a "CA" system is simply one that is not distributed (or stops working when partitioned).
• "NoSQL is AP, SQL is CP." The choice belongs to the configuration, not to the label. MongoDB with default settings reads and writes through one primary and behaves CP-like; Cassandra lets you choose per query how many replicas must answer (ONE, QUORUM, ALL).

In normal operation the real trade-off is between consistency and latency: waiting for more replicas to confirm makes every request slower. The PACELC model (Daniel Abadi) adds exactly that: if Partitioned, choose Availability or Consistency; Else, choose Latency or Consistency. How to make this choice per piece of data, and the range of consistency models between "linearizable" and "eventual", is on Data in Distributed Systems.

BASE

Where relational databases promise ACID, many distributed NoSQL systems are described as BASE: Basically Available (the system answers even when parts fail), Soft state (data may change without new input while replicas converge), Eventually consistent. It is a design philosophy rather than a precise guarantee: many products let you choose stronger consistency for the operations that need it.

Sharding

Sharding splits one data set over several servers (shards), each holding a part of the rows, so that storage and load grow by adding servers. It is also called horizontal partitioning. (Partitioning a table into parts on one server, which relational databases support for managing large tables, is related but does not add servers.) Each row goes to a shard based on its shard key (also called the partition key):

• Hash sharding: a hash of the key decides the shard. Spreads data evenly, but a range query ("orders from July") has to ask every shard.
• Range sharding: each shard holds a range of keys (customers A–F, G–M, ...). Range queries stay on one shard, but new keys can all land on the same shard, such as when the key is a date.

Choosing the shard key is the most important design decision in a sharded database, and hard to change later. A good key has many distinct values, spreads reads and writes evenly, and matches the most common query, so that query reads one shard. A bad key creates a hot partition: one shard that gets most of the traffic (all orders of today, one famous user) while the others sit idle. For the shop's orders, customer_id is a reasonable key; order_date or status would be poor ones.

Relational databases can be sharded too, by the application or with extensions such as Citus for PostgreSQL, but joins and transactions across shards are then expensive. Many NoSQL databases shard automatically, which is a large part of their appeal.

The same order three ways in Python

This standard-library example puts the difference side by side: a relational query that joins four tables, a key-value lookup in a dictionary, and an embedded document that holds a whole order. The last lines show the price of embedding: a question across documents means reading them all.

import sqlite3

# Relational: separate tables, joined at query time
db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE customers (customer_id INTEGER PRIMARY KEY, name TEXT);
CREATE TABLE products (product_id INTEGER PRIMARY KEY, name TEXT, price INTEGER);
CREATE TABLE orders (order_id INTEGER PRIMARY KEY, customer_id INTEGER, status TEXT);
CREATE TABLE order_items (order_id INTEGER, product_id INTEGER, quantity INTEGER);
INSERT INTO customers VALUES (3, 'Mei Chen');
INSERT INTO products VALUES (1, 'USB-C cable 2 m', 9), (8, 'Noise-cancelling headset', 129);
INSERT INTO orders VALUES (104, 3, 'shipped');
INSERT INTO order_items VALUES (104, 8, 1), (104, 1, 3);
""")
rows = db.execute("""
    SELECT c.name, p.name, oi.quantity * p.price
    FROM orders o
    JOIN customers c    ON c.customer_id = o.customer_id
    JOIN order_items oi ON oi.order_id = o.order_id
    JOIN products p     ON p.product_id = oi.product_id
    WHERE o.order_id = ?
    ORDER BY p.name""", (104,)).fetchall()
print("relational:", rows)

# Key-value: one opaque value per key, found in one step
sessions = {"session:8f3a": "customer=3;cart=1,8"}
print("key-value: ", sessions["session:8f3a"])
print("no such key:", sessions.get("session:0000"))

# Document: the whole order in one nested structure, no joins needed
orders = {
    104: {
        "status": "shipped",
        "customer": {"id": 3, "name": "Mei Chen"},
        "items": [
            {"name": "Noise-cancelling headset", "quantity": 1, "unit_price": 129},
            {"name": "USB-C cable 2 m", "quantity": 3, "unit_price": 9},
        ],
    }
}
order = orders[104]
total = sum(item["quantity"] * item["unit_price"] for item in order["items"])
print("document:  ", order["customer"]["name"], len(order["items"]), "items, total", total)

# The price of embedding: which orders contain a USB-C cable?
# Relational: one query on order_items. Documents: read every order.
hits = [order_id for order_id, doc in orders.items()
        if any(item["name"] == "USB-C cable 2 m" for item in doc["items"])]
print("scan:      ", hits)

relational: [('Mei Chen', 'Noise-cancelling headset', 129), ('Mei Chen', 'USB-C cable 2 m', 27)]
key-value:  customer=3;cart=1,8
no such key: None
document:   Mei Chen 2 items, total 156
scan:       [104]

JSON in relational databases

The line between the two worlds has blurred. Relational databases now store and query JSON: PostgreSQL has the jsonb type with indexes, MySQL a JSON type, SQL Server functions such as JSON_VALUE and OPENJSON, and SQLite built-in JSON functions. So a relational database can keep a flexible part (a product's varying specifications, an order's gift options) next to strictly defined columns, in one transaction.

In SQLite, json_extract(doc, path) reads a value from a JSON text by a path such as '$.customer.name' ($ is the document, . steps into an object, [0] into a list); the ->> operator is a shorter form. json_each turns a JSON list into rows, which can then be joined and aggregated like any table:

CREATE TABLE order_docs (
  order_id INTEGER PRIMARY KEY,
  doc      TEXT NOT NULL CHECK (json_valid(doc))
);

INSERT INTO order_docs VALUES
  (104, '{"status": "shipped", "customer": {"id": 3, "name": "Mei Chen"},
          "items": [{"product": "Noise-cancelling headset", "quantity": 1, "unit_price": 129},
                    {"product": "USB-C cable 2 m", "quantity": 3, "unit_price": 9}]}'),
  (107, '{"status": "shipped", "customer": {"id": 2, "name": "Kwame Mensah"},
          "items": [{"product": "Laptop stand", "quantity": 1, "unit_price": 35}],
          "gift_note": "Happy birthday!"}');

SELECT order_id,
       json_extract(doc, '$.customer.name')  AS customer,
       json_array_length(doc, '$.items')     AS items,
       doc ->> '$.gift_note'                 AS gift_note
FROM order_docs
ORDER BY order_id;

SELECT o.order_id,
       json_extract(i.value, '$.product') AS product,
       json_extract(i.value, '$.quantity') * json_extract(i.value, '$.unit_price') AS amount
FROM order_docs AS o, json_each(o.doc, '$.items') AS i
ORDER BY o.order_id, product;

order_id  customer      items  gift_note
--------  ------------  -----  ---------------
104       Mei Chen      2      NULL
107       Kwame Mensah  1      Happy birthday!

order_id  product                   amount
--------  ------------------------  ------
104       Noise-cancelling headset  129
104       USB-C cable 2 m           27
107       Laptop stand              35

Order 104 has no gift_note, so the path finds nothing and the result is NULL: schema-on-read inside a relational database. The CHECK (json_valid(doc)) constraint at least refuses text that is not valid JSON.

Choosing: SQL, NoSQL or both

Large systems rarely use only one kind of database. Using several, each for what it does best, is called polyglot persistence. A web shop might keep customers, orders and payments in a relational database, sessions and a cache in Redis, the product search index in a search engine such as OpenSearch or Elasticsearch, and "customers who bought this also bought" in a graph. Each extra system has to be run, secured, backed up and kept in sync, so add one only when it solves a real problem.

When in doubt, start relational. It answers the widest range of questions, keeps data valid, and scales further than most applications ever need; move a specific workload to a NoSQL store when you can name the limit you hit.