Scalability and Caching Patterns

Scalability, the ability of a system to handle more load by adding resources, is a quality attribute you design for (see Quality Attributes and Trade-offs). The infrastructure side is explained elsewhere: vertical and horizontal scaling, autoscaling, load balancers, how much capacity you need in Capacity Planning and Performance, and HTTP caching in browsers and content delivery networks on HTTP Caching. This page is about the application's side: what makes a design scale, and how to cache and split data inside your own system. The examples follow Kestrel Energy's customer portal, its meter-api service and the PostgreSQL server db01.

What it takes to scale out

Scaling up (a bigger server) needs no design change but has a ceiling and keeps a single point of failure. Scaling out (more servers) has no hard ceiling, but only works if the application allows it. Three conditions:

• Stateless instances. Any instance must be able to serve any request, so sessions, uploads and work in progress live in backing services (see stateless services).
• Nothing shared that does not scale. Instances that share a lock, a file on one disk or a counter in one table row queue up behind it, however many instances you add.
• Data that can be divided. Eventually the database is the bottleneck. Read load can move to read replicas and caches; write load can only grow by partitioning the data.

Kestrel's portal runs on two cloud virtual machines, web01 and web02, behind a load balancer; adding web03 is a configuration change. Every one of them, plus meter-api and the billing system, queries db01. Doubling the web servers doubles the number of database connections and queries, so the honest answer to "can the portal scale?" is "as far as db01 can". The rest of this page is mostly about protecting that database.

Where to cache, and the hit ratio

A cache keeps a copy of data closer to where it is used, so that repeated requests do not pay the full cost again. Caches exist at every level: the browser and a content delivery network (HTTP caching), a reverse proxy, the application's own memory, a shared cache server such as Redis or Memcached that all instances use, and the database's buffer in memory. An in-process cache is the fastest but each instance has its own copy; a shared cache server costs a network round trip but every instance sees the same entries.

The number that decides whether a cache helps is the cache hit ratio: the share of lookups that the cache can answer from its stored entries, hits divided by hits plus misses. A miss still pays the full cost, plus the cache lookup. The portal's usage graph at the evening peak:

HIT_MS, MISS_MS = 0.5, 40.0      # cache lookup vs. usage query on db01
REQUESTS = 300                   # usage-graph requests per second at the evening peak

print("hit ratio  avg latency  queries/s on db01")
for hit_ratio in (0.0, 0.50, 0.90, 0.95, 0.99):
    avg = hit_ratio * HIT_MS + (1 - hit_ratio) * (HIT_MS + MISS_MS)   # a miss checks the cache first
    print(f"{hit_ratio:8.0%}  {avg:8.1f} ms  {REQUESTS * (1 - hit_ratio):12.0f}")

hit ratio  avg latency  queries/s on db01
      0%      40.5 ms           300
     50%      20.5 ms           150
     90%       4.5 ms            30
     95%       2.5 ms            15
     99%       0.9 ms             3

The database load falls with the miss ratio: going from 90% to 99% hits looks like a small step but cuts the queries on db01 by a factor of ten. A low hit ratio, typical for data that is read once or keyed too finely (per request, per second), means the cache only adds latency and memory. Measure it per cache and alert when it drops: a sudden fall after a release often means a key format changed.

Cache-aside

Cache-aside, also called lazy loading, is a caching pattern in which the application itself looks for data in the cache, and on a miss reads it from the data store and puts a copy in the cache for the next request; the cache knows nothing about the database. The Azure Architecture Center describes it as "Load data on demand into a cache from a data store." It is the most common pattern because it works with any cache and any database, and only data that is actually requested ends up in the cache.

The portal's usage graph is a good candidate: it is requested often, is expensive to compute, and only changes when a new reading arrives, every 15 minutes. The cache key is the customer and the month, and each entry gets a time to live (TTL) of 15 minutes: after that it counts as expired and the next request reloads it. (The same words mean a hop counter in an IP packet; see time to live.) With a fake clock so the example is repeatable:

TTL = 15 * 60            # seconds: a new reading arrives every 15 minutes

class Clock:             # a fake clock keeps the example repeatable
    now = 0

class UsageCache:
    def __init__(self, load):
        self.load, self.entries = load, {}
        self.hits = self.misses = 0

    def get(self, key):
        entry = self.entries.get(key)
        if entry and Clock.now < entry[1]:
            self.hits += 1
            return entry[0]
        self.misses += 1                       # 1. miss
        value = self.load(key)                 # 2. read from the database
        self.entries[key] = (value, Clock.now + TTL)   # 3. store with an expiry time
        return value

db_queries = []
def load_usage(key):
    db_queries.append(key)
    customer, month = key
    return f"{customer} {month}: 412 kWh"

cache = UsageCache(load_usage)
# (seconds since 18:00, customer) for one evening of portal visits
visits = [(0, "K-1042"), (40, "K-1042"), (65, "K-2077"), (300, "K-1042"),
          (610, "K-2077"), (950, "K-1042"), (980, "K-1042"), (1500, "K-2077")]
for Clock.now, customer in visits:
    cache.get((customer, "2026-10"))

total = cache.hits + cache.misses
print(f"{total} requests, {cache.hits} hits, {cache.misses} misses, "
      f"hit ratio {cache.hits / total:.0%}, {len(db_queries)} database queries")

8 requests, 4 hits, 4 misses, hit ratio 50%, 4 database queries

For a single function in one process, Python's standard library already has a cache-aside implementation: functools.lru_cache(maxsize=…) keeps the most recently used results and reports hits and misses with cache_info(). It has no expiry time, so use it for data that does not change while the process runs, such as reference tables loaded at startup.

Read-through, write-through and write-behind

In cache-aside, the application does all the work. Some caching products can do part of it themselves. The Azure Architecture Center notes: "Many commercial caching systems provide read-through and write-through or write-behind operations", and that an application can emulate read-through by implementing cache-aside.

Read-through

A read-through cache loads missing data from the underlying data store itself: the application only ever asks the cache, and on a miss the cache calls a loader that it was configured with. The effect is the same as cache-aside, but the loading logic lives in one place instead of in every piece of code that reads.

Write-through

A write-through cache writes every change to the data store as part of the write operation, and updates its own copy at the same time, so that the cache never holds data the database does not have. AWS's caching guidance contrasts the two basic approaches: "A cache-aside cache is updated after the data is requested. A write-through cache is updated immediately when the primary database is updated." The cost is that every write pays for both, and data that is written but never read still fills the cache.

Write-behind

A write-behind cache (also called write-back) confirms a write as soon as the cache has it, and writes the changes to the data store later, usually in batches, after a delay. Writes become very fast and the database gets fewer, larger writes. But until the flush, the only copy of the change is in the cache, so a crash loses it. Ten portal logins update a customer's "last seen" time, and then the cache node fails:

class Store:
    def __init__(self, mode, flush_every=4):
        self.mode, self.flush_every = mode, flush_every
        self.cache, self.database, self.dirty = {}, {}, []
        self.db_writes = 0

    def write(self, key, value):
        self.cache[key] = value
        if self.mode == "write-through":          # database first, then confirm
            self.database[key] = value
            self.db_writes += 1
        else:                                      # write-behind: confirm now, flush later
            self.dirty.append(key)
            if len(self.dirty) == self.flush_every:
                self.flush()

    def flush(self):
        for key in self.dirty:
            self.database[key] = self.cache[key]
        self.db_writes += 1                        # one batched write
        self.dirty = []

    def crash(self):                               # the cache node dies
        self.cache, lost, self.dirty = {}, len(self.dirty), []
        return lost

# "last seen" timestamps for 10 portal logins, then the cache node crashes
for mode in ("write-through", "write-behind"):
    store = Store(mode)
    for n in range(10):
        store.write(f"K-{1040 + n}", f"2026-10-06T19:{n:02}")
    lost = store.crash()
    print(f"{mode:13}  database writes: {store.db_writes:2}  "
          f"in database: {len(store.database):2}  lost: {lost}")

write-through  database writes: 10  in database: 10  lost: 0
write-behind   database writes:  2  in database:  8  lost: 2

Five times fewer database writes, and two customers whose "last seen" time is slightly wrong: an acceptable trade for that data. For an invoice, a payment or a meter reading, a lost write means lost money or a wrong bill, so those are written to the database first. The choice in one table:

Cache invalidation

A saying attributed to Phil Karlton goes: "There are only two hard things in Computer Science: cache invalidation and naming things." Cache invalidation is removing or replacing cached entries when the original data changes, so that readers stop getting the old copy. It is hard because the cache and the data store are two copies without a shared transaction: every strategy accepts some window in which they differ. The common strategies, often combined:

• Expiry (TTL). Every entry expires after a fixed time. Simple and self-healing: even a missed invalidation is corrected within one TTL. The TTL is the maximum staleness you accept, so choose it from the data: 15 minutes for usage that changes every 15 minutes, seconds for a stock level.
• Delete on write. The code that changes the data deletes the cached entry right after the database commit; the next read reloads it. Delete rather than overwrite: two writers updating the cache in the wrong order can leave the older value behind.
• Versioned keys. Put a version in the key (tariffs:v42). A change creates a new version, and readers simply stop asking for the old key, which expires on its own.
• Events. The owner of the data publishes what changed, and every service that caches it deletes its entry. KBS publishes TariffChanged, and each portal instance clears its in-process tariff cache (see Event-Driven Architecture).

Kestrel's tariff table changes a few times a year, but a wrong price on the portal is a complaint and possibly a legal problem. So it is cached in every portal instance with event-based invalidation, plus a TTL of one hour as a safety net in case an event is lost.

Cache stampedes

A cache stampede (also called a thundering herd or dog-piling) happens when a popular cache entry expires or is cleared and many requests miss at the same moment, so they all run the same expensive query against the data store at once, which can overload it just when it should be protected. The tariff overview on the portal's home page is requested about every two milliseconds at peak, and rebuilding it takes 200 ms:

QUERY_MS = 200                       # rebuilding the tariff overview takes 200 ms
arrivals = range(0, 200, 2)          # 100 requests in the 200 ms after the entry expired

def naive():
    ready_at, queries = None, 0
    for t in arrivals:
        if ready_at is None or t < ready_at:   # still a miss: nobody has stored it yet
            queries += 1
            if ready_at is None:
                ready_at = t + QUERY_MS
    return queries

def single_flight():
    in_flight, queries, waited = None, 0, 0
    for t in arrivals:
        if in_flight is None:                  # first miss starts the only query
            in_flight, queries = t + QUERY_MS, 1
        elif t < in_flight:                    # later misses wait for that result
            waited += 1
    return queries, waited

print(f"naive cache-aside: {naive()} identical queries hit the database")
queries, waited = single_flight()
print(f"single-flight:     {queries} query, {waited} requests waited for its result")

naive cache-aside: 100 identical queries hit the database
single-flight:     1 query, 99 requests waited for its result

Three defenses, usually combined:

• Request coalescing (single-flight): the first miss for a key loads it, and concurrent misses for the same key wait for that result instead of querying themselves. Across several instances this needs a short lock in the shared cache.
• Jittered expiry. Entries written at the same moment expire at the same moment. Adding a random amount to each TTL, the same idea as jitter for retries, spreads the reloads. At Kestrel this matters twice: usage entries all filled after the readings at :00 would all expire at :15, exactly when meter-api is busy with the next burst. A TTL of 15 minutes plus or minus two spreads them out.
• Refresh before expiry. A background job, or the first request after a "soft" expiry time, reloads a hot entry while the old value is still served, so readers never all miss.

Partitioning and sharding

Caches and replicas take read load off a database. Write load and data volume can only grow by dividing the data: sharding spreads rows over several servers by a shard key, by hash or by range, and the databases page explains how to choose the key and avoid hot partitions. Partitioning one table into parts on a single server is the gentler step that comes first.

For Kestrel the numbers are concrete: 400,000 meters times 96 readings a day is 38.4 million rows a day. The likely first step is partitioning meter-api's readings table by month inside PostgreSQL, which keeps indexes small and lets old months be archived by dropping a partition. Sharding readings over several database servers by a hash of the meter's EAN code is the option after that. Sharding by postcode would be a poor key: a new housing estate puts thousands of new meters on the same shard.

Consistent hashing

Hash sharding usually means hash(key) % shards. That spreads keys evenly, but changing the number of shards changes the result for almost every key, so adding a shard means moving almost all of the data. Consistent hashing is a way of assigning keys to servers in which keys and servers are placed on the same circular range of hash values, and each key belongs to the next server point clockwise from it, so that adding or removing a server moves only the keys in that server's part of the circle.

Going from four to five shards, for 10,000 meter IDs, hashed with CRC-32 because Python's built-in hash() of a string changes every time a process starts:

import bisect
import zlib

def h(text):                          # stable hash; Python's hash() changes per run
    return zlib.crc32(text.encode())

meters = [f"8716874{n:011}" for n in range(10_000)]

def modulo_shard(key, shards):
    return h(key) % shards

class Ring:
    def __init__(self, shards, vnodes=100):
        self.points = sorted((h(f"shard{s}-{v}"), s)
                             for s in range(shards) for v in range(vnodes))
        self.hashes = [p for p, _ in self.points]

    def shard(self, key):             # first point clockwise from the key's hash
        i = bisect.bisect(self.hashes, h(key)) % len(self.points)
        return self.points[i][1]

def moved(before, after):
    return sum(before(k) != after(k) for k in meters) / len(meters)

ring4, ring5 = Ring(4), Ring(5)
print(f"modulo, 4 -> 5 shards:      "
      f"{moved(lambda k: modulo_shard(k, 4), lambda k: modulo_shard(k, 5)):.1%} of meters move")
print(f"consistent hashing, 4 -> 5: {moved(ring4.shard, ring5.shard):.1%} of meters move")
print("minimum possible:           20.0% (the new shard's fair share)")

modulo, 4 -> 5 shards:      80.2% of meters move
consistent hashing, 4 -> 5: 22.2% of meters move
minimum possible:           20.0% (the new shard's fair share)

With modulo, four out of five meters change shard, and all that data must be copied while the system runs. With the ring, little more than the unavoidable fifth moves. Distributed caches, key-value stores and some databases use consistent hashing or a variant of it internally; you mostly meet it when choosing or operating such a product, or when you shard by hand.