Capacity Planning and Performance

Capacity planning answers two questions: how much load can the system handle while still meeting its service level objectives, and when will the load outgrow it? The database view, with storage growth and connection limits, is on Database Administration; adding capacity automatically is covered under autoscaling and the HorizontalPodAutoscaler. This page is about the measurements and the math behind both. The examples use Kestrel Energy's meter-api, which receives smart-meter readings, and its database server db01.

Utilization and saturation

Utilization is the share of time a resource is busy doing work, or, in a second common meaning, the share of its capacity in use: a CPU busy 70% of the time, a disk 70% full, 140 of 200 database connections open. Brendan Gregg's USE method, introduced on What Is Observability?, uses the first meaning and pairs it with saturation: the extra work the resource cannot handle yet, usually waiting in a queue.

The difference matters because utilization can look fine while users suffer. A resource that is 100% busy is not a problem in itself; a resource with a growing queue is. During the meter-api incident of 6 October 2026 (see Incident Response in Practice), each instance received about 150 readings per second but wrote only about 60 to the database. The test setup's Prometheus recorded it:

promtool query instant http://localhost:9090 'sum by (instance) (rate(meter_readings_received_total[5m]))'
promtool query instant http://localhost:9090 'sum by (instance) (rate(meter_readings_processed_total[5m]))'

{instance="meter-api-01:8080"} => 149.2736842105263 @[1791306953.621]
{instance="meter-api-02:8080"} => 149.0140350877193 @[1791306953.621]
{instance="meter-api-01:8080"} => 59.39999999999999 @[1791306953.686]
{instance="meter-api-02:8080"} => 59.59649122807017 @[1791306953.686]

When more arrives than leaves, the difference piles up. The queue depth is a gauge, and deriv() gives its slope per second:

promtool query instant http://localhost:9090 'meter_readings_queue_depth'
promtool query instant http://localhost:9090 'deriv(meter_readings_queue_depth[5m])'

meter_readings_queue_depth{env="production", instance="meter-api-01:8080", job="meter-api", team="metering"} => 33467 @[1791306931.747]
meter_readings_queue_depth{env="production", instance="meter-api-02:8080", job="meter-api", team="metering"} => 33778 @[1791306931.747]
{env="production", instance="meter-api-01:8080", job="meter-api", team="metering"} => 89.98651629072681 @[1791306931.876]
{env="production", instance="meter-api-02:8080", job="meter-api", team="metering"} => 88.89268170426065 @[1791306931.876]

About 33,500 readings were waiting on each instance, growing by about 90 per second, which is exactly the 150 − 60 that did not leave. Before the incident the same queue held between 5 and 40 readings. That is saturation, and it is the number to alert on, not the CPU graph.

Useful utilization and saturation pairs for common resources, as PromQL (the first two use the standard node_exporter metrics):

# CPU utilization per host: share of time not idle
1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

# CPU saturation: runnable tasks per CPU (load average divided by CPUs)
node_load1 / on (instance) count by (instance) (node_cpu_seconds_total{mode="idle"})

# Database connections in use, as a share of the configured maximum
pg_connections_in_use / pg_connections_max

# Readings queue: saturation as a share of the queue limit
meter_readings_queue_depth / meter_readings_queue_limit

The load average is explained on Logs and Monitoring in the Linux topic.

Headroom

Headroom is the capacity a system has left between its expected peak load and the load at which it stops meeting its objectives, after allowing for the loss of a component. It is the buffer that absorbs a traffic spike, a failed server or a month of growth while new capacity is ordered.

Two rules make headroom concrete. First, measure against the peak, not the average: Kestrel's portal is busiest on cold Monday mornings in winter, after a price change, when everyone checks their usage. Second, plan for one component to be missing, the N+1 principle. Two web servers that each run at 65% are fine until one of them fails:

import math

node_capacity = 40      # req/s per web node at which p99 is still under 300 ms
ceiling = 0.80          # never plan to run a node above 80% of that
peak = 52               # portal peak, req/s (a cold Monday morning in January)
growth = 0.04           # peak grows 4% per month

def check(nodes, peak):
    normal = peak / (nodes * node_capacity)
    one_down = peak / ((nodes - 1) * node_capacity)
    ok = "OK" if one_down <= ceiling else "NOT N+1"
    return f"{nodes} nodes: {normal:4.0%} normal, {one_down:4.0%} with one down  {ok}"

print(check(2, peak))
print(check(3, peak))

# How long until three nodes are no longer N+1 at this growth?
limit = (3 - 1) * node_capacity * ceiling            # 64 req/s
months = math.log(limit / peak) / math.log(1 + growth)
print(f"Headroom with 3 nodes: {limit - peak:,.0f} req/s, gone in {months:.1f} months")

2 nodes:  65% normal, 130% with one down  NOT N+1
3 nodes:  43% normal,  65% with one down  OK
Headroom with 3 nodes: 12 req/s, gone in 5.3 months

The node capacity of 40 requests per second is not a guess: it comes from a load test (see below). The 80% ceiling leaves room for the queueing effect explained in latency under load. And 5.3 months is the deadline for the next step, which must include the time to order, install and test: the lead time of new capacity is part of the plan.

Headroom also applies to limits that are not hardware. During the incident, db01's connections were climbing by about six per minute toward its configured maximum of 200. How much time is left is simply the remaining room divided by the slope:

promtool query instant http://localhost:9090 '(pg_connections_max - pg_connections_in_use) / deriv(pg_connections_in_use[10m]) / 60'

{env="production", instance="db01", job="postgres", team="platform"} => 10.774039059584526 @[1791306953.369]

About ten minutes, at 19:16 that evening. That kind of number is what makes an incident commander roll back instead of waiting for a diagnosis. PostgreSQL's max_connections is typically 100 by default; Kestrel raised it to 200 for db01, and every service that connects to it shares that budget.

Forecasting growth

A forecast extends today's trend into the future. Prometheus has a built-in one: predict_linear(v[range], seconds) fits a straight line through a gauge's samples in the range (simple linear regression) and returns its value that many seconds ahead. The function is meant for gauges only. The test setup's fake db01 exports its database size and the size of the volume it lives on. The first attempt prints nothing at all:

promtool query instant http://localhost:9090 'pg_database_size_bytes / pg_volume_size_bytes'

An empty result, not an error: pg_database_size_bytes has a datname label that the volume metric lacks, so no series match on both sides. Telling PromQL to ignore that label fixes it:

promtool query instant http://localhost:9090 'pg_database_size_bytes / ignoring (datname) pg_volume_size_bytes'
promtool query instant http://localhost:9090 'predict_linear(pg_database_size_bytes[10m], 14 * 86400) / ignoring (datname) pg_volume_size_bytes'
promtool query instant http://localhost:9090 '(pg_volume_size_bytes - ignoring (datname) pg_database_size_bytes) / ignoring (datname) deriv(pg_database_size_bytes[10m]) / 86400'

{env="production", instance="db01", job="postgres", team="platform"} => 0.4502875936869532 @[1791306959.278]
{env="production", instance="db01", job="postgres", team="platform"} => 0.9122791513960442 @[1791306959.347]
{env="production", instance="db01", job="postgres", team="platform"} => 16.65825283664234 @[1791306965.849]

The database fills 45% of its volume, will fill 91% in 14 days, and the volume is full in 16.7 days. Hamid Rahimi, the database administrator, raises a change to grow the volume this week. Two warnings come with these numbers:

• The window must match the horizon. The test setup's fake database grows at a perfectly steady rate, so ten minutes of data was enough. A real database grows unevenly over a day and a week; forecasting two weeks ahead needs days or weeks of history, or a recording rule that keeps daily values.
• Not all growth is linear. A straight line through exponential growth under-forecasts more the further you look ahead. For business growth, compare month over month in percent; if the percentage is steady, think in doubling times.

The same function powers "will be full within an hour" alerts; see Alerting for a rule built on it, and for a mistake with label matching that it exposed.

Queues and Little's Law

Little's Law, which the Kanban page uses for work items, applies just as well to requests: the average number of items in a system equals the average arrival rate times the average time each item spends in it (L = λ × W). It holds for any stable system over a long enough period, whatever the distribution of arrivals. In capacity planning it answers "how many do I need?": connections in a pool, threads in a server, slots in a queue.

Kestrel has about 400,000 smart meters that each report every 15 minutes. If the readings are spread evenly, that is 444 per second; if every meter reports in the first minute of the quarter hour, which is what happens when their clocks are synchronized, it is 6,667:

# Little's Law: requests in flight = arrival rate x time in system
service_time = 0.120          # seconds per POST /v1/readings, incl. the db01 write

def in_flight(rate_per_s):
    return rate_per_s * service_time

meters = 400_000
interval = 15 * 60            # every meter reports once per 15 minutes

spread = meters / interval    # readings spread evenly over the 15 minutes
burst = meters / 60           # every meter reports in the first minute

print(f"Spread out: {spread:7,.0f} req/s -> {in_flight(spread):5.0f} in flight")
print(f"Burst:      {burst:7,.0f} req/s -> {in_flight(burst):5.0f} in flight")

pool = 60                     # db01 connections the meter-api instances may use
print(f"Pool of {pool}: busy {in_flight(spread) / pool:.0%} when spread,"
      f" needs {in_flight(burst) / pool:.1f}x the pool in a burst")

Spread out:     444 req/s ->    53 in flight
Burst:        6,667 req/s ->   800 in flight
Pool of 60: busy 89% when spread, needs 13.3x the pool in a burst

The same load, arriving in bursts, needs fifteen times the capacity. Spreading periodic work with random delays is cheaper than buying that capacity; see jitter. And a pool that is 89% busy on average has little room left for any burst that does get through; the next section shows how waiting grows as a resource fills up.

Little's Law also works backward. Divide the number waiting by the rate at which they leave, and you get how long each one waits. For the meter-api queue during the incident:

promtool query instant http://localhost:9090 'meter_readings_queue_depth / on (instance) rate(meter_readings_processed_total[5m])'

{instance="meter-api-01:8080"} => 608.5858585858587 @[1791306953.813]
{instance="meter-api-02:8080"} => 589.833382396232 @[1791306953.813]

About 600 seconds: a reading that arrived at that moment waited ten minutes before it reached the database. No metric measured that delay directly; Little's Law derived it from two that were there.

Latency under load

A request that arrives while the server is busy has to wait. That wait is the queueing delay: the time a request spends waiting for a busy resource before work on it starts, on top of the service time it needs once it gets its turn. At low utilization it is nearly zero. As utilization rises it grows, and near 100% it grows without limit, because any short burst of arrivals creates a backlog that the server never has spare time to clear.

The simplest model of a queue, one server with random arrivals and random service times (written M/M/1 in queueing theory), gives a formula: the average time in the system is the service time divided by (1 − utilization). Real systems differ in detail, but the shape is the same:

# Single-server queue (M/M/1): random arrivals, random service times.
# Mean time in system = S / (1 - utilization); waiting = that minus S.
S = 0.120  # service time of one request, in seconds

print("busy   waiting   in system   x service time")
for util in (0.50, 0.70, 0.80, 0.90, 0.95, 0.99):
    in_system = S / (1 - util)
    print(f"{util:4.0%}  {(in_system - S) * 1000:6.0f} ms  {in_system * 1000:7.0f} ms"
          f"   {in_system / S:6.1f}")

busy   waiting   in system   x service time
 50%     120 ms      240 ms      2.0
 70%     280 ms      400 ms      3.3
 80%     480 ms      600 ms      5.0
 90%    1080 ms     1200 ms     10.0
 95%    2280 ms     2400 ms     20.0
 99%   11880 ms    12000 ms    100.0

The point where the curve turns from gently rising to steep is called the knee of the curve: the utilization beyond which a small increase in load causes a large increase in response time. It is not a sharp mathematical point; for one server with random traffic it lies roughly between 70% and 85%, and systems with many parallel workers can run closer to full. Planning to stay below the knee is why the headroom example uses an 80% ceiling. Google's SRE book makes the same observation from practice: "many systems degrade in performance before they achieve 100% utilization", and "latency increases are often a leading indicator of saturation."

The tail grows first

Tail latency is the response time of the slowest requests, usually expressed as a high percentile such as the 99th (p99) or 99.9th. Averages hide it; the statistics page shows how. Queueing makes the tail worse first, because the unlucky requests that arrive just after a burst wait for all of it. A simulation of the same single-server queue, 100,000 requests per utilization level:

import random

def simulate(util, n=100_000, service=0.120, seed=7):
    """Single-server queue with random arrivals and service times.
    Returns the response times (wait + service) in seconds, sorted."""
    rng = random.Random(seed)
    arrival_rate = util / service
    wait, times = 0.0, []
    for _ in range(n):
        work = rng.expovariate(1 / service)
        times.append(wait + work)
        gap = rng.expovariate(arrival_rate)        # until the next request
        wait = max(0.0, wait + work - gap)         # Lindley's recursion
    return sorted(times)

def pct(sorted_values, p):
    return sorted_values[int(p / 100 * (len(sorted_values) - 1))]

print("util   p50 ms   p99 ms  p99/p50")
for util in (0.50, 0.80, 0.90, 0.95):
    t = simulate(util)
    p50, p99 = pct(t, 50) * 1000, pct(t, 99) * 1000
    print(f"{util:4.0%}  {p50:7.0f}  {p99:7.0f}  {p99 / p50:7.1f}")

util   p50 ms   p99 ms  p99/p50
 50%      166     1142      6.9
 80%      417     2636      6.3
 90%      840     5088      6.1
 95%     1691     9460      5.6

At 50% the median request takes 166 ms but 1 in 100 takes over a second; at 95% that 1 in 100 takes more than nine seconds. If your objective is "99% of requests within 300 ms", the tail is what you have to plan for. This model is pessimistic (real service times vary less than this), so use it for the shape and measure the numbers with a load test. Google's SRE book adds a related warning about queues in front of thread pools: with a queue ten times the size of the pool and 100 ms of work per request, a request that finds the queue full takes 1.1 seconds, "most of which time is spent on the queue". Long queues turn overload into latency instead of fast errors; see load shedding.

Load, stress and soak tests

Every capacity number on this page needs one measured input: how much one unit (a server, a pod, a database) can handle while still meeting the objective. That comes from testing with generated traffic. The names below are common usage rather than a formal standard, and teams use them a little differently:

• A load test runs the system at its expected load, typically the forecast peak, for long enough to measure response times, error rates and resource use, to check that it meets its objectives there.
• A stress test runs the system above its expected peak, to see how it behaves and fails there: with fast errors, slow responses or crashes. Pushed step by step until it breaks, it is also called a breakpoint test.
• A soak test (or endurance test) runs a normal load for hours or days, to find problems that only appear over time, such as memory or connection leaks, growing queues, full disks and slow degradation.

A spike test, a sudden jump in load such as the quarter-hour meter burst, is often added as a fourth kind. Popular open-source tools include k6, Apache JMeter, Locust and Gatling. Google's SRE book recommends stress testing explicitly, in its breakpoint form: "Load test components until they break", because the breaking point is "fundamental to the capacity planning process", and because individual components break at different loads, so each needs its own test. How to write and run such tests with k6 is on Performance and Load Testing.

The connection leak behind INC-22047 is a textbook case for a soak test. The release pipeline ran a five-minute load test, during which the leak added a few connections; at production traffic it added six per minute, and db01 was minutes from its limit when the release was rolled back, 24 minutes after the deploy. The postmortem added a two-hour soak test before each release.

A capacity plan in practice

Capacity planning is the regular practice of forecasting demand and adding capacity before it runs out, using the measurements on this page: utilization, headroom, growth and what one unit can handle. Its result, a capacity plan, does not have to be a big document. For each important service, Kestrel keeps one page with these lines, reviewed every quarter and after every incident that involved saturation:

Look for load that competes for the same resource at the same time. At 02:30 the nightly billing export opens its own connections to db01 while meter readings keep arriving every 15 minutes; a plan that checks each workload alone misses the overlap. And remember that utilization is not only CPU: connections, file handles, licenses, API rate limits at a supplier and the disk all run out too, and each needs its own line.