Metrics

Metrics are time series

A monitoring metric is a numeric measurement taken again and again. Each measurement is a sample: a timestamp and a number (Prometheus stores every value as a 64-bit floating-point number). A time series is the sequence of samples for one specific thing, in time order, such as "requests answered with status 200 on the /login route of web01".

Metrics are cheap because a sample is tiny and compresses well. Prometheus needs only a few bytes per sample, so Kestrel can keep 30 days of every service's numbers on one modest monitoring server. The price of that cheapness: a metric tells you how many and how fast, never which customer or which request. For that you need logs and traces.

Labels: the dimensions of a metric

A label is a key-value pair attached to a metric, such as route="/login" or code="200", that splits it into separate time series. The metric name plus the complete set of labels identifies one series. Labels are what let you ask "errors per route" or "latency on web02 only". They resemble Kubernetes labels, and Prometheus can copy those over, but a metric label belongs to a time series, not to an object.

This is what an application's metrics look like when Prometheus reads them. The output below comes from a small test exporter that imitates Kestrel's portal on web01, 15 seconds after it started. The format is plain text: a # HELP line with a description, a # TYPE line, then one line per series:

$ curl -s http://web01:8080/metrics

# HELP http_requests_total Total HTTP requests handled, by method, route and status code.
# TYPE http_requests_total counter
http_requests_total{method="GET",route="/account",code="200"} 70
http_requests_total{method="GET",route="/account",code="404"} 2
http_requests_total{method="GET",route="/invoices",code="200"} 40
http_requests_total{method="GET",route="/invoices/{invoice_id}/pdf",code="200"} 11
http_requests_total{method="GET",route="/invoices/{invoice_id}/pdf",code="404"} 1
http_requests_total{method="GET",route="/login",code="200"} 48
http_requests_total{method="POST",route="/login",code="200"} 32
# HELP http_request_duration_seconds Time spent handling HTTP requests.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{method="GET",route="/login",le="0.005"} 0
http_request_duration_seconds_bucket{method="GET",route="/login",le="0.01"} 1
http_request_duration_seconds_bucket{method="GET",route="/login",le="0.025"} 18
http_request_duration_seconds_bucket{method="GET",route="/login",le="0.05"} 40
http_request_duration_seconds_bucket{method="GET",route="/login",le="0.1"} 48
http_request_duration_seconds_bucket{method="GET",route="/login",le="0.25"} 48
http_request_duration_seconds_bucket{method="GET",route="/login",le="0.5"} 48
http_request_duration_seconds_bucket{method="GET",route="/login",le="1"} 48
http_request_duration_seconds_bucket{method="GET",route="/login",le="2.5"} 48
http_request_duration_seconds_bucket{method="GET",route="/login",le="5"} 48
http_request_duration_seconds_bucket{method="GET",route="/login",le="10"} 48
http_request_duration_seconds_bucket{method="GET",route="/login",le="+Inf"} 48
http_request_duration_seconds_sum{method="GET",route="/login"} 1.612
http_request_duration_seconds_count{method="GET",route="/login"} 48
…

http_requests_total is one metric with seven series so far: one per combination of method, route and status code that has occurred. Prometheus adds its own labels when it stores them, such as job="portal" and instance="web01:8080", so web01 and web02 never mix (see Prometheus).

Naming conventions

Prometheus' naming guidelines are widely followed, also outside Prometheus:

• Use base units: seconds, bytes, meters, and say so in the name (http_request_duration_seconds, not _ms). One metric uses one unit; mixing milliseconds and seconds in one metric makes every query wrong.
• A counter of things ends in _total: meter_readings_received_total.
• Start with a prefix for the application or domain: meter_, billing_, pg_.
• Put metadata in an info metric: a series with the value 1 whose labels carry the data, such as the running version.

# HELP kestrel_build_info Build information of the running service.
# TYPE kestrel_build_info gauge
kestrel_build_info{version="2.14.3",commit="9f3c2ab"} 1

The four metric types

Prometheus' text format knows four types (plus "untyped"). The type tells you, and the query language, which operations make sense:

Counters

A counter is a metric whose value only increases, by counting events since the process started: requests handled, readings received, bytes sent, errors. Its raw value is rarely interesting ("368,748 readings since Tuesday's restart"). What matters is how fast it grows. Prometheus computes that with rate(), explained in PromQL:

$ promtool query instant http://localhost:9090 'rate(http_requests_total{job="meter-api", route="/v1/readings", code="202"}[5m])'

{code="202", env="production", instance="meter-api-01:8080", job="meter-api", method="POST", route="/v1/readings", team="metering"} => 37.23508771929824 @[1791306532.603]
{code="202", env="production", instance="meter-api-02:8080", job="meter-api", method="POST", route="/v1/readings", team="metering"} => 37.16491228070175 @[1791306532.603]

About 37 accepted readings per second on each meter-api instance, averaged over the last 5 minutes. Counters are robust: if Prometheus misses a scrape, the next sample still contains every event, so nothing is lost.

Do not confuse this with Python's collections.Counter, which counts how often each value occurs in a collection.

Counter resets

A counter lives in the memory of the process that counts. When meter-api restarts, for a deploy or after a crash, the counter starts again at zero. That drop is a counter reset. Because a counter may never go down otherwise, any decrease must be a reset, and rate() and increase() correct for it automatically. A script that simply subtracts the first value from the last does not:

# meter_readings_received_total on meter-api-01, one sample every 15 seconds.
# The process restarted between 17:31:00 and 17:31:15, so the counter began again at 0.
samples = [
    ("17:30:30", 368_748),
    ("17:30:45", 371_001),
    ("17:31:00", 373_239),
    ("17:31:15", 1_502),      # restarted: counter reset
    ("17:31:30", 3_755),
    ("17:31:45", 6_001),
]
seconds = 15 * (len(samples) - 1)

naive = samples[-1][1] - samples[0][1]
print(f"last - first: {naive} readings  (nonsense)")

increase = 0
for (_, previous), (time, value) in zip(samples, samples[1:]):
    if value < previous:      # a counter never goes down, so this is a reset:
        increase += value     # count from zero up to the new value
        print(f"reset detected at {time}")
    else:
        increase += value - previous
print(f"increase: {increase} readings in {seconds} s")
print(f"rate: {increase / seconds:.1f} readings per second")

last - first: -362747 readings  (nonsense)
reset detected at 17:31:15
increase: 10492 readings in 75 s
rate: 139.9 readings per second

The reset-aware rate is about 140 readings per second, a bit below the real 150. The readings counted between the last scrape before the restart and the restart itself were never scraped, and are lost. That is the reason never to restart a service in a loop to "clear" a problem: every restart blurs its own metrics.

Gauges

A gauge is a metric whose value can go up and down: the current reading of something, like the fuel gauge in a car. Queue depth, connections in use, free disk space, temperature and "Unix time of the last successful run" are gauges. You use the value directly, compare it with a limit, or look at its trend.

$ curl -s http://db01:9187/metrics

# HELP pg_connections_in_use Client connections currently open on the PostgreSQL server.
# TYPE pg_connections_in_use gauge
pg_connections_in_use 63
# HELP pg_connections_max Value of the max_connections setting.
# TYPE pg_connections_max gauge
pg_connections_max 200

$ curl -s http://billing01:9101/metrics

# HELP billing_export_last_success_timestamp_seconds Unix time of the last successful nightly billing export.
# TYPE billing_export_last_success_timestamp_seconds gauge
billing_export_last_success_timestamp_seconds 1791247297
# HELP billing_export_duration_seconds Duration of the last billing export run.
# TYPE billing_export_duration_seconds gauge
billing_export_duration_seconds 697.4

1791247297 is 6 October 2026, 00:41:37 UTC (02:41:37 in the Netherlands): the moment last night's 02:30 billing export finished, after 697 seconds. Publishing a timestamp as a gauge is the standard way to monitor a batch job (see jobs that are not request-driven).

Histograms

Latency is not one number but a distribution: most requests are quick, some are slow. Storing every single duration would cost as much as logs. A histogram instead counts observations in a fixed set of buckets, each with an upper bound le ("less than or equal"), and also keeps the _sum of all observed values and their _count. In Prometheus' classic histograms the buckets are cumulative: the bucket le="0.25" counts every request of 0.25 s or less, including those already in le="0.1". The le="+Inf" bucket always equals _count.

The readings endpoint of meter-api-01 after 40 minutes:

$ curl -s http://meter-api-01:8080/metrics | grep 'route="/v1/readings"'

http_requests_total{method="POST",route="/v1/readings",code="202"} 87920
http_requests_total{method="POST",route="/v1/readings",code="400"} 1881
http_requests_total{method="POST",route="/v1/readings",code="500"} 1218
http_requests_total{method="POST",route="/v1/readings",code="503"} 2806
http_request_duration_seconds_bucket{method="POST",route="/v1/readings",le="0.005"} 0
http_request_duration_seconds_bucket{method="POST",route="/v1/readings",le="0.01"} 24
http_request_duration_seconds_bucket{method="POST",route="/v1/readings",le="0.025"} 4935
http_request_duration_seconds_bucket{method="POST",route="/v1/readings",le="0.05"} 31013
http_request_duration_seconds_bucket{method="POST",route="/v1/readings",le="0.1"} 56677
http_request_duration_seconds_bucket{method="POST",route="/v1/readings",le="0.25"} 87503
http_request_duration_seconds_bucket{method="POST",route="/v1/readings",le="0.5"} 93614
http_request_duration_seconds_bucket{method="POST",route="/v1/readings",le="1"} 93824
http_request_duration_seconds_bucket{method="POST",route="/v1/readings",le="2.5"} 93825
http_request_duration_seconds_bucket{method="POST",route="/v1/readings",le="5"} 93825
http_request_duration_seconds_bucket{method="POST",route="/v1/readings",le="10"} 93825
http_request_duration_seconds_bucket{method="POST",route="/v1/readings",le="+Inf"} 93825
http_request_duration_seconds_sum{method="POST",route="/v1/readings"} 9873.34
http_request_duration_seconds_count{method="POST",route="/v1/readings"} 93825

Reading it: 87,503 of the 93,825 requests took 0.25 s or less; 210 took between 0.5 s and 1 s (93,824 − 93,614); one took between 1 s and 2.5 s. The mean is _sum / _count = 9873.34 / 93825 ≈ 0.105 s.

Estimating percentiles from buckets

Buckets do not record exact durations, so a percentile computed from them is an estimate. Prometheus' histogram_quantile() finds the bucket that contains the wanted rank and assumes the requests are spread evenly inside it (linear interpolation). The same calculation in Python, for the 1,000 requests in the diagram:

import math

# Cumulative bucket counts for 1,000 portal requests: (upper bound "le" in seconds, count)
buckets = [(0.1, 900), (0.25, 960), (0.5, 990), (1.0, 998), (math.inf, 1000)]


def bucket_quantile(q, buckets):
    """Estimate a quantile from cumulative buckets, the way histogram_quantile() does."""
    total = buckets[-1][1]
    rank = q * total
    lower, below = 0.0, 0            # the lowest bucket is assumed to start at 0
    for upper, count in buckets:
        if count >= rank:
            if upper == math.inf:    # rank in the +Inf bucket: return the highest finite bound
                return lower
            # linear interpolation: assume requests are spread evenly inside the bucket
            return lower + (upper - lower) * (rank - below) / (count - below)
        lower, below = upper, count


for q in (0.5, 0.95, 0.99, 0.999):
    print(f"p{q * 100:g}: {bucket_quantile(q, buckets):.4f} s")

p50: 0.0556 s
p95: 0.2250 s
p99: 0.5000 s
p99.9: 1.0000 s

Three things follow from this:

• The estimate is only as good as the buckets around it. The p95 of 0.225 s assumes the 60 requests between 0.1 s and 0.25 s are spread evenly; they might all have taken 0.24 s.
• The estimate can never be higher than the highest finite bound. The p99.9 says 1 s, but the two slowest requests may have taken 30 s. Add buckets beyond the slowest time you care about.
• Put a bucket boundary exactly at your target. Kestrel's portal target is "99% of requests within 300 ms", so its histogram needs a bucket le="0.3": then "share of requests within 0.3 s" is an exact count, not an interpolation (see SLIs, SLOs and SLAs).

Summaries, and why you cannot average percentiles

A summary also measures a distribution, but the application itself computes chosen percentiles over a recent time window and exposes them as series such as {quantile="0.95"}, next to _sum and _count. The percentiles are exact for that one process, and cheap to query. The catch: percentiles from several processes cannot be combined. Prometheus' documentation marks summary quantiles as "Not aggregatable", and the reason is plain arithmetic:

import math


def p95(values):
    ordered = sorted(values)
    return ordered[math.ceil(0.95 * len(ordered)) - 1]


# Five minutes of portal latencies in seconds
web01 = [0.08] * 900 + [1.2] * 100    # 1,000 requests, some wait for the slow PDF route
web02 = [0.10] * 200                  # 200 requests, all quick

print(f"p95 web01: {p95(web01)} s")
print(f"p95 web02: {p95(web02)} s")
print(f"average of the two p95s:  {(p95(web01) + p95(web02)) / 2:.2f} s  (wrong)")
weighted = (p95(web01) * len(web01) + p95(web02) * len(web02)) / (len(web01) + len(web02))
print(f"weighted average:         {weighted:.2f} s  (also wrong)")
print(f"p95 of all requests:      {p95(web01 + web02)} s")

p95 web01: 1.2 s
p95 web02: 0.1 s
average of the two p95s:  0.65 s  (wrong)
weighted average:         1.02 s  (also wrong)
p95 of all requests:      1.2 s

Neither the plain nor the weighted average of the two p95s is the p95 of all requests, and no formula can get it from the two numbers alone. Histograms do not have this problem: you add up the bucket counts of web01 and web02 first, then estimate the percentile from the combined buckets. That is why histogram_quantile(0.95, sum by (le) (…)) is the standard pattern, and why most teams use histograms rather than summaries.

Why averages hide problems

The percentile and the mean are explained with worked examples in Statistics for IT. For monitoring, one example says it all. Of 1,000 portal requests, 940 take 80 ms and 60 wait 2 seconds for an invoice PDF:

import math

# 1,000 portal requests: most are fast, a few wait for a slow invoice PDF
latencies_ms = [80] * 940 + [2000] * 60


def percentile(values, p):
    """Nearest-rank percentile: the value at position ceil(p/100 * n)."""
    ordered = sorted(values)
    rank = math.ceil(p / 100 * len(ordered))
    return ordered[rank - 1]


mean = sum(latencies_ms) / len(latencies_ms)
print(f"mean: {mean:.1f} ms")
for p in (50, 90, 95, 99):
    print(f"p{p}:  {percentile(latencies_ms, p)} ms")
slow = sum(1 for ms in latencies_ms if ms > 300)
print(f"over 300 ms: {slow / len(latencies_ms):.0%} of requests")

mean: 195.2 ms
p50:  80 ms
p90:  80 ms
p95:  2000 ms
p99:  2000 ms
over 300 ms: 6% of requests

A mean of 195 ms is comfortably under a 300 ms target, while 6% of customers wait two full seconds. The p95 shows it at once. The real portal, measured on Kestrel's test monitoring server before the simulated incident, shows the same pattern. The p95:

$ promtool query instant http://localhost:9090 'histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="portal"}[5m])))'

{} => 0.3909934497816592 @[1791306532.932]

The mean at the same moment, from _sum and _count:

$ promtool query instant http://localhost:9090 'sum(rate(http_request_duration_seconds_sum{job="portal"}[5m])) / sum(rate(http_request_duration_seconds_count{job="portal"}[5m]))'

{} => 0.12238675623800384 @[1791306533.063]

The p95 (0.391 s) is three times the mean (0.122 s). Split by route, the culprit is clear:

$ promtool query instant http://localhost:9090 'histogram_quantile(0.95, sum by (route, le) (rate(http_request_duration_seconds_bucket{job="portal"}[5m])))'

{route="/account"} => 0.2355757575757576 @[1791306533]
{route="/invoices"} => 0.36451612903225783 @[1791306533]
{route="/invoices/{invoice_id}/pdf"} => 0.9635135135135133 @[1791306533]
{route="/login"} => 0.2337923728813559 @[1791306533]

The invoice PDF route drives the tail. A mean would have hidden it; labels and percentiles together find it in two queries.

Labels and cardinality

Cardinality is the number of distinct time series a metric (or a whole monitoring system) has: the number of possible combinations of its label values. Each label multiplies it. Every series costs memory in the monitoring server and storage on disk, and queries that touch many series get slow. Prometheus' guidelines put it bluntly: do not use labels for "user IDs, email addresses, or other unbounded sets of values".

import math

# Possible values per label of http_requests_total for the portal
labels = {"method": 4, "route": 25, "code": 8, "instance": 2}
series = math.prod(labels.values())
print(f"{' x '.join(map(str, labels.values()))} = {series:,} series")

# The same counter, plus one label per customer
labels["customer_id"] = 300_000
series_with_ids = math.prod(labels.values())
print(f"with customer_id: {series_with_ids:,} series")

# A classic histogram with 12 buckets has 14 series per label combination
# (12 _bucket series plus _sum and _count)
print(f"as a 12-bucket histogram: {series_with_ids * 14:,} series")

4 x 25 x 8 x 2 = 1,600 series
with customer_id: 480,000,000 series
as a 12-bucket histogram: 6,720,000,000 series

1,600 series is nothing; 480 million would take down any Prometheus server. At Kestrel the risk is concrete: a developer who adds the meter's EAN code as a label to "see which meters fail" adds one series per meter. Put identifiers like that in logs and traces, which are built for one record per event. Keep labels for small, fixed sets: route templates, status codes, instances, regions.

You can measure cardinality with PromQL. The number of series per metric name of the meter-api job on Kestrel's test server:

$ promtool query instant http://localhost:9090 'count by (__name__) ({job="meter-api"})'

http_requests_total => 18 @[1791308503.672]
http_request_duration_seconds_bucket => 72 @[1791308503.672]
http_request_duration_seconds_sum => 6 @[1791308503.672]
http_request_duration_seconds_count => 6 @[1791308503.672]
meter_readings_queue_depth => 2 @[1791308503.672]
meter_readings_queue_limit => 2 @[1791308503.672]
meter_readings_received_total => 2 @[1791308503.672]
meter_readings_processed_total => 2 @[1791308503.672]
kestrel_build_info => 2 @[1791308503.672]
up => 2 @[1791308503.672]
scrape_duration_seconds => 2 @[1791308503.672]
scrape_samples_scraped => 2 @[1791308503.672]
scrape_samples_post_metric_relabeling => 2 @[1791308503.672]
scrape_series_added => 2 @[1791308503.672]
…

The histogram dominates: 72 bucket series for 3 routes × 2 instances × 12 buckets. The up and scrape_* series are added by Prometheus for every target. To spot a label that is exploding, count its distinct values:

$ promtool query instant http://localhost:9090 'count(count by (route) (http_requests_total))'

{} => 7 @[1791308503.896]

Seven routes, because the application labels requests with the route template (/invoices/{invoice_id}/pdf), not the raw path. If this number grows every day, someone is putting raw paths with IDs in a label.

Scrape interval and resolution

Prometheus reads each target's metrics at a fixed scrape interval. The built-in default is 1 minute; the example configuration that ships with Prometheus, and most setups, use 15 seconds. The interval is the resolution of your data: between two scrapes Prometheus knows nothing.

That matters differently for the two basic types. A gauge is a snapshot: a spike that starts and ends between two scrapes is simply never seen. A counter keeps counting between scrapes, so the next scrape still shows that events happened, even if you cannot tell exactly when. One minute of db01, with a burst of reconnects from meter-api between two scrapes:

# Open connections on db01, the real value in each second of one minute,
# and a counter of connections opened since the server started.
in_use = []
opened_total = 51_200
opened_at_scrape = {}
for second in range(60):
    if 20 <= second < 26:               # meter-api reconnects all at once
        in_use.append(190)
        opened_total += 22
    else:
        in_use.append(60)
        opened_total += 1
    if second % 15 == 14:               # the value Prometheus sees at the next scrape
        opened_at_scrape[second + 1] = opened_total

scrapes = range(0, 60, 15)              # Prometheus scrapes at :00, :15, :30 and :45
print(f"real maximum:       {max(in_use)} connections")
print(f"gauge as scraped:   {[in_use[s] for s in scrapes]}")
increases = []
previous = 51_200
for second, value in opened_at_scrape.items():
    increases.append(value - previous)
    previous = value
print(f"counter increase per 15 s: {increases}")

real maximum:       190 connections
gauge as scraped:   [60, 60, 60, 60]
counter increase per 15 s: [15, 141, 15, 15]

The gauge never shows the 190 connections, but the counter's jump from 15 to 141 new connections per interval does. For limits that must never be crossed, the application itself should count the event ("connections refused") or record the peak as its own gauge, rather than relying on a scrape to land at the right moment.

What goes wrong with metrics

• A function on the wrong type. rate() belongs on counters, and predict_linear() or deriv() on gauges. Prometheus usually still returns a number, just a meaningless one.
• Mixed units. One service reports milliseconds, another seconds, and the dashboard adds them. Use base units everywhere.
• Averaging percentiles. Sum the histogram buckets first, then compute the percentile (see summaries).
• Matching on a bucket bound as written. The test exporter writes le="1", but Prometheus 3 stores bucket bounds as normalized floating-point numbers, so the series is called le="1.0" and a matcher le="1" finds nothing:

http_request_duration_seconds_bucket{env="production", instance="web02:8080", job="portal", le="0.5", method="GET", route="/login", team="portal"} => 1489 @[1791306125.802]
http_request_duration_seconds_bucket{env="production", instance="web02:8080", job="portal", le="1.0", method="GET", route="/login", team="portal"} => 1490 @[1791306125.802]

• A label that slowly explodes. Cardinality problems rarely arrive in one deploy; they grow with every new customer or invoice. Check the series count per job regularly (the Prometheus page shows how).