PromQL

What PromQL is

PromQL (Prometheus Query Language) is the functional query language of Prometheus. You use it to select time series by name and labels, to calculate with them — rates, ratios, sums, percentiles, predictions — and to compare the results with a threshold. The same expressions appear everywhere in a Prometheus setup: typed into the query page of the Prometheus web interface, in Grafana panels (Dashboards), in recording and alerting rules, in scripts through the HTTP API, and on the command line with promtool query.

PromQL is not SQL. There are no tables and no SELECT; an expression describes a set of series and what to do with them, and Prometheus evaluates it at one moment (an instant query) or at regular steps over a period (a range query, which is what a graph does). Every query on this page was run against the Kestrel test setup described on the Prometheus page, with simulated traffic. Some ran before a simulated incident at 17:09 UTC, others during it, when meter-api failed 10% of its readings and the portal slowed down.

Four types of result

Most mistakes in PromQL are type mistakes: passing an instant vector where a function needs a range vector, or trying to graph a range vector. The error messages say so, as the last section shows.

Selectors and label matchers

The simplest query is a metric name. It returns an instant vector: for every series with that name, one sample — the most recent value at the evaluation time. "Most recent" has a limit: Prometheus looks back at most five minutes (the lookback delta). A series that has not been scraped for longer, or that was marked stale because its target disappeared, is left out.

Curly braces narrow the selection. Each condition inside is a label matcher: a label name, an operator and a value that a series' label must satisfy for the series to be selected. All matchers must hold at once. One meter-api instance, one route, every status code:

$ promtool query instant http://localhost:9090 'http_requests_total{job="meter-api", instance="meter-api-01:8080", route="/v1/readings"}'

http_requests_total{code="202", env="production", instance="meter-api-01:8080", job="meter-api", method="POST", route="/v1/readings", team="metering"} => 10905 @[1791306119.366]
http_requests_total{code="400", env="production", instance="meter-api-01:8080", job="meter-api", method="POST", route="/v1/readings", team="metering"} => 226 @[1791306119.366]
http_requests_total{code="503", env="production", instance="meter-api-01:8080", job="meter-api", method="POST", route="/v1/readings", team="metering"} => 6 @[1791306119.366]
http_requests_total{code="500", env="production", instance="meter-api-01:8080", job="meter-api", method="POST", route="/v1/readings", team="metering"} => 1 @[1791306119.366]

Every line is one series: the full label set, then =>, the value and the evaluation time as a Unix timestamp. The values are counters — totals since the process started — which is why the raw numbers mean little; the next sections turn them into rates. There are four operators:

# Equal and not equal
http_requests_total{job="portal", code!="200"}

# Regular expression: all 5xx codes (fully anchored, so "5.." never matches "5000")
http_requests_total{job="meter-api", code=~"5.."}

# Negative regular expression: every route except the invoice routes
http_requests_total{job="portal", route!~"/invoices.*"}

# A name with dots (allowed since Prometheus 3.0) goes inside the braces, quoted
{"kestrel.portal.logins", "http.route"="/login"}

The third of these, narrowed to successful GET requests, during normal traffic:

$ promtool query instant http://localhost:9090 'http_requests_total{job="portal", route!~"/invoices.*", method="GET", code="200"}'

http_requests_total{code="200", env="production", instance="web02:8080", job="portal", method="GET", route="/account", team="portal"} => 1254 @[1791306109.334]
http_requests_total{code="200", env="production", instance="web02:8080", job="portal", method="GET", route="/login", team="portal"} => 836 @[1791306109.334]
http_requests_total{code="200", env="production", instance="web01:8080", job="portal", method="GET", route="/account", team="portal"} => 1282 @[1791306109.334]
http_requests_total{code="200", env="production", instance="web01:8080", job="portal", method="GET", route="/login", team="portal"} => 855 @[1791306109.334]

Regular expressions are fully anchored

PromQL uses regular expressions in the RE2 syntax, and a matcher must match the whole label value: code=~"5.." is treated as ^5..$. That is different from grep and from Python's re.search(), which find a match anywhere in the text. Python's re.fullmatch() behaves like PromQL:

import re

codes = ["200", "202", "404", "500", "503", "5000"]
pattern = "5.."   # as in code=~"5.."

print("search:   ", [c for c in codes if re.search(pattern, c)])
print("fullmatch:", [c for c in codes if re.fullmatch(pattern, c)])  # what PromQL does
print("route=~\"/invoices\" matches /invoices/{invoice_id}/pdf:",
      bool(re.fullmatch("/invoices", "/invoices/{invoice_id}/pdf")))
print("route=~\"/invoices.*\" matches /invoices/{invoice_id}/pdf:",
      bool(re.fullmatch("/invoices.*", "/invoices/{invoice_id}/pdf")))

search:    ['500', '503', '5000']
fullmatch: ['500', '503']
route=~"/invoices" matches /invoices/{invoice_id}/pdf: False
route=~"/invoices.*" matches /invoices/{invoice_id}/pdf: True

So route=~"/invoices" finds only the route that is exactly /invoices; add .* for "starts with".

To find out which labels and values exist before writing a query, ask Prometheus — the web interface autocompletes the same information:

$ promtool query labels http://localhost:9090 route
$ promtool query series http://localhost:9090 --match='http_requests_total{instance="web01:8080",route="/login"}'

/account
/healthz
/invoices
/invoices/{invoice_id}/pdf
/login
/v1/meters/{meter_id}/readings
/v1/readings
{__name__="http_requests_total", code="200", env="production", instance="web01:8080", job="portal", method="GET", route="/login", team="portal"}
{__name__="http_requests_total", code="200", env="production", instance="web01:8080", job="portal", method="POST", route="/login", team="portal"}
{__name__="http_requests_total", code="400", env="production", instance="web01:8080", job="portal", method="POST", route="/login", team="portal"}
{__name__="http_requests_total", code="404", env="production", instance="web01:8080", job="portal", method="GET", route="/login", team="portal"}
{__name__="http_requests_total", code="503", env="production", instance="web01:8080", job="portal", method="POST", route="/login", team="portal"}
{__name__="http_requests_total", code="503", env="production", instance="web01:8080", job="portal", method="GET", route="/login", team="portal"}

Notice the route values: templates such as /invoices/{invoice_id}/pdf, not real URLs with invoice numbers. That keeps the number of series small (see Metrics). And one detail of Prometheus 3: the bucket label le of classic histograms is stored as a normalized number, so an exporter's le="1" is found with le="1.0", not le="1".

Range vectors and time

A duration in square brackets after a selector turns it into a range vector: for every matching series, all samples within that time window before the evaluation time. [5m] covers the last five minutes; the window is open on the left and closed on the right, so a sample exactly five minutes old is not included. With a 15-second scrape interval, [5m] usually holds 20 samples. Durations use the units ms, s, m, h, d, w and y.

A range vector cannot be graphed or compared directly; it is the input for functions that summarize a window. The …_over_time functions do exactly that for gauges — the average number of database connections and the highest queue depth over the last five minutes:

$ promtool query instant http://localhost:9090 'avg_over_time(pg_connections_in_use[5m])'
$ promtool query instant http://localhost:9090 'max_over_time(meter_readings_queue_depth[5m])'

{env="production", instance="db01", job="postgres", team="platform"} => 80.2 @[1791306533.132]
{env="production", instance="meter-api-01:8080", job="meter-api", team="metering"} => 40 @[1791306533.201]
{env="production", instance="meter-api-02:8080", job="meter-api", team="metering"} => 40 @[1791306533.201]

Looking back: offset and @

offset shifts a selector back in time, which makes comparisons possible. How many more connections does db01 have now than five minutes ago?

$ promtool query instant http://localhost:9090 'pg_connections_in_use - pg_connections_in_use offset 5m'

{env="production", instance="db01", job="postgres", team="platform"} => 31 @[1791306539.063]

The @ modifier pins a selector to an absolute Unix time, and a subquery [1h:1m] evaluates an instant expression every minute over the last hour so that an …_over_time function can summarize it:

# Request rate now, compared with the same moment one week ago
sum(rate(http_requests_total{job="portal"}[5m]))
  / sum(rate(http_requests_total{job="portal"}[5m] offset 1w))

# Evaluate at a fixed time (Unix seconds): 6 October 2026, 17:16 UTC
sum by (instance) (rate(http_requests_total{job="meter-api", code=~"5.."}[5m] @ 1791306960))

# The highest 5-minute error ratio of the last hour, sampled every minute (a subquery)
max_over_time(job:http_requests_errors:ratio_rate5m{job="meter-api"}[1h:1m])

The modifier belongs directly after the selector it shifts, inside any function. Putting it after the whole expression is a parse error:

# Invalid: offset must follow the selector, not the function or aggregation
sum(rate(http_requests_total{job="portal"}[5m])) offset 1w

Counters: rate() and increase()

Most important metrics are counters: totals that only go up while the process runs, such as http_requests_total. The current total is rarely interesting; how fast it grows is. rate() takes a range vector of a counter and returns, per series, the average increase per second over that window. It handles counter resets — when a process restarts and its counters start again at zero — and extrapolates to the edges of the window. increase() returns the total increase over the window instead; it is the same calculation multiplied by the length of the window in seconds.

$ promtool query instant http://localhost:9090 'rate(http_requests_total{job="meter-api", route="/v1/readings", code="202"}[5m])'
$ promtool query instant http://localhost:9090 'increase(http_requests_total{job="meter-api", route="/v1/readings", code="202"}[5m])'

{code="202", env="production", instance="meter-api-01:8080", job="meter-api", method="POST", route="/v1/readings", team="metering"} => 37.23508771929824 @[1791306532.603]
{code="202", env="production", instance="meter-api-02:8080", job="meter-api", method="POST", route="/v1/readings", team="metering"} => 37.16491228070175 @[1791306532.603]
{code="202", env="production", instance="meter-api-01:8080", job="meter-api", method="POST", route="/v1/readings", team="metering"} => 11170.526315789473 @[1791306532.667]
{code="202", env="production", instance="meter-api-02:8080", job="meter-api", method="POST", route="/v1/readings", team="metering"} => 11149.473684210525 @[1791306532.667]

Each meter-api instance accepts about 37 readings per second, and increase() is exactly 300 times rate(): 37.235… × 300 = 11,170.5…. The odd decimals are not a rounding error. A 5-minute window with a 15-second scrape interval holds 20 samples, but they span only 285 seconds (19 intervals). Prometheus measured an increase of 10,612 requests over those 285 seconds and extrapolated it to the full window: 10,612 ÷ 285 = 37.2350877… per second. That is also why increase() on a counter of whole requests can return a fraction.

What rate() does, step by step

This script follows the same steps as Prometheus for a single series (simplified: no native histograms, no zero-point details beyond the basic rule). It reproduces the 285-second effect and shows what a restart does:

def extrapolated_rate(samples, range_start, range_end, is_rate=True):
    """samples: (time in s, counter value) inside the window, oldest first.
    Follows the steps of rate()/increase() in Prometheus for a float counter."""
    if len(samples) < 2:
        return None                      # rate() needs at least two samples
    first_t, first_v = samples[0]
    last_t, last_v = samples[-1]
    result = last_v - first_v
    previous = first_v
    for _, value in samples[1:]:
        if value < previous:             # counter reset: the process restarted
            result += previous           # add what was counted before the reset
        previous = value
    sampled = last_t - first_t
    average_gap = sampled / (len(samples) - 1)
    to_start = first_t - range_start
    to_end = range_end - last_t
    threshold = average_gap * 1.1       # a sample "missing" at an edge?
    if to_start >= threshold:
        to_start = average_gap / 2
    if result > 0 and first_v >= 0:      # never extrapolate below zero
        to_start = min(to_start, sampled * first_v / result)
    if to_end >= threshold:
        to_end = average_gap / 2
    factor = (sampled + to_start + to_end) / sampled
    if is_rate:
        factor /= range_end - range_start
    return result * factor

# A 5-minute window ending at t=300, scraped every 15 s: 20 samples at 15..300
# (a sample exactly at the start of the window, t=0, is not included).
steady = [(t, 5_000 + 37 * t) for t in range(15, 301, 15)]
print("steady: increase in samples", steady[-1][1] - steady[0][1])
print("  rate     =", round(extrapolated_rate(steady, 0, 300), 3), "per second")
print("  increase =", round(extrapolated_rate(steady, 0, 300, is_rate=False), 1))

# The same traffic, but the process restarts at t=150 and counts from 0 again.
restarted = [(t, 5_000 + 37 * t if t < 150 else 37 * (t - 150) + 20) for t in range(15, 301, 15)]
print("restart: raw last - first =", restarted[-1][1] - restarted[0][1])
print("  rate     =", round(extrapolated_rate(restarted, 0, 300), 3), "per second")

steady: increase in samples 10545
  rate     = 37.0 per second
  increase = 11100.0
restart: raw last - first = 15
  rate     = 35.123 per second

Without reset handling, the restart would make the counter look as if it had grown by 15 requests in five minutes. With it, the rate is almost right; the requests between the last scrape before the restart and the restart itself are lost, which is why it is 35.1 and not 37. This is the reason for the rule "take the rate first, then aggregate". After sum(http_requests_total) the restart of one instance is just a dip in a total, which rate() would read as a reset of the whole sum. Rates of change on the Math page explains the underlying difference quotient.

• Window length. The window must hold at least two samples, and preferably four or more: with a 15-second scrape, use at least [1m]; [5m] is a good default. Too short a window gives gaps and jumpy lines; too long a window reacts late (the incident below took about five minutes to show fully in a 5-minute rate).
• irate() uses only the last two samples in the window. It shows spikes in a graph, but is too nervous for alerts.
• Only counters. The documentation says rate() should only be used with counters. For gauges use deriv() or the …_over_time functions.

Prometheus warns, but quietly

rate() on a gauge is not an error, just wrong: meter_readings_queue_depth goes up and down, and every decrease looks like a counter reset. Prometheus returns a result — promtool prints it without comment — but the HTTP API adds an infos field based on the metric name:

$ curl -s 'localhost:9090/api/v1/query' --data-urlencode 'query=rate(meter_readings_queue_depth[5m])' | jq .

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "env": "production",
          "instance": "meter-api-01:8080",
          "job": "meter-api",
          "team": "metering"
        },
        "value": [
          1791306539.176,
          "0.687719298245614"
        ]
      },
…
    ]
  },
  "infos": [
    "PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: \"meter_readings_queue_depth\" (1:6)"
  ]
}

The web interface shows such annotations above the result; Grafana shows them as a small warning on the panel. They are easy to miss, so do not rely on them: know your metric types.

Aggregation: sum by and friends

A rate per instance, route, method and status code is far too much detail for a dashboard. Aggregation operators combine series: sum, avg, min, max, count, topk, bottomk, quantile and a few more. by (labels) says which labels to keep — one result per combination of their values; without (labels) says which to drop and keeps the rest. It works like GROUP BY in SQL.

# One number: all requests per second, over all series
sum(rate(http_requests_total[5m]))

# Keep only the job label
sum by (job) (rate(http_requests_total[5m]))

# Keep every label except instance (add up the instances, keep the routes and codes)
sum without (instance) (rate(http_requests_total[5m]))

# The busiest database: highest connection count per environment
max by (env) (pg_connections_in_use)

Portal traffic per route, and the three busiest routes across all jobs:

$ promtool query instant http://localhost:9090 'sum by (route) (rate(http_requests_total{job="portal"}[5m]))'
$ promtool query instant http://localhost:9090 'topk(3, sum by (job, route) (rate(http_requests_total[5m])))'

{route="/account"} => 8.996491228070173 @[1791306532.728]
{route="/invoices"} => 4.990332481871344 @[1791306532.728]
{route="/invoices/{invoice_id}/pdf"} => 1.5970873286549705 @[1791306532.728]
{route="/login"} => 9.996491803868645 @[1791306532.728]
{job="meter-api", route="/v1/readings"} => 76.02134080733286 @[1791306532.795]
{job="meter-api", route="/v1/meters/{meter_id}/readings"} => 11.999999999999998 @[1791306532.795]
{job="portal", route="/login"} => 9.99649753036437 @[1791306532.795]

Ratios: the error rate

Arithmetic between two instant vectors works series by series: each series on the left is paired with the series on the right that has exactly the same labels. The error ratio per job — 5xx responses divided by all responses — is the most common example:

$ promtool query instant http://localhost:9090 'sum by (job) (rate(http_requests_total{code=~"5.."}[5m])) / sum by (job) (rate(http_requests_total[5m]))'

{job="portal"} => 0 @[1791306532.859]
{job="meter-api"} => 0.00020173305338995023 @[1791306532.859]

Before the incident meter-api failed 0.02% of its requests. A range query evaluates the same expression at regular steps, which is what a graph does. Here the stored version of that ratio (a recording rule, see below) over the ten minutes around the start of the incident at 17:09:07:

$ promtool query range --start=2026-10-06T17:06:00Z --end=2026-10-06T17:16:00Z --step=1m http://localhost:9090 'job:http_requests_errors:ratio_rate5m{job="meter-api"}'

job:http_requests_errors:ratio_rate5m{job="meter-api"} =>
0 @[1791306360]
0.0001258397890776202 @[1791306420]
0.00014402456396671407 @[1791306480]
0.00020690924976805424 @[1791306540]
0.009496889557716248 @[1791306600]
0.02617204241279303 @[1791306660]
0.04302056588532482 @[1791306720]
0.06019226996900255 @[1791306780]
0.07581968560062544 @[1791306840]
0.08114435362272837 @[1791306900]
0.08166018569954767 @[1791306960]

The service was failing 10% of its readings from 17:09 on, but a 5-minute rate needed until about 17:14 to show it (0.0095 at 17:10, 0.08 at 17:15). Every rate-based alert inherits that delay; the burn-rate alerts page shows how to balance speed against noise.

Comparison operators filter: … > 0.05 keeps only series whose value is above 0.05 and drops the rest — this is how an alerting rule's expression decides what fires. Add bool (… > bool 0.05) to get 1 or 0 instead.

Vector matching: on, ignoring and group_left

Pairing by identical labels fails when the two sides have different labels. "Which share of portal traffic goes to each route?" divides a per-route rate by a total without a route label — and returns nothing (the empty line), because no pair matches:

$ promtool query instant http://localhost:9090 'sum by (route) (rate(http_requests_total{job="portal"}[5m])) / sum(rate(http_requests_total{job="portal"}[5m]))'
$ promtool query instant http://localhost:9090 'sum by (route) (rate(http_requests_total{job="portal"}[5m])) / ignoring (route) group_left sum(rate(http_requests_total{job="portal"}[5m]))'

{route="/account"} => 0.3516917102351942 @[1791306533.462]
{route="/invoices"} => 0.19508723881753293 @[1791306533.462]
{route="/invoices/{invoice_id}/pdf"} => 0.06243476848198267 @[1791306533.462]
{route="/login"} => 0.3907862824652902 @[1791306533.462]

ignoring (route) tells Prometheus to leave route out when pairing (on (labels) does the opposite: pair only on the listed labels), and group_left allows many series on the left to share one series on the right. The second query does that: 39% of portal requests are logins. Without group_left, a match of many to one is refused, and many to many is never allowed:

$ promtool query instant http://localhost:9090 'http_requests_total{job="meter-api"} / on (job) http_requests_total{job="meter-api"}'

query error: execution: found duplicate series for the match group {job="meter-api"} on the right hand-side of the operation: [{__name__="http_requests_total", code="200", env="production", instance="meter-api-02:8080", job="meter-api", method="GET", route="/healthz", team="metering"}, {__name__="http_requests_total", code="200", env="production", instance="meter-api-02:8080", job="meter-api", method="GET", route="/v1/meters/{meter_id}/readings", team="metering"}];many-to-many matching not allowed: matching labels must be unique on one side

Division by zero is not an error either: it returns +Inf (or NaN for 0/0). Comparing the 5xx rate now with ten minutes earlier, when there were no 5xx responses:

$ promtool query instant http://localhost:9090 'sum(rate(http_requests_total{job="meter-api",code=~"5.."}[1m])) / sum(rate(http_requests_total{job="meter-api",code=~"5.."}[1m] offset 10m))'

{} => +Inf @[1791306945.937]

Percentiles: histogram_quantile()

Latency is measured with histograms: counters of how many requests fell into each duration bucket (le, "less than or equal"), cumulative, plus _sum and _count; see Metrics. histogram_quantile() estimates a quantile, such as the 95th percentile, from such buckets: it finds the bucket in which the requested rank falls and interpolates linearly within it. The recipe for a classic histogram is always the same — rate of the buckets, sum by le plus the labels you want to keep, then the quantile:

$ promtool query instant http://localhost:9090 'histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="portal"}[5m])))'
$ promtool query instant http://localhost:9090 'histogram_quantile(0.95, sum by (route, le) (rate(http_request_duration_seconds_bucket{job="portal"}[5m])))'
$ promtool query instant http://localhost:9090 'sum(rate(http_request_duration_seconds_sum{job="portal"}[5m])) / sum(rate(http_request_duration_seconds_count{job="portal"}[5m]))'

{} => 0.3909934497816592 @[1791306532.932]
{route="/account"} => 0.2355757575757576 @[1791306533]
{route="/invoices"} => 0.36451612903225783 @[1791306533]
{route="/invoices/{invoice_id}/pdf"} => 0.9635135135135133 @[1791306533]
{route="/login"} => 0.2337923728813559 @[1791306533]
{} => 0.12238675623800384 @[1791306533.063]

Before the incident, 95% of portal requests took at most about 0.391 seconds, while the mean — _sum divided by _count — was 0.122 seconds. The mean hides the slow tail; the per-route query shows where it comes from: the PDF download of an invoice, with a p95 near one second.

During the incident, the portal's p95 rose to 2.155 seconds:

$ promtool query range --start=2026-10-06T17:06:00Z --end=2026-10-06T17:16:00Z --step=2m http://localhost:9090 'histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="portal"}[5m])))'

{} =>
0.3908953227691752 @[1791306360]
0.3897239333102146 @[1791306480]
0.8188247011952179 @[1791306600]
1.7207677165354307 @[1791306720]
2.1553592814371245 @[1791306840]
2.1419863861386133 @[1791306960]

Read such numbers as estimates. The portal's buckets end at 1, 2.5, 5 and 10 seconds; 2.155 lies inside the bucket from 1 to 2.5 seconds, and the precise value is an interpolation, not a measured request. If the requested rank falls in the highest bucket (+Inf), the function returns the upper bound of the highest finite bucket, so the estimate can never be larger than your largest bucket boundary. Put a boundary at the thresholds you care about, such as an objective of 0.3 seconds.

Never average percentiles

The bucket counts of web01 and web02 can be added; their percentiles cannot. This script implements the same interpolation as Prometheus and compares the right and the wrong way to get one p95 for two servers — a busy, fast web01 and a quiet web02 with one slow route:

INF = float("inf")

def histogram_quantile(q, buckets):
    """buckets: [(le, cumulative count), ...] sorted, last le is +Inf.
    Linear interpolation inside the bucket, as Prometheus does."""
    total = buckets[-1][1]
    rank = q * total
    for i, (le, count) in enumerate(buckets[:-1]):
        if count >= rank:
            break
    else:
        return buckets[-2][0]              # rank falls in +Inf: upper bound of the last finite bucket
    lower_le, lower_count = (0.0, 0) if i == 0 else buckets[i - 1]
    return lower_le + (le - lower_le) * (rank - lower_count) / (count - lower_count)

BOUNDS = [0.1, 0.25, 0.5, 1.0, 2.5, INF]
# Requests per second in each bucket, as rate(..._bucket[5m]) would give them.
web01 = list(zip(BOUNDS, [38.0, 46.0, 48.5, 49.6, 50.0, 50.0]))   # busy and fast
web02 = list(zip(BOUNDS, [4.0, 6.0, 7.0, 8.5, 9.8, 10.0]))        # quiet, one slow route

p95_web01 = histogram_quantile(0.95, web01)
p95_web02 = histogram_quantile(0.95, web02)
merged = [(le, a + b) for (le, a), (_, b) in zip(web01, web02)]  # sum by (le)

print(f"p95 web01                     {p95_web01:.3f} s")
print(f"p95 web02                     {p95_web02:.3f} s")
print(f"average of the two p95s       {(p95_web01 + p95_web02) / 2:.3f} s   (wrong)")
print(f"p95 of the summed buckets     {histogram_quantile(0.95, merged):.3f} s")
print(f"p99.9 of the summed buckets   {histogram_quantile(0.999, merged):.3f} s   (capped)")

p95 web01                     0.400 s
p95 web02                     2.154 s
average of the two p95s       1.277 s   (wrong)
p95 of the summed buckets     0.788 s
p99.9 of the summed buckets   2.500 s   (capped)

The average of the two p95s (1.277 s) treats web02's ten requests per second as if they weighed as much as web01's fifty. Summing the buckets first (sum by (le)) gives the p95 over all requests: 0.788 s. The last line shows the cap at the highest finite bucket. The statistics page explains why tail latency deserves this care. A few more histogram queries:

# Classic histogram: keep le in the by clause
histogram_quantile(0.95, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))

# Native histogram: one series per histogram, no _bucket suffix and no le
histogram_quantile(0.95, sum by (job) (rate(http_request_duration_seconds[5m])))

# Share of portal requests faster than 0.5 s (a latency SLI); note le="0.5"
sum(rate(http_request_duration_seconds_bucket{job="portal", le="0.5"}[5m]))
  / sum(rate(http_request_duration_seconds_count{job="portal"}[5m]))

The second form is for native histograms, which keep all buckets in one series: no _bucket suffix and no le label to keep. The third is often more useful than a percentile: it says directly which share of requests met a latency target — the basis of a latency service level indicator.

Other functions you will use

absent() answers a question no other query can: "is this series missing?". It returns nothing while the series exists, and a single series with value 1 when it does not — which is what an alert on a silently vanished job or a never-pushed batch result needs:

# Nothing while the billing job has an up series; {job="billing"} 1 when it has none
absent(up{job="billing"})

# The same for the timestamp the nightly export pushes: 1 only if it was never pushed or has gone
absent(billing_export_last_success_timestamp_seconds{job="billing"})

In the test setup, the first query returned an empty result, because the billing job exists. The same query for a job name that does not exist returned one series with the value 1 and a job label holding that name: absent() copies the labels it can derive from the = matchers, so an alert built on it still knows what is missing. A few more functions that appear in Kestrel's rules and dashboards:

# Hours since the last successful billing export (a gauge holding a Unix time)
(time() - billing_export_last_success_timestamp_seconds) / 3600

# Will the readings queue reach its limit within an hour, at the trend of the last 15 minutes?
predict_linear(meter_readings_queue_depth[15m], 3600) > meter_readings_queue_limit

# Per-second rate of the last two samples only: for graphs of spiky counters, not for alerts
irate(http_requests_total{job="meter-api", route="/v1/readings"}[1m])

# Clamp a ratio to the range 0..1 before showing it as a percentage
clamp(job:http_requests_errors:ratio_rate5m, 0, 1)

• time() is the evaluation time in Unix seconds; subtracting a timestamp metric gives an age.
• predict_linear() fits a straight line through a gauge's samples (linear regression) and extrapolates it; the documentation says it should only be used with gauges. "Disk full in four hours" is the classic use.
• label_replace() creates or rewrites a label with a regular expression. It solves the next problem.

Arithmetic drops the metric name, because the result is no longer that metric. Dividing four differently named recording-rule series by the same number therefore produces four series with identical labels, which Prometheus refuses. Copying part of the name into a label first keeps them apart (these figures were recorded with only 12 minutes of history, which is why the 30m, 1h and 6h windows agree):

$ promtool query instant http://localhost:9090 '{__name__=~"job:slo_errors_per_request:ratio_rate.*"} / 0.001'
$ promtool query instant http://localhost:9090 'label_replace({__name__=~"job:slo_errors_per_request:ratio_rate.*"}, "window", "$1", "__name__", ".*_rate(.+)") / 0.001'

query error: execution: vector cannot contain metrics with the same labelset
{job="meter-api", window="5m"} => 81.78246894964485 @[1791306945.873]
{job="meter-api", window="30m"} => 42.66863068151602 @[1791306945.873]
{job="meter-api", window="1h"} => 42.66863068151602 @[1791306945.873]
{job="meter-api", window="6h"} => 42.66863068151602 @[1791306945.873]

Recording rules

A recording rule makes Prometheus evaluate a PromQL expression at a regular interval and store the result as a new time series under a name you choose. Queries, dashboards and alerts then read that cheap, precomputed series instead of recalculating an expensive expression every time. Recording rules also give an expression one official definition: the error ratio that the dashboard shows and the one the alert fires on are guaranteed to be the same.

Rules live in YAML files listed under rule_files in prometheus.yml, in named groups. The general-purpose recording rules of the Kestrel test setup are in one file (the SLI rules in rules/slo-meter-api.yml are on SLIs, SLOs and SLAs):

rules/recording.yml

groups:
  - name: kestrel-http
    interval: 15s
    rules:
      - record: job_route:http_requests:rate5m
        expr: sum by (job, route) (rate(http_requests_total[5m]))

      - record: job:http_requests_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_request_duration_seconds:p95_5m
        expr: |
          histogram_quantile(0.95,
            sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
          )

The names follow the convention from the Prometheus documentation, level:metric:operations:

• level — the labels that remain after aggregation: job, or job_route for job and route.
• metric — the metric name, without _total once a rate has been taken; for a ratio the two metrics joined with _per_, or a descriptive name.
• operations — what was done, newest first: rate5m, ratio_rate5m, p95_5m.

A reader who sees job:http_requests_errors:ratio_rate5m in an alert knows, without opening the rule file, that it is a ratio of 5-minute rates aggregated per job. Check the files with promtool before loading them:

$ promtool check rules rules/*.yml

Checking rules/alerts.yml
  SUCCESS: 7 rules found

Checking rules/recording.yml
  SUCCESS: 3 rules found

Checking rules/slo-meter-api.yml
  SUCCESS: 6 rules found

Once loaded, the new series can be queried like any other — here during the incident, with the portal's p95 above two seconds:

$ promtool query instant http://localhost:9090 'job_route:http_requests:rate5m'
$ promtool query instant http://localhost:9090 'job:http_request_duration_seconds:p95_5m'

job_route:http_requests:rate5m{job="meter-api", route="/healthz"} => 0.39999999999999997 @[1791306971.087]
job_route:http_requests:rate5m{job="meter-api", route="/v1/meters/{meter_id}/readings"} => 11.996491228070175 @[1791306971.087]
job_route:http_requests:rate5m{job="meter-api", route="/v1/readings"} => 76.03508771929823 @[1791306971.087]
job_route:http_requests:rate5m{job="portal", route="/account"} => 8.992977061988302 @[1791306971.087]
job_route:http_requests:rate5m{job="portal", route="/invoices"} => 5.003508771929824 @[1791306971.087]
job_route:http_requests:rate5m{job="portal", route="/invoices/{invoice_id}/pdf"} => 1.5999999999999999 @[1791306971.087]
job_route:http_requests:rate5m{job="portal", route="/login"} => 10 @[1791306971.087]
job:http_request_duration_seconds:p95_5m{job="meter-api"} => 0.4098674067221705 @[1791306971.153]
job:http_request_duration_seconds:p95_5m{job="portal"} => 2.1419863861386133 @[1791306971.153]

One trap: promtool check rules reports a recording rule that appears twice in a group as a lint error, prints FAILED — and still exits with status 0. Only --lint-fatal turns it into a failure (exit status 3) that stops a pipeline:

$ promtool check rules name-rules.yml
$ promtool check rules --lint-fatal name-rules.yml
$ echo $?

Checking name-rules.yml
  FAILED:
lint error 1 duplicate rule(s) found.
Metric: job_route:http_requests:rate5m
Label(s):
Might cause inconsistency while recording expressions
Checking name-rules.yml
  FAILED:
lint error 1 duplicate rule(s) found.
Metric: job_route:http_requests:rate5m
Label(s):
Might cause inconsistency while recording expressions
3

Reading PromQL errors

Parse errors name the position (line:column) and the problem. Six recorded errors, one per query, in the order of the commands:

$ promtool query instant http://localhost:9090 'sum(rate(http_requests_total[5m])'
$ promtool query instant http://localhost:9090 'rate(sum(http_requests_total)[5m])'
$ promtool query instant http://localhost:9090 'rate(http_requests_total)'
$ promtool query instant http://localhost:9090 'rate(http_requests_total{job="meter-api"}[5x])'
$ promtool query instant http://localhost:9090 'percentile(0.95, http_request_duration_seconds)'
$ promtool query instant http://localhost:9090 '{job=~".*"}'

query error: bad_data: invalid parameter "query": 1:34: parse error: unclosed left parenthesis
query error: bad_data: invalid parameter "query": 1:30: parse error: ranges only allowed for vector selectors
query error: bad_data: invalid parameter "query": 1:6: parse error: expected type range vector in call to function "rate", got instant vector
query error: bad_data: invalid parameter "query": 1:43: parse error: bad number or duration syntax: "5"
query error: bad_data: invalid parameter "query": 1:1: parse error: unknown function with name "percentile"
query error: bad_data: invalid parameter "query": 1:1: parse error: vector selector must contain at least one non-empty matcher

For long expressions, promtool can format a query into an indented layout that shows its structure. The command is still experimental and refuses to run without the --experimental flag, which comes before the subcommand:

$ promtool promql format 'histogram_quantile(0.95,sum by(le,route)(rate(http_request_duration_seconds_bucket{job="portal",route!="/login"}[5m])))'
$ promtool --experimental promql format 'sum by(job)(rate(http_requests_total{job="meter-api",code=~"5.."}[5m]))/sum by(job)(rate(http_requests_total{job="meter-api"}[5m]))>0.05'

This command is experimental and requires the --experimental flag to be set.
    sum by (job) (rate(http_requests_total{code=~"5..",job="meter-api"}[5m]))
  /
    sum by (job) (rate(http_requests_total{job="meter-api"}[5m]))
>
  0.05

The formatter also sorts the label matchers (code before job), which does not change the meaning. When a query is correct but slow, the cause is nearly always the number of series it touches: narrow the selector with more matchers, use a shorter range, or put the expression in a recording rule.