Dashboards

What dashboards are for

A dashboard is a page of graphs, numbers and tables that answer the standing questions about one service or system, laid out so that a person can read them in a few seconds. The general idea is introduced in Monitoring, Logging and Observability; this page is about building them well.

A dashboard has two jobs. During an incident it is a diagnostic tool: which part is slow, since when, and does it line up with the deploy at 14:10? In calmer times it gives context: how much traffic is normal on a Monday, how fast the database grows, how much of the error budget is left. What a dashboard should not be is the detection mechanism. Google's Site Reliability Engineering (SRE) book puts it sharply: SRE teams "carefully avoid any situation that requires someone to 'stare at a screen to watch for problems'". People get tired, look away and miss slow changes; an alert rule does not. Detection belongs to alerting, and a good alert links to the dashboard that explains it.

At Kestrel Energy the network operations center (NOC) has a wall screen. Tomasz Nowak keeps it, but he uses it as a shared picture for the room, not as the thing that wakes people up. The pager is driven by alert rules; the wall shows where to look next.

Grafana and data sources

Grafana is the most widely used open-source tool for building dashboards on top of monitoring data. It stores no metrics itself: it sends queries to other systems and draws the results. As of October 2026 the current release is Grafana 13.2.3 (13.0 arrived in April 2026). Grafana Labs moved Grafana, Loki and Tempo from the Apache 2.0 license to the GNU Affero General Public License version 3 (AGPL-3.0) in 2021; running it unmodified for your own dashboards is not affected by that, but check with your legal team before you modify and offer it as a service to others.

A data source is a configured connection from Grafana to one storage back end: its type, address, credentials and a few query settings. Typical data sources in an observability stack are Prometheus for metrics, Loki or Elasticsearch for logs, a trace store such as Tempo or Jaeger for traces, and SQL databases or cloud monitoring services for everything else. Each panel queries one data source in that source's own language: PromQL for Prometheus, LogQL for Loki.

Data sources can be created by hand in the web interface, but a provisioning file is better, because it is repeatable and can be reviewed. Kestrel's Prometheus data source, in a file such as /etc/grafana/provisioning/datasources/prometheus.yaml (illustrative):

apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus-kestrel
    type: prometheus
    access: proxy
    url: http://monitor01:9090
    isDefault: true
    jsonData:
      timeInterval: 15s

timeInterval is the "Scrape interval" setting of the Prometheus data source. Set it to the interval Prometheus really uses (15 seconds at Kestrel): Grafana needs it to calculate $__rate_interval, as explained below. The uid is the stable name that dashboards use to refer to this data source.

Panels and visualizations

A panel is one rectangle on a Grafana dashboard: one or more queries against a data source, plus a visualization that decides how the results are drawn, with its own title, units and thresholds. The visualization is a per-panel choice; the same query can be drawn as a line graph, a single number or a table. The ones you will use most:

Two settings deserve attention on every panel. The unit (seconds, requests per second, percent, bytes) makes Grafana print "390 ms" instead of "0.39" and keeps axes honest. Prometheus metrics use base units, so latency arrives in seconds and a ratio as a number between 0 and 1; choose the unit that matches the data rather than multiplying in the query. Thresholds color a value or draw a line, for example at the latency target of a service level objective, so the reader sees at once what "too slow" means.

Heatmaps for latency

A heatmap shows a distribution over time: the horizontal axis is time, the vertical axis is a value range (for example latency buckets), and the color of each cell says how many observations fell into that range in that time slice. A percentile line tells you "p95 went from 0.4 to 2.1 seconds"; a heatmap shows how: did everything get slower, or did a second band of slow requests appear while most stayed fast? Two bands usually mean two populations, such as cache hits and misses or two routes with different work.

Prometheus histograms store cumulative bucket counters: le="0.25" counts every request up to 0.25 seconds, including those already counted in le="0.1". A heatmap cell needs the count inside one bucket, so Grafana subtracts neighbors (choose the query format "Heatmap" for this). The same calculation by hand, for the buckets meter-api exposed for POST /v1/readings:

# Cumulative buckets of meter-api POST /v1/readings, as scraped (le = upper bound in seconds).
cumulative = [
    ("0.005", 0), ("0.01", 24), ("0.025", 4935), ("0.05", 31013),
    ("0.1", 56677), ("0.25", 87503), ("0.5", 93614), ("1", 93824),
    ("2.5", 93825), ("5", 93825), ("10", 93825), ("+Inf", 93825),
]
total = cumulative[-1][1]

lower, previous = "0", 0
for upper, count in cumulative:
    in_bucket = count - previous  # what one heatmap cell shows
    if in_bucket:
        bar = "#" * round(40 * in_bucket / total)
        line = f"{lower:>5} – {upper:<5} s {in_bucket:>6}  {in_bucket / total:>6.2%}  {bar}"
        print(line.rstrip())
    lower, previous = upper, count

0.005 – 0.01  s     24   0.03%
 0.01 – 0.025 s   4911   5.23%  ##
0.025 – 0.05  s  26078  27.79%  ###########
 0.05 – 0.1   s  25664  27.35%  ###########
  0.1 – 0.25  s  30826  32.85%  #############
 0.25 – 0.5   s   6111   6.51%  ###
  0.5 – 1     s    210   0.22%
    1 – 2.5   s      1   0.00%

Each printed row is one cell of a single heatmap column; a heatmap draws such a column for every time step, using increase() per step instead of the totals since start-up. The query behind a latency heatmap for the portal:

sum by (le) (increase(http_request_duration_seconds_bucket{job="portal"}[$__rate_interval]))

(This query is marked as not checkable because $__rate_interval is a Grafana variable: Prometheus only sees the query after Grafana has replaced it with a duration.) The vertical resolution of a heatmap is only as good as the bucket boundaries chosen in the application; a heatmap cannot show detail between 0.1 and 0.25 seconds if there is no bucket in between.

The queries behind the panels

Every graph is a query evaluated many times. Grafana asks Prometheus for a range query: the same PromQL expression evaluated at evenly spaced points (the step) across the visible time range, one point per step. You can see exactly what a time-series panel would plot by running the range query yourself. The portal's p95 latency every two minutes around the start of the meter-api incident of 6 October 2026, from the test Prometheus:

promtool query range --start=2026-10-06T17:06:00Z --end=2026-10-06T17:16:00Z --step=2m http://localhost:9090 'histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="portal"}[5m])))'

{} =>
0.3908953227691752 @[1791306360]
0.3897239333102146 @[1791306480]
0.8188247011952179 @[1791306600]
1.7207677165354307 @[1791306720]
2.1553592814371245 @[1791306840]
2.1419863861386133 @[1791306960]

Six points, two minutes apart (the numbers after @ are Unix timestamps; 1791306360 is 17:06 UTC). A time-series panel would join them into a line that jumps from 0.39 to 2.16 seconds. Note that 2.155 seconds is not a measured request time: it is interpolated inside the bucket from 1 to 2.5 seconds, as histogram_quantile always does.

The same moment shows why latency panels need percentiles. Before the incident, the mean and the 95th percentile of the portal differed by a factor of three:

promtool query instant http://localhost:9090 'sum(rate(http_request_duration_seconds_sum{job="portal"}[5m])) / sum(rate(http_request_duration_seconds_count{job="portal"}[5m]))'
promtool query instant http://localhost:9090 'histogram_quantile(0.95, sum by (route, le) (rate(http_request_duration_seconds_bucket{job="portal"}[5m])))'

{} => 0.12238675623800384 @[1791306533.063]
{route="/account"} => 0.2355757575757576 @[1791306533]
{route="/invoices"} => 0.36451612903225783 @[1791306533]
{route="/invoices/{invoice_id}/pdf"} => 0.9635135135135133 @[1791306533]
{route="/login"} => 0.2337923728813559 @[1791306533]

A panel that showed only the mean (0.12 seconds) would say everything is fine. Split by route, the p95 shows that the invoice PDF route takes almost a second for one request in twenty. Why the mean hides this is explained on Metrics and in Statistics for IT.

Why $__rate_interval exists

rate() needs at least two samples inside its range, and is only reliable with several. Grafana chooses the step from the time range and the width of the panel, and exposes it as $__interval. A fixed range causes two different problems:

• Too short: with a 1-minute scrape interval, rate(x[1m]) often finds only one sample in the window and returns nothing, so the graph has gaps or is empty.
• Shorter than the step: zoom out to two days and each point is, say, 30 minutes apart. rate(x[5m]) then looks at 5 minutes out of every 30 and skips the rest. A short outage that falls in the gap never appears on the graph.

Grafana's Prometheus data source solves both with $__rate_interval, defined as the larger of $__interval plus one scrape interval, and four scrape intervals. The first part closes the gaps between steps; the second guarantees at least four samples.

def rate_interval(interval_s, scrape_s):
    """Grafana's $__rate_interval in seconds."""
    return max(interval_s + scrape_s, 4 * scrape_s)


def fmt(seconds):
    minutes, secs = divmod(seconds, 60)
    return f"{minutes}m{secs:02d}s" if minutes else f"{secs}s"


print(f"{'Scrape':>7}{'$__interval':>13}{'$__rate_interval':>18}")
for scrape in (15, 60):
    for interval in (5, 15, 60, 300, 1800):
        print(f"{fmt(scrape):>7}{fmt(interval):>13}{fmt(rate_interval(interval, scrape)):>18}")

 Scrape  $__interval  $__rate_interval
    15s           5s             1m00s
    15s          15s             1m00s
    15s        1m00s             1m15s
    15s        5m00s             5m15s
    15s       30m00s            30m15s
  1m00s           5s             4m00s
  1m00s          15s             4m00s
  1m00s        1m00s             4m00s
  1m00s        5m00s             6m00s
  1m00s       30m00s            31m00s

The effect of the gap is easy to simulate. Two hours of meter-api counters, sampled every 15 seconds, with a four-minute outage in which 20% of requests fail. A wide panel draws one point every 30 minutes:

SCRAPE = 15        # seconds between samples
STEP = 30 * 60     # a wide time range: one point per 30 minutes

# Two hours of meter-api counters, one sample every 15 seconds.
total, errors = [], []
count_total = count_errors = 0
for i in range(2 * 3600 // SCRAPE + 1):
    minute = i * SCRAPE / 60
    count_total += 100
    if 47 <= minute < 51:  # a four-minute outage: 20 of every 100 requests fail
        count_errors += 20
    total.append(count_total)
    errors.append(count_errors)


def error_ratio(at, window):
    """Share of failed requests in the window that ends at `at` seconds."""
    end = at // SCRAPE
    start = max(0, (at - window) // SCRAPE)
    return (errors[end] - errors[start]) / (total[end] - total[start])


print("Panel point at:           ", "   ".join(f"{t // 60:>3} min" for t in range(STEP, 7201, STEP)))
for window, label in ((300, "[5m]"), (STEP + SCRAPE, "[$__rate_interval]")):
    points = [error_ratio(at, window) for at in range(STEP, 7201, STEP)]
    print(f"{label:<26}", "   ".join(f"{p:>7.2%}" for p in points))

Panel point at:             30 min    60 min    90 min   120 min
[5m]                         0.00%     0.00%     0.00%     0.00%
[$__rate_interval]           0.00%     2.64%     0.00%     0.00%

With [5m] the outage between minute 47 and 51 is invisible: every point looks at a quiet five minutes. With $__rate_interval (30 minutes 15 seconds here) the point at 60 minutes covers the outage, diluted over half an hour to 2.64%. The graph is not precise at this zoom level, but it is not lying. Zoom in to see the real shape. In a panel query:

sum by (route) (rate(http_requests_total{job="meter-api"}[$__rate_interval]))

Dashboard variables

A dashboard variable is a named value that the reader picks from a drop-down list at the top of a dashboard, and that Grafana substitutes into every query before sending it. One dashboard then serves every instance, environment or service instead of one copy per host. The selected values also appear in the address bar (?var-instance=web01:8080), so a link in a ticket opens the dashboard with the same selection.

Grafana offers several variable types:

• Query: values come from the data source, for example all values of the label instance for job="portal". In current Grafana you choose the query type "Label values" in the editor; the classic label_values(metric, label) function is marked as deprecated.
• Custom (a fixed list such as production,acceptance), Text box (free input, for example a trace ID) and Constant (a hidden fixed value).
• Data source (switch between, say, two Prometheus servers) and Interval (let the reader choose a window such as 5m or 1h).
• Filters: dashboard-wide key/value filters added to every query of a data source (older documentation calls them ad hoc filters), and Switch, a toggle between two values.

In a query you refer to a variable as $instance or ${instance}; ${instance:<format>} chooses how multiple values are joined. The old [[instance]] syntax is deprecated and will be removed, so replace it when you meet it in an imported dashboard.

A variable that allows several values (or "All") turns into a regular expression such as (web01:8080|web02:8080) for Prometheus, so it must be used with the regex matcher =~:

sum by (instance) (rate(http_requests_total{job="portal", instance=~"$instance"}[$__rate_interval]))

Annotations: marking deploys and incidents

An annotation is a marker on the time axis of Grafana graphs: a vertical line or shaded region with a short text, such as "CHG-0714: meter-api 2.14.3 deployed". The first question in almost every incident is "what changed?", and an annotation answers it on the graph itself: if latency rises one minute after a deploy line, you know where to look.

Annotations come from two places:

• Queries: Grafana draws a marker wherever a query returns a value. Kestrel's services expose kestrel_build_info{version="2.14.3",commit="9f3c2ab"} 1, an "info" metric whose labels change when a new version starts, so a query on that metric can mark every rollout.
• The HTTP API: the deploy pipeline posts an annotation with the change number when it starts and finishes. This also works for things no metric records, like a firewall change by the NOC.

A deploy job could send this request (illustrative; the token is a Grafana service account token read from a file, never written into the pipeline definition):

curl -s -X POST https://grafana.example.com/api/annotations \
  -H @/run/secrets/grafana-annotations-auth-header \
  -H "Content-Type: application/json" \
  -d '{"tags": ["deploy", "meter-api", "CHG-0714"], "text": "CHG-0714: meter-api 2.14.3 deployed"}'

{"id":1874,"message":"Annotation added"}

The secret file holds the line Authorization: Bearer …, so the token never appears on a command line. Without a time in the request, Grafana uses the current time. Dashboards then show every annotation tagged deploy, or only those tagged with the service they are about. Yara Haddad, Kestrel's release manager, likes this because the change record, the pipeline and the graphs all carry the same CHG number.

A golden-signals layout

Most good service dashboards follow the same reading order, from "are users OK?" to "why not?". The four golden signals (latency, traffic, errors and saturation) give the middle part its structure; the SRE book notes that dashboards normally include some form of them. Kestrel's "meter-api overview":

1. Service level first. Stat panels for the service level indicator over the objective's window, the error budget left and the current burn rate. If these are fine, users are fine, whatever the rest says.
2. Traffic, errors, latency side by side, with the same time axis, so a change in one can be compared with the others at a glance.
3. Saturation: the resources that run out first. For meter-api that is the readings queue and the database connection pool on db01, not the CPU.
4. Drill-down: a logs panel filtered on the same service, and links (Grafana calls them data links) from a graph to the matching traces or to a more detailed dashboard.

The PromQL behind the middle rows, written without Grafana variables so it can be pasted into Prometheus as it is:

# Traffic: requests per second, by route
sum by (route) (rate(http_requests_total{job="meter-api"}[5m]))

# Errors: share of requests answered with a 5xx status (a recording rule)
job:http_requests_errors:ratio_rate5m{job="meter-api"}

# Latency: 95th and 99th percentile over all instances
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="meter-api"}[5m])))

histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="meter-api"}[5m])))

# Saturation: queue depth, and how much faster readings arrive than they are processed
sum(meter_readings_queue_depth{job="meter-api"})

sum(rate(meter_readings_received_total[5m])) - sum(rate(meter_readings_processed_total[5m]))

Recording rules such as job:http_requests_errors:ratio_rate5m (see PromQL) make dashboards faster: the expensive aggregation is computed once every evaluation interval instead of every time someone opens the page. The traffic query for the portal, as a panel would show its current values:

promtool query instant http://localhost:9090 'sum by (route) (rate(http_requests_total{job="portal"}[5m]))'

{route="/account"} => 8.996491228070173 @[1791306532.728]
{route="/invoices"} => 4.990332481871344 @[1791306532.728]
{route="/invoices/{invoice_id}/pdf"} => 1.5970873286549705 @[1791306532.728]
{route="/login"} => 9.996491803868645 @[1791306532.728]

Four series, one line each on the traffic panel: about 25 requests per second in total. Set the legend to {{route}} so the lines are labeled by route rather than by the full label set.

Dashboards as code

A dashboard built only by clicking lives in one Grafana database. Someone changes a query on Friday afternoon, the graph breaks, and nobody knows what it looked like before. Dashboards as code means keeping dashboard definitions as text files under version control, reviewed in pull requests and loaded into Grafana automatically, the same way infrastructure as code treats servers.

Grafana stores every dashboard as a JSON document (the "JSON model", visible under the dashboard settings and exportable). A trimmed panel from meter-api's dashboard (illustrative; the exact fields differ between Grafana versions):

{
  "type": "timeseries",
  "title": "Latency · p95 and p99",
  "datasource": { "type": "prometheus", "uid": "prometheus-kestrel" },
  "fieldConfig": { "defaults": { "unit": "s" } },
  "targets": [
    {
      "refId": "A",
      "expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job=\"meter-api\"}[$__rate_interval])))",
      "legendFormat": "p95"
    }
  ],
  "gridPos": { "h": 8, "w": 8, "x": 16, "y": 4 }
}

Three common ways to manage these files:

• File provisioning: Grafana reads dashboards from a folder at start-up and keeps them in sync. The folder is filled from Git by the deployment.
• Terraform or OpenTofu with the Grafana provider: dashboards, folders, data sources and alert contact points as resources, planned and applied like the rest of the infrastructure (see Terraform and OpenTofu: First Steps).
• Generated JSON: libraries such as Grafonnet (Jsonnet) build dashboards from code, which pays off when twenty services need the same layout.

A provisioning file that loads every dashboard from one folder (illustrative):

apiVersion: 1
providers:
  - name: kestrel-dashboards
    folder: Kestrel
    type: file
    disableDeletion: true
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards/kestrel

With allowUiUpdates: false, changes made in the web interface cannot be saved over the provisioned version: people experiment in a copy and propose the change in Git. That feels strict at first, but it is the only way to know that the dashboard in production is the one that was reviewed.

Habits that make dashboards misleading

A dashboard can be correct in every number and still lead people to the wrong conclusion. The usual causes:

• Averages instead of percentiles. Shown above: a mean of 0.12 seconds next to a p95 of almost 1 second on one route.
• Mixed units. One panel in milliseconds, the next in seconds; one ratio as 0–1, another as 0–100. Set the unit on every panel.
• A truncated y-axis. Grafana scales the axis to the data by default, which can turn a tiny change into a cliff (see the figure below). For rates and counts, set the minimum to zero. When you deliberately zoom in, for example on an availability between 99.8% and 100%, draw the objective as a threshold line so the scale is obvious.
• Fixed rate windows. [1m] with a 1-minute scrape interval gives empty graphs; [5m] on a two-day view skips most of the data. Use $__rate_interval.
• Time zones. promtool and the HTTP API give Unix timestamps, amtool and Alertmanager's notifications mostly show UTC, the Prometheus and Alertmanager logs use the server's local time, Grafana shows the browser's local time by default, and the incident channel speaks Dutch time. Agree on one zone for incident work and show it on the dashboard.
• Too many panels, no owner. Fifty panels nobody reads hide the five that matter. Delete panels that have not helped in a real investigation.
• Pie charts of time-based data. A pie of "errors per route" hides when they happened; a stacked time series or a table sorted by count shows it.