Prometheus

What Prometheus is

Prometheus is an open-source monitoring system that collects numeric metrics by fetching them from the systems it monitors, stores them as time series on its own disk, evaluates rules over them and answers queries in its query language, PromQL. It joined the Cloud Native Computing Foundation (CNCF) in 2016 and graduated in 2018. It is licensed under Apache 2.0.

As of October 2026 the newest release is Prometheus 3.15.0 (25 September 2026). A new minor version appears every six weeks; for slower upgrades there is a long-term support (LTS) line: 3.13, supported until 31 July 2027. The previous LTS, 3.5, reached its end of life in July 2026.

A Prometheus setup is a few cooperating programs:

• The Prometheus server scrapes, stores, evaluates recording and alerting rules and serves queries — in its own web interface on port 9090, to Grafana (Dashboards) and to scripts through its HTTP API.
• Exporters expose metrics for systems that do not do it themselves.
• Alertmanager receives firing alerts from Prometheus and groups, routes and silences them (see Alerting).
• The Pushgateway holds metrics pushed by batch jobs until Prometheus scrapes them.

Prometheus is built for numbers that describe the health of systems: request rates, error ratios, latencies, queue depths, free disk space. It is not a log store, not an event database and not a place for per-customer data — a label with 400,000 meter IDs creates 400,000 time series. Use logs and traces for those.

The examples on this page were recorded on monitor01, a small Prometheus server in Kestrel Energy's test setup that scrapes meter-api, the customer portal, db01 and the billing job. All metrics come from small simulated exporters, so the numbers are invented; the Prometheus behavior is real.

The pull model: targets and scrapes

In the pull model, the monitoring system fetches metrics from the monitored systems at a regular interval, instead of the systems sending (pushing) their data to it. Every system Prometheus fetches from is a target: one URL, usually http://host:port/metrics. Targets with the same purpose form a job — the job portal has the targets web01, web02 and web03.

A scrape is one such fetch: an HTTP GET of a target's metrics endpoint, after which Prometheus parses the text and stores one sample per series, all with the same timestamp. Kestrel scrapes every 15 seconds (the built-in default is one minute; the example configuration shipped with Prometheus uses 15 seconds). Each stored series gets two labels from the scrape itself: job (the job name) and instance (the target's host and port).

Why pull?

Pulling has practical advantages for operations:

• A dead target is visible. If a scrape fails, Prometheus knows immediately. With push, silence could mean "healthy and quiet" or "dead".
• Prometheus controls the load. A misbehaving application cannot flood it with data; the scrape interval and limits decide how much comes in.
• Easy to inspect and duplicate. You can open a target's /metrics in a browser or with curl, and a second Prometheus can scrape the same targets for redundancy or testing without changing them.

The price: Prometheus must be able to reach every target. Across a firewall or network address translation (NAT) boundary, the usual advice is to run a Prometheus server on the same network as the targets rather than to switch to pushing.

The up series

For every target and every scrape, Prometheus writes a series called up: 1 if the scrape succeeded, 0 if it failed. It is the cheapest and most important health signal there is. In the test setup, web03:8080 is in the configuration but nothing listens on it:

$ promtool query instant http://localhost:9090 'up'

up{env="production", instance="web02:8080", job="portal", team="portal"} => 1 @[1791306109.103]
up{instance="localhost:9093", job="alertmanager"} => 1 @[1791306109.103]
up{env="production", instance="meter-api-02:8080", job="meter-api", team="metering"} => 1 @[1791306109.103]
up{instance="localhost:9090", job="prometheus"} => 1 @[1791306109.103]
up{env="production", instance="web01:8080", job="portal", team="portal"} => 1 @[1791306109.103]
up{env="production", instance="billing01", job="billing", team="billing"} => 1 @[1791306109.103]
up{env="production", instance="db01", job="postgres", team="platform"} => 1 @[1791306109.103]
up{env="production", instance="web03:8080", job="portal", team="portal"} => 0 @[1791306109.103]
up{env="production", instance="meter-api-01:8080", job="meter-api", team="metering"} => 1 @[1791306109.103]

The number after @[ is the evaluation time as a Unix timestamp; the order of the lines is not sorted. The target health page in Prometheus' web interface (under Status) shows the same, with the error message. The data behind it comes from the HTTP API, which is handy for scripts:

$ curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | [.labels.job, .labels.instance, .health, .lastError] | @tsv'

alertmanager	localhost:9093	up	
billing	billing01	up	
meter-api	meter-api-01:8080	up	
meter-api	meter-api-02:8080	up	
portal	web02:8080	up	
portal	web03:8080	down	Get "http://web03:8080/metrics": dial tcp 192.0.2.13:8080: connect: connection refused
portal	web01:8080	up	
postgres	db01	up	
prometheus	localhost:9090	up	

"Connection refused" means the host answered but no process listens on the port: the service is down or listens elsewhere. A timeout would point to a firewall or a host that is gone. Without jq, a few lines of Python do the same with a saved API answer:

import json

# Part of a saved answer from GET /api/v1/targets (fields trimmed).
answer = json.loads("""
{"status": "success", "data": {"activeTargets": [
 {"labels": {"job": "portal", "instance": "web01:8080"}, "health": "up", "lastError": ""},
 {"labels": {"job": "portal", "instance": "web02:8080"}, "health": "up", "lastError": ""},
 {"labels": {"job": "portal", "instance": "web03:8080"}, "health": "down",
  "lastError": "Get \\"http://web03:8080/metrics\\": dial tcp 192.0.2.13:8080: connect: connection refused"},
 {"labels": {"job": "postgres", "instance": "db01"}, "health": "up", "lastError": ""},
 {"labels": {"job": "billing", "instance": "billing01"}, "health": "up", "lastError": ""}
]}}
""")

targets = answer["data"]["activeTargets"]
per_job = {}
for target in targets:
    up, total = per_job.get(target["labels"]["job"], (0, 0))
    per_job[target["labels"]["job"]] = (up + (target["health"] == "up"), total + 1)

for job in sorted(per_job):
    up, total = per_job[job]
    print(f"{job:<9} {up}/{total} up")
for target in targets:
    if target["health"] != "up":
        print(f"DOWN {target['labels']['instance']}: {target['lastError']}")

billing   1/1 up
portal    2/3 up
postgres  1/1 up
DOWN web03:8080: Get "http://web03:8080/metrics": dial tcp 192.0.2.13:8080: connect: connection refused

Besides up, every scrape produces a few more series about itself, such as scrape_duration_seconds and scrape_samples_scraped. A scrape duration that creeps toward the scrape timeout (10 seconds by default) is an early warning.

Exporters and the exposition format

Applications that Kestrel writes itself, such as meter-api and the portal, use a Prometheus client library to serve their own /metrics. Everything else needs an exporter. An exporter is a small program that collects metrics from a system that does not speak Prometheus — a database, a Linux host, a network device, a message queue — and exposes them on an HTTP endpoint for Prometheus to scrape. Well-known ones are node_exporter for Linux hosts (version 1.12.1 as of October 2026), postgres_exporter, blackbox_exporter (probes HTTP, TCP, DNS and ICMP from the outside) and snmp_exporter for network devices that speak SNMP.

Exporters and Prometheus components have their own conventional ports, which is useful to recognize in firewall rules:

The exposition format

The exposition format is the plain-text format in which a target presents its metrics to Prometheus: per metric a # HELP line with a description, a # TYPE line with the metric type, and one line per series with the metric name, labels in curly braces and the current value. This is what the database exporter on db01 returns, preceded by the HTTP headers of a scrape of meter-api:

$ curl -si http://meter-api-01:8080/metrics | head -5
$ curl -s http://db01:9187/metrics

HTTP/1.0 200 OK
Server: kestrel-exporter 
Date: Tue, 06 Oct 2026 17:03:29 GMT
Content-Type: text/plain; version=0.0.4; charset=utf-8
Content-Length: 5289
# HELP pg_connections_in_use Client connections currently open on the PostgreSQL server.
# TYPE pg_connections_in_use gauge
pg_connections_in_use 63
# HELP pg_connections_max Value of the max_connections setting.
# TYPE pg_connections_max gauge
pg_connections_max 200
# HELP pg_database_size_bytes Disk space used by the database.
# TYPE pg_database_size_bytes gauge
pg_database_size_bytes{datname="kestrel"} 48319284244
# HELP pg_volume_size_bytes Size of the volume that holds the PostgreSQL data directory.
# TYPE pg_volume_size_bytes gauge
pg_volume_size_bytes 107374182400

• Content-Type: text/plain; version=0.0.4 marks the classic Prometheus text format. Prometheus also understands OpenMetrics 1.0 (a standardized variant of the same text) and a binary Protocol Buffers format. OpenMetrics 2.0 exists as a release candidate and is still marked experimental; Prometheus 3.15 can already scrape it.
• Names use underscores and end in their unit (_bytes, _seconds); counters end in _total. The four metric types (counter, gauge, histogram, summary) are explained on the Metrics page.
• Since Prometheus 3.0, names may contain any UTF-8 character, such as the dots of OpenTelemetry names. Underscores remain the convention, and tutorials that say names may only contain letters, digits, underscores and colons describe Prometheus 2.

The format is simple enough to write by hand, which is why a few lines in any language make an exporter. The rules that catch people out are in label values — a backslash, a double quote and a line feed must be escaped — and the last line must end with a line feed too:

def escape_label(value):
    """Label values escape backslash, double quote and line feed."""
    return value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n")

def render(name, kind, help_text, samples):
    lines = [f"# HELP {name} {help_text}", f"# TYPE {name} {kind}"]
    for labels, value in samples:
        pairs = ",".join(f'{key}="{escape_label(val)}"' for key, val in labels.items())
        lines.append(f"{name}{{{pairs}}} {value}" if pairs else f"{name} {value}")
    return "\n".join(lines) + "\n"   # the last line must end with a line feed too

text = render(
    "billing_export_files", "gauge",
    "Files written by the last billing export run, by target share.",
    [({"share": "\\\\fs01\\exports", "result": "ok"}, 3),
     ({"share": "\\\\fs02\\exports", "result": "failed"}, 0)],
)
text += render("billing_export_last_success_timestamp_seconds", "gauge",
               "Unix time of the last successful nightly billing export.",
               [({}, 1791247297)])
print(text, end="")
print("ends with a line feed:", text.endswith("\n"))

# HELP billing_export_files Files written by the last billing export run, by target share.
# TYPE billing_export_files gauge
billing_export_files{share="\\\\fs01\\exports",result="ok"} 3
billing_export_files{share="\\\\fs02\\exports",result="failed"} 0
# HELP billing_export_last_success_timestamp_seconds Unix time of the last successful nightly billing export.
# TYPE billing_export_last_success_timestamp_seconds gauge
billing_export_last_success_timestamp_seconds 1791247297
ends with a line feed: True

Configuring Prometheus

Prometheus reads one YAML file, by convention prometheus.yml. This is the file of the test setup, used for every recording in this topic:

prometheus.yml

# Prometheus configuration for Kestrel Energy
global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s

storage:
  tsdb:
    retention:
      time: 30d
      size: 40GB

rule_files:
  - rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: alertmanager
    static_configs:
      - targets: ["localhost:9093"]

  - job_name: meter-api
    static_configs:
      - targets: ["meter-api-01:8080", "meter-api-02:8080"]
        labels:
          env: production
          team: metering
    metric_relabel_configs:
      # Debug-only metrics are not worth storing.
      - source_labels: [__name__]
        regex: "meter_api_debug_.*"
        action: drop

  - job_name: portal
    static_configs:
      - targets: ["web01:8080", "web02:8080", "web03:8080"]
        labels:
          env: production
          team: portal

  - job_name: postgres
    file_sd_configs:
      - files: ["targets/postgres.json"]
        refresh_interval: 1m
    relabel_configs:
      # Use the host name from the target file as instance, without the port.
      - source_labels: [hostname]
        target_label: instance
      - action: labeldrop
        regex: hostname

  - job_name: billing
    file_sd_configs:
      - files: ["targets/billing.json"]
    relabel_configs:
      - source_labels: [hostname]
        target_label: instance
      - action: labeldrop
        regex: hostname

• global sets defaults: how often to scrape, how long a scrape may take, and how often rules are evaluated. Without this block the defaults are 1 minute, 10 seconds and 1 minute.
• storage.tsdb.retention keeps data for 30 days or 40 GB, whichever limit is reached first (see Storage and retention).
• rule_files loads recording and alerting rules (recording rules, Alerting). alerting.alertmanagers says where firing alerts go.
• scrape_configs has one entry per job. Labels under static_configs (here env and team) are added to every series of those targets; Prometheus also scrapes itself and Alertmanager.

In the test setup meter-api runs as two instances, meter-api-01 and meter-api-02; other pages refer to their host as api01.

Checking the configuration

promtool, which ships with Prometheus, checks the file and every rule file it refers to:

$ promtool check config prometheus.yml

Checking prometheus.yml
  SUCCESS: 3 rule files found
 SUCCESS: prometheus.yml is valid prometheus config file syntax

Checking rules/alerts.yml
  SUCCESS: 7 rules found

Checking rules/recording.yml
  SUCCESS: 3 rules found

Checking rules/slo-meter-api.yml
  SUCCESS: 6 rules found

Two typical mistakes, each on its own: a duration without a unit (scrape_interval: 15), and a job copied without changing its name. Both make promtool exit with status 1, which is what you want in a pipeline:

Checking prometheus.yml
  FAILED: parsing YAML file prometheus.yml: not a valid duration string: "15"

Checking prometheus.yml
  FAILED: parsing YAML file prometheus.yml: found multiple scrape configs with job name "meter-api"

Reloading without a restart

Prometheus rereads its configuration on SIGHUP (for example systemctl reload prometheus, if the unit defines it) or on an HTTP POST to /-/reload, which only works when Prometheus was started with --web.enable-lifecycle. Here someone wrote 15sec instead of 15s and reloaded without checking:

$ curl -s -X POST localhost:9090/-/reload

failed to reload config: couldn't load configuration (--config.file="prometheus.yml"): parsing YAML file prometheus.yml: unknown unit "sec" in duration "15sec"

The Prometheus log says the same:

time=2026-10-06T19:38:10.516+02:00 level=INFO source=main.go:1721 msg="Loading configuration file" filename=prometheus.yml
time=2026-10-06T19:38:10.516+02:00 level=ERROR source=main.go:1420 msg="Error reloading config" err="couldn't load configuration (--config.file=\"prometheus.yml\"): parsing YAML file prometheus.yml: unknown unit \"sec\" in duration \"15sec\""

Two things to notice. Prometheus keeps running with the old configuration, which is safe but easy to miss: the change you thought you made is not active. And curl exited with status 0 although the reload failed, because -s without -f does not turn an HTTP error into an exit code. Run promtool check config first, use curl -sf in scripts, and watch the metric prometheus_config_last_reload_successful, which drops to 0 after a failed reload — a good alert.

Relabeling

Relabeling rewrites labels with rules in the configuration. There are two places, at two moments:

• relabel_configs runs before the scrape, on the target's labels. It decides which targets are scraped at all and what their instance and other labels are. In the postgres job above it copies the hostname label from the target file into instance (so series say instance="db01" instead of db01:9187) and then drops the helper label.
• metric_relabel_configs runs after the scrape, on every scraped series. In the meter-api job it drops a debug metric that nobody needs.

The second rule in action: the exporter still offers the debug metric, Prometheus does not store it, and the difference between scraped and stored samples shows exactly one dropped sample per scrape per instance (the second command prints an empty line: no series):

$ curl -s http://meter-api-01:8080/metrics | grep debug
$ promtool query instant http://localhost:9090 'meter_api_debug_cache_entries'
$ promtool query instant http://localhost:9090 'scrape_samples_scraped{job="meter-api"} - scrape_samples_post_metric_relabeling{job="meter-api"}'

# HELP meter_api_debug_cache_entries Entries in the meter lookup cache (debug only).
# TYPE meter_api_debug_cache_entries gauge
meter_api_debug_cache_entries 18567

{env="production", instance="meter-api-01:8080", job="meter-api", team="metering"} => 1 @[1791308508.355]
{env="production", instance="meter-api-02:8080", job="meter-api", team="metering"} => 1 @[1791308508.355]

The API shows the labels of each target before and after relabeling as discoveredLabels and labels — the first place to look when a target has the wrong labels or does not appear at all:

$ curl -s 'localhost:9090/api/v1/targets' | python3 -m json.tool

                "discoveredLabels": {
                    "__address__": "db01:9187",
                    "__always_scrape_classic_histograms__": "false",
                    "__convert_classic_histograms_to_nhcb__": "false",
                    "__meta_filepath": "targets/postgres.json",
                    "__metrics_path__": "/metrics",
                    "__scheme__": "http",
                    "__scrape_interval__": "15s",
                    "__scrape_native_histograms__": "false",
                    "__scrape_timeout__": "10s",
                    "env": "production",
                    "hostname": "db01",
                    "job": "postgres",
                    "team": "platform"
                },
                "labels": {
                    "env": "production",
                    "instance": "db01",
                    "job": "postgres",
                    "team": "platform"
                },
…

Labels that start with __ are internal: they steer the scrape (__address__, __metrics_path__, __scheme__) or carry discovery metadata (__meta_…) and disappear after relabeling unless a rule copies them into a normal label.

Service discovery

Typing every target into static_configs works for a handful of servers. It breaks down when machines come and go with autoscaling or when pods move around a Kubernetes cluster. Service discovery is the automatic finding of targets from a source that already knows them — a file, DNS, a cloud provider's API, the Kubernetes API — so that new instances are scraped without anyone editing the configuration.

The simplest form is file-based discovery, which the postgres and billing jobs above use. Prometheus watches a JSON or YAML file and picks up changes without a reload, so a configuration-management tool such as Ansible or a cron job can write it:

targets/postgres.json

[
  {
    "targets": ["db01:9187"],
    "labels": {
      "hostname": "db01",
      "env": "production",
      "team": "platform"
    }
  }
]

Prometheus has built-in discovery for Kubernetes, the large cloud providers (Amazon EC2, Azure, Google Compute Engine and, since 3.14, Oracle Cloud Infrastructure), Consul, DNS SRV records, Docker and more. Each attaches __meta_… labels describing what it found, and relabeling turns those into a selection and useful labels. A common Kubernetes pattern (illustrative) scrapes only pods that ask for it with an annotation and copies their namespace and name into labels:

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Only pods that ask for it with the annotation prometheus.io/scrape: "true"
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        regex: "true"
        action: keep
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

The prometheus.io/scrape annotation is a widespread convention, not a Kubernetes feature: the keep rule is what gives it meaning. In clusters, Prometheus is often installed through the Prometheus Operator, which generates this configuration from ServiceMonitor and PodMonitor objects. The HorizontalPodAutoscaler is a different consumer of metrics: it reads them through the Kubernetes metrics APIs, not from Prometheus directly.

The Pushgateway and batch jobs

Kestrel's nightly billing export starts at 02:30, runs for about twelve minutes and exits. A scrape every 15 seconds would catch it while it runs, but its most important number — "when did the export last succeed?" — has to outlive the process. The Pushgateway is a small Prometheus component that accepts metrics pushed over HTTP by short-lived jobs and keeps exposing them until Prometheus scrapes them. At the end of a successful run the export could push its timestamp like this (illustrative):

$ cat <<'METRICS' | curl --data-binary @- http://pushgateway.example.com:9091/metrics/job/billing_export
# TYPE billing_export_last_success_timestamp_seconds gauge
billing_export_last_success_timestamp_seconds 1791247297
METRICS

An alert then checks the age of that timestamp; the Alerting page has the rule. In the test setup the export's metrics are scraped from a small exporter on billing01 instead; the alert is the same.

The Prometheus documentation is blunt about when to use the Pushgateway: usually only to capture the outcome of a service-level batch job — one that is not tied to a particular machine. It warns of three pitfalls when it is used more widely:

• It becomes a single point of failure and a bottleneck for every job that pushes through it.
• You lose the up series per instance: Prometheus only sees that the gateway is up, not whether the job behind it is.
• It never forgets: a series pushed once stays there, and is scraped again and again, until someone deletes it through the gateway's API.

For batch jobs that belong to one machine, such as a backup script or an update run on db01, the documentation recommends writing the metrics to a file that node_exporter's textfile collector exposes with the host's other metrics.

Storage and retention

Prometheus stores samples in its own time-series database (TSDB) in the directory given by --storage.tsdb.path. New samples are kept in memory in the head block and written to a write-ahead log (WAL) in 128 MB segments, so after a crash Prometheus replays the WAL and loses at most the last few seconds. Every two hours the head is cut into a persistent block on disk, and blocks are later compacted into larger ones. About 40 minutes after the start, the test server's data directory still has only the WAL and head chunks, and no persisted block yet:

$ ls -l data/ data/wal/ data/chunks_head/
$ promtool tsdb list data/

data/:
total 28
drwxr-xr-x 2 amara amara  4096 Oct  6 19:37 chunks_head
-rw-r--r-- 1 amara amara     0 Oct  6 19:03 lock
-rw-r--r-- 1 amara amara 20001 Oct  6 19:43 queries.active
drwxr-xr-x 2 amara amara  4096 Oct  6 19:03 wal

data/chunks_head/:
total 128
-rw-r--r-- 1 amara amara 131072 Oct  6 19:37 000001

data/wal/:
total 1824
-rw-r--r-- 1 amara amara 1861665 Oct  6 19:44 00000000
BLOCK ULID   MIN TIME     MAX TIME     DURATION     NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE

Retention

Without settings, Prometheus keeps 15 days of data. You can limit by time, by size or both; whichever limit is reached first wins, and expired blocks may take up to two hours to disappear. Since Prometheus 3.15 the command-line flags for retention are deprecated in favor of the storage.tsdb.retention block in prometheus.yml shown above, which can also be changed with a reload:

$ prometheus --help 2>&1 | grep -A12 -E -- '--storage.tsdb.path'

      --storage.tsdb.path="data/"  
                                 Base path for metrics storage. Use with server
                                 mode only.
      --storage.tsdb.retention.time=STORAGE.TSDB.RETENTION.TIME  
                                 [DEPRECATED] How long to retain samples
                                 in storage. If neither this flag nor
                                 "storage.tsdb.retention.size" is set,
                                 the retention time defaults to 15d.
                                 Units Supported: y, w, d, h, m, s, ms.
                                 This flag has been deprecated, use the
                                 storage.tsdb.retention.time field in the config
                                 file instead. Use with server mode only.
      --storage.tsdb.retention.size=STORAGE.TSDB.RETENTION.SIZE  

The documentation advises setting a size limit to at most 80–85% of the disk, because compaction temporarily needs room for both the old and the new blocks. For a first estimate of the disk you need, it gives a rough rule: Prometheus stores on average 1 to 2 bytes per sample, so the space is retention time × samples per second × bytes per sample. With the 1,966 series the test server held:

# Rough disk sizing from the Prometheus storage documentation:
# needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample
DAY = 24 * 3600

def disk_gb(series, scrape_interval_s, retention_days, bytes_per_sample):
    samples_per_second = series / scrape_interval_s
    return retention_days * DAY * samples_per_second * bytes_per_sample / 1e9

cases = [
    ("our setup, 15 s", 1_966, 15),
    ("our setup, 60 s", 1_966, 60),
    ("+ 40 hosts at ~1,000 series each", 1_966 + 40 * 1_000, 15),  # an assumption
]
print(f"{'situation':<36}{'samples/s':>10}{'30 days, 1 B':>14}{'30 days, 2 B':>14}")
for label, series, interval in cases:
    print(f"{label:<36}{series / interval:>10,.0f}"
          f"{disk_gb(series, interval, 30, 1):>11.1f} GB{disk_gb(series, interval, 30, 2):>11.1f} GB")

situation                            samples/s  30 days, 1 B  30 days, 2 B
our setup, 15 s                            131        0.3 GB        0.7 GB
our setup, 60 s                             33        0.1 GB        0.2 GB
+ 40 hosts at ~1,000 series each         2,798        7.3 GB       14.5 GB

The number of series drives everything — disk, memory and query speed. Prometheus reports its own count, which is worth a graph and an alert:

$ curl -s localhost:9090/metrics | grep -E '^prometheus_tsdb_head_(series|chunks) '

prometheus_tsdb_head_chunks 2021
prometheus_tsdb_head_series 1966

Local storage is deliberately simple: one server, one disk, no clustering and no replication. NFS is not supported. Kestrel's wish to compare this month with the same month last year therefore needs something else: Prometheus sends its samples with remote write to a long-term store that keeps years of data, such as Thanos, Grafana Mimir or VictoriaMetrics. Remote-Write 1.0 is the published specification; version 2.0 is still marked experimental as of October 2026.

Native histograms

A classic histogram is stored as one series per bucket plus _sum and _count: twelve buckets times three routes times two instances is already 72 series for one metric in the test setup. A native histogram stores the whole distribution as a single series whose samples contain all buckets, with automatically chosen, exponentially growing bucket boundaries. That gives much finer resolution at a fraction of the series count, and you no longer have to guess bucket boundaries in advance.

Native histograms have been a stable feature since Prometheus 3.8, but they are opt-in per scrape configuration, and the target must offer them — in practice through the Protocol Buffers format (OpenMetrics 2.0 text can carry them too, still experimentally). The old feature flag --enable-feature=native-histograms that many articles mention has done nothing since 3.9.

scrape_configs:
  - job_name: meter-api
    scrape_native_histograms: true
    static_configs:
      - targets: ["meter-api-01:8080", "meter-api-02:8080"]

The simulated exporters in the test setup speak only the text format and therefore offer classic histograms; how queries differ is shown on the PromQL page.

Running Prometheus well

A healthy start-up log is worth recognizing, so that an unhealthy one stands out. Prometheus logs in logfmt (key=value pairs); the important lines of a start (trimmed):

$ prometheus --config.file=prometheus.yml --storage.tsdb.path=/home/amara/kestrel-observability/data --web.listen-address=localhost:9090 --web.enable-lifecycle

time=2026-10-06T19:03:13.177+02:00 level=INFO source=main.go:933 msg="Starting Prometheus Server" mode=server version="(version=3.15.0, branch=HEAD, revision=5241a27fe3c6983549fccc32f6e65917408c63cd)"
…
time=2026-10-06T19:03:13.429+02:00 level=INFO source=web.go:727 msg="Start listening for connections" component=web address=localhost:9090
time=2026-10-06T19:03:13.429+02:00 level=INFO source=main.go:1499 msg="Starting TSDB ..."
…
time=2026-10-06T19:03:13.436+02:00 level=INFO source=head.go:846 msg="Replaying WAL, this may take a while" component=tsdb
…
time=2026-10-06T19:03:13.438+02:00 level=INFO source=main.go:1523 msg="TSDB started"
time=2026-10-06T19:03:13.438+02:00 level=INFO source=main.go:1721 msg="Loading configuration file" filename=prometheus.yml
time=2026-10-06T19:03:13.438+02:00 level=INFO source=main.go:1137 msg="TSDB retention updated" duration=30d size=40GiB percentage=0
…
time=2026-10-06T19:03:13.443+02:00 level=INFO source=main.go:1484 msg="Server is ready to receive web requests."
time=2026-10-06T19:03:13.443+02:00 level=INFO source=manager.go:235 msg="Starting rule manager..." component="rule manager"

After a crash or an unclean stop, "Replaying WAL" can take minutes on a large server; during that time Prometheus does not answer queries. That is normal, not a hang. The log level is set with runtime.log_level in prometheus.yml since 3.15, which deprecated the --log.level flag.

Habits that keep a Prometheus setup trustworthy:

• Run two. For high availability run two identical Prometheus servers that scrape the same targets and evaluate the same rules. Both send alerts to all Alertmanagers, which remove the duplicates as long as the Alertmanagers are clustered with each other (the --cluster.* flags, port 9094); do not put a load balancer between Prometheus and Alertmanager.
• Monitor the monitor. Alert when targets are down, when a reload failed, when rule evaluation fails and when the number of series jumps. Something outside — a second Prometheus, or a simple external check — should notice when Prometheus itself is gone.
• Keep labels bounded. Every new label value is a new series. IDs of customers, meters, sessions or requests belong in logs and traces.
• Change through review. Keep prometheus.yml and rule files in Git, run promtool check config and promtool test rules in the pipeline, and reload only what passed.

With two Alertmanagers for redundancy, each Prometheus lists both:

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["monitor01:9093", "monitor02:9093"]