Logs at Scale

When grep on one server stops working

On one server, logs are files and the journal, and grep or journalctl find what you need. That stops working as soon as a request passes through several machines, containers come and go, or a log you need was rotated away last night. The answer is centralized logging: every machine sends its logs to one searchable store. How to write useful log lines in the first place (structure, levels, correlation IDs, what not to log) is the subject of Logs; this page is about moving, storing and searching them in bulk.

Bulk is the operative word. A rough estimate for Kestrel Energy, assuming meter-api writes one access-log line for each of the roughly 444 smart-meter readings it receives per second (400,000 meters, one reading every 15 minutes):

SECONDS_PER_DAY = 86_400

# Assumptions for Kestrel (averages over a day)
sources = [
    # name, log lines per second, average bytes per line
    ("meter-api access log", 444, 320),
    ("portal access log", 25, 450),
    ("portal application log", 40, 380),
    ("db01 PostgreSQL log", 3, 250),
]
COMPRESSION = 8          # assumed: text logs often compress 5 to 10 times
RETENTION_DAYS = 30

total = 0
for name, lines_per_s, size in sources:
    gb_per_day = lines_per_s * size * SECONDS_PER_DAY / 1e9
    total += gb_per_day
    print(f"{name:<24}{gb_per_day:>7.2f} GB/day")
print(f"{'Total ingested':<24}{total:>7.2f} GB/day")
print(f"Stored for {RETENTION_DAYS} days: {total * RETENTION_DAYS:.0f} GB raw, "
      f"about {total * RETENTION_DAYS / COMPRESSION:.0f} GB compressed")

# One developer turns on DEBUG in meter-api: eight extra lines per request
debug = 444 * 8 * 300 * SECONDS_PER_DAY / 1e9
print(f"DEBUG in meter-api adds {debug:.0f} GB/day ({debug / total:.1f}x today's total)")

meter-api access log      12.28 GB/day
portal access log          0.97 GB/day
portal application log     1.31 GB/day
db01 PostgreSQL log        0.06 GB/day
Total ingested            14.63 GB/day
Stored for 30 days: 439 GB raw, about 55 GB compressed
DEBUG in meter-api adds 92 GB/day (6.3x today's total)

A medium-sized company produces gigabytes of logs per day without trying, and one forgotten debug setting multiplies that several times. Hosted log services typically charge per gigabyte ingested and per gigabyte stored; on your own hardware the same volume costs disks, memory and administrator time. Every design decision on this page is a trade-off between what you can search quickly and what that costs.

The log pipeline

Whatever the product, central logging has the same parts:

• Shipper (agent): a small program on every host or in every Kubernetes node that reads log files, the journal or container output, adds metadata (service, host, environment), parses lines, drops what is not needed and forwards the rest. It buffers when the store is unreachable, so a short outage of the log store does not lose logs.
• Store: receives, compresses, indexes and keeps the logs for a set retention period.
• Query interface: Grafana for Loki, Kibana for Elasticsearch, OpenSearch Dashboards for OpenSearch, or a command line client.

Grafana Alloy is Grafana Labs' telemetry collector: a distribution of the OpenTelemetry Collector with built-in Prometheus and Loki pipelines, which collects logs, metrics and traces and forwards them to Loki, Prometheus, Tempo or any OpenTelemetry-compatible back end. It replaces two older agents that many tutorials still show: Grafana Agent reached end of life on 1 November 2025, and Promtail, Loki's original shipper, on 2 March 2026, with all further development moved to Alloy. Other widely used shippers are the OpenTelemetry Collector itself, Fluent Bit and Vector.

At Kestrel, Alloy runs on web01, web02, api01, api02 and db01. It reads the JSON lines of the portal and meter-api and the PostgreSQL log, attaches the labels service_name, env and host, drops DEBUG lines, replaces the last octet of client IP addresses before anything leaves the host, and pushes the result to Loki on monitor01.

Grafana Loki: index the labels, not the lines

Grafana Loki is an open-source log store (AGPL-3.0; version 3.7.8 as of October 2026) designed "like Prometheus, but for logs". In its own documentation's words, Loki "does not index the contents of the logs, but only indexes metadata about your logs as a set of labels for each log stream". The lines themselves are compressed into chunks and kept in cheap object storage such as Amazon S3 or Google Cloud Storage (or a local file system for small setups).

A log stream is the sequence of log lines that share exactly the same set of labels; in Loki each unique label set is one stream, with its own index entry and its own chunks. {service_name="portal", env="production", host="web01"} is one stream; the same labels with host="web02" are another. This is the same idea as a time series in Prometheus, and it has the same weak spot: every new label value creates new streams.

Loki's label guidance is short: static labels are good, and labels should have low cardinality, "ideally limited to tens of values". For very large tenants (more than 10 TB a day) the documentation advises staying below 100,000 active streams and below a million streams in 24 hours. What happens when a high-cardinality value becomes a label:

import random

random.seed(7)
HOSTS = {"portal": ["web01", "web02"], "meter-api": ["api01", "api02"]}
LEVELS = ["debug", "info", "info", "info", "warn", "error"]

# 10,000 log records with their candidate labels
records = []
for _ in range(10_000):
    service = random.choice(sorted(HOSTS))
    records.append({
        "service_name": service,
        "host": random.choice(HOSTS[service]),
        "level": random.choice(LEVELS),
        "customer_id": f"KE-{random.randint(1, 50_000):06d}",  # fake customer numbers
    })


def streams(label_names):
    """Each distinct combination of label values is one Loki stream."""
    return {tuple(r[name] for name in label_names) for r in records}


for labels in (["service_name"], ["service_name", "host"],
               ["service_name", "host", "level"],
               ["service_name", "host", "customer_id"]):
    count = len(streams(labels))
    print(f"{', '.join(labels):<34}{count:>6} streams{len(records) / count:>9.1f} lines each")

service_name                           2 streams   5000.0 lines each
service_name, host                     4 streams   2500.0 lines each
service_name, host, level             16 streams    625.0 lines each
service_name, host, customer_id     9756 streams      1.0 lines each

With customer_id as a label almost every line becomes its own stream: thousands of tiny chunks, a huge index and slow queries, the opposite of what Loki is built for. The customer number is still useful; it just belongs inside the line, where a query can filter on it. Loki 3 can also store such values as structured metadata: kept next to the line and filterable, but not part of the stream's identity.

LogQL: querying Loki

LogQL is Loki's query language. It is modeled on PromQL: a query starts with a stream selector in curly braces (mandatory, so Loki knows which chunks to read) and continues with a pipeline of stages, each introduced by |, that filter, parse and reshape the lines. Stream selectors use the same four matchers as PromQL: =, !=, =~ and !~.

{service_name="meter-api", env="production"}

This returns every line of every meter-api stream in production, newest first, for the selected time range.

Line filters

A line filter is a pipeline stage that keeps or drops whole log lines by their text, before any parsing: |= keeps lines that contain a string, != drops them, and |~ and !~ do the same with a regular expression. Line filters are the cheapest stage, so put them first:

{service_name="meter-api"} |= "upstream timeout" != "/healthz"

{service_name="portal"} |~ `"status":5[0-9]{2}`

Loki's documentation uses this to argue against a level label: filtering with != "level=debug" on the text is cheap, so there is no need to split every stream by level.

Parsers and label filters

For structured lines, a parser stage extracts fields: | json for JSON lines, | logfmt for key=value lines, and | pattern or | regexp for free text. After parsing, every field can be compared like a label, including numerically:

{service_name="meter-api"} |= "error" | json | status >= 500

{service_name="billing-export"} | json | level="error"

{service_name="portal"} | json | trace_id="0af7651916cd43dd8448eb211c80319c"

{service_name="portal"} | json | duration_ms > 2000 | line_format "{{.status}} {{.route}} {{.duration_ms}} ms"

The third query is how a trace ID from a slow trace finds its log lines, without the trace ID ever being a label. The last one rewrites each matching line into a short summary with line_format. Lines that a parser cannot read get an __error__ label; add | __error__="" to leave them out of numeric comparisons.

What Loki does with the first query, in miniature: the label index picks the streams, and everything after that is a scan of the lines in those streams.

import json
from collections import Counter

# A tiny log store: stream labels -> list of (timestamp, line)
STORE = {
    (("host", "api01"), ("service_name", "meter-api")): [
        ("17:09:12", '{"level":"info","route":"/v1/readings","status":202,"duration_ms":41}'),
        ("17:09:40", '{"level":"error","route":"/v1/readings","status":503,"duration_ms":2310}'),
        ("17:10:05", '{"level":"error","route":"/v1/readings","status":503,"duration_ms":2288}'),
        ("17:10:31", '{"level":"info","route":"/healthz","status":200,"duration_ms":1}'),
    ],
    (("host", "api02"), ("service_name", "meter-api")): [
        ("17:09:20", '{"level":"error","route":"/v1/readings","status":500,"duration_ms":97}'),
        ("17:10:44", '{"level":"info","route":"/v1/readings","status":202,"duration_ms":38}'),
        ("17:11:02", '{"level":"error","route":"/v1/readings","status":503,"duration_ms":2297}'),
    ],
    (("host", "web01"), ("service_name", "portal")): [
        ("17:09:30", '{"level":"info","route":"/account","status":200,"duration_ms":180}'),
        ("17:10:15", '{"level":"error","route":"/account","status":502,"duration_ms":30001}'),
    ],
}

# {service_name="meter-api"} |= "error" | json | status >= 500   then count per minute
selected = [s for s in STORE if ("service_name", "meter-api") in s]  # uses the label index
scanned = matched = 0
per_minute = Counter()
for stream in selected:
    for timestamp, line in STORE[stream]:
        scanned += 1
        if "error" not in line:          # line filter: plain substring test
            continue
        fields = json.loads(line)        # parser: | json
        if fields["status"] >= 500:      # label filter on an extracted field
            matched += 1
            per_minute[timestamp[:5]] += 1

print(f"streams selected: {len(selected)} of {len(STORE)}, lines scanned: {scanned}, matched: {matched}")
for minute in sorted(per_minute):
    print(minute, per_minute[minute])

streams selected: 2 of 3, lines scanned: 7, matched: 4
17:09 2
17:10 1
17:11 1

The portal stream is never opened, but every meter-api line in the time range is. That is Loki's bargain: a tiny index and cheap ingestion, paid for with brute-force scanning at query time. Loki makes the scan fast by running it in parallel over many chunks, and you make it fast by choosing a narrow selector and a short time range.

Metric queries: logs as numbers

Wrapping a log query in a range aggregation turns lines into numbers that Grafana can graph and that can feed alert rules. count_over_time counts lines in each window, rate gives lines per second, and unwrap takes a numeric field from the line for aggregations such as percentiles:

sum by (host) (count_over_time({service_name="meter-api"} |= "upstream timeout" [5m]))

sum by (route) (rate({service_name="portal"} | json | status >= 500 [5m]))

quantile_over_time(0.95, {service_name="portal"} | json | __error__="" | unwrap duration_ms [5m]) by (route)

These are useful for applications that log but have no metrics yet. Where a real metric exists, prefer it: counting lines means reading them, which is far more expensive than reading a counter from Prometheus.

On the command line, Loki's logcli client runs the same queries (illustrative, against Kestrel's Loki):

export LOKI_ADDR=http://monitor01:3100
logcli query --since=1h --limit=3 '{service_name="meter-api"} |= "upstream timeout"'

2026-10-06T19:11:02+02:00 {host="api02", service_name="meter-api", env="production"} {"level":"error","route":"/v1/readings","status":503,"duration_ms":2297,"msg":"upstream timeout db01"}
2026-10-06T19:10:05+02:00 {host="api01", service_name="meter-api", env="production"} {"level":"error","route":"/v1/readings","status":503,"duration_ms":2288,"msg":"upstream timeout db01"}
2026-10-06T19:09:40+02:00 {host="api01", service_name="meter-api", env="production"} {"level":"error","route":"/v1/readings","status":503,"duration_ms":2310,"msg":"upstream timeout db01"}

Elasticsearch and OpenSearch: index every word

Elasticsearch is a distributed search and analytics engine, built on the Apache Lucene library, that stores JSON documents and indexes every field for fast full-text search and aggregations. For logs it is usually combined with Kibana (the user interface) and a shipper; the combination used to be called the ELK stack (Elasticsearch, Logstash, Kibana). As of October 2026 the current release is 9.5.5, with 9.4 and 8.19 still receiving updates.

OpenSearch is an open-source fork of Elasticsearch and Kibana created in 2021, licensed under Apache 2.0 and, since September 2024, governed by the OpenSearch Software Foundation under the Linux Foundation. Its interface is OpenSearch Dashboards. For log work the two behave much the same; the current OpenSearch release is 3.9.0 (as of October 2026).

The inverted index

An inverted index is a data structure that maps every term (word) to the list of documents that contain it, like the index at the back of a book. To find all log lines with "timeout", the engine looks up one entry instead of reading the lines; to find lines with both "timeout" and "portal", it intersects two lists:

import re
from collections import defaultdict

LINES = [
    "meter-api POST /v1/readings 202 41ms",
    "meter-api POST /v1/readings 503 2310ms upstream timeout db01",
    "portal GET /account 200 180ms",
    "portal GET /invoices 200 240ms",
    "meter-api POST /v1/readings 503 2288ms upstream timeout db01",
    "portal GET /account 502 30001ms upstream timeout api01",
]

# Build the inverted index: every term -> the line numbers that contain it
index = defaultdict(set)
for number, line in enumerate(LINES):
    for term in re.findall(r"[a-z0-9/.-]+", line.lower()):
        index[term].add(number)


def search(*terms):
    """Lines that contain all terms: intersect the posting lists, read no lines."""
    hits = set.intersection(*(index[t] for t in terms))
    return sorted(hits)


print(f"{len(LINES)} lines, {len(index)} distinct terms, "
      f"{sum(len(p) for p in index.values())} postings")
for query in (["timeout"], ["timeout", "portal"], ["db01"]):
    print(f"{' + '.join(query):<17}-> lines {search(*query)}")

6 lines, 21 distinct terms, 39 postings
timeout          -> lines [1, 4, 5]
timeout + portal -> lines [5]
db01             -> lines [1, 4]

Six short lines already need 39 index entries ("postings"). Real engines compress them well, but indexing every field of every line costs processor time at ingestion and often as much disk as the data itself. In return, a search for one rare value across a month of logs ("which lines mention meter M-30011?") answers in seconds, where Loki would scan everything in the selected streams.

Elasticsearch has its own cardinality trap: with dynamic mapping, every new field name in a JSON log becomes a new field in the index. An application that logs a map keyed by customer number creates thousands of fields until the index hits its limit (index.mapping.total_fields.limit, 1,000 by default) and rejects documents. Define the mapping for log indices, or use a fixed schema such as the Elastic Common Schema.

Searching Elasticsearch

Kibana's search bar accepts the short Kibana Query Language (KQL), and OpenSearch Dashboards accepts the same style:

service.name : "portal" and http.response.status_code >= 500 and message : "timeout"

Programs and the Dev Tools console use the JSON-based Query Domain-Specific Language (DSL). The same search, limited to the last hour (illustrative; the field names follow the Elastic Common Schema):

GET logs-portal-production/_search
{
  "size": 1,
  "sort": [{ "@timestamp": "desc" }],
  "query": {
    "bool": {
      "must": [{ "match": { "message": "timeout" } }],
      "filter": [
        { "range": { "@timestamp": { "gte": "now-1h" } } },
        { "range": { "http.response.status_code": { "gte": 500 } } }
      ]
    }
  }
}

{
  "took": 38,
  "timed_out": false,
  "hits": {
    "total": { "value": 214, "relation": "eq" },
    "max_score": null,
    "hits": [
      {
        "_index": ".ds-logs-portal-production-2026.10.06-000041",
        "_source": {
          "@timestamp": "2026-10-06T17:10:15.204Z",
          "service.name": "portal",
          "host.name": "web01",
          "log.level": "error",
          "http.response.status_code": 502,
          "message": "upstream timeout api01 after 30001 ms",
          "trace.id": "0af7651916cd43dd8448eb211c80319c"
        }
      }
    ]
  }
}

Clauses under filter only include or exclude documents and can be cached; clauses under must also contribute to a relevance score, which matters for a web shop's product search and hardly at all for logs. The index name starting with .ds- shows a data stream: logs are written to a series of time-based backing indices that index lifecycle management rolls over and eventually deletes (OpenSearch calls its equivalent Index State Management).

Choosing: labels or a full-text index

Many organizations end up with both: a label-indexed store for the large volume of application logs that engineers query by service and time, and a fully indexed store (often inside the SIEM) for the smaller volume of security-relevant logs that analysts search in every possible way. Kestrel chose Loki for application logs because its metrics already live in Prometheus and its dashboards in Grafana; authentication and firewall logs also go to the security team's SIEM.

Cost, retention and sampling

The biggest savings come before the store. In order of effect:

1. Do not ship what nobody reads. Drop debug lines and health-check requests (GET /healthz every few seconds from every load balancer) at the shipper.
2. Sample routine lines. Keep every warning and error, and a fraction of successful requests. Metrics already count the successes; the sample shows what a normal request looks like. See Logs for when sampling is acceptable.
3. Set retention per log type. Application logs for weeks, security and audit logs longer, debug output for days at most.
4. Move old data to cheaper storage. Loki keeps everything in object storage anyway; Elasticsearch and OpenSearch move older indices to cheaper tiers with lifecycle policies.

Sample by trace or request ID, not line by line: hash the ID and keep the request if the hash falls below the sampling rate, so all lines of one request are kept or dropped together. But there is a catch, visible in the simulation below:

import hashlib
import random

random.seed(42)


def keep(record, rate=0.10):
    """Keep every warning and error; keep about `rate` of the rest, decided per trace."""
    if record["level"] in ("warn", "error"):
        return True
    digest = hashlib.sha256(record["trace_id"].encode()).digest()
    return int.from_bytes(digest[:8], "big") / 2**64 < rate


# 2,000 portal requests, each written as 3 log lines with the same trace ID
records = []
for _ in range(2_000):
    trace_id = f"{random.getrandbits(128):032x}"
    failed = random.random() < 0.02
    for step in ("received", "queried meter-api", "responded"):
        level = "error" if failed and step == "responded" else "info"
        records.append({"trace_id": trace_id, "level": level, "msg": step})

kept = [r for r in records if keep(r)]
errors = sum(r["level"] == "error" for r in records)
kept_errors = sum(r["level"] == "error" for r in kept)
by_trace = {}
for r in kept:
    by_trace.setdefault(r["trace_id"], []).append(r["msg"])
complete = sum(len(steps) == 3 for steps in by_trace.values())
failed_ids = {r["trace_id"] for r in records if r["level"] == "error"}
failed_complete = sum(len(by_trace[t]) == 3 for t in failed_ids)

print(f"lines: {len(records)} -> {len(kept)} kept ({len(kept) / len(records):.1%})")
print(f"error lines kept: {kept_errors} of {errors}")
print(f"requests with all three lines kept: {complete}")
print(f"failed requests with all three lines kept: {failed_complete} of {len(failed_ids)}")

lines: 6000 -> 635 kept (10.6%)
error lines kept: 40 of 40
requests with all three lines kept: 199
failed requests with all three lines kept: 2 of 40

Volume drops by almost 90%, and every error line survives. But for 38 of the 40 failed requests, the two lines before the error were dropped: when they were written, nobody knew the request would fail. Keeping the full story of failed requests requires buffering lines per request until it ends and deciding then, which is the same trade-off as tail sampling in distributed tracing. A practical middle way is to log one complete summary line per request (route, status, duration, trace ID) that is never sampled, and sample only the chattier lines.

Retention is enforced by the store: in Loki by its compactor, with a global period and optional overrides; in Elasticsearch and OpenSearch by lifecycle policies that delete old indices. Make the retention a decision written down per log type, not a default nobody chose.

Access, security and privacy

A central log store holds a copy of what every system saw, so it needs the protection of a production database:

• Personal data. Client IP addresses, customer numbers and email addresses in logs are usually personal data under the GDPR, and storage limitation means you must not keep them longer than needed. Deleting single lines from compressed chunks or indices is expensive, so the practical tools are redaction at the shipper and short retention.
• Access control. Not everyone who may read meter-api's logs may read the authentication logs. Loki separates data by tenant; Elasticsearch and OpenSearch have index- and field-level permissions.
• Integrity. Logs that may serve as evidence after a security incident must be protected against changes by the people whose actions they record; the security monitoring page explains what to log for that purpose and why.
• Secrets. A password or token that reaches the central store has been copied to every replica and backup of it. Treat it as leaked and rotate it.