Distributed Tracing

Why trace a request

A Kestrel customer clicks "Show my usage" in the portal. The request reaches the portal on web01, which calls meter-api on api01, which queries PostgreSQL on db01 twice. It takes 2.1 seconds. Metrics say the portal's 95th percentile went up; the logs of three hosts each show part of the story. Neither says where those 2.1 seconds went.

Distributed tracing is recording the path of individual requests through all the services and components they touch, with the timing of each step, so you can see where a request spent its time and where it failed. The result for one request is a trace; the Cloud topic introduces the idea with a short example. This page covers how it works inside: what a span contains, how the trace ID travels, which traces are kept, and how to read one.

Spans

A span is one timed operation within a trace: an incoming HTTP request, an outgoing call, a database query, a wait for a lock. Spans nest. The span for meter-api's handling of a request is the parent of the spans for its two database queries. Each span records:

Simplified, the meter-api span of the example looks like this when the service exports it (field names follow OpenTelemetry's semantic conventions for HTTP; the real export format has more structure):

{
  "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
  "spanId": "b7c1e09a4f3d2a61",
  "parentSpanId": "5f3e2b1c9d0a4e77",
  "name": "GET /v1/meters/{meter_id}/readings",
  "kind": "SERVER",
  "startTime": "2026-10-06T17:21:04.026Z",
  "endTime": "2026-10-06T17:21:06.034Z",
  "status": "UNSET",
  "attributes": {
    "http.request.method": "GET",
    "http.route": "/v1/meters/{meter_id}/readings",
    "url.path": "/v1/meters/M-104317/readings",
    "url.scheme": "https",
    "server.address": "api.example.com",
    "http.response.status_code": 200
  }
}

Trace IDs and span IDs

A trace ID is the identifier that every span of one trace shares. In the W3C Trace Context standard it is 16 bytes, written as 32 lowercase hexadecimal characters; a span ID is 8 bytes, 16 hex characters. Both are random. A trace ID carries no meaning and is not a secret, which makes it a good correlation ID for log lines and error pages.

The root span

The root span is the first span of a trace, the one without a parent. Its duration is the duration of the whole request as the first instrumented component saw it. Where the root span starts depends on where instrumentation starts: if the load balancer is instrumented, the root is there; if not, it is the portal's SERVER span on web01, and any time spent before that is invisible.

Span kinds, names and status

OpenTelemetry gives each span a kind: SERVER for handling an incoming request, CLIENT for an outgoing call, PRODUCER and CONSUMER for sending and receiving messages through a queue, and INTERNAL for work inside a process. A CLIENT span on web01 and the matching SERVER span on api01 describe the same call from two sides; the time between them is network and queueing.

The name must be low-cardinality, like a metric label: GET /v1/meters/{meter_id}/readings with the route template, never the raw path with a meter ID in it. The semantic conventions say an instrumentation "MUST NOT default to using URI path" as the name. The concrete path goes in an attribute (url.path), where it does no harm to grouping.

The status follows the same conventions: a server span with a 5xx response gets status Error; with a 4xx response its status stays unset, because a 404 is usually the client's mistake, not a failure of the server. A client span with a 4xx should be marked as an error, since the caller did not get what it wanted.

Reading a trace

Tracing tools draw a trace as a waterfall: one bar per span on a shared time axis, each indented under its parent. The "Show my usage" request, as Ravi Kumar found it while investigating slow usage pages:

The longest bar is the root, but it is not the problem: it is long because its children are. What matters is each span's self time: its duration minus the time covered by its children. Tools often compute this for you; the logic is short:

# One trace of "Show my usage", times in ms from the start of the root span
# (span_id, parent_id, name, start, end)
spans = [
    ("a1", None, "GET /usage  (portal, web01)", 0, 2140),
    ("b2", "a1", "GET /v1/meters/{meter_id}/readings  (client)", 20, 2040),
    ("c3", "b2", "GET /v1/meters/{meter_id}/readings  (meter-api)", 26, 2034),
    ("d4", "c3", "pool.acquire  (wait for db01 connection)", 30, 1830),
    ("e5", "c3", "SELECT readings  (db01)", 1832, 1922),
    ("f6", "c3", "SELECT tariffs  (db01)", 1924, 2016),
    ("g7", "a1", "render usage.html", 2046, 2132),
]


def self_time(span_id, start, end):
    """Duration minus the time covered by this span's children."""
    covered = 0
    position = start
    children = sorted((s[3], s[4]) for s in spans if s[1] == span_id)
    for child_start, child_end in children:   # merge overlapping children
        child_start = max(child_start, position)
        if child_end > child_start:
            covered += child_end - child_start
            position = child_end
    return (end - start) - covered


print(f"{'span':<50}{'total':>7}{'self':>7}")
for span_id, parent, name, start, end in sorted(
    spans, key=lambda s: self_time(s[0], s[3], s[4]), reverse=True
):
    print(f"{name:<50}{end - start:>7}{self_time(span_id, start, end):>7}")

span                                                total   self
pool.acquire  (wait for db01 connection)             1800   1800
SELECT tariffs  (db01)                                 92     92
SELECT readings  (db01)                                90     90
render usage.html                                      86     86
GET /usage  (portal, web01)                          2140     34
GET /v1/meters/{meter_id}/readings  (meter-api)      2008     26
GET /v1/meters/{meter_id}/readings  (client)         2020     12

The ranking makes the finding obvious: 1.8 seconds of waiting for a database connection, against less than 0.2 seconds of actual queries. Making the queries faster would not help. Hamid Rahimi and Ravi Kumar instead looked at why the pool of 20 connections was always in use, and found a background job holding connections open while it slept.

Patterns worth recognizing in a waterfall:

• One long span with high self time: the work, or the wait, is there. Check its attributes and events.
• Gaps: time in a parent that no child covers. Either the parent is busy itself, or something is not instrumented (a cache lookup, a DNS resolution, garbage collection).
• A staircase of small, identical spans: the same query run once per item, one after the other (the "N+1 queries" pattern). Each span is fast; together they are slow.
• Sequential calls that could run in parallel: two independent calls drawn one after the other.
• A long gap between a CLIENT span and its SERVER span: time lost in the network, a proxy, or a queue in front of the receiving service.

Context propagation

Spans from different services only form one trace if each service knows which trace it is part of. Context propagation is the mechanism that carries the trace context (trace ID, the ID of the calling span and the sampling decision) from one service to the next, in the headers or metadata of each request or message, and within a service from function to function. Each service reads the incoming context, creates its own spans as children of it, and injects an updated context into its own outgoing calls.

The traceparent header

The traceparent header is the HTTP header defined by the W3C Trace Context standard that carries the trace context between services, in the form version-traceid-parentid-flags. Trace Context Level 1 has been a W3C Recommendation since 23 November 2021, and it is the default propagation format of OpenTelemetry. The example from the standard itself:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

• 00: the version. Version ff is invalid.
• 0af7…319c: the trace ID, 32 lowercase hex characters. All zeros is forbidden.
• b7ad6b7169203331: the parent ID, the span ID of the caller's span (16 hex characters, not all zeros).
• 01: the trace flags. In version 00 the only flag is sampled (bit value 1): the caller is recording this trace.

The standard tells a receiver to ignore a header it cannot parse and start a new trace. That is what a service must do, and what this parser does. For each valid header it also builds the header for an outgoing call: the same trace ID and flags, with the ID of the service's own CLIENT span as the new parent:

import random
import re

TRACEPARENT = re.compile(r"([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})")
rng = random.Random(2026)    # seeded for this page; real code uses secrets.token_hex(8)


def parse(header):
    """Return (trace_id, parent_id, sampled), or None if the header must be ignored."""
    match = TRACEPARENT.fullmatch(header.strip())
    if not match:
        return None
    version, trace_id, parent_id, flags = match.groups()
    if version == "ff" or set(trace_id) == {"0"} or set(parent_id) == {"0"}:
        return None
    sampled = int(flags, 16) & 0x01 == 1
    return trace_id, parent_id, sampled


def child_header(trace_id, sampled):
    """traceparent for an outgoing call: same trace, new span ID as the parent."""
    span_id = f"{rng.getrandbits(64):016x}"
    return f"00-{trace_id}-{span_id}-{'01' if sampled else '00'}"


incoming = [
    "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01",   # the spec's example
    "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-00",   # not sampled
    "00-00000000000000000000000000000000-b7ad6b7169203331-01",   # all-zero trace ID
    "00-0AF7651916CD43DD8448EB211C80319C-b7ad6b7169203331-01",   # uppercase hex
]
for header in incoming:
    result = parse(header)
    if result is None:
        print("ignored, start a new trace:", header)
        continue
    trace_id, parent_id, sampled = result
    print(f"trace {trace_id}  parent {parent_id}  sampled={sampled}")
    print("  call meter-api with:", child_header(trace_id, sampled))

trace 0af7651916cd43dd8448eb211c80319c  parent b7ad6b7169203331  sampled=True
  call meter-api with: 00-0af7651916cd43dd8448eb211c80319c-51c9bc701e7ea419-01
trace 0af7651916cd43dd8448eb211c80319c  parent b7ad6b7169203331  sampled=False
  call meter-api with: 00-0af7651916cd43dd8448eb211c80319c-f38b2ffc80a4df5a-00
ignored, start a new trace: 00-00000000000000000000000000000000-b7ad6b7169203331-01
ignored, start a new trace: 00-0AF7651916CD43DD8448EB211C80319C-b7ad6b7169203331-01

Real code uses the OpenTelemetry SDK for this (see OpenTelemetry) and generates span IDs with a cryptographically random source such as secrets.token_hex(8); the seeded generator only keeps this page's output stable. Note the uppercase case: the standard requires lowercase hex, so the header is ignored and the trace breaks at that service.

tracestate and baggage

Two related headers often travel with traceparent. tracestate carries vendor-specific data as a list of key=value pairs (at most 32), for example tracestate: rojo=00f067aa0ba902b7,congo=t61rcWkgMzE. baggage carries application-defined key-value pairs that every downstream service can read, such as a tenant or a feature flag.

Where propagation breaks

A trace is only as complete as its weakest hop. Every component that receives a request and sends another one must copy the context across. Meter readings show the trickiest case: meter-api accepts a reading, puts it on a queue, and a consumer writes it to db01 later. HTTP headers do not survive that; the context must travel as an attribute of the message:

Other places where traces break:

• A reverse proxy, API gateway or old load balancer that drops unknown headers.
• A thread pool or background task that runs code without the context of the request that queued it.
• A service that starts a fresh trace on purpose. The standard allows a service at the edge of a trusted network to restart the trace, so that outside callers cannot influence internal trace IDs; that is a deliberate choice, not a bug.

Sampling

Recording and storing every span of every request is expensive at high volume. OpenTelemetry's documentation suggests considering sampling from about 1,000 traces per second. Healthy requests look alike, so a sample of them represents the rest well. Errors and slow requests are rare and each one is interesting, so a good sampling strategy keeps those.

Head sampling

Head sampling decides whether to record a trace at its very start, before anything is known about how the request will go, and passes the decision downstream in the sampled flag. The usual form derives the decision from the trace ID, so any service that sees the same ID reaches the same answer:

import random

RATIO = 0.05                        # keep 5% of traces
LIMIT = int(RATIO * 2**56)          # threshold on the last 56 bits of the trace ID


def head_sample(trace_id):
    """The same trace ID gives the same decision in every service."""
    return int(trace_id[-14:], 16) < LIMIT


rng = random.Random(7)              # seeded: the same "random" trace IDs on every run
trace_ids = [f"{rng.getrandbits(128):032x}" for _ in range(100_000)]
kept = [t for t in trace_ids if head_sample(t)]
print(f"traces started: {len(trace_ids):,}")
print(f"traces kept:    {len(kept):,} ({len(kept) / len(trace_ids):.2%})")

# portal, meter-api and the readings consumer decide independently, yet agree:
example = kept[0]
print(example, [head_sample(example) for service in ("portal", "meter-api", "consumer")])

# 30 of the 100,000 requests failed. How many of their traces survive?
failed = rng.sample(trace_ids, 30)
print(f"failed requests with a trace: {sum(head_sample(t) for t in failed)} of 30")

traces started: 100,000
traces kept:    4,935 (4.93%)
6b0d549b6f03675a1600a35a099950d8 [True, True, True]
failed requests with a trace: 0 of 30

Head sampling is simple, cheap and needs no extra infrastructure. Its weakness shows in the last line: with 5% sampling, a failed request has only a 5% chance of having a trace, and here none of the 30 failures do. As the OpenTelemetry documentation puts it, with head sampling alone "you cannot ensure that all traces with an error within them are sampled". In practice, downstream services usually just follow the sampled flag in traceparent (parent-based sampling); deriving the decision from the trace ID makes services agree even when they did not receive the flag.

Tail sampling

Tail sampling decides whether to keep a trace after all or most of its spans have arrived, so the decision can use the whole trace: its duration, its status, its attributes. Services send all their spans to a collector (in OpenTelemetry, the Collector's tail sampling processor), which holds each trace in memory for a while and then applies rules:

import random

rng = random.Random(7)


def finished_trace():
    """A complete trace as a tail sampler sees it, after the last span arrived."""
    trace_id = f"{rng.getrandbits(128):032x}"
    duration_ms = round(rng.lognormvariate(4.6, 0.8))      # most around 100 ms
    has_error = rng.random() < 0.0003
    return trace_id, duration_ms, has_error


def tail_sample(trace_id, duration_ms, has_error):
    if has_error:
        return "error"
    if duration_ms > 1000:
        return "slow"
    if int(trace_id[-14:], 16) < 0.05 * 2**56:              # 5% of the rest
        return "baseline"
    return None


reasons = {"error": 0, "slow": 0, "baseline": 0}
errors = slow = 0
for _ in range(100_000):
    trace = finished_trace()
    errors += trace[2]
    slow += trace[1] > 1000 and not trace[2]
    reason = tail_sample(*trace)
    if reason:
        reasons[reason] += 1

print(f"error traces kept: {reasons['error']} of {errors}")
print(f"slow traces kept:  {reasons['slow']} of {slow}")
print(f"normal traces kept: {reasons['baseline']:,}")
print(f"total kept: {sum(reasons.values()):,} of 100,000")

error traces kept: 39 of 39
slow traces kept:  199 of 199
normal traces kept: 4,986
total kept: 5,224 of 100,000

Every error and every slow trace is kept, plus a 5% baseline of normal traffic to compare them with. The cost is operational: the collector must buffer every span of every trace until the decision, all spans of one trace must reach the same collector instance, and that component needs its own monitoring and capacity.

Large systems often combine the two: head sampling to cap the volume (say 25%), then tail sampling in the collector to keep the interesting part of that.

Connecting traces with logs and metrics

Traces explain single requests; metrics show the trend; logs carry the detail. The links between them are what make an investigation fast:

• Trace ID in every log line. The structured logging example in Logs uses the trace ID as its request_id. From a slow span you jump to exactly the log lines of that request, on every host.
• The same names everywhere. The span name GET /v1/meters/{meter_id}/readings and the metric label route="/v1/meters/{meter_id}/readings" use the same template, so a slow route on a dashboard leads to traces of that route.
• Trace IDs on error pages and in tickets. Mei Chen's service desk quotes the reference from the error page in the ticket; the engineer opens the trace directly instead of searching by time and customer.

Instrumentation and tracing back ends

Spans are produced by instrumentation: code in the application, or a library or agent that wraps the web framework, the HTTP client and the database driver. Today that is almost always OpenTelemetry, whose software development kits (SDKs) exist for all mainstream languages and send spans over the OpenTelemetry Protocol (OTLP) to a collector or back end; see OpenTelemetry. The back end stores the traces and draws the waterfalls. Common choices, as of October 2026:

• Jaeger (open source, a graduated Cloud Native Computing Foundation project). Jaeger v2 is built on the OpenTelemetry Collector; the separate v1 components (agent, collector, query) are no longer published since v2.14.0 in January 2026.
• Grafana Tempo (open source).
• The cloud providers' tracing services: AWS X-Ray, Azure Application Insights and Google Cloud Trace, which all accept OpenTelemetry data.

What goes wrong

• Raw paths as span names. GET /v1/meters/M-104317/readings makes every meter a separate operation in the tracing tool, and nothing can be grouped. Use the route template; put the path in an attribute.
• Broken propagation. A trace that ends at meter-api while the consumer's spans show up as separate traces means a hop does not pass the context on. Check proxies, queues and background tasks.
• Clock differences. Span times come from each host's clock. If api01 runs 300 ms behind web01, the child span seems to start before its parent. Keep every host on NTP.
• Personal data in attributes. Full URLs with query strings, SQL statements with literal values and customer names end up in the tracing store. Apply the same rules as for logs.
• Every 404 marked as an error. Follow the semantic conventions: for SERVER spans, only 5xx is an error, otherwise error searches drown in clients asking for things that do not exist.
• Relying on head sampling for rare failures. At 5%, most failed requests have no trace. Keep errors with tail sampling, or sample at a higher rate on low-volume services.