OpenTelemetry

What OpenTelemetry is

OpenTelemetry (often OTel) is an open-source, vendor-neutral set of specifications, application programming interfaces (APIs), software development kits (SDKs) and tools for producing telemetry — traces, metrics and logs — and delivering it to whatever back end you choose. The Cloud topic introduces the idea: instrument once, decide later where the data goes. This page shows how that works in practice at Kestrel Energy.

OpenTelemetry is a Cloud Native Computing Foundation (CNCF) project. It was accepted in May 2019, became incubating in August 2021 and graduated on 11 May 2026, the CNCF's top maturity level. Older articles that call it "incubating" are out of date.

The project consists of several parts that fit together:

• The API: the interfaces code calls to create spans, record metrics and emit logs.
• The SDK per language: the implementation behind the API — sampling, batching and exporting.
• Instrumentation: ready-made instrumentation for popular frameworks and libraries, and zero-code agents that apply it without changing the application.
• OTLP, the OpenTelemetry Protocol that carries the data.
• The Collector: a stand-alone program that receives, processes and forwards telemetry.
• Semantic conventions: agreed names for attributes and metrics, so that data from different languages looks the same.

What OpenTelemetry does not include is a place to store and look at the data. Traces go to a trace store such as Jaeger or Grafana Tempo, metrics to Prometheus or a cloud monitoring service, logs to a log store (see Logs at Scale). Many commercial observability products accept OTLP directly.

How stable is each signal?

Not every part has the same maturity. The status as published by the project, as of October 2026:

The specification being stable does not mean every language is. Per language the picture differs: Java and .NET are stable for traces, metrics and logs; Python is stable for traces and metrics while its logs support is still in development; Go's logs are a release candidate. Check the language's own page before you depend on a signal.

The API and the SDK

The split between API and SDK is the design choice that makes OpenTelemetry work for libraries. A database driver or web framework depends only on the API. If the application that uses it never installs an SDK, every API call is a cheap no-op: no data, no overhead, no dependency on any back end. When the application does install and configure the SDK, the same calls start producing real spans and measurements.

The SDK is where the decisions are made:

• Resource attributes describe the thing that produces the telemetry: service.name, service.version, service.instance.id, deployment.environment.name. Every span, metric and log record carries them, so a back end can show "all errors of meter-api version 2.14.3 on api01".
• Sampling decides which traces are kept (see Distributed Tracing).
• Exporters decide where the data goes: usually OTLP to a local Collector.

All SDKs read the same environment variables, which is the easiest way to configure them from a systemd unit, a container or a Kubernetes manifest:

Instrumentation: zero-code and manual

Instrumentation is the code that produces telemetry. There are two ways to get it, and most services use both.

Zero-code instrumentation

Zero-code instrumentation (also called automatic instrumentation) adds telemetry to an application without changing its source code: an agent or wrapper loads instrumentation for the frameworks and libraries the application already uses, such as its HTTP server, HTTP client and database driver. OpenTelemetry offers it for Java, .NET, Python, JavaScript (Node.js), PHP and Go, and the OpenTelemetry Operator can inject it into Kubernetes pods.

Kestrel's meter-api is a Java service on api01. Ravi Kumar adds the Java agent to its start command — a JVM option, not a code change — and points it at the Collector on the same host. In the systemd unit these would be Environment= lines; typed by hand they look like this (illustrative):

$ export OTEL_SERVICE_NAME=meter-api
$ export OTEL_RESOURCE_ATTRIBUTES=service.instance.id=api01,deployment.environment.name=production
$ export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
$ export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
$ java -javaagent:/opt/otel/opentelemetry-javaagent.jar -jar /opt/meter-api/meter-api.jar

From then on every incoming HTTP request becomes a server span with standard attributes, every JDBC query to db01 a client span, and the agent records HTTP metrics such as http.server.request.duration. Nobody wrote a line of tracing code.

Manual instrumentation

Zero-code instrumentation sees the edges of a service — requests in, calls out — but not the business logic in between. For that you add spans and metrics yourself with the API. The customer portal calculates a monthly cost estimate that is sometimes slow; the team wraps it in its own span and counts how often it runs. This Python sketch needs the opentelemetry-api package and is not run here:

portal/estimates.py

from opentelemetry import metrics, trace

tracer = trace.get_tracer("kestrel.portal")
meter = metrics.get_meter("kestrel.portal")
estimates = meter.create_counter(
    "portal.estimates.calculated",
    unit="{estimate}",
    description="Monthly cost estimates calculated for customers.",
)


def monthly_estimate(daily_kwh, price_per_kwh):
    with tracer.start_as_current_span("calculate monthly estimate") as span:
        span.set_attribute("kestrel.days.count", len(daily_kwh))
        estimate = round(sum(daily_kwh) / len(daily_kwh) * 30 * price_per_kwh, 2)
        estimates.add(1, {"kestrel.contract.type": "variable"})
        return estimate

• start_as_current_span makes the new span a child of whatever span is active — here the server span that zero-code instrumentation created for the HTTP request — so it shows up nested inside the request in the trace view.
• The counter's unit {estimate} is an annotation in curly braces: a countable thing, not a physical unit.
• Attributes carry low-cardinality facts (contract type, number of days). The customer number does not go into a metric attribute: every distinct value would create a new time series (see Metrics).

OTLP: the protocol

OTLP (OpenTelemetry Protocol) is the protocol that SDKs, Collectors and back ends use to send traces, metrics and logs to each other. It is stable for all three signals and comes in two transports:

Clients may compress with gzip and should retry with exponential backoff when the server answers that it is overloaded. A server may also report a partial success: part of the batch was rejected, and the client must not resend it.

You rarely build OTLP by hand, but the JSON form is readable and handy for testing. It follows the Protocol Buffers JSON mapping with a few OTLP-specific rules: trace and span IDs are hexadecimal strings (not base64), enumerations such as the span kind are numbers, keys are in lowerCamelCase and 64-bit integers such as nanosecond timestamps are written as strings. The script below builds the body of one OTLP/HTTP trace request for a portal span and prints every value with its path:

import json

def attr(key, value):
    """One OTLP attribute: a key plus a typed value."""
    if isinstance(value, int):
        return {"key": key, "value": {"intValue": str(value)}}  # 64-bit integers as strings
    return {"key": key, "value": {"stringValue": value}}

start_ns = 1_791_306_600_000_000_000  # 2026-10-06 17:10:00 UTC in nanoseconds
span = {
    "traceId": "4f1d6c0e9a8b47d2b3e5a17c90d2e641",  # 16 bytes as hex, not base64
    "spanId": "a3c9e1f04b7d2a65",                   # 8 bytes as hex
    "name": "GET /invoices",
    "kind": 2,                                     # SPAN_KIND_SERVER: numbers only
    "startTimeUnixNano": str(start_ns),
    "endTimeUnixNano": str(start_ns + 364_000_000),  # 364 ms later
    "attributes": [attr("http.request.method", "GET"),
                   attr("http.route", "/invoices"),
                   attr("http.response.status_code", 200)],
}
payload = {"resourceSpans": [{
    "resource": {"attributes": [attr("service.name", "portal"),
                                attr("service.instance.id", "web01")]},
    "scopeSpans": [{"scope": {"name": "kestrel.portal"}, "spans": [span]}],
}]}

def flatten(node, path=""):
    """Yield (path, value) for every leaf, like the tool gron does."""
    if isinstance(node, dict):
        for key, value in node.items():
            yield from flatten(value, f"{path}.{key}" if path else key)
    elif isinstance(node, list):
        for i, value in enumerate(node):
            yield from flatten(value, f"{path}[{i}]")
    else:
        yield path, node

for path, value in flatten(payload):
    print(path.replace("resourceSpans[0].", "").replace("scopeSpans[0].spans[0].", "span."),
          "=", json.dumps(value))
body = json.dumps(payload).encode()
print(f"POST /v1/traces  Content-Type: application/json  {len(body)} bytes")

resource.attributes[0].key = "service.name"
resource.attributes[0].value.stringValue = "portal"
resource.attributes[1].key = "service.instance.id"
resource.attributes[1].value.stringValue = "web01"
scopeSpans[0].scope.name = "kestrel.portal"
span.traceId = "4f1d6c0e9a8b47d2b3e5a17c90d2e641"
span.spanId = "a3c9e1f04b7d2a65"
span.name = "GET /invoices"
span.kind = 2
span.startTimeUnixNano = "1791306600000000000"
span.endTimeUnixNano = "1791306600364000000"
span.attributes[0].key = "http.request.method"
span.attributes[0].value.stringValue = "GET"
span.attributes[1].key = "http.route"
span.attributes[1].value.stringValue = "/invoices"
span.attributes[2].key = "http.response.status_code"
span.attributes[2].value.intValue = "200"
POST /v1/traces  Content-Type: application/json  657 bytes

Note the three levels: a resource (which service and instance), a scope (which instrumentation library produced the data) and then the spans themselves. Metrics and logs use the same nesting (resourceMetrics, resourceLogs).

The Collector

The OpenTelemetry Collector is a stand-alone program that receives telemetry, processes it and exports it to one or more back ends. Applications could export straight to a back end, but a Collector next to them takes work and risk out of every service:

• Applications hand data to a local process quickly and get on with their work; the Collector does the batching, retrying and buffering when the back end is slow.
• Redaction, filtering and enrichment happen in one place, in configuration, instead of in every code base.
• Changing or adding a back end is a Collector change; the applications keep sending OTLP to the same local address.
• The Collector can also collect what applications do not send themselves: host metrics, log files, Prometheus endpoints.

Two deployment patterns are common, often combined. An agent Collector runs on every host (or as a DaemonSet or sidecar in Kubernetes) and receives data on localhost. A gateway Collector is a central service that agents forward to; it holds the back-end credentials and does expensive work such as tail sampling. Kestrel runs an agent on web01, web02 and api01 and a gateway on monitor01.

The Collector comes in several official distributions, as of October 2026: Core (otelcol, a small set of components), Contrib (otelcol-contrib, with hundreds of community components), Kubernetes (otelcol-k8s), OTLP (otelcol-otlp, OTLP in and out only) and an eBPF profiling distribution. You can build your own with only the components you need using the OpenTelemetry Collector Builder (ocb), and vendors ship their own distributions, among them the AWS Distro for OpenTelemetry and Grafana Alloy. The current release is v0.162.0 (late September 2026); the version number still starts with 0, but releases come every few weeks and the core modules are versioned 1.x.

Pipelines: receivers, processors, exporters

A Collector configuration has a few kinds of components:

• A receiver is how data gets into the Collector: it listens for pushed data (the otlp receiver) or fetches it (the prometheus receiver scrapes metrics endpoints; the filelog receiver reads log files).
• A processor changes the data on its way through: it limits memory use, drops what nobody needs, removes or hashes sensitive attributes, adds attributes or samples traces.
• An exporter sends the data onward: to another Collector, a back end or, for debugging, the Collector's own log.
• A connector joins two pipelines: it is the exporter of one and the receiver of the next, for example to count spans and turn the counts into metrics. Extensions add capabilities that do not touch the data, such as a health-check endpoint or authentication.

A pipeline ties them together for one signal: traces, metrics or logs. Two rules from the documentation explain most "why is nothing happening?" questions. First, configuring a component does not enable it: it only runs when a pipeline in service.pipelines lists it. Second, the order of the processors in a pipeline is the order in which they process the data.

A Collector configuration

This is the agent configuration on web01 (illustrative, with the component names current as of October 2026). It accepts OTLP from the portal on localhost only, protects itself against running out of memory, drops health-check spans, hashes the customer number that the portal puts on spans, stamps the environment on everything and forwards to the gateway over TLS:

/etc/otelcol-contrib/config.yaml on web01

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
      http:
        endpoint: 127.0.0.1:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 400
    spike_limit_mib: 100
  filter/healthz:
    error_mode: ignore
    trace_conditions:
      - 'span.attributes["http.route"] == "/healthz"'
  attributes/customer:
    actions:
      - key: kestrel.customer
        action: hash
  resource:
    attributes:
      - key: deployment.environment.name
        value: production
        action: upsert

exporters:
  otlp_grpc:
    endpoint: monitor01.example.com:4317
    tls:
      ca_file: /etc/otelcol-contrib/kestrel-ca.pem
    sending_queue:
      batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, filter/healthz, attributes/customer, resource]
      exporters: [otlp_grpc]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource]
      exporters: [otlp_grpc]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resource]
      exporters: [otlp_grpc]

• Receivers on 127.0.0.1. The Collector itself defaults to localhost; many examples on the internet bind to 0.0.0.0 "for convenience", which opens the port on every interface, including public ones. An agent only needs localhost.
• memory_limiter comes first, so it can refuse data before other processors spend memory on it.
• filter/healthz uses the type/name syntax: the type is filter, the name lets you have several. It drops spans whose condition is true. filter and attributes are in both the Core and the Contrib distribution (Core contains a handful of processors: batch, memory_limiter, attributes, resource, span, probabilistic_sampler and filter).
• action: hash replaces the value with an unkeyed hash. That is pseudonymization, not anonymization: the same customer always gets the same hash, so traces stay linkable — and because customer numbers have a short, predictable format, anyone with the traces can hash every possible number and reverse it (see the keyed hash on the Logs page). Use it only where the trace store is as well protected as the original data; otherwise use action: delete, or pseudonymize with a keyed hash in the application.
• sending_queue: batch: turns on batching in the exporter's queue; the empty value means "default settings". This replaces the separate batch processor that older examples put in every pipeline.

The gateway on monitor01 receives from all agents and fans the data out per signal. Only the parts that differ are shown: traces to a trace store, metrics to Prometheus (see below) and logs to a log store, each through its own named exporter:

exporters:
  otlp_grpc/traces:
    endpoint: traces.example.com:4317
  otlp_http/prometheus:
    endpoint: http://localhost:9090/api/v1/otlp
  otlp_http/logs:
    endpoint: https://logs.example.com/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp_grpc/traces]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp_http/prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp_http/logs]

The order of processors matters more than it looks. This simulation runs six spans through three processors twice, in two different orders. A limit of "three spans per batch" stands in for a sampler or any processor that keeps only part of the data:

import hashlib

# Six spans arriving at the Collector on web01, oldest first.
SPANS = [
    {"name": "GET /healthz", "attrs": {"http.route": "/healthz"}},
    {"name": "POST /login", "attrs": {"http.route": "/login", "kestrel.customer": "KE-000417"}},
    {"name": "GET /healthz", "attrs": {"http.route": "/healthz"}},
    {"name": "GET /invoices", "attrs": {"http.route": "/invoices", "kestrel.customer": "KE-000417"}},
    {"name": "GET /healthz", "attrs": {"http.route": "/healthz"}},
    {"name": "GET /account", "attrs": {"http.route": "/account", "kestrel.customer": "KE-000982"}},
]

def drop_health_checks(spans):   # like a filter processor
    return [s for s in spans if s["attrs"]["http.route"] != "/healthz"]

def hash_customer(spans):        # like an attributes processor with action: hash
    out = []
    for s in spans:
        attrs = dict(s["attrs"])
        if "kestrel.customer" in attrs:
            attrs["kestrel.customer"] = hashlib.sha256(attrs["kestrel.customer"].encode()).hexdigest()[:12]
        out.append({"name": s["name"], "attrs": attrs})
    return out

def keep_three(spans):           # a crude limit: at most 3 spans per batch
    return spans[:3]

def run(pipeline):
    spans = SPANS
    for processor in pipeline:
        spans = processor(spans)
    print(" -> ".join(p.__name__ for p in pipeline))
    for s in spans:
        print(f"   {s['name']:<14} {s['attrs'].get('kestrel.customer', '-')}")

run([drop_health_checks, hash_customer, keep_three])
run([keep_three, drop_health_checks, hash_customer])

drop_health_checks -> hash_customer -> keep_three
   POST /login    271fba008ef0
   GET /invoices  271fba008ef0
   GET /account   ef5367228adc
keep_three -> drop_health_checks -> hash_customer
   POST /login    271fba008ef0

In the second order the limit spends its budget on health checks that are dropped a step later, and two real requests are lost. Filter early, sample late, and put anything that removes sensitive data before every exporter.

Validating and debugging

Check a configuration before restarting the service; a Collector that refuses to start stops all telemetry from that host. The validate command reads the file and checks every reference (illustrative output, with a typo in a processor name):

$ otelcol-contrib validate --config=/etc/otelcol-contrib/config.yaml

Error: invalid configuration: service::pipelines::traces: references processor "attributes/customers" which is not configured

When data does not arrive where you expect, add the debug exporter to the pipeline for a few minutes. With verbosity: basic it writes one line per batch to the Collector's log, readable with journalctl -u otelcol-contrib; detailed prints every span and attribute, which is useful on a test system and a privacy problem in production.

exporters:
  debug:
    verbosity: basic
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, filter/healthz, attributes/customer, resource]
      exporters: [otlp_grpc, debug]

2026-10-06T17:12:03.512Z	info	Traces	{"otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 4}
2026-10-06T17:12:08.519Z	info	Traces	{"otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 2, "spans": 11}

Spans appear in the debug output but not in the trace store? Then the problem is between the exporter and the back end: look for export errors and retries in the same log. No debug lines at all? Then nothing arrives: check the application's OTEL_EXPORTER_OTLP_ENDPOINT, the protocol (gRPC to 4318 or HTTP to 4317 fails) and whether the pipeline lists the receiver. The Collector also exposes metrics about itself — by default on port 8888 — which Prometheus can scrape to alert on dropped or failed exports.

Semantic conventions

Semantic conventions are OpenTelemetry's agreed names, types, units and meanings for attributes, span names and metrics, so that the same thing is called the same everywhere. The Java agent on meter-api and the Python code in the portal both describe an HTTP request with http.request.method, http.route and http.response.status_code; one dashboard query works for both. The current version is 1.44.0 (as of October 2026).

Two details are worth remembering. http.route is the route template, which the conventions require to be low-cardinality; the full URL with a meter ID in it belongs in other attributes or not at all. And http.server.request.duration is measured in seconds, with recommended bucket boundaries from 5 ms to 10 s.

OpenTelemetry and Prometheus

Kestrel's metrics live in Prometheus, which normally pulls metrics, while OpenTelemetry SDKs normally push them. There are two ways to connect them:

• Scrape: the Collector (with its prometheus exporter) or the SDK exposes a /metrics endpoint, and Prometheus scrapes it as usual.
• Push with OTLP: Prometheus has its own OTLP receiver. It is off by default, because Prometheus runs without authentication unless you configure it, and is enabled with the flag --web.enable-otlp-receiver. It then accepts OTLP/HTTP metrics on /api/v1/otlp/v1/metrics; the gateway above uses it. Older articles mention a feature flag otlp-write-receiver; that has been replaced since Prometheus 3.0.

Pushed data needs translating, because OpenTelemetry and Prometheus name things differently. By default (translation_strategy: UnderscoreEscapingWithSuffixes) Prometheus replaces dots with underscores and adds the unit and, for counters, _total — the Prometheus naming habits. The resource attribute service.name becomes the job label and service.instance.id the instance label. A simplified version of the name translation:

import re

UNIT_WORDS = {"s": "seconds", "ms": "milliseconds", "By": "bytes"}

def prometheus_name(name, unit, kind):
    """Simplified version of the default OTLP translation in Prometheus
    (UnderscoreEscapingWithSuffixes): escape, add the unit, add _total."""
    result = re.sub(r"[^a-zA-Z0-9_:]", "_", name)
    if not unit.startswith("{"):              # {reading} is only an annotation
        word = UNIT_WORDS.get(unit, unit)
        if not result.endswith("_" + word):
            result += "_" + word
    if kind == "counter":
        result += "_total"
    return result

METRICS = [  # (OpenTelemetry name, unit, instrument)
    ("http.server.request.duration", "s", "histogram"),
    ("meter.readings.received", "{reading}", "counter"),
    ("system.network.io", "By", "counter"),
    ("billing.export.duration", "s", "gauge"),
]
for name, unit, kind in METRICS:
    print(f"{name:<29} {unit:<10} {kind:<10} -> {prometheus_name(name, unit, kind)}")

print()
for attribute in ["http.route", "http.response.status_code", "deployment.environment.name"]:
    print(f"{attribute:<29} -> label {re.sub(r'[^a-zA-Z0-9_]', '_', attribute)}")

http.server.request.duration  s          histogram  -> http_server_request_duration_seconds
meter.readings.received       {reading}  counter    -> meter_readings_received_total
system.network.io             By         counter    -> system_network_io_bytes_total
billing.export.duration       s          gauge      -> billing_export_duration_seconds

http.route                    -> label http_route
http.response.status_code     -> label http_response_status_code
deployment.environment.name   -> label deployment_environment_name

Other resource attributes do not become labels on every series — that would multiply the label set of everything. Prometheus stores them once per service instance on a special series called target_info, and you join them in when you need them, or you promote a few to labels in prometheus.yml. The documentation also recommends accepting out-of-order samples, because batching Collectors can deliver them late:

storage:
  tsdb:
    out_of_order_time_window: 30m
otlp:
  promote_resource_attributes:
    - service.version
    - deployment.environment.name

Queries then look like any other PromQL:

# Requests per second, from the OpenTelemetry histogram
sum by (job, http_route) (rate(http_server_request_duration_seconds_count[2m]))

# Add a resource attribute that was not promoted, from target_info
rate(http_server_request_duration_seconds_count[2m])
  * on (job, instance) group_left (telemetry_sdk_language)
target_info

The second query multiplies by target_info, whose value is always 1, matching on job and instance and copying one label across; see PromQL for how on and group_left work.

Outdated advice and common mistakes

OpenTelemetry moves quickly, and search results are full of configurations that worked two years ago. As of October 2026:

• Exporter names. In Collector v0.144.0 (January 2026) the otlp exporter was renamed otlp_grpc and otlphttp became otlp_http. The old names still work as deprecated aliases, so old configurations keep running — until the aliases are removed.
• Filter processor. The traces: span: and metrics: metric: lists are deprecated in favor of trace_conditions, metric_conditions and log_conditions with an explicit context prefix such as span..
• Batching. Older guides put a batch processor in every pipeline. Batching now lives in the exporter's sending_queue; a queuebatch processor (v0.158.0) replaces the legacy batch processor where you need one.
• Logs are stable, OpenTelemetry has graduated in the CNCF, and profiles are not stable yet — three statements that many older articles get the wrong way round.
• Jaeger. Tutorials with Jaeger agents and Jaeger client libraries describe Jaeger v1, which is no longer published. Jaeger v2 is itself built on the Collector; instrument with OpenTelemetry and send OTLP (see Distributed Tracing).