OpenTelemetry

CNCF Incubating project; second-most-active CNCF project after Kubernetes. Vendor-neutral spec for traces, metrics, logs. SDKs in 11+ languages, Collector for processing/routing, OTLP wire protocol. Replaces vendor-specific agents (Datadog, New Relic, Jaeger client libs).

Overview

Instrument once, send anywhere

OpenTelemetry (OTel) is the CNCF Incubating standard for application telemetry: traces, metrics and logs, vendor-neutral. It started as the 2019 merger of OpenTracing and OpenCensus and has been CNCF Incubating since 2021. By 2024 it was the second-most-active CNCF project after Kubernetes itself (commits, contributors, releases).

> [!IMPORTANT] Switching backends becomes a Collector config change. The app is never re-instrumented.

What changed with OTel

Pre-OTel (2010s) — Every vendor had its own SDK: Datadog tracing client, New Relic agent, Jaeger client libraries, AppDynamics agent, Dynatrace OneAgent — Rewrite instrumentation; re-instrument the app for each vendor
Post-OTel (2020+) — App instruments once with the OTel SDK; telemetry flows to the OTel Collector, which exports to the backend you choose — Change Collector config; sending the same data to two backends is one config line

Industry adoption

CNCF — Kubernetes itself emits OTel; most CNCF projects (Argo CD, Flux, Istio, Envoy) have native OTel support
Vendors — Datadog, New Relic, Honeycomb, Splunk, Dynatrace and AppDynamics all accept OTLP natively; most are migrating off proprietary agents toward OTel-based instrumentation
OSS backends — Jaeger, Tempo, Prometheus, Loki, OpenSearch and ClickHouse all support OTLP ingestion

Signal status (as of 2026)

Traces — GA: stable spec, mature SDKs
Metrics — GA: stable spec, mature SDKs
Logs — GA, stable as of 2024; catching up, some semantic conventions still evolving
Profiling — Experimental continuous-profiling signal, similar to Pyroscope and Parca; spec stabilizing 2025-2026

Why OTel wins, and where it has friction

Vendor neutrality: multi-backend, switch-backend and multi-vendor strategies are trivial; ends instrumentation lock-in — Configuration complexity: Collector config can run to hundreds of lines; receivers, processors and exporters interact; steep learning curve
Auto-instrumentation: Java and .NET byte-code agents instrument hundreds of libraries; Python, Node and Go wrappers cover common frameworks; often useful traces with zero code changes — Logs less mature: stable, but semantic conventions still solidifying; structured-logging patterns inconsistent across SDKs
Semantic conventions: standard attribute names (http.method, db.system, messaging.destination.name) let backends build dashboards without per-vendor schema knowledge — Performance overhead: auto-instrumentation adds 5-15% CPU in measured benchmarks; tunable but real
Active community: hundreds of contributors; backing from Google, Microsoft, AWS, Datadog, Honeycomb, Splunk and others — Deployment complexity: SDK, agent collectors, gateway collectors and backends; more moving parts than a drop-in vendor agent

When to pick OTel

Greenfield observability with no existing investment: the default in 2026 — Tiny apps: a drop-in vendor agent is faster to start; adopt OTel if you outgrow it
Multi-backend strategy: "Datadog for production, Tempo for dev and staging" is trivial — Single-vendor commitment: Datadog's proprietary agent has more features than OTel as of 2026 (continuous profiling, network monitoring)
Vendor migration: instrument with OTel, route via the Collector, migrate gradually — Vendor agents are evolving toward OTel compatibility but are still ahead in some areas
Multi-cloud: same instrumentation across AWS, GCP and Azure

Three concepts every answer needs

Three signals — Traces, metrics and logs, plus emerging profiling; each has its own SDK API but shares the Collector pipeline
Collector pipeline — Receivers, then processors, then exporters; pipeline-as-code, configurable per signal; agent plus gateway deployment pattern
OTLP wire protocol — Replaces vendor protocols: app to Collector via OTLP; Collector to backend via OTLP or a vendor-specific exporter

> [!TIP] Lead with these three in any OTel interview answer, then go deep on whichever the interviewer probes.

Key concepts

Three signals: traces, metrics, logs: OTel standardizes three observability signal types. Traces: distributed traces span service boundaries. A trace = collection of spans; each span represents one unit of work. Each span has: name, start/end time, status (OK/Error), attributes (key-value), events (timestamped logs within span), links (to other traces). Trace context propagation: W3C Trace Context standard (traceparent, tracestate HTTP headers). All major HTTP/gRPC clients propagate by default in instrumented code. Metrics: time-series numeric data. Three primary types: - Counter: monotonically increasing (request count, error count). - UpDownCounter: can decrease (in-flight requests, queue depth). - Histogram: distribution of values (request latency, payload size). Metrics are aggregated in-process before export (delta or cumulative aggregation; configurable). Logs: structured log records. Less mature than traces/metrics but stable as of 2024. OTel logs preserve the trace context — log records emitted during a span's scope link to that span. In Honeycomb / Datadog UI: click a span, see related logs. Three signals correlate via trace context: traces have IDs; metrics tagged with exemplars (linkable spans); logs reference trace IDs. Powerful debugging when you can pivot from "high error rate" metric to "the failing span" trace to "the error log line" log.
Auto-instrumentation: Most app teams don't want to hand-write tracer.startSpan() everywhere. Auto-instrumentation hooks common frameworks transparently. Java — javaagent JAR: Auto-instruments 100+ libraries: HTTP servers (Tomcat, Jetty, Spring), HTTP clients, JDBC, Kafka, Redis, gRPC, Hibernate, Cassandra, etc. Zero code changes. .NET — .NET Auto-Instrumentation similar: Python — opentelemetry-instrument wrapper: Wraps Flask, Django, FastAPI, requests, psycopg2, redis-py, kafka-python, etc. Node.js — opentelemetry-auto-instrumentations-node: Go — manual or via build tool. Go doesn't have JIT/byte-code rewriting; auto-instrumentation requires explicit imports per library: Rust — manual; ecosystem younger. Auto-instrumentation overhead: 5-15% CPU for typical web apps. Tunable via sampling, attribute filtering, span limits. Combine auto + manual: auto-instrumentation for HTTP/DB/external calls; manual for business-domain-specific spans (auth_check, fraud_detection, etc.). What's exposed by auto-instrumentation per HTTP request: - Server span: http.method, http.route, http.status_code, http.user_agent, network.peer.address, http.request.body.size. - DB span: db.system (postgresql), db.statement (full SQL — beware PII!), db.operation, db.connection_string (sanitized). - Outbound HTTP span: http.method, http.url, http.status_code, http.response_content_length. This is the killer feature — most observability value comes from auto-instrumentation; manual is for filling gaps.
OTel Collector — pipeline-as-code: Stateless service that receives OTel data, transforms it, exports to backends. Configured via YAML. Production Collector config: Pipeline order matters: - memory_limiter first (back-pressure before processing). - attributes/redact early (fewer attributes flow through). - batch last (efficient export). tail_sampling needs all spans of a trace at one Collector instance. Use load_balancing_exporter on agent collectors to send all spans of a trace to the same gateway instance.
Sampling — head vs tail: Sampling controls how much trace data you keep. Without sampling, high-traffic services produce billions of spans daily — storage cost prohibitive. Head sampling: decision made at trace start. Cheap; doesn't require all spans of a trace. Probabilistic head sampling — keep N% of traces: Pro: zero cost for dropped traces (no spans created). Con: random — no guarantee of catching slow traces or errors. A 1% sample misses 99% of error traces. Tail sampling: decision made after all spans of a trace arrive. Can sample based on trace properties (slow, error, specific route). Behavior: keep 100% of error traces, 100% of slow traces, 1% of normal traces. Storage ~1-5% of total; signal ~100%. Tradeoffs: - Head: cheap, simple, random. - Tail: expensive (Collector buffers all spans for decision_wait), informed (keeps interesting traces). Tail sampling requires: - Gateway Collector pattern (all spans of a trace go to same Collector). - Sufficient memory to buffer spans during decision_wait window. - load_balancing_exporter on agent Collectors to colocate spans by traceID. Head + tail combination: head sampling at SDK (drop 50% of bot traffic via attribute matching), tail sampling at Collector (keep error/slow + 1% baseline). Adaptive sampling (newer): adjust sampling rate based on traffic volume. Honeycomb's Refinery, Datadog's adaptive sampling. Less mature in OSS OTel as of 2026. Sampling decisions affect cost dramatically. 100k RPS service, 100% sampling = ~100M spans/hour. Tail sampling at 1% baseline + errors + slow = ~1-5M spans/hour. Storage cost differential is 10-100x.
Recipe: Kubernetes-native OTel deployment: Production deployment on EKS with agent + gateway pattern. Step 1 — install OpenTelemetry Operator. Manages Collector deployments via CRD. Step 2 — agent Collector as DaemonSet. One per node; collects from local pods. Step 3 — gateway Collector as Deployment. Centralized; does heavy processing. Step 4 — application instrumentation via Operator's Instrumentation CRD: Step 5 — annotate pods for auto-instrumentation: Operator injects javaagent via init container. Pod restart picks up instrumentation. Result: zero-code-change instrumentation across all annotated pods. Collector pipeline routes to backends.
Recipe: tail sampling for cost control: Tail sampling is the difference between $1k/month and $50k/month observability bills at high RPS. Without sampling: 100k RPS service × 10 spans/request × 30 days × 24h × 3600s = 26B spans/month. At Datadog APM pricing (~$0.10/spans/M), that's $260k/month. Untenable. With tail sampling: keep 100% of errors, 100% of slow (>1s), 1% baseline. Typically 2-5% of spans retained. Bill drops to $5-13k/month — same insight, fraction of cost. Tail sampling Collector config: Critical: tail sampling requires all spans of a trace at one Collector instance. Otherwise the Collector decides on partial trace and drops spans wrongly. Solution — load_balancing_exporter on agents: Agent computes hash(traceID) → picks gateway. All spans of a trace land at the same gateway instance. Tail sampling can decide correctly. Memory budgeting: gateway buffers spans for decision_wait. At 100k RPS / 5 gateway instances / 10s buffer / 100KB/trace = ~20GB memory per gateway. Provision generously. decision_wait tradeoff: short (1-5s) = lower memory + miss late spans. Long (30-60s) = higher memory + catch all spans. 10s typical for most architectures. When tail sampling fails: - Late-arriving spans (slow services that haven't responded by decision_wait) are dropped. Tune decision_wait per service architecture. - High cardinality of traceIDs (true uniqueness) means hash collisions are rare; if you see related traces split across gateways, check load_balancing_exporter is using traceID as routing_key. Cost monitoring: Collector itself emits metrics — otelcol_processor_tail_sampling_count_traces_sampled vs total. Track sampling rate; alert if it drifts.

When to use it

Greenfield observability — no existing investment in vendor agents
Multi-backend strategy (production vendor + OSS dev/staging)
Vendor migration (legacy Datadog → modern stack, Splunk → Honeycomb)
Multi-cloud / vendor-neutral instrumentation
Replacing legacy Jaeger/Zipkin clients (still works but OTel SDK is the future)

Interview questions

What is OpenTelemetry and what problem does it solve?
Walk through a production OTel deployment on Kubernetes.
Compare head sampling vs tail sampling — when to use each.
How does OTel handle correlation between traces, metrics, and logs?
Quick-fire interview answers — OpenTelemetry essentials.