AIOps Telemetry Pipelines

AIOps telemetry pipelines ingest and normalize five signal types -- metrics, logs, traces, events, and topology -- from hundreds of services before machine learning can operate on them. OpenTelemetry has become the de-facto standard for collecting and routing all three core observability signals through a vendor-neutral protocol.

Overview

The telemetry pipeline in one picture

A telemetry pipeline collects, routes, transforms and delivers observability signals from running systems to the backends where AIOps algorithms analyze them. Before any anomaly detection, root cause analysis or alert correlation can run, raw signals must be gathered at scale, normalized to one format and reduced in volume.

The five signal types

Metrics — Numeric time series sampled at intervals: counters (total HTTP requests), gauges (current memory), histograms (latency buckets) — Prometheus, CloudWatch, Datadog
Logs — Timestamped text records, from free text to JSON — Elasticsearch, Splunk, Loki, CloudWatch Logs
Traces — One request's end-to-end journey: spans linked by a shared trace ID — Jaeger, Tempo, X-Ray, Zipkin
Events — Discrete state changes: deployments, feature flag toggles, auto-scaling, config changes — CI/CD, flag platforms
Topology — How services depend on each other — CMDB, Kubernetes namespace maps, Istio or Linkerd telemetry

> [!NOTE] Correlating an error-rate spike with a deployment five minutes earlier is a core AIOps use case, and without topology an alert on Service A cannot be traced to a failing dependency.

OpenTelemetry as the standard

OpenTelemetry (OTel) is the vendor-neutral standard for collection. Its wire protocol, OTLP, carries metrics, logs and traces in one format, so you do not need a separate agent per signal type.

Normalization

Different APM tools name the same concept differently, so AIOps platforms map everything to a canonical schema before ML can correlate across sources.

Data quality requirements

Completeness — A metric that stops reporting looks identical to one that dropped to zero
Freshness — Determines detection latency
Cardinality control — Prometheus label explosion can create millions of unique series and crash the TSDB
Metadata richness — Service name, team owner, environment and severity drive correlation and routing

Scale and reduction

A 500-service production system generates more than 500,000 metric samples per minute, more than 10,000 log lines per second and more than 1,000 trace spans per second.

> [!IMPORTANT] Aggregation, sampling and filtering in the pipeline are mandatory before signals reach anomaly detection.

Trace sampling strategies

Head-based — Keep or drop at the start of the trace — Fast, but cannot know whether the trace will contain errors
Tail-based — Holds the complete trace in a buffer and decides after it finishes — Captures all errors, at the cost of memory and processing overhead
Adaptive — Adjusts the sampling rate dynamically — Rate follows traffic volume and error rate

Pipeline topologies

> [!TIP] At high scale, Kafka decouples ingestion rate from downstream processing capacity.

Interview questions

Walk me through the end-to-end architecture of an OpenTelemetry pipeline from a microservice to an AIOps backend.
A 500-service production system is generating 500,000 metric samples per minute. How do you design the telemetry pipeline to make this workable for ML-based anomaly detection?
Explain the tradeoffs between head-based and tail-based trace sampling in an AIOps context.
What makes event signals different from the other four telemetry types, and how do they change AIOps correlation workflows?