APM Platforms

Application Performance Monitoring — integrated traces + metrics + logs + RUM + profiling in one product. Datadog (market leader), New Relic (Java/.NET strong), Dynatrace (auto-discovery), Honeycomb (high-cardinality, BubbleUp). Vendor APM vs OSS Grafana stack.

Overview

One product for every signal

Application Performance Monitoring (APM) platforms bundle traces, metrics, logs, real user monitoring (RUM), synthetic monitoring, profiling and SLO tracking into one integrated product. The category emerged around 2008 (New Relic and AppDynamics both launched in 2008) and matured through the 2010s.

The 2026 landscape

Datadog — 2010; IPO 2019; ~$2.5B revenue in 2024 — Market leader: polished UX, deep cloud integrations, continuous profiling; RUM, APM, Logs, Synthetic and DBM in one product
New Relic — 2008 — Long-time #2; moved to per-data-ingest pricing in 2020; strongest in the Java and .NET ecosystem
Dynatrace — 2005 — Enterprise focus: OneAgent, a single binary that auto-discovers everything; Davis AI for causation analysis
Honeycomb — 2016, by ex-Facebook engineers — Built for high cardinality and multi-tenancy; BubbleUp anomaly-attribute analysis
AppDynamics — Acquired by Cisco 2017 — Java and .NET enterprise; less active development
Splunk Observability Cloud — Splunk acquired SignalFx 2019; Cisco acquired Splunk 2024 — Real-time streaming analytics
OSS alternative — Grafana stack: Prometheus, Tempo, Loki, Grafana, plus Pyroscope for profiling

The OTel impact

Before 2024, APM platforms had vendor-specific SDKs, so instrumentation locked you to a vendor. OpenTelemetry (see the OTel topic) standardized the SDK and wire protocol; by 2024-2026 every major APM vendor accepts OTLP natively. Switching vendors is a Collector config change, not re-instrumentation.

> [!NOTE] Vendors still recommend their proprietary agents for advanced features (Datadog continuous profiling, RUM, network monitoring), but core APM (traces, metrics, logs) is OTel-compatible.

Vendor APM vs OSS

UX: vendor UIs are far more polished than self-hosted Grafana; hours saved per debugging session add up — Cost: vendor APM at scale runs $50k-1M+/year; OSS at the same scale $5-30k/year in compute and storage, plus engineering time
Operational simplicity: the vendor handles ingestion, storage, query and scaling — Data sovereignty: telemetry holds business-sensitive data; some regulated and government orgs cannot send it to a vendor cloud
Integration breadth: Datadog ships 600+ integrations (AWS services, databases, queues, frameworks, major SaaS); self-hosting means building your own — Customization: custom storage tiers, retention policies and query patterns vendors don't support
Advanced features: continuous profiling (Datadog), causation analysis (Dynatrace Davis), BubbleUp (Honeycomb), business transaction tracking (AppDynamics) are hard to replicate — No vendor lock: OTLP-instrumented apps stay portable
Compliance: SOC 2, HIPAA, FedRAMP-Ready (some vendors); self-hosting needs equivalent investment

When to use which

Under 200 hosts: operational simplicity dominates and vendor cost is acceptable — Over 500 hosts: vendor cost crosses $200k+/year and OSS savings dominate
App-team-led observability: teams want to ship, and an SRE hand-off is friction — The team has K8s and observability ops capacity (an SRE or platform team)
Compliance needs (SOC 2, HIPAA, FedRAMP) covered by vendor certifications — Multi-cloud or hybrid, where vendor data egress is operationally complex
Polished UX is critical for support, sales and executive dashboards — A strong cost focus

> [!TIP] Most large orgs use both: vendor APM (Datadog or New Relic) for application teams, OSS Prometheus and Grafana for infrastructure, where volume is too high for vendor pricing.

Three concepts every answer needs

1. Integration is the value — Vendors integrate the signals plus RUM and synthetic in one UI for traces, metrics, logs and frontend performance
2. OTel changes the lock-in story — Apps instrument with OTel; the backend is a Collector config change; switching vendors is far easier since 2024
3. Cost vs features — Vendor APM at $50-200/host/month vs OSS at ~$10-50/host/month: operational savings vs feature polish

Key concepts

Datadog APM — features and pricing: Datadog APM is the most-used commercial APM in 2026. Bundled with Datadog's broader platform. Core APM features: - Trace ingestion via dd-trace SDK or OTLP. - Trace search by traceID, service, route, status, custom attributes. - Flame graph visualization with span detail. - Service map auto-generated from trace flow. - Trace analytics: aggregations over trace data ("p99 latency by route by service"). Adjacent features: - Continuous profiling: in-prod CPU/memory/goroutine profiling. ~5% overhead. Identifies hot functions, lock contention. - Watchdog: auto-anomaly detection on traces and metrics. ML-based. - DBM (Database Monitoring): per-query performance tracking. SQL Server, PostgreSQL, MySQL, MongoDB, Redis. - Network Monitoring: TCP/IP-level observability between services. - RUM: browser SDK for frontend perf + JS errors. - Synthetic Monitoring: scripted user journeys + API uptime. - Logs (Datadog Logs): full-text indexed; pivots from APM. - Metrics (Datadog Metrics): per-host dashboards + custom metrics. - Error Tracking: error grouping, regression detection, ownership routing. Pricing model (2026): - Pro plan: $15-23/host/month. - Enterprise: $23-40/host/month + per-feature. - APM add-on: $31/host/month (1M ingested spans). - Continuous Profiling: $2/host/month. - DBM: $70/host/month per host monitoring DB. - Logs: $0.10/M ingested + $1.27-3.75/M indexed (depends on retention). - RUM: $1.50/1k sessions. - Synthetic: $5/10k API tests, $12/1k browser tests. Cost example (200 hosts moderate APM + Logs + RUM): - Pro: 200 × $23 = $4.6k/month. - APM: 200 × $31 = $6.2k/month. - Logs (50GB/day ingested, 30d retention): ~$3k/month. - RUM (10M sessions/month): $15k/month. - Total: ~$30k/month, ~$360k/year. This scales aggressively. At 1000 hosts with full feature set: $1M+/year. Why teams pay: - UI is best-in-class. Engineers find issues fast. - Service map auto-built; understand architecture without effort. - Watchdog catches anomalies you didn't know to alert on. - Continuous profiling catches inefficiencies metrics miss. - Vendor handles scale/HA. Common Datadog patterns: Trace correlation with logs: every Datadog log includes trace_id when emitted in a span scope. Click span → "View related logs" auto-filters by trace_id. Trace-level alerts: alert on trace.http.request.errors{service:payments}.as_rate() > 0.05 for 5min. SLO tracking: define SLO target; Datadog computes burn rate; alerts on fast-burn (5%+ budget in 1h) and slow-burn (full month budget in 30d). Deployment tracking: tag traces with version label (e.g., version:v1.2.3). Compare metrics across versions; auto-detect regression after deploy. Datadog vs OTel migration: dd-trace SDK has more features than OTel SDK currently (continuous profiling integration, automatic propagation across some non-standard protocols). Migration to pure OTel sometimes loses features. Pragmatic path: instrument with OTel; export to Datadog via OTLP. Keep dd-trace for specific advanced features (profiling, RUM). Hybrid SDK approach.
Honeycomb — BubbleUp and high-cardinality: Honeycomb's distinctive feature: BubbleUp. Solves "why is this slow?" at scale. The problem: in production microservices, p99 latency is 5x p50. Why? Average aggregation hides it. Cardinality is too high to drill manually. BubbleUp approach: 1. Engineer selects an outlier set in a heatmap (slow traces, error traces). 2. Honeycomb compares the outlier set's attribute distribution against the rest of the population. 3. Surfaces attributes where outliers differ significantly. "Outliers have user.region=eu-west-1 (90% of outliers vs 30% of population). Outliers have plan_tier=enterprise (40% of outliers vs 5% of population)." Result: hypothesis generated automatically. Engineer pivots to investigate that segment. Equivalent in Datadog/New Relic: manual filter + groupBy iteration. Maybe surfaces in Watchdog (Datadog) for known-pattern anomalies. Honeycomb's BubbleUp is automatic for arbitrary attribute combinations. Why high-cardinality matters: Multi-tenant SaaS: tenant_id is high-cardinality (10k+ unique). Datadog/Prometheus charge per-cardinality; Honeycomb storage scales linearly with row count, not cardinality. ML inference: model_version, input_features, prediction_class — all high-cardinality. Honeycomb storage handles; many APMs can't. Per-user debugging: user_id as attribute on every span. Find traces for support-reported user issue trivially. Honeycomb's storage model: Columnar (similar to ClickHouse). Compressed columns. Aggregations are fast even on high-cardinality columns. Trade-off: full-text search across log content is less ergonomic than Elastic. Honeycomb is trace-and-metric-first; logs are events with attributes (no full-text). Pricing model: Per-event (= per span). Tiered by retention. Sample free tier: 20M events/month, 60-day retention. Useful for OSS / small teams. Pro: $130/month for 100M events. Enterprise: custom pricing. Typically $20-100k/year for moderate-large orgs. Cheaper than Datadog for trace-heavy workloads at scale; Datadog wins on RUM + Logs combined pricing. When Honeycomb wins: Multi-tenant SaaS where per-tenant debugging matters. ML applications with high-cardinality features. Teams that value debugging UX over generic dashboard polish. Engineering culture that's data-savvy and willing to learn Honeycomb's query language. When Honeycomb loses to Datadog: Need integrated RUM + Synthetic + Logs. Need polished out-of-box dashboards. Don't have high-cardinality data; standard label dimensions work fine. Honeycomb's adoption is concentrated in observability-savvy orgs (Slack, Vox Media, Tailscale, observability tooling vendors themselves). Smaller market share than Datadog but loyal users.
Continuous profiling — the fourth pillar: Continuous profiling samples application performance in production. Measures: CPU, memory allocations, goroutine waiting time, lock contention, file I/O. Different signal from traces and metrics. What it shows: Per-function CPU time. "checkPolicy() consumed 35% of CPU last hour." Memory allocation hotspots. "createSession() allocated 200MB." Lock contention. "DB connection pool mutex held 8% of time." Off-CPU analysis. "Goroutines waiting on syscalls 15% of time." How it works: In-process sampler runs at low frequency (10-100 Hz). Captures stack trace per sample. Aggregates samples → flame graph. Overhead: 1-5% CPU. Acceptable for production. Stack traces include line numbers; click flame graph element → source code line. Tools: Datadog Continuous Profiler. Bundled with Datadog APM. Per-language profilers (Go, Java, Python, .NET, Node, Ruby, PHP). $2/host/month add-on. Pyroscope (Grafana Labs, OSS). Independent of Grafana stack but integrates. Apache 2.0. Self-host or Grafana Cloud Pyroscope. Parca (CNCF Sandbox). eBPF-based profiling. No SDK in app; profiles via kernel-level instrumentation. Whole-host profiling. Pyrra. Pyroscope-compatible profiling for K8s. When you need it: CPU spikes that metrics show but root cause is unclear. "Service is using 80% CPU; metrics don't show why." Memory growth without OOM. Slow leak. Profiler shows which function is allocating. Tail latency mystery. Median is fine; p99 is bad. Profile high-latency requests; find the slow function. Why traces alone don't suffice: Traces show external calls (DB, HTTP). Profiling shows in-process work (loops, JSON parsing, regex evaluation, string copies). A 500ms span "process_request" with no children: tracing dead end. Profiler shows internal breakdown. Profiling vs APM trace span timing: Spans measure wall-clock duration of explicit boundaries. Profiling samples actual CPU usage continuously. Span: "stripe_call took 1500ms". Profiler: "while waiting for stripe, the calling function did nothing useful (off-CPU)." Different views. Production profiling concerns: Stack trace privacy. Stack traces may include data passed through functions. Some tools redact; verify. Symbol resolution. Production binaries are stripped; need separate symbol upload. Otherwise profile shows hex addresses. Sampling consistency across services. Different sample rates → can't compare directly. Cost: storage for profiles is meaningful. Compressed flame graph = ~100KB; 1000 hosts × 1 profile/min × 1 day = ~140GB/day of profile data. S3 storage for Pyroscope: cheap. Vendor: bundled. OSS alternative for profiling: Pyroscope: self-host on K8s. Helm chart. Apps instrument with language-specific Pyroscope SDK or OTel Profiler (CNCF Incubating; spec stabilizing 2025-2026). OTel profiling signal will likely standardize the field by 2027+. Until then, vendor-specific (Datadog Profiler, Pyroscope) or eBPF-based (Parca) are the options.
RUM and Synthetic monitoring: Two complementary frontend monitoring approaches. Real User Monitoring (RUM): Browser SDK (or mobile SDK) tracks every user session. Captures: - Page load metrics: TTFB, FCP, LCP, FID, CLS, INP. - Resource timing: every script, image, API call timing. - JavaScript errors with stack traces. - User interactions: clicks, scrolls, route changes. - Custom user actions. - Session replay (some vendors): pixel-level recording of user sessions for debugging. Embedded as tag: What RUM answers: - "What's the slowest page load for actual users?" - "Where do users hit JS errors?" - "Why do users abandon checkout?" Pros: - Real user data; perfect representativeness. - Catches issues that synthetic tests miss (specific browser bugs, network conditions, ad blockers). Cons: - Privacy concerns. PII in URLs, form data; redaction essential. - Performance overhead on the page. Modern SDKs are <50KB; older were heavier. - Cost per session at scale ($1-5/1k sessions for Datadog/New Relic). Vendors: Datadog RUM, New Relic Browser, Sentry Performance, FullStory, LogRocket, Rookout. OSS: limited (no first-class OSS RUM at scale). Synthetic monitoring: Scripted browser tests + API uptime checks running on schedule from multiple regions. Configures via UI or YAML: What Synthetic answers: - "Is the homepage up from all regions?" - "Did the checkout flow break in production?" - "Did p99 latency for the API endpoint regress?" Pros: - Predictable; runs on schedule regardless of traffic. - Catches issues before users report. - Multi-region coverage tests CDN, network paths. Cons: - Doesn't catch user-specific edge cases. - False positives: scheduled test from us-east-1 fails because that one region's network was glitchy. - Cost per test ($5-12/1k tests). Vendors: Datadog Synthetic, Pingdom, Catchpoint, Checkly, UptimeRobot. Combination pattern: Synthetic for known-critical journeys (uptime, checkout, login). Few tests, high reliability. RUM for breadth (catch issues real users hit). Sample every session. Both tied to APM trace context — RUM session attached to backend trace; synthetic test attached to backend trace. Engineer can see "real user experienced 5s page load" → "click for backend trace" → drill to slow span. OSS alternative: Lighthouse CI (Google) for synthetic in CI pipelines; no first-class OSS RUM.
Recipe: migrate from Datadog APM to Grafana stack: Cost-driven migration. Multi-quarter project. Common at $200k+/year Datadog spend with mature K8s ops. Phase 1: parallel ingest (1-3 months). Apps emit OTel telemetry. OTel Collector exports to Datadog AND Grafana stack simultaneously: Both systems receive identical telemetry. Compare for parity. Phase 2: dashboard parity (4-12 weeks). Datadog dashboards → Grafana. Manual rewrite required: - Datadog Notebook → Grafana Dashboard. - Datadog query syntax → PromQL / LogQL / TraceQL. - Datadog widgets → Grafana panels. Auto-import plugins exist (community Grafana plugins) but coverage is partial. Estimate: 30-100 critical dashboards × ~30-60 min each = 30-100 engineering hours. Phase 3: alert parity (4-8 weeks). Datadog Monitors → Prometheus alert rules + Loki alert rules. Each monitor reviewed: - PromQL/LogQL equivalent. - Threshold tuning (Datadog and Prometheus aggregate slightly differently). - Alertmanager routing config. Critical: run new alerts in audit mode (notification to alert-test channel, not page) during parallel phase. Tune thresholds before cutover. Estimate: 50-200 alerts × ~15-30 min each = 12-100 hours. Phase 4: APM-specific feature replacement. Datadog Service Map → Tempo's metricsGenerator service-graph + Grafana node-graph plugin. Functional equivalent; less polished UX. Datadog Watchdog → Grafana Mimir's anomaly detection (newer; less mature). Datadog Continuous Profiler → Pyroscope (Grafana Labs OSS). App-level changes to swap profiling SDK. Datadog DBM → Database-specific Prometheus exporters (postgres_exporter, mysqld_exporter). No equivalent of query-level monitoring out of box. Datadog RUM → Sentry / FullStory / commercial RUM (no first-class OSS RUM at scale). Datadog Synthetic → Checkly / Lighthouse CI (limited OSS). Phase 5: cutover (2-4 weeks). Disable Datadog ingestion in OTel Collector. Datadog account read-only mode for 30-60 days for fallback. Then cancel. Cost analysis: Datadog (typical mid-large): $200-600k/year for 200-host fleet with full APM + Logs + RUM. Grafana stack at same scale: - Compute: ~$3-10k/month for Tempo/Mimir/Loki. - S3: ~$200-1k/month. - Engineering: 0.5-1 full-time observability engineer (~$200-400k/year fully loaded). Total: ~$240-450k/year. Marginal savings; significant operational shift. At 1000+ host scale: Datadog $1-2M/year vs Grafana stack $250k-500k/year. 2-4x savings. Operational shift: Datadog: app teams use UI; no infrastructure to operate. Grafana: SRE / platform team operates Tempo, Mimir, Loki, Grafana, Pyroscope. Helm upgrades, scaling, capacity planning, S3 cost management. Engineering capacity required: 0.5-2 engineers depending on scale. When migration succeeds: Existing K8s expertise + observability ops capacity. Cost has reached pain point ($500k+/year). Multi-cloud / on-prem requirements that complicate Datadog egress. Engineering culture comfortable with OSS. When migration fails: Insufficient SRE / platform team. Dashboards/alerts atrophy. Underestimate replication time. 6 months becomes 18 months. Don't replace advanced features (Watchdog, BubbleUp). Engineering productivity drops. App teams resent the UX downgrade. Adoption plateaus. Hybrid as alternative: Keep Datadog for app teams (Logs + RUM + Synthetic). Move infrastructure metrics to Prometheus + Grafana (volume too high for Datadog cost). Volume gets cut in half; Datadog spend drops 40-60%; less migration work. Many large orgs (Shopify, Cloudflare) run hybrid permanently. Pure migration is uncommon at hyperscale.

When to use it

Integrated observability without operational overhead
App teams that want to focus on shipping, not running observability infra
Polished UX requirements (customer-facing dashboards, executive reporting)
Advanced features (continuous profiling, business transaction tracking, BubbleUp)
Compliance requirements (SOC 2, HIPAA, FedRAMP) where vendor certifications cover

Interview questions

When do you pick Datadog over OSS Grafana stack?
What advanced APM features matter most?
How does APM vendor cost scale with usage?
Walk through OTel migration from vendor SDK to vendor-neutral.
Quick-fire interview answers — APM platforms.