CPU Runaway Process Investigation

Identify and fix runaway CPU consumers using profiling tools across Python, JVM, Go, and Node.js.

Overview

Overview

A CPU runaway process uses far more CPU than its workload should, often pinning one or more cores at 100%. It degrades co-located services and can end in OOM kills or instance failures. Unlike memory leaks, runaways are often intermittent and triggered by specific inputs, which makes them hard to reproduce in development.

Step 1: identify the process

  • top (or htop), sorted with the P key — Live per-process CPU, refreshed every 2 seconds
  • ps aux --sort=-%cpu \ — head -20 — Point-in-time snapshot, good for logging or scripts
  • kubectl top pods — CPU per pod
  • kubectl top pods --containers — CPU per container
  • CloudWatch agent procstat config — Per-process metrics for dashboards and alarms, by name or PID file

Step 2: profile what it is doing

Sampling the running process is the key diagnostic step.

  • Python — py-spy — Attaches without code changes or restart: py-spy top --pid 12345 shows the hottest functions live
  • JVM — async-profiler — Samples the native stack too, CPU and wall-clock: java -jar async-profiler.jar -d 30 -f profile.html $(pgrep java) writes a flame graph
  • Go — pprof (built in) — With net/http/pprof imported, fetch http://localhost:6060/debug/pprof/profile?seconds=30 and open it with go tool pprof
  • Node.js — V8 profiler — --prof flag, or Chrome DevTools protocol via --inspect

Common causes

  • Infinite loop (recursion without a correct base case, an event loop that feeds itself) — One core pegged at 100% immediately
  • Hot lock contention — Many threads spinning on a mutex; profiler shows high time in synchronization primitives
  • Catastrophic regex backtracking — A pattern like (a+)+ on crafted input takes exponential time
  • Large JSON parse or serialize in a hot path — CPU spikes tied to big payloads

Container CPU throttling

In Kubernetes, CFS enforces resources.limits.cpu through cpu.cfs_quota_us. When a container spends its quota for a period, it is throttled even if the host has idle CPUs.

> [!TIP] Check container_cpu_cfs_throttled_seconds_total in Prometheus, or cat /sys/fs/cgroup/cpu/cpu.stat inside the container (on cgroup v2 the file is /sys/fs/cgroup/cpu.stat).

Key concepts

CPU Flame Graph
A visualization of profiler samples showing which code paths consume CPU — width represents time spent, and the call stack is shown bottom-up, making hotspots immediately visible.
CFS Throttling
Linux kernel enforcement of CPU limits in containers — when a container exhausts its CFS quota in a scheduling period, it is throttled even if the host has idle CPUs.
Catastrophic Backtracking
A regex execution failure mode where ambiguous alternation or repetition patterns cause exponential time complexity on certain inputs, pinning the CPU.
py-spy
A sampling profiler for Python that attaches to a running process without code modification or restart, showing live CPU usage by function.
async-profiler
A low-overhead sampling profiler for JVM applications that captures native stack traces accurately for both CPU and wall-clock time analysis.

When to use it

  • Diagnosing a process pinning CPU at 100% with no obvious cause
  • Profiling a Python, JVM, Go, or Node.js service to identify CPU hotspots
  • Explaining why a container is slow despite the host having available CPU (CFS throttling)
  • Investigating catastrophic regex backtracking or N+1 queries causing CPU spikes
  • Setting up CloudWatch or Prometheus alarms for per-process CPU usage

Interview questions

  • How do you profile a running Java application for CPU hotspots without restarting it?
  • What is CFS throttling in Kubernetes and how do you diagnose whether it is affecting your service?
  • How do you identify and fix catastrophic regex backtracking?

Sign up free to read the full topic

More Troubleshooting topics

Start free on Cariara · Coding problems · System design studio