containerd and Container Runtimes

Overview

Overview

containerd began as an internal component inside Docker. It was donated to the CNCF in 2017 and graduated as a top-level CNCF project in 2019.

It is now the default container runtime in most Kubernetes distributions, including EKS, GKE, AKS and kubeadm-provisioned clusters. It replaced dockershim, which was removed from the kubelet in Kubernetes 1.24.

Two OCI specs, one bridge

• OCI Image Specification — A manifest referencing a config blob and compressed layer tarballs in a content-addressable layout — Pulls and stores images
• OCI Runtime Specification — The config.json bundle: namespaces, cgroups, capabilities and the process to run — Generates the bundle for runc

containerd bridges the two specs by converting images into runtime bundles.

containerd, the shim and runc

• containerd — Long-running daemon — Manages images, snapshots, containers as records and tasks as running processes
• containerd-shim — Lives as long as the container — Proxies IO and collects exit codes, so containerd need not be the parent of every container
• runc — Short-lived CLI — Sets up namespaces and cgroups, forks the container init process, then exits

> [!IMPORTANT] containerd is the high-level runtime. runc is the low-level OCI runtime that actually creates the Linux primitives.

Kubernetes and the CRI

The kubelet talks to containerd through the Container Runtime Interface, a gRPC API defined by Kubernetes. The CRI plugin inside containerd translates CRI calls into containerd operations.

The kubelet calls RunPodSandbox to create the shared network and IPC namespace for a Pod, then CreateContainer and StartContainer for each container. The sandbox is a "pause" container whose only job is to hold the network namespace alive for the Pod's lifetime.

Which CLI

• ctr — Native containerd API, bundled with the daemon — Debugging and inspecting internal state
• nerdctl — containerd API, Docker-compatible UX — docker run style work without dockerd
• crictl — CRI gRPC — Seeing pods, containers and images exactly as the kubelet sees them

Snapshotters

Docker storage drivers are global and monolithic. containerd snapshotters are pluggable and can be selected per container.

• overlayfs — The overwhelmingly common choice on Linux 4.0 and later
• devmapper — Better IO isolation on block storage
• native — Safe fallback when the kernel lacks overlayfs support

> [!TIP] Knowing which snapshotter is in use is critical when diagnosing disk space issues or running containerd inside containers, which is common in CI.

Key concepts

OCI Image Spec vs OCI Runtime Spec

The Open Container Initiative defines two independent specifications that together describe the full lifecycle of a container. The Image Specification defines how a container image is stored and distributed. An OCI image consists of a manifest (a JSON document listing a config blob and one or more layer blobs by digest), a config blob (a JSON document describing the image's environment, entrypoint, cmd, and layer diff IDs), and layer blobs (gzip-compressed tarballs of filesystem changes). All blobs are stored in a content-addressable layout keyed by SHA-256 digest. The Runtime Specification defines the bundle that a low-level runtime like runc receives when it is told to create a container. A bundle is a directory containing a rootfs subdirectory (the unpacked container filesystem) and a config.json file (a JSON document specifying linux namespaces, cgroups, capabilities, seccomp, mounts, and the process to run). containerd bridges the two specs by pulling OCI images into its content store, using the snapshotter to unpack layers into a prepared rootfs, then generating a config.json from the image config plus any overrides, and presenting the resulting bundle to runc.

containerd-shim-runc-v2 and Shim Architecture

The containerd shim is a small supervisor process that decouples containerd from the running container process. When containerd creates a task, it forks a containerd-shim-runc-v2 binary and passes it the bundle path and a Unix socket address. The shim then forks runc create, which configures the container, and runc start, which executes the init process. runc exits after each phase; the shim remains alive. The shim serves several critical purposes. First, it allows containerd itself to be restarted without killing running containers, because the shim is the actual parent of the container init process (or uses subreaping via the shim's own init). Second, it multiplexes the container's stdio streams via a FIFO or vsock so clients can attach and detach without interrupting the container. Third, it monitors the container exit event and reports it back to containerd over the shim's gRPC socket so containerd can update the task state. The v2 shim API is a plugin interface, meaning containerd can load alternative shim implementations. This is how kata-containers and gVisor integrate: they provide custom shims (containerd-shim-kata-v2, containerd-shim-runsc-v1) that replace runc with a VM-based or sandbox-based runtime while presenting the same shim API to containerd.

Kubelet CRI gRPC Call Sequence

The Container Runtime Interface is a gRPC API that kubelet uses to drive any conformant container runtime. containerd implements CRI via its CRI plugin. The call sequence for starting a Pod has three required phases. Phase one is RunPodSandbox. kubelet calls RunPodSandbox with a PodSandboxConfig that includes the pod name, namespace, labels, and network configuration. containerd creates a pause container (the sandbox) that holds the network namespace, IPC namespace, and UTS namespace for the pod. The CNI plugin is invoked here to wire up networking. RunPodSandbox returns a sandbox ID. Phase two is CreateContainer. For each container in the Pod spec, kubelet calls CreateContainer with the sandbox ID and a ContainerConfig describing the image, command, env, mounts, and resources. containerd pulls the image if absent, prepares a snapshot, and records the container. CreateContainer returns a container ID but does not start the process. Phase three is StartContainer. kubelet calls StartContainer with the container ID. containerd calls into the shim which calls runc start, launching the init process inside the already-prepared namespaces of the sandbox.

Snapshotters: overlayfs, devmapper, and native

The snapshotter subsystem in containerd is responsible for constructing the layered filesystem that becomes the container rootfs. Unlike Docker's monolithic storage driver, containerd snapshotters are pluggable and selected in /etc/containerd/config.toml. The overlayfs snapshotter is the default on Linux kernels 4.0 and above. It uses the kernel overlayfs union filesystem to stack read-only image layers under a single read-write upper layer without copying files. This makes container creation very fast (microseconds for the merge) and efficient for disk space because layers are shared across all containers using the same image. The devmapper snapshotter uses Linux device mapper thin provisioning to give each container its own block device snapshot. This is slower to create than overlayfs but provides true block-level isolation and supports selinux labeling more reliably on older kernels. It was the required driver for RHEL 7 nodes before overlayfs support matured. The native snapshotter simply does a full directory copy for each layer, providing a reliable but disk-intensive fallback. It is used in environments that do not support overlayfs, such as running containerd inside a Docker container in CI without privileged mode.

nerdctl vs Docker CLI

nerdctl is a Docker-compatible CLI for containerd developed by the containerd project. It provides the same commands and flags as docker (nerdctl run, nerdctl build, nerdctl ps, nerdctl push) but communicates directly with containerd via the containerd gRPC API, bypassing dockerd entirely. This makes nerdctl suitable for environments where Docker daemon is not installed, such as Kubernetes nodes running pure containerd. nerdctl supports most Docker features including BuildKit integration for builds, docker-compose compatible files via nerdctl compose, rootless containers via rootlesskit, lazy image pulling via stargz-snapshotter, and namespaces for multi-tenant workloads. Key behavioral differences from Docker: nerdctl uses containerd namespaces (default is "default" for CLI, "k8s.io" for Kubernetes). Containers created by kubelet live in the k8s.io namespace and are visible with nerdctl --namespace k8s.io ps, not in the default namespace. Docker does not have this namespace concept.

ctr for containerd Debugging

ctr is the low-level containerd CLI bundled with the daemon binary. Unlike nerdctl which aims for Docker UX compatibility, ctr exposes the raw containerd API surface and is primarily a debugging tool. It lacks features like automatic DNS setup, port mapping, and user-friendly defaults that nerdctl provides. ctr operates on containerd namespaces. The default namespace for ctr is "default". Kubernetes uses the "k8s.io" namespace. You must specify the namespace explicitly with the -n flag to interact with Kubernetes containers. ctr is useful for inspecting the content store directly, checking snapshots, examining task state, and triggering garbage collection. It is the correct tool when you need to understand containerd's internal state rather than just managing containers at a high level.

containerd Content Store and Garbage Collection

The content store is containerd's immutable blob store, located at /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/. Every image layer tarball, image manifest, and image config blob is stored here as a file named by its SHA-256 digest. The store is strictly append-only and content-addressed, meaning the same blob is never stored twice regardless of how many images reference it. Leases are the mechanism containerd uses to prevent premature garbage collection of blobs that are in the process of being used. When a client pulls an image, it creates a lease that pins the content blobs until the pull is complete and the image record is committed to metadata. If the pull fails and the lease expires, the garbage collector can safely remove partially pulled blobs. The garbage collector marks all content referenced by image records and snapshot chains in the metadata store, then sweeps any blobs not in the marked set. On Kubernetes nodes, the kubelet's image garbage collection (configured via --image-gc-high-threshold and --image-gc-low-threshold) triggers containerd image removal when disk usage exceeds thresholds, which in turn allows the containerd GC to reclaim blob space.

When to use it

• When migrating a Kubernetes cluster from dockershim to a direct CRI runtime and you need to verify containerd configuration, socket paths, and sandbox image settings in /etc/containerd/config.toml
• When debugging a node-level container failure in Kubernetes where kubectl describe pod gives insufficient detail and you need crictl pods, crictl inspect, or crictl logs to examine CRI-level state
• When running containers in environments where Docker daemon overhead is unacceptable and you want to use nerdctl or ctr to manage containers directly against containerd
• When building a custom Kubernetes node image or distro and you need to select and configure the appropriate snapshotter (overlayfs vs devmapper) for the underlying storage
• When implementing a CI pipeline that builds and pushes OCI images without Docker, using tools like buildkit or buildah that push directly to containerd's content store
• When investigating container startup latency and you need to trace the full containerd to shim to runc call path using containerd events or metrics
• When diagnosing disk exhaustion on Kubernetes nodes and you need to inspect containerd's content store, snapshot usage, and trigger garbage collection via ctr content ls and ctr snapshots ls

Interview questions

• Walk me through the complete stack of what happens when you run docker run nginx on a modern Linux machine with Docker 24+ installed.
• What is the purpose of the containerd shim, and what happens if you kill it while a container is running?
• Describe the exact sequence of CRI gRPC calls that the kubelet makes to containerd when scheduling a new Pod with two containers.
• What are the key differences between nerdctl and docker CLI, and when would you choose one over the other?
• How does the containerd snapshotter differ from Docker's storage driver model, and what problems does the snapshotter design solve?

More DevOps topics

• Configuration Drift
• Container Errors — OOMKilled, Exit Codes, Probes and Stale Images
• Container Fundamentals
• Container Image Hardening
• Container Probes — Liveness, Readiness, Startup
• Container Runtime Security
• Continuous Delivery vs Deployment
• Continuous Integration
• Control Plane Errors — API Server, RBAC, Webhooks, Quotas and Finalizers
• Cost-Optimized CI Runner Fleet
• Cross-Compilation and Sysroots
• Crossplane — Cloud Control Plane