Firecracker microVMs

Overview

Overview

Firecracker was open-sourced by Amazon Web Services in 2018. It is the reference implementation of a microVM: a virtual machine stripped of legacy hardware emulation and sized to run one workload rather than a general-purpose OS.

AWS needed to run millions of short-lived Lambda functions with sub-second cold starts and hard multi-tenant isolation. Containers share the host kernel, so they did not isolate enough; QEMU-class hypervisors started too slowly and used too much memory at Lambda scale.

Why each design choice matters

• VMM written in Rust — Eliminates classes of memory-safety bugs that plague C hypervisors such as QEMU; buffer overflows, use-after-free and data races are caught at compile time
• Minimal device model — Only virtio-net, virtio-blk and vsock, plus a serial console and an i8042 controller for orderly shutdown; no USB, sound or legacy ISA to probe
• Seccomp-BPF filters — Even arbitrary code execution in the VMM is limited to the syscalls Firecracker needs
• jailer — chroot, network namespace and cgroup confinement make each microVM an isolated island

> [!IMPORTANT] The VMM processes untrusted guest requests continuously, so a smaller emulated surface and a memory-safe language directly reduce the chance of a guest escaping to the host.

Ecosystem

• AWS Lambda — The original workload
• Kata Containers — Supports Firecracker as one of its VMM backends
• firecracker-containerd — A containerd shim so Kubernetes can schedule workloads into microVMs without modification

> [!TIP] Firecracker knowledge matters for platform engineering, serverless infrastructure and multi-tenant container security roles.

Key concepts

KVM Backend and VMM Process Model

Firecracker uses the Linux Kernel-based Virtual Machine subsystem as its hypervisor layer. KVM exposes /dev/kvm as a character device. The VMM opens this device, creates a VM file descriptor with the KVM_CREATE_VM ioctl, maps guest physical memory as anonymous mmap regions, creates one or more virtual CPU file descriptors with KVM_CREATE_VCPU, and then enters a run loop that calls KVM_RUN in a tight loop per vCPU thread. Each vCPU thread calls KVM_RUN and blocks until the guest exits to the host. Exits happen for I/O port accesses, MMIO accesses to virtual device registers, and hypercalls. The VMM handles each exit, updates device state, and re-enters the guest. This model means the VMM is entirely single-address-space with the guest memory mapped directly into the VMM process. Firecracker creates exactly one process per microVM. There is no privileged daemon. The jailer then wraps this process with namespaces and seccomp.

REST API Configuration Over Unix Socket

All microVM configuration in Firecracker happens through a REST API served exclusively over a Unix domain socket. There is no TCP listener, no authentication layer beyond filesystem permissions on the socket path, and no persistent daemon. The API is documented via an OpenAPI spec bundled with each release. The typical configuration sequence uses curl or a purpose-built client: The socket path is typically placed inside the jailer chroot so that only the orchestrator process, which holds a file descriptor opened before the chroot, can communicate with the VMM. After InstanceStart no further configuration changes are allowed on most resources.

The Jailer: Namespaces, cgroups, and seccomp

The jailer is a separate binary distributed alongside Firecracker. It is invoked with the Firecracker binary path, a unique ID, the uid and gid to run as, and optionally a cgroup version flag. The jailer performs the following steps in order before exec-ing Firecracker. First it creates or joins a cgroup v2 hierarchy under /sys/fs/cgroup/firecracker// and writes memory and CPU limits. Second it creates a new network namespace so the microVM process has no access to host network interfaces by default. Third it creates a new PID namespace. Fourth it chroots into a minimal directory that contains only the Firecracker binary, the kernel image, the rootfs image, and the socket path, all bind-mounted or copied in. Fifth it drops all Linux capabilities except those strictly required. Sixth it installs a seccomp-BPF filter that creates an allowlist of the roughly 24 syscalls Firecracker needs. The seccomp filter is generated at compile time from a JSON policy file and compiled into the binary. Any syscall not on the allowlist causes the process to receive SIGSYS and terminate immediately. This means even if a guest exploits a VMM bug to gain code execution, the attacker cannot call execve, open, socket, or any other syscall to escape.

virtio Devices: net, blk, and vsock

Firecracker implements three virtio devices. The virtio-net device backs each guest network interface with a TAP device on the host. The orchestrator creates the TAP device, connects it to a bridge or configures routing rules, and passes the TAP name to the Firecracker API. Inside the guest the interface appears as a standard Ethernet device. The virtio-blk device backs block storage. Each drive is a file or a block device on the host. The guest sees it as /dev/vda, /dev/vdb, and so on. Drives can be marked read-only, which is common for the root filesystem in serverless environments where the rootfs is a shared immutable snapshot. vsock is implemented as a virtio-vsock device with CID-based addressing. It allows the guest and the host to exchange data over a socket-like API without using the network stack at all. AWS Lambda uses vsock to implement the metadata service that Lambda functions query for environment variables and credentials. For vsock the host side uses the AF_VSOCK socket family. The guest connects to CID 3 (host CID) on whatever port the host service listens on.

Guest Kernel Requirements and Boot Sequence

Firecracker loads the Linux kernel using a built-in ELF loader. The kernel must be an uncompressed vmlinux ELF binary, not the bzImage format used by most distribution kernels. AWS publishes reference kernels but any kernel built from source with a suitable config works. Required kernel config options include CONFIG_KVM_GUEST for paravirtual optimizations, CONFIG_VIRTIO and CONFIG_VIRTIO_PCI for device drivers, CONFIG_VIRTIO_BLK and CONFIG_VIRTIO_NET for the specific devices Firecracker emulates, CONFIG_EXT4_FS if the rootfs is ext4, and CONFIG_9P_FS with CONFIG_9P_FS_POSIX_ACL if using virtiofs for overlay mounts. The boot sequence after InstanceStart is: Firecracker writes the kernel ELF segments into guest RAM, sets the vCPU entry point to the kernel entry address, writes e820 memory map and device tree blobs at well-known addresses, then starts the vCPU threads. The kernel initializes, mounts the virtio-blk rootfs, and runs /sbin/init or the configured init process. No BIOS, no GRUB, no ACPI tables. The entire process typically completes in 125 milliseconds.

Snapshotting and Memory Cloning

Firecracker supports pausing a running microVM, serializing its full state to disk, and restoring it later. The snapshot consists of the VM state file (vCPU register state, device state, memory metadata) and a memory file (a dump of all guest RAM pages). Restoring a snapshot is faster than booting because there is no kernel initialization phase. AWS Lambda uses this to implement the snapshot-and-clone pattern. A base microVM is fully initialized with the runtime and the function code loaded. When a new invocation arrives for a cold start, Lambda restores from the snapshot rather than booting from scratch, bringing restore time down to under 10 milliseconds for small memory configurations. For true cloning at scale, operators use Linux userfaultfd or copy-on-write block devices so multiple restored microVMs share the same physical memory pages until they diverge.

firecracker-containerd and CRI Integration

firecracker-containerd is an open-source project from AWS that implements the containerd Snapshotter and Task API interfaces on top of Firecracker. It allows containerd, and by extension Kubernetes via the CRI, to run OCI container images inside Firecracker microVMs transparently. The architecture has three components. The firecracker-containerd runtime is a containerd shim v2 that launches a Firecracker microVM for each task. Inside the microVM runs a minimal guest agent that receives task commands over vsock. The devmapper snapshotter prepares thin-provisioned device-mapper block devices from OCI layer snapshots and attaches them to the microVM as additional virtio-blk drives, which the guest agent mounts as overlayfs layers. With this configuration, kubectl run commands that specify the firecracker RuntimeClass result in each pod running inside an isolated Firecracker microVM while the container image, resource limits, and lifecycle remain entirely standard Kubernetes constructs.

When to use it

• Running untrusted third-party code in a serverless or FaaS platform where container-level isolation is insufficient and full VM boot times are too slow
• Multi-tenant container workloads on Kubernetes where different customers must have hard kernel isolation, achieved through firecracker-containerd or Kata Containers with the Firecracker backend
• High-density edge compute where hundreds of microVMs must coexist on a single host, each with a memory footprint starting at 128 MB and booting in under 200 milliseconds
• Reproducible CI environments where each test run gets a fresh, identical microVM snapshot cloned in milliseconds rather than provisioned from scratch
• Security-sensitive data processing pipelines where the blast radius of a compromised workload must be limited to the contents of a single microVM with no shared kernel state
• Lightweight development sandbox environments where developers need full VM isolation but cannot tolerate the two-minute boot cycle of traditional VMs

Interview questions

• Why do serverless platforms like AWS Lambda use microVMs instead of containers for tenant isolation?
• Walk through the full Firecracker boot sequence from the moment an orchestrator makes its first API call to a workload process running inside the guest.
• What specific isolation actions does the jailer take, and what attack surface does each action close?
• Compare the security models of gVisor and Firecracker. When is each the right choice?
• How does firecracker-containerd integrate with the Kubernetes CRI to run container images inside microVMs?

More DevOps topics

• etcd and the API Server — The Write Path, Raft, and Why 3 vs 5 Members Matters
• Event-Driven Architecture
• Event-Driven Automation and StackStorm
• External Secrets Operator
• Feature Flags
• FinOps & Cloud Cost Engineering
• Flux v2
• GCC Toolchain Internals
• GCC Upgrades and ABI Migration
• Git
• Git Migration to GitHub
• GitHub Actions