Lease and Reservation Control Planes

Arbitrating scarce, non-fungible resources — a GPU zone, a hardware bench, a test cluster — with time-bounded leases: why leases must expire, fencing tokens, queueing and fairness, preemption, and modelling the whole thing as a CRD plus controller.

Overview

Scarce, non-fungible resources

Some resources cannot be autoscaled: a GPU zone with a specific interconnect topology, a hardware bench with one board on it, a test cluster pinned to the Kubernetes version an upgrade must be validated against, a vendor sandbox with a single account. There is a fixed number of them, they are not interchangeable, and no API call makes another one appear this afternoon.

Every organization that owns hardware ends up building an arbitration layer for them. Most build it badly at first: a spreadsheet, a Slack channel, a convention that you rename the machine to your username.

The lease: ownership with an expiry

The distinction from a lock is the entire design.

Released — Explicitly, by the holder — On its own, when renewals stop
Depends on — The holder surviving to release it — The holder renewing while alive
Holder failure — Permanent; on a single-instance resource, an outage with no automated recovery — A bounded event

> [!IMPORTANT] That single property is why Kubernetes uses leases for node heartbeats and leader election rather than locks.

The Kubernetes Lease object

The reference implementation, worth knowing at field level: coordination.k8s.io/v1.

holderIdentity — Who holds it
leaseDurationSeconds — For how long
acquireTime, renewTime — When acquired and last renewed
leaseTransitions — Count of holder changes

It is deliberately spartan: no queue, no priority, no fairness, no notion of what the lease is for. It records who holds what until when; every scheduling policy lives in a controller above it.

Node heartbeats — Every Node has a Lease in kube-node-lease that the kubelet renews cheaply instead of rewriting full node status
Leader election — HA control plane components contend for one Lease: 15-second lease duration, 10-second renew deadline, 2-second retry period

Fencing: the correctness problem

A lease with a timeout is not by itself safe. The holder cannot reliably detect its own expiry, because what causes expiry (a stop-the-world GC pause, a suspended VM, a page-fault storm, a network partition) also prevents it from noticing.

Martin Kleppmann named the fix: a fencing token, a monotonically increasing number issued at acquisition, presented with every operation, and enforced by the resource, which rejects anything lower than it has already seen. leaseTransitions is precisely such a counter.

> [!WARNING] The resource must do the checking. A token the resource ignores provides nothing.

The policy layer

This is where most of the engineering actually goes.

Queue discipline — FIFO, priority, weighted fair share, per-team quota, or a layering of all four
Preemption — Reallocate a fixed pool to urgent work instead of leaving it held by a six-hour batch job; needs checkpointing, because preemption that destroys work teaches users to game the system
Reservations versus on-demand — Planned needs versus opportunistic ones; overcommit is whether reservations may exceed capacity on the bet that not everyone shows up
Observability — Utilization, queue depth, 95th-percentile wait, who holds what now, and whether buying two more units would change any of it

Where this connects

The "Hardware-in-the-Loop CI" topic covers exclusive leasing of physical test benches for CI on real silicon. This topic is the general control plane behind it: lease semantics, the fencing argument, queueing policy, and modeling it all as a CRD with a controller so it is declarative, self-service, and auditable rather than a wiki page and a lock file.

Key concepts

Lease versus lock: A lock is held until explicitly released and therefore depends on the holder surviving to release it. A lease is ownership with an expiry that the holder must renew. Because holders crash, get evicted, and get SIGKILLed without running cleanup, only the lease bounds the damage — a permanently held lock on a single-instance resource is an outage requiring manual intervention.
Kubernetes Lease object: coordination.k8s.io/v1 Lease. spec.holderIdentity names the current owner, spec.leaseDurationSeconds is how long a renewal is valid, spec.acquireTime and spec.renewTime record acquisition and the last heartbeat, and spec.leaseTransitions counts holder changes. Renewal is an update to renewTime. There is no queue, priority, or fairness in the object itself.
Node heartbeat leases: Each Node has a matching Lease in the kube-node-lease namespace that the kubelet renews frequently by updating only renewTime. The node lifecycle controller treats a lease that stops being renewed past the node monitor grace period, roughly forty seconds by default, as an unhealthy node. Updating a tiny Lease instead of full node status is what keeps heartbeat traffic affordable at cluster scale.
Leader election timing: HA control plane replicas contend for one Lease. Defaults are lease duration 15 seconds, renew deadline 10 seconds, retry period 2 seconds. The invariant is that a leader which cannot renew by the renew deadline must stop acting before the lease duration expires, so the incoming leader never overlaps with an outgoing one that has not yet noticed it lost.
Fencing token: A monotonically increasing number issued on each lease acquisition, presented by the holder on every operation, and enforced by the resource, which rejects any operation carrying a token lower than the highest already accepted. It is what makes a lease safe against a holder that paused past its expiry and resumed believing it still owns the resource. leaseTransitions and object resourceVersion both serve this role in Kubernetes.
Renewal, TTL, and grace period: Three separate numbers. The renewal interval is how often the holder heartbeats. The TTL or lease duration is how long a renewal remains valid. The grace period is the extra time before reclamation begins after expiry. The renewal interval must be well below the TTL so a single missed heartbeat is survivable, and the reclaim path must be idempotent because it will occasionally run against a holder that is merely slow.
Queue discipline and starvation: FIFO is starvation-free and urgency-blind. Strict priority serves urgency and starves the bottom indefinitely unless requests age upward in priority as they wait. Weighted fair share allocates a contended resource proportionally per tenant over time. Per-tenant quota caps consumption regardless of demand. Production systems layer quota, then fair share, then priority, with ageing to bound worst-case wait.
Preemption with checkpointing: Reclaiming a held lease for higher-priority work. Safe preemption signals the holder, allows a grace period to checkpoint, then revokes and requeues the preempted request automatically. Without a checkpoint path preemption is destruction of work, and tenants respond by inflating priority and holding leases longer than they need, which defeats the fairness the mechanism was built for.

When to use it

GPU and accelerator pools where jobs need whole nodes or whole interconnect zones and the number of them is fixed by what is physically racked
Hardware test benches, device farms, and lab equipment that can serve exactly one job at a time and must be released reliably even when the job crashes
Long-lived test and staging clusters pinned to a specific version or configuration, where two teams running conflicting experiments at once invalidates both results
Single-instance external dependencies — a vendor sandbox account, a licensed software instance, a shared data warehouse slot — that cannot be duplicated per consumer
Any leader election or single-writer pattern in your own controllers, where exactly one replica must act and a failed leader must be replaced without a human

Interview questions

Why must a lease expire, and what exactly goes wrong with a lock that does not?
Explain fencing tokens. Why is a lease with a timeout not sufficient on its own?
Design a lease and reservation control plane as a CRD plus controller. What are the types and what does the reconcile loop do?
How do you handle queueing, fairness, and starvation across teams sharing one pool?
How do you preempt a lease without corrupting in-flight work, and how do reservations and overcommit fit in?
What do you instrument, and how do you use those metrics to answer whether to buy more hardware?