Design GPU Credits

Cloud GPU Billing

Design the credits system behind a cloud GPU service: users buy credits, launch instances that burn credits by the second, and get stopped when the credits run out.

Overview

A cloud GPU service sells time on graphics cards. A user buys a pile of credits, launches an instance of some GPU type, and the instance eats credits for as long as it runs. When the credits are gone, the instance must stop, and the user must be able to see exactly where every credit went. The interview asks you to design the part that keeps that count honest.

The problem looks like a billing ledger with a timer attached. Two things make it harder than a bank balance. First, spending is continuous: an instance owes money every second it runs, but nobody calls a method every second, so the system bills in intervals and has to decide what happens between them. Second, many instances can be charging the same account at the same moment, so the check 'is there enough left?' and the deduction have to be one atomic step.

Interviewers use this problem to see whether you keep money out of floating point, whether you make the balance check and the charge atomic, whether you can explain why a ledger of movements beats a mutable counter, and whether you handle the moment the credits run out without losing either the user's work or the provider's revenue.

Scoping the problem

Is a credit a currency unit or a fixed amount of GPU time? — A rate card in credits per hour makes each GPU type a price, which is the simpler model — A currency unit; each GPU type has a rate in credits per hour
How fine is billing: per hour, per minute, per second? — Coarser billing overcharges short jobs; finer billing needs more charges — Per second, with a 60-second minimum per run
Who calls billing, and how often? — The gap between calls is the amount the account can go negative by — A scheduler calls bill(now) every few minutes
What happens when the credits run out mid-run? — Terminate at once, or warn and give a grace period — Charge what is left and terminate; a low-balance event warns first
Are there cheaper preemptible instances? — The provider can take them back, so the design needs a preempted state — Yes: a spot policy with a discount
Is the balance the source of truth, or the ledger? — A ledger can rebuild any balance and explains every charge — The ledger; balances are derived from it

The main flow: launch, bill, run out

Every charge is one call to Ledger.post, which checks the balance and moves the amount in a single locked step. The service never reads a balance and then writes a new one.

Why a ledger of movements, not a counter

A balance field that goes up and down cannot say why it has the value it has. A ledger records each movement once, with a source and a destination, and derives balances from those records. Every credit a user spends shows up as revenue, and every credit they hold came from a sale, so the balances of all accounts always add to zero. That sum is a cheap check that no bug created or lost credits.

alice buys credits — system:sales — user:alice — 10.00 — 0
30 minutes of A100 at 4.00 per hour — user:alice — system:revenue — 2.00 — 0
next 30 minutes — user:alice — system:revenue — 2.00 — 0
final 15 minutes — user:alice — system:revenue — 1.00 — 0
spot H100 runs out of credits — user:alice — system:revenue — 5.00 (partial) — 0

The table uses an example rate card (T4 1.00, A100 4.00, H100 8.00 credits per hour). These are demo numbers, not any vendor's prices.

Money and time in code

Number type for credits — Decimal, rounded to 0.01 with ROUND_HALF_EVEN — The Python docs note that 1.1 + 2.2 is not 3.3 in binary floating point and that decimal suits accounting; quantize gives a fixed number of places
Clock — An integer number of seconds passed into every call — Tests are exact and repeatable; production passes the real clock
Minimum bill — 60 seconds per run — Amazon EC2 bills Linux and Windows instances in one-second increments with a 60-second minimum, so the rule is realistic
Where the lock lives — One lock inside the ledger; one in the service for the instance table; one per instance — The check and the deduction happen under the same lock, and a scheduled bill() and a stop() cannot charge the same seconds twice
Pre-authorization at launch — The account must cover 60 seconds of the chosen GPU — A launch that would be terminated on the first bill is refused up front

What you will be able to do

Explain why the balance check and the charge must be one locked step, and show the code that does it.
Argue for a ledger of postings over a mutable balance, and use the zero-sum check as proof that the ledger is consistent.
Keep money in Decimal with a stated rounding rule, and say why floats are wrong here.
Handle the moment the credits run out: partial charge, terminate, notify.
Add a pricing policy (spot, reserved) without touching the billing loop.

Functional requirements

Buy credits: an amount is added to the user account and recorded in the ledger
Launch an instance of a GPU type under a pricing policy (on demand or spot); refused when the account cannot cover the minimum bill
Metered billing: bill(now) charges every running instance for the seconds since its last charge, at its hourly rate
Stop settles the final charge; every run is billed for at least 60 seconds
When a charge exceeds the balance, the remaining credits are taken and the instance is terminated
Spot instances can be preempted by the provider; the bill is settled and the state recorded
A low-balance event fires once when an account drops below a threshold, until credits are bought again
Every credit movement is an immutable posting with a source, a destination, an amount and a memo

Non-functional requirements

The balance check and the deduction are atomic: many instances can charge one account at once without overdrawing it
Credits are exact: Decimal with a fixed rounding rule, never float
The ledger is auditable: balances are derived from postings and the sum of all balances is always zero
Billing is deterministic under test: the clock is an argument, not a global
A stopped, terminated or preempted instance is never charged again

Interview questions

Why a ledger of postings instead of a balance field you add to and subtract from?
Two instances bill the same account at the same moment. How do you stop the balance going negative?
Billing runs every few minutes. What goes wrong between two runs, and how would you bill by events instead?
How would you add reserved capacity paid up front, or a spot discount?
How do you keep this correct across a crash or across many servers?