AI Image Generation Service

Text-to-Image Service

Design a service that turns a text prompt into images with a diffusion model: jobs wait in a queue, GPU workers run them in batches, credits are charged only for delivered images, and unsafe prompts and images are blocked.

Overview

A text-to-image service takes a prompt such as 'a lighthouse at dusk, watercolor' and returns a few images a few seconds later. The model behind it is usually a latent diffusion model: it starts from random noise and removes a little of it in each of many steps, guided by the prompt, until an image appears.

Three things make this a system design problem rather than a model problem. First, every job occupies a GPU for seconds, so a request cannot be served like a normal web call; it becomes a job in a queue. Second, GPUs are the whole cost, so the design must keep them busy (batching) without letting one heavy user or a traffic spike starve everyone else. Third, the service charges credits and must block harmful content, so each job needs a clear life cycle: accepted, run, checked, delivered or refunded.

Questions to ask first

How many users and jobs? — 1 million daily users, 4 jobs each, 4 images a job
Image size and quality? — 1024 x 1024, a fixed default step count; users pick a style
How fast must results arrive? — Most within about 20 seconds at normal load
How do users pay? — Credits bought in advance; a job costs credits by size and count
Free and paid tiers? — Yes; paid jobs go first, but free jobs must not starve
Can we reject work when busy? — Yes, free-tier submissions may be turned away during spikes
How long are images kept? — 30 days for free users, longer for paid users

What you will be able to do

Explain what a diffusion model does per job and which settings drive its GPU cost.
Design the job life cycle with a queue, leases and idempotent submits, and write it in SQL.
Batch jobs on the GPU and schedule paid and free users fairly.
Size the GPU fleet, and survive a spike that is faster than new GPUs can arrive.
Put safety checks before and after the GPU, and charge credits only for delivered images.

Key concepts

Latent diffusion
Async job queue
Leases and SKIP LOCKED
GPU batching
Fair scheduling
Credit reservation
Warm pool and load shedding
Prompt and output safety
Signed URLs
Content provenance (C2PA)

Functional requirements

Submit a prompt with options (size, style, number of images) and get a job ID at once
See job progress and get image URLs when the job finishes, by polling or a server-sent event stream
Cancel a job that has not started, with a full refund
Charge credits only for jobs whose images are delivered; refund blocked and failed jobs
Block prompts that break the content policy before any GPU time is spent
Check every generated image before it is shown
Store images and serve them through time-limited URLs
Record the seed and settings so a user can generate a close variation later

Non-functional requirements

Queue wait under 10 seconds at normal load (a target we chose); under a spike, show an honest wait estimate
A job is charged at most once, even with client retries and worker crashes
GPU utilization around 80% at peak (a target we chose): idle GPUs are the main waste
No image reaches a user without passing the output check
The API stays up when the GPU fleet is saturated: it queues or sheds, it does not time out

Interview questions

Why is generation asynchronous, and how does the job flow work?
How do you make sure a user is charged exactly once, and only for delivered images?
How many GPUs do you need?
How do you handle a sudden spike when new GPUs take minutes to start?
How do you batch jobs on the GPU without being unfair?
How do you stop unsafe images?
Can a user regenerate exactly the same image?
What happens if the jobs database becomes the bottleneck?