Design a Text-to-Image Service

A user types a sentence and, a few seconds later, gets a picture. Behind that is a GPU doing dozens of denoising steps, a queue of people waiting for it, and several safety checks. The earlier chapter on diffusion explained how an image gets generated. This chapter is about running that as a service.

A print shop has a fixed number of presses and each job takes about the same time. When it is quiet, your job starts at once. As the shop gets busier, queues form, and near full capacity a small rush makes everyone wait a long time. Doubling presses is expensive, so shop owners want to keep presses busy but not so busy that customers leave. GPUs behave the same way.

• -Prompt safety: classify and sometimes rewrite the prompt, blocking requests for disallowed content before any GPU time is spent.
• -Queue: a job queue with priority tiers, so paying or interactive users are not stuck behind a large batch job. The API is asynchronous: submit a job, then poll or get notified.
• -GPU workers: load the model, run the denoising steps, and return an image. The number of steps and the resolution set the cost of a job.
• -Image safety: classify the output for unsafe content, likenesses of real people, or copyrighted material, according to policy, since a safe prompt can still produce an unsafe image.
• -Storage and delivery: store the image, and serve it through a CDN with expiring links.
• -Caching: the same prompt, seed, and settings yield the same image, so identical requests can reuse a stored result.

Intuition says that running GPUs at 95% is efficient. Queueing theory says it is dangerous. When requests arrive at random, the waiting time rises steeply as utilization approaches 100%. The code uses a standard formula, the Erlang C formula, for a queue with 8 identical workers where each image takes 4 seconds. Press Run and watch the average wait as utilization rises from 50% to 98%.

At 50% utilization the average wait is 0.06 seconds, and a user gets an image in about 4.06 seconds. At 70% it is 0.45 seconds, and at 85% it is 1.91 seconds, still tolerable. At 90% it is 3.51 seconds, at 95% it is 8.44, and at 98% it is 23.40 seconds, nearly six times the generation time. The curve bends sharply, so the last few percentage points of utilization are paid for in long waits. The cost lines at the bottom show the trade: an image costs $0.0033 at full use, and twice that at 50%. Most services aim for a target such as 70% to 85%, scale out before the knee, and use a queue with load shedding for spikes.

• -Autoscale on queue depth and wait time, not on CPU. GPU workers take minutes to start, so scale before the knee.
• -Priority tiers and fairness: separate queues for interactive and batch jobs, and per-user limits so one user cannot fill the queue.
• -Show progress and a queue position. People tolerate a known wait much better than an unknown one.
• -Degrade gracefully: lower the number of steps or the resolution under heavy load, or return a queued status.
• -Handle failed jobs with retries and idempotent job ids, so a retry does not charge or generate twice.

• -Human preference: show raters two images for the same prompt and ask which is better. Pairwise comparison is the most trusted measure, as in the evals module.
• -Automatic scores such as prompt-image similarity are useful for regressions, but they correlate imperfectly with what people like.
• -Safety evals: an attack set of prompts built to elicit disallowed images, with attack success and over-refusal rates reported together.
• -Operations: queue wait, generation time, failure rate, and cost per image, watched as percentiles.