Why Go Async? Background Jobs & Task Queues

A web server's job is to answer quickly. A request that triggers something slow, such as encoding a video, generating a PDF report or sending a batch of emails, ties up a server thread for seconds while the user, and everyone queued behind them, waits. The numbers show how badly. Say a server has 50 threads and each request holds one. If a request takes 50 milliseconds, the server can finish 50 / 0.05 = 1,000 requests per second. If every request takes 5 seconds, it can finish only 50 / 5 = 10. The fix is to stop doing slow work inline.

At a busy counter-service restaurant you order, pay, and are handed a buzzer. The cashier does not cook your meal before serving the next customer, which would stretch the line out of the door. The order goes onto a rail, the kitchen works through it at its own pace, and the buzzer tells you when it is ready.

• -The cashier is the web server: it takes the request and answers quickly.
• -The order ticket on the rail is a message in a queue.
• -The kitchen is the pool of workers, which pull tickets off the rail independently.
• -The buzzer is how the user finds out the result: a notification, or a status they can check.

The web server's job becomes: accept the request, hand the slow work to a queue, and immediately respond that it has been accepted. A separate pool of workers pulls tasks off the queue and processes them at their own pace, without holding up any user-facing request. The response is usually HTTP 202 Accepted, often with a job id the user can use to check progress. The diagram shows the flow:

The card below describes the Worker component as it appears in the lab:

Not every background job is triggered by a user request. Some work needs to run on the clock regardless of what anyone does: regenerating a recommendation cache overnight, sending a weekly billing summary, purging sessions that expired hours ago. A task scheduler (cron, or a managed equivalent like AWS EventBridge) fires these jobs at fixed intervals and drops them onto the same worker queue as any request-triggered job, so a worker never needs to know or care whether a task exists because a user asked for something or because the clock struck midnight.

The card below describes the scheduler component:

Handing work off to a queue does not make capacity problems disappear, it just moves them. If producers enqueue work faster than the worker pool can process it, the queue keeps growing. An unbounded queue turns that growth into an unbounded memory leak and, since work is now sitting for longer and longer before a worker ever reaches it, a slowly climbing effective latency for anyone waiting on the result. Backpressure is the general term for a system pushing back on producers once it cannot keep up, instead of silently absorbing unlimited work and hoping capacity catches up later.

Here is the arithmetic. Suppose producers add 120 jobs a second and the workers finish 100 a second. The queue grows by 20 a second, which is 72,000 jobs after an hour. A job added at that moment has 72,000 jobs in front of it, and at 100 a second it waits 720 seconds, twelve minutes, before a worker even starts on it. Nothing crashed, yet the system is effectively broken. The simulation below lets you set the arrival and processing rates and watch the queue grow or stay healthy.

A bounded queue is the first piece: capping how much work can sit waiting turns an unbounded memory problem into a bounded, predictable one. Once that cap is reached, the system has to decide what to do with the next arrival, and that decision is called load shedding.

• -Reject new work (fail fast): return an explicit error, commonly a 429 Too Many Requests, so the caller knows immediately to retry later or back off, rather than waiting on a request that was going to time out anyway.
• -Drop the oldest queued item: useful when only the freshest data actually matters, like a live metrics feed where a stale reading is worse than a missing one.
• -Signal the producer to slow down directly: TCP itself does this at the transport layer, and some queue protocols expose an explicit "pause sending" signal so a well-behaved producer throttles itself instead of being rejected.