Why Go Async? Background Jobs & Task Queues
You'll learn to
- -Recognize workloads that don't need an instant response
Recall from Module 2 that a web server thread handling a request should stay fast and stateless. A request that triggers something slow (encoding a video, generating a PDF report, sending a batch of emails) blocks that thread for seconds while the user (and every other request queued behind it) waits. The fix is to stop doing slow work inline.
Background Jobs
The web server's job becomes: accept the request, hand the slow work off to a queue, and immediately respond "got it, we're on it." A separate pool of workers pulls tasks off that queue and processes them independently, at their own pace, without holding up any user-facing request.
A worker is a background helper that handles jobs that would take too long to do while a user is waiting. Imagine you're at a photo printing kiosk. You upload 50 photos and hit 'Print.' The kiosk doesn't make you stand there for 20 minutes while it prints. It says 'Got it! We'll notify you when your prints are ready' and you walk away. Meanwhile, a worker in the back room picks up your print job from a queue and processes it. In software, workers do the same: they pull tasks from a message queue (like SQS or Redis) and process them in the background. Sending emails, resizing images, generating PDF reports, syncing data to another system, running AI models. The web server's only job is to say 'Task received!' and move on to serve the next request immediately.
Examples: Celery (Python), Sidekiq (Ruby), Bull/BullMQ (Node.js), AWS Lambda (event-driven), AWS SQS + ECS workers
A task scheduler is your system's alarm clock. Just like you set an alarm for 7:00 AM to wake up, a task scheduler runs specific jobs at specific times or intervals. 'Every night at 2:00 AM, clean up expired sessions from the database.' 'Every Monday at 6:00 AM, generate the weekly sales report and email it to the CEO.' 'Every 5 minutes, check if any premium subscriptions have expired.' These jobs happen automatically, on schedule, without any human or user triggering them. It's the invisible maintenance crew that keeps your system healthy while everyone is sleeping.
Examples: AWS EventBridge Scheduler, Kubernetes CronJob, Celery Beat (Python), Quartz Scheduler (Java), node-cron, GitHub Actions scheduled workflows
Scheduled Tasks
Not every background job is triggered by a user request. Some work needs to run on the clock regardless of what anyone does: regenerating a recommendation cache overnight, sending a weekly billing summary, purging sessions that expired hours ago. A task scheduler (cron, or a managed equivalent like AWS EventBridge) fires these jobs at fixed intervals and drops them onto the same worker queue as any request-triggered job, so a worker never needs to know or care whether a task exists because a user asked for something or because the clock struck midnight.
A user uploads a video and your API responds "processing complete" immediately, before the video is actually encoded. Good idea?
"Yes, it makes the API feel fast."
"No: that response has to reflect reality. The API should respond immediately with something honest, like 'accepted, processing' plus a job ID or status endpoint the client can poll (or a websocket/webhook callback), not a false 'complete.' Going async changes when work happens, not what you're allowed to claim happened."