API Gateways for LLM Traffic
You'll learn to
- -Understand routing, rate limiting, and failover at the gateway layer
- -Reason about multi-provider routing strategies: cost, latency, capability and context length
- -Implement cost-based routing with automatic failover, both from health flags and from live errors
- -Name what failover costs you: a bigger bill and different answers, and how to watch for both
A company calling five different LLM providers, OpenAI, Anthropic, Google, Cohere, and a self-hosted open-source model, directly from application code ends up with five different rate limit schemes, five different pricing models, and five different failure modes scattered across the codebase. An LLM gateway exists to absorb all of that into one place, so the application only ever talks to one consistent interface.
An Analogy: A Travel Agent Who Knows Five Airlines
You tell a travel agent where you want to go. They check five airlines, pick the cheapest one that has seats, and book it. If the flight is canceled, they rebook you on the next best option, and you never have to learn each airline's booking system. You deal with one agent, and the agent absorbs all the differences between airlines.
- -The travel agent is the gateway: one front door, one way of asking.
- -The airlines are the LLM providers, each with its own price, limits and outages.
- -Choosing the cheapest airline with seats is cost-based routing among healthy providers.
- -Rebooking after a cancellation is failover.
What a Gateway Actually Does
- -Unified interface: one API shape regardless of which provider actually serves the request underneath.
- -Cost-based routing: send each request to the cheapest currently healthy provider, automatically.
- -Failover: if the chosen provider is down or erroring, retry on the next one without the caller ever noticing.
- -Rate limiting: enforced consistently per user or per tenant, regardless of which provider ends up serving the request.
The prices in the code are illustrative, since real prices differ and change often, but the arithmetic is worth doing. At 10,000 requests of 1,000 tokens each, the cheapest provider here costs 10,000 x 0.80 = $8,000, and the dearest costs 10,000 x 5.00 = $50,000: 6.25 times as much. When the two cheapest providers go down, the cost per 1,000 tokens rises from 0.80 to 4.50, which is 5.6 times as much. The application sees no errors at all, only a bill that grows, so alert on cost per request as well as on errors.
Notice what this code does not need: no change to the calling application at all. The gateway absorbs every provider outage as an internal routing decision. The application just keeps getting responses, at a slightly higher cost during the outage, with zero code change required anywhere else.
Failover That Reacts to Real Errors
The code above chooses from a health flag that something else has set. Real calls also fail in the middle of a request, so a gateway needs to react to the failure itself: try the cheapest healthy provider, and if the call fails or times out, mark it down and try the next one, all within the same request:
Beyond Price
- -Capability: send easy requests to a small, cheap model and hard ones to a large model. Price per token is not the only difference between providers.
- -Latency: a provider that is cheap but slow can be the wrong choice for a chat interface.
- -Context length: a long document may fit only some models, so route on the size of the request.
- -Per-tenant limits and budgets: enforce them at the gateway, once, whichever provider serves the call.
- -Visibility: log tokens, cost and latency for every request, so you can see who is spending what.
What Failover Costs You
Failover is not free. A cheaper or different model may answer differently, and a prompt tuned for one provider can behave worse on another. Test every provider you might fail over to, with the same evaluation set, before an outage forces your hand. Also add timeouts, retries with backoff and a circuit breaker, as covered in the resilience chapters of the system design course, so that a slow provider cannot hold up every request.
Health Checks Have to Be Fast
A gateway is only as good as how quickly it notices a provider is unhealthy. Checking every 15 seconds or faster keeps the window where the gateway is still confidently routing to a dead provider short. This connects directly forward to the disaster recovery chapter later in this tier, where health-check interval becomes the actual unit that failover time is measured in.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!