Skip to content
GenAI Learn/Safety, Model Choice & First Agents
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Choosing the Right Model for the Job

4 min read

You'll learn to

  • -Weigh capability, latency and cost as three competing axes, and pick a point on them for each request
  • -Route requests by complexity, then add a fallback chain for when the first choice fails
  • -Respect a response-time promise (an SLA) and fail honestly when every option is exhausted

A team uses its most capable, most expensive model for everything. A customer-support lookup that costs about half a cent gets the same model as a complex financial analysis that costs several times more. The CFO (chief financial officer) wants a 60% cost cut with no SLA violations. An SLA, or service level agreement, is a promise about service quality, here usually a maximum response time. This chapter is the model-selection half of that problem: the previous chapter's cost levers, applied to deciding which model to call at all.

An Analogy: Choosing How to Travel

A bicycle, a car and a plane are not ranked best to worst. Each wins on some trips. The bicycle is cheap and flexible for a short ride, the plane is fast over long distances but costly, and the car sits between. Nobody flies to the corner shop. Models are the same: you choose by trip, not by always taking the plane.

Three Competing Axes

  • -Capability: how well the model handles hard reasoning, nuanced instructions or specialized domains. The most capable models in a provider's lineup charge a real premium for this.
  • -Latency: smaller, distilled or more specialized models usually respond faster, which matters directly for anything with a tight user-facing SLA.
  • -Cost: usually tracks capability, though not perfectly. Some providers offer very good price-to-capability ratios at a particular size.

No single model wins on all three. If one did, every other tier in a provider's lineup would not exist. Model selection is the deliberate act of picking the point on that trade-off that matches what a particular request needs, not defaulting to the most capable option.

Complexity-Based Routing

This is the InferenceRouter idea from the cost chapter, seen from the capability side. Classify an incoming request's complexity, using a lightweight classifier or simple signals such as query length, numerical reasoning or a keyword from a known-complex category, and route it accordingly. "What are your business hours" and "reconcile these three years of transactions against the new tax policy" are not the same task. Treating them identically wastes capability on the first or under-serves the second.

Fallback Chains: What Happens When Your First Choice Fails

A production system that relies on one model provider has a single point of failure. An outage, a rate limit hit during a traffic spike or a timeout takes the whole feature down. A fallback chain is an ordered list of alternatives: if the primary model errors or exceeds its latency budget, retry against a secondary model, possibly from another provider, and degrade capability gracefully instead of failing the request outright. It is the defensive instinct behind circuit breakers and retries from the HLD course's resilience module, applied to LLM calls. The demo implements routing, a fallback chain and a response-time promise in about thirty lines.

Press Run: simple requests go to the fast tier, a failure falls back, and the SLA can rule an option out
A Rough SLA-Aware Routing Policy
< 40 words, no numbers
Route to the fast, cheap tier
Multi-step reasoning detected
Route to the capable tier
Primary provider times out
Fall back to a secondary model automatically
All tiers exhausted
Return a graceful, honest failure, never a silent wrong answer

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!