Choosing the Right Model for the Job

A team uses their most capable, most expensive model for everything: a customer support lookup costing roughly half a cent per call gets exactly the same model as a complex financial analysis costing several times more per call. The CFO wants a 60% cost reduction with zero SLA violations. This chapter is the model-selection half of that problem (the previous chapter's cost levers, applied specifically to the question of which model to call in the first place).

• -Capability: how well the model handles genuinely hard reasoning, nuanced instructions, or specialized domains. The most capable models in a provider's lineup command a real premium for this.
• -Latency: smaller, distilled, or more specialized models are typically faster to respond, which matters directly for anything with a tight, user-facing SLA.
• -Cost: usually correlates with capability, though not perfectly, some providers offer genuinely strong price-to-capability ratios at a specific size point.

No single model wins on all three simultaneously; if one did, every other tier in a provider's lineup would not exist. Model selection is the deliberate act of picking a point on that tradeoff surface that matches what a specific request actually needs, not defaulting to "always pick the most capable option available."

This is the same InferenceRouter idea from the cost chapter, viewed from the capability angle instead of the cost angle: classify an incoming request's complexity (a lightweight classifier, or simple heuristics like query length, presence of numerical reasoning, or a keyword match against a known-complex category) and route accordingly. "What are your business hours" and "reconcile these three years of transaction records against this new tax policy change" are not the same task, and treating them identically wastes capability on the former or under-serves the latter, whichever direction the default happens to point.

A production system that depends on a single model provider has a single point of failure: a provider outage, a rate limit hit during a traffic spike, or a timeout takes the entire feature down with it. A fallback chain defines an ordered sequence of alternatives, if the primary model errors or exceeds a latency budget, automatically retry against a secondary model (potentially from a different provider entirely), gracefully degrading capability rather than failing the request outright. This is the same defensive instinct behind circuit breakers and retries covered in the HLD course's resilience module, applied specifically to LLM calls.