Skip to content
GenAI Learn/Prompting & Model Behavior
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

The Economics of LLM Calls: Tokens & Pricing

5 min read

You'll learn to

  • -Estimate the cost of a call from its input and output tokens and per-token prices
  • -Stack tiered routing, caching and batching and check the arithmetic of a 70% cut
  • -Explain why the levers stack instead of competing

A CEO tells the team that an AI assistant costing $52,000 a month must drop to $15,000, with no visible loss of quality. A 70% cut with no quality drop sounds impossible, until you notice the detail buried in the request: 70% of the actual queries are simple lookups that never needed the most expensive model. This chapter covers the handful of concrete levers that make that cut genuinely achievable.

An Analogy: The Taxi, the Bus and the Delivery Van

Not every journey needs a taxi. A quick trip to the corner shop goes by foot, a commute by bus, and only the urgent or unusual trips need a taxi. Repeated trips to the same place can be combined. Anything that can wait goes in the next delivery run. Cost control for LLMs is the same habit: match the tool to the trip.

The Baseline: Cost Is (Input Tokens × Input Price) + (Output Tokens × Output Price)

A simple, stdlib-only cost calculator, showing why output-heavy tasks are disproportionately expensive.

This prints a roughly 10x cost ratio between tiers, a realistic order of magnitude between a flagship model and a smaller, faster model from the same provider. Multiply either number across a million monthly queries and the gap between "route everything to the premium model" and "route intelligently" becomes a very large line item.

Lever 1: Tiered Routing

Not every query needs the most capable model. A simple FAQ lookup and a complex multi-step financial analysis do not deserve the same treatment: one wastes capability and money, the other truly needs it. Tiered routing classifies incoming requests, often with a small cheap classifier or even a rule based on query length and keywords, and sends simple requests to a cheaper, faster model, saving the expensive model for requests that need it.

Lever 2: Caching

If the same question is asked repeatedly, such as a common FAQ or a popular product description, there is no reason to pay for a fresh model call each time. An exact-match cache stores the response keyed by the literal input and serves repeats instantly for free. A semantic cache goes further, using embeddings (from the previous chapter) to catch queries phrased differently but meaning the same, so "how do I get a refund" and "can I get my money back" hit the same cached answer. Caching gets its own full chapter in the production tier ahead.

Lever 3: Batch Processing

Work that does not need an instant answer, such as nightly report generation, bulk content classification or offline data labeling, can often go through a provider's batch API. It is processed asynchronously within a longer window, commonly 24 hours, at a meaningful discount, frequently 50% off. The provider can slot that work into otherwise idle capacity instead of guaranteeing real-time priority.

Illustrative Cost Levers, Stacked
~10x
Typical cost gap between a flagship and a budget model tier
~50%
Typical discount for asynchronous batch processing
70%+
Share of real-world traffic that is often simple enough to route to a cheaper tier
$0
Marginal cost of a cache hit, exact-match or semantic

Now do the arithmetic for the CEO's problem. The demo starts at $52,000, routes 70% of queries to a model a tenth as expensive, treats one in five of the remaining requests as repeats served from cache, and moves 15% of what is left to half-price batch work. The shares are illustrative, but the structure is the point.

Press Run: three stacked levers take $52,000 to under $15,000

These levers compose. Routing simple queries to a cheap model, caching the genuinely repeated ones among what remains, and batching anything that is not latency-sensitive are independent, stackable savings, not competing alternatives. That is how a real 70% reduction gets assembled without users noticing a quality drop.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!