Skip to content
GenAI Learn/Tokens, Embeddings & Memory
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

How LLMs See Text: Tokenization

6 min read

You'll learn to

  • -Explain that a model reads tokens, integer ids for pieces of text, and not words or characters
  • -Trace Byte-Pair Encoding merging frequent pairs into subword tokens, and try a live tokenizer
  • -Explain why token count, not character count, sets both your context limit and your bill, and see resent history compound

An LLM never sees words, sentences or characters the way you do. Before a single number reaches the model, every input is turned into a sequence of tokens: integer ids that point into a fixed vocabulary of text pieces the model was trained on. This matters at once and in practice. Token count is what a context window actually limits, and token count is what you are billed for, not characters or words.

An Analogy: Syllables and Lego Bricks

A word like "unbelievable" is easier to handle as pieces you already know: un, believ, able. Your reading brain does something similar with long, unfamiliar words. A tokenizer breaks text into a set of reusable pieces, a bit like Lego bricks. Common words are one brick. Rare words are built from several smaller bricks. Nothing is ever impossible to build.

Why Not Just Use Characters, or Whole Words?

Both extremes have a real problem. Splitting by single character keeps the vocabulary tiny, but sequences become enormous. A 500-word paragraph turns into thousands of tokens, and the model must work much harder to rebuild meaning from such a fine-grained stream. Splitting by whole word keeps sequences short, but the vocabulary explodes. Every inflection, typo and invented word needs its own slot, and any word never seen in training becomes an unrecoverable gap. Byte-Pair Encoding (BPE), the algorithm behind GPT's tokenizer and the close relatives behind most other major LLMs, is the practical middle.

Byte-Pair Encoding, Traced by Hand

BPE starts with every word as single characters. It then repeatedly finds the most frequent adjacent pair of pieces across the whole training text and merges that pair into one new piece. Repeat that thousands of times and common words become single tokens, common chunks like "-ing" or "-tion" become single tokens, and anything truly new falls back to smaller, still recognizable pieces. Run it on a tiny five-word corpus and watch the same behavior a real tokenizer's training uses, small enough to follow by eye.

A minimal BPE trainer, run on five words. Watch "low" and "lower" and "newer" converge on a shared "er" subword.

Running this prints five merges. First comes ('l','o'), because "low" appears in three of the five words. Then ('lo','w') forms the whole piece "low". Next is ('e','r'), because "er" is in "lower", "newer" and "wider". Then ('er','_') closes it as a full suffix, and finally ('low','_') closes the standalone word. After five merges, "lower" is already down to two pieces, ['low', 'er_'], which share "low" with "lowest" and "er_" with "newer" and "wider". A real tokenizer does this tens of thousands of times over billions of words. That is why a common word like "the" is usually one token, while a rare, invented or foreign word is split into more, smaller pieces.

Now try a real-style tokenizer yourself. Type a few sentences, including a long unusual word and a number, and watch how many tokens each one costs compared with how many words it has.

Type Something, Watch It Get Tokenized

A simplified illustration of the idea: short, common words usually stay whole. Longer or rarer words split into pieces.

Theweathertodayisnice.

characters

26

tokens

7

est. cost

$0.00004

Cost shown at an illustrative $0.005 / 1K tokens. Real per-token pricing varies by model and provider, and input/output tokens are usually priced differently.

Context Windows: The Hard Ceiling

A model's context window is the maximum number of tokens, input and output together, that it can process in one call. Go over it and the call fails outright. Or, worse and much more common in production, the application silently trims the oldest part of the conversation to make it fit. That is the classic "chatbot randomly cuts off mid-sentence" bug, and the exact scenario the first GenAI Lab level puts in front of you.

Context Window Sizes, Roughly (Late-Model Generation)
~128K tokens
Now closer to a budget/legacy tier than a flagship limit
~200K-500K tokens
Where current-generation flagship models typically sit
1M+ tokens
The newest long-context models, growing fast
~750 words
Roughly what 1,000 tokens holds in plain English prose

Why This Drives Cost, Not Just Limits

Every major provider bills per token, and output tokens usually cost noticeably more than input tokens, often three to five times as much. Generating a token needs a full forward pass through the model (one run through all of its layers), while input can be processed in a more parallel batch. So a system that asks for a long, chatty explanation costs meaningfully more than one built to return a short, structured answer, even with an identical prompt.

There is a second, subtler cost. Conversation history is not free just because it already happened. Every earlier turn you resend is re-tokenized and re-billed on every later call. The demo adds it up for ten turns of about 100 tokens each.

Press Run: ten turns cost 5.5 times what one pass over them would suggest

A ten-turn conversation with no memory management does not cost ten times one turn. It costs one plus two plus three, all the way up to ten times a turn's tokens, since each new call resends everything before it. That compounding cost is exactly why the next chapter exists.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!