How LLMs See Text: Tokenization
You'll learn to
- -Explain that a model reads tokens, integer ids for pieces of text, and not words or characters
- -Trace Byte-Pair Encoding merging frequent pairs into subword tokens, and try a live tokenizer
- -Explain why token count, not character count, sets both your context limit and your bill, and see resent history compound
An LLM never sees words, sentences or characters the way you do. Before a single number reaches the model, every input is turned into a sequence of tokens: integer ids that point into a fixed vocabulary of text pieces the model was trained on. This matters at once and in practice. Token count is what a context window actually limits, and token count is what you are billed for, not characters or words.
An Analogy: Syllables and Lego Bricks
A word like "unbelievable" is easier to handle as pieces you already know: un, believ, able. Your reading brain does something similar with long, unfamiliar words. A tokenizer breaks text into a set of reusable pieces, a bit like Lego bricks. Common words are one brick. Rare words are built from several smaller bricks. Nothing is ever impossible to build.
Why Not Just Use Characters, or Whole Words?
Both extremes have a real problem. Splitting by single character keeps the vocabulary tiny, but sequences become enormous. A 500-word paragraph turns into thousands of tokens, and the model must work much harder to rebuild meaning from such a fine-grained stream. Splitting by whole word keeps sequences short, but the vocabulary explodes. Every inflection, typo and invented word needs its own slot, and any word never seen in training becomes an unrecoverable gap. Byte-Pair Encoding (BPE), the algorithm behind GPT's tokenizer and the close relatives behind most other major LLMs, is the practical middle.
Byte-Pair Encoding, Traced by Hand
BPE starts with every word as single characters. It then repeatedly finds the most frequent adjacent pair of pieces across the whole training text and merges that pair into one new piece. Repeat that thousands of times and common words become single tokens, common chunks like "-ing" or "-tion" become single tokens, and anything truly new falls back to smaller, still recognizable pieces. Run it on a tiny five-word corpus and watch the same behavior a real tokenizer's training uses, small enough to follow by eye.
Running this prints five merges. First comes ('l','o'), because "low" appears in three of the five words. Then ('lo','w') forms the whole piece "low". Next is ('e','r'), because "er" is in "lower", "newer" and "wider". Then ('er','_') closes it as a full suffix, and finally ('low','_') closes the standalone word. After five merges, "lower" is already down to two pieces, ['low', 'er_'], which share "low" with "lowest" and "er_" with "newer" and "wider". A real tokenizer does this tens of thousands of times over billions of words. That is why a common word like "the" is usually one token, while a rare, invented or foreign word is split into more, smaller pieces.
Now try a real-style tokenizer yourself. Type a few sentences, including a long unusual word and a number, and watch how many tokens each one costs compared with how many words it has.
Type Something, Watch It Get Tokenized
A simplified illustration of the idea: short, common words usually stay whole. Longer or rarer words split into pieces.
characters
26
tokens
7
est. cost
$0.00004
Cost shown at an illustrative $0.005 / 1K tokens. Real per-token pricing varies by model and provider, and input/output tokens are usually priced differently.
Context Windows: The Hard Ceiling
A model's context window is the maximum number of tokens, input and output together, that it can process in one call. Go over it and the call fails outright. Or, worse and much more common in production, the application silently trims the oldest part of the conversation to make it fit. That is the classic "chatbot randomly cuts off mid-sentence" bug, and the exact scenario the first GenAI Lab level puts in front of you.
Why This Drives Cost, Not Just Limits
Every major provider bills per token, and output tokens usually cost noticeably more than input tokens, often three to five times as much. Generating a token needs a full forward pass through the model (one run through all of its layers), while input can be processed in a more parallel batch. So a system that asks for a long, chatty explanation costs meaningfully more than one built to return a short, structured answer, even with an identical prompt.
There is a second, subtler cost. Conversation history is not free just because it already happened. Every earlier turn you resend is re-tokenized and re-billed on every later call. The demo adds it up for ten turns of about 100 tokens each.
A ten-turn conversation with no memory management does not cost ten times one turn. It costs one plus two plus three, all the way up to ten times a turn's tokens, since each new call resends everything before it. That compounding cost is exactly why the next chapter exists.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!