How LLMs See Text: Tokenization

An LLM never sees words, sentences, or characters the way you do. Before a single number reaches the model, every input is converted into a sequence of tokens, integer IDs pointing into a fixed vocabulary of text pieces the model was trained on. Getting this right matters immediately and practically: token count is what a context window actually limits, and token count is what you are billed for, not characters, not words.

Both extremes have a real problem. Tokenizing by individual character keeps the vocabulary tiny, but it makes sequences enormous: a 500-word paragraph becomes thousands of character tokens, and the model has to work far harder to reconstruct meaning from such a fine-grained stream. Tokenizing by whole word keeps sequences short, but the vocabulary explodes, every inflection, typo, and made-up word needs its own slot, and any word never seen during training becomes an unrecoverable gap. Byte-Pair Encoding (BPE), the algorithm behind GPT's tokenizer and close relatives behind most other major LLMs, is the practical middle ground.

BPE starts every word as a sequence of individual characters, then repeatedly finds the most frequent adjacent pair of pieces across the whole training corpus and merges it into a single new piece. Repeat that thousands of times and you end up with a vocabulary where common whole words are single tokens, common sub-word chunks (like "-ing" or "-tion") are single tokens, and anything genuinely novel falls back to smaller, still-recognizable pieces instead of an unrecoverable gap. Run it yourself on a tiny five-word corpus and watch the exact same merging behavior a real tokenizer's training run uses, just at a scale you can trace by eye.

Running this prints exactly five merges: ('l','o') because "low" appears in three of the five words, then ('lo','w') to form the whole piece "low", then ('e','r') because "er" shows up in "lower", "newer", and "wider", then ('er','_') to close it off as a full suffix, and finally ('low','_') closing off the standalone word. After just five merges, "lower" is already down to two pieces, ['low', 'er_'], sharing its "low" piece with "lowest" and its "er_" piece with "newer" and "wider". A real tokenizer runs this exact process tens of thousands of times over billions of words, which is exactly why common words like "the" or "running" usually end up as one or two tokens, while a rare, made-up, or foreign word gets split into more, smaller pieces, never an unrecoverable gap.

A model's context window is the maximum number of tokens, input and output combined, it can process in a single call. Exceed it and the call fails outright, or (worse, and far more common as a production bug) the application silently truncates the oldest part of the conversation to fit, which is exactly the "chatbot randomly cuts off mid-sentence" bug that shows up constantly in real production systems, and is the exact scenario the first GenAI Lab level puts in front of you.

Every major provider bills per token, and almost universally, output tokens cost noticeably more than input tokens, often three to five times as much, because generating a token requires a full forward pass through the model while the input can be processed in a more parallelizable batch. This asymmetry matters for design decisions you will make constantly in this phase: a system that asks a model to return a long, verbose explanation costs meaningfully more than one engineered to return a short, structured answer, even if the input prompt is identical in both cases.