Temperature & Sampling Controls

At every step of generation, an LLM does not output a single guaranteed next token. It outputs a probability distribution over its entire vocabulary, tens of thousands of candidate tokens, each with some probability of coming next. Sampling settings control how that distribution actually gets turned into one chosen token, and getting them wrong produces exactly the symptom that opens this chapter's corresponding lab level: an AI content generator that is either robotically repetitive, or randomly hallucinating brand names that do not exist.

Before sampling, the model's raw output scores (logits) are converted into probabilities using softmax. Temperature divides every logit by a value T before that conversion. A low temperature (below 1) sharpens the distribution, the already-most-likely token becomes even more dominant, pushing the model toward its single best guess every time. A high temperature (above 1) flattens the distribution, giving lower-probability tokens a genuinely higher chance of being chosen, which reads as more varied, more surprising, and, past a certain point, less coherent.

At T=0.2, "sunny" claims 98% of the probability mass, essentially deterministic. At T=1.0 (the un-scaled distribution), the choice is meaningfully spread across the top few plausible candidates. At T=2.0, even "purple", a nonsense answer to "the weather today is", climbs to a non-trivial 1.3%, which is exactly the kind of occasional off-the-wall hallucination high-temperature settings are prone to producing.

Top-p sampling takes a different approach: instead of reshaping the whole distribution, it restricts sampling to the smallest set of top candidates whose cumulative probability adds up to p (say, the top 90%), then samples only from that shortlist. At low p, only the handful of most likely tokens are even eligible, regardless of how flat or sharp the underlying distribution is. Temperature and top-p are frequently used together in practice: temperature reshapes the distribution's overall sharpness, and top-p trims off the long, unreliable tail of implausible candidates before sampling from what remains.

Neither temperature nor top-p directly stops a model from repeating itself, a distinct failure mode where the same phrase or sentence structure loops. Frequency and presence penalties subtract a small amount from a token's score every time it has already appeared in the output so far, frequency penalty scaling with how many times, presence penalty applying a flat penalty just for having appeared at all, making already-used tokens progressively less attractive and directly countering repetitive output, independent of temperature.