Sampling

At each step a model has a spread of possible next tokens with different probabilities. Sampling is the act of choosing one from that spread. A little randomness makes the writing natural instead of robotic, but it also means the model can pick a plausible-sounding wrong token.

This is the mechanism under hallucination. When the model has no real information to pin the answer down, it still samples something, and what it fills in can be confident and false. Giving it the right context narrows the spread it samples from.