Language Model Basics
A language model’s KV cache saves repeated work during generation
A KV cache reuses attention information during language-model generation. Learn what it saves, what memory it consumes and what it does not remember.
AI · VISION · EVERYDAY SYSTEMS
Language Model Basics
Language Model Basics
Top-p sampling changes which tokens an AI model may choose next
Top-p sampling selects a changing set of likely next tokens. A small numerical example explains how it differs from choosing a fixed number of candidates.
Language Model Basics
A Model’s Context Window Also Has to Leave Room for Its Answer
A model’s context window includes more than your document. Budget for instructions, history, tools, and the answer you want it to produce.
Language Model Basics
A Token Count Is Not a Word Count
Measure text with the tokenizer used by the model instead of assuming that words, characters and tokens have a fixed one-to-one relationship.