KV cache

A language model’s KV cache saves repeated work during generation

A KV cache reuses attention information during language-model generation. Learn what it saves, what memory it consumes and what it does not remember.