A language model generates an answer one token at a time. Much of the context is the same from one step to the next. Recomputing every earlier key and value representation would repeat work, so many inference systems retain them in a key-value cache.
Hugging Face’s explanation of caching describes how previously computed keys and values from attention layers are reused as generation continues. New entries are added for new tokens. This is an inference mechanism; it is not a database of completed answers.
Faster generation has a memory cost
The saved representations occupy memory. Longer contexts and more concurrent requests can therefore change a serving system’s memory needs even when the model weights stay the same. The exact requirement depends on architecture, precision and cache strategy.
Imagine testing the same model with a short question and then a lengthy document. A change in available memory or generation speed is not automatically evidence that the model itself became larger. The active workload and its cache also matter.
Separate three meanings of remembering
A KV cache stores intermediate attention information for inference. Application conversation history stores messages that may be sent back to the model. Training changes model parameters. These are different mechanisms, even when a product description casually calls all of them memory.
Reusing a compatible prompt prefix across requests is a further serving optimisation that depends on implementation. Do not assume every application keeps or shares caches in the same way, or infer its data-retention policy from the word cache alone.
When comparing deployment options, record prompt length, output length, concurrency and cache configuration alongside latency and memory use. Compare representative requests rather than only the fastest short demonstration.
The useful question is how the system avoids repeated computation within its supported workload. A KV cache can make that process more efficient, but it does not guarantee factual answers or give the model permanent knowledge of the conversation.