A text model does not need to choose from the same number of candidate tokens at every step. With top-p sampling, the candidate set depends on how the probabilities are distributed at that moment.
Hugging Face’s generation documentation describes retaining the smallest set of the most probable tokens whose cumulative probability reaches the configured top-p threshold. The model then samples within the allowed set, subject to the rest of the generation configuration.
Follow one invented distribution
Suppose four possible next tokens have probabilities of 0.50, 0.25, 0.15 and 0.10. For a threshold of 0.80, work down the sorted candidates:
- The first contributes 0.50: below the threshold.
- The second brings the total to 0.75: still below it.
- The third brings the total to 0.90: the threshold is reached, so all three remain candidates.
The retained mass can exceed 0.80 because a whole candidate is included. The example is deliberately simplified; real vocabularies contain many more tokens.
At the next step, the distribution may be much more concentrated. A smaller set could then reach the same threshold. That changing set size distinguishes this approach from a fixed top-k count.
Test the control in the generation mode that uses it
Check the library version, model and sampling configuration. A parameter present in a configuration file may not affect every decoding mode. Keep other settings fixed when testing its effect, and save several outputs rather than selecting a single attractive example.
Evaluate what the application actually needs: adherence to the brief, useful variation, factual accuracy or another criterion. A smaller candidate set does not become a fact checker. It can still sample a fluent continuation containing an incorrect claim.
Record both the numerical setting and the observed behaviour. “More controlled” is too vague to reproduce; “fewer unexpected wording changes across these examples, with the same factual errors still present” is a result that a later test can examine.