An impressive average can conceal an awkward product reality: the system works well in the language of most test examples and poorly in the language of a smaller customer group. If that group needs the feature, the overall score is not enough to judge readiness.
Start with the tasks users will actually attempt. Summarising a support message, extracting an address, and rewriting an instruction require different checks. A broad benchmark may inform research, but it does not replace examples that represent the product’s own use.
Keep language groups visible
Record the language of each evaluation example and review results within each group. Include the writing styles people use, such as mixed-language messages, abbreviations, or regional vocabulary, where relevant. Obtain and handle examples appropriately rather than copying private customer messages into an unapproved service.
In a fictional evaluation, 90 successful English examples and two successful examples out of ten in another language could produce an encouraging total. Reporting only that total would hide the smaller group’s experience. The sample is also too limited to support confident conclusions about every speaker.
Ask competent reviewers about meaning
Fluent wording can still change a request, omit a condition, or misread a culturally specific expression. Use reviewers who understand the language and task. Give them a clear rubric and a way to flag uncertain cases rather than forcing every answer into an instant pass or fail.
Compare systems using the same representative test set, and retain examples of important failures. If the product changes its prompt, model, or retrieval sources, repeat the relevant checks. A past result describes a particular configuration, not a permanent property of the system.
The outcome may be a narrower launch, a human review step, or further development. A language-specific limitation is actionable when it remains visible. Folding it into one average makes it easier to overlook and harder to improve.