The new model produces its first words faster, but the finished report arrives later. It calls another service twice, retries a formatting error and leaves more corrections for the reviewer. The model became faster; the task did not.
Choose the start and finish that matter
Define the user’s waiting period. Does it begin when a file is uploaded, when processing starts or when a request reaches the model? Does it end at the first token, the completed answer or an approved result?
Record those boundaries explicitly. Comparing one system’s first-token time with another system’s full-task time produces an impressive but meaningless difference.
Measure the stages separately
Break the workflow into retrieval, model work, tool calls, validation and any human review. Preserve an overall time as well as stage measurements. Overlapping operations need careful interpretation; adding their individual durations may overstate the elapsed time.
Use a representative set of authorised test inputs. Include small and large documents, missing information and cases likely to trigger a retry. A demonstration containing only the easiest request does not describe ordinary service behaviour.
Compare quality under the same conditions
A shorter answer may be faster because it omitted required work. Keep the task and acceptance criteria stable when comparing versions. Record failures and rework rather than removing them from the timing table.
Look beyond an average. A system that is usually quick but occasionally stalls may create a different experience from one with a steadier response. Report the distribution and the number of test cases in terms your team can interpret.
For a small pilot, a simple table can contain input identifier, overall duration, retry count, acceptance result and reviewer corrections. Avoid collecting sensitive content in timing logs when an identifier is sufficient.
Choose improvements based on the actual bottleneck. A faster model will not fix a slow external lookup or an unclear approval process. The useful performance claim is about a task completed to the required standard, with the measurement boundary visible.