An AI classifier can achieve an apparently respectable overall accuracy while repeatedly confusing the two categories that matter most to a project. A confusion matrix exposes that pattern by showing which known labels were assigned to which predicted labels.
In scikit-learn’s confusion-matrix documentation, rows represent the true labels and columns represent predicted labels. Other presentations may arrange the axes differently. Read the labels before interpreting a diagonal or an off-diagonal cell.
Read four cells before trusting one percentage
Consider an invented test of a classifier sorting support messages into “billing” and “delivery”. Of 50 billing messages, it labels 40 correctly and sends 10 to delivery. Of 50 delivery messages, it labels 45 correctly and sends 5 to billing. It is correct on 85 of 100 messages, but the two error directions are not equally common.
With true labels as rows and predicted labels as columns, the billing row contains 40 and 10; the delivery row contains 5 and 45. The off-diagonal entries show the mistakes. Their practical meaning depends on what happens next: a wrong route might delay a response, trigger an unsuitable workflow or require manual correction.
Check whether the displayed values are counts or normalised proportions. A row-normalised display helps compare how each true class is handled, but it can conceal the fact that one class had very few examples. Keep the underlying test counts available.
Then inspect actual examples from the problematic cells. Are the reference labels reliable? Do messages genuinely overlap? Is the model missing a distinction present in the instructions or training data? The matrix locates a pattern; it does not explain the cause by itself.
Use a held-out evaluation set appropriate to the intended task, and record the label order and model version. After changing the system, compare the same kinds of errors as well as the headline score. An improvement is more meaningful when it addresses the mistakes users will actually experience.