A model can achieve an impressive test score without learning a useful prediction task. Sometimes the evaluation quietly gives it information that would not exist when a real decision has to be made. This is data leakage, and it can make a weak system look remarkably accurate.
Imagine predicting whether an order will be returned while including a field completed only after the return is processed. The model may find a strong pattern, but the pattern is unavailable at the moment the prediction is needed.
Ask what was known at prediction time
For every input, record where it comes from and when it becomes available. A field’s name may sound harmless while its update process reveals future information. Derived totals and status codes deserve the same scrutiny as obvious outcome fields.
Leakage also occurs during preparation. The scikit-learn guide to common pitfalls explains why training and test data must be kept separate when learning preprocessing transformations. Fitting a scaler or selecting features using the complete dataset can let the test set influence the system being tested.
Split according to the real question
A random row split is not automatically appropriate. If many rows belong to the same person, machine or organisation, closely related examples can appear on both sides. If the task concerns future periods, a chronological evaluation may better reflect the intended use.
Consider a maintenance model with repeated readings from each machine. Decide whether you want to predict later readings from known machines or performance on completely new machines. Those are different claims and may require different evaluation designs.
- Separate data before fitting learned preprocessing steps.
- Keep related entities together where the question requires it.
- Check timestamps and post-outcome fields.
- Reserve a final evaluation set from repeated tuning decisions.
When a score drops after correcting leakage, that is useful information. It shows that the earlier result overstated performance. Do not restore the contaminated setup merely because its chart is more attractive.
A credible evaluation describes its limits: the population, period, split method and information available at prediction time. The point is not to obtain the highest number on a convenient dataset. It is to learn whether the model can make the decision you actually intend to ask of it.