A dataset contains ten images from each of a hundred objects. A random image-level split can place pictures of the same object in both training and testing. The test then asks a different question from how well the model handles an entirely unseen object.
Scikit-learn’s GroupKFold keeps groups non-overlapping across its folds. The important design decision comes before the splitter: define which observations are related in a way that matters to the evaluation.
Choose the unit you want to generalise to
The group might be a person, machine, recording session, household or source document. If several crops come from one photograph, the original photograph may be the relevant grouping unit. A row identifier alone does not capture that relationship.
Write the evaluation question in ordinary language. “Predict for a new patient” implies a different separation from “predict another measurement for a patient already observed.” Neither is automatically the right task; the intended use decides.
Audit the split before training
Count distinct groups in each partition and verify that the group-ID intersection is empty where separation is required. Inspect label balance and group sizes too. Keeping groups intact can create uneven partitions, so report those limitations rather than quietly returning to a leaking split.
Fit preprocessing and feature-selection steps using training data only within the evaluation procedure. Group separation does not prevent every other form of leakage.
For a small illustrative audit, select three source documents and trace every derived paragraph or image to its partition. If one source appears on both sides of the intended boundary, fix the lineage or split before trusting the score.
A lower result after correcting the split may be more useful than an impressive earlier number. It measures a task closer to the one you meant to test, and gives the next modelling decision a more honest starting point.