A classification dataset contains a rare label, and a random split leaves very few examples of it in the test set. Stratification can help preserve the label distribution across the split. That useful property is narrower than a guarantee that the two datasets are equivalent.
In scikit-learn’s train_test_split documentation, the stratify argument uses the supplied class labels when splitting. It does not independently inspect every feature and balance every meaningful subgroup.
State what you are preserving
Consider an illustrative dataset with two classes, one much less common than the other. Stratifying on the target helps retain similar class proportions in training and testing, subject to sample counts and rounding. It says nothing by itself about whether age ranges, locations or collection devices are similarly represented.
Nor does it solve dependence between observations. If several rows belong to the same person, device or household, a row-level stratified split can still place related examples on both sides. That may make evaluation look better than performance on genuinely new groups.
Time creates another boundary. A model intended to predict future events may need evaluation on later data rather than a random mixture of earlier and later records. The split should represent the intended use, not just produce attractive label counts.
Inspect the split before fitting
Record the split method, seed where applicable, class counts and any grouping or time constraints. Check whether the smallest class can support the chosen split. Do not repeatedly change the seed until the test score looks favourable.
Fit preprocessing steps using training data only, then apply them to the held-out data through the appropriate workflow. Stratification does not protect against leakage introduced by preprocessing the full dataset first.
A useful review question is: “What future case does this test set stand for?” If the answer is a new patient, a later month or a different site, make that boundary explicit. Preserving labels is helpful, but the evaluation design still needs to match the question the model is meant to answer.