The Cross-Validation Bug Worth 26 Points of Fake Accuracy

Michi ships a split command whose whole purpose is keeping one entity on one side of a train/test split — the standard defense against group leakage. It had done that correctly since v1.9. Meanwhile bench, tune, and ensemble cross-validated without honouring groups at all: the tool was contradicting its own advice on exactly the pattern that advice exists for.

What group leakage does to a score

Suppose each customer contributes several rows and the label is a property of the customer, not the row. A random k-fold split scatters one customer's rows across train and test. The model doesn't need to learn anything generalizable — it can recognize the customer and recall the answer.

The measurement, on a real dataset with this shape:

| Cross-validation | Balanced accuracy | | --- | --- | | Random folds (leaking) | 0.836 | | Grouped folds (honest) | 0.580 |

26 points of pure illusion. The 0.836 is not a model that works; it is a model that memorized which rows belong to whom. Every downstream decision made on that number — model choice, threshold, ship/no-ship — was made on noise.

Why the fix is not just "pass groups through"

Two details matter beyond wiring the flag:

1. The splitter must be group-aware and stratified where possible. StratifiedGroupKFold keeps class balance while respecting entity boundaries; plain GroupKFold is the fallback when stratification doesn't apply.
2. The grouping column must never reach the model as a feature. A customer ID handed to a tree is a lookup table. Grouping the folds and then feeding the ID as input reintroduces the leak through the front door.

For hyperparameter search there is a third: a group-aware splitter passed to scikit-learn's search without the groups themselves fails deep inside library internals, with an error that names nothing useful. The fix names the flag.

The part that generalizes

This bug survived a long time because it is invisible to a normal test suite. Nothing crashes. Nothing returns an obviously wrong shape. The number just comes back higher than it should, which is the direction nobody investigates. Two adjacent flags in the same codebase were declared and never wired — silently doing nothing — and they hid for the same reason: there is nothing to assert against a feature that was never connected.

What caught them was not more tests of behavior but an audit of the surface: walk every command's signature and assert each parameter is actually referenced in its own body. That audit is now itself a test.

The timing mattered too. Michi froze its public surface under semantic versioning at 1.0, and this was fixed before the freeze rather than after — because freezing a defect makes it a feature, and compatibility guarantees are only worth having when what they preserve is correct.