Spatial cross-validation when the data won’t spread out
Random k-fold overstates performance on clustered ecological data. The obvious alternatives broke on my dataset too — here is the fold design QueleaGuard ended up with, and why.
Before QueleaGuard trains a single research model, it needs a way to tell whether that model is any good. For ecological point data, that question is harder than it looks, and I went through two designs that failed before settling on one I trust.
Why random folds are not an option
Observations that are close together in space tend to look alike: similar rainfall, similar vegetation, similar distance to water. If a random split puts one record in training and its neighbour a few hundred metres away in testing, the model is partly being tested on data it has effectively already seen. The well-established result (Roberts et al., 2017; Valavi et al., 2019) is that this underestimates predictive error, and that spatially blocked cross-validation is required instead.
So the question was not whether to block spatially, but how to block a dataset that is extremely uneven.
Attempt one: use the grid I already had
The analysis already divides the area around Ahero into 328 grid cells of about 5.5 km. Blocking on those cells was the natural first idea. But only 93 of the 328 cells contain any records, and one cell — a persistent site on the Winam Gulf shore — holds 62 of the 266 records on its own. Grouping folds by cell would have produced folds dominated or emptied by a handful of cells.
Attempt two: cluster all the points
Next I clustered every record’s coordinates with k-means into five groups. One of the resulting folds contained no presence records at all.
In hindsight the reason is simple. Background points are spread across the whole 50 km analysis extent, while presences concentrate near the rice scheme and the persistent site. Clustering all points together can carve out a region that is pure background. A fold with no presences cannot be evaluated: precision, recall, and F1 are undefined.
What worked: anchor the clusters on presences
The fix was to fit k-means (k = 5, fixed seed) on presence coordinates only, which places the five centroids where the birds were actually recorded, and then assign every record — presence and background — to its nearest centroid. Because each centroid sits among presences, a presence-free fold cannot occur by construction.
| Fold | Presence | Background | Total |
|---|---|---|---|
| 0 | 8 | 42 | 50 |
| 1 | 9 | 5 | 14 |
| 2 | 11 | 20 | 31 |
| 3 | 36 | 22 | 58 |
| 4 | 69 | 44 | 113 |
The folds are uneven, and I left them that way. Fold 4 is large because it contains the persistent shoreline site; fold 1 is small. Rebalancing them would mean moving records away from their spatial neighbours, which is precisely what blocking exists to prevent.
Checking that the folds do what they claim
- All 62 records from the persistent site landed in a single fold, so that cluster cannot leak between training and testing.
- Only one grid cell was split across two folds. Its two records are 4.6 km apart — further than the largest 2 km neighbourhood used by the rice-landscape features, so they do not share the same feature window. That does not rule out every form of spatial autocorrelation, and I have not claimed it does.
- The fold file covers the modelling dataset exactly: every record once, no duplicates, five fold IDs, both classes present in every fold.
- Re-running the same clustering on the same data reproduces the assignment with zero mismatches.
A related choice: keep coordinates out of the model
The same reasoning shaped the baseline predictor set. Latitude and longitude are deliberately excluded. With data this concentrated, a model given raw coordinates could learn where the clusters are instead of which environmental conditions are associated with quelea — and spatial folds would then be measuring how well it memorised geography. The baseline uses ten environmental and landscape predictors plus season, chosen after checking for redundancy between candidates.
What this design does not fix
Fold 1 has 14 records, 9 of them presences. Any metric computed on it will be noisy, and the honest response is to report per-fold results alongside the average rather than let one number stand in for all of them. The dataset is small — 133 presences — and no fold design changes that.
No model has been trained on these folds yet. The point of fixing the evaluation design first is that when results do arrive, I will not be tempted to choose the validation scheme that makes them look best.
Originally published on DEV Community.