The API chose my sampling design
How an undocumented sort order in a public biodiversity API put 97% of my background points in January — and what I changed so that kind of bias gets caught before a model ever sees it.
Exploratory analysis is supposed to be the calm part of a project. For QueleaGuard, it was where I found that 122 of my 126 background records — 96.8% — fell in January or February. Among the real quelea observations, only 20.3% did.
Nothing had crashed. Every validation check up to that point had passed. If I had gone straight to training, the model would almost certainly have looked good. That is exactly what made it dangerous.
Why the background points exist at all
QueleaGuard investigates where red-billed quelea occur around Ahero Rice Irrigation Scheme. The occurrence data comes from GBIF, which aggregates citizen-science sightings, mostly from eBird. That data is presence-only: it records where birds were seen, never where someone looked and saw nothing.
To model occurrence you need something to contrast presences against. I used an approximate target-group background: other bird species recorded in the same area stand in for “a birder was here and did not record quelea.” It corrects for the fact that observations cluster around places people actually go birding, rather than pretending effort is spread evenly across the landscape.
It is an approximation. Individual records are a noisier proxy for effort than complete checklists, and I wrote that limitation down when I adopted the method. But it only works if the background sample is drawn fairly across space and time. Mine was not.
Why a calendar bias is so harmful here
Almost every environmental feature in the dataset moves with the seasons: rainfall, temperature, dewpoint, vegetation greenness. If nearly all background points are in January and February while presences are spread across the year, a model can separate the two classes by learning the calendar. It would report strong performance, and the performance would say nothing about quelea ecology.
A confound like this does not produce an error. It produces a convincing result.
Finding the cause
The effort-proxy pool came from a script that queried GBIF’s occurrence/search endpoint one year at a time, capped at 150 records per year through pagination. Looking at the raw pool, several years — 2000, 2011, and 2012 — were 150 out of 150 January.
GBIF does not publish a guaranteed default sort order for that endpoint. Empirically, results within a year-only query came back front-loaded from January, and in any year with more records than the cap, the cap cut the list off before it reached the rest of the year. My sampling design had been decided by the order in which a server happened to return rows.
The uncomfortable part is that I had already fixed a year-level version of the same problem when I first built the pool. This was the same class of bug one level down, and it survived because no one — me — had run a month-level check on the pool.
The fix: stop depending on the order
The endpoint accepts a month parameter alongside year. Instead of working around an undocumented ordering, I removed the dependency on it: one query per year–month pair, 16 years × 12 months = 192 queries, each targeting around 13 records.
| Before | After | |
|---|---|---|
| Effort-proxy records | Year-capped, January-heavy | 2,371, stratified by month |
| Largest single-month share | 90.7% | 8.8% |
| Records per month | Extremely uneven | 182–208 |
While rebuilding the pool I also closed a gap found during an earlier validation pass. Three background records appeared to come from the same eBird checklist as a confirmed quelea sighting — which directly contradicts what a background point is meant to represent. The rebuilt pipeline now excludes any effort-proxy record that shares a grid cell and date with a confirmed presence before sampling, instead of hoping validation catches it afterwards. That rule removed 94 records that would have reintroduced the same contradiction.
Checking before moving on
I held back the expensive part — re-extracting rainfall, meteorology, and vegetation features for the new records — until the corrected sample passed its own checks:
- Month distribution: the largest single month is 18.8% of presences (November) and 16.8% of background records (May). No class is concentrated in one season any more.
- Cross-class contradictions: zero cell-and-date pairs shared between presences and background.
- Year coverage: both classes span 2000–2026, weighted towards 2021–2024 in both, which matches the growth of eBird rather than an artefact of the query.
- Spatial coverage: background points now fall in 81 grid cells instead of 49.
The month-biased files were kept, renamed, and documented as superseded rather than deleted. If anyone wants to see how much the bias would have mattered, the evidence is still there.
What I took from it
- Whenever a data request has a cap, ask what decides which records come back. If the answer is “whatever the server does,” the sample is not under your control.
- Stratify on the dimensions the model will be sensitive to. Season matters to every feature here, so month had to be part of the sampling design, not something I hoped would average out.
- Check the distribution of every dimension the model can see, for each class separately. A pooled check would have hidden this.
- Write the problem down instead of quietly patching it. The decision log entry is now the most useful explanation of why the dataset looks the way it does.
QueleaGuard has not trained its research models yet, so I cannot tell you how much this changed the results. What I can say is that whatever those results turn out to be, they will not be a measurement of what month it is.
Originally published on DEV Community.