Coverage is not capability: what Whisper taught me about African speech
A language appearing in a model’s supported list says little about whether the model will work for your speakers. Why Whisper struggled with the speech I needed, and what I now check before building on any speech model.
When I built the first version of Scribe AI in July 2026, speech recognition was the part I expected to be easy. OpenAI’s Whisper was open, widely used, and described as multilingual. I ran it locally through Faster-Whisper, wired it into the live pipeline, and moved on to the parts I thought would be hard.
That expectation did not survive the speech I actually needed to handle. On the African languages and accents I cared about, the results were disappointing — not reliable enough to build on. The problem was not a bug in my pipeline. When I went back to the Whisper paper, it turned out to be close to what the model’s own training data predicts.
What “multilingual” meant in practice
Whisper was trained on 680,000 hours of labelled audio. The paper’s dataset statistics show how that total divides: 438,218 hours of English speech recognition, 125,739 hours of translation into English, and 117,113 hours of speech recognition spread across 96 other languages.
Within that last group the distribution is extremely uneven. Swahili, one of Africa’s most widely spoken languages, is listed with about 5.4 hours of transcribed speech. Dholuo does not appear in Whisper’s language list at all.
“The amount of pre-training speech recognition data for a given language is very predictive of zero-shot performance on that language.” — Radford et al., the Whisper paper
So the model was behaving exactly as its training implies. The mistake was mine: I had read “multilingual” as a property of the model, when it is really a distribution over languages, and the speech I needed sat in its thin tail.
Wider coverage, the same question
Whisper is not the only option, and the field has moved. In 2023 Meta released Massively Multilingual Speech (MMS), extending speech recognition to more than 1,100 languages, largely by training on recorded readings of the New Testament — around 32 hours per language. In late 2025 Meta released Omnilingual ASR, which it describes as covering more than 1,600 languages, under the Apache 2.0 licence.
These are real achievements, and they changed what is possible for languages that previously had no speech recognition at all. But they sharpen the lesson rather than retiring it. Meta reports a character error rate below 10 for about 78% of the languages it evaluated with Omnilingual ASR; among low-resource languages, the figure falls to about 36%. Coverage is a list. Capability is a distribution of error rates — usually measured on someone else’s test set.
Three questions now matter more to me than whether a language appears on the list:
- Does it work on my speakers? Training and benchmark audio is mostly clean and often read aloud. A sermon, a phone call, or a support conversation is spontaneous, accented, frequently noisy, and often switches language mid-sentence. The only reliable way to know is to measure word or character error rate on a small set of your own recordings, transcribed by hand.
- Can I use it for what I am building? Licences differ. The original MMS models were released for non-commercial use only — fine for research, a blocker for a product.
- Can it run where my users are? Scribe runs on whatever laptop a church production desk has, with no internet. That rules out large models on modest CPUs, whatever their accuracy. A model that is accurate in a data centre and unusable on the target hardware is not a solution for that setting.
Where that left Scribe — and me
For Scribe, the practical response was architectural. Speech recognition sits in its own module, separate from detection and display, so that a better model can be swapped in later; a formal interface for doing that is still on the roadmap. And every detection still passes through a human operator before it reaches the screen. Neither fixes recognition quality, but together they keep the system safe to use while recognition remains imperfect.
For me, the experience raised a larger question. If a developer in Kenya wants to build a voice product for people who speak Dholuo, or Kalenjin, or simply Kenyan-accented English, what do they build it on? Searching for an answer is what led me into African speech and language AI as a field, and eventually to start Sauti Labs. I have written about that separately.
Originally published on DEV Community.
Sources
- Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al.) (opens in a new tab) — arXiv
- Whisper language list (tokenizer.py) (opens in a new tab) — openai/whisper on GitHub
- Scaling Speech Technology to 1,000+ Languages (Pratap et al., MMS) (opens in a new tab) — arXiv
- Meta’s new open source models speak more than 1,100 languages (opens in a new tab) — The Decoder
- Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages (opens in a new tab) — arXiv
- Meta’s Omnilingual ASR brings speech recognition to 1,600 languages (opens in a new tab) — The Decoder