Researchers have developed DunDun, an automated test of whether artificial intelligence voices pronounce Yorùbá tones correctly. Its name and central idea draw on the dùndún talking drum.
The metric, called DunDun, is designed for text-to-speech systems that turn written sentences into synthetic audio. It addresses a weakness in a common evaluation method known as character error rate. That method can judge a generated sentence as readable while missing pitch errors that change the meaning of a Yorùbá word.
Researchers Moses Daudu, Adeola Enitan Bamidele and Honor-Jesus Bezaleel described the system in a paper titled Tone on a Budget: A Reference-Free Metric for Lexical Tone in Massively Multilingual Text-to-Speech, submitted to the arXiv research repository in September.
The problem is especially important in Yorùbá because pitch carries meaning. The researchers use the example of three words written with the same basic letters: ọkọ can mean husband, ọkọ̀ can mean vehicle and ọkọ́ can mean hoe. The tone on the second syllable distinguishes them.
A conventional character error calculation normally compares the text produced by speech-recognition software with the sentence that the voice system was asked to read. If that recognition output drops the tone marks, a voice can pronounce the wrong tone and still receive a favourable score. In practical terms, the system may be credited for producing the right letters after saying a different word.
DunDun works from the tone-marked Yorùbá text supplied to the speech generator. It reads the intended high, mid and low tones from the diacritics, then analyses the pitch of the resulting audio. Because the expected tones are already present in the written input, the test does not require a separate reference recording of every sentence or a tone-labelled audio collection.
The name acknowledges the dùndún, the variable-tension drum whose changing pitch can imitate aspects of spoken Yorùbá. A drummer alters the tension of the instrument’s cords while striking it, raising and lowering the pitch. The Metropolitan Museum of Art identifies its own 20th-century Yorùbá ìyá ìlù as a double-headed, waisted variable-tension drum made from wood, cloth, hide and metal.
The researchers tested DunDun in three ways. When they artificially flattened the pitch of speech recordings, the DunDun score fell sharply while character error rate did not change. They also reversed the expected high and low tone labels in 300 native recordings; the resulting score dropped below the relevant chance level, showing that the metric was responding to the relationship between pitch and the correct label.
In a separate listening exercise, three native speakers compared 67 pairs of recordings and selected the tone-correct version 89.6 per cent of the time. The paper cautions that the sample was too small to establish whether DunDun’s scores consistently matched the listeners’ choices from trial to trial.
The team also applied the metric while adapting a multilingual text-to-speech model with different amounts of clean, tone-marked Yorùbá audio. Character error rate fell from 5.6 per cent before the additional training to 2.7 per cent after five hours of material, and to 1.7 per cent after 15 hours.
Tone performance behaved differently. The zero-shot model, before any Yorùbá-specific fine-tuning in the experiment, already scored close to the native-speech reference point used by the researchers. The authors therefore warn that DunDun reaches a practical ceiling near that reference point and should not be read as proof that a machine has achieved “super-native” speech.
The distinction matters for developers assessing African-language voice tools. A system can improve its handling of letters and word segments without making an equivalent improvement in tone. Conversely, a multilingual model may preserve pitch distinctions better than its general transcription error suggests. Reporting both measures gives developers a clearer view of what additional training data actually changes.
DunDun also has firm limits. It depends on accurately tone-marked input, so missing or incorrect diacritics can create the wrong answer key. Its pitch analysis relies on locating the relevant sounds within the audio, and alignment errors can affect the result. The metric is suited to languages whose writing reliably marks tone and cannot simply be applied to every African language.
The authors demonstrated that difference with Swahili, a non-tonal language for which the ordinary character error measure already captured the gains from extra training. They have released the DunDun metric and its validation procedure, allowing other researchers to examine the method and test it on compatible speech systems.
The work gives a familiar Yorùbá instrument a new role in the evaluation of modern voice technology. The dùndún’s pitch-based speech imitation is reflected in a tool built to check whether synthetic voices preserve the distinctions that written tone marks are meant to protect.
