Investigating data partitioning strategies
for crosslinguistic low-resource ASR evaluation

Certain model evaluation practices are considered standard or quite common in natural language processing (NLP), such as using popular benchmarks (Bowman et al., 2015), pre-defined data partitions (Collins, 2002), or random splits (Gorman and Bedrick, 2019). All of these practices rely on metrics calculated over test sets as indices of model performance. It is not generally acknowledged that a particular numerical result might be meaningful only for the specific train/test split that produced that result. A single aggregated metric does not necessarily paint the full picture of a model architecture’s potential (Lewis et al., 2021).

Automatic speech recognition (ASR) provides a case in point. Given a data set produced by multiple speakers, the common data partitioning strategy is “hold speaker(s) out", namely holding out all utterances from one or more speakers (Panayotov et al., 2015; Gauthier et al., 2016) as the test set, with the utterances from the remaining speakers serving as the training set. Cross-validation is generally not applied; the speakers in the test set are fixed. In other words, an ASR system is usually evaluated with just a single train/test split in which there is no speaker overlap between the training and test sets.

This common data partitioning strategy might fare well with a large data set, with recordings of dozens or hundreds of speakers, where the quantity of data and the wide array of speakers enable the training of models that are assumed to be speaker independent. The same practice, however, is not ideal for low-resource scenarios, where the number of speakers is much smaller. With endangered languages (Meek, 2012) in particular, there is much less flexibility in deciding how much data and what kind of utterances to include. Thus, observed ASR accuracy may depend heavily on which speakers appear in the test sets rather than being representative of the model architecture’s general performance.

This study investigates alternative data partitioning methods for low-resource ASR. Leveraging data from five typologically distinct languages, including one endangered language, we ask: (1) How dependent is ASR performance on the identity of the held-out speaker? (2) Can alternative data partitioning strategies yield less variable estimates of a model’s generalizability? (3) What factors other than speaker identity contribute to differences in model performance? (4) How can we operationalize lessons learned to improve ASR evaluation for under-resourced and endangered languages?

While the hold-speaker(s)-out data partitioning method is prevalent in ASR (1; E. Gauthier, L. Besacier, S. Voisin, M. Melese, and U. P. Elingui (2016); A. Zeyer, P. Bahar, K. Irie, R. Schlüter, and H. Ney (2019); I. Kipyatkova and A. Karpov (2016)), there are a number of exceptions. Focusing on Wolof, Laleye et al. (2016) divided the utterances into three groups based on the categories of their content, then used two categories for training and one for testing.  Chiu et al. (2021) tested the performance of an English ASR system trained on short audio segments with utterances of longer duration from YouTube and found poor generalization performance. With five low-resource languages as the test cases, Morris et al. (2021) re-partitioned the data where each speaker occurred in both the training and the test sets; the results showed considerable variability when compared to those derived from holding out one or a fixed set of speakers.

We used data sets for four widely spoken low-resource languages, Fongbe (Laleye et al., 2016), Wolof (Gauthier et al., 2016), Swahili (Gelas et al., 2012), and Iban (Juan et al., 2014), which were previously released as ASR corpora. They include segmented audio with corresponding transcripts, as well as additional written texts for training the language model (Table 1 in Appendix A.1).

In addition, we explored a data set of a critically endangered language indigenous to North America, which we refer to here as Language H. The audio recordings for Language H are the product of ongoing linguistic fieldwork started in 2005. All the recordings were produced by a single female elder speaker, which is common for speech corpora for critically endangered languages, making Language H a unique test bed for our study. Each transcription typically goes through several stages of correction and consultation with the elder before being considered complete; thus some transcriptions have been examined more thoroughly than others. Based only on differences in transcription quality, the audio data was divided into two sets, which we will call “verified" vs. “coarse" data respectively (details are presented in Appendix A.2).

We first consider the degree of variability in WER depending on which speaker is held out. In Figure 1, for Fongbe, Wolof, and Iban, we see a high degree of variability. The WER range across held-out speakers spans from 12.71 for Iban to 54.74 for Fongbe. (See Table 3 in Appendix A.5.) Perhaps surprisingly, for Swahili and Language H, where we held out recording sessions rather than individual speakers, we also observe great variability in model performance. The WER range across sessions is 17.59 for Swahili and is above 25 for both data sets of Language H. Thus it does not appear to be the case that variability in WER is due entirely to the identity of the speaker; other factors such as recording setting or domain could contribute to this variability. These observations speak to our original point, namely that observations from “hold speaker(s) out” low-resource ASR evaluation are not representative of the model’s generalizability.

One might suspect that the observed WER variability across speakers in each data set is (only) because of the varying amount of audio available for each speaker. Although there is a relationship between average WER and the total utterance duration per speaker (when looking at total utterance duration as a sole predictor in the regression) for Iban ($p<0.005$), this relationship does not hold for Fongbe ($p=0.75$) or Wolof ($p=0.91$). While there is a positive correlation between duration and WER for the coarse data of Language H ($p<0.05$), this correlation does not exist for Swahili ($p=0.45$) or for the verified data of Language H ($p=0.99$). This indicates that the total amount of utterance duration alone is not enough to yield (high) WER variability across speakers.

In contrast, results from random splits are much less variable. While not surprising, this is notable in that the average WER when holding out a speaker or session is comparable to that of random splits, or any one random split. Thus a single random split can alone be enough to provide a reasonable estimate of the performance that would be derived by averaging over all random splits or over all possible held-out speakers/sessions. In contrast, the WER for a model tested on single held-out speaker/session may not be a reliable estimate of the WER of that model on any other speaker.

On the other hand, splitting data heuristically and adversarially, creating test sets that consist of “more challenging" cases than the training sets, does not necessarily lead to higher WER. The results across the data split methods are mostly comparable except for when utterance duration or perplexity is used as the heuristic in certain cases. Splitting the data by maximizing transcript distribution distance also yields minimal variability.

The regression analysis (Table 2, Appendix A.4) further indicates that most of the five features we investigated have significant effects on ASR model performance. Compared to other features, the ratios of utterance duration and intensity between the train and the test sets play strong roles more consistently in predicting WER variability. The fact that utterance duration has an effect on WER when controlling for the effects of other factors points to the potential limitation of evaluating models with held-out speakers in low-resource settings, where speakers contribute varying amounts of data.

With data for four widely-spoken low-resource languages and one critically endangered language indigenous to North America, our work demonstrates that there is a real risk of grossly over- or underestimating the performance of an ASR model architecture when evaluating on held-out speakers (and sessions) when only minimal resources are available. By contrast, random splits provide a more accurate and less variable estimate of the overall performance. Moreover, while cross-validation is advisable when partitioning by speaker, a single random split appears to provide an adequate estimate for expected WER on unseen data.

We note that these findings also hold for data sets partitioned according to recording session rather than speaker, suggesting that this phenomenon is not limited to diversity in speaker characteristics. This has implications particularly for ASR in support of endangered language documentation, in which the number of speakers is few but the recording conditions are highly variable. We propose that future work on small ASR corpora for under-resourced languages carry out multiple evaluations on various data partitioning strategies in order to present a more complete picture of ASR model architecture performance.