Effectiveness of Mining Audio and Text Pairs from Public Data
for Improving ASR Systems for Low-Resource Languages

Current state-of-the-art speech recognition systems often employ end-to-end (E2E) models Li and others (2022); Graves (2012b, a); Soltau et al. (2017); Gulati et al. (2020); Babu et al. (2021) which combine acoustic, pronunciation, and language models into a single network. Such models are often large (order millions of parameters) and require compute-heavy training on large datasets of labelled audio. While reducing the word-error-rate (WER) of high-resource languages such as English, such models increase the performance gap of speech systems for low-resource languages, further disadvantaging adoption of AI models for low-resource languages.

A robust approach to address this gap is to collect labelled datasets for low-resource languages. However, creation of high quality datasets can be expensive given the logistics of collecting data across a large diversity of languages and dialects Gumperz (1961). Another approach to reduce the gap between languages is self-supervised learning. Models such as Wav2Vec Baevski et al. (2020) can be pretrained on large easier-to-obtain unlabelled datasets and then fine-tuned with smaller labelled datasets. This was demonstrated for Indian languages in Javed et al. (2022b) with pretraining on 40 languages and fine-tuned models for 9 languages. However, the WER reported for Indian languages is still much higher than what is achieved with equivalent models for high-resource languages. Another approach is cross-lingual transfer of knowledge from high to low-resource languages. Scharenborg et al. (2017) Specifically, labelled datasets for high-resource languages can be transliterated to low-resource languages, similar to how language understanding tasks are created for low-resource languages by translation Khare et al. (2021). However, the resultant accuracy still leaves large WER gaps w.r.t. high-resource languages Khare et al. (2021). Further, transfer learning is dependent on non-native speakers and content which may cause robustness issues on more diverse benchmarks. Thus, while self-supervised and transfer learning can effectively reduce WER, they still leave a role for larger datasets to further reduce the large gap to high-resource languages.

In this paper, we propose an alternative technique to create diverse datasets for low-resource languages. Our approach is motivated by a recent work in Neural Machine Translation (NMT) for Indian languages Ramesh et al. (2022), where billions of sentences in Indian languages were collected from public sources, and then pairs of sentences with high semantic similarity were mined. The obtained bi-text pairs were used to train NMT systems which improved the accuracy by several BLEU Papineni et al. (2002) points for 12 Indian languages, in spite of inexact alignment of the pairs. Similarly, in this paper, we mine audio segments and corresponding text segments for Indian languages from publicly available sources. Specifically, we process the archives of India’s public broadcaster, All India Radio (AIR), which broadcasts radio bulletins in various Indian languages from 45 stations from February 2018. The geographical distribution of these stations and the number of hours of archival audio data are shown in Figure 1. In 12 Indian languages, the archive has over 9,700 hours of audio and is recognized as a major source of Indian language data GIZ (2021).

While both audio bulletins and PDF transcripts are available from AIR, extracting audio and corresponding text pairs, say at a sentence granularity, is not straightforward. We extracted text from the PDF documents with OCR which was error-prone given use of proprietary non-UTF font encodings, presence of formatted content with formats varying from one station to other, presence of metadata such as speaker cues, etc. Audio bulletins too had challenges such as presence of intro and outro segments often containing music, non-transcribed speech such as announcements and external news clips, presence of background music, and code-mixed speech. Given these irregularities, a core requirement is a robust and systematic approach to align document-scale audio and text segments across languages and programs.

We propose an alignment methodology with three steps. First, we process the audio bulletin with an ASR system (IndicWav2Vec Javed et al. (2022b)) to obtain frame level emissions which are collapsed into a character sequence using CTC alignment. Similarly, we pass the PDF documents through OCR and text cleanup to obtain the transcription. Second, we adapt the Needleman-Wunsch algorithm Needleman and Wunsch (1970), which was originally designed to align protein or nucleotide sequences, to align the outputs of ASR and text extraction. Through experimentation, we set the parameters of the algorithm to allow for insertions and deletion of characters. Importantly, the alignment is at document-scale, which on average is about 8 mins long with tens of thousands of characters. Third, we segment the extracted text by sentence boundaries and the audio bulletin by the time intervals based on alignments identified using CTC Graves (2012a) and Needleman-Wunsch algorithm Needleman and Wunsch (1970). We retain each pair of segmented text and audio if a defined Levenstein distance similarity ratio is above a chosen threshold $\tau$. Thus, the output of alignment is a sequence of audio and text pairs corresponding to sentences of text.

We apply these methods to the AIR archive to obtain the Shrutilipi (In Sanskrit, Shruti means ’Sound’ and Lipi means ’script’.) dataset with 6,457 hours of data at a value of $\tau =0.8$, which is more than 60% of the raw audio. The corresponding number at $\tau =0.95$ is 3,239 hours. We evaluate the quality of Shrutilipi with 4,050 pairs graded by 21 annotators. The annotators found quality of alignment to be consistently high for $\tau =0.95$ across stations. Further, the annotators localized a majority of alignment errors either at the start or end of segments. We show that training of ASR systems is more forgiving of such errors. We report on the speaker and content diversity of Shrutilipi. For speaker diversity, we obtain speaker-related embeddings by training a speaker verification task. These embeddings are significantly more diverse for Shrutilipi w.r.t. the MUCS dataset. For content diversity, we find that Shrutilipi has significantly more unique references to named entities across categories w.r.t. to the MUCS dataset.

We evaluate the value of Shrutilipi by training ASR systems for 7 Indian languages using the Wav2Vec architecture Baevski et al. (2020) with existing baselines Javed et al. (2022b). On the IndicSUPERB benchmark Javed et al. (2022a), we show that addition of Shrutilipi to the training dataset of Wav2Vec decreases WER averaged across 7 languages by 5.82%. For Hindi, where we have 7 public benchmarks, the average WER falls from 18.8% to 13.2%. This observed reduction is on top of improvements made in Javed et al. (2022b) of pretraining, thus demonstrating that gains with mining data can compose with those from self-supervised learning. The improvement in WER extends to efficient models: We show a 2.26% reduction for a Conformer Gulati et al. (2020) model which is 10$\times$ smaller than Wav2Vec. The above results are on the entire Shrutilipi data created for $\tau =0.8$. For Hindi, we compare accuracy of models for different values of $\tau =0.8,0.9,0.95$ and found that WER was least for $\tau =0.8$. Finally, we create a hard ASR benchmark for Hindi by introducing background noise, and show that training with Shrutilipi leads to a lower increase in WER.

In summary, we propose a methodology to perform long-audio alignment and apply it to the public archives of AIR to create Shrutilipi. We demonstrate the quality and utility of Shrutilipi with human evaluations, and WER reductions of ASR models on public and noisy benchmarks. We hope that this template can be replicated with other sources of data for Indian languages, and for other languages around the world, to bridge the widening gap between ASR systems for high and low-resource languages.

Though the methodology presented in this work applies generally in mining audio and text pairs for long audio, we will specifically focus on the dataset available from All India Radio (AIR). In this section, we detail this dataset, its language-wise statistics, and some of the issues that make mining text and audio pairs non-trivial.

AIR is the national radio broadcaster of India, a Prasar Bharati division, that streams radio programs in all major Indian languages. The AIR website¹¹1http://newsonair.in/ hosts thousands of hours of audio and PDF transcripts for programs. In this paper, we work with data for 12 Indian languages, collectively representing 1.1B number of speakers in the Indian subcontinent. There are multiple stations creating content in each language categorised into two types - the News Services Division and Regional News. The stations are geographically scattered across India (see Figure 1), indicating diversity in regional representation. The audio data comes from news bulletins that are aired either daily or bi-daily depending on the station. The bulletins vary in length, but are typically 5 min, 10 min, or 15 min in duration. The audio data is in MP3 format sampled at 44KHz and is either mono-channel or stereo. We standardize the data by converting it into 16KHz WAV mono-channel format. The transcriptions for each bulletin are available as PDF documents. Almost every radio station has a different document style.

We collect bulletins aired between 21 Feb 2018 and 17 May 2021. After extraction, we filter out bulletins which have corrupted audio or transcript files. After this basic filter, we collect a total of 72,580 bulletins with 9,695 hours of audio, averaging about 8 mins per bulletin. We show the detailed statistics of the All India Radio dataset in Table 1.

In this section, we propose a novel technique to mine audio and text pairs at the document scale. We describe the alignment technique which consists of 3 main components (i) ASR Prediction using CTC, (ii) Needleman-Wunsch alignment, and (iii) Filtering noisy audio-text pairs.

In this section, we discuss applying the mining procedure to the AIR dataset to create Shrutilipi in 12 languages. We detail the text processing pipeline, the parameter choices for the alignment algorithm, and statistics of the mined data.

In this section, we evaluate Shrutilipi along three axes: (i) is the data of good quality? (ii) is the data diverse? and (iii) is it effective on downstream ASR?

We consider creation of speech datasets from diverse publicly available datasets from All India Radio (AIR). Given irregularities in data, we present a technique to mine audio and text-pairs at document scale by using CTC-based ASR models and the Needleman-Wunsch algorithm. By applying this technique on the AIR archives, we create the Shrutilipi dataset, which consists of 6,457 hours of labelled audio for 12 Indian languages. We show that Shrutilipi is of good quality and has significantly higher diversity in speakers and content in comparison to other public datasets. We evaluate its effectiveness on downstream ASR by evaluating on multiple benchmarks, training on efficient models, and showing robustness to noise. We hope that this methodology is applicable to other public datasets and other languages as well to advance speech technology for low-resource languages.

We would like to thank the Ministry of Electronics and Information Technology (MeitY⁹⁹9https://www.meity.gov.in/) of the Government of India and the Centre for Development of Advanced Computing (C-DAC¹⁰¹⁰10https://www.cdac.in/index.aspx?id=pune), Pune for generously supporting this work and providing us access to multiple GPU nodes on the Param Siddhi Supercomputer. We would like to thank the EkStep Foundation and Nilekani Philanthropies for their generous grant which went into hiring human resources as well as cloud resources needed for this work. We would like to thank Megh Makhwana from Nvidia for helping in training Conformer-based ASR models. We would like to thank the EkStep Foundation for providing the Tarini dataset. We would like to thank Janki Nawale and Anupama Sujatha from AI4Bharat for helping in coordinating the annotation task, and extend thanks to all the annotators of AI4Bharat team.