Improving Natural-Language-based Audio Retrieval
with Transfer Learning and Audio & Text Augmentations

Natural-language-based audio retrieval is concerned with ranking audio recordings depending on their content’s similarity to textual descriptions. Retrieval tasks like this are typically solved by converting recordings and textual descriptions into high-level representations and then aligning them in a shared audio-caption space; ranking can then be done based on the distance between embeddings. These systems’ retrieval performance highly depends on the quality of the audio and text embedding models, which must extract features that accurately and discriminatively represent the high-level content. Current state-of-the-art approaches [recent_work_1, recent_work_2, clap] create such feature extractors by training models with millions of parameters directly from raw input features, i.e., deep learning. These large embedding models require a large number of training examples, such as the 400 million image-text pairs used to train CLIP [clip], a cutting-edge image-retrieval model. However, publicly available audio-caption datasets like Clotho and AudioCaps are significantly smaller.

This work showcases how to use off-the-shelf pre-trained audio and text neural networks to create a state-of-the-art retrieval model under this limiting condition. We evaluate our approach in the context of task 6b of the 2022’s DCASE Challenge [dcase2022_task6b] and demonstrate how the already well-performing model can be further improved by using a range of audio and text augmentation methods and pre-training on AudioCaps.

The idea of aligning text and audio features for content-based retrieval is not new: Early audio retrieval methods connected bag-of-words text queries and MFCC features via density or discriminative models [previous_work_2]. However, the handcrafted features and the relatively small vocabulary limited these methods’ performance. Current methods build on top of learnable feature extractors that produce high-level audio and text representations from raw input features. Xie et al. [recent_work_2], for example, used a convolutional recurrent neural network to extract frame-wise acoustic embeddings and aligned those to Word2Vec features via a linear transformation. Recently, language-based audio retrieval has received increased attention due to the newly introduced task 6b in the 2022’s DCASE challenge [dcase2022_task6b]. The task’s objective was to create a retrieval system that takes natural-language queries as input and retrieves the ten best-matching recordings from the test set. The top ranking systems among the nine submitted ones leveraged large pre-trained audio and text embedding models like CNN14 [panns] and BERT [bert]. While most systems applied SpecAugment [specaugment], other data augmentation methods, especially text augmentations, have received little to no attention. We address this paucity and study a range of audio and text augmentation methods in the context of audio retrieval.

Our model uses separate audio and caption embedding networks ${\varphi }_a(\cdot )$ and ${\varphi }_c(\cdot )$ to embed tuples of spectrograms and descriptions $\left\{\right(a_i,c_i){\}}_{i=1}^N$ into a shared $D$-dimensional space in a manner that representations of matching audio-caption pairs are close. This behavior is achieved by contrastive training, which equalizes the embeddings of matching audio-caption pairs $(a_i,c_i)$, while pushing the representations of mismatching pairs $(a_i,c_{j;j\ne i})$ apart. The agreement between audio $a_i$ and description $c_j$ is estimated via the normalized dot product in the shared embedding space:

The similarity matrix $C\in R^{N\times N}$ holds the agreement of matching pairs on the diagonal and the agreement of mismatching pairs off-diagonal. We train the system using the NT-Xent [NTxent] loss, which is defined as the average Cross Entropy ($CE$) loss over the audio and text dimension; the ground truth is given by the identity matrix $I\in R^{N\times N}$:

To reduce overfitting of the audio embedding model and improve generalization, we employ three regularization techniques during training: Gain augmentation, MixStyle [mixstyle] along the frequency dimension (Freq-MixStype), and SpecAugment [specaugment]. Figure 3 gives an overview of the audio augmentation pipeline.

Gain Augmentation tries to make the model invariant changes in volume by randomly altering the loudness of the raw audio input signal. Volume manipulations are done by multiplying the waveform with factor

$g$ controls the change in volume (in dB), and its value is randomly drawn from a uniform distribution in the range $[-g_{\text{max}},g_{\text{max}}]$.

SpecAugment [specaugment] randomly masks time and frequency stripes in the input spectrogram, thereby reducing the audio embedding model’s reliance on specific input patterns. The number of stripes along the time and frequency dimensions is controlled via hyperparameters $n_f$ and $n_t$, respectively. Parameters $w_f$ and $w_t$ control the maximum width of the time and frequency stripes, respectively. The actual width and the offset of the stripes are chosen from a uniform distribution; masked values are replaced with zeros. We omitted the warping transformation proposed in the original work as it is computationally expensive and only leads to marginal improvements.

Freq-MixStyle [mixstyle] tries to transfer device-style characteristics between recordings by exchanging statistics along the frequency dimension of spectrograms. To this end, the original spectrogram is normalized along the frequency dimension and un-normalized with adjusted mean and standard deviation statistics. The adjusted statistics are a convex combination of the original statistics $({\mu }_i,\sigma -i)$ and the statistic of a randomly selected spectrogram $({\mu }_j,{\sigma }_j)$:

The coefficient $\lambda$ is drawn from a symmetric beta distribution in a manner that the original statistics always receive a higher weight:

$\alpha$ controls the shape of the Beta distribution. Freq-MixStyle is applied to each input example with a probability of $p_{\text{MS}}$.

We apply Back Translation [backtranslation] and Easy Data Augmentation [EDA] (in that order) to reduce overfitting of the sentence embedding model. Examples of these augmentations are given in Table 3.

Back Translation (BT) [backtranslation] introduces variation into the input sentence without changing its semantics by translating the input sentence to a foreign language and back to the source language. We translate the training captions from English to German, French, or Spanish, and back to English.

Easy Data Augmentation (EDA) [EDA] chooses one of four word-level manipulations and applies the selected operation to each word with a certain probability (indicated in parenthesis): insertion of a random word ($p_{\text{ins}}$), deletion ($p_{\text{del}}$), swap with another word in the sentence ($p_{\text{swp}}$), or replacement with a synonym according to WordNet [wordnet] ($p_{\text{syn}}$). EDA is applied with a probability of $p_{\text{EDA}}$.

We first established a baseline without augmentation and then conducted a series of experiments to investigate the impact of using pre-trained weights for the audio embedding model, augmenting the audio and text inputs, and pre-training on AudioCaps. The model architecture, the exact experimental setup, and the results are discussed below.

This study set out to investigate transfer learning and data augmentation strategies to alleviate the data scarcity problem in natural-language-based audio retrieval. Our research has shown that using pre-trained audio and text embedding models greatly increases the retrieval performance on ClothoV2. We enriched this already well-performing retrieval system with a range of augmentation methods and showed that augmenting both text and audio inputs significantly reduces overfitting. Finally, we further showed that pre-training on AudioCaps leads to additional improvements.

The LIT AI Lab is financed by the Federal State of Upper Austria.