Video-Guided Curriculum Learning for Spoken Video Grounding

Harwath et al.(Harwath et al., 2020b) first investigate whether unsegmented raw speech audio can highlight the relevant regions in the images. This work demonstrates that this problem is quite challenging since the portions of the speech signals refer to shorter objects involving many categories. Nortje et al.(Nortje and Kamper, 2020) investigate how to study a shared embedding space of spoken words and images with a multi-modal few-shot learning. To further explore the relationships between images and speech, Hsu et al. (Hsu et al., 2020) collect a new spoken audio captions dataset from MSCOCO and introduce a model that directly generates fluent spoken audio captions of given images without any auxiliary text supervision. These works focus on the visual grounding of spoken language with static images. However, for humans to understand surrounding world, visual perception is inherently extended in time, as many spoken languages are related to the events or actions evolving over time rather than still images (Chrupala, 2021). Thus, recently some works have tackled this problem by focusing on the constrained domain of cooking videos (Monfort et al., 2021; Boggust et al., 2019; Rouditchenko et al., 2020). For example, (Boggust et al., 2019) explore the correlations among visual images and the corresponding spoken descriptions in cooking videos with unsupervised semantic learning. Rouditchenko et al. (Rouditchenko et al., 2020) investigate the relationship between spoken words and visual entities in videos with a contrastive loss. However, their model is trained only on instructional videos, limiting in the types of visual scenes and spoken languages. Onsescu et al. (Oncescu et al., 2021) introduce a new speech-video retrieval dataset. In this work, we employ the ActivityNet Speech dataset collected from ActivityNet, which is a open domain video collection and contains over 20k untrimmed videos.

Lee(Lee et al., 2018) first propose a open-domain spoken question answering (SQA) dataset. Li(Li et al., 2018) also introduce a spoken dataset for Stanford Question Answering Dataset (SQuAD). Many methods have been proposed to solve SQA problems(Chen et al., 2021; You et al., 2021b, 2020b, 2020a). (You et al., 2021b) propose a temporal-alignment attention mechanism to learn cross-modality alignment between speech and text embedding spaces. (You et al., 2021a) propose a knowledge distillation which can utilize the teacher trained on manual transcriptions to guide the training of the student on ASR transcriptions.

Text-to-video temporal grounding is first studied in (Hendricks et al., 2017; Gao et al., 2017), aiming to localize the time interval in a video semantically relevant to the text description. Various models have been proposed to solve this problem, which can be divided into proposal-based and proposal-free methods. Proposal-based methods first extract candidate proposals with sliding windows and then rank the candidates with text query (Chen and Jiang, 2019; Zhang et al., 2019a, b). Zhang et al. (Zhang et al., 2019a) construct a graph to explicitly model moment-wise temporal relations. (Zhang et al., 2019b) learn fine-grained representation learning by devising a syntactic GCN to leverage the syntactic structure of queries and apply multi-head attention to capture long range semantic dependencies from video context. (Ma et al., 2021) use convolutional neural network to generate coarse video candidate moments and then adjust the boundary. (Hu et al., 2021) design a hierarchical semantic tree to explore the semantic relationship between different visual moments. (Qu et al., 2020) propose a fine-grained iterative attention module to incorporate information from text and video mutually. Although these methods have achieved promising results, they are suffering from time-consuming and computation-extensive problems.

Recently, thanks to the development of machine reading comprehension in NLP, proposal-free models have been proposed to alleviate the above mentioned problems. For example, (Chen et al., 2019) use a cross-gated attended recurrent module to exploit the interactions between text query and video, and then use a segment localizer to directly predict start and end moment. (Ghosh et al., 2019) apply an extractive approach which can leverage cross-modal interactions between text and video to predict the start and end frames. (Zhang et al., 2020) propose a simple but effective query-guided highlighting (QGH) strategy which can guide the model to search for matching video span within a highlighted region.

Inspired by human learning process, Bengio et al. (Bengio et al., 2009) propose curriculum leaning that gradually increased the training data complexity. Many studies follow the original formulation that applies curriculum strategy at the data level (Shi et al., 2015; Pentina et al., 2015). There are also some methods that apply curriculum strategy at the model level, which gradually increase the model’s capacity during the training process (Karras et al., 2018; Sinha et al., 2020). Other works explore the curriculum learning at the task level. (Sarafianos et al., 2017) first group the tasks into strongly and weakly-correlated ones and then transfer the knowledge from the strongly-correlated tasks to the weakly-correlated tasks. Jiang et al. (Jiang et al., 2014) first propose a self-paced learning with diversity methodology, which formalizes the preference for both easy and diverse samples into a general regularizer. (Liang et al., 2016) combine the curriculum learning and self-paced learning together for webly labeled video data learning with noisy labels. (Platanios et al., 2019) propose a continuous curriculum learning in neural machine translation. In this paper, we propose a curriculum learning strategy similar to the data level methods, which gradually shift the given video content from ground truth part to the entire part during pretraining audio encoder utilizing contrastive learning.

Given a spoken audio $A$ and its corresponding video clip $V$, the goal of the spoken language video grounding task is to localize the start and end video frames that semantically meet the description of the audio. In order to simulate the real-world scenarios, we randomly select an environmental sound (Piczak, 2015) that we consider as noise and merge it with the clean speech at a random ratio. Compared with traditional text-to-video grounding, the new task is far more challenging due to the unsegmented and unaligned noisy spoken audio. To bridge this gap, we propose a novel curriculum learning strategy that gradually understands the noisy spoken language with corresponding video content. Essentially, our method can imitate the process of babies trying to understand the relationship between spoken language and external visual events.

As depicted in Fig 2, our baseline model can be divided into four modules. Specifically, after encoding the video and spoken language into the same latent space, we can obtain the visual features as $V=\{v_i{\}}_{i=1}^{N_v}$, and the audio features as $A=\{a_i{\}}_{i=1}^{N_a}$, where $N_v$ and $N_a$ are the number of the extracted visual and audio frames, respectively. Then, we apply a self-attention layer to obtain a holistic understanding of the video content and utilize a context-query attention module to integrate the visual and audio features. Finally, we employ a prediction layer based on recurrent network to predict the temporal grounding result. Our main contribution is the audio pre-training process based on the video-guided curriculum learning. After obtaining the audio pre-trained model, we exploit it to replace the audio encoder in feature extraction process. More details about the curriculum learning strategy are presented in Section 5.

Audio encoder We represent the noisy audio input as a log Mel filterbank spectrogram, with a 16 kHz sampling rate, a 25 ms Hamming window, a 10 ms window stride, and 128 Mel filter bands. We apply two 1-D convolution blocks and a single layer unidirectional GRU to encode the spectrogram. Then we use two trainable CNN blocks with residual layers (Harwath et al., 2020b) to process the audio features into $A=\{a_i{\}}_{i=1}^{N_a}\in R^{N_a\times d}$, where d denotes the feature dimension.

Video encoder For the untrimmed visual features extracted by pretrained C3D model(Tran et al., 2015), we use a 1D convolutional layer and one layer bi-directional GRU layer as our visual encoder, which can capture a long range information. We encode the visual features into the same dimension latent space with audio features as $V=\{v_i{\}}_{i=1}^{N_v}\in R^{N_v\times d}$.

Following the previous machine reading comprehension works, we formulate the interaction between video context and audio query from the perspective of Context-Query Attention (CQA) module, which have been successfully used in many text-to-video grounding methods (Zhang et al., 2020; Zhao et al., 2021). We first input the visual features into a self-attention layer to get a holistic understanding of the entire video content: $\hat{V}=SelfAtt\left(V\right)$. Then we use the CQA to calculate the cross-modal similarity score $S\in R^{N_v\times N_a}$ between the visual and audio features. The video-to-audio attention ${\beta }_1\in R^{N_v\times d}$ and audio-to-video attention ${\beta }_2\in R^{N_v\times d}$ can be obtained by:

where $S_r$ and $S_c$ are the normalized result of similarity score $S$ by SoftMax along row and column axis, respectively. Finally these two attention matrices are fused to integrate the visual-audio attention vectors:

where FFN is the feed-forward layer and $\odot$ is the element-wise multiplication.

We use two unidirectional GRU to predict start and end boundaries $F_s$ and $F_e$. Inspired by (Zhang et al., 2020), the end boundary probability is calculated based on the start boundary probability, since an event’s start and end boundaries are always related:

Instead of solely relying on sophisticated start and end boundaries prediction, we introduce an additional unidirectional GRU to predict inside frame probability $F_i$, helping the model distinguish whether a frame belongs to an event. Then we apply three feed-forward layers to predict the boundary and the inside frame probability scores:

where $P_s$ and $P_e$ denote scores of the start and end boundaries at each position, $P_i$ denotes the scores of the inside frames whether relate to the spoken language description.

CPC has been widely used in audio pre-training, which can extract high-level phonetic representations and acoustic units from raw observations by predicting the future samples with powerful autoregressive models. However, most of the previous works predict the future audio waveform features, which is not suitable for our task. Thus in this paper, we propose to directly train CPC on audio log-Mel spectrogram $\tilde{A}$. The spectrogram is first processed by a stack of 2 convolutional layers as encoder $g_{enc}$, each downsamples the input by a factor of 2. Thus we can get a sequence of latent variables $z_t=g_{enc}\left({\tilde{A}}_t\right)$. Then we use a single layer undirectional GRU to summarize the information of all $z_{\le t}$ and obtain a context representation as $c_t=GRU\left(z_{\le t}\right)$.

Given a prediction of $K$ steps, a set $Z$ of $N-1$ random negative samples and one positive sample $z_{t+k}$, we use $c_t$ to predict the k-th future step $z_{t+k}$, and the Info Loss can be optimized as:

where $W_k$ is the linear projection matrix for different step k. After pretraining, we can use the encoder $g_{enc}$ and GRU layer to replace the according audio encoder layer in Section 4.3.

Motivation: We all have such experiences, when the speech is disturbed by external noise, it is easy for us to misunderstand some keywords, i.e. ”sing” or ”seen” as shown in Fig 6. Such mistakes will lead us to completely misunderstand the original sentences. However, if we see the corresponding events while hearing the noise speech, the visual information will help us to rectify these errors. Besides, when babies first learn to understand the relationship between noisy speech and visual actions in the external world, their parents will point out the related visual events while repeating the descriptive speech. The introduction of visual information can help babies learn how to distinguish between noise and real useful voice information progressively.

Inspired by this, in this section, we propose a video-guided curriculum learning strategy, which can make use of visual perception to help audio pretrain with CPC and effectively suppress the environmental noise. The overall architecture of VGCL is shown in Figure 3. Specifically, given a piece of audio, we first use self-attention to have a comprehensive understanding of its corresponding video content as $\tilde{V}$, then we use mask mechanism to keep the critical part of the video content. Concretely, the left and right mask boundary $M_l$ and $M_r$ can be calculated as follow:

where $L$ is the video length, ${\tau }_s$ and ${\tau }_e$ are the ground truth video start and end points, $\gamma \sim U(0,1)$, $\kappa$ is the number of the curriculum learning stages, $t\in (0,\kappa )$. We use the mask matrix to obtain the masked video content as ${\tilde{V}}_{mask}$.

In the initial training stage, the visual input is the ground truth video part. Then we continue training the model with t increasing sequentially. For simplicity, we increase t from 0 to $\kappa$ step by step, which means the input video part is gradually shifting from ground truth to the entire fragment. By understanding the information in this vital video content, we can learn the connections between phonemes and corresponding actions in the video, retaining key features and suppressing environmental noise during CPC autoregressive encoding. Meanwhile, we also find in the experiments that the effectiveness of solely adding ground truth video part is not satisfying due to the lack of video context information. Therefore, we concatenate the entire video clip with the masked video content in the time dimension: ${\tilde{V}}_{full}=[\tilde{V};{\tilde{V}}_{mask}]$.

Then we project audio into the query features $Q_a$, and project the concatenated visual features into key and value features $K_v$ and $V_v$. Finally we apply cross-modal attention to exploit related information from visual features:

where $\delta$ denotes the Relu function. Then we use $\hat{z}$ to calculate ${\hat{c}}_t=GRU\left({\hat{z}}_{\le t}\right)$, but still use ${\hat{c}}_t$ to predict $z_{t+k}$. The Info loss can be optimized as:

where ${\hat{W}}_k$ is the linear projection matrix for different step k. After the pre-training process, we can obtain a mature audio encoder, which can extract vital visual information from the video clip and use it to rectify ambiguous phonemes from noisy speech input. We use the VGCL module to replace the first audio 1D convolution encoder as described in Fig 2. The experiments demonstrate that with the addition of VGCL module, our full model can effectively suppress the noise and improve the localization performance compared with baseline network.

Pacing function is widely used in curriculum learning strategies, which can control the training steps for different training stages. Concretely, we divide our training process into $\kappa$ stages, and define three different pacing functions to control the training steps as $t\in (0,\kappa )$ increases: linear, exponential, and logarithmic (Liu et al., 2020). Linear function refers to exploiting the same training steps for each stage. Exponential function means that for early t, the training steps are small and become more prominent as the input video boundaries get blurred. Logarithmic function is entirely different from the exponential function as the training steps will gradually decrease as the t increases. We compare and analyze these different pacing functions with experiments to demonstrate the impact of different curriculum learning strategies on audio pretraining.

Based on the abovementioned audio pretrain process with video-guided curriculum learning and main model structure, we can get the start and end boundary probability scores: $P_s$ and $P_e$, where the probability distributions can be calculated as $\widehat{P_m}=SoftMax\left(P_m\right)$, $m\in \{s,e\}$; and one inside frame probability scores: $P_i$, where the distributions can be calculated as $\widehat{P_i}=Sigmoid\left(P_i\right)$. Then we use these probability distributions to define two loss functions, boundary loss and inside loss:

Boundary loss: We utilize cross-entropy loss function to optimize the start and end boundary predictions, the loss item can be defined as:

Inside loss: We also introduce the inside frame loss to let the model distinguish whether a frame belongs to answer or not, which can further optimize the boundary prediction. Specifically, the inside loss item can be defined as:

where $y_t$ is the ground truth label for each frame, if the t-th frame belongs to the answer fragment, then the value of $y_t$ is 1, otherwise is 0.

The overall loss function of the main model can be summarized as $L_{total}=L_{bound}+L_{in}$.

At the inference time, the start and end boundary predictions can be calculated by maximize the probability scores $P_s$ and $P_e$.

For visual features, we follow the previous works (Zhang et al., 2020; Mun et al., 2020) and use the features extracted by a publicly available pre-trained C3D model (Tran et al., 2015). For each speech audio, we first randomly select a piece of noise from ESC-50 (Piczak, 2015), and then add the noise to a random part of the original audio with a proportion $\alpha \in (0.5,0.7)$. ESC-50 dataset is a labeled collection of 2000 environmental audio and consists of 5-second-long recordings organized into 50 semantical classes, which can be divided into five main classes: animals, natural soundscapes/water sounds, human non-speech sounds, interior/domestic sounds and exterior/urban noises.

We downsample the noisy audio recordings to 16KHz and extract 128-dim mel-spectrogram features. For the convenience of model training, we uniformly sample segments from each mel-spectrogram feature with a fixed length $N_a=1024$. We use the Deepspeech (Authors, 2021) to translate the noise speech audio into text and obtain the word embeddings with the 300d Glove model (Pennington et al., 2014).

We set the frame number of video $N_v$ as 64, and the frame length of audio $N_a$ as 1024 for ActivityNet Speech dataset. The hidden size of all layers for both audio and visual features is set to 1024. We split the features into 8 chunks to improve the model stability, and average the results of all chunks at last. The prediction step K for audio pretraining process is set to 3, and the default audio length is set to 256. For curriculum learning, the default pacing function is linear function, and $\kappa =10$. During training, we apply Adam optimizer to optimize the model with a warmup strategy, the initial learning rate is set to 0.001. Following the previous works (Gao et al., 2017; Mun et al., 2020), we apply ”R@n, IoU = m” and ”mIoU” as our evaluation metrics.

Since the spoken video grounding is a new task, none of the previous works focus on handling it. Therefore, we set the model as depicted in Section 4 as our baseline model, which directly uses raw speech to localize the video fragments without any audio pretrain process, also we implement VSLNet (Zhang et al., 2020) with audio by ourselves. Besides, we compare our proposed method with another audio pretrain model, VQCPC (Wang et al., 2021), which combine VQ-VAE (van den Oord et al., 2017) and CPC together to extract speech representations. We also use the DeepSpeech to translate the noise audio into text to test the performance of the baseline model with these ASR transcripts.

The main results are shown in Table 2. Specifically, our proposed method can outperform the baseline model with a large margin, especially achieving score 1.23$\%$ absolute improvements in $IoU=0.7$. The results of row 2 and row 4 indicate that the vanilla CPC is more suitable for this task. Further, as shown in row 4 and row 6, with the introduction of the video content, the performance is improved with a notable margin in both IoU and mIoU scores, which demonstrates that the video information can facilitate the audio denoising. The result in row 7 and row 8 indicate that although the text transcripts recognized by ASR have lots of errors, the introduction of Glove word embedding model can still guarantee video grounding performance. However, with the equipment of the proposed curriculum learning strategy, our VGCL can surpass the text-based grounding model in both accuracy and efficiency.

Model components To investigate the effect of individual components in our proposed VGCL module, we conduct a series of ablation studies, the results are shown in Table 3. To demonstrate the effectiveness of the curriculum learning, we remove it and simply use the entire video content to do cross-attention with audio features, denotes as ”w/o CL”. The scores drop a lot, which shows that without the highlight of ground truth visual features, the model can barely extract useful information from complex video contents. To further verify the effectiveness of the introduction of the video, we remove the entire video content, leaving only the masked video part, which denotes ”w/o entire video”. Also we remove the video self-attention layer, which denotes as ”w/o self attention”. The results show that both the entire video content and the self-attention layer can help extract context visual information, which is vital for audio pretraining. The default audio length when training CPC is set to 256, here we test the performance when the audio length is set to 128. The score decreases 0.3 in $IoU=0.7$, indicating that the suitable length will benefit the audio pretraining process, short audio segments are difficult to extract relative information from the input video.

Analysis on Pacing Functions We conduct a series of experiments to compare the models pretrained with different pacing functions and different curriculum learning steps. From the Table 4 we can see that the performances of the models trained with linear function and exponential function are equivalent, and slightly outperforms the model trained with logarithmic function. As we described in Section 5.3, logarithmic function focuses more on the easier samples while exponential function focuses more on harder samples. In addition, in order to explore the impact of different training steps in the pre-training stage, we test the effect of the pretrained audio encoder when $t\in (1,\kappa )$ with different pacing functions, as shown in Fig 5. The results show that logarithmic function performs better when t is small, and exponential function achieves the best performance when the pre-training is finished. These also demonstrate that more training steps on harder curriculum stages is beneficial to our task.

Analysis on Different Proportion of Noise To further verify the effectiveness of our model, we compare the performances of the models trained under different proportions of noise, as shown in Fig 4. The results demonstrate that under different proportions of noise, our VGCL can facilitate the contrastive prediction process and obtain a mature audio encoder for spoken video grounding tasks. It is noticeable that the improvement after changing the model from baseline to our VGCL when the audio is clean is lower than the situation where the audio is noisy. From the figure we can also find that when the audio is clean, the text-based method performs better. However, when the proportion of noise is getting larger, our model can surpass the text-based method by a certain margin. This indicates that our VGCL can utilize the corresponding video content to rectify the noisy phonemes and improve the grounding performance, which is superior in real application.

To further qualitatively compare our method with baseline model and text-to-video grounding model based on ASR transcripts, we analyze two examples from Activitynet Speech dataset, as shown in Fig 6 and Fig 7. From the success case in Fig 6 we can see that compared with the other two models, our VGCL can accurately locate the correct event boundaries. The ASR transcripts have a large deviation in the key verbs due to the interference of noise, leading to the wrong localized moments. This demonstrates the difficulty for the model to learn a reliable correspondence between ASR transcripts from noisy audio and video actions.

From the failure case in Fig 7 we can observe that a woman is riding a machine in many frames, however, the key world ”continue” emphasizes that the model should focus on the latter frames. Our model fails to predict the ground truth action boundaries, but still have a high score for the right start position. This example indicates that our model still lacks of the ability for understanding some key adjectives and adverbs, which needs to be strengthened in future.