Partially Relevant Video Retrieval

Before constructing video clips, we first downsample the input in the temporal domain to reduce the length of the feature sequence, which helps reduce the computational complexity of the model. Specifically, given a sequence of frame feature vectors $V$ as the input, we downsample them into a fixed number of feature vectors, where each feature vector is obtained by mean pooling over the corresponding multiple consecutive frame features. Then, the video is described by a sequence of new feature vectors $U\in R^{d_v\times n_u}$, where $n_u$ indicates the number of the corresponding feature vectors. In order to make the features more compact, we employ an FC layer with a ReLU activation. Moreover, we also use standard Transformer (Vaswani et al., 2017) with a learned positional embedding to improve the temporal dependency of the features. Formally, through an FC layer and a one-layer Transformer, we obtain $U^{\prime }\in R^{d\times n_u}$:

where $PE$ denotes the output of the positional embedding. The reduced feature array $U^{\prime }$ is further used for clip-scale video representation learning.

For clip construction, we employ a multi-scale sliding window strategy to generate video clips, as illustrated in Fig. 3. Note that different from previous work (Zhang et al., 2020a) where video clips are of equal length and non-overlapping, our video clips are of varied lengths and overlapping. Concretely, we apply sliding windows of different sizes over $U^{\prime }$ along its temporal dimension with a stride of 1. Given a sliding window of size $k$, a clip feature is obtained by mean pooling over the features within the given window. The resultant feature sequence is denoted as ${\Phi }_k$. Consequently, by jointly employing sliding windows of varied sizes as $\{1,2,...,n_u\}$, we are able to obtain $\{{\Phi }_1,{\Phi }_2,...,{\Phi }_{n_u}\}$. Putting them together, a video can be represented a sequence of video clips $C\in R^{d\times n_c}$:

where $c_i\in R^d$ denotes the feature representation of $i$-th clip, $n_c$ is the number of all generated clips which meets $n_c=n_u\left(n_u+1\right)/2$.

As the initial frame features are extracted independently, they naturally lack temporal dependency. To bring such dependency back, we again utilize Transformers. Specifically, given the frame feature sequence $V$, we first utilize an FC layer with ReLU activation to reduce the dimensionality of the input, followed by a standard Transformer with a positional embedding layer. The re-encoded frame features, denoted $F\in R^{d\times n_v}$, are computed as:

Note that the network structures of Transformer, FC and PE are the same as that in the clip-scale branch, but their trainable parameters are not shared. This allows each branch to learn parameters suitable for their own scale.

Given a video as a sequence of video clips, we first measure the cross-modal similarity of each video clip with the query, and then aggregate the instance similarities to obtain the clip-scale similarity. Specifically, given a sequence of video clips $C=\{c_1,c_2,...,c_{n_c}\}$, we use cosine similarity between each instance representation and the query representation, followed by a max-pooling operator on the similarities. More formally, the clip-scale similarity is obtained as:

where $cos(\cdot )$ denotes the cosine similarity function. The max-pooling determines the clip with the highest similarity, and we then utilize its similarity with the query as the whole video’s similarity with the query. Besides, we select this clip as key clip, which is used for the latter frame-scale similarity learning.

To obtain the frame-scale similarity, we first aggregate a sequence of frame feature vectors to a feature vector under the guidance of the key clip obtained in Section 3.4.1, and then compute its similarity with the query as the frame-scale similarity. Specifically, given a sequence of video frames $F=\{f_1,f_2,...,f_{n_v}\}$, we devise a Key Clip Guided Attention (KCGA) to aggregate frame features. The implementation of KCGA borrows the idea of multi-head self-attention (MHSA) mechanism in Transformer (Vaswani et al., 2017). MHSA first projects the input into queries, keys, values, and then computes the output as a weighted sum of the values. The weight assigned to each value is computed by a compatibility function of the query with the corresponding key. Different from MHSA utilizes the same input to construct queries, keys, and values, here we take the feature vector of the key clip as the query, and the video frame features as keys and values. Formally, the aggregated frame feature vector is obtained as:

where $\tilde{c}\in R^{d\times 1}$ indicates the feature vector of the key clip, $W_k\in R^{d\times d}$ and $W_k\in R^{d\times d}$ are two trainable projection matrices. The dot product measures the similarity between frames and the key clip, resulting in larger values for frames that are more similar to the key clip. Therefore, frames that are more similar to the key clip will have greater attention weights.

Finally, the frame-scale similarity is measured as the cosine similarity between the aggregated frame feature vector $r$ and query feature vector $q$, namely:

In this section, we first introduce the definition of positive and negative pairs for similarity learning. Inspired by MIL (Dietterich et al., 1997; Maron and Lozano-Pérez, 1997), we define that a query and video pair is positive if the video contains certain content that is relevant to the query, and negative if no relevant content in the video.

Based on the above definition, we jointly use the triplet ranking loss (Faghri et al., 2018; Dong et al., 2021) and InfoNCE loss (Miech et al., 2020; Zhang et al., 2021) that are widely used in retrieve related tasks, and found them complementary. Given a positive video-query pair $(v,q)$, the triplet ranking loss over the mini-batch $B$ is defined as:

where $m$ is the margin constant, $S(\cdot )$ denotes the similarity function which we can use the clip-scale similarity $S(\cdot {)}_c$ or the frame-scale similarity $S(\cdot {)}_f$. Besides, $q^{-}$ and $v^{-}$ respectively indicate a negative sentence sample for $v$ and a negative video sample for $s$. The negative samples are randomly sampled from the mini-batch at the beginning of the training, while being the hardest negative samples after 20 epochs.

Given a positive video-query pair $(v,q)$, the infoNCE loss over the mini-batch $B$ is computed as:

where $N_q$ denotes all negative queries of the video $v$ in the mini-batch, while $N_v$ denotes all negative videos of the query $q$ in the mini-batch.

As the previous work (Li et al., 2020) has concluded that using one loss per similarity function performs better than using one loss on the summation of multiple similarities, we employ the above two losses on both clip-scale similarity and frame-scale similarity, instead of their sum. Finally, our model is trained by minimizing the following overall training loss:

where $L_c^{trip}$ and $L_f^{trip}$ denote the triplet ranking loss using the clip-scale similarity $S(\cdot {)}_c$ and frame-scale similarity $S(\cdot {)}_f$ respectively, and accordingly for $L_c^{nce}$ and $L_f^{nce}$. ${\lambda }_1$ and ${\lambda }_2$ are hyper-parameters to balance the contribution of infoNCE loss.

In order to verify the viability of our proposed model for PRVR, queries that are partially relevant to videos are required. As videos in popular T2VR datasets such as MSR-VTT (Xu et al., 2016), MSVD (Chen and Dolan, 2011) and VATEX (Wang et al., 2019) are supposed to be fully relevant to the queries, they are not suited for our experiments. Here, we re-purpose three datasets commonly used for VCMR, i.e., TVR (Lei et al., 2020), Activitynet Captions (Krishna et al., 2017), and Charades-STA (Gao et al., 2017), considering their natural language queries partially relevant with the corresponding videos (a query is typically associated with a specific moment in a video). Table 1 summarizes the brief statistics of these datasets, including average lengths of moments and videos, and the average moment length proportion in the whole video (moment-to-video ratio). Note that as we focus on retrieving videos, moment annotations provided by these datasets are not used in our proposed new PRVR task.

TV show Retrieval (TVR) (Lei et al., 2020) is a multimodal dataset originally for video corpus moment retrieval, where videos are paired with subtitles that are generated by automatic speech recognition. It contains 21.8K videos collected from 6 TV shows, and each video is associated with 5 natural language sentences that describe a specific moment in the video. As a moment is typically a part of a video, we assume that sentences are partially relevant to videos, and use them to evaluate our model. Following (Zhang et al., 2020a, 2021), we utilize 17,435 videos with 87,175 moments for training and 2,179 videos with 10,895 moments for testing.

ActivityNet Captions (Krishna et al., 2017) is originally developed for dense video captioning task, and is now a popular dataset for single video moment retrieval. It contains around 20K videos from Youtube, and the average length of videos is the largest among the three datasets we used. On average, each video has around 3.7 moments with corresponding sentence descriptions.We use the popular data partition used in (Zhang et al., 2020a, 2021).

Charades-STA (Gao et al., 2017) is a dataset for single video moment retrieval. It contains 6,670 videos with 16,128 sentence descriptions. Each video has around 2.4 moments with corresponding sentence descriptions on average. We utilize the official data partition for model training and evaluation.

To evaluate PRVR models, we utilize the rank-based metrics, namely $R@K$ ($K=1,5,10,100$), which are commonly used for the conventional text-to-video retrieval (Wang et al., 2020b; Dong et al., 2021). $R@K$ is the fraction of queries that correctly retrieve desired items in the top $K$ of the ranking list. The performance is reported in percentage (%). Higher $R@K$ means better performance. For overall comparison, we also report the Sum of all Recalls (SumR).

We use PyTorch as our deep learning environment, and we will release our source code. For video feature, on TVR, we utilize the feature provided by (Lei et al., 2020), that is, 3,072-D visual feature obtained by the concatenation of frame-level ResNet152 (He et al., 2016) feature and segment-level I3D (Carreira and Zisserman, 2017) feature. For ease of reference, we refer to it as ResNet152-I3D. On ActivityNet-Captions and Charades-STA, we only utilize the same I3D feature, which are respectively provided by (Zhang et al., 2020a) and (Mun et al., 2020). For sentence feature, we use the 768-D RoBERTa feature provided by (Lei et al., 2020) on TVR, where RoBERTa is finetuned on the queries and subtitle sentences of TVR. On ActivityNet-Captions and Charades-STA, we use the 1,024-D RoBERTa feature extracted by ourselves using the open RoBERTa toolkit¹¹1https://pytorch.org/hub/pytorch_fairseq_roberta/. Due to the limited space of the paper, we present more detailed implementation details in the supplementary material.

As models specifically designed for PRVR are non-existing, we compare with models targeted at conventional T2VR and models developed for VCMR. Given the rich literature, we have to be selective, choosing open-source models for fair and reproducible comparison. In particular, we choose the following nine T2VR models, i.e., VSE++ (Faghri et al., 2018), W2VV (Dong et al., 2018), CE (Liu et al., 2019a), W2VV++ (Li et al., 2019), DE (Dong et al., 2019), HTM (Miech et al., 2019), HGR (Chen et al., 2020b), DE++ (Dong et al., 2021) and RIVRL (Dong et al., 2022), and the following two VCMR models, i.e., XML (Lei et al., 2020) and ReLoCLNet(Zhang et al., 2021). Both XML and ReLoCLNet are two-stage, where a first-stage module is used to retrieve candidate videos followed by a second-stage module to localize specific moments in the candidate videos. As moment annotations are unavailable for PRVR, we have re-trained XML and ReLoCLNet (with their moment localization modules removed) using the same video features as ours.

Table 2 summarizes the performance comparison on TVR. Our proposed model consistently outperforms all conventional T2VR models with a clear margin. Even the best performing model RIVRL among the T2VR models, our model outperforms it by 36.8 in terms of SumR. As these models focus on the whole similarity between videos and queries, the results allow us to conclude that such similarity modeling is sub-optimal for PRVR. For the models of the second group, i.e., ReLoCLNet and XML, they perform better than the conventional T2VR models, but they are still worse than ours. ReLoCLNet and XML focus on retrieving moment, which to some extent model the partial relevance, but they compute the similarity only in terms of a specific scale. By contrast, we compute the similarity in terms of both clip scale and frame scale. The results demonstrate the effectiveness of our proposed multi-scale similarity learning for PRVR. Note that when using the extra subtitle feature provided by (Lei et al., 2020), our model obtains better performance (R@1 of 24.0 and SumR of 220.8).

To gain a further understanding of the individual models, we define moment-to-video ratio (M/V) for query, which is measured by its corresponding moment’s length ratio in the entire video. The smaller M/V indicates less relevant content while more irrelevant content with respect to the query. Besides, the smaller M/V to some extent means a lower relevance of a query to its corresponding video, while the larger one indicates a higher relevance. According to M/V, queries can be automatically classified into different groups, which enables a fine-grained analysis of how a specific model responds to the different types of queries. On TVR, the 10,895 test queries are split according to their M/V into six groups, with the performance of each group shown in Fig. 5.

Unsurprisingly, our model consistently performs the best in all groups. Observing the figure from left to right, the average performance of the twelve compared models increases along with the M/V, from 106.8, 114.2 114.3, 118.6, 125.8 and 127.7. The performance in the group with the lowest M/V is the smallest, while the group with the highest M/V is the largest. The result allows us to conclude that the current video retrieval baseline models better address queries of larger relevance to the corresponding video. By contrast, the performance we achieved is more balanced in all groups. This result shows that our proposed model is less sensitive to irrelevant content in videos.

The performance of different models on Activitynet Captions and Charades-STA are summarized in Table 3 and Table 4, respectively. On both datasets, our model is still at the leading position.The results again verify the effectiveness of our model for measuring partial relevance between videos and queries. Interestingly, we observe that HTM performs badly on TVR and Activitynet Captions, while on Charades-STA it achieves the best SumR score among the T2VR models. We speculate it is due to the fact that Charades-STA has the least training data among the three datasets. Besides, the model structure of HTM is very simple, respectively using an FC layer with gating mechanism to embed videos and sentences into a common space, showing an advantage of training on small-scale data. For our proposed model, it consistently performs the best on the three datasets of the varying number of training samples, which to some extent shows that our model is not sensitive to the scale of training data.

To examine the usefulness of the multi-scale branches, we compare the counterpart without the clip-scale branch or the frame-scale branch. As shown in Table 6, removing any branch results in clear performance degeneration. The result not only demonstrates the effectiveness of the multi-scale solution, but also shows the complementary of the clip-scale and the frame-scale branches.

Additionally, we also compare the model w/o key clip guide, which is implemented by replacing key clip guided attention with a simple attention. The simple attention is implemented as Eq. 1 without any guide. As Table 6 shows, our model with the full setup still performs better, which shows the importance of key clip guided attention for PRVR.

To validate the choice of joint use of the two losses, we compare the results of using either triplet ranking loss or infoNCE loss. As shown in Table 6, triplet ranking loss and InfoNCE give comparable results when used alone, but they are much worse than the model with the full setup of jointly using both. The result demonstrates the benefit of using these two losses jointly.

The influence of the hyper-parameter $\alpha$ in Eq. 11 is studied as follows. We try $\alpha$ with its value ranging from 0.1 to 0.9 with an interval of 0.1. As shown in Fig. 7, when the $\alpha$ is larger than 0.3, the performance of using multi-scale similarity are all over 170, which consistently outperform the counterparts using the frame-scale or the clip-scale similarity alone.