Temporal Contrastive Learning with Curriculum

Data. Let $X=\left\{x_i\right\},i\in [1,N]$ denote the training dataset, where $N$ is the total number of training video instances. Each video is a stack of RGB frames of size $3\times T\times S^2$, where $T$ is the total number of frames, and $S$ is the spatial dimension of each frame. First, we re-sample the input video at a desired frame per second (fps) rate $f$. We then sample a clip from the video with $t$ consecutive frames to obtain a $3\times t\times s^2$ clip, where $s$ is the input spatial dimensions to the model.

For a reasonably long video, the semantic context at the start of a video can be very dissimilar to the end of that video. This is why sampling positive clips over the entire duration of a video and blindly encouraging their respective embeddings to be situated close to one another is not reasonable. To tackle this problem, we propose a sampling method that imposes a dynamic constraint on the temporal span from which positive clips are sampled. The proposed sampling technique only picks positive samples from inside the defined temporal span, and any clip sampled from a different video is considered a negative. A visual illustration of the proposed sampling technique is shown in Fig. 2. Given, a video clip with a temporal length (number of frames) $T$, and a temporal span ($TS$), the method first picks a random starting frame $s\in [1,T-TS]$. Accordingly, the temporal window for positives is defined as $w\in [s,s+TS]$. We then sample $\rho$ clips of $t$ consecutive frames from the sampling window $w$, and $\rho$ random spatial augmentations $\{t^1,t^2,...,t^{\rho }\}$ from $\tau$ which are applied on the sampled clips to get $\rho$ positive samples $x^{+}=\{x_i^1,x_i^2,...x_i^{\rho }\}$. In the multi-instance momentum contrastive setting, one positive sample is treated as the query and $\rho -1$ positive samples as keys. Following [chen2020simple], we adopt a symmetric version of the loss, where each clip in $x^{+}$ is considered as query and $\rho -1$ clips are considered positives $\left\{k^{+}\right\}$. The modified loss for Multi-instance MoCo is represented as follows (the total loss is averaged over $\rho$):

In the following sub-section we describe how we use curriculum learning to progressively update $TS$.

In video contrastive settings, if a pair of sampled clips $(x_i^a,x_i^b)$ are very close or overlap with one another, they are more likely to contain semantically similar content. We define pairs that are temporally close as ‘easy’ positive pairs. On the other hand, pairs $(x_i^a,x_i^c)$ that are temporally far apart are considered ‘hard’ positives. Here, we propose a hardness measure that gradually increases the temporal span, $TS$, of positive samples over the training epochs, effectively hardening the positive samples for our contrastive loss. The proposed curriculum learning component is illustrated in Fig. 3 (left), where the training starts with a short temporal span and is then increased over the training iterations. We investigate the proposed component in two settings. In first setting, we increase the hardness over the entire training phase. In the second setting, we limit the number of epochs over which hardening occurs ($E_{CL}$), and a constant hardness is used beyond that threshold (see Fig. 3 (right)). The temporal span at a given epoch $e$ is formulated as:

where $T{S}_m$ is the maximum temporal span, $T{S}_i$ is the initial temporal span, and $E_{CL}$ is the total number of epochs over which hardening is performed.

We define an auxiliary task of predicting the temporal distance (number of frames) between any two positive clips (key and query) given their learned embeddings. This is done to help the model learn better contextual information regarding the location of the positive samples. To predict the distance between a query embedding $q$ and a key embedding $k$, we add a single linear layer that takes the concatenation of $q$ and $k$ as input and generates the context similarity prediction.

Let $\varphi \left(x\right)$ be a function that returns the frame number at which a positive clip ($x$) starts within the original video. We define the distance between the query clip $x_i^q$ and any positive key clip $x_i^k$ as $d_k=abs\left(\varphi \right(x_i^q)-\varphi (x_i^k\left)\right)$, and the predicted distance by the model as $d_k^{\prime }$. For $\rho$ positive keys in the proposed setting, we define the context similarity loss as:

Accordingly, the overall loss for ConCur is defined as:

where $L_{MI}$ is the multi-instance MoCo loss presented earlier in Eq. 3. The overall diagram for our method is illustrated in Fig. 4.

Following other works in the area [feichtenhofer2021large, pan2021videomoco], we pre-train our model with Kinetics400 and Kinetics200, and fine-tune on UCF101 and HMDB51 datasets. In the following, we present a short description of each of the datasets used in this paper.

In this sub-section, we describe the detailed settings of our training along with other implementation details about contrastive pre-training of the model, downstream fine-tuning, and the evaluation protocol. We also summarize the video retrieval settings and the linear evaluation protocol.

First, we pre-train the video encoder with the proposed ConCur loss on Kinetics400. We report the performance of the method with two popular video encoders namely R(2+1d) [tran2018closer] and C3D [tran2015learning]. While all the ablation and sensitivity studies in this paper are conducted using the R(2+1)D encoder, other encoders are used in the final results and comparisons to the state-of-the-art. Following the self-supervised learning literature [chen2020simple, he2020momentum], we use an MLP projection head after the last layer (global pooling layer) of the video encoder, which embeds the output of the video representation into a vector of dimension 128. The MLP layer used here is a simple 2-layer fully connected network of sizes $(n\times 2048)$ and $(2048\times 128)$. Here, $n$ is the output vector dimension of the video encoder. We use a ReLU activation layer between the fully-connected layers. The output of the MLP projection head is normalized before using the proposed loss function. Note that the MLP projection head is used only in the pre-training stage and is not involved in fine-tuning. Instead, a single linear layer is added on top of the output of the video encoder with output dimensions equal to the number of classes for the final prediction task.

The model is implemented with PyTorch framework and trained with 8 Nvidia V100 GPUs. The pre-training is done with a batch-size of 16 clips per GPU. Momentum SGD is used for the training with a momentum value of 0.9. Following [chen2020simple], a temperature value $\alpha$ of 0.07 is used. Shuffling BatchNorm [he2020momentum] is utilized for the training. A base learning rate of 0.02 is used for training which is linearly warmed up from 0 to 0.02 over 5 epochs, and a cosine learning rate decay is used to reduce the learning rate for the rest of the training epochs. A weight decay regularization is used with a value of 0.0001. We pre-train the randomly initialized encoder for a total of 400 epochs for the final model. For all the ablation studies the encoder is trained for 100 epochs. Unless mentioned otherwise, we use $t=16$ as the number of frames with a spatial resolution of $s=112$ for the input to the model. The clips are selected after the original videos are re-sampled to 15 fps. For the momentum encoder, we use the key dictionary of size 65536, and a momentum value of 0.999.

For the data augmentation, we largely follow the augmentation protocol introduced in [feichtenhofer2021large]. We follow Inception-style [szegedy2015going] cropping with a minimum crop area of 0.2 and a maximum crop area of 0.76 following [feichtenhofer2021large]. We apply random horizontal flip, color distortion, and Gaussian blur following SimCLR [chen2020simple] and MoCo v2 [chen2020improved]. The random gray scaling is applied with a probability of 0.2 and random horizontal flip with probability of 0.5. Following [feichtenhofer2021large] we apply color distortion with a probability of 0.8 and color strength of $\{$brightness, contrast, saturation, hue$\}=0.4s,0.4s,0.4s,0.1s$, with a default value of $s=0.5$. The Gaussian blur is applied with a probability of 0.5 using a spatial kernel with a standard dev. $\in$ [0.1, 2.0].

Following the standard video evaluation protocol [tran2018closer, luo2020video, xu2019self], we uniformly sample 10 clips from the test video alone its temporal axis. We then resize the smaller dimension to 112. Next, we take 3 spatial crops per clip (total 30 clips) with resolution $112\times 112$ to cover full spatial space. Final prediction for a video is the average over all the clips.

Here, we study the key parameters of our curriculum learning module. Note that these studies were done on the smaller K200 dataset and trained for 100 epochs.

First, we conduct an experiment for different temporal spans for sampling the positives. The results are presented in Fig. 5 (left), where it can be seen that the performance of the model increases when we increase the temporal span from 32 to 100. However, the accuracy drops when we use larger values. For example, a temporal span of 125 performs worse than 32. The accuracy is the lowest when the length of the full video is used as the temporal span. The reason behind this is likely the significant semantic contextual difference between the first and last parts of a long video. For the rest of the experiments, we set $TS=100$.

Next, we experiment with different $E_{CL}$ values to identify its impact and optimal value. We present the results in Fig. 5 (right) where we observe that $E_{CL}=50$ gives the best results, followed closely by $25$ and $75$. However, curriculum learning over a very short period (e.g., $E_{CL}=10$) did not result in any notable improvements compared to not using curriculum learning. Moreover, large $E_{CL}$ values also do not help the model. This is in line with our expectation of identifying an optimum level of hardness, where harder or simpler self-supervisory signals hurt the performance [appalaraju2020towards, sarkar2020self].

Here, we perform ablation experiments on K200 to investigate the effect of the key contributions of our work, namely curriculum learning and Context Similarity loss. The results are presented in Table 1.

As we observe from the table, removing the CS loss from ConCur reduces the accuracy by around 0.6%, which shows the importance of the context similarity loss in the proposed method. It should be noted that in recent years, improvements made by the state-of-the-art are in a similar range, often improving prior works by approximately 0.5% to 2.0%. Next we remove the curriculum learning component, effectively allowing the positives to be sampled from the entire duration of the video, which shows a drop of 0.9% accuracy on the downstream task. Finally, we train the model by removing both CL loss and curriculum learning and observe a drop of 1.6% accuracy. This shows the effectiveness of each of the components of our proposed method.

We present the main results in comparison with state-of-the-art methods on two video-related tasks, action recognition and video retrieval, which we perform on UCF101 and HMDB datasets as discussed earlier.

As shown in Table 2, when we use the C3D encoder in ConCur and pre-train it on UCF101, we achieve SOTA for both UCF101 and HMDB51 datasets in the full fine-tuning scheme. When pre-trained on the larger K400 dataset, our model achieves 1.2% and 3.6% improvements relative to the previous SOTA results on UCF101 and HMDB51 respectively. When we use R(2+1)D pre-trained on UCF101, ConCur outperforms other methods on HMDB51, but not on UCF101, as we achieve the second-best results behind RTT [jenni2020video]. However, ConCur achieves SOTA results on both of the datasets when R(2+1)D encoder is pre-trained on the larger K400 dataset. On HMDB51, we obtain a considerable improvement of 9% in accuracy, and a 2.6% improvement on UCF101.

Finally we present qualitative results using the video retrieval setup, where we pull the nearest neighbours of the query clip with high similarity. We show the input query and the retrieved clips with higher cosine similarity values in Fig. 6. As can be seen, videos with similar semantic appearance and motions are captured, indicating that our method learns important semantically relevant video features.