CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillation

Knowledge Modeling:  To perform knowledge distillation between modalities, we first need to model the knowledge learned in each modality in a proper way. It needs to take advantage of the existing contrastive learning framework to avoid introducing excessive computational overhead. Moreover, since the distillation is performed cross-modally for self-supervised learned knowledge, conventional methods that rely on individual features/logits are no longer applicable.

Knowledge Distillation:  Based on the aforementioned probability distributions, an intuitive way to perform knowledge distillation would be to directly establish consistency constraints between skeleton modalities. Different from previous knowledge distillation approaches that transfer the knowledge of a fixed and well-trained teacher model to the student, in our approach, the knowledge is continuously updated during self-supervised pre-training and each modality acts as both student and teacher.

Cross-modal Positive Mining:  Cross-modal positive mining is the most important component in CrosSCLR [li20213d], where the most similar negative sample is selected to boost the positive sets for contrastive learning in complementary modalities. The contrastive loss for modality B is reformulated as:

where $u$ is the index of most similar negative sample in modality A.

CMD with ${\tau }_t=0$ and $K=N$:  Setting temperature ${\tau }_t=0$ and $K=N$, the key distribution $p(z_k^a,{\tau }_t)$ in Eq. 4 will be an one-hot vector with the only $1$ at index $u$, and thus the loss works on modality B will be like:

We can find that the loss $L_{\text{CMD}}^b$ is essentially doing contrastive learning in modality B with the positive sample mined by modality A. Compared with Eq. 6, the only difference is that when the mined $m_u^b$ is taken as the positive sample, the key embedding $z_k^b$ is excluded from the denominator. The same result holds for modality A. Thus we draw a conclusion that the cross-modal positive mining performed in CrosSCLR [li20213d] can be regarded as a special case of our approach with the temperature of teacher ${\tau }_t=0$ and the number of neighbors $K=N$.

Network Architecture:  In our approach, we adopt a 3-layer Bidirectional GRU (BiGRU) as the base-encoder, which has a hidden dimension of 1024. Before the encoder, we additionally add a Batch Normalization [ioffe2015batch] layer to stabilize the training process. Each skeleton sequence is represented in a two-actor manner, where the second actor is set to zeros if only one actor exists. The sequences are further resized to a temporal length of 64 frames.

Self-supervised Pre-training:  During pre-training, we adopt MoCo v2 [chen2020improved] to perform single-modal contrastive learning. The temperature hyper-parameter in the InfoNCE [van2018representation] loss is 0.07. In cross-modal mutual distillation, the temperatures for teacher and student are set to 0.05 and 0.1, respectively. The number of neighbors $K$ is set to 8192. The SGD optimizer is employed with a momentum of 0.9 and a weight decay of 0.0001. The batch size is set to 64 and the initial learning rate is 0.01. For NTU RGB+D 60 [shahroudy2016ntu] and NTU RGB+D 120 [liu2020ntu] datasets, the model is trained for 450 epochs, the learning rate is reduced to 0.001 after 350 epochs, and the size of the memory bank $N$ is 16384. For PKU-MMD II [liu2017pku] dataset, the total epochs are increased to 1000, and the learning rate drops at epoch 800. We adopt the same skeleton augmentations as ISC [thoker2021skeleton].

NTU RGB+D 60 [shahroudy2016ntu]:  NTU-RGB+D 60 (NTU-60) is a large-scale multi-modality action recognition dataset which is captured by three Kinect v2 cameras. It contains 60 action categories and 56,880 sequences. The actions are performed by 40 different subjects (actors). In this paper, we adopt its 3D skeleton data for experiments. Specifically, each human skeleton contains 25 body joints, and each joint is represented as 3D coordinates. Two evaluation protocols are recommended by the authors: cross-subject (x-sub) and cross-view (x-view). For x-sub, action sequences performed by half of the 40 subjects are used as training samples and the rest as test samples. For x-view, the training samples are captured by camera 2 and 3 and the test samples are from camera 1.

NTU RGB+D 120 [liu2020ntu]:  Compared with NTU-60, NTU-RGB+D 120 (NTU-120) extends the action categories from 60 to 120, with 114,480 skeleton sequences in total. The number of subjects is also increased from 40 to 106. Moreover, a new evaluation protocol named cross-setup (x-set) is proposed as a substitute for x-view. Specifically, the sequences are divided into 32 different setups according to the camera distances and background, with half of the 32 setups (even-numbered) used for training and the rest for testing.

PKU-MMD [liu2017pku]:  PKU-MMD is a new benchmark for multi-modality 3D human action detection. It can also be used for action recognition tasks [lin_ms2l_mm]. PKU-MMD has two phases, where Phase II is extremely challenging since more noise is introduced by large view variation. In this work, we evaluate the proposed method on Phase II (PKU-II) under the widely used cross-subject evaluation protocol, with 5,332 skeleton sequences for training and 1,613 for testing.

Evaluation Metrics:  We report the top-1 accuracy for all datasets.

Linear Evaluation Protocol:  For linear evaluation protocol, we freeze the pre-trained encoder and add a learnable linear classifier after it. The classifier is trained on the corresponding training set for 80 epochs with a learning rate of 0.1 (reduced to 0.01 and 0.001 at epoch 50 and 70, respectively). We evaluate the proposed method on the NTU-60, NTU-120, and PKU-II datasets. As shown in Table 1, we include the recently proposed CrosSCLR [li20213d], ISC [thoker2021skeleton], and AimCLR [guo2022aimclr] for comparison. Our approach outperforms previous state-of-the-art methods by a considerable margin on all the three benchmarks. Note that ISC and the proposed CMD share the same BiGRU encoder, which is different from the ST-GCN [yan2018spatial] encoder in CrosSCLR. For a fair comparison, we additionally train a variation of CrossSCLR with BiGRU as its base-encoder (denoted as CrosSCLR-B). We can find that our method still outperforms it on all the three datasets, which shows the superiority of the proposed cross-modal mutual distillation.

KNN Evaluation Protocol:  An alternative way to use the pre-trained encoder for action classification is to directly apply a K-Nearest Neighbor (KNN) classifier to the learned features of the training samples. Following [su2020predict], we assign each test sample to the most similar class where its nearest neighbor is in (i.e. KNN with k=1). As shown in Table 3, we perform experiments on the NTU-60 and NTU-120 benchmarks and compare the results with previous works. For both datasets, our approach exhibits the best performance, surpassing CrosSCLR-B [li20213d] by 4.5%~6% in the more challenging cross-subject and cross-setup protocols.

Transfer Learning Evaluation Protocol:  In transfer learning evaluation protocol, we examine the transferability of the learned representation. Specifically, we first utilize the proposed framework to pre-train the encoder on the source dataset. Then the pre-trained encoder along with a linear classifier are finetuned on the target dataset for 80 epochs with a learning rate of 0.01 (reduced to 0.001 at epoch 50). We select NTU-60 and NTU-120 as source datasets, and PKU-II as the target dataset. We compare the proposed approach with previous methods LongT GAN [zheng2018unsupervised], MS${}^2$L [lin_ms2l_mm], and ISC [thoker2021skeleton] under the cross-subject protocol. As shown in Table 3, our approach exhibits superior performance on the PKU-II dataset after large-scale pre-training, outperforming previous methods by a considerable margin. This indicates that the representation learned by our approach is more transferable.

Semi-supervised Evaluation Protocol:  In semi-supervised classification, both labeled and unlabeled data are included during training. Its goal is to train a classifier with better performance than the one trained with only labeled samples. For a fair comparison, we adopt the same strategy as ISC [thoker2021skeleton]. The pre-trained encoder is fine-tuned together with the post-attached linear classifier on a portion of the corresponding training set. We conduct experiments on the NTU-60 dataset. As shown in Table 4, we report the evaluation results when the proportion of supervised data is set to 1%, 5%, 10%, and 20%, respectively. Compared with previous methods LongT GAN [zheng2018unsupervised], MS${}^2$L [lin_ms2l_mm], ASSL [si2020adversarial], and ISC [thoker2021skeleton], our algorithm exhibits superior performance. For example, with the same baseline, the proposed approach outperforms ISC and CrosSCLR-B by a large margin. We also take 3s-CrosSCLR [li20213d], 3s-Colorization [yang2021skeleton], and recently proposed 3s-AimCLR [guo2022aimclr] into comparison. In these methods, test-time multimodal ensembling is performed and the results of using 1% and 10% labeled data are reported. We can find that our 3s-CMD still outperforms all of these methods after ensembling multiple skeleton modalities.

Number of neighbors:  The number of nearest neighbors controls the abundance of contextual information used in the proposed cross-modal mutual distillation module. We test the performance of the learned representation with respect to different numbers of nearest neighbors $K$ under the linear evaluation protocol. As shown in Figure 3, on the downstream classification task, the performance of the pre-trained encoder improves as $K$ increases. When $K$ is large enough ($K\ge 8192$), continuing to increase its value hardly contributes to the performance. This is because the newly added neighbors are far away and contain little reference value for describing the distribution around the query sample. In addition, we can also find that when the value of $K$ varies from 64 to 16384, the performance of our approach is consistently higher than that of CrosSCLR-B [li20213d] and our baseline. This demonstrates the superiority and robustness of the proposed approach.

Modality Selection:  In our approach, we consider three kinds of skeleton modalities for self-supervised pre-training as in [li20213d]. They are joint, motion, and bone, respectively. Our approach is capable of performing knowledge distillation between any two of the above modalities. As shown in Table 5, we report the performance of the representation obtained by pre-training with different combinations of skeleton modalities. Note that the joint modality is always preserved since it is used for evaluation. There are several observations as follows: i) Cross-modal knowledge distillation helps to improve the performance of the representation in student modalities. ii) Under the linear evaluation protocol, knowledge distillation between joint and motion achieves the optimal performance, exceeding the baseline by 3.7%. iii) Under the KNN evaluation protocol, the learned representation shows the best results when all the three modalities are involved in knowledge distillation, which outperforms the baseline with an absolute improvement of 7.2%.

Bidirectional Distillation:  In addition to modality selection, we also verify the effectiveness of bidirectional distillation. It enables the modalities involved in the distillation to interact with each other and progress together, forming a virtuous circle. In Table 5, the last column of each evaluation protocol reports the performance gain of bidirectional mutual distillation over unidirectional distillation in the joint modality. Results show that regardless of which skeleton modalities are used during pre-training, bidirectional mutual distillation further boosts the performance, especially under the KNN evaluation protocol.

Qualitative Results:  We visualize the learned representation of the proposed approach and compare it with that of the baseline. The t-SNE [van2008visualizing] algorithm is adopted to reduce the dimensionality of the representation. To obtain clearer results, we select only 1/4 of the categories in the NTU-60 dataset for visualization. The final results are illustrated in Figure 4. For both joint and motion modalities, the representation learned by our approach is more compactly clustered than those learned by the baseline in the feature space. This brings a stronger discrimination capability to the representation, explaining the stunning performance of our approach in Table 3.