MAPLE: Masked Pseudo-Labeling autoEncoder for Semi-supervised Point Cloud Action Recognition

The task of point cloud action recognition is, given a point cloud video of humans, to predict the human behavior and actions in the video. Semi-supervised point cloud action recognition is consistent with the point cloud action recognition task in the inference phase, but usually adopts a different paradigm in the training phase as follows.

1) We have a dataset $D$, which contains a labeled subset $D_l=\left\{\right(x_i,y_i\left)\right\}$ and an unlabeled subset $D_u=\left\{x_j\right\}$. Both $D_l$ and $D_u$ are sampled i.i.d. from the same distribution $p\left(x\right)$ and in general the size of $D_l$ is smaller than the size of $D_u$. Let $x_i=\{X_t\in R^{(3+C)\times N}{\}}_{t=1}^T$ denote the matrix sequence of a point cloud video, where N indicates the number of points in each frame, $T$ indicates the number of frames in this video, $R^3$ and $R^C$ indicates the spatial coordinates and features dimension of one point. It is worth noting that there are no point cloud features (ie., $C=0$) in the given dataset (NTU RGB+D 60 (Shahroudy et al., 2016), NTU RGB+D 120 (Liu et al., 2020a) and MSR-Action3D (Li et al., 2010)).

2) The training of model $f_{\theta }(\cdot )$ has an optimization function of the following form:

where $L_l$ is the loss function (e.g., cross-entropy loss, mean squared error loss, Hinge loss, etc.) for classification of the labeled dataset $D_l$, and $L_u$ is the optimization objective designed for unlabeled dataset $D_u$ (the design of this function varies from paper to paper, and we discuss our designed $L_u$ in later subsection.), $\alpha$ is the positive scalar weight for $L_u$ and $\theta$ is the learnable parameters of $f_{\theta }(\cdot )$.

This subsection presents our backbone for point cloud action recognition named Decoupled spatial-temporal TransFormer (DestFormer). The design of DestFormer is based on P4Transformer  (Fan et al., 2021), and its main purpose is to serve as a basic backbone for the semi-supervised learning framework MAPLE and learn discriminative motion representations with fewer annotations. As shown in Fig. 7, the DestFormer consists of four parts: data preparation, spatial extractor, temporal aggregator, and prediction head. Note that embedding features in figures with different colors (e.g. “t1” and “t3”) correspond to the different keyframes of the input action sequence.

Data Preparation. For the input point cloud video $x_i$, we construct some local areas (e.g. “a”,“b”,“c”) on adjacent frames (e.g. “t1”,“t2”) as what Point 4D Convolution (P4Conv) (Fan et al., 2021) do. The calculation of the local areas is based on the Farthest Point Sampling (FPS) algorithm (Qi et al., 2017b), and the exhaustive calculation process is declared in (Fan et al., 2021).

Spatial Extractor (SE). SE is designed to extract the short-term global feature $g_i=\{G_{\zeta \cdot (t-1)+1}\in R^D{\}}_{t=1}^{T/\zeta }$ from the local areas. We first extracts the short-term local feature $s_i=\{S_{\zeta \cdot (t-1)+1}\in R^{D\times \left(N/\kappa \right)}{\}}_{t=1}^{T/\zeta }$ through P4Conv (Fan et al., 2021), where $D$ is the dimension of short-term local feature, $\kappa \ge 1.0$ denotes the spatial scaling rate and $\zeta \ge 1.0$ denotes the spatial scaling rate. P4Conv plays the role of aggregating the local information between adjacent $\zeta$ frames. After that, we feed the short-term local feature $s_i$ frame by frame into the Spatial Transformer (ST) modules for extracting the merged short-term local feature $m_i=\{M_{\zeta \cdot (t-1)+1}\in R^{D\times \left(N/\kappa \right)}{\}}_{t=1}^{T/\zeta }$ which aggregate the information of different spatial part.

Temporal Aggregator (TA). TA consists of a pooling layer and a transformer-based  (Vaswani et al., 2017) Temporal Encoder (TE). we first prepare the short-term global feature $g_i$ from the merged short-term local feature $m_i$ through the pooling layer (e.g. maximum pooling). Then we aggregate the long-term global $z_i$ from the short-term global feature $g_i$ with our TE module.

Prediction Head. Following the TA module is the pooling layer (e.g. maximum pooling) and classification head, which consists of Layer Normalization layers (LayerNorm), linear layers, and Gaussian Error Linear Units (GELUs). Its role is to project the global feature $v_i$ into the label space and generate the corresponding pseudo labels for classification.

This subsection elaborates our Masked Pseudo-Labeling autoEncoder (MAPLE) framework for semi-supervised point cloud action recognition. As shown in the left part of Fig. 3, we adopt our DestFormer backbone and the cross-entropy loss for the supervised training with labeled point cloud videos. The right part of Fig. 3 shows the complete training process of our MAPLE with unlabeled point cloud videos. Similar to the reconstruction process of autoencoders, the spatial extractor encodes the point cloud videos as the short-term local embedding feature $g_i$. The temporal encoder of our MAPLE projects the visible subset of embedding feature $g_i$ into latent space Z, and the temporal decoder is responsible for reconstructing from the latent representation $z_i$ and the learnable mask tokens $M$. However, different from classical autoencoders, our MAPLE implicitly reconstructs the original signal through the pseudo-label generated from the classification head, rather than reconstructing the original signal itself. We introduce the training process in detail as follows:

Masking the short-term global feature $g_i$ that is extracted from the Spatial Extractor (SE) modules is the first step of our framework. Like what ImageMAE does in (He et al., 2021), we directly discard a subset (e.g., 50%) of the original short-term global feature $g_i$ with random sampling. The motivation of masking is to help the model efficiently understand the order of actions via reconstructing the complete action sequence from the mutilated one.

Temporal Encoder (TE) is a lightweight transformer that only contains several self-attention blocks in the second step of our MAPLE. We directly feed the masked short-term global features $g_i$ into the TE module without adding positional embedding, since its temporal positional embedding is already added when fed into the SE module.

Temporal Decoder (TD) is also a lightweight transformer that is used to reconstruct the removed embedding feature $g_i$ in the third step of our MAPLE. Before feeding the latent representation $z_i$ into the TD modules, we first insert the shared and learnable mask token $M$ at the position of the original abandoned features and then add the new temporal positional embedding to the full set of sequences. Note that the shared mask without new temporal positional embedding cannot reconstruct the action information at different times.

Reconstruction Target. As shown in the final step of our MAPLE in Fig. 3 (b), the target of reconstruction is calculated with pseudo-label $P_i$ instead of the original feature $g_i$. We feed both the original feature $g_i$ and the reconstructed feature $r_i$ into the classification head to obtain their corresponding pseudo-label $P_i$ and ${\hat{P}}_i$. Note that the original pseudo-label $P_i$ is generated without backprop for maintaining the stability of the training process. Following this target, our unsupervised loss can be defined with the Kullback-Leibler divergence:

where $KL$ is the function of Kullback-Leibler divergence, $|D_u|$ is the size of the unlabeled dataset, $f(\cdot )$ is the model, $P_i$ is the pseudo-label generated from the original feature $g_i$, ${\hat{P}}_i$ is the reconstructed pseudo-label generated from the reconstructed feature $r_i$. $Algorithm$ 1 and $Algorithm$ 2 present the training and inference process of our MAPLE, respectively.

Compare to reconstructing the original feature, reconstructing the pseudo-label not only improves the performance of classification but also makes the training stage more stable. We compare these two strategies in detail in section 4.5.

Our experiments are performed on three main human action recognition datasets: MSR-Action3D (Li et al., 2010), NTU RGB+D 60 (Shahroudy et al., 2016), and NTU RGB+D 120 (Liu et al., 2020a).

MSR-Action3D (Li et al., 2010) dataset captured with Kinect v1 depth camera, which contains 567 videos and 23k frames in total (270 videos for training and 297 videos for testing). This dataset contains twenty actions: high arm wave, horizontal arm wave, and so on. For our semi-supervised point cloud action recognition, 7.5%, 15.0%, 22.5%, 30.0%, and 37.5% of training videos of each action are selected for the labeled dataset $D_l$ and the rest for the unlabeled dataset $D_u$. More detailed information is available in the supplementary material.

NTU RGB+D 60 (Shahroudy et al., 2016) is a large dataset that was captured with Kinect v2 depth camera. It consists of 56K videos and 4M frames captured from 80 views and with 40 performers. Sixty action categories and two types of evaluation (i.e. cross-subject and cross-view) are defined in this dataset. In this paper, we evaluate our model with a cross-subject setting. For our semi-supervised point cloud action recognition task, 5%, 10%, 20%, 30%, and 40% of training videos of each action are selected for the labeled dataset $D_l$.

NTU RGB+D 120 (Liu et al., 2020a) is an extension of NTU RGB+D 60 and the largest dataset for human action recognition. It consists of 114K videos and 8M frames captured from 155 views and with 106 performers. The dataset captured by Kinect v2 depth camera has the modalities of RGB, Depth, 3DJoints, and IR. One hundred and twenty action categories and two types of evaluation (i.e. cross-subject and cross-setup) are defined on this dataset. To harmonize with the above dataset, we still evaluate our model with a cross-subject setting and select the same percentage of the labeled dataset $D_l$ as NTU RGB+D 60.

This subsection presents the training hyperparameters and implementation details of our DestFormer and MAPLE.

Network Structure. The DestFormer $f(\cdot )$ are introduced in Section 3.2. By default, The spatial scaling rate $\kappa$ of P4Conv is set to 2 and the spatial scaling rate $\zeta$ is set to 32. The spatial transformer is designed with 4 self-attention blocks and the temporal encoder is designed with only 3 self-attention blocks. Each black spatial transformer and temporal encoder contains 8 heads. As the Temporal Decoder used in MAPLE, it consists of 8 self-attention blocks to strengthen its ability for reconstruction.

MAPLE Training. In the whole process of training, the basic learning rate $\eta$ is set to 0.01. The warm-up strategy is used for the first 10 epochs with the initial $\eta ={10}^{-6}$ and the decreased learning rate $\eta$ of the final 5 epochs is set to 0.001. The mini-batch of $D_l$ and $D_u$ is set to 14 for the MSR-Action3D dataset, and 32 for NTU RGB+D 60 and NTU RGB+D 120. The masking ratio is set to 75% for all datasets. In Step 1 (Pre-training) of our training, the DestFormer is trained on the labeled dataset of MSR-Action3D, NTU RGB+D 60 and NTU RGB+D 120 with the epoch of 40, 20, and 20, respectively. In Step 2 (Training of MAPLE), we set the positive scalar weight $\alpha$ as 0.5 for MSR-Action3D, and 0.2 for NTU RGB+D 60 and NTU RGB+D 120.

Compared Approaches. In the following section 4.4, we use leading semi-supervised learning algorithms that have proven to be generally effective as our compared approaches:

1) Supervised-only. We train the model only with labeled dataset $D_l$. The performance of the best-trained model is used as the lower bounds of semi-supervised learning.

2) Pseudo Labels (Lee and others, 2013). The main idea is to further train the model with the pseudo hard labels of unlabeled data. This algorithm can be summarized in two steps as follows. First, we get the pre-training model with the supervised-only method, then we predict the pseudo hard labels of unlabeled data. Finally, the model can be retrained with these hard labels.

3) Virtual Adversarial Training (VAT) (Miyato et al., 2019). It is inspired by adversarial learning and its regularization only needs unlabeled data. In the training process, it first adds small adversarial perturbation ${ϵ}_{vat}$ to the unlabeled data for changing the final prediction, and then forces the model $f_{\theta }(\cdot )$ against this type of perturbation with the following consistency loss:

4) Conditional Entropy Minimization (EntMin) (Grandvalet and Bengio, 2004). This approach encourages the model to output the confident pseudo labels $y$ for unlabeled input data. In other words, the predictions $y$ closed to the one-hot vector are encouraged. The conditional entropy minimization loss can be defined as:

Note that the EntMin is almost not used alone for semi-supervised learning because the model can easily increase the weights of the classification head to generate a confident prediction. It always adopt with the VAT loss, i.e. $L_u={\alpha }_{vat}{L}_{vat}+{\alpha }_{entmin}{L}_{entmin}$, where ${\alpha }_{vat}$ and ${\alpha }_{entmin}$ are the positive scalar weight for loss of VAT and EntMin, respectively.

To demonstrate the validity of our spatial-temporal backbone, this subsection compares the action recognition performance and computational efficiency of our DestFormer and the P4transformer model (Fan et al., 2021) with the supervised-only setting on MSR-Action3D, NTU RGB+D 60 and NTU RGB+D 120 datasets. The action recognition accuracy (%) of each backbone on three benchmark datasets is listed in Table 1. The time complexity (GFLOPs) and average inference time (ms) of each point cloud video are listed in Table 2.

In Table 1, we observe that our DestFormer has less annotation dependence and better classification performance in the supervised-only setting. Especially on the NTU RGB+D 120 dataset, our DestFormer model generally obtains a greater than 5.7% increase in action recognition accuracy.

In Table 2, we notice that our DestFormer is more efficient in computational complexity, which obtain about 30% and 23% decrease for time complexity (GLOPs) and inference time (ms), respectively.

In this section, we first evaluate our MAPLE method by comparing it with leading semi-supervised methods (e.g. Pseudo Label, VAT, and EntMin) for semi-supervised point cloud action recognition on three mainstream datasets. Then we further combined our MAPLE method with those methods (VAT+EntMin+MAPLE) and obtain better performance for semi-supervised point cloud action recognition. Specifically, we use $L_{vat}$ and $L_{entmin}$ as unsupervised loss functions $L_u$ in the early training stage until the model has almost stabilized and then adopt the $L_{maple}$ as the unsupervised loss function. Please refer to the supplementary materials for the detailed training process of “VAT+EntMin+MAPLE”.

The results on MSR-Action3D and NTU RGB+D 60 datasets are shown in Tables 3 and  4 respectively. The results on NTU RGB+D 120 are shown in the supplementary material. From the semi-supervised results of each semi-supervised method, we can find that our MAPLE method is effective for semi-supervised learning and slightly outperforms the previous methods’ performance under most settings. We also observe that our MAPLE method can be combined with other leading semi-supervised methods, and obtain significant performance increases under each setting. Especially on the 7.5% labeled MSR-Action3D setting, it brings significant improvement in action recognition performance (+8.08% Acc).

We investigate the effectiveness of our proposed MAPLE method on benchmark datasets in this section. We first analyze the influence of the masking ratio and the depth of the temporal decoder, then illustrate the importance of reconstruction with pseudo-label during the training process. At last, we show the feature distributions of each method to prove the effectiveness of our MAPLE method.

Masking ratio. Fig. 4 shows the accuracy of classification on NTU RGB+D 60 5% labeled dataset with different masking ratios. The high masking ratio (75%) of reconstruction achieves the peak of classification performance, which is as high as the masking ratio of Image MAE (He et al., 2021). This phenomenon is the exact opposite of the natural language processing field, whose best masking ratio is 15% typically. However, this behavior verifies our hypothesis that most frames of the action sequence are redundant and we can reconstruct the complete action sequence from the small residual part of the sequence.

Depth of Temporal Decoder. We investigate the effect of decoder depth on the final classification performance and the result on NTU RGB+D 60 5% labeled dataset with different depth of temporal decoder. For more detail, please refer to the supplementary materials.

Reconstruction with Pseudo-Label. To demonstrate the importance of implicit reconstruction through pseudo-label, we show the results with and without pseudo-label on the MSR-Action3D dataset in the supplementary materials.

The t-SNE visualization. To further explore the mechanism of MAPLE, we visualize the feature distributions of the labeled and unlabeled video sequences of the MSR-Action3D training dataset with t-SNE in Fig. 5. From the t-SNE visualization, we can find that the model trained with only supervised action sequences is often hard to distinguish the decision boundaries of the unlabeled action sequence. Although the benchmark methods (Pseudo Label, VAT, and EntMin) have better distributions, there are still some outliers that make the decision boundaries ambiguous. Compared to previous approaches, our MAPLE and “VAT+EntMin+MAPLE” form tighter data clusters and clear decision boundaries which benefit from the semi-supervision learning with our masked autoencoder. To summarize, the visualization shows that combining semi-supervised learning with masked pseudo-labeling autoencoder is possible to learn numerous action concepts from unlabeled point cloud videos and improves the performance of action recognition.