Skeleton Prototype Contrastive Learning with Multi-Level Graph Relation Modeling for Unsupervised Person Re-Identification

Early skeleton-based works extract hand-crafted descriptors in terms of certain geometric, morphological or anthropometric attributes of human body. Barbosa et al. [barbosa2012re] compute 7 Euclidean distances between the floor plane and joint or joint pairs to construct a distance matrix, which is learned by a quasi-exhaustive strategy to extract discriminative features for person re-ID. Munaro et al. [munaro2014one] and Pala et al. [pala2019enhanced] further extend them to 13 ($D_{13}$) and 16 skeleton descriptors ($D_{16}$) respectively, and leverage support vector machine (SVM), $k$-nearest neighbor (KNN) or Adaboost classifiers for person re-ID. Since such solutions using 3D skeletons alone are hard to achieve satisfactory performance, they usually combine other modalities such as 3D point clouds [munaro20143d] and 3D face descriptors [pala2019enhanced] to improve person re-ID accuracy.

Most recently, a few works exploit deep learning paradigms to learn gait representations from skeleton data for person re-ID in a supervised or self-supervised manner. Liao et al. [liao2020model] propose PoseGait, which feeds 81 hand-crafted pose features of 3D skeletons into CNN for human recognition. Rao et al. [rao2020self] devise a self-supervised attention-based gait encoding (AGE) model with multi-layer LSTM to encode gait features from unlabeled skeleton sequences, and then fine-tune the learned features with the supervision of labels for person re-ID. In [rao2021self], they further propose a locality-awareness approach (SGELA) that combines various pretext tasks ($e.g.,$ reverse sequential reconstruction) and contrastive learning scheme to enhance self-supervised gait representation learning for the person re-ID task. The latest self-supervised work is SM-SGE [rao2021sm], which utilizes a skeleton graph based reconstruction and inference mechanism to encode discriminative skeleton structure and motion features for the person re-ID task.

Our work differs in following aspects. We for the first time propose an unsupervised learning paradigm to learn discriminative skeleton representations from unlabeled skeleton data to directly perform person re-ID. Our approach requires neither any form of feature engineering ($e.g.,$ hand-crafted skeleton descriptors [barbosa2012re, munaro2014one, pala2019enhanced]) nor any skeleton label for training model [liao2020model, rao2021multi] or fine-tuning representations [rao2020self, rao2021self, rao2021sm]. Unlike most existing works that directly represent skeletons as sequences of body joints [rao2020self, rao2021self] or pose descriptors [liao2020model], this work not only explores more comprehensive body-structure modeling by devising unified and generalized multi-level skeleton graphs, but also systematically mines valuable body-component relations at various levels. Besides, the proposed approach needs neither self-supervised pretext tasks ($e.g.,$ reconstruction [rao2021sm]) for pre-training or supervised classification networks for discriminative representation learning. Instead, it exploits the unsupervised skeleton prototype contrastive learning to mine representative skeleton features and class-related semantics from unlabeled 3D skeleton data. To the best of our knowledge, this is also the first attempt to leverage skeleton graph based relation modeling and contrastive learning to realize both skeleton representation learning and unsupervised person re-ID.

To capture latent body structural information and learn an effective representation for each body-component node in skeleton graphs, we propose to focus on features of structurally-connected neighbor nodes, which enjoy higher correlations ( referred as structural relations) than distant pairs. For instance, adjacent nodes usually have closer spatial positions and similar motion tendency. Therefore, we devise a multi-head structural relation layer (MSRL) to learn relations of neighbor nodes and aggregate the most correlative spatial features to represent each body-component node.

We first devise a basic structural relation head based on the graph attention mechanism [velickovic2018graph], which can focus on more correlative neighbor nodes by assigning larger attention weights, to capture the internal relation $e_{i,j}^l$ between adjacent nodes $i$ and $j$ in the same graph as:

where $W_v^l\in R^{D\times D_h}$ denotes the weight matrix to map the $l^{th}$ level node features $v_i^l\in R^D$ into a higher level feature space $R^{D_h}$, $W_r^l\in R^{2D_h}$ is a learnable weight matrix to perform relation learning in the $l^{th}$ level graph, $\parallel$ indicates concatenating features of two nodes, and LeakyReLU$(\cdot )$ is a non-linear activation function. Then, to learn flexible structural relations to focus on more correlative nodes, we normalize relations using the $\mathrm{softmax}$ function as follows:

where $N_i$ denotes directly-connected neighbor nodes (including $i$) of node $i$ in graph. We use structural relations $A_{i,j}^l$ to aggregate features of most relevant nodes to represent node $i$:

where $\sigma (\cdot )$ is a non-linear function and ${\overline{v}}_i^l\in R^{D_h}$ is feature representation of node $i$ computed by a structural relation head.

To sufficiently capture potential structural relations ($e.g.,$ position similarity and movement correlations) between each node and its neighbor nodes, we employ multiple structural relation heads, each of which independently executes the same computation of Eq. 3 to learn a potentially different structural relation , as shown in Fig. 4. We averagely aggregate features learned by $m$ different structural relation heads as the representation of node $i$ as follows:

where ${\hat{v}}_i^l\in R^{D_h}$ denotes the multi-head feature representation of node $i$ in $G^l$, $m$ is the number of structural relation heads, $(A_{i,j}^l{)}^s\in R$ represents the structural relation between node $i$ and $j$ computed by the $s^{th}$ structural relation head, and $(W_v^l{)}^s$ denotes the corresponding weight matrix to perform feature mapping in the $s^{th}$ head. Here we use average rather than concatenation operation to reduce feature dimension and allow for more structural relation heads. MSRL enables our model to capture the relations of correlative neighbor nodes ( see Eq. 1 and 2) and integrates key spatial features into node representations of each graph ( see Eq. 3 and 4). However, it only considers the local relations of the same-level components in graphs and is insufficient to capture global collaboration between different level body components, which motivates us to propose the full-level collaborative relation layer.

Motivated by the natural property of human walking, $i.e.$, gait, which could be represented by the dynamic cooperation among body joints or between different body components [murray1964walking], we expect our model to infer the degree of collaboration (referred as collaborative relations) among body-component nodes in multi-level graphs, so as to capture more unique and recognizable walking patterns from the motion of skeletons. For this purpose, we propose a full-level collaborative relation layer (FCRL) to capture relations between a node and all motion-related nodes of the same level and that between a node and its spatially corresponding higher level body component or other potential components. As shown in Fig. 2 and Fig. 4, we compute collaborative relation matrix ${\hat{A}}^{a,b}\in R^{n_a\times n_b}$ ($a,b\in \{1,2,3\},a\le b$) between the $a^{th}$ level nodes $V^a$ and the $b^{th}$ level nodes $V^b$ as following:

where ${\hat{A}}_{i,j}^{a,b}$ is the collaborative relation between node $i$ in $G^a$ and node $j$ in $G^b$. Here we use the inner product of multi-head node feature representations (see Eq. 4) that retain key spatial information of nodes to measure the degree of collaboration. Instead of merely considering body relations between adjacent level graphs [rao2021multi], FCRL can capture the global collaborative relations among both adjacent and non-adjacent graphs, and meanwhile provides full collaboration inference between a node and all potential motion-correlated nodes in the same graph.

To enhance structural semantics of multiple graphs ($e.g.,$ global graph patterns) and adaptively integrate key correlative features in component collaboration, we exploit collaborative relations to fuse body-component node features across different spatial levels. We update the node representation (${\hat{v}}_i^a$) of $a^{th}$ level graph by fusing collaborative node features (${\hat{v}}_j^b$) learned from different graphs:

where $W_C^{a,b}\in R^{D_h\times D_h}$ is a learnable weight matrix to integrate features of collaborative node ${\hat{v}}_j^b$ of $b^{th}$ level into $a^{th}$ level node ${\hat{v}}_i^a$. $n_b$ denotes the number of nodes in the $b^{th}$ level graph, and ${\lambda }_C^{a,b}$ represents the fusion coefficient between $a^{th}$ level and $b^{th}$ level graphs, which can be adjusted according to their inherent correlations ($e.g.,$ level similarity). We denote the fused $l^{th}$ level graph features of the $i^{th}$ skeleton as $F_i^l\in R^{n_l\times D_h}$ by concatenating all node representations. Inspired by [rao2021sm], we retain graph representations of each individual level and adopt their concatenation to represent a skeleton as follows:

where $M_i\in R^{(n_1+n_2+n_3)\times D_h}$ is the multi-level graph representation of the $i^{th}$ skeleton $S_i$, and $[;]$ indicates the concatenation of graph features. By combining all graph-level representations that integrate structural and collaborative body relation features (see Eq. 1-Eq. 6), we encourage the model to capture richer features of body structure and skeleton patterns at various levels. We show the effectiveness of the proposed structural-collaborative body relation learning by the relation visualization in Sec. 5.3 and Appendix.

Given multi-level graph representations ($M_1,\cdots ,M_f$) of an input skeleton sequence ($S_{1:f}=(S_1,\cdots ,S_f)$), we first integrate graph features into a sequence-level skeleton graph representation:

where $\overline{M}$ is the multi-level graph representation of skeleton sequence $S_{1:f}$, which incorporates structural-collaborative features and temporal dynamics of $f$ consecutive multi-level skeleton graphs, and $w_i$ denotes the importance of $i^{th}$ skeleton graph representation. Here we assume that each skeleton equally contributes to representing graph features of a sequence, $i.e.$, $w_i=1$. For clarity, we use $\overline{M}=\{{\overline{M}}_i{\}}_{i=1}^{N_1}$ to represent multi-level graph representations of skeleton sequences in the training set ${\Phi }_t$, which are exploited as skeleton instances in the proposed SPC scheme.

Then, to gather skeleton instances $\overline{M}$ that contain similar features to find the representative skeleton prototypes, we leverage the DBSCAN algorithm [ester1996density], which can discover clusters with arbitrary shapes or semantics, to perform clustering as:

where $\overline{M}={\overline{M}}^1\cup {\overline{M}}^2\cup \cdots \cup {\overline{M}}^z\cup {\overline{M}}^o$, $z$ is the number of clusters ($i.e.,$ pseudo classes), ${\overline{M}}^k=\{{\overline{M}}_i^k{\}}_{i=1}^{x_k}$, $k\in \{1,\cdots ,z\}$, is the cluster that contains $x_k$ instances belonging to the $k^{th}$ pseudo class, and ${\overline{M}}^o=\{{\overline{M}}_i^o{\}}_{i=1}^{x_o}$ denotes the set of outlier instances that do not belong to any cluster. We compute the centroid of each cluster, which averagely aggregates features of skeleton instances in the cluster, to obtain corresponding skeleton prototype:

where ${\overline{M}}_i^k\in R^{(n_1+n_2+n_3)\times D_h}$ is the $i^{th}$ skeleton instance in the $k^{th}$ cluster, and $P^k$ denotes the $k^{th}$ skeleton prototype.

To focus on the typical and discriminative features of skeleton prototypes as well as facilitate learning high-level skeleton semantics from different prototypes, we propose to enhance the inherent similarity of a skeleton instance to corresponding skeleton prototype and maximize its dissimilarity to other skeleton prototypes with a skeleton prototype contrastive loss as:

where $N$ represents the number of all training instances, $z$ denotes the number of skeleton prototypes, $x_k$ is the number of skeleton instances belonging to the $k^{th}$ prototype $P^k$, and $\tau$ represents the temperature for contrastive learning, where higher value of $\tau$ produces a softer probability distribution over prototypes and retains more similar information among clusters [hinton2015distilling]. The skeleton prototype contrastive loss $L_{\text{SPC}}$ can be viewed as a special form of InfoNCE loss [oord2018representation], which aims to maximize the dot product based similarity between the query for each skeleton instance and its positive key for the same prototype while maximizing the dissimilarity to negative keys for other prototypes.

The objective of skeleton prototype contrastive learning is to maximize the intra-prototype similarity and inter-prototype dissimilarity to mine discriminative skeleton features and identity-related semantics in an unsupervised manner. Such learning process can be interpreted as to simultaneously enhance the tightness of skeleton instances belonging to the same prototype and the looseness among different prototypes. Considering that a better representation space should have smaller intra-class distance (higher tightness) and larger inter-class distance (higher looseness), we can measure the tightness and looseness levels of the learned representations with regard to ground-truth classes to verify whether the model has learned effective features for different classes from the unlabeled data. To this end, we propose two metrics named mean intrA-Class Tightness (mACT) and mean inteR-Class Looseness (mRCL) to quantitatively evaluate the merits of the learn skeleton representations.

algocf[htbp]     \end@dblfloat

Mean Intra-Class Tightness (mACT): To obatin mACT, we first compute intra-class tightness (ACT) of the $k^{th}$ class with

where $D_{k\text{-intra-class}}$ represents the average intra-class distance between the representation, $v_j^k$, of $j^{th}$ sample and the representation centroid, $c_k$, belonging to the $k^{th}$ class, $D_{\text{global-average}}$ denotes the global average distance between each representation and all centroids, $s_i$ represents the number of instances in the $i^{th}$ class, $C$ is the number of ground-truth classes, and $D(v_j^i,c_z)=1-\frac{v_j^i\cdot c_z}{\parallel v_j^i{\parallel }_2\parallel c_z{\parallel }_2}$ is the cosine distance between $v_j^i$ and $c_z$. Note that $1/D_{k\text{-intra-class}}$ can be used to represent the $k$-class tightness, but this result does NOT consider the factor of global feature distribution that leads to different representation distances. Therefore, we use $D_{\text{global-average}}/D_{k\text{-intra-class}}$ to indicate the degree of intra-class tightness. When $D_{k\text{-intra-class}}$ is smaller than $D_{\text{global-average}}$, the representations in $k^{th}$ class are denser than the average, thus indicating a higher ACT of $k^{th}$ class. Then, we average the ACT values of all classes to represent the mean intra-class tightness (mACT) of representations by $\text{mACT}=\frac{1}{C}{\sum }_{k=1}^C\text{ACT}\left(k\right)$.

Mean Inter-Class Looseness (mRCL): To measure the mRCL of representations, we compute the ratio of the average distance between different class centroids, $D_{\text{centroid-average}}$, to the global average distance, $D_{\text{global-average}}$ (see Eq. 12), with

where higher mRCL can be obtained with greater average distance $D_{\text{centroid-average}}$ between different representation centroids compared to the global average representation distance $D_{\text{global-average}}$. Full details of mACT and mRCL are provided in the Appendix. We conduct both quantitative and qualitative analysis of the proposed skeleton prototype contrastive learning scheme using the mACT and mRCL metrics, which shows that it can encourage the model to capture class-related semantics for more effective skeleton representations, as detailed in Sec. 5.4 and Appendix.

As presented in Table II and Table III, the proposed SPC-MGR enjoys distinct advantages over existing self-supervised and unsupervised methods on all datasets. Compared with AGE model [rao2020self] that learns skeleton features based on body-joint sequence representations, our approach consistently achieves higher person re-ID performance by a large margin of $7.2$ to $37.9\%$ top-1 accuracy and $6.0$ to $12.8\%$ mAP on different datasets, which demonstrates that the proposed multi-level skeleton graph representations with structural-collaborative body relation learning are more effective on modeling discriminative skeleton features for the person re-ID task. Our approach significantly outperforms the state-of-the-art skeleton contrastive learning method SGELA [rao2021self] by up to $25.2\%$ top-1 accuracy and $11.0\%$ mAP on KS20, KGBD, IAS-A, and IAS-B testing sets. On two testing sets of BIWI, both SGELA and the proposed approach obtain comparable mAP, while our SPC-MGR can achieve superior overall performance with higher top-1 ($7.2$-$8.3\%$), top-5 ($5.5$-$17.5\%$), and top-10 accuracy ($5.4$-$25.8\%$). Finally, our approach also performs better than the latest graph-based skeleton representation learning method SM-SGE [rao2021sm] by a distinct margin of $2.6$-$13.1\%$ top-1 accuracy and $2.5$-$12.2\%$ mAP on all datasets. In contrast to direct inter-sequence contrastive learning [rao2021self] or manually devising pretext tasks for skeleton representation learning [rao2020self, rao2021sm], our approach can automatically mine most representative skeleton features by contrasting sequence-level representations (instances) and cluster-level representations (prototypes), which enables our model to learn better skeleton representations for person re-ID. Moreover, our model requires only 0.01M parameters and evidently lower computational complexity for skeleton representation learning compared with existing self-supervised and unsupervised methods, as shown in Table II, which demonstrates its superior efficiency for person re-ID tasks.

We also compare the performance of our approach with state-of-the-art skeleton-based counterparts with the cross-view evaluation (CVE) setup of KS20. As shown in Table IV, our approach remarkably outperforms the latest self-supervised and multi-view skeleton-based methods SGELA [rao2021self] and SM-SGE [rao2021sm] by an average margin of $6.5$-$32.1\%$ top-1 accuracy, $12.8$-$41.0\%$ top-5 accuracy, $17.5$-$40.0\%$ top-10 accuracy, and $5.2$-$22.8\%$ mAP on 24 out of 25 testing combinations of probe views and gallery views, which demonstrates that our model can learn more discriminative skeleton representations with better robustness against viewpoint variations for cross-view person re-ID.

Compared with $D_{13}$ [munaro2014one] and $D_{16}$ [pala2019enhanced] that extract hand-crafted geometric and anthropometric skeleton descriptors, our model achieves a significant improvement of person re-ID performance by $1.5$-$23.8\%$ top-1 accuracy on KGBD, BIWI-S, and BIWI-W. Despite gaining similar performance on IAS-A and IAS-B, these methods are inferior to our approach by at least $7.3\%$ top-1 accuracy on more challenging datasets such as KS20 and KGBD that contains more viewpoints and individuals. Furthermore, with unlabeled 3D skeletons as the only input, the proposed approach can obtain comparable or even superior performance to two state-of-the-art supervised methods PoseGait [liao2020model] and MG-SCR [rao2021multi] on five out of six testing sets (KS20, IAS-A, IAS-B, BIWI-S, BIWI-W). Interestingly, with massive labels as the supervision, these methods still fail to obtain satisfactory person re-ID accuracy and even perform worse than hand-crafted methods on datasets with frequent view, shape, and appearance changes in KS20, IAS, and BIWI with the probe-gallery person re-ID setup. Considering that our approach does not require any manual annotation and can achieve highly competitive and more balanced performance with a significantly smaller size of network parameters, as shown in Table II, it can be a more general solution to skeleton-based person re-ID and related tasks. We will show broader applications of our approach to more general person re-ID scenarios in Sec. 5.5.

To analyze the effects of different graph levels, Table VI evaluates the performance of our approach when combining different graphs for person re-ID under the same optimal model setting. Exploiting low-level graph representations can achieve evidently better person re-ID performance than using high-level graphs by $1.3$-$19.0\%$ top-1 accuracy on most datasets, which suggests that the low-level graphs ($e.g.,$ part-level) that contain more body-component nodes and body structural/relational information can benefit learning more discriminative skeleton representations for person re-ID. Combining different levels of graphs can improve the person re-ID performance compared with solely using single-level graphs (first three rows in Table VI) in most cases, while the model exploiting all level graphs is the best performer among different datasets. This further justifies that the proposed multi-level graphs that model body structure and motion at various levels enable our approach to capture more recognizable structural features and pattern information for the person re-ID task.

As shown in Table VII, we show the effects of different amounts of structural relation heads on our approach. It is observed that introducing more learnable structural relation heads with larger $m$ values can improve the performance of our approach on different datasets, as it can encourage capturing more spatially or motion related features of adjacent body components. However, too many structural relation heads , $m=16$, for example, may cause the model to learn redundant relation information and degrade model performance on relatively limited training data such as KS20.

The hyper-parameter ${\lambda }_C^{a,b}$ controls the extent of fusing collaborative node features between $a^{th}$ level and $b^{th}$ level graphs. Here we use a unified ${\lambda }_C$ to equally fuse features between different graph levels, so as to better analyze the overall effects of collaborative fusion. As reported in Table VIII, improving ${\lambda }_C$ enables our model to achieve better Re-ID performance in terms of both top-1 accuracy and mAP on all datasets. Such results verify the necessity of adequate collaborative graph fusion to learn more effective multi-level skeleton graph representations for person re-ID. Based on this observation, we set ${\lambda }_C^{a,b}=1$ to fuse skeleton graph features.