Optimising 2D Pose Representation: Improve Accuracy, Stability and Generalisability Within Unsupervised 2D-3D Human Pose Estimation

Fully supervised approaches seek to learn mappings from paired 2D-3D data which contain ground truth 2D locations of key points and their corresponding 3D coordinates. Martinez et al. [Martinez_2017_ICCV] introduced a baseline fully connected regression model which learned 3D coordinates from their relative 2D locations. Exemplar approaches such as Chen et al. [Chen20173DHP] and Yang et al. [YANG2019371] use large dictionaries/databases of 3D poses with a nearest-neighbor search to determine an optimal 3D pose. Pavllo et al. [Pavllo2019CVPR_Facebook] used temporal convolutions over 2D key points to predict the pose of the central or end frame in a time series, whereas Mehta et al. [VNect_SIGGRAPH2017] utilized multi-task learning to combine a convolutional pose regressor with kinematic skeleton fitting for real-time 3D HPE.

Weakly-Supervised approaches do not use explicit 2D-3D correspondences and instead use either augmented 3D data during training or unpaired 2D-3D data to learn human body priors (shape or articulation). Pavlakos et al. [Pavlakos] and Ronchi et al. [ordinal_paper_2] proposed learning of 3D poses from 2D with ordinal depth relationships between key points (e.g. the right wrist is behind the right elbow). Wandt and Rosenhahn [Wandt2019RepNet] introduced a weakly-supervised adversarial approach where they transformed their predicted and ground truth 3D poses into a kinematic chain [Wandt2018AKC] before being seen by a Wasserstein critic network [improved_wasserstein]. Yang et al. [Yang20183DHP] lifted 2D poses from in the wild scenes where no ground truth data was available with a critic network that compared these predictions against ground-truth 3D skeletons. By contrast, Zhou et al. [Zhou_2017_ICCV] did not use a critic network but transfer learning that used mixed 2D and 3D labels in a unified network to predict 3D poses in scenarios where no 3D data is available. Drover et al. [Drover_2018_ECCV_Amazon] investigated if 3D poses can be learned through 2D self-consistency alone, where they would rotate a predicted 3D pose and reproject it back into 2D before passing it back to the model for comparison. This led to the discovery that a 2D critic network was also needed and self-consistency alone is not sufficient.

Unsupervised approaches do not utilize any 3D data during training, unpaired or otherwise. Kudo et al. [kudo2018unsupervised] introduced one of the first unsupervised adversarial networks which utilized random reprojections and a 2D critic network, under the assumption that any predicted 3D pose once rotated and reprojected should still produce a believable 2D pose. Chen et al. [amazon_paper_2] expanded this work and that of [Drover_2018_ECCV_Amazon] by introducing an unsupervised adversarial approach with a self-consistency cycle. They also provided ablation studies highlighting a 7% improvement when using temporal cues during training. Yu et al. [Yu_2021_ICCV] built upon [amazon_paper_2] highlighting that these temporal constraints may hinder a models performance due to balancing multiple training objectives simultaneously and proposed splitting the problem into both a lifting and scale estimation module. They also found that adding temporal motion consistency can boost the performance of their model by 6%.

Park et al. [Park_2018_BMVC] introduced one of the first studies that investigated if the human pose could be constructed of multiple interrelated structures. By utilizing relational networks, which were passed limb pairs during training ([left arm and right arm], [left arm and right leg], etc.), they aimed to discover shared feature information between limbs that would aid in the 2D-3D lifting process. Kyoungoh et al. [kyoungoh] utilized a propagating LSTM architecture where each LSTM would sequentially add another key point to its input. This intrinsic approach allowed joint relationships to be learnt0 over time as each key point was added. Jiang Hao [exemplar_upper_lower] worked on an exemplar approach where their 3D dictionary was split into torso and legs key points during lookup. Though their idea behind this was to speed up the nearest-neighbor search process, they may have improved the generalisability of their model as it could find similar legs in one training example and similar torsos within another, increasing the number of possible library combinations. Zeng et al. [Ailing_ECCV_2020_SRNET] investigated splitting a pose into different parts to learn localized correlations between key points first then global correlations between these localized segments afterward. Unlike our study however, [Ailing_ECCV_2020_SRNET] is a supervised network that still assumes that an entire 2D pose is one independent structure via feature sharing between localized groups throughout their network. Our study utilizes completely unsupervised networks, where two of the pose representations we investigate utilize no feature sharing, with the idea that localized key-point groups can be assumed to be independent of one another.

Our 2D poses consist of $N$ key points $(x_i,y_i)$, $i=\mathrm{1...}N$, with the root key point, the midpoint between the left and right hip joint, being located at the origin $(0,0)$. Because of the fundamental scale ambiguity associated with monocular images, it is impossible to infer absolute depth from a single view alone [depth]. Therefore, we used max-normalization on each of our 2D poses to scale their 2D key point coordinates between -1 and 1. In our study we investigate using the following five 2D pose representations and how they affect the unsupervised 2D-3D HPE process:

• Full 2D Pose Representation: The standard approach where a model is taught to predict the 3D ordinates from the entire 2D pose. The theory being that a good model should learn to correlate co-dependent key points and disassociate independent ones. In this scenario, we will have one network take an entire 2D pose as input and predict the 3D ordinates for the full pose.

• Split and Recombine Leg and Torso key points: A concept introduced by [Ailing_ECCV_2020_SRNET], the idea behind this representation is that we should first learn localized key point correlations and then the global context of a pose. In this scenario, two separate networks will first learn the localized correlations between the leg and torso key points and output a vector of features learned. A further network will learn to combine these features and predict the 3D ordinates for the full pose.

• Independent Leg and Torso key points: Although there is intuition behind first learning local and then global context. For many actions (e.g. waving, eating, etc) the legs and torso of a person may be more independent than co-dependent. To see if this is true another 2D pose representation will be that of completely independent torso and leg key points. In this scenario, we will have two independent lifting networks one that accepts the 2D key points and predicts the 3D ordinates for the torso and the other the 2D key points and 3D ordinates of the legs.

• Split and Recombine 5 Limbs key points: Although many human actions require both hands or legs to move in sync, in reality, humans can easily move each limb independently of one another. In this approach, 5 separate networks will learn the localized correlations of each limb (left arm, right arm, torso, right leg and left leg) key points and output a vector of features learned. A further network will learn to combine these features and predict the 3D ordinates for the full pose.

• Independent 5 Limbs key points: Similar to the Independent leg and torso representation, this approach will look at the possibility of treating the entire pose as 5 independent limbs. In this scenario, 5 separate networks will each take as input and predict the 3D ordinates for each limb independently (i.e. one network will take 3 key points as input belonging to the right arm and predict the 3D ordinates for the right arm).

The lifting networks ($G$) used are fully connected neural networks whose architecture was based on [Martinez_2017_ICCV] and can be seen in Figure LABEL:fig:residualblock in Appendix LABEL:architecture_figures. These predict one 3D ordinate relative to the root for each 2D key point provided:

where $(x,y)$ are the 2D coordinates provided to the network, $\hat{z}$ are the predicted 3D ordinate for each key point and $w$ are the weights of our model learned during training. Once our lifting networks had made their predictions they are concatenated with the original 2D key points to create our final predicted 3D pose $(x,y,\hat{z})$. A more detailed architecture explanation of each pose representation model can be seen in Appendix LABEL:architecture_figures.

Similar to prior work [amazon_paper_2, Drover_2018_ECCV_Amazon, Yu_2021_ICCV], we utilize a self-consistency cycle through random 3D rotations to reproject our predicted 3D poses to new synthetic 2D viewpoints. Let $Y\in R^{N\times 2}$ be our full 2D pose. Once a prediction $G\left(Y\right)$ is made and a full 3D pose $(x,y,\hat{z})$ obtained, a random rotation matrix $R$ will be created by uniformly sampling an azimuth angle between [$-\frac{8\pi }{9},\frac{8\pi }{9}$] and an elevation angle between [$\frac{-\pi }{18},\frac{\pi }{18}$]. The predicted 3D pose will be rotated by this matrix and reprojected back to 2D via projection $P$, obtaining a new synthetic viewpoint of the pose and the matrix $\tilde{Y}\in R^{N\times 2}$ where $\tilde{Y}=PR\left[\right(x,y,\hat{z}\left)\right]$. To enable our model to learn consistency, if we now pass $\tilde{Y}$ to the same lifting network, perform the inverse rotation $R^{-1}$ on the newly predicted 3D pose $(\tilde{x},\tilde{y},\tilde{\hat{z}})$ and reproject it back into 2D, we should obtain our original matrix of 2D key points $Y$. This cycle allows our lifting networks to learn self-consistency during training where they seek to minimize the following components in the loss function:

In the case of our independent leg and torso and independent 5 limb 2D pose approaches, each network will receive its own $L_{2D}$ loss based on the error between the key points that they predicted. As an example, part of the $L_{2D}$ loss for the right arm lifter within our independent 5 limbs representation would include the difference between the original 2D key point coordinate of the right wrist, and its 2D coordinate once $\tilde{\text{Y}}$ was inversely rotated and reprojected. This error would not be included in the $L_{2D}$ loss for the left arm lifter as it does not predict the 3D ordinate for this key point.

During our study, we found that increasing self-consistency was key to reducing the evaluation error (see Appendix LABEL:consistency_cycle). Therefore, we introduce new self-consistency constraints during training based on rotations around the $y$ axis at ${90}^{\circ }$ increments. Let $(x,y,\hat{z})$ be the predicted 3D pose from our model. If we assume a fixed camera position and rotate our pose ${90}^{\circ }$, then the depth component of our pose ($\hat{z})$ prior to rotation will now lie on the $x$ axis from our cameras viewpoint. A visual example of this can be seen in Figure 1.

As we have normalized the axis of our poses between -1 and 1, a ${90}^{\circ }$ clockwise rotation of the 3D pose $(x,y,\hat{z})$ will produce the pose $(\hat{z},y,-x)$. Therefore, providing $(\hat{z},y)$ as input to our generators should result in $-x$ as its predictions. This fact allows for the inclusion of three additional consistency constraints in the loss function of our generators which teach consistency at a ${90}^{\circ }$ clockwise rotation, a ${90}^{\circ }$ anticlockwise rotation and a ${180}^{\circ }$ rotation, which are as follows:

The left parts of these constraints are summed in the final loss function to produce $L_{{90}^{\circ }}$ and included in the optimization function of our models. Similar to $L_{2D}$ in the case of our independent 2D pose representation networks, each lifting network will have its own $L_{{90}^{\circ }}$ depending on the 2D key points that it predicted for. Although we could include three similar constraints for ${90}^{\circ }$ rotational increments around the $x$ axis, we found that these hinder the performance of the model. This is due to ${90}^{\circ }$ $x$ axis rotations producing a birds eye and ground-up view of a 2D pose, which contains little variation between 2D key points.

Although self-consistency is important, alone it is insufficient for generating realistic 3D skeletons [amazon_paper_2]. Therefore, we utilize a 2D discriminator $D$, that takes as input a 2D pose and outputs a probability value of that pose being plausible or unrealistic. The architecture of our discriminator is a fully connected neural network made of 3 residual blocks [Martinez_2017_ICCV] and ending with a linear prediction function. It learns to discriminate between the real 2D poses within our data $Y$, and our reprojected 2D pose $\tilde{Y}$. This provides feedback to the generators during training, enabling the learning of geometric priors such as joint angles and limb length ratios. Our discriminator $D$ utilized the least squares loss [LSGAN] as we found it performed better than the standard adversarial loss during our experiments, and defined as:

Unlike the consistency constraints, we do not provide a unique version of $L_{adv}$ to each lifting network in our independent 2D pose representation networks and instead provide the same loss (with a different weight) to each network. This is due to two reasons; Firstly, although our independent lifting networks are being trained separately, we still want them to produce a believable reprojected 2D pose together and having one discriminator see the entire pose will provide this feedback during training. Secondly, we found that trying to discriminate between segments of a 2D pose provided poor feedback during training. An example being that 2D legs and arms are normally represented as straight or bent lines, making it hard for a discriminator to tell plausible and implausible limbs apart.

As discussed, our lifting networks were trained adversarially with random reprojection and ${90}^{\circ }$ consistency constraints. The network parameters are then updated to optimize the total loss for each lifting model given by:

where, $w_1$, $w_2$ and $w_3$ are the relative weights for the adversarial loss component, reprojection consistency loss component and ${90}^{\circ }$ consistency loss component respectively. The numerical values of the weights for each network are defined in Appendix LABEL:architecture_figures. We trained our models completely unsupervised following [amazon_paper_2] using a batch size of 8192 and the Adam optimizer [Kingma2015AdamAM] with a learning rate of 0.0002. Our experiments use $N=16$ key points. During discriminator training we employ label flipping with a 10% chance.

Human3.6M (H36M) [h36m_pami] is one of the largest and most widely used 3D human pose datasets, containing 3.6 million 3D human poses. It consists of both video and motion capture (MoCap) data from 4 viewpoints of 5 female and 6 male subjects performing specific actions (e.g. talking on the phone, taking a photo, eating, etc.). There are two main evaluation protocols for the H36M dataset, which use subjects 1, 5, 6, 7 and 8 for training and subject 9 and 11 for evaluation. Both protocols report the Mean Per Joint Position Error (MPJPE), which is the Euclidean distance in millimeters between the predicted and ground truth 3D coordinates. We report the protocol-II performance of our model on all frames of the validation set which employs rigid alignment between the ground truth and predicted pose before evaluation. The results of our different 2D pose representation models can be seen in Table 1.

From our results, we can see that representing a 2D pose as an independent torso and legs improves performance the most, as this representation achieved a 20% lower average error than our full 2D pose network. Additionally, our split-combine leg and torso and split-combine 5 limbs representation also showed improved performance, decreasing the average error by 6% and 9% respectively. Our independent 5 limbs model did not improve results which may be due to this representations models using 3 or 4 key points to predict 3D ordinates which may not contain enough information to learn the task at hand. By looking at our leg and torso results we can see it additionally performed well against several fully supervised models. In fact, by using this representation we achieved the second highest performance in the Discussing, Posing, Smoking and Walking action. Although improving the Discussing, Posing and Smoking action seems intuitive, as we would assume the arms and legs of a person to be independent during these actions, it is surprising that we achieved such high results in walking as we would expect to see larger levels of co-dependence between the torso and legs during this movement.

MPI-INF-3DHP [mono-3dhp2017] is a markerless MoCap dataset containing the 3D human poses of 8 actors performing 8 different activities. We use this dataset to highlight the effect that 2D pose representations have on a model’s generalisability to unseen poses when trained on the H36M dataset. The evaluation metrics used are the percentage of correctly positioned key points (PCK3D) and area under the curve (AUC) as defined by [mono-3dhp2017]. As our predicted poses are normalized, we upscale their 3D coordinates by the original 2D normalizing factor before evaluation. Our results can be seen in Table 2.

Similar to our H36M results, all of our 2D representation approaches, aside from our independent 5 limbs representation, performed better on unseen poses when compared to a full 2D pose representation approach. Our independent leg and torso representation improved results the most with a 4% increase in PCK3D and 6% increase in AUC when compared to the full 2D pose network. Additionally, this representation achieved the second highest overall AUC metric, even compared to other supervised and weakly-supervised approaches. Our reasoning for improved generalisability stems from the individualistic nature of our independent leg and torso model. For example, let us say we have an unseen 2D pose we want to lift into 3D $\tilde{\text{Y}}$ which happens to contain identical 2D leg key points to pose ${\text{Y}}_A$ and identical 2D torso key points to pose ${\text{Y}}_B$ which were both in the training data. The leg network during training will have seen and learned to predict the 3D ordinates of the leg-keypoints within ${\text{Y}}_A$ without needing the torso key points. Similarly, the torso network saw the torso key points within ${\text{Y}}_B$ and learned to predict these 3D ordinates with no knowledge of the leg key points. Therefore this new example $\tilde{\text{Y}}$ is not a difficult pose to predict the 3D ordinates for as the model has seen both segments of the pose before just within different training examples. Our full 2D pose by contrast saw both the leg and torso key points of ${\text{Y}}_A$ and ${\text{Y}}_B$ during training and learned to correlate them together during the 2D-3D lifting process. Meaning that when we now provide it with $\tilde{\text{Y}}$ it struggles to predict accurate 3D ordinates as though it has seen parts of this pose before the entire pose as a whole is new. We believe this also explains why our independent model had higher generalisability performance than both our split-recombine approaches. Although features are first obtained from localized groups in these approaches, the combination of these localized groups’ features leads to a poorer generalizing model as it has learned to correlate sub-structures together during prediction.