FusePose: IMU-Vision Sensor Fusion in Kinematic Space for Parametric Human Pose Estimation

Generally, IMU measures accelerations and angular velocities, then orientations can be solved leveraging filter algorithms [2, 8, 6, 40, 46]. The commercial solution of [50] aggregates 17 IMUs to estimate human pose in a Kalman Filter. The seminal work of [47] uses a custom made suit to target human motion un everyday surroundings. There are also some works combine depth vision data with IMUs [13, 56]. The approach of [31] tries to integrate IMUs with 2D posees detected in one or two views. Reconstructing human poses from only IMUs is an under-constrained problem, as the information IMUs provide is insufficient for accurate human pose estimation. Thus, combining IMUs with vision data such as videos and images is worth employing. IMU-vision sensor fusion can take advantage of the supplementary strengths of the two data sources, i.e. the global drift-free position from vision data and local occlusion-free limb orientation even under fast motion from IMU data. The early work on the combination of video and inertial sensors is implemented by Pons-Moll [38] via optimizing a local hybrid tracker. Marcard et al. [49] further develop this strategy under a setting of 8 viewpoint videos and 13 IMUs. Malleson et al. [30] integrate constraints from IMUs, multi-view images and a human pose prior model to optimize generated parametric human pose. These methods explore the raw data of IMUs via minimizing the rotation or the acceleration energy terms. However, they only utilize rough vision cues such as image silhouettes or estimated 2D human pose.

As the deep networks ensure the high quality of extracted image features, deep feature-based methods for 3D human pose estimation develop well in recent years. The 2D heatmaps can serve as appropriate intermediate cues for either multi-view features aggregation or IMU-vision fusion. Gilbert et al. [10] back-project multiple viewpoint features to probabilistic visual hull (PVH). The IMU orientations are processed to 3D joint locations by a forward kinematic solver and then embedded into 3D pose estimation via tensor concatenation. Huang et al. [16] propose to transform IMU-bone orientations to volume features, which are then concatenated with vision volume heatmaps. Zhang et al. [55] process the IMU orientations to bone vectors for either promoting the 2D joint heatmap refinement in terms of multi-view geometry or optimizing the 3D occupancy of joints via picture structure model (PSM). Although high accuracy the above-mentioned deep learning-based methods achieve, they do not leverage the raw orientation and acceleration data from IMUs. Instead, they utilize the transformed IMU information such as bone vectors for the convenience of fusing with deep vision features.

In this work, we fuse the IMU raw data and reliable intermediate features of multi-view vision data under a parametric human pose representation, ensuring both the efficiency of sensor fusion and the kinematic consistency of estimated 3D human skeletons.

3D human pose estimation is often formulated as learning from sensor data such as monocular or multi-view images to estimate the 3D keypoint coordinates of a human skeleton [24, 18, 32]. This location-based strategy recognizes each human joint as a point from heatmap and each bone as a line from a predefined human kinematic tree. However, there exist problems unsolved under this point-line human pose representation. First, directly regressing human joints from heatmaps which are generated by convolutional neural network may introduce bone stretching issue[15, 14] or left-right asymmetry[5]. Second, the twist rotation of the bone is ignored and can not be integrated into the whole framework, resulting in that the estimated 3D human pose may be inadequate to represent the true human motion[25]. Thus, a parametric human skeleton is paramount for more plausible human pose.

Parametric human pose[35, 11, 28, 48] represents the human body with skeletons with a certain number of keypoints and the parameters attached to them. The pose parameters are often based on a human model with local coordinates for joints or bones. This model-based methods have great merit for more than human skeleton estimation such as robotic control[4], motion retargeting[45], and human mesh recovery[20, 3, 7, 33]. The works on model-based parametric human pose estimation can be divided into two categories: optimization-based methods and direct regression methods.

Optimization-based methods mainly focus on searching for the optimal solution of human pose parameters[52]. The approach of [23] fit SMPL [29] to 2D detections via solving optimization problem. Energy functions are built to minimize the difference between generated human model and image features. The solution procedure often needs good initialization and a relatively long time for iteratively optimization, while mismatching the requirement of real scene application.

With the development of deep networks, increasing studies focus on directly regressing either location-based pose[18, 32, 42] or parametric pose[45, 41, 36, 20] in an end-to-end manner. Kanazawa et al. [20] propose Human Mesh Recovery (HMR) to reconstruct a mesh representation that is parameterized shape and 3D joint angles. Shi et al. [41] proposed an approach that learns to predict joint rotations directly from training data. Despite the great contribution made by [20] and [41], these methods rely heavily on an extra adversarial loss to learn real-or-fake prior information and ensure that the regressed results lie on the manifold of natural pose parameters.

From the rest skeleton $P^T$, we can initialize all the local systems $S$ by defining cartesian coordinates. A vector $b^g$ in global system can be transferred to $s_j$ by $b^j=R_j^g{b}^g$, where $R_j^g$ denotes the rotation from global system $s_g$ to local system $s_j$. Also, the relative rotation from the parent $pa\left(j\right)$ of $j$ to $j$ can be calculated by:

The purpose of FK is to reconstruct 3D joint locations from pose parameters along the kinematic chain. For example, all the joints in the path from joint $j$ to root joint are formulated as $\left\{p{a}^i\right(j){\}}_{i=1}^{n_j}$, then the bone vector in global system $b_j^g$ can be calculated by:

where the bone vector in local system $b_j^j$ and all the relative rotation are calculated from the rest T-pose, as described above. $n_j$ is simplified as $n$ here and later. The 3D location of $j$ therefore can be calculated by:

As illustrated in Figure 2 (a), the bone vector $b_j$ from $pa\left(j\right)$ to $j$ is not only affected by the rotation $R_j$ of its related joint $j$, but also by the rotations of all the upstream joints $\left\{p{a}^i\right(j){\}}_{i=1}^n$, matching the kinematics of rigid bodies and the real anatomic motion of human.

While the forward kinematics is well-defined and easy to solve, the inverse kinematics (IK) is an ill-posed problem because the 3D locations of joints fail to model the rotation information. In this work, we propose a canonical solution of IK based on axis-angle representation.

Same as the process of FK, Inverse Kinematics (IK) is performed along the kinematic chain. When calculating the rotation parameter $R_j$ of joint $j$, the rotation parameters of all his parents $\left\{p{a}^i\right(j){\}}_{i=1}^n$ are supposed be already solved. As the bone vector $b_j^g$ can be derived from the current 3D pose, the only unknown variable in Eq. (2) is $R_j$. Figure 2 (b) shows how we solve a canonical axis-angle representation of the local rotation from two 3D bone vectors. The axis $K$ and angle $\theta$ are then transferred to the rotation matrix by the Rodriguez formula:

Despite this solution is unable to represent the true rotation of the local system related to the bone vector, it provides a unique mapping from 3D joints location to pose parameters. A more detailed introduction of IK will be expanded in section III-C and Alg 1.

Since the IK and the FK are both differentiable, we implement them as two parameter-free layers in the deep neural network: IK Layer and FK Layer. Thus, they are both lightweight enough to be inserted into any 3D pose estimation framework as a useful module.

The whole framework of our approach is illustrated in Figure 3. We first process the information from whether multi-view images (MVIs) or IMUs to pose parameters based on the kinematic human model. Then, a total of three methods are proposed to fuse those pose parameters for bolstering the performance of 3D human pose estimation. In the rest of this section, we will elaborate on pose parameterization and the three parts of the whole frameworks, relatively.

After the $P^{vis}$ is obtained, pose parameters are generated by an IK layer. The root global transition $T_1$ is determined as the root location in $P_{vis}$. For better properties of computation consistency and stability, following [37], we convert the rotation matrix of local system $R_j^{vis}$ to quaterion $q_j^{vis}$ as pose representation parameters.

The pose parameters $Q^{vis}=\{q_j^{vis}{\}}_{j=2}^J$ have certain distributions based on the articulation range of human joints with known degree of freedoms (DoFs). The main reason we explore the pose parameters in this work instead of 3D joint locations is to more conveniently and efficiently aggregate the pose information from MVIs and IMUs. We will elaborate on the detail of the aggregation algorithms in section III-C.

The calculated bone global rotation $Q^{imu}=\{q_k^{imu}{\}}_{k=1}^K$ is still difficult to collaborate with the pose parameters generated from MVIs. Hence, it is also indispensable to align the IMU and vision information as the same representation for robust and efficient consolidation. There exist two available operations for IMU-vision alignment. The first one is directly applying the global rotations from IMUs to the bone vectors in T-pose, and generating the current frame bone vectors $B^{imu}=\{b_k^{imu}{\}}_{k=1}^K$ which are the same representation with the bone vectors derived from 3D joint locations. $K$ is the number of IMUs attached to human bones. In this work, $K$ is set as 8 following [55], employing IMUs in thighs, calves, upper arms and forearms. The bone vector rotation is performed by $b_k^{imu}={\overline{q}}_k^{imu}⊳b_j^j$, where $⊳$ denotes applying the rotation of a quaternion on a 3D vector and $j$ is the endpoint of the bone $k$ attached with IMU. The second operation for IMU-vision alignment is converting the global rotation $Q^{imu}$ to the local rotation ${\overline{Q}}^{imu}=\{{\overline{q}}_k^{imu}{\}}_{k=1}^K$ by:

where $q_j^g$ denotes the transformation from global $s_g$ to local $s_j$. Note that the local rotations are also part of the unknown variables in the IK process. Thus, ${\overline{Q}}^{imu}$ can be efficiently embedded into the IK layer and can collaborate well with pose parameters $Q^{vis}$ from vision data.

As illustrated in the middle of Figure 3, we propose three approaches to fusing information from vision and IMUs. Next, we will dive into the elaboration of these three approaches.

Obviously, this approach can only facilitate the results of joints related to IMUs by a decent margin. Assuming there exist large estimation errors on the upstream joints, NaiveFuse can hardly improve the results. Another drawback of NaiveFuse is that it is inadequate to just naively utilize the transformed 3D bone vector to reconstruct the real motion parameter of human bone which is a rigid body. For better leveraging the rigid bone rotation information from IMUs on a deeper level, we propose another fusion method in a kinematic manner.

The whole process of KineFuse is summarized in Alg 1. Given the predicted 3D pose $P^{vis}$ from vision, the calibrated and aligned IMU rotation information ${\overline{Q}}^{imu}$ and the rest T-pose $P^T$, the KineFuse is performed based on IK layer and output the fused pose parameters $T_1$ and $Q^{KF}$. In order to get to the final fused 3D pose $P^{KF}$, the fused pose parameters are fed into FK layer as described in section III-A.

The most important improvement in KineFuse is that it can simultaneously ensure the consistency of bone length and the accuracy of limb joints estimation. To be specific, the results of joints that are not related to IMUs can also be improved on the ground that the bone stretching errors are eliminated via IK and FK process. Furthermore, the estimation errors of joints that are related to IMUs can be explicitly reduced on the basis of more accurate ancestor joints, together with IMU information which entangles with twist rotation.

However, in KineFuse there exists an important issue that, it just serves as a post-processing approach in test-time, same as NaiveFuse. IMU information should also be utilized in the network training stage to help increase the robustness of the whole framework. Thus, in the last approach AdaDeepFuse, we explore efficient IMU-vision fusion by a deep learning-based supervised way.

Different from NaiveFuse and KinFuse, AdaDeepFuse are trained in an end-to-end manner with supervision in either the 3D human pose or pose parameters. We claim that the error from IMUs and 3D pose estimated from MVIs can be adaptively compensated via training a neural network. The AdaFuse module can determine what extent $Q^{vis}$ or ${\overline{Q}}^{imu}$ contributes to more accurate results. We fuse pose parameters instead of bone vectors since the quaternion space is continuous for interpolation and robust for network training. The total loss $L$ function for training AdaDeepFuse consists of two terms of loss function for 3D human pose $L_{pose}$ and loss function for pose parameters $L_{param}$ as :

where $\alpha$ is the weight of pose parameter loss and set as $1\times {10}^{-2}$ in this work. Both the two loss terms are implemented as computing the Smoothed L1 loss of results compared to ground truths. The ground truth 3D pose $P^{GT}$ is collected from MoCap equipment. The ground truth pose parameters $Q^{GT}$ are derived from $P^{GT}$ via IK layer.

We first compare the 3D human pose estimation performance of our framework to the state-of-the-arts on the Total Capture dataset. The results are listed in Table I. The last four rows show the methods implemented in this work, including the baseline AlgTri algorithm and three proposed IMU-vision sensor fusion approaches. Among all previous methods, first we can see that the LSTM-AE[43] and IMUPVH[10] achieve decent performance in the case of using temporal information. The error the state-of-the-art among the methods that only utilize vision data and single frame as input, i.e. Fusion-RPSM[39], is $29mm$ and larger than $25.9mm$ achieved by our trained baseline AlgTri. This is because the multiple viewpoints information is aggregated well using view confidences weighted triangulation algorithm. The state-of-the-art performance using only vision data is boosted by 14.3%.

Among the methods that perform IMU-vision sensor fusion, our approach NaiveFuse ($24.5mm$), KineFuse ($23.3mm$) and AdaDeepFuse ($22.5mm$) achieve state-of-the-art performances. There are three main points need to be noted. First, the NaiveFuse and KineFuse both utilize IMU information for only testing and serve as post-processing approaches. They can be conviniently applied to any existing 3D human pose estimation framework. They both outperforme the method of GeoFuse-SN-ORPSM[55] ($25.5mm$) which also only uses IMU for testing only. Second, the last proposed approach AdaDeepFuse uses IMU for both training and testing. It outperforms the method of GeoFuse-ORN-OPRSM[55] ($24.6mm$). We increase the performance by 8.5%. Third, the Recursive Pictorial Structure Model (RPSM) in [55] is proposed to perform 3D IMU-vision information integration based on enumeration algorithm, which is time-consuming and relies heavily on computing and memory resources. However, our approaches are computationally friendly since the proposed IK and FK layers are both lightweight and parameter-free. The AdaFuse module is also a shallow network. Thus, our framework has a wider spectrum of applications in the real scene.

As shown in Figure 4, the angle curve between vision and IMUs is undulate due to pose estimation errors of $P^{vis}$. An appropriate value of threshold ${\theta }^t$ is crucial for figuring out suitable hard cases and then integrating IMU information to improve the performance. We randomly select $30\%$ frames from Total Capture training set to perform an ablation study to determine the value of ${\theta }^t$. We list some possible values of ${\theta }^t$ and their respective estimation errors in Table II. We find that ${\theta }^t=0.25$ achieves the best performance in our experiment setting. Note that NaiveFuse and KineFuse are employed in this experiment.

We can draw several conclusions by observing the results. First, we can find that NF only brings error improvement for the joints that are related to the used IMUs, while having no impact on other joints ($0.0\%$ of blue bar in ’Others’). This is the limitation of directly replacing local bone vectors while ignoring the holistic human skeleton. As a contrast, KF and ADF both prompt better performance on these IMU-unrelated joints ($8.3\%$ of orange and green bars in ’Others’). This is mainly because IK and FK ensure the bone lengths consistency and avoid the bone stretch issue in the fused 3D pose. This conclusion can be also achieved by observing Figure 6, where we draw the output 3D poses of different methods and show them. The bone lengths in the 3D pose of AlgTri and NF disagree with that of ground truth, while this phenomenon does not occur in the results of KF and ADF. Based on the more accurate ancestor joints, the errors of IMU-related joints on limbs are further improved in KF, compared to NF. The whole average joint error improvement is also increased from $5.4\%$ to $10.0\%$. The value is further boosted to $13.1\%$ in ADF, indicating the efficiency of the adaptive deep learning-based fusion strategy.

In order to figure out which IMU plays more important roles in the fusion process, we conduct experiments on the usage of IMUs based on KineFuse. The results are listed in Table IV. It can be observed that the IMUs in upper limbs perform better than the IMUs in lower limbs. This is because the error improvement of the upstream joints will further correct the downstream children. In this work, using all IMUs in limbs serves as the final setting for the most accurate results.

We also conduct experiments on Human3.6M dataset for generalization evaluation. Due to the absolute accuracy of simulative IMU information, we just validate NaiveFuse and KineFuse on the testing set. The qualitative and quantitative results are shown in Table V and Figure 7. We can obtain similar conclusions with that from experiments on Total Capture. The IMU-related joint errors are improved from $30.1mm$ to $23.5mm$ by NaiveFuse, while to $19.2mm$ by KineFuse. In addition, results of other joints are also decreased by KineFuse from $20.4mm$ to $18.4mm$ while fixed in NaiveFuse. Also, in Figure 7, the bone length consistency is ensured in KineFuse while not in AlgTri and NaiveFuse.