Methods for hand pose and shape reconstruction have many applications in human-computer interaction, augmented reality, virtual reality, robotics, and motion generation [41, 40, 9]. Recent research demonstrates impressive results on the task of supervised 3D hand reconstruction from monocular RGB images (\eg [31, 44, 16]). However, generalizing to in-the-wild settings, with fully unconstrained and uncontrollable environmental conditions, would require large amounts of training data captured under the same conditions. As of today, accurate 3D keypoint annotation of in-the-wild data is an open research problem and, therefore, no large-scale in-the-wild dataset with accurate 3D annotations exists. For these reasons, techniques that leverage sparsely annotated data [15] or weakly labelled data [26, 4, 24] have seen much interest. However, such methods rely on pseudo 2D or 3D annotations, which in turn require human effort for acquisition, or may introduce label noise that bounds model performance [4, 24]. Therefore, a promising solution to avoid pseudo-labels entirely, is to make use of unlabelled data, for example via contrastive learning [35, 46]. In the context of sequence data, we observe that existing methods often struggle with heavy occlusions, for instance brought on by hand-object interaction. Consider the example from Fig. 1: while the hand pose throughout the grasp is quasi-static, the images change drastically from frame to frame, which causes existing methods to output incorrect hand poses. In this paper, we explore how to learn better representations that capture human motion’s inherent temporal consistency, improving the hand reconstruction stability through time. We do so by leveraging single-view unlabelled videos of hands grasping objects to improve 3D hand reconstruction in the most challenging setting of heavy occlusions.

Unlike single images, videos contain temporal information that can help to predict coherent hand reconstructions through time by learning correlations between time-adjacent frames. Combining this idea with the recent progress of contrastive representation learning methods [32, 6, 17], we introduce a time-coherent contrastive learning pipeline, dubbed TempCLR. Our approach consists of two stages, as shown in Figure 2: a pre-training stage where we perform time-coherent contrastive learning on unlabelled videos and a second stage, where the pre-trained encoder is fine-tuned on the 3D hand reconstruction task using labelled data. In particular, TempCLR contributes two key ideas: 1) a time-coherent augmentation method to impose strong spatial augmentations on each frame of a video while maintaining temporal integrity; and 2) a probabilistic sampling strategy that accounts for the differences in frames along the temporal dimension. In contrast to a vanilla time-contrastive learning approach [46], which repels any non-neighboring frame in a sequence, our sampling strategy takes into consideration that temporally-closer frames often represent more similar hand poses in range of motion. Based on this insight, TempCLR gives more attention to attracting temporally close frames and only repels temporally distant frames. Figure 1 shows that our approach is able to produce smoother hand reconstructions along time, where a state-of-the-art approach [8] fails to do so.

We evaluate TempCLR in different settings and on different datasets. First, we demonstrate that our pre-training improves the performance over a fully-supervised baseline [8] by $15.9\%$ and $7.6\%$ in 3D mesh error on the HO-3D and FreiHAND datasets (\cfTab. 1 and Tab. 2). Next, we show that our single-view time-contrastive method improves over a vanilla time-contrastive approach [46] on FreiHAND. Through cross-dataset evaluation and in-the-wild qualitative results, we show improvements in generalization capabilities. Finally, we demonstrate that our method yields smoother hand reconstructions along the temporal dimension compared to other SotA approaches.

Our contributions can be summarized as follows:

1. [noitemsep]

2. A novel single-view time-contrastive learning approach for 3D hand reconstruction. Our method leverages time-coherent augmentations and a probabilistic sampling strategy to capture long-range dependencies.

3. We experimentally show that by leveraging in-the-wild unlabelled monocular videos, TempCLR outperforms existing methods across different metrics.

4. We provide empirical evidence that our method leads to smoother hand poses estimated over time.

To allow methods to generalize to unconstrained settings, recently, there has been attention on reducing the reliance on 3D annotation [1, 44, 4, 2, 15, 26, 35, 46, 38, 37]. For example, Hasson \etal[15] leverage sparsely annotated data by introducing a photometric loss formulation to learn from partially labelled sequences. Liu \etal[26] propose a specialized transformer-based architecture used to collect pseudo labels from in-the-wild videos. These pseudo labels are then used to train the same architecture. Zimmermann \etal[46] explore the benefits of multi-view and single-view time-contrastive learning applied on the hand reconstruction task. Spurr \etal[35] introduce an equivariant contrastive objective formulation where geometric transformations applied on the image are reversed in the latent space. In this paper, we introduce a self-supervised approach to leverage supervision on unlabelled monocular videos in the wild.

The most relevant methods to us are [15, 26, 35, 46], which leverage unlabelled or partially labelled data. Compared to [15, 26], our method requires neither human intervention for tuning pseudo-labels [26], nor sparsely annotated videos [15]. Similarly to ours, the methods in [35, 46] use a contrastive formulation. However, [35] relies on unlabelled in-the-wild still images while we rely on unlabelled in-the-wild videos. In addition, [46] leverage a multi-view time-contrastive formulation while our approach is based on monocular videos. Furthermore, to go beyond [46], we introduce a simple-yet-effective time-coherent augmentation method and sampling strategy that reflects the differences in frames along time. Experiments show that this novel combination is crucial for time-contrastive learning.

Figure 2 shows a schematic of our method, TempCLR, which consists of two stages: a pre-training stage, and a fine-tuning stage. In the pre-training stage, we leverage a time-contrastive objective to train the image encoder on unlabelled videos. This stage is to obtain additional supervision for the encoder from diverse in-the-wild videos of hand in motion. In the second stage, we train the whole hand reconstruction architecture through supervised fine-tuning. In Section 3.1, we describe our time-contrastive pre-training, motivating the importance of our data augmentation and probabilistic sampling technique. Then, in Sec. 3.2 we present our hand reconstruction model.

In Section 4.1, we first introduce experiment details such as the datasets, the evaluation metrics, and the implementation details. In Sec. 4.2, we compare our method to state-of-the-art approaches on both hand-grasping-objects and hand-only settings. In Sec. 4.3, we ablate TempCLR and provide qualitative results. Also, we show the effectiveness of TempCLR when 3D annotations are scarce. Finally, in Sec. 4.4, we perform cross-dataset evaluation to demonstrate generalization under domain shifts.

We introduce, TempCLR, a time-contrastive method for hand pose and shape estimation that yields stable 3D reconstructions through time. We introduce time-coherent augmentations and probabilistic pair sampling to better account for the temporal information provided by unlabelled videos. We thoroughly investigate our method, showing that it better captures temporal features and improves reconstruction stability through time. We demonstrate that our TempCLR achieves state-of-the-art results on the HO-3D and FreiHAND datasets. Finally, by means of cross-dataset evaluation, we show the potential of our method’s generalization capabilities.

Here we report the details describing the TempCLR training procedure, and we clarify how we adjust PeCLR  [35] to our setting. We use ResNet [18] as the backbone, which takes monocular RGB images of size $224\times 224$ as input. We employ Adam [22] optimizer for training.

To select positive frames for the contrastive training, the window size is set to $15$ for HO-3D and 100DOH. For HanCo  [46], because the frame rate is not available and the number of frames per sequence is much smaller compared to other datasets, we fix the positives’ window size to $5$. For TempCLR geometric agumentation, we augment the sampled images using rotation $r\in [-{45}^{\circ },{45}^{\circ }]$, scaling $s\in [0.6,2.0]$, and translation $t\in [-\text{im\_size}\times 0.3,\text{im\_size}$ $\times 0.3]$ in pixel. The randomness of these augmentations is fixed per sequence, such that the same transformation is applied to each frame. In addition, we apply random appearance transformation, independently to each frame, in terms of channel noise $n\in [0.6,1.4]$, sobel filter with a kernel size of 3, and color drop.

Here, we report additional experiments and analysis. Then we compare TempCLR with a temporal model.

Figure 7 demonstrates the temporal model. A sequence of frames $I_1,\cdots ,I_t$ is fed into a ResNet18 [18] encoder, which functions as a feature extractor and outputs a vector $f_i$ for each frame. These feature vectors $f\left(I_1\right),...,f\left(I_t\right)$ are sent to a Gated Recurrent Unit [7] (GRU) layer which yields a latent feature vector $g_i$ for each frame, $g\left(f\right(I_1\left)\right),...,g\left(f\right(I_t\left)\right)$, based on the previous frames. Then, each of these latent vectors are fed into $T$ regressors with iterative feedback as in [8]. The training loss function is a linear combination of 2D re-projection loss, 3D joint errors, and pose and shape parameter loss:

All the variables with a hat denote the network predictions, while all the variables without a hat denote the ground truth. Moreover, $J^{2\text{D}}\in R^{21\times 2}$ and $J^{3\text{D}}\in R^{21\times 3}$ represent the twenty one 2D and 3D keypoints, respectively.

We compare our time-contrastive approach with the fully-supervised ExPose [8] baseline and the temporal model, solely trained on HO-3D labelled sequences. Table 7 shows that the temporal model reaches better performances compared to the single-frame architecture. However, our time-contrastive approach, without employing any additional in-the-wild data during pre-training, improves the performance over the temporal architecture by $4.5\%$ in PA-V2V and PA-EPE.

In this section we compare the qualitative results from TempCLR and PeCLR  [35]. Next, we analyze the limitations of TempCLR. Finally, we provide additional qualitative results (Fig. 11-13) produced by our baseline model, PeCLR, and TempCLR.

In particular, the goal of PeCLR contrastive training is to attract the embeddings of different transformations of an image representing the same hand pose. However, artificial variations (\egrotation, scaling, translation) are not expressive enough to account for all the possible changes within consecutive frames. All of these small differences in the image space may actually lead to significantly different latent representations. For example, consider two consecutive frames of a sequence, where the hand is more occluded by an object in the second frame. Since there is no temporal component in PeCLR, these slightly different but similar hand poses are likely to be repelled during training. On the other hand, the TempCLR contrastive objective accounts for this small variations of the images by attracting the latent representation of frames closer in time. Hence, as shown in Fig. 4 of the main text, images with similar hand poses clustered together in the TempCLR latent space, leading to coherent hand reconstructions.

Further, TempCLR fails when there is a drastic variation in image scale across a video (see the bottom split of Fig. 10). This problem arises since our time-coherent augmentation strategy does not account for these unexpected geometric transformations happening over time. A solution could be to separate a sequence into sub-sequences by detecting sudden changes in images. For example, optical flow could be used to measure such changes. In this way, the attraction of latent representations of frames with different geometric variations would be avoided and the motion cues of the video preserved.