Our method is related to the image-to-image translation task that converts one source image to a target image conditional on a specific style, resolution, contents, etc (Isola et al., 2017). Existing image-to-image translation methods usually adopt the Conditional Generative Adversarial Networks (Mirza and Osindero, 2014; Li et al., 2019) to take the specific image (e.g., semantic segmentation map, sketch, edge map, etc.) as input for image generation. Such models can generate the desired image by editing the properties of the input image. Therefore, Compared with unconstrained GANs, image-to-image translation has a wider range of applications. For example, Pix2pix (Isola et al., 2017) used a conditional discriminator to make the generated image and the input image match as much as possible, so it can generate realistic corresponding images. Based on pix2pix, pix2pixHD (Wang et al., 2018a) generated high-resolution images by using multi-scale generators and discriminators. Vid2vid (Wang et al., 2018b) was aimed at the temporal inconsistency of the generated videos. By adding optical flow constrained to the generator and discriminator, vid2vid can generate temporally coherent videos.

The personalized motion transfer is to translate the pose map to the corresponding RGB image, where the person’s appearance is implicitly embedded into the generator. Therefore, many image-to-image translation works can be used for personalized motion transfer. However, such methods cannot be directly applied to HVMT for the generation of varied poses and precise details of persons.

Personalized HVMT. On the basis of the image-to-image translation methods, some works have specific improvements based on the characteristics of this task. In order to conform to the size of the source person, EDN (Chan et al., 2019) matched the poses between the driving person and the source person by designing global pose normalization. In addition, EDN designed a face GAN to generate a more precise face region. DIW (Wang et al., 2021) encoded non-uniformly sampled pose maps of the past ten frames to capture appearance change details and refine pose features of the single frame. As mentioned earlier, personalized methods suffer from the inability to generalize to unseen person generations and need massive computing resources.

General-purpose HVMT. To overcome the limitations of personalized HVMT, recent studies focus on general-purpose HVMT. In order to better preserve appearance details, some works adopt the warp operation. These methods focusing on the animation of general objects, such as FOMM (Siarohin et al., 2019) and MRAA (Siarohin et al., 2021), utilized predicted keypoints or regions to compute the flow field between the source image and the driving image, and then warped the feature of the source image to obtain the target image. However, these methods do not take temporal consistency into consideration, which results in the temporal inconsistency of the synthesized video. LWGAN (Liu et al., 2019b) warped the source information in both image and feature spaces based on the SMPL model. However, the SMPL model (Loper et al., 2015) was only suitable for smooth human bodies, it cannot represent the human body with complex clothes. Different from warping the feature of the source image, C2F (Wei et al., 2021) estimated the optical flow of the clothing regions and directly warped the clothing region according to optical flow. Unfortunately, when the driving pose is greatly different from the source pose, such methods usually failed due to inaccurate flow estimations. Different from the above methods, our proposed method does not use warp operation. This design can avoid complex transformation calculations and can better cope with arbitrary poses.

In addition, some methods consider the diversity of different regions of the human body and treat each region of the human body separately. SSF (Gafni et al., 2021) encoded each region of the body separately, then integrated features of different regions to directly generate the final target image. SHUP (Balakrishnan et al., 2018) first transformed each part of the source person to the spatial position of the driving pose, then directly generated the whole person image. Moreover, PGTM (Huang et al., 2021) predicted the UV map (Güler et al., 2018) of the source person conditioned on driving pose, then filled it with each region textures of the source person image. However, predicting dense UVs in different poses accurately is challenging. StylePeople (Grigorev et al., 2021) used parameterized human body model SMPL-X (Pavlakos et al., 2019) to model the pose and shape of subjects, then used neural rendering to render region appearance details. Our method also encodes each region of the source image separately. However, unlike these methods, we directly generate corresponding target region images, then use the whole composition network to integrate the generated regions for generating the final target image. Compared with directly generating the whole person image, generating the region image first can better preserve the appearance details.

As mentioned before, the reason why we first generate each source person region in the driving pose is that different regions of the human body have significantly different texture patterns. It’s difficult to directly generate the whole human body. To achieve this, we need to first generate the layout of the source people in the driving pose through Layout GAN, as shown in Figure 2 (a).

Layout GAN. In order to represent the human pose, we first use the human keypoint detection method (Cao et al., 2021; Liu et al., 2021a, 2022b, 2022a) to predict the 2-D keypoint $K\in R^{2\times 25}$ of the human body. Then according to the predefined connection strategy, we can get the pose connection map $P_d^t\in R^{3\times H\times W}$ of the driving person, where $H\times W$ is the resolution of the image. In order to represent the human body layout, we use the human parsing methods (Li et al., 2020; Liu et al., 2019c; Zeng et al., 2021) to obtain the 18-channel semantic segmentation maps of the person image. Further, according to the similarity of textures in various regions, we merge the semantic segmentation maps into six classes, i.e., head, top, bottom, shoes, limbs, and background. So we can get the body layout $L_s\in R^{6\times H\times W}$ of the source person image and $L_d^t$ of the driving person image.

For the network structure, we also use vid2vid (Wang et al., 2018b) as in (Wei et al., 2021). Specifically, we use three encoders $E_1^l$, $E_2^l$ and $E_3^l$ to encode the source body layout $L_s$, the concatenated driving poses connection map $\{P_d^{t-2},P_d^{t-1},P_d^t\}$ and the concatenated results previously generated $\{{\hat{L}}_d^{t-2},{\hat{L}}_d^{t-1}\}$, to obtain the feature maps $F_1^l$, $F_2^l$ and $F_3^l$ respectively. Then decoder $D_1^l$ is proposed to decode the added feature maps $F_1^l$, $F_2^l$ and $F_3^l$ to get raw result ${\hat{L}}_{d,raw}^t$. Similarly, $D_2^l$ decodes added features $F_2^l$ and $F_3^l$ to obtain optical flow $O$ and its weight $w$, and finally obtains the final result ${\hat{L}}_d^t$. The entire Layout GAN can be formulated as

where $+,\ast$ represent element-wise addition and element-wise multiplication, respectively. $\left\{\right\}$ indicates that the inputs are concatenated along the channel dimension. $\mathrm{Warp}(I,o)$ refers to the affine transformation of $I$ according to the optical flow $O$. It should be emphasized that the warp operation in the formula is only to integrate the generated results from previous moment and make generated video frames coherent temporally, not for the driving image and the source image.

Region GAN. After obtaining the human body layout ${\hat{L}}_d^t$ of the source person conditioned on driving pose from Layout GAN, Region GAN is proposed to generate five person regions conditioned on the driving pose.

For the network structure, we still adopt the vid2vid structure. Since Region GAN is only for generating raw region images of the source person, here we only use one generator to generate five regions of the human body. This can not only save computational resources but also prevent the model from over-fitting. Specifically, we use the encoder $E_1^r$ to encode the $i$-th region mask ${\hat{L}}_{d,i}^t\in R^{1\times H\times W}$ of the human body to obtain the feature map $F_1^r$. In addition, we use another encoder $E_2^r$ to encode the $i$-th region of the source person image and the previous generations to obtain the feature map $F_2^r$. Here we directly concatenate ${\hat{I}}_{d,i}^{t-1}$, ${\hat{I}}_{d,i}^{t-2}$and $I_{s,i}$ along the channel dimension to utilize the previous generations for enhancing the temporal consistency of appearance details. The original Region GAN can be expressed as

Global Alignment Module. The scale and spatial position of the person in the video may vary greatly, so the scales and positions of ${\hat{I}}_{d,i}^t$ and $I_{s,i}$ may not match, then the extracted features $F_1^l$ and $F_2^l$ are also not aligned. Therefore, simply adding $F_1^l$ and $F_2^l$ does not fuse the features well. To scale it, we propose the Global Alignment Module (GAM), a simple but effective Affine transformation to transform the $I_s$ before feeding it to $E_2^r$, so that the scale and location of $I_s$ are consistent with the $L_d^t$.

See the green rectangle of Figure 2 (a). First, we obtain the human masks $M_d^t$ and $M_s$ according to ${\hat{L}}_d^t$ and $L_s$, respectively. According to the human mask, we can obtain the foreground image ${FG}_s$ of the source image. We express the process of aligning the foreground of the source foreground image with the following formulation:

where ${\overline{FG}}_s$ represents the global aligned foreground image of the source image. $R$ represents the scaling factor, which is obtained according to $H_d/H_s$, $H_d$ and $H_s$ represent the height of $M_d$ and $M_s$, respectively. $c=[c_x,c_y{]}^T$ represents the position offset, which can be obtained by $c_d-c_s$, where $c_d$ and $c_s$ represent the center position coordinates of $M_d^t$ and $M_s$ , respectively. So the Region GAN can be expressed as

The loss of Region GAN is the sum of the loss of five synthesized region images, and each loss consists of three items, e.g., the reconstruction loss, the perceptual loss (Johnson et al., 2016), and the adversarial loss.

Reconstruction loss. It directly constrains the generated image ${\hat{I}}_{d,i}^t$ in the pixel space to be closer to ground truth $I_{d,i}^t$. Compared with the L2 loss, L1 loss can focus on the subtle differences of the image. Its formulation is as follows:

Perceptual loss. It regularizes the generated image ${\hat{I}}_{d,i}^t$ and ground truth $I_{d,i}^t$ to be closer in multi-dimensional feature space. The perceptual loss includes the feature content loss and feature style loss, which can be expressed as

where $j$ represents the $j$-th layer of the pre-trained VGG-19 (Simonyan and Zisserman, 2015) model, and $G$ represents the Gram matrix of features.

Adversarial loss. It aims to force the synthesized images to conform to a similar distribution of real images. In order to make the network focus on multi-scale image details, we also use the multi-scale conditional discriminator proposed in pix2pixHD (Isola et al., 2017). It takes a synthesized image and the corresponding human region mask as input. Its expression is

Therefore, the entire loss function of the generator is shown below.

where ${\lambda }_{rec}$ and ${\lambda }_{per}$ are the weights for the reconstruction loss and perceptual loss, respectively.

For the discriminator, its loss function is

After generating the each region image, we can obtain the foreground image of the target image using the following formula:

However, this is only the sum of the individual regions of the human body generated by Region GAN, each region is still insufficient in connections and some details. In addition, we still need to add the background to the generated person image. Therefore, the Whole Generation Network is proposed to further enhance the details of the person, improve the unreasonable areas and add the background image to the person image.

As shown in Figure 2 (b), for the Whole Composition Network, we use three encoders to extract information. a) We use encoder $E_3^w$ to encode the concatenated $\{{\hat{L}}_d^{t-2\sim t},P_d^{t-2\sim t}\}$ to obtain feature map $F_3^w\in R^{c\times h\times w}$. Three adjacent frames as input can implicitly improving the temporal consistency of generated frames. b) We use the encoder $E_2^w$ to encode concatenated $\{{\widehat{FG}}_d^{t-2},{\widehat{FG}}_d^{t-1},{\widehat{FG}}_{d,r}^t\}$ to get the feature map $F_2^w\in R^{c\times h\times w}$. c) In order to exploit the true foreground information of the source person image again, we use the encoder $E_1^w$ to encode ${\overline{FG}}_s$ to get the feature map $F_1^w\in R^{c\times h\times w}$. Because ${\hat{L}}_d^t$, $P_d^t$ and ${\widehat{FG}}_{d,r}^t$ are all in the driving pose, the obtained feature maps $F_3^w$ and $F_2^w$ can be directly added for feature fusion.

Texture Alignment Module. However, ${\widehat{FG}}_{d,r}^t$ and ${\overline{FG}}_s$ are the same person with the same textures but having different poses. So the Texture Alignment Module (TAM) is proposed to better fuse feature maps $F_1^w$ and $F_2^w$ for refining local details. TAM calculates the texture similarity between each position of $F_1^w$ and each position of $F_2^w$, then adds the aligned $F_1^w$ to $F_2^w$.

As shown in the blue rectangle of Figure 2 (b), we first reshape $F_1^w$ into $H_1\in R^{c\times hw}$, and we can get $H_2\in R^{c\times hw}$ by $F_2^w$ in the same way. Then we calculate the similarity of each position in $H_1$ to each position in $H_2$ to get the affinity matrix $A\in R^{hw\times hw}$. Here we use the cosine distance to represent the texture similarity. The calculation formula is as follows:

where $\cdot$ means matrix multiplication, $H_{1,i}$ represents the feature of $H_1$ at the $i$-th position, and $H_{2,j}$ represents the feature of $H_2$ at the $j$-th position. Next, we multiply $H_1$ by the affinity matrix to get the alignment vector, then add it point-to-point with $H_2$, finally reshape it to the original dimension to get the fused feature map $F_{1,2}^w\in R^{c\times h\times w}$, which can be formulated as

After getting the fused texture feature $F_{1,2}^w$, we add it to the pose feature $F_3^w$ and send it to decoder $D_1^w$ to get the final foreground image ${\widehat{FG}}_d^t$. The entire foreground image generation process can be expressed as

Although a realistic person image is synthesized, we still need to add a reasonable background to the foreground person image. Therefore, we send the additive feature map $F_2^w$ and $F_3^w$ into decoder $D_2^w$ to get the soft-mask $m$, then multiply it with $BG$ to get the corresponding background image. Finally, we add the background to the foreground image to get the final image. Its calculation formula can be expressed as

It should be noted that we also use the same strategy as Layout GAN, using the predicted optical flow of adjacent frames to further refine the generated image. In addition to using the same loss function as Region GAN, we also design another discriminator specifically for the facial region to generate realistic faces.

Training. In the training phase, in order to get the paired training data as supervision information, we select a forward frame in a video as the source image and this video as the driving video. Before training our model, we first use pre-trained models to extract the poses and the human parsing maps of frames. We train our REMOT step by step. First, we train Layout GAN and Region GAN separately for 10 epochs. Then we train the Whole Composition Network for 10 epochs using the output of Region GAN.

Inference. In the inference stage, the choice of driving video is not limited, as long as it is a clear video of any solo person. For convenience, our model can perform end-to-end inference.

Implementation Details. Like (Wei et al., 2021), all frames are resized and cropped to 256x192 before training the models. We use Openpose (Cao et al., 2021) to get the 2-D keypoints of the human body, use SCHP (Li et al., 2020) to get the human parsing, and use the optical flow obtained by FlowNet2 (Ilg et al., 2017) as ground truth. We set ${\lambda }_{rec}$, ${\lambda }_{per}$ in Eq10 to 10 for balancing several losses. We use Adam optimizers (Kingma and Ba, 2015) with learning rate of 0.0002, where ${\beta }_1=0.5,{\beta }_2=0.999$ on Nvidia Tesla P40.

Dataset. In order to comprehensively evaluate the performance of our proposed model, we choose two public datasets for training and testing.

iPER Dataset. The iPER dataset was proposed by (Liu et al., 2019b), which was collected in the laboratory environment. The dataset includes 30 actors with a total of 206 videos. Its action types of an actor include A-pose and random actions. The scale of iPER is relatively large and the actions are relatively simple. Same as the original protocol of iPER, 164 videos are used for training and the remaining 42 videos for testing.

SoloDance Dataset. Different from the iPER dataset, SoloDance dataset contains 179 solo dance videos in real scenes collected online (Wei et al., 2021), and each video has 300 frames. The dataset has 143 subjects, and its action types are mainly dances such as hip-hop and modern dance. Compared with the iPER dataset, the actions in this dataset are more complex and the types of clothes are more diverse. However, this dataset is not as large as the iPER dataset. Following the setting of (Wei et al., 2021), we also use 153 videos for training and 26 videos for testing.

Compared Approaches. We select several representative state-of-the-art HVMT methods for comparison with our proposed approach. They are personalized methods END (Chan et al., 2019), directly generation method FSV2V (Wang et al., 2019), feature warping method LWGAN (Liu et al., 2019b), and image warping method C2F (Wei et al., 2021). We also use 3,000 video frames for each subject to train the exclusive EDN model, and the training strategies of other methods are the same as our method.

Metrics. In order to comprehensively compare the quality of generated images, we use four image-level evaluation metrics: SSIM (Wang et al., 2004), PSNR, Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018), Fréchet Inception Distance (FID) (Heusel et al., 2017) and one metric for evaluating the temporal consistency of generated videos, i.e., Temporal Consistency Metric (TCM) (Yao et al., 2017). SSIM measures the structural similarity between the generated image and the real image in pixel space. PSNR evaluates the generation quality based on pixel-wise errors. LPIPS assesses the perceptual distance based on feature vectors. FID represents the Inception (Szegedy et al., 2016) feature distance between two image sets. As for TCM, it evaluates the temporal consistency of generated videos by calculating the distance between the current video frame and the warped neighbored frame with optical flow.

A quantitative comparison of our method with other methods on the iPER dataset is shown in Table 1. It shows that our proposed method outperforms other methods in terms of image quality and temporal consistency. EDN trains the exclusive model for each specific person, so it achieves satisfactory results on all metrics. Because FSV2V directly generates the whole person image, resulting in unreasonable person images, it performs the worst among all methods. For LWGAN, there is no constraint on temporal consistency, resulting in jitters in the generated human video. Therefore, LWAGN does not perform well on the TCM metric. Since half of the videos of rotating actions in the iPER dataset, it is difficult for C2F to handle such a situation with drastic pose differences. Thanks to proposed region-to-whole strategy, our method can be applied to more situations, thus achieving the best results on all metrics.

Table 2 provides the comparison of our method with other methods on the SoloDance dataset. It can be seen that we have achieved the best results on the Mask-SSIM, Mask-PSNR, LPIPS and TCM. (Mask means masking the background of the image according to the human parsing map) This shows that our method has advantages in the quality of generated person images and the temporal consistency of generated videos. The reason why we don’t achieve the best results on other metrics may be that the scale of the SoloDance dataset is not large enough for training the model composed of entirely GANs compared to the iPER dataset. In addition, C2F directly retains the clothes of the source image based on predicted optical flow, which makes it achieve the best results on the FID metric. However, this also leads to generating unreasonable person images in the case of dramatic pose differences between the source person and the driving person. (It can be seen in the areas marked by the yellow dotted boxes of Figures 3 and 4.) Overall, our method can achieve the state-of-the-art on both two datasets.

As shown in Figures 3 and 4, we randomly select three video frames of different poses from synthesized videos on iPER and SoloDance datasets. Although EDN can generate realistic images in some cases, unseen poses often lead to unreasonable person images (Marked with green boxes in Figures 3 and  4). Therefore, EDN is limited by the pose diversity of the particular person in the training set. It can be seen that due to the inaccuracy of the SMPL predicted by HMR (Kanazawa et al., 2018), LWGAN will generate uneven texture and jitter (Marked with red boxes in Figures 3 and  4). For C2F, when the poses of the driving image and the source image are different dramatically, optical flow prediction becomes inaccurate. So the synthesized clothes cannot be warped precisely, which results in unrealistic person images. (Marked with yellow boxes in Figures 3 and  4) This may seriously affect human visual perception. Compared with other methods, our method can better handle the situation of drastic pose changes while preserving the details. In addition, the person images generated by our model generally have clearer faces. This is due to our progressive model in which the initial face image provides an important template for the final clear face.

In order to verify the roles of the Global Alignment Module (GAM) and Texture Alignment Module (TAM) on the generated results and the effectiveness of the Whole Composition Network (WCN), we perform the ablation experiments on the iPER and SoloDance datasets. The variants of the framework are as follows:

• RGN w/o GAM. It means that the foreground image is generated without using the GAM.

• REMOT w/o WCN. It means that the foreground image is generated through the RGN using the GAM.

• REMOT w/o TAM. It means that the REMOT model directly adds the feature $F_1^w$ to the feature $F_2^w$ point-to-point, instead of using the TAM.

• REMOT. It refers to our proposed complete REMOT model.

The experimental results are shown in Tables 3 and 4. Since the RGN only generates person foreground images, we only calculate the Mask-SSIM and Mask-PSNR. Experimental results in Table 3 show that the GAM does not significantly improve the quality of the generated images on the iPER dataset. This is because in iPER videos, the motion of actors is relatively simple and the scale and position of actors remain unchanged, which means that the source image and the driving image are originally in a weak alignment. However, the GAM plays a greater role because of the large range of actions of the subjects in the SoloDance dataset. Obviously, on both datasets, WCN can significantly improve the quality of generated images. Compared with direct feature addition for fusion, TAM can better fuse the features of the generated raw foreground image with the features of the source image.

As can be seen from Figure 5, WCN plays an important role in enhancing facial details and clothing textures. Compared with the direct addition of features, TAM is more beneficial to generate realistic person images.