StableFace: Analyzing and Improving Motion Stability for Talking Face Generation

Video-driven talking face generation aims to reenact the existing facial image with external driving source, e.g., another talking image or video [thies2016face2face, ling2020toward, wiles2018x2face, kim2018deep, nagano2018pagan, xue2020realistic, tang2022generative]. Although these works can synthesize talking face images with good visual quality, video-driven methods require an additional driving video, and still need another lip-sync audio to compose a complete talking face video. Audio-driven talking face generation employs speech as the driving source and synthesizes face video in sync with the input speech content. In this paper, we focus on audio-driven talking face generation, since it has some advantages over its video-driven counterparts: 1) they are more essential to talking face generation since the goal is to generate talking face from talking audio, and 2) they do not need additional driving video which makes them more generally applicable to different scenarios.

Recently, transformer [vaswani2017attention] has been becoming the most popular in sequence modeling long-term correlations in various research fields [devlin2018bert, ren2019fastspeech, fan2022faceformer]. Typically, the transformer consists two main conponents, incorporate a multi-head self attention mechanism over the tokens and a feedforward module. Due to its capability in long-range relationship modeling, transformer-based models have been applied to various kinds of vision tasks, such as visual recognition [dosovitskiy2020image], 3D facial animation [fan2022faceformer, chen2022transformers2a], inpainting [liu2021fuseformer, ren2022dlformer], object detection [carion2020end, zhu2020deformable], image synthesis [zeng2021improving], image harmonization [guo2021image] etc.

In the fileds close to the talking face video generation, as far as our knowledge concerns, most of recently proposed methods deal with 3D meshes generation [fan2022faceformer, li2020learning] or facial blendshape coefficients estimation [chen2022transformers2a], while fewer of them focus on photo-realistic talking face generation. Fan et. al. propose FaceFormer [fan2022faceformer] designs an autoregressive transformer-based architecture that predicts moving coordinates for 3D facial mesh animation. Li et. al. [li2020learning] propose a two-stream transformer to model the motion distribution and capture long-term dependency. Chen et. al. [chen2022transformers2a] propose a MOE-based transformer, to generate facial animation coefficients from speechwhich should be fed into graphic engine to generate animations. Despite these methods achieve significant improvements in animation generation, they cannot be adapted to photo-reslistic video generation framework because the input and output data are both 4 dimensions, i.e., $R^{T\times H\times W\times C}$. Therefore, in this paper, we treat photo-realisitc talking face generation as a frame-based sequence-to-sequence generation task and build a transformer-based architecture to capture dependencies among consecutive video frames.

Besides the lip-sync quality and high image fidelity, in the talking face videos, the frame visual consistency (e.g., illumination, or color) and motion jitters also harm its naturalness and realness. Previous works have tried different approaches to improve the quality of synthesized images to make people feel more realistic over each frame or improve the lip-sync quality of the talking videos. To improve frame consistency, some talking face generation methods introduce autoregressive generation strategy [song2021tacr, lahiri2021lipsync3d] by ”looking at previously generated frames” in the neural renderer, or incorporate a temporal discriminator to judge the video quality on multiple consecutive frames [chen2019hierarchical, vougioukas2020realistic, prajwal2020lip]. However, these methods are not designed for improving motion stability, and are not efficient enough to mitigate the motion jittering problem (which can be clearly observed in their released talking face videos). Yu et. al. [yu2020multimodal] utilize motion flow to warp previous images to improve consistency. While achieving better temporal consistency, the optical flow only represents the motion between two adjacent frames, but not long-range dynamics. Meanwhile, inaccurate and unstable motion flows also hinder stable motion generation.

Beyond talking face generation task, some works try to solve the temporal inconsistency problems [lai2018learning, lei2020blind] that are incurred by illumination or color changing across the adjacent frames. Unfortunately, this kind of temporal jitters of appearance in consecutive frames is not the same as motion jitters which care more in the object motions and less in appearance. Some other works try to improve video stability for hand-held captured videos due to the high-frequency camera jitters occur in video capturing procedure [matsushita2006full, liu2009content, wang2018deep, li2022deep] but not the motion jittering problem in video synthesis.

To analyze and mitigate the motion jittering problem, we investigate previous approaches that can handle talking face generation on personalized talking face videos, and then design a baseline. In brief, the baseline is audio-driven, two-stage, and uses 3D face parameters as the intermediate representations between audio input and video output. In this section, we introduce each component of our baseline as follows.

We consider the generation procedure as a sequence-to-sequence mapping problem ${\hat{I}}_{1:T}=G(s_{1:T},x_{1:T},\theta )$, where $s_{1:T}$ and $x_{1:T}$ represent the input 3D face representations and eroded images from time stamp of 1 to $T$, and $\theta$ is the network parameters. Since the neural renderer acquires background visual information (hair, backgrounds) from the eroded background images, it is easy for the neural renderer to synthesize the background-related regions. However, the mouth and jaw parts are more challenging to deal with because 1) the synthesized mouths and jaw should match the background images, and 2) the motions of the lip and jaw are complex and diverse due to the diverse content of input audio. With systematic analyses, we find several causes that may incur motion jitters in the synthesized videos, including causes from $s_{1:T}$, $x_{1:T}$, and $\theta$.

First, jitters from the 3D face representations $s_{1:T}$. We adopt the choice of 3D face shapes to bridge the neural renderer and audio processing module. However, the facial parameters are extracted by the 3D face model on single image or predicted by audio processing module, they are not stable across consecutive frames and can have jitters. Consequently, the jitters from the input parameters incur the jitters of 3D face shapes, which are then input to the neural renderer. A straightforward solution to alleviate this jitters is to smooth the input of the neural render. Instead of smoothing the 3D parameters in parametric space, we propose to smooth the 3D face shapes in geometry space due to the fact that the 3D face shapes are the direct input of the neural render.

Second, training-inference mismatch. In training, we extract the 3D face parameters from the background images and render the face shapes as the input of neural render. Similar to [song2021tacr, thies2020neural, wen2020photorealistic], we mask-out the mouth part and concatenate the eroded image with the face shape in channel-wise manner, and treat them we the input of neural renderer. However, the eroded image and the face shape are from the same target image in training, which relieves the optimizing procedure. In inference, the eroded image is from the same background video, which is not complementary with the new face shape where the mouth area has been changed due to the new audio. Consequently, the neural render should generate a realistic face with mouth part from the face shape while the rest from the eroded image, which makes it difficult for the neural renderer to deal with this distortion and producing more errors around the mouth boundary.

Third, lack of dependency modeling among video frames in the neural renderer. Each image frame in a video is not independent but correlated with its adjacent frames. However, the current neural renderer generates each frame independently, without considering the information from adjacent frames. Therefore, synthesizing each frame without ”looking behind and ahead” probably produces motion jitters in face video. To address this issue, we propose to apply a dependency modeling module to enforce dependencies learning and improve the motion stability for the current neural render.

As mentioned in Sec I, quantitatively evaluating the motion stability of a talking face video is valuable and challenging. However, as far as our knowledge is concerned, there is no available objective metric to measure the the motion stability in previous works [lahiri2021lipsync3d, thies2020neural, song2021tacr, lu2021live], thus forcing us to solely rely on subjective user study. To bridge the gap, we resort to develop an effective metric that measure the motion stability in talking face videos.

To mesure the motion stability, we start from the motion jitters in talking face videos. Motion jitters describe the irregular motion variations in a video, and the more stable motion indicates the better smoothness of the motion variations. Based on this analysis, we propose to use the variance of the motion variations to measure motion jitters. For simplicity we represent the motion variations by accelerations. To acquire the accelerations, the first thing to do is acquiring the motion in each video frame. Intuitive choice to represent motion in video is using optical flow [fleet2006optical] which measures the motion of each pixel between two frames. However, optical flow has two drawbacks. First, the pixels in optical flow have no accurate semantic labels, thus we cannot distinguish the mouth from eyes or nose via optical flow map. Second, optical flow estimates the pixels motion between two consecutive frames, thereby imposing difficulties in forming the trajectory of a certain object of interest, such as the lip. Therefore, in this work, we adopt facial key points to represent the key motions of talking faces. With the coordinates of each key point, we can easily calculate the motions and acquire their trajectories. To achieve this, we first track the facial landmarks via face alignment tool [wood2021fake], and use them to calculate the motions.

Denote the landmarks at each frame $t$ as $Z_t\in R^{N\times 2}$, $N$ is the number of key points which is determined by the facial alignment tool. The motion velocity and acceleration of key point at $i$ at timestep $t$ is formulated as:

where $v_{-1}^i=v_0^i=v_{T+1}^i=0$.

Intuitively, if the video suffers from significant motions jitters, the variations of motion will change dramatically, which means the statistical variance of the accelerations is much higher. The variance of the motion variations can be written as follows:

where ${\overline{a}}^i$ is the mean of $a_t^i$ for $t=1,2,...,T$. From our experimental results, the variance of accelerations is negatively correlated with the subjective score on motion stability (better motion stability, lower variance of acceleration). In this paper, we use the reciprocal of the variance of the acceleration as the motion stability index of the point. Our proposed objective metric for motion stability index is written as follows:

where $K$ is the number of selected key points, and $ϵ={10}^{-5}$.

It is worth to mention that the scores of MSI in lips region differ from the scores in the jaws region which is because the different scale of motions in a face video. Hence, it is preferable to separately measure MSI in mouth region and jaw. To eliminate the interference of acceleration brought by different face sizes, we crop the video via unified strategy where a fixed bounding box is determined according to the ratio of the mouth width over the cropped image width. Specifically, we use the first 5 frames and the ratio of 0.25 to generate the cropping box of each video. For fair comparison, we rescale the face images into 256$\times$256 for each method.

Our neural renderer is optimized in supervised manner to generate a realistic image that are similar to the ground-truth image given with paired audio. We used two kinds of losses, namely reconstruction loss (unbalanced pixel-wise reconstruction loss and VGG perceptual loss [johnson2016perceptual]), and adversarial loss. The unbalanced reconstruction loss is employed to penalize the errors between the synthesized image sequence and the ground-truth image sequence (denoted as $I_{1:T}$). To achieve this, we utilize the mouth masks (denoted as $M_{1:T}$) to indicate different loss weights in the images where 2 for the weight in eroded region and 1 for the rest. A pre-trained VGG model is utilized to calculate the perceptual loss. The reconstrcution loss can be formulated as:

where the weight mask for mouth and non-mouth region are set to 2 and 1, respectively.

The adversarial losses for generator $G$ and discriminator $D$ in our work are implemented as the following formulation:

In our experiment, $D$ shares the same structure as [thies2020neural]. The full objective of neural renderer is:

where ${\lambda }_1$ and ${\lambda }_2$ are set to 20 and 1, respectively.

For Testset 1 and 2, we extract speech audio from the demos of NVP [thies2020neural], LipSync3D [lahiri2021lipsync3d], LiveSP [lu2021live], and AD-NeRF [guo2021adnerf] and animate the faces from Testset 1 and 2. Note that the audios and videos are not matched (without ground-truth videos), we cannot evaluate the results as we do in GRID. To this end, we conduct comparisons via both subjective experiments and objective metrics.

For comparison without ground-truth video setting in which the talking face videos are generated with audios from other source, we conduct both subjective and objective experiments. In subjective experiments, we choose MOS (mean opinion score) as our main rule. 18 experienced users are invited to score the given videos at five grades according to perceived quality of motion stability⁴⁴4Are the motions around the lips and jaw stable across frames? , lip-sync quality⁵⁵5How does the motion of the lips match the speech in lip-audio accuracy? , and video realness⁶⁶6Does the video look real in your mind? . Different grade choices in MOS are: Very Good (5), Good (4), Average (3), Poor (2), Very Poor (1). A detailed annotation guideline and several real/generated videos are provided to ensure users can fully understand the difference between grades. Meanwhile, an additional labeling practice is conducted before the evaluation. For objective evaluation, we adopt MSI to measure the motion stability around lips and jaw regions respectively; Sync-C (SyncNet Confidence) to measure the lip-sync score and distance via SyncNet [chung2016out].

The quantitative results of comparison with ground-truth setting are presented in Table II. It can be concluded that our method outperforms state-of-the-art methods in all objective metrics, i.e., SSIM, CPBD, and WER, on the GRID dataset. With NLMD which measures the normalized landmark distance between synthesized images and the ground-truth, we eleminate the errors that may be brought by image resolution, and present the comparison results to fairly demonstrate the better performance of our method on self-reenactment. In terms of NLMD which, we achieve comparable performance to LipSync3D. Note that LipSync3D generates face images in resolution of 128$\times$128, while our method work on images of 256$\times$256. Our synthesized videos have better quality of sharpness and less word error rate. In Fig. 4, we visualize some key-frame results of our methods on GRID and Obama’s video. Our method can achieve plausible results with shapr details and high image quality under different illumination conditions, skin color, and etc.

We proceed to evaluate the efficacy of our MSI metric. In Table IV, we present the Pearson correlation coefficients between our MSI and the subjective scores of motion stability, where the MSI results and subjective scores of motion stability on 99 videos. Instead of measuring motion stability via the variance of motions or variance of accelerations across frames, we take the reciprocal value of the motion variations to achieve higher correlations between objective metric and subjective scores on motion stability. It can be found that the correlation coefficients between MSI metric and motion stability in videos reach to 0.424 and 0.438 for lip and jaw respectively, which are higher than the statistical measures.

For comparison without groudtruth videos, we present our quantitative results in Table III. In the left of the table, we show the statistical MOS in subjective study. Different from previous works that only evaluate the frame visual quality and lip-sync quality, we add a new evaluating item and show that our synthesized videos looks much better than previous other works. In the right of the table, we evaluate those videos with objective metrics. Considering that different scale of motions around the lips and jaw in talking face video, we separately measure MSI for the lips and jaw for each video. We first compare the results of motion stability, our method achieves higher MOS of 3.72 compared to the second best results. From the results of MSI (Lip) and MSI (Jaw), our method consistently outperforms the competing methods, demonstrating the effectiveness of our solutions on solving the motion jittering problem in talking face generation task. According to the results of motion stability and video realness, it can be concluded that the method that generates videos with better motion stability tends to attain better results in video realness.

To qualitatively compare the generated results of each method, we extract key frames from the videos of each method and show the results in Fig. 5. Among the images synthesized by AD-NeRF [guo2021adnerf], some of them suffer from clear cracks between head and torso. The reason is that AD-NeRF synthesizes the head and torso separately, without considering spatial consistency within each frame. Meanwhile, from their video results suffer from severe body jitters. NVP [thies2020neural] generates videos with nearly muted lip movements while LiveSP [lu2021live] fails to synthesizes accurate lips in sync with the audios. In contrast, our method predicts more accurate facial expressions, and generates videos with higher quality and more accurate lip movements. (We strongly suggest reviewers to watch the enclosed video for better visual comparison experiences on motion jitters.)

To rationalize each designing choice of our method, we conduct extensive ablation experiments by removing each component or all of our three components. From the comparison results of Sync-C in Table V, we find that our dependency module helps to improve the lip-sync quality on both datasets. Meanwhile, in evaluating image similarity with SSIM and shape similarity with NLMD, our full model achieves clear improvements over the method without augmented erosion, dependency module, or adaptive smoothing module. It can be seen that the SSIM results get worse after removing each of our proposed components, which verifies the effectiveness of each design.

We compare our full model with the one without audio fusion. Fig. 6 gives the comparison results, from which we conclude that audio features have the better capability in helping the neural renderer synthesize more accurate lips. In Fig. 7, we concatenate the vertical slice in each frame along the time for each video. The position and size of the vertical slice are shown in the first row and kept the same for each video. We have two observations from the figure. First, without each component, we can observe some jitter patterns from the vertical slice figure. Compared to the baseline, the vertical slip figure of the full model is much more smooth and close to the real video. Meanwhile, the jaggy patterns can also be observed compared to other methods without each component. Second, it may be concluded from the figure that the jitters are more likely to occur in the lip regions (green box) than the nose region (red box) because the lip motions are much more than the nose which is almost static relative to the head.

In Fig. 8, we visualize the user preference results in terms of motion stability, lip-sync quality, and realness. From each sub-figure, it can be observed that the preference percentage motion stability reduces when we remove the adaptive smoothing module, dependency modeling or augmented erosion. Fig. 9 shows the user preference studies over different smoothing strategy. From the results, we can tell that our adaptive smoothing choice clearly attains more votes than that of fixed weight or global but learnable weight, especially on lip-sync quality and video realness.