Leveraging Synthetic Data to Learn Video Stabilization Under Adverse Conditions ^††thanks: The paper is under review.

Video stabilization methods can be categorized broadly into non-learning-based and learning-based approaches. Non-learning-based video stabilization methods do not perform training. For instance, Grundmann et al. [11] stabilize the shaky camera trajectory using L1-norm optimization under constraints. Similarly, Bradley et al. [4] address the stabilization task as a constrained convex optimization problem. These non-learning-based methods stabilize videos without the need for training data to tune the model’s parameters. However, they tend to give less pleasant results, can work only under predefined conditions, and the model’s parameters must be tuned manually.

The learning-based approaches, on their turn, can be classified into unsupervised and supervised approaches. Non-supervised video stabilization methods require training videos but do not demand for annotated pairs of shaky-stable videos. Deep Iterative FRame INTerpolation (DIFRINT) [6] is an unsupervised learning-based approach that can be trained in an end-to-end manner. It can stabilize videos without cropping original frames. Essentially, this method utilizes the frame interpolation technique to synthesize middle frames for video stabilization. Supervised learning approaches, on the other hand, require human labeled ground-truth data, which is the main limitation of applying them to video stabilization. Some works approached this problem using a mechanical stabilizer to generate ground-truth stable videos with their shaky counterparts. As an example, StabNet [32] followed this procedure, generating videos for training CNNs for video stabilization. The network learns a warping transformation of multi-grids given the shaky video frames and the previously stabilized ones. On the other hand, Liu et al. [22] apply a learning-based hybrid-space fusion to compensate for optical flow inaccuracy. They synthesize stabilized frames by fusing the warped content estimated from neighboring frames. Yu et al. [35] learn to stabilize videos using optical flow. They compute the per-pixel warp field from the optical flow of the shaky video, allowing to better handle moving objects and occlusion. In contrast to the work of Yu et al. [35], Ali et al. [1] propose a full-frame supervised video stabilization method that does not require optical flow. Their novel pipeline for dataset generation uses a linearly moving window on high-resolution images.

Despite the substantial progress made by these methods, they present a partial solution to the video stabilization problem since it is assumed that they will work under normal weather conditions and sufficient illumination. However, finding resilient features in adverse conditions is rather challenging. For example, rain particles, foggy weather conditions, and low illumination pose clear challenges to finding robust features. Thus, it leads to inaccurate motion estimation and low-quality video stabilization. At the same time, collecting diverse real-world videos for training video stabilization methods is cumbersome and time-consuming. Moreover, obtaining data with perfect ground-truth is not feasible given the physical limitation of recording the same scene from the same perspective via steady and shaky cameras.

Our video stabilization method belongs to the supervised learning-based category. However, unlike other methods, we use only synthetic data for training. To the best of our knowledge, this is the first work to utilize synthetic videos for video stabilization. No pre-training or fine-tuning on real data is required, and by using only a small-scale training dataset, our method is more robust than state-of-the-art methods.

Estimating the affine or homography transformation between two images are two common approaches to align one image to another. For that aim, there are different ways to find these matrices like applying a feature extractor (e.g., SIFT [23], ORB [28], SURF [3], and OAN [36]) and an outlier rejection algorithm (e.g., RANSAC [9] and MAGSAC [2]). At the same time, there are unsupervised [37] and supervised [7] methods that estimate, specifically, the homography matrix.

Although the traditional approach, i.e., feature extraction, does well at standard conditions, it performs poorly under challenging conditions such as adverse weather conditions and low illumination. Moreover, supervised approaches cannot reflect scene parallax [33] and generating a suitable training data is rather hard. Unsupervised approaches tried to solve these problems but they still fail under large baseline alignment which makes it unpractical for applications such as image stitching and video stabilization under sharp camera movements.

Unlike previous methods, our model learns the affine transformation in a supervised manner using specially generated synthetic training data for this task and it does not need any real data in training time. Estimating the affine transformation is of a special importance in video stabilization because it allows us to recover the camera translation, rotation, and scale. Although it is possible to decompose the homography matrix to extract these information, it is not accurate. Moreover, training a model to estimate the affine transformation is much easier compared to estimating the homography transformation. As have been discussed by other similar works [11, 18], although homography can better model the camera motion between frames for a limited number of frames, it starts to cause many artifacts like skew and perspective as the number of frames becomes larger. At the same time, higher degrees of freedom transformations (e.g., homography) overfit easily even with some regularization. Thus, utilizing homography transformation is both harder to train (more parameters and easier to overfit) and more subject to artifacts.

The use of synthetic data in the computer vision community has shown to be a promising solution to overcome the lack of suitable data for training supervised learning models [5, 27, 30, 21, 14, 31]. Adapting a specific video game to generate synthetic data with its corresponding ground-truth for the task of semantic segmentation was presented by Richter et al. [27], who modified the game Grand Theft Auto V for that purpose. At the same time, another work by Shafaei et al. [30] investigated the use of photo-realistic video games to generate synthetic data and their corresponding ground-truths for image segmentation and depth estimation. Using open-source animation movies was another approach discussed by Butler et al. [5], who showed how to obtain an optical flow large-scale dataset, MPI-Sintel, following a systematic and easy process. Dosovitskiy et al. [8] presented CARLA, a popular open-source simulator widely used for autonomous driving research. Recently, the UrbanScene3D simulator was proposed by Liu et al. [21] and developed for computer vision and robotics research. The system supports autonomous driving and flying research in different environments. GANcraft is another work by Hao et al. [12] for generating photorealistic images.

Despite the great advancements of the previous simulators, they present a partial solution for the data generation issue because of the lack of control on the environmental elements. They also fail to randomize the scene elements which will lead to some clear repetitions (to scene elements) when a large-scale dataset is required to be generated by these methods. In our simulator, we utilize procedural content generation to generate 3D virtual worlds. Additionally, our simulator can generate a special training data for the task of video stabilization. In our experiments, we show that generating appropriate training data and creating a pipeline that utilizes synthetic data, can achieve superior results. In fact, our algorithm can still provide better results in real videos even though our engine does not generate state-of-the-art photo-realistic videos. Additionally, our synthetic aware algorithm and our specially designed synthetic data both contribute to teaching the model to accurately estimate the affine transformation while not overfitting to the synthetic data distribution being trained on. Thus, it mitigates the domain gap and achieves satisfactory results on real data.

The first stage of our pipeline consists in estimating the camera motion throughout the video. Most existing 2D-based stabilization approaches apply key-point feature extraction and tracking to solve this task [17, 11, 19]. However, both feature extraction and tracking may fail under adverse weather conditions due to repetitive textures and partial occlusions caused by rain and snow particles or textureless objects under foggy weather and at night-time. To overcome this issue and properly recover the camera motion in $V$, we propose estimating parameters $t_x$, $t_y$, $\theta$, and $s$ of an affine transformation matrix

for every consecutive pairs of frames using deep neural networks with synthetic data. Thus, we abdicate the feature extraction procedure entirely since, using our proposed engine, we can generate the ground-truth affine transformation needed for training as described in Section 3.3.

Two identical networks are utilized: the $f_{tr}:R^{4\times W\times H}\to R^2$ network that estimates the $x$ and $y$ translations, i.e., $x^{tr}=[t_x,t_y]$; and $f_{rs}:R^{4\times W\times H}\to R^2$ that outputs the rotation angle and scale values $x^{rs}=[\theta ,s]$, where $W=H=256$ is the center cropped image width and height. It is important to note that $f_{tr}$ and $f_{rs}$ share the same architecture, but not the same weights.

Both networks consist of a feature extractor implemented as four convolutional layers followed by a pooling and a dropout layer, and a regressor, which is a fully connected network composed of three linear layers that process the extracted features to estimate the parameters. The number of output channels of each layer is illustrated in Figure 2-a. For each training step, we feed the networks with an input $I=[v_i;v_{i+1};O_i]\in R^{4\times W\times H}$, where $v_i,v_{i+1}\in R^{W\times H}$ are two consecutive grayscale frames from the input video $V$ and $O_i\in R^{2\times W\times H}$ is the optical flow map for the pair $(v_i,v_{i+1})$ obtained via the FlowNet2 [13, 25]. Then, we estimate the parameters for $A_i$ as ${\hat{x}}^{tr}=f_{tr}\left(I\right)$ and ${\hat{x}}^{rs}=f_{rs}\left(I\right)$.

A key contribution of our approach is the usage of specially-designed synthetic data to learn affine transformation. Let $P_i=\{p_1,p_2,\dots ,p_K\}$ denote the 2D coordinates of $K$ mark points at the frame $v_i$ from a generated synthetic video. Since these mark points in frame $v_i$ and $v_{i+1}$ are static in the world space, we can compute an affine transformation $A_i$ with $4$ degrees of freedom using $P_i$ and $P_{i+1}$. Following this strategy, we get the ground-truth values $t_x$, $t_y$, $\theta$, and $s$ for a single pair of images. The detailed process of the ground-truth data generation is described in Section 3.3.

Finally, with the estimated parameters $\hat{x}=[{\hat{t}}_x,{\hat{t}}_y,\hat{\theta },\hat{s}]$ for each image pair in the video, we can compute the estimated camera trajectory $\hat{T}=\{{\hat{t}}_1,{\hat{t}}_2,\dots ,{\hat{t}}_{N-1}\}$, where ${\hat{t}}_i={\sum }_{j=1}^i{\hat{x}}_j$, and ${\hat{x}}_j$ represents the estimated parameters for the pair of frames $(v_j,v_{j+1})$. It is important to note that, similar to Grundmann et al. [11], we do not use ${\hat{t}}_i$ directly to warp the shaky images; we warp the frames by applying smoothed affine transformation composed using the smoothed translation, ration, and scale parameters as detailed in the following section.

After estimating the shaky camera trajectory, we need to smooth it. In contrast to other methods like the work of Grundmann et al. [11], where they tackle the camera trajectory smoothing as an optimization problem, we deploy the Savitzky-Golay filter [29] on the averaged envelop of the shaky camera trajectory to smooth it, as described in the sequel.

The goal of our network is to infer the affine transformation matrix given two consecutive frames. Thus, we need to generate the affine transformation ground-truth to supervise the training process. Our idea is two-fold: a) create stationary hypothetical labeled objects in the 3D world scene; b) record their coordinates in the screen space of the recording camera. In that way, we can guarantee that the coordinates of these objects in frames $(v_i,v_{i+1})$ correspond to exactly the same static elements in the 3D world seen by the recording camera at frames $v_i$ and $v_{i+1}$. In other words, we create a number of invisible objects and save their coordinates in the camera space for each frame. In order to do so, for each frame a number of random points are sampled from the screen camera space. Then, a ray is cast from each of these points to infinity. At each ray’s intersection point with the scene, a hypothetical invisible object is created. The object remains stationary for a number of seconds before being destroyed. For each frame, the object position in world space is transferred to the camera space and saved in XML format. Figure 3 demonstrates how these hypothetical objects are created.

If the current camera view does not include the hypothetical object, or if it exceeds its time limit duration $\beta$, it is removed. Each object is given a Unique Identifier (UID) over its lifetime. Later in the post processing stage, for each two consecutive frames $(v_i,v_{i+1})$ using the UIDs of these hypthetical objects and their screen locations, the affine transformation ground-truth is calculated for each pair $(v_i,v_{i+1})$ as described in Algorithm 1. It should be noted that generating millions of hypothetical objects is still a valid option. However, our algorithm provides a solution to create only objects in the field view of the recording agent improving the overall performance.

The available video stabilization benchmarks such as DeepStab [32], Stabfr [38], Selfie Video [34], and LiuSigg2013 [20] exclusively contain videos under normal weather condition and at a sufficient illumination. To assess the performance of the state-of-the-art video stabilization methods under foggy, rainy, snowy, and night-time conditions, we created the VSAC105Real dataset.

Our dataset is composed of videos collected from YouTube using search queries like “Fog”, “Rain”, “Snow”, “Night”, “Adverse”, and “Severe”. We manually inspected all the videos and selected the ones with shaking camera movement. Then, we cut the videos to ensure continuous temporal criteria and the query attribute. VSAC105Real dataset comprises $105$ videos spanning normal, rainy, foggy, snowy, and night-time attributes. The first four attributes were selected to study the effect of severe weather conditions on video stabilization quality. Similarly, the night-time was chosen to understand the effect of low illumination on video stabilization. Table 1 shows a comparison among different video stabilization datasets and VSAC105Real dataset. The VSAC105Real dataset has the advantage in terms of the average number of frames. Moreover, it includes a diverse set of challenging attributes where videos are evenly distributed across the classes, i.e., $21$ videos per class. A visual comparison among VSAC105Real and other video stabilization datasets is depicted in Figure 6.

Using Silver, we generate two different synthetic training datasets: VSNC35Synth under normal weather conditions and VSAC65Synth under both normal and adverse weather conditions.

Table 2 shows the results of comparing our method to several video stabilization approaches. As can be seen, our method presented the best values on average in comparison to all the baselines in terms of stability average, distortion, cropping ratio, and success rate. Even though our method did not surpass the baselines at each class individually, it still achieved competitive results. Every baseline performs badly in at least one class, while our method is more robust across classes, hence holding the final best results on the VSAC105Real.

Preserving the content while compensating for camera shakiness is another important feature of our algorithm. Our method achieved the best results as compared to other state-of-the-art methods. The superiority of our method can be linked to the accurate affine transformation matrix estimation and the smoothing stage. Moreover, our method achieved the highest success rate compared to the competitors as shown in Table 2. It is worth noting that all baselines failed to stabilize most of the shaky videos at foggy weather conditions. The reason for this outcome is that participating media, like fog, work as a low pass filter that removes high-quality features that most videos stabilization algorithms depend on to estimate the camera trajectory. We highlight that even though our model was not trained on any samples under the foggy weather condition, it was capable of learning useful features from both raw images and optical flow.

Furthermore, we evaluate our model on Selfie Video dataset [34], contains videos under normal weather condition and standard illumination. Our model achieves comparable results to other methods while maintaining the best distortion score as shown in Table 3. It should be noted that our model was trained from scratch on synthetic data only.

We analyzed different design options and showed the effectiveness of each component of our proposed pipeline. The results are reported in Table 4. First, since we use two networks in our pipeline, we trained using a single CNN with four losses: horizontal translation loss, vertical translation loss, rotation loss, and scale loss. As a result, the network was unable to converge well. One problem could be that the translation losses were larger than others. However, even after applying weights for the losses to make them comparable to each other, the network could not learn well (column Single Network in Table 4).

To emphasize the advantages of using our learning-based model, for affine transformation matrix estimation over applying SIFT, we apply SIFT to find the affine transformation matrix while keeping the smoothing part of our model intact. As expected, SIFT did not perform well (column SIFT in Table 4). Standard feature extractors like SIFT struggle to extract reliable and robust features under adverse conditions. Rain and snow particles, low illumination at night, and foggy weather make finding and matching features rather hard. This leads to inaccurate affine transformations, thus low-quality stabilized videos. To further evaluate the advantages of using our smoothing algorithm over $l_1$ directed smoothing, as done in [11], we apply $l_1$ directed smoothing on the predicted camera trajectory, while keeping our learning-based model for the affine transformation matrix estimation. As expected, the model does not preform very well as compared to using our proposed smoothing algorithm (column $l_1$ directed smoothing in Table 4) because ours considers more sophisticated camera paths and is not limited to constant, linear and parabolic motions like [11].

To show the need for optical flow in the affine transformation learning, we train using only graysale images rather than both grayscale and optical flow modalities. As anticipated, not utilizing optical flow information decreased the video stabilization quality (column No Optical Flow in Table 4).

At last, to demonstrate the effect of our training synthetic data on the quality of video stabilization, we trained our model on more data, including normal weather, adverse weathers, and night-time videos. The training was performed using VSAC65Synth dataset from scratch, and no real data were used. The results indicated no significant improvement on the baseline that was trained on VSNC35Synth dataset. Thus, we can see in Table 4, under More Data column, that a few synthetic videos, with accurate ground-truth, were sufficient to train our model to learn affine transformation. Thus, increasing the number and diversity of the training videos did not boost the overall performance.

To further investigate the accuracy of our estimated affine transformation matrix, we carry a special set of experiments shown in Table 5. We compare our learning-based affine transformation estimation model to three types of affine estimation approaches: i) the traditional ones, including ORB and SIFT with RANSAC, MAGSAC, and LMEDS for outliers rejection; ii) supervised; and iii) unsupervised approaches. We note that traditional approaches can be used to estimate the affine transformation directly, but supervised and unsupervised methods are designed to estimate the homography matrix. Thus, for fair comparison, we extract the affine transformation from the estimated homography one. We utilize the dataset of [37] which contains $\mathrm{4,200}$ pairs of images where each image-pair includes 6-matching human-annotated pairs of points. The dataset covers regular-texture (RE), low-texture (LT), low-light (LL), small-foregrounds (SF), and large-foregrounds (LF). We use $l_2$ distance to measure the error between the warped and ground-truth points similar to [37, 33]. Table 5 demonstrates that our method can estimate the affine transformation better for both standard and challenging conditions. Figure 7 shows a qualitative comparison among our model, traditional (e.g., SIFT and ORB), supervised [7] and unsupervised [37] methods. While other methods fail under such challenging conditions, our method performs well because it learned how to extract resilient features using our specially designed synthetic data.