SSORN: Self-Supervised Outlier Removal Network for Robust Homography Estimation

Our model is built upon a Siamese structure. It takes two grayscale images $I_a$ and $I_b$ as input, and produces a homography matrix $H_{ab}$ from $I_a$ to $I_b$ as output. The entire architecture is composed of four components: a feature extractor $f(\cdot )$, a feature matching module $m(\cdot )$, an outlier removal module $r(\cdot )$, and a homography estimator $h(\cdot )$. The resulting model is illustrated in Fig. 1.

Unlike other components in our model, the feature matching module has no trainable parameters. It explicitly computes a cost volume $C$ between two feature maps $F_a$ and $F_b$. Intuitively, the cost volume can be thought of as a similarity matrix in 3D form. It stores dense feature matching costs of two sets of feature vectors. The process of computing the cost volume is shown in Fig. 2. For clarity, we use the superscripts “2D” and “3D” to denote 2D and 3D tensors, respectively.

To compute the 3D cost volume $C^{3D}$ from $F_a^{3D}$ to $F_b^{3D}$, we first reshape $F_a^{3D}$ and $F_b^{3D}$ into corresponding 2D tensors $F_a^{2D}$ and $F_b^{2D}$ respectively. Then, the matching cost $C^{2D}(i,j)$ between the $i-$th feature vector in $F_a^{2D}$ and the $j-$th feature vector in $F_b^{2D}$ is implemented as the correlation between the feature vectors:

where $\odot$ denotes the dot product, $T$ is the transpose operator, and $D$ is the dimension of feature vectors. Accordingly, the full cost volume $C^{2D}$ is defined as:

where $\otimes$ denotes the matrix product. Finally, the 2D cost volume $C^{2D}$ will be reshaped into the corresponding 3D cost volume $C^{3D}$.

In the traditional view, outliers usually mean incorrect correspondences. In a similar spirit, we can treat those incorrect matching costs in a cost volume as outliers in the cost volume. Since a cost volume stores dense matching costs, it often contains a large number of outliers due to the existence of many similar image features. It is difficult for the homography estimator to predict an exact homography matrix from this cost volume. To this end, we propose the outlier removal model, which treats the outlier removal problem as a cost volume denoising problem. We believe that, for a specific homography matrix, there exists a specific pattern (i.e. a clean cost volume) corresponding to that homography matrix. The goal of the outlier removal module is to recover this pattern from the noisy cost volume produced by the feature matching module.

Apparently, the cost volume is different from natural images, thus the priors for the natural image denoising problem are not suitable for the cost volume denoising problem. To address the training problem, we propose a novel self-supervised loss function as described in Sec. III-D. Theoretically, any deep denoising model can be used as the outlier removal module. In practice, we adopt the UNet structure as the backbone of the outlier removal model. After training with the proposed self-supervised loss function, the outlier removal module is able to remove most of the outliers in the original cost volume $C$ produced by the feature matching module, obtaining a clean cost volume $C^{\prime }$, as shown in Fig. 3. As expected, for a specific homography matrix, the outlier removal module successfully finds a specific pattern (i.e. the clean cost volume $C^{\prime }$) corresponding to that homography matrix.

Given a pair of images $I_a$ and $I_b$ and the ground truth homography matrix $H_{ab}^{gt}$, one simple way to train the model is to use the following supervised loss:

where $\parallel \cdot {\parallel }_2^2$ is L2 norm.

However, this way does not take full advantage of the outlier removal module, because the above loss function does not ensure that the outlier removal module learns the ability to remove outliers in the cost volume. To this end, we propose an additional self-supervised loss function to train the outlier removal module. We exploit the fact that two image pairs related by the same homography matrix should correspond to a similar clean cost volume. We force the outlier removal module to produce similar outputs for these two image pairs. Fig. 4 illustrates the training process of the whole model.

Formally, given two image pairs $(I_a,I_b)$ and $(I_c,I_d)$ related by the same ground truth homography matrix $H^{gt}$, the self-supervised loss for training the outlier removal module is designed as:

where $\parallel \cdot {\parallel }_1$ is L1 norm. The first half of the equation forces the outlier removal module to produce similar cost volumes, and the second half of the equation is used to avoid trivial solutions, i.e. $C_{ab}^{\prime }=C_{cd}^{\prime }=0$.

To train the homography estimator, we still need the following supervised loss:

Combining the above two losses, we get the final loss for training the whole model:

If the idea that a specific homography corresponds to a specific clean cost volume is true, then we seem to be able to derive the grund truth cost volume directly from a ground truth homography, and thus allowing for simple supervised training. In theory, given a ground truth homography and the coordinate of a pixel on one image, we can always compute the coordinate of the unique corresponding pixel on another image, from which we can derive the ground truth cost volume. However, this derivation does not take into account the local self-similarity property of images, that is, adjacent pixels usually have similar (deep) features. In other words, a pixel on one image often has multiple high-scoring matches on another image, not just one match (see the visualization of the clean cost volume in Fig. 3). It is not trivial to take the local self-similarity property of images into account when deriving the ground truth cost volume from a ground truth homography. This motivates us to propose the self-supervised training, which avoids the difficulty of directly deriving the ground truth cost volume.

Although there exist several datasets [22, 2, 21] for homography estimation, these datasets are too small for training deep homography estimation models. To this end, Detone et al. [8] proposed a synthetic dataset S-COCO based on the COCO dataset [18]. S-COCO is generated by applying random projective transformations to the COCO dataset. By using the same data generation rule, we can easily generate two image pairs with the same homography matrix for training our model.

Based on S-COCO, we additionally generate two datasets, called I-COCO and O-COCO, for testing the robustness of different models to illumination and occlusion. In I-COCO, we apply the photometric distortion techniques in [19] to S-COCO. The illumination change is controlled by a parameter $\delta$ ranging from 8 to 32 with an interval of 8. In O-COCO, we simulate occlusion by overlaying a random image patch of size $p$ on one of the input images, where $p$ ranges from 0 to 80 with an interval of 10.

In addition to the above three synthetic datasets, we also conduct experiments on a real dataset, HPatches [2]. The dataset contains two subsets: the illumination subset (i.e. I-HPatches) and the viewpoint subset (i.e. V-HPatches). As this dataset only contains 580 pairs of images, we only use it for testing.

Fig. 5 shows some examples from the above datasets. All images are resized to $256\times 256$ for training and testing. To measure the performance of different models, we adopt the Mean Average Corner Error (MACE) [8] as the evaluation metric.

We compare our model with 6 recently proposed deep learning models (i.e. HomographyNet [8], UnsupervisedNet [25], SRHEN [17], Zhang’s et al. [36] and Koguciuk’s et al. [15] models) and two representative keypoint based methods (i.e. SIFT [20] and SuperPoint [9]).

To ensure that the performance difference between deep learning methods comes from better network structure and loss function design, rather than better CNN backbone, we implement these methods to use the same ResNet34 structure as the backbone. Using the same backbone and the same training settings, most deep learning models achieve very similar performance, but this is not the case for SRHEN and our model (see experiments in the following sections). For keypoint based methods, we compare two feature matchers: the nearest neighbor (NN) with Lowe’s ratio test (ratio is set to 0.9) and SuperGlue (SG). MAGSAC++ [3] is used as the robust estimator.

We adopt the 4-point parameterization [1] to represent the homography matrix instead of $3\times 3$ matrix form. The weight parameters ${\lambda }_1$ and ${\lambda }_2$ in Eq. 8 are empirically set to 0.5 and 0.25, respectively.

We first conduct experiments on the S-COCO dataset. Fig. 6 reports the MACE of different methods on this dataset. In general, deep learning methods outperform keypoint based methods on this dataset, mainly due to the powerful feature representation capability of deep learning features. Compared with those deep models without the feature matching module, SRHEN with the feature matching module achieves a better result. This demonstrates the necessity of the feature matching module not only in the traditional homography estimation pipeline but also in deep homography estimation models. Our model contains not only the feature matching module but also an outlier removal module. It leverages the strength of the traditional homography estimation pipeline and the strength of deep learning features. It obtains a much better result than SRHEN on the S-COCO dataset, which indicates that the outlier removal module is as important as the feature matching module for deep homography estimation models.

To analyze the robustness of different models to different geometric transformations, we generated four subsets according to the data generation rule in [8]. Each subset corresponds to a specific type of transformation, namely translation, scale, rotation and perspective transformaitons. We plot the results in Fig. 7. It manifests that all methods perform well when the magnitude of the geometric transformation is small. However, the performance degrades as the magnitude of the geometric transformation increases. Specifically, deep learning models that do not employ the feature matching module perform poorly for large geometric transformations, especially for large translations. Overall, our model is more robust to these four types of geometric transformations than other methods.

To investigate the robustness or the generalization ability of deep learning models to illumination and occlusion, we conduct experiments on the I-COCO and O-COCO datasets. We employ two experimental configurations. In the first experiment, we train all deep learning models on the S-COCO dataset and test them on the I-COCO/O-COCO dataset. In the second experiment, all models are trained and tested on the I-COCO/O-COCO dataset. The experimental results are shown in Fig. 8 and Fig. 9.

Interestingly, we observe that the deep learning models (i.e. HomographyNet, UnsupervisedNet, Zhang’s et al. and Kugociuk’s et al. models) without the feature matching module obtain significantly different results under the two experimental configurations. This demonstrates that learning the mapping function directly from image features to the homography matrix may lead to the generalization problem. In contrast, SRHEN and our model achieve more stable results under these two experimental configurations. Compared to SRHEN, our model is more robust to illumination and occlusion, as the outlier removal module is able to recover the “correct” clean cost volume in most cases (see Fig. 10).

In the previous sections, we have tested different methods on three synthetic datasets. In order to investigate the generalization ability of these methods to real images, we conduct experiments on the HPatches dataset. Since there are no large-scale real dataset for training the deep learning models, we train all the models on the S-COCO dataset and test them on the HPatches dataset. The experimental results are reported in Fig. 11.

As shown in Fig. 11, our model achieves a distinct advantages on the HPatches dataset among all deep learning models. However, it is inferior to SIFT and SuperPoint. A major reason may be that the distributions of training images and test images are quite different. The images in the HPatches dataset contains a lot of local illumination variations and non-planar regions. These challenging factors are difficult to simulate by synthetic means. We show some success and failure cases of our model in Fig. 12.

Currently, deep learning models do not achieve satisfactory results on the HPatches dataset. However, they have the advantage of being able to generalize to a variety of scenarios when training on a large number of images, as demonstrated by previous experiments on the synthetic datasets. We believe that the performance of our model, as well as other deep learning models, on the HPatches dataset will be further improved if large-scale real dataset are available for training in the future.

In this section we conduct experiments to investigate the effectiveness of each proposed technique in our model. To this end, we perform ablation experiments on four variants of our model:

• FH, which only contains the feature extractor and the homography estimation. It learns the mapping function from image features to the homography matrix.

• FMH, which augments FH with a feature matching module. It learns the mapping function from the cost volume produced by the feature matching module to the homography matrix.

• FMRH-s, which further augments FMH with an outlier removal module. It mimics the traditional homography estimation pipeline in a single deep learning model. It is trained with the training loss in Eq. 9.

• FMRH-ss, has exactly the same network structure as FMRH-s, but is trained with the training loss in Eq. 8.

Table II reports comparison results on the S-COCO dataset between these four variants. FH, without using the feature matching module and the outlier removal module, is similar to HomographyNet with different feature learning structures. Thus, FH performs similarly to HomographyNet on the S-COCO dataset. The performance comparison between FH and FMH manifests the benefit of leveraging the feature matching module to bridge the gap between image features and the homography matrix. Although FMRH-s is further equiped with an outlier removal module based on FMH, the outlier removal module can not be well trained with the supervised loss in Eq. 9. As a consequence, FMRH-s is only slightly better than FMH. The performance is improved marginally from FMRH-s to FMRH-ss, which is benefited from the proposed self-supervised loss for training the outlier removal module.

Besides the network structure and the loss function, in practice, we find that the backbone of the feature extractor and the outlier removal module also has a significant impact on the performance of our model. Early deep learning models usually employ VGG as the backbone of the feature extractor. However, recent studies [36, 15] have shown that replacing VGG with ResNet-34 always results in better performance in the homography estimation task. Therefore, we choose ResNet-34 as the backbone of the feature extractor in our model. Using the ResNet-34, our model achieves a MACE of 0.73 on the S-COCO dataset. While using the VGG, the MACE is 2.36. For fair comparison, in previous experiments, we implemented all deep learning models to use the same ResNet-34 as the backbone of the feature extractor. As for the outlier removal module,we started with a classic deep denoising model DnCNN [37], but later found that this model performed slightly worse than the currently used model UNet.