ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer

Differing from detector-based matching methods, which typical involve detecting [8, 6, 23, 29], describing [25, 45, 22, 21, 52] and matching [34, 3, 56, 54, 40, 2, 1] a set of keypoints, detector-free matching consumes a pair of images and output correspondences in one shot. Thanks to the removal of keypoint detection stage, detector-free matching is able to capture richer contexts from original images, thus exhibits strong potential to match in extreme situations, such as low texture areas and repetitive patterns.

Despite the potential merits of detector-free matching, its popularity hardly outperforms detector-based methods during early deep learning times due to the intrinsic difficulties in robust and distinctive features. Recently, with the help of deep neural network, possibility is explored to build high performance detector-free matching frameworks based on deep features, which can roughly be classified into two categories: cost volume-based methods [30, 50, 48, 31, 18, 47] and Transformer-based methods [39, 15, 42]. Both kinds of methods leverage strong signals in deep features’ correlation, either in form of correlation layer or cross attention, to guide correspondence search and feature update. Our method follows works on Transformer-based methods and employs multilevel cross attention for mutual feature update, encoding two-view contexts into original features for both global and local consensus.

Balancing receptive field and interaction granularity is a long-standing issue for both cost volume-based and Transformer-based matching. To ensure global receptive field, cost volume based methods are often designed to perform convolution on large global correlation volume, while Transformer-based methods need to conduct attention among all pixel tokens in image pairs. Due to the high cost of global interaction, the input features are usually downsampled into coarse resolution [18, 50, 15] or being projected into low rank [39], which to some degree limits the networks’ capability for fined grained feature update.

Complementary to global interaction, some methods propose to perform local interaction only within a certain region instead of a global field, enabling to process fine level features given a limited computation budget. This practice is especially common in cost volume based methods and are referred to as local correlation layer [50, 14, 48, 43], where the cost volumes/vectors are only constructed around neighbourhood of each correspondence estimation. Intuitively, the idea of complementary global-local interaction can also be introduced to Transformer-based matcher. In our method, a global-local attention block is proposed for message passing across images, ensuring both global receptive field and fine level feature processing. Specially, instead of fixing span for local attention, we design an adaptive mechanism to determine the size of area that each pixel should attend to.

Flow maps depicts correspondence coordinates, which can either be absolute or relative, for each location in an image. Predicting correspondence coordinates from an image pair has been intensively investigated by works in optical flow estimation [10, 14, 55, 43] and general dense image matching [50, 48, 47]. In these works, the flow maps are regressed from structured correlation volumes which are implicitly position-aware. Recently, a Transformer-based method, COTR [15], proves that flows can also be retrieved from positional-embedded features after several turns of attention update.

Naturally, the reliability of flow estimation in each location is not equal and predicting associated confidence scores is essential for many scenarios. As an elegant framework for uncertainty prediction, some works [48, 58, 11, 13, 5] propose to use probabilistic model to jointly explain both flows estimations and their confidence. Inspired by above works, we propose a network that regresses a flow map for each attention block to guide local attention region and adjust the attention span adaptively based on uncertainty prediction.

Before introducing the structure of our network, we first clarify necessary notations and concepts.

As the first part of our network, a convolutional neural network (CNN) is used to extract $1/8$ down-sampled initial features $F_A,F_B\in R^{\frac{H}{8}\times \frac{W}{8}}$ for each image. As is shown in previous works [23, 29, 6, 21, 22, 45, 25, 52], CNN exhibits strong capability to capture local context and generates high-level features, which can be directly used to perform nearest neighbour matching. However, since these features are processed independently for each image and critical cross view contexts are missed. To enrich features with long range and cross view contexts, the initial features are further fed into our proposed Transformer module for updating.

Our Transformer-module starts with a fast initialization block, which conducts (1) positional encoding and (2) two-view contexts initialization.

The basic structure of our Transformer network is global-local attention (GLA) block. As is shown in Figure 2, for each GLA block, attention is performed upon a 3-level coarse-to-fine feature pyramid built by strided average pooling.

For the $i$-th GLA block, global attention is conducted on coarsest downsampled features in resolution $[H_0,W_0]$, while local attention with adaptive span is used to pass message between medium-resolution features in resolution $[\frac{H}{16},\frac{W}{16}]$ and fine level features in resolution $[\frac{H}{8},\frac{W}{8}]$. Note that we keep the coarsest resolution as a constant, making the complexity of global full attention unaffected by input size. Retrieved messages $M^c,M^m,M^f$ from coarse/medium/fine level are upsampled to same $[\frac{H}{8},\frac{W}{8}]$ resolution, concatenated and fused with an MLP to update source features.

The FFN in our network is defined as

LN stands for layer normalization. Specially, we adopt a $3\times 3$ convolution ${\text{Conv}}_3$ in FFN for locality modeling, which compensate for the absence of self attention within each feature map. Empirically, we find $3\times 3$ convolution in FFN works better than the combination of linear projection FFN and self attention, more details can be found in Appendix A.5.

A key problem for local attention is how to define the size of neighbourhood region. A naive approach is to define neighbourhood with a fixed radius $r$ for all pixels, neglecting the fact that the optimal attention span for different regions varies. For instance, it is sufficient to match regions with distinctive features using small contexts, while regions that are harder to match require larger contexts. Instead of using fixed attention span for all pixels, we propose to adaptively adjust the attention span according to the uncertainty of flow estimation. This design lets each area balance their local receptive fields with uncertainty awareness. Regions with high confidence in flow estimation can sharply focus on a small region for fine level matching, while larger contexts are extracted in low confidence areas for better convergence.

Formally, for the $i$-th GLA block, we first regress flow maps ${\varphi }_A^i,{\varphi }_B^i$ from input features $F_A^i,F_B^i$ in fine level with an MLP, while the medium level flow map are obtained by strided average pooling. For each scale level, we partition the corresponding query map $Q$ into cells with size $S\times S$. For each cell, we use the mean flow estimation to generate a rectangle region upon $K,V$ map and uniformly sample a fixed number of tokens. Attention is performed between each cell and the sampled tokens. The detailed process is defined in Algorithm 1. An illustration for local attention is given in Fig. 3. Since number of sampled tokens for each location is fixed, the whole process retains linear complexity.

We inherit the scheme in LoFTR [39] to generate final correspondences, including a coarse matching stage and a sub-pixel refinement stage.

After being updated by $N$ GLA blocks, we flatten the output features into ${\tilde{F}}_A\in R^{n\times c},{\tilde{F}}_B\in R^{m\times c}$ and construct correlation matrix $C=\tau {\tilde{F}}_A{\tilde{F}}_B^T\in R^{n\times m}$, where $\tau$ is a temperature parameter and $n,m$ are feature numbers of two images. By applying dual-direction softmax in both column/row dimensions, a score matrix is given by $S={\text{softmax}}_{row}\left(C\right)\cdot {\text{softmax}}_{col}\left(C\right)$, from which we retain coarse-level matches $M_c$ by mutual nearest neighbour (MNN) and filtering scores below a certain threshold $\theta$. The coarse matches $M_c$ are further fed into a correlation-based refinement block, which is the same with LoFTR [39], to obtain the final matching results.

We formulate the final loss from three parts, (1) coarse matches loss $L_c$, (2) fine-level loss $L_f$ and (3) flow estimation loss $L_{flow}$

For coarse level loss $L_c$, the ground truth matches $M_{gt}$ is determined by reprojection using depth and camera poses in datasets. We supervise the dual-softmax score matrix $S$ with cross entropy loss

The fine-level loss is supervised directly with L2-distance between each refined coordinates $M_f(i,j)$ and ground truth reprojection coordinates, which are further normalized by the coordinate variance.

For flow estimation supervision, we minimize the log-likelihood for each estimated distribution. Formally, given flow estimation $\Phi$ from each layer and ground truth flow $D^{gt}$, $L_{flow}$ is defined as

In our implementation, this log-likelihood formulation can be further substituted and decomposed into a more compact form, which is elaborated in Appendix B.

Our network shares the same ResNet-18 [12] CNN feature extractor with LoFTR. After feature extraction and flow initialization, we use 4 GLA blocks for updating. For adaptive attention span, we set $n=5$, meaning that using 5 standard deviation to crop local neighbourhood region for each token. We uniformly sample $8\times 8$ features in each cropped local region.

We train two different models specified for indoor and outdoor scenes respectively. Both models are optimized using Adam with learning rate $1\times {10}^{-3}$ for 30 epochs on 8 V-100 GPUs. Indoor model is trained on ScanNet [4] dataset with batch size 24, where the training consumes 5 days. Outdoor model is trained on MegaDepth [19] with batch size 8, taking 2 days to converge. More details about implementation are introduced in Appendix A.3.

We resort to two popular datasets, ScanNet [4] and MegaDepth [19], introduced below, to demonstrate the matching ability of our method in indoor scenes and outdoor scenes, respectively. We also provide additional results on YFCC100M [44] and Image Matching Challenge(IMC) 2022 in Appendix C.

Apart from evaluation on two-view pose estimation task, we further integrate our network into a visual localization pipeline, and use two popular datasets, InLoc [41] and Aachen Day-Night v1.1 [57, 36, 35] datasets, to demonstrate performance on multi-view matching in indoor scenes and outdoor scenes, respectively.

To validate the effectiveness of different design components of our method, we conduct ablation experiments on ScanNet dataset [4] following the same setting in Section 4.1. Specifically, we compare three designs of attention structure:

• Single-Level Attn.: A design with only global attention at coarsest feature maps without the need of flow estimation. In this design, global context is well captured, whereas essential locality in motion smoothness is omitted and fine-grained message exchange becomes difficult.

• Multi-Level Attn.: A design with the hierarchical attention framework proposed in this paper, except that the size of local attention span is fixed to 13 px, i.e., the statistical mean of the adaptive attention span used in our network.

• Adaptive Span Attn.: Our full design that enables hierarchical attention with adaptive attention span. By this means, the need of context for different pixels is dynamically decided regarding different matchability.

As presented in Table 6, both hierarchical global-local attention and adaptive attention span improve overall performance by a considerable margin, validating the essentiality of our network designs.

As for uncertainty estimation, we split all pixels into two categories, matchable and unmatchable pixels, identified by ground-truth camera poses and depths, and report their mean standard deviation ($\sigma$). On one hand, mean $\sigma$ decreases with iterations for matchable pixels, as the network becomes more certain about its flow prediction in later stages. On the other hand, the network gradually increases uncertainty values of unmatchable pixels to prevent over-confidence to a certain region.