我们提出了深刻的Magsac ++结合了传统和深度强大的估算的优势。我们介绍了一种新的损失功能，可利用部分染色的协调特征，例如筛选，以几何实际合理的方式利用部分染色的协调功能。新损失有助于学习有关底层场景几何的高阶信息。此外，我们为Ransac提出了一个新的采样器，总是选择具有最概率的最高概率的样本。在每次不成功的迭代之后，概率通过贝叶斯方法以原则方式更新。深网络的预测被利用在采样器内部。从新的损失，拟议的采样器和一些技术进步受益，Deep Magsac ++在来自公共可用数据集的成千上万的图像对上的准确性和运行方面都优于最先进的和基本矩阵估计。

We propose the fully differentiable $\nabla$-RANSAC.It predicts the inlier probabilities of the input data points, exploits the predictions in a guided sampler, and estimates the model parameters (e.g., fundamental matrix) and its quality while propagating the gradients through the entire procedure. The random sampler in $\nabla$-RANSAC is based on a clever re-parametrization strategy, i.e.\ the Gumbel Softmax sampler, that allows propagating the gradients directly into the subsequent differentiable minimal solver. The model quality function marginalizes over the scores from all models estimated within $\nabla$-RANSAC to guide the network learning accurate and useful probabilities.$\nabla$-RANSAC is the first to unlock the end-to-end training of geometric estimation pipelines, containing feature detection, matching and RANSAC-like randomized robust estimation. As a proof of its potential, we train $\nabla$-RANSAC together with LoFTR, i.e. a recent detector-free feature matcher, to find reliable correspondences in an end-to-end manner. We test $\nabla$-RANSAC on a number of real-world datasets on fundamental and essential matrix estimation. It is superior to the state-of-the-art in terms of accuracy while being among the fastest methods. The code and trained models will be made public.

提出了一种新的算法来加速RANSAC模型质量计算。该方法基于将联合对应空间（例如，2D-2D点对应）分区为一对常规网格。网格电池被最小的样本模型映射，估计在Ransac内，以拒绝早期与模型参数不一致的对应关系。所提出的技术是一般的。即使将点映射到点集，例如，它也适用于任意变换，例如，作为对末极线的基本矩阵映射。该方法在来自公共可用数据集的数千个图像对上测试了来自基本和基本矩阵，同字和径向扭曲的相同估计。平均而言，它将RANSAC运行时间降低了41％，无需准确性劣化。它可以直接插入最先进的Ransac框架中，例如，vsac。

自RANSAC以来，大量研究一直致力于提高其准确性和运行时间。尽管如此，在完成通常昂贵的模型估计和质量计算之前，只有少数方法旨在识别无效的最小样品。为此，我们提出了NEFSAC，这是一种有效的算法，用于对运动不一致和条件不足的最小样品的神经过滤。我们仅基于图像对应关系的像素坐标来预测最小样品的最小样品的概率。我们的神经滤波模型学习了导致不稳定姿势的样品的典型运动模式，并以可能的动作进行规律性，以偏爱条件良好的样品。新颖的轻量级体系结构实现了最小样本的主要不变性，以进行姿势估计，而新颖的培训方案解决了极端阶级失衡的问题。 NEFSAC可以插入任何现有的基于RANSAC的管道中。我们将其集成到USAC中，并表明即使在极端的火车测试域间隙下，它也会始终如一地提供强大的加速度 - 例如，该模型也训练了用于拍照库的自主驾驶场景。我们从三个公开可用的现实世界数据集中测试了超过100k图像对的NEFSAC，发现它导致了一个数量级的速度，同时通常比单独使用USAC更准确。源代码可从https://github.com/cavalli1234/nefsac获得。

Affine correspondences have traditionally been used to improve feature matching over wide baselines. While recent work has successfully used affine correspondences to solve various relative camera pose estimation problems, less attention has been given to their use in absolute pose estimation. We introduce the first general solution to the problem of estimating the pose of a calibrated camera given a single observation of an oriented point and an affine correspondence. The advantage of our approach (P1AC) is that it requires only a single correspondence, in comparison to the traditional point-based approach (P3P), significantly reducing the combinatorics in robust estimation. P1AC provides a general solution that removes restrictive assumptions made in prior work and is applicable to large-scale image-based localization. We propose two parameterizations of the P1AC problem and evaluate our novel solvers on synthetic data showing their numerical stability and performance under various types of noise. On standard image-based localization benchmarks we show that P1AC achieves more accurate results than the widely used P3P algorithm.

在本文中，我们建议超越建立的基于视觉的本地化方法，该方法依赖于查询图像和3D点云之间的视觉描述符匹配。尽管通过视觉描述符匹配关键点使本地化高度准确，但它具有重大的存储需求，提出了隐私问题，并需要长期对描述符进行更新。为了优雅地应对大规模定位的实用挑战，我们提出了Gomatch，这是基于视觉的匹配的替代方法，仅依靠几何信息来匹配图像键点与地图的匹配，这是轴承矢量集。我们的新型轴承矢量表示3D点，可显着缓解基于几何的匹配中的跨模式挑战，这阻止了先前的工作在现实环境中解决本地化。凭借额外的仔细建筑设计，Gomatch在先前的基于几何的匹配工作中改善了（1067m，95.7升）和（1.43m，34.7摄氏度），平均中位数姿势错误，同时需要7个尺寸，同时需要7片。与最佳基于视觉的匹配方法相比，几乎1.5/1.7％的存储容量。这证实了其对现实世界本地化的潜力和可行性，并为不需要存储视觉描述符的城市规模的视觉定位方法打开了未来努力的大门。

This paper introduces SuperGlue, a neural network that matches two sets of local features by jointly finding correspondences and rejecting non-matchable points. Assignments are estimated by solving a differentiable optimal transport problem, whose costs are predicted by a graph neural network. We introduce a flexible context aggregation mechanism based on attention, enabling SuperGlue to reason about the underlying 3D scene and feature assignments jointly. Compared to traditional, hand-designed heuristics, our technique learns priors over geometric transformations and regularities of the 3D world through end-to-end training from image pairs. SuperGlue outperforms other learned approaches and achieves state-of-the-art results on the task of pose estimation in challenging real-world indoor and outdoor environments. The proposed method performs matching in real-time on a modern GPU and can be readily integrated into modern SfM or SLAM systems. The code and trained weights are publicly available at github.com/magicleap/SuperGluePretrainedNetwork.

Video provides us with the spatio-temporal consistency needed for visual learning. Recent approaches have utilized this signal to learn correspondence estimation from close-by frame pairs. However, by only relying on close-by frame pairs, those approaches miss out on the richer long-range consistency between distant overlapping frames. To address this, we propose a self-supervised approach for correspondence estimation that learns from multiview consistency in short RGB-D video sequences. Our approach combines pairwise correspondence estimation and registration with a novel SE(3) transformation synchronization algorithm. Our key insight is that self-supervised multiview registration allows us to obtain correspondences over longer time frames; increasing both the diversity and difficulty of sampled pairs. We evaluate our approach on indoor scenes for correspondence estimation and RGB-D pointcloud registration and find that we perform on-par with supervised approaches.

This paper proposes a generalizable, end-to-end deep learning-based method for relative pose regression between two images. Given two images of the same scene captured from different viewpoints, our algorithm predicts the relative rotation and translation between the two respective cameras. Despite recent progress in the field, current deep-based methods exhibit only limited generalization to scenes not seen in training. Our approach introduces a network architecture that extracts a grid of coarse features for each input image using the pre-trained LoFTR network. It subsequently relates corresponding features in the two images, and finally uses a convolutional network to recover the relative rotation and translation between the respective cameras. Our experiments indicate that the proposed architecture can generalize to novel scenes, obtaining higher accuracy than existing deep-learning-based methods in various settings and datasets, in particular with limited training data.

我们提出了一个简单的基线，用于直接估计两个图像之间的相对姿势（旋转和翻译，包括比例）。深度方法最近显示出很强的进步，但通常需要复杂或多阶段的体系结构。我们表明，可以将少数修改应用于视觉变压器（VIT），以使其计算接近八点算法。这种归纳偏见使一种简单的方法在多种环境中具有竞争力，通常在有限的数据制度中具有强劲的性能增长，从而实质上有所改善。

部分重叠点云的实时登记具有对自治车辆和多助手SLAM的合作看法的新兴应用。这些应用中点云之间的相对转换高于传统的SLAM和OCOMOTRY应用程序，这挑战了对应的识别和成功的注册。在本文中，我们提出了一种用于部分重叠点云的新颖注册方法，其中使用有效的点亮特征编码器学习对应关系，并使用基于图形的注意网络改进。这种注意网络利用关键点之间的几何关系，以改善点云中的匹配，低重叠。在推断时间下，通过通过样本共识稳健地拟合对应关系来获得相对姿态变换。在基蒂数据集和新的合成数据集上进行评估，包括低重叠点云，位移高达30米。所提出的方法在Kitti DataSet上使用最先进的方法实现了对映射性能，并且优于低重叠点云的现有方法。此外，所提出的方法可以比竞争方法更快地实现更快的推理时间，低至410ms，低至410ms。我们的代码和数据集可在https://github.com/eduardohenriquearnold/fastreg提供。

现代计算机视觉已超越了互联网照片集的领域，并进入了物理世界，通过非结构化的环境引导配备摄像头的机器人和自动驾驶汽车。为了使这些体现的代理与现实世界对象相互作用，相机越来越多地用作深度传感器，重建了各种下游推理任务的环境。机器学习辅助的深度感知或深度估计会预测图像中每个像素的距离。尽管已经在深入估算中取得了令人印象深刻的进步，但仍然存在重大挑战：（1）地面真相深度标签很难大规模收集，（2）通常认为相机信息是已知的，但通常是不可靠的，并且（3）限制性摄像机假设很常见，即使在实践中使用了各种各样的相机类型和镜头。在本论文中，我们专注于放松这些假设，并描述将相机变成真正通用深度传感器的最终目标的贡献。

Estimating 6D poses of objects from images is an important problem in various applications such as robot manipulation and virtual reality. While direct regression of images to object poses has limited accuracy, matching rendered images of an object against the input image can produce accurate results. In this work, we propose a novel deep neural network for 6D pose matching named DeepIM. Given an initial pose estimation, our network is able to iteratively refine the pose by matching the rendered image against the observed image. The network is trained to predict a relative pose transformation using a disentangled representation of 3D location and 3D orientation and an iterative training process. Experiments on two commonly used benchmarks for 6D pose estimation demonstrate that DeepIM achieves large improvements over stateof-the-art methods. We furthermore show that DeepIM is able to match previously unseen objects.

小天体的任务在很大程度上依赖于光学特征跟踪，以表征和相对导航。尽管深度学习导致了功能检测和描述方面的巨大进步，但由于大规模，带注释的数据集的可用性有限，因此培训和验证了空间应用程序的数据驱动模型具有挑战性。本文介绍了Astrovision，这是一个大规模数据集，由115,970个密集注释的，真实的图像组成，这些图像是过去和正在进行的任务中捕获的16个不同物体的真实图像。我们利用Astrovision开发一组标准化基准，并对手工和数据驱动的功能检测和描述方法进行详尽的评估。接下来，我们采用Astrovision对最先进的，深刻的功能检测和描述网络进行端到端培训，并在多个基准测试中表现出改善的性能。将公开使用完整的基准管道和数据集，以促进用于空间应用程序的计算机视觉算法的发展。

关键点匹配是多个图像相关应用的关键组件，例如图像拼接，视觉同时定位和映射（SLAM）等。基于手工制作和最近出现的深度学习的关键点匹配方法仅依赖于关键点和本地功能，同时在上述应用中丢失其他可用传感器（如惯性测量单元（IMU））的视觉。在本文中，我们证明IMU集成的运动估计可用于利用图像之间的关键点之前的空间分布。为此，提出了一种注意力制剂的概率视角，以自然地将空间分布集成到注意力图神经网络中。在空间分布的帮助下，可以减少用于建模隐藏特征的网络的努力。此外，我们为所提出的关键点匹配网络提出了一个投影损耗，它在匹配和未匹配的关键点之间提供了平滑的边缘。图像匹配在Visual Slam数据集上的实验表明了呈现的方法的有效性和效率。

Erroneous feature matches have severe impact on subsequent camera pose estimation and often require additional, time-costly measures, like RANSAC, for outlier rejection. Our method tackles this challenge by addressing feature matching and pose optimization jointly. To this end, we propose a graph attention network to predict image correspondences along with confidence weights. The resulting matches serve as weighted constraints in a differentiable pose estimation. Training feature matching with gradients from pose optimization naturally learns to down-weight outliers and boosts pose estimation on image pairs compared to SuperGlue by 6.7% on ScanNet. At the same time, it reduces the pose estimation time by over 50% and renders RANSAC iterations unnecessary. Moreover, we integrate information from multiple views by spanning the graph across multiple frames to predict the matches all at once. Multi-view matching combined with end-to-end training improves the pose estimation metrics on Matterport3D by 18.8% compared to SuperGlue.

培训和测试监督对象检测模型需要大量带有地面真相标签的图像。标签定义图像中的对象类及其位置，形状以及可能的其他信息，例如姿势。即使存在人力，标签过程也非常耗时。我们引入了一个新的标签工具，用于2D图像以及3D三角网格：3D标记工具（3DLT）。这是一个独立的，功能丰富和跨平台软件，不需要安装，并且可以在Windows，MacOS和基于Linux的发行版上运行。我们不再像当前工具那样在每个图像上分别标记相同的对象，而是使用深度信息从上述图像重建三角形网格，并仅在上述网格上标记一次对象。我们使用注册来简化3D标记，离群值检测来改进2D边界框的计算和表面重建，以将标记可能性扩展到大点云。我们的工具经过最先进的方法测试，并且在保持准确性和易用性的同时，它极大地超过了它们。

We present a method for solving two minimal problems for relative camera pose estimation from three views, which are based on three view correspondences of i) three points and one line and the novel case of ii) three points and two lines through two of the points. These problems are too difficult to be efficiently solved by the state of the art Groebner basis methods. Our method is based on a new efficient homotopy continuation (HC) solver framework MINUS, which dramatically speeds up previous HC solving by specializing HC methods to generic cases of our problems. We characterize their number of solutions and show with simulated experiments that our solvers are numerically robust and stable under image noise, a key contribution given the borderline intractable degree of nonlinearity of trinocular constraints. We show in real experiments that i) SIFT feature location and orientation provide good enough point-and-line correspondences for three-view reconstruction and ii) that we can solve difficult cases with too few or too noisy tentative matches, where the state of the art structure from motion initialization fails.

Incremental Structure-from-Motion is a prevalent strategy for 3D reconstruction from unordered image collections. While incremental reconstruction systems have tremendously advanced in all regards, robustness, accuracy, completeness, and scalability remain the key problems towards building a truly general-purpose pipeline. We propose a new SfM technique that improves upon the state of the art to make a further step towards this ultimate goal. The full reconstruction pipeline is released to the public as an open-source implementation.

在本文中，我们解决了估算图像之间尺度因子的问题。我们制定规模估计问题作为对尺度因素的概率分布的预测。我们设计了一种新的架构，ScaleNet，它利用扩张的卷积以及自我和互相关层来预测图像之间的比例。我们展示了具有估计尺度的整流图像导致各种任务和方法的显着性能改进。具体而言，我们展示了ScaleNet如何与稀疏的本地特征和密集的通信网络组合，以改善不同的基准和数据集中的相机姿势估计，3D重建或密集的几何匹配。我们对多项任务提供了广泛的评估，并分析了标准齿的计算开销。代码，评估协议和培训的型号在https://github.com/axelbarroso/scalenet上公开提供。