We propose the fully differentiable $\nabla$-RANSAC.It predicts the inlier probabilities of the input data points, exploits the predictions in a guided sampler, and estimates the model parameters (e.g., fundamental matrix) and its quality while propagating the gradients through the entire procedure. The random sampler in $\nabla$-RANSAC is based on a clever re-parametrization strategy, i.e.\ the Gumbel Softmax sampler, that allows propagating the gradients directly into the subsequent differentiable minimal solver. The model quality function marginalizes over the scores from all models estimated within $\nabla$-RANSAC to guide the network learning accurate and useful probabilities.$\nabla$-RANSAC is the first to unlock the end-to-end training of geometric estimation pipelines, containing feature detection, matching and RANSAC-like randomized robust estimation. As a proof of its potential, we train $\nabla$-RANSAC together with LoFTR, i.e. a recent detector-free feature matcher, to find reliable correspondences in an end-to-end manner. We test $\nabla$-RANSAC on a number of real-world datasets on fundamental and essential matrix estimation. It is superior to the state-of-the-art in terms of accuracy while being among the fastest methods. The code and trained models will be made public.
translated by 谷歌翻译
我们提出了深刻的Magsac ++结合了传统和深度强大的估算的优势。我们介绍了一种新的损失功能,可利用部分染色的协调特征,例如筛选,以几何实际合理的方式利用部分染色的协调功能。新损失有助于学习有关底层场景几何的高阶信息。此外,我们为Ransac提出了一个新的采样器,总是选择具有最概率的最高概率的样本。在每次不成功的迭代之后,概率通过贝叶斯方法以原则方式更新。深网络的预测被利用在采样器内部。从新的损失,拟议的采样器和一些技术进步受益,Deep Magsac ++在来自公共可用数据集的成千上万的图像对上的准确性和运行方面都优于最先进的和基本矩阵估计。
translated by 谷歌翻译
自RANSAC以来,大量研究一直致力于提高其准确性和运行时间。尽管如此,在完成通常昂贵的模型估计和质量计算之前,只有少数方法旨在识别无效的最小样品。为此,我们提出了NEFSAC,这是一种有效的算法,用于对运动不一致和条件不足的最小样品的神经过滤。我们仅基于图像对应关系的像素坐标来预测最小样品的最小样品的概率。我们的神经滤波模型学习了导致不稳定姿势的样品的典型运动模式,并以可能的动作进行规律性,以偏爱条件良好的样品。新颖的轻量级体系结构实现了最小样本的主要不变性,以进行姿势估计,而新颖的培训方案解决了极端阶级失衡的问题。 NEFSAC可以插入任何现有的基于RANSAC的管道中。我们将其集成到USAC中,并表明即使在极端的火车测试域间隙下,它也会始终如一地提供强大的加速度 - 例如,该模型也训练了用于拍照库的自主驾驶场景。我们从三个公开可用的现实世界数据集中测试了超过100k图像对的NEFSAC,发现它导致了一个数量级的速度,同时通常比单独使用USAC更准确。源代码可从https://github.com/cavalli1234/nefsac获得。
translated by 谷歌翻译
Affine correspondences have traditionally been used to improve feature matching over wide baselines. While recent work has successfully used affine correspondences to solve various relative camera pose estimation problems, less attention has been given to their use in absolute pose estimation. We introduce the first general solution to the problem of estimating the pose of a calibrated camera given a single observation of an oriented point and an affine correspondence. The advantage of our approach (P1AC) is that it requires only a single correspondence, in comparison to the traditional point-based approach (P3P), significantly reducing the combinatorics in robust estimation. P1AC provides a general solution that removes restrictive assumptions made in prior work and is applicable to large-scale image-based localization. We propose two parameterizations of the P1AC problem and evaluate our novel solvers on synthetic data showing their numerical stability and performance under various types of noise. On standard image-based localization benchmarks we show that P1AC achieves more accurate results than the widely used P3P algorithm.
translated by 谷歌翻译
This paper introduces SuperGlue, a neural network that matches two sets of local features by jointly finding correspondences and rejecting non-matchable points. Assignments are estimated by solving a differentiable optimal transport problem, whose costs are predicted by a graph neural network. We introduce a flexible context aggregation mechanism based on attention, enabling SuperGlue to reason about the underlying 3D scene and feature assignments jointly. Compared to traditional, hand-designed heuristics, our technique learns priors over geometric transformations and regularities of the 3D world through end-to-end training from image pairs. SuperGlue outperforms other learned approaches and achieves state-of-the-art results on the task of pose estimation in challenging real-world indoor and outdoor environments. The proposed method performs matching in real-time on a modern GPU and can be readily integrated into modern SfM or SLAM systems. The code and trained weights are publicly available at github.com/magicleap/SuperGluePretrainedNetwork.
translated by 谷歌翻译
提出了一种新的算法来加速RANSAC模型质量计算。该方法基于将联合对应空间(例如,2D-2D点对应)分区为一对常规网格。网格电池被最小的样本模型映射,估计在Ransac内,以拒绝早期与模型参数不一致的对应关系。所提出的技术是一般的。即使将点映射到点集,例如,它也适用于任意变换,例如,作为对末极线的基本矩阵映射。该方法在来自公共可用数据集的数千个图像对上测试了来自基本和基本矩阵,同字和径向扭曲的相同估计。平均而言,它将RANSAC运行时间降低了41%,无需准确性劣化。它可以直接插入最先进的Ransac框架中,例如,vsac。
translated by 谷歌翻译
Erroneous feature matches have severe impact on subsequent camera pose estimation and often require additional, time-costly measures, like RANSAC, for outlier rejection. Our method tackles this challenge by addressing feature matching and pose optimization jointly. To this end, we propose a graph attention network to predict image correspondences along with confidence weights. The resulting matches serve as weighted constraints in a differentiable pose estimation. Training feature matching with gradients from pose optimization naturally learns to down-weight outliers and boosts pose estimation on image pairs compared to SuperGlue by 6.7% on ScanNet. At the same time, it reduces the pose estimation time by over 50% and renders RANSAC iterations unnecessary. Moreover, we integrate information from multiple views by spanning the graph across multiple frames to predict the matches all at once. Multi-view matching combined with end-to-end training improves the pose estimation metrics on Matterport3D by 18.8% compared to SuperGlue.
translated by 谷歌翻译
在本文中,我们建议超越建立的基于视觉的本地化方法,该方法依赖于查询图像和3D点云之间的视觉描述符匹配。尽管通过视觉描述符匹配关键点使本地化高度准确,但它具有重大的存储需求,提出了隐私问题,并需要长期对描述符进行更新。为了优雅地应对大规模定位的实用挑战,我们提出了Gomatch,这是基于视觉的匹配的替代方法,仅依靠几何信息来匹配图像键点与地图的匹配,这是轴承矢量集。我们的新型轴承矢量表示3D点,可显着缓解基于几何的匹配中的跨模式挑战,这阻止了先前的工作在现实环境中解决本地化。凭借额外的仔细建筑设计,Gomatch在先前的基于几何的匹配工作中改善了(1067m,95.7升)和(1.43m,34.7摄氏度),平均中位数姿势错误,同时需要7个尺寸,同时需要7片。与最佳基于视觉的匹配方法相比,几乎1.5/1.7%的存储容量。这证实了其对现实世界本地化的潜力和可行性,并为不需要存储视觉描述符的城市规模的视觉定位方法打开了未来努力的大门。
translated by 谷歌翻译
We present a method for solving two minimal problems for relative camera pose estimation from three views, which are based on three view correspondences of i) three points and one line and the novel case of ii) three points and two lines through two of the points. These problems are too difficult to be efficiently solved by the state of the art Groebner basis methods. Our method is based on a new efficient homotopy continuation (HC) solver framework MINUS, which dramatically speeds up previous HC solving by specializing HC methods to generic cases of our problems. We characterize their number of solutions and show with simulated experiments that our solvers are numerically robust and stable under image noise, a key contribution given the borderline intractable degree of nonlinearity of trinocular constraints. We show in real experiments that i) SIFT feature location and orientation provide good enough point-and-line correspondences for three-view reconstruction and ii) that we can solve difficult cases with too few or too noisy tentative matches, where the state of the art structure from motion initialization fails.
translated by 谷歌翻译
通过Perspective-N点(PNP)从单个RGB图像找到3D对象是计算机视觉中的长期问题。在端到端的深度学习的驱动下,最近的研究表明将PNP解释为一个可区分的层,因此可以通过反向传播梯度W.R.T.可以部分学习2d-3d点对应。对象姿势。然而,由于确定性姿势本质上是非差异的,因此学习整个不受限制的2D-3D点无法与现有方法融合。在本文中,我们提出了EPRO-PNP,这是用于一般端到端姿势估计的概率PNP层,该阶段估计输出了SE(3)歧管上的姿势分布,从本质上讲,将分类软效量带到连续域。 2d-3d坐标和相应的权重被视为通过最大程度地减少预测姿势分布和目标姿势分布之间的KL差异来学习的中间变量。基本原则统一了现有方法并类似于注意机制。 EPRO-PNP显着胜过竞争基线,缩小基于PNP的方法与LineMod 6DOF姿势估计和NUSCENES 3D对象检测基准的差距。
translated by 谷歌翻译
This paper studies the challenging two-view 3D reconstruction in a rigorous sparse-view configuration, which is suffering from insufficient correspondences in the input image pairs for camera pose estimation. We present a novel Neural One-PlanE RANSAC framework (termed NOPE-SAC in short) that exerts excellent capability to learn one-plane pose hypotheses from 3D plane correspondences. Building on the top of a siamese plane detection network, our NOPE-SAC first generates putative plane correspondences with a coarse initial pose. It then feeds the learned 3D plane parameters of correspondences into shared MLPs to estimate the one-plane camera pose hypotheses, which are subsequently reweighed in a RANSAC manner to obtain the final camera pose. Because the neural one-plane pose minimizes the number of plane correspondences for adaptive pose hypotheses generation, it enables stable pose voting and reliable pose refinement in a few plane correspondences for the sparse-view inputs. In the experiments, we demonstrate that our NOPE-SAC significantly improves the camera pose estimation for the two-view inputs with severe viewpoint changes, setting several new state-of-the-art performances on two challenging benchmarks, i.e., MatterPort3D and ScanNet, for sparse-view 3D reconstruction. The source code is released at https://github.com/IceTTTb/NopeSAC for reproducible research.
translated by 谷歌翻译
我们提出了算法贡献,可提高在异常值影响的几何回归问题中稳健的修剪效率。该方法在很大程度上依赖于快速排序算法,我们提出了两个重要的见解。首先,部分排序足以进行x-TheThepile值的增量计算。其次,线性拟合问题中的正常方程可能会通过在排序过程中跨x-TheStile边界上记录交换操作来逐渐更新。除了线性拟合问题外,我们还展示了如何将技术额外应用于封闭形式的非线性能量最小化问题,从而在几何最佳目标下实现有效的修剪拟合。我们将方法应用于两种不同的摄像机切除算法,并展示了高效和可靠的几何修剪拟合。
translated by 谷歌翻译
Camera relocalization involving a prior 3D reconstruction plays a crucial role in many mixed reality and robotics applications. Estimating the camera pose directly with respect to pre-built 3D models can be prohibitively expensive for several applications with limited storage and/or communication bandwidth. Although recent scene and absolute pose regression methods have become popular for efficient camera localization, most of them are computation-resource intensive and difficult to obtain a real-time inference with high accuracy constraints. This study proposes a simple scene regression method that requires only a multi-layer perceptron network for mapping scene coordinates to achieve accurate camera pose estimations. The proposed approach uses sparse descriptors to regress the scene coordinates, instead of a dense RGB image. The use of sparse features provides several advantages. First, the proposed regressor network is substantially smaller than those reported in previous studies. This makes our system highly efficient and scalable. Second, the pre-built 3D models provide the most reliable and robust 2D-3D matches. Therefore, learning from them can lead to an awareness of equivalent features and substantially improve the generalization performance. A detailed analysis of our approach and extensive evaluations using existing datasets are provided to support the proposed method. The implementation detail is available at https://github.com/aislab/feat2map
translated by 谷歌翻译
Video provides us with the spatio-temporal consistency needed for visual learning. Recent approaches have utilized this signal to learn correspondence estimation from close-by frame pairs. However, by only relying on close-by frame pairs, those approaches miss out on the richer long-range consistency between distant overlapping frames. To address this, we propose a self-supervised approach for correspondence estimation that learns from multiview consistency in short RGB-D video sequences. Our approach combines pairwise correspondence estimation and registration with a novel SE(3) transformation synchronization algorithm. Our key insight is that self-supervised multiview registration allows us to obtain correspondences over longer time frames; increasing both the diversity and difficulty of sampled pairs. We evaluate our approach on indoor scenes for correspondence estimation and RGB-D pointcloud registration and find that we perform on-par with supervised approaches.
translated by 谷歌翻译
准确的相机姿势估计是许多应用程序(例如自动驾驶,移动机器人技术和增强现实)的基本要求。在这项工作中,我们解决了在给定环境中从单个RGB图像中估算全局6 DOF摄像头的问题。以前的作品考虑图像的每个部分都有价值对于本地化。但是,许多图像区域,例如天空,遮挡和重复的非固定模式,不能用于本地化。除了添加不必要的计算工作外,从此类地区提取和匹配功能还会产生许多错误的匹配,从而降低了本地化准确性和效率。我们的工作解决了这一特定问题,并通过利用有趣的3D模型的有趣概念来显示,我们可以利用歧视性环境零件并避免出于单个图像本地化而避免无用的图像区域。有趣的是,通过避免从树木,灌木丛,汽车,行人和遮挡等不可靠的图像区域中选择关键点,我们的工作自然而然地作为离群过滤器。这使我们的系统高效,在最小的对应关系中,由于异常值的数量很少,因此需要高度准确。我们的工作超过了室外剑桥地标数据集的最新方法。仅在推理上依靠单个图像,它的精度方法超过了构成姿势先验和/或参考3D模型的精度方法,同时更快。通过选择仅100个对应关系,它超过了从数千个对应关系进行定位的类似方法,同时更有效。特别是,与这些方法相比,它实现了,在旧院面场景中,本地化提高了33%。此外,它甚至可以从图像顺序中学习的直接姿势回归器
translated by 谷歌翻译
相对摄像头姿势估计,即使用在不同位置拍摄的一对图像来估算翻译和旋转向量,是增强现实和机器人技术系统中系统的重要组成部分。在本文中,我们使用独立于摄像机参数的暹罗体系结构提出了端到端的相对摄像头姿势估计网络。使用剑桥地标数据和四个单独的场景数据集和一个结合四个场景的数据集对网络进行培训。为了改善概括,我们提出了一种新颖的两阶段训练,以减轻超参数以平衡翻译和旋转损失量表的需求。将提出的方法与基于CNN的一阶段培训方法(例如RPNET和RCPNET)进行了比较,并证明了所提出的模型在Kings College,Old Hospital和St Marys上提出的翻译量估计提高了16.11%,28.88%和52.27%教堂场景分别。为了证明纹理不变性,我们使用生成的对抗网络研究了提出的方法的概括,将数据集扩展到不同场景样式,作为消融研究。此外,我们对网络预测和地面真相构成的异性线进行定性评估。
translated by 谷歌翻译
3D point cloud registration is a fundamental problem in computer vision and robotics. Recently, learning-based point cloud registration methods have made great progress. However, these methods are sensitive to outliers, which lead to more incorrect correspondences. In this paper, we propose a novel deep graph matching-based framework for point cloud registration. Specifically, we first transform point clouds into graphs and extract deep features for each point. Then, we develop a module based on deep graph matching to calculate a soft correspondence matrix. By using graph matching, not only the local geometry of each point but also its structure and topology in a larger range are considered in establishing correspondences, so that more correct correspondences are found. We train the network with a loss directly defined on the correspondences, and in the test stage the soft correspondences are transformed into hard one-to-one correspondences so that registration can be performed by a correspondence-based solver. Furthermore, we introduce a transformer-based method to generate edges for graph construction, which further improves the quality of the correspondences. Extensive experiments on object-level and scene-level benchmark datasets show that the proposed method achieves state-of-the-art performance. The code is available at: \href{https://github.com/fukexue/RGM}{https://github.com/fukexue/RGM}.
translated by 谷歌翻译
在本文中,我们解决了估算图像之间尺度因子的问题。我们制定规模估计问题作为对尺度因素的概率分布的预测。我们设计了一种新的架构,ScaleNet,它利用扩张的卷积以及自我和互相关层来预测图像之间的比例。我们展示了具有估计尺度的整流图像导致各种任务和方法的显着性能改进。具体而言,我们展示了ScaleNet如何与稀疏的本地特征和密集的通信网络组合,以改善不同的基准和数据集中的相机姿势估计,3D重建或密集的几何匹配。我们对多项任务提供了广泛的评估,并分析了标准齿的计算开销。代码,评估协议和培训的型号在https://github.com/axelbarroso/scalenet上公开提供。
translated by 谷歌翻译
This paper proposes a generalizable, end-to-end deep learning-based method for relative pose regression between two images. Given two images of the same scene captured from different viewpoints, our algorithm predicts the relative rotation and translation between the two respective cameras. Despite recent progress in the field, current deep-based methods exhibit only limited generalization to scenes not seen in training. Our approach introduces a network architecture that extracts a grid of coarse features for each input image using the pre-trained LoFTR network. It subsequently relates corresponding features in the two images, and finally uses a convolutional network to recover the relative rotation and translation between the respective cameras. Our experiments indicate that the proposed architecture can generalize to novel scenes, obtaining higher accuracy than existing deep-learning-based methods in various settings and datasets, in particular with limited training data.
translated by 谷歌翻译
3D点云登记在遥感,摄影测量,机器人和几何计算机视觉中排名最基本的问题。由于3D特征匹配技术的准确性有限,因此可能存在异常值,有时即使在非常大的数字中,则在该对应中也是如此。由于现有的强大的求解器可能会遇到高计算成本或限制性的稳健性,因此我们提出了一种名为VoCra(具有成本函数和旋转平均的投票的新颖,快速,高度强大的解决方案,为极端异常率的点云注册问题。我们的第一款贡献是聘请Tukey的双重强大的成本来引入新的投票和对应分类技术,这证明是在异常值中区分真正的入世性,即使是极端(99%)的异常率。我们的第二次贡献包括基于强大的旋转平均设计时效的共识最大化范例,用于在通信中寻求Inlier候选人。最后,我们使用Tukey的Biweight(GNC-TB)应用毕业的非凸性,以估计所获得的Inlier候选者的正确变换,然后使用它来找到完整的Inlier集。进行了应用于两个实体数据问题的标准基准和现实实验,并且我们表明我们的求解器VORCA对超过99%的异常值较高,而且比最先进的竞争对手更多的时间效率。
translated by 谷歌翻译