Single-view depth prediction is a fundamental problem in computer vision. Recently, deep learning methods have led to significant progress, but such methods are limited by the available training data. Current datasets based on 3D sensors have key limitations, including indoor-only images (NYU), small numbers of training examples (Make3D), and sparse sampling (KITTI). We propose to use multi-view Internet photo collections, a virtually unlimited data source, to generate training data via modern structure-from-motion and multi-view stereo (MVS) methods, and present a large depth dataset called MegaDepth based on this idea. Data derived from MVS comes with its own challenges, including noise and unreconstructable objects. We address these challenges with new data cleaning methods, as well as automatically augmenting our data with ordinal depth relations generated using semantic segmentation. We validate the use of large amounts of Internet data by showing that models trained on MegaDepth exhibit strong generalization-not only to novel scenes, but also to other diverse datasets including Make3D, KITTI, and DIW, even when no images from those datasets are seen during training. 1

尽管在过去几年中取得了重大进展，但使用单眼图像进行深度估计仍然存在挑战。首先，训练度量深度预测模型的训练是不算气的，该预测模型可以很好地推广到主要由于训练数据有限的不同场景。因此，研究人员建立了大规模的相对深度数据集，这些数据集更容易收集。但是，由于使用相对深度数据训练引起的深度转移，现有的相对深度估计模型通常无法恢复准确的3D场景形状。我们在此处解决此问题，并尝试通过对大规模相对深度数据进行训练并估算深度转移来估计现场形状。为此，我们提出了一个两阶段的框架，该框架首先将深度预测到未知量表并从单眼图像转移，然后利用3D点云数据来预测深度​​移位和相机的焦距，使我们能够恢复恢复3D场景形状。由于两个模块是单独训练的，因此我们不需要严格配对的培训数据。此外，我们提出了图像级的归一化回归损失和基于正常的几何损失，以通过相对深度注释来改善训练。我们在九个看不见的数据集上测试我们的深度模型，并在零拍摄评估上实现最先进的性能。代码可用：https：//git.io/depth

在简单的数据集中，在简单的数据集中开发和广泛地进行了深度多视图立体声（MVS）方法，在那里他们现在优于经典方法。在本文中，我们询问控制方案中达到的结论是否仍然有效，在使用互联网照片集合时仍然有效。我们提出了一种评估方法，探讨了深度MVS方法的三个方面的影响：网络架构，培训数据和监督。我们进行了几个关键观察，我们广泛地定量和定性地验证，无论是深度预测和完整的3D重建。首先，复杂的无监督方法无法在野外训练数据。我们的新方法使三个关键要素成为可能：上采样输出，基于Softmin的聚合和单一的重建损失。其次，监督基于深度堤map的MVS方法是用于重建几个互联网图像的最新技术。最后，我们的评估提供了比通常的结果非常不同。这表明在不受控制的方案中的评估对于新架构很重要。

The success of monocular depth estimation relies on large and diverse training sets. Due to the challenges associated with acquiring dense ground-truth depth across different environments at scale, a number of datasets with distinct characteristics and biases have emerged. We develop tools that enable mixing multiple datasets during training, even if their annotations are incompatible.In particular, we propose a robust training objective that is invariant to changes in depth range and scale, advocate the use of principled multi-objective learning to combine data from different sources, and highlight the importance of pretraining encoders on auxiliary tasks. Armed with these tools, we experiment with five diverse training datasets, including a new, massive data source: 3D films. To demonstrate the generalization power of our approach we use zero-shot cross-dataset transfer, i.e. we evaluate on datasets that were not seen during training. The experiments confirm that mixing data from complementary sources greatly improves monocular depth estimation. Our approach clearly outperforms competing methods across diverse datasets, setting a new state of the art for monocular depth estimation.

A significant weakness of most current deep Convolutional Neural Networks is the need to train them using vast amounts of manually labelled data. In this work we propose a unsupervised framework to learn a deep convolutional neural network for single view depth prediction, without requiring a pre-training stage or annotated ground-truth depths. We achieve this by training the network in a manner analogous to an autoencoder. At training time we consider a pair of images, source and target, with small, known camera motion between the two such as a stereo pair. We train the convolutional encoder for the task of predicting the depth map for the source image. To do so, we explicitly generate an inverse warp of the target image using the predicted depth and known inter-view displacement, to reconstruct the source image; the photometric error in the reconstruction is the reconstruction loss for the encoder. The acquisition of this training data is considerably simpler than for equivalent systems, requiring no manual annotation, nor calibration of depth sensor to camera. We show that our network trained on less than half of the KITTI dataset gives comparable performance to that of the state-of-the-art supervised methods for single view depth estimation. 1 1 Find the model and other imformation on the project github page: https://github. com/Ravi-Garg/Unsupervised_Depth_Estimation

Self-supervised monocular depth estimation has shown impressive results in static scenes. It relies on the multi-view consistency assumption for training networks, however, that is violated in dynamic object regions and occlusions. Consequently, existing methods show poor accuracy in dynamic scenes, and the estimated depth map is blurred at object boundaries because they are usually occluded in other training views. In this paper, we propose SC-DepthV3 for addressing the challenges. Specifically, we introduce an external pretrained monocular depth estimation model for generating single-image depth prior, namely pseudo-depth, based on which we propose novel losses to boost self-supervised training. As a result, our model can predict sharp and accurate depth maps, even when training from monocular videos of highly-dynamic scenes. We demonstrate the significantly superior performance of our method over previous methods on six challenging datasets, and we provide detailed ablation studies for the proposed terms. Source code and data will be released at https://github.com/JiawangBian/sc_depth_pl

建立新型观点综合的最近进展后，我们提出了改善单眼深度估计的应用。特别是，我们提出了一种在三个主要步骤中分开的新颖训练方法。首先，单眼深度网络的预测结果被扭转到额外的视点。其次，我们应用一个额外的图像综合网络，其纠正并提高了翘曲的RGB图像的质量。通过最小化像素-WISE RGB重建误差，该网络的输出需要尽可能类似地查看地面真实性视图。第三，我们将相同的单眼深度估计重新应用于合成的第二视图点，并确保深度预测与相关的地面真理深度一致。实验结果证明，我们的方法在Kitti和Nyu-Deaft-V2数据集上实现了最先进的或可比性，具有轻量级和简单的香草U-Net架构。

Predicting depth is an essential component in understanding the 3D geometry of a scene. While for stereo images local correspondence suffices for estimation, finding depth relations from a single image is less straightforward, requiring integration of both global and local information from various cues. Moreover, the task is inherently ambiguous, with a large source of uncertainty coming from the overall scale. In this paper, we present a new method that addresses this task by employing two deep network stacks: one that makes a coarse global prediction based on the entire image, and another that refines this prediction locally. We also apply a scale-invariant error to help measure depth relations rather than scale. By leveraging the raw datasets as large sources of training data, our method achieves state-of-the-art results on both NYU Depth and KITTI, and matches detailed depth boundaries without the need for superpixelation.

Learning based methods have shown very promising results for the task of depth estimation in single images. However, most existing approaches treat depth prediction as a supervised regression problem and as a result, require vast quantities of corresponding ground truth depth data for training. Just recording quality depth data in a range of environments is a challenging problem. In this paper, we innovate beyond existing approaches, replacing the use of explicit depth data during training with easier-to-obtain binocular stereo footage.We propose a novel training objective that enables our convolutional neural network to learn to perform single image depth estimation, despite the absence of ground truth depth data. Exploiting epipolar geometry constraints, we generate disparity images by training our network with an image reconstruction loss. We show that solving for image reconstruction alone results in poor quality depth images. To overcome this problem, we propose a novel training loss that enforces consistency between the disparities produced relative to both the left and right images, leading to improved performance and robustness compared to existing approaches. Our method produces state of the art results for monocular depth estimation on the KITTI driving dataset, even outperforming supervised methods that have been trained with ground truth depth.

Per-pixel ground-truth depth data is challenging to acquire at scale. To overcome this limitation, self-supervised learning has emerged as a promising alternative for training models to perform monocular depth estimation. In this paper, we propose a set of improvements, which together result in both quantitatively and qualitatively improved depth maps compared to competing self-supervised methods.Research on self-supervised monocular training usually explores increasingly complex architectures, loss functions, and image formation models, all of which have recently helped to close the gap with fully-supervised methods. We show that a surprisingly simple model, and associated design choices, lead to superior predictions. In particular, we propose (i) a minimum reprojection loss, designed to robustly handle occlusions, (ii) a full-resolution multi-scale sampling method that reduces visual artifacts, and (iii) an auto-masking loss to ignore training pixels that violate camera motion assumptions. We demonstrate the effectiveness of each component in isolation, and show high quality, state-of-the-art results on the KITTI benchmark.

现有的深度完成方法通常以特定的稀疏深度类型为目标，并且在任务域之间概括较差。我们提出了一种方法，可以通过各种范围传感器（包括现代手机中的范围传感器或多视图重建算法）获得稀疏/半密度，嘈杂和潜在的低分辨率深度图。我们的方法利用了在大规模数据集中训练的单个图像深度预测网络的形式的数据驱动的先验，其输出被用作我们模型的输入。我们提出了一个有效的培训计划，我们在典型的任务域中模拟各种稀疏模式。此外，我们设计了两个新的基准测试，以评估深度完成方法的普遍性和鲁棒性。我们的简单方法显示了针对最先进的深度完成方法的优越的跨域泛化能力，从而引入了一种实用的解决方案，以在移动设备上捕获高质量的深度捕获。代码可在以下网址获得：https：//github.com/yvanyin/filldepth。

深度完成旨在预测从深度传感器（例如Lidars）中捕获的极稀疏图的密集像素深度。它在各种应用中起着至关重要的作用，例如自动驾驶，3D重建，增强现实和机器人导航。基于深度学习的解决方案已经证明了这项任务的最新成功。在本文中，我们首次提供了全面的文献综述，可帮助读者更好地掌握研究趋势并清楚地了解当前的进步。我们通过通过对现有方法进行分类的新型分类法提出建议，研究网络体系结构，损失功能，基准数据集和学习策略的设计方面的相关研究。此外，我们在包括室内和室外数据集（包括室内和室外数据集）上进行了三个广泛使用基准测试的模型性能进行定量比较。最后，我们讨论了先前作品的挑战，并为读者提供一些有关未来研究方向的见解。

作为许多自主驾驶和机器人活动的基本组成部分，如自我运动估计，障碍避免和场景理解，单眼深度估计（MDE）引起了计算机视觉和机器人社区的极大关注。在过去的几十年中，已经开发了大量方法。然而，据我们所知，对MDE没有全面调查。本文旨在通过审查1970年至2021年之间发布的197个相关条款来弥补这一差距。特别是，我们为涵盖各种方法的MDE提供了全面的调查，介绍了流行的绩效评估指标并汇总公开的数据集。我们还总结了一些代表方法的可用开源实现，并比较了他们的表演。此外，我们在一些重要的机器人任务中审查了MDE的应用。最后，我们通过展示一些有希望的未来研究方向来结束本文。预计本调查有助于读者浏览该研究领域。

最新的多视图深度估计方法是在深度视频或多视图立体设置中采用的。尽管设置不同，但这些方法在技术上是相似的：它们将多个源视图与关键视图相关联，以估算关键视图的深度图。在这项工作中，我们介绍了强大的多视图深度基准，该基准构建在一组公共数据集上，并允许在两个设置中对来自不同域的数据进行评估。我们评估了最近的方法，并发现跨领域的性能不平衡。此外，我们考虑了第三个设置，可以使用相机姿势，目的是用正确的尺度估算相应的深度图。我们表明，最近的方法不会在这种情况下跨数据集概括。这是因为它们的成本量输出不足。为了解决这一问题，我们介绍了多视图深度估计的强大MVD基线模型，该模型构建在现有组件上，但采用了新颖的规模增强程序。它可以应用于与目标数据无关的强大多视图深度估计。我们在https://github.com/lmb-freiburg/robustmvd上为建议的基准模型提供了代码。

不同的环境对长期自主驾驶的户外强大的视觉感知构成了巨大挑战，以及对不同环境影响的学习算法的概括仍然是一个公开问题。虽然最近单眼深度预测得到了很好的研究，但很少有很多工作，专注于不同环境的强大的基于学习的深度预测，例如，由于缺乏如此多环境的现实世界数据集和基准测试，不断变化照明和季节。为此，基于CMU Visual Location DataSet建立了第一个跨赛季单眼深度预测数据集和基准赛季。为了基准不同环境下的深度估计性能，我们使用几个新配制的指标调查来自Kitti基准的代表性和最近的最先进的开源监督，自我监督和域适应深度预测方法。通过对所提出的数据集进行广泛的实验评估，定性和定量分析了多种环境对性能和鲁棒性的影响，表明即使微调，长期单眼深度预测也仍然具有挑战性。我们进一步提供了承诺的途径，即自我监督的培训和立体声几何约束有助于提高改变环境的鲁棒性。数据集可在https://seasondepth.github.io上找到，并且在https://github.com/seasondepth/seasondepth上提供基准工具包。

本文提出了一个开放而全面的框架，以系统地评估对自我监督单眼估计的最新贡献。这包括训练，骨干，建筑设计选择和损失功能。该领域的许多论文在建筑设计或损失配方中宣称新颖性。但是，简单地更新历史系统的骨干会导致25％的相对改善，从而使其胜过大多数现有系统。对该领域论文的系统评估并不直接。在以前的论文中比较类似于类似的需要，这意味着评估协议中的长期错误在现场无处不在。许多论文可能不仅针对特定数据集进行了优化，而且还针对数据和评估标准的错误。为了帮助该领域的未来研究，我们发布了模块化代码库，可以轻松评估针对校正的数据和评估标准的替代设计决策。我们重新实施，验证和重新评估16个最先进的贡献，并引入一个新的数据集（SYNS-Patches），其中包含各种自然和城市场景中的密集室外深度地图。这允许计算复杂区域（例如深度边界）的信息指标。

现代计算机视觉已超越了互联网照片集的领域，并进入了物理世界，通过非结构化的环境引导配备摄像头的机器人和自动驾驶汽车。为了使这些体现的代理与现实世界对象相互作用，相机越来越多地用作深度传感器，重建了各种下游推理任务的环境。机器学习辅助的深度感知或深度估计会预测图像中每个像素的距离。尽管已经在深入估算中取得了令人印象深刻的进步，但仍然存在重大挑战：（1）地面真相深度标签很难大规模收集，（2）通常认为相机信息是已知的，但通常是不可靠的，并且（3）限制性摄像机假设很常见，即使在实践中使用了各种各样的相机类型和镜头。在本论文中，我们专注于放松这些假设，并描述将相机变成真正通用深度传感器的最终目标的贡献。

Monocular Depth Estimation (MDE) is a fundamental problem in computer vision with numerous applications. Recently, LIDAR-supervised methods have achieved remarkable per-pixel depth accuracy in outdoor scenes. However, significant errors are typically found in the proximity of depth discontinuities, i.e., depth edges, which often hinder the performance of depth-dependent applications that are sensitive to such inaccuracies, e.g., novel view synthesis and augmented reality. Since direct supervision for the location of depth edges is typically unavailable in sparse LIDAR-based scenes, encouraging the MDE model to produce correct depth edges is not straightforward. In this work we propose to learn to detect the location of depth edges from densely-supervised synthetic data, and use it to generate supervision for the depth edges in the MDE training. %Despite the 'domain gap' between synthetic and real data, we show that depth edges that are estimated directly are significantly more accurate than the ones that emerge indirectly from the MDE training. To quantitatively evaluate our approach, and due to the lack of depth edges ground truth in LIDAR-based scenes, we manually annotated subsets of the KITTI and the DDAD datasets with depth edges ground truth. We demonstrate significant gains in the accuracy of the depth edges with comparable per-pixel depth accuracy on several challenging datasets.

We present an unsupervised learning framework for the task of monocular depth and camera motion estimation from unstructured video sequences. In common with recent work [10,14,16], we use an end-to-end learning approach with view synthesis as the supervisory signal. In contrast to the previous work, our method is completely unsupervised, requiring only monocular video sequences for training. Our method uses single-view depth and multiview pose networks, with a loss based on warping nearby views to the target using the computed depth and pose. The networks are thus coupled by the loss during training, but can be applied independently at test time. Empirical evaluation on the KITTI dataset demonstrates the effectiveness of our approach: 1) monocular depth performs comparably with supervised methods that use either ground-truth pose or depth for training, and 2) pose estimation performs favorably compared to established SLAM systems under comparable input settings.

Photometric differences are widely used as supervision signals to train neural networks for estimating depth and camera pose from unlabeled monocular videos. However, this approach is detrimental for model optimization because occlusions and moving objects in a scene violate the underlying static scenario assumption. In addition, pixels in textureless regions or less discriminative pixels hinder model training. To solve these problems, in this paper, we deal with moving objects and occlusions utilizing the difference of the flow fields and depth structure generated by affine transformation and view synthesis, respectively. Secondly, we mitigate the effect of textureless regions on model optimization by measuring differences between features with more semantic and contextual information without adding networks. In addition, although the bidirectionality component is used in each sub-objective function, a pair of images are reasoned about only once, which helps reduce overhead. Extensive experiments and visual analysis demonstrate the effectiveness of the proposed method, which outperform existing state-of-the-art self-supervised methods under the same conditions and without introducing additional auxiliary information.