深度立体声匹配近年来取得了重大进展。然而，最先进的方法基于昂贵的4D成本体积，这限制了它们在现实世界中的应用。要解决此问题，已经提出了3D相关映射和迭代差异更新。关于在现实世界平台中，如自动驾驶汽车和机器人，通常安装LIDAR。因此，我们进一步将稀疏的LIDAR点引入了迭代更新，这减轻了网络更新从零状态的差异的负担。此外，我们提出以自我监督的方式培训网络，以便可以在任何捕获的数据上培训，以获得更好的泛化能力。实验和比较表明，呈现的方法是有效的，并通过相关方法实现了可比的结果。

立体声Vision是一种有效的深度估算技术，具有广泛适用性在自主城市和公路驾驶中。虽然已经为立体声开发了各种基于深度学习的方法，但是具有固定基线的双目设置的输入数据受到限制。解决这样的问题，我们介绍了一个端到端网络，用于处理来自三曲面的数据，这是窄和宽立体对的组合。在这种设计中，用网络的共享权重和中间融合处理两对具有公共参考图像的双目数据。我们还提出了一种用于合并两个基线的4D数据的引导添加方法。此外，介绍了实际和合成数据集的迭代顺序自我监督和监督学习，使三曲系统的训练实用，无需实际数据集的地面真实数据。实验结果表明，三曲差距网络超越了个别对被馈送到类似架构中的场景。代码和数据集：https：//github.com/cogsys-tuebingen/tristeReonet。

Recent work has shown that depth estimation from a stereo pair of images can be formulated as a supervised learning task to be resolved with convolutional neural networks (CNNs). However, current architectures rely on patch-based Siamese networks, lacking the means to exploit context information for finding correspondence in illposed regions. To tackle this problem, we propose PSM-Net, a pyramid stereo matching network consisting of two main modules: spatial pyramid pooling and 3D CNN. The spatial pyramid pooling module takes advantage of the capacity of global context information by aggregating context in different scales and locations to form a cost volume. The 3D CNN learns to regularize cost volume using stacked multiple hourglass networks in conjunction with intermediate supervision. The proposed approach was evaluated on several benchmark datasets. Our method ranked first in the KITTI 2012 and 2015 leaderboards before March 18, 2018. The codes of PSMNet are available at: https: //github.com/JiaRenChang/PSMNet.

深度估计是某些领域的关键技术之一，例如自动驾驶和机器人导航。但是，使用单个传感器的传统方法不可避免地受到传感器的性能的限制。因此，提出了一种融合激光镜头和立体声摄像机的精度和健壮方法。该方法完全结合了LiDAR和立体声摄像机的优势，这些摄像头可以保留LIDAR高精度和图像的高分辨率的优势。与传统的立体声匹配方法相比，对象和照明条件的质地对算法的影响较小。首先，将LIDAR数据的深度转换为立体声摄像机的差异。由于LiDAR数据的密度在Y轴上相对稀疏，因此使用插值方法对转换的差异图进行了更采样。其次，为了充分利用精确的差异图，融合了差异图和立体声匹配以传播准确的差异。最后，将视差图转换为深度图。此外，转换后的差异图还可以提高算法的速度。我们在Kitti基准测试中评估了拟议的管道。该实验表明，我们的算法比几种经典方法具有更高的精度。

Existing deep learning based stereo matching methods either focus on achieving optimal performances on the target dataset while with poor generalization for other datasets or focus on handling the cross-domain generalization by suppressing the domain sensitive features which results in a significant sacrifice on the performance. To tackle these problems, we propose PCW-Net, a Pyramid Combination and Warping cost volume-based network to achieve good performance on both cross-domain generalization and stereo matching accuracy on various benchmarks. In particular, our PCW-Net is designed for two purposes. First, we construct combination volumes on the upper levels of the pyramid and develop a cost volume fusion module to integrate them for initial disparity estimation. Multi-scale receptive fields can be covered by fusing multi-scale combination volumes, thus, domain-invariant features can be extracted. Second, we construct the warping volume at the last level of the pyramid for disparity refinement. The proposed warping volume can narrow down the residue searching range from the initial disparity searching range to a fine-grained one, which can dramatically alleviate the difficulty of the network to find the correct residue in an unconstrained residue searching space. When training on synthetic datasets and generalizing to unseen real datasets, our method shows strong cross-domain generalization and outperforms existing state-of-the-arts with a large margin. After fine-tuning on the real datasets, our method ranks first on KITTI 2012, second on KITTI 2015, and first on the Argoverse among all published methods as of 7, March 2022. The code will be available at https://github.com/gallenszl/PCWNet.

立体声匹配是许多视觉和机器人应用程序的基本构建块。信息性和简洁的成本量表示对于高准确性和效率的立体声匹配至关重要。在本文中，我们提出了一种新颖的成本量构建方法，称为“注意串联量”（ACV），该方法从相关线索中产生了注意力权重，以抑制冗余信息并增强串联体积中与匹配相关的信息。 ACV可以无缝嵌入大多数立体声匹配网络中，所得网络可以使用更轻巧的聚合网络，同时获得更高的精度。我们进一步设计了快速版本的ACV版本以实现实时性能，名为FAST-ACV，它产生了很高的可能性差异假设，以及来自低分辨率相关线索的相应注意力权重，可显着降低计算和记忆成本，同时保持令人满意的精度。我们快速ACV的核心思想是音量注意传播（VAP），它可以自动从上采样相关量中选择准确的相关值，并将这些准确的值传播到周围环境像素具有模棱两可的相关线索。此外，我们分别基于我们的ACV和Fast-ACV设计了高度准确的网络ACVNET和实时网络快速ACVNET，该网络在几个基准上实现了最新性能（即，我们的ACVNET排名第二，第二名在Kitti 2015和场景流以及所有已发布方法中的Kitti 2012和Eth3d的第三次；我们的快速ACVNET几乎优于现场流的所有最新实时方法，Kitti 2012和2015年，与此同时，与此同时更好的概括能力）

立体声匹配是许多视觉和机器人应用程序的基本构建块。信息性和简洁的成本量表示对于高准确性和效率的立体声匹配至关重要。在本文中，我们提出了一种新颖的成本量构建方法，该方法从相关线索中产生了注意力，以抑制冗余信息并增强串联量中与匹配相关的信息。为了产生可靠的注意力权重，我们提出了多层次自适应补丁匹配，以提高在不同差异区域以不同差异的匹配成本的独特性。提出的成本量被命名为注意串联量（ACV），可以将其无缝嵌入大多数立体声匹配网络中，结果网络可以使用更轻巧的聚合网络，同时实现更高的精度，例如。仅使用聚合网络的1/25参数可以实现GWCNET的更高精度。此外，我们根据ACV设计了一个高度准确的网络（ACVNET），该网络（ACVNET）在几个基准上实现了最先进的性能。

时间一致的深度估计对于诸如增强现实之类的实时应用至关重要。虽然立体声深度估计已经接受了显着的注意，导致逐帧的改进，虽然相对较少的工作集中在跨越帧的时间一致性。实际上，基于我们的分析，当前立体声深度估计技术仍然遭受不良时间一致性。由于并发对象和摄像机运动，在动态场景中稳定深度是挑战。在在线设置中，此过程进一步加剧，因为只有过去的帧可用。在本文中，我们介绍了一种技术，在线设置中的动态场景中产生时间一致的深度估计。我们的网络增强了具有新颖运动和融合网络的当前每帧立体声网络。通过预测每个像素SE3变换，运动网络占对象和相机运动。融合网络通过用回归权重聚合当前和先前预测来提高预测的一致性。我们在各种数据集中进行广泛的实验（合成，户外，室内和医疗）。在零射泛化和域微调中，我们证明我们所提出的方法在数量和定性的时间稳定和每个帧精度方面优于竞争方法。我们的代码将在线提供。

立体声匹配是计算机愿景中的一个重要任务，这些任务是几十年来引起了巨大的研究。虽然在差距准确度，密度和数据大小方面，公共立体声数据集难以满足模型的要求。在本文中，我们的目标是解决数据集和模型之间的问题，并提出了一个具有高精度差异地面真理的大规模立体声数据集，名为Plantstereo。我们使用了半自动方式来构造数据集：在相机校准和图像配准后，可以从深度图像获得高精度视差图像。总共有812个图像对覆盖着多种植物套装：菠菜，番茄，胡椒和南瓜。我们首先在四种不同立体声匹配方法中评估了我们的Plandstereo数据集。不同模型和植物的广泛实验表明，与整数精度的基础事实相比，Plantstereo提供的高精度差异图像可以显着提高深度学习模型的培训效果。本文提供了一种可行和可靠的方法来实现植物表面密集的重建。 PlantSereo数据集和相对代码可用于：https://www.github.com/wangqingyu985/plantstereo

深度完成旨在预测从深度传感器（例如Lidars）中捕获的极稀疏图的密集像素深度。它在各种应用中起着至关重要的作用，例如自动驾驶，3D重建，增强现实和机器人导航。基于深度学习的解决方案已经证明了这项任务的最新成功。在本文中，我们首次提供了全面的文献综述，可帮助读者更好地掌握研究趋势并清楚地了解当前的进步。我们通过通过对现有方法进行分类的新型分类法提出建议，研究网络体系结构，损失功能，基准数据集和学习策略的设计方面的相关研究。此外，我们在包括室内和室外数据集（包括室内和室外数据集）上进行了三个广泛使用基准测试的模型性能进行定量比较。最后，我们讨论了先前作品的挑战，并为读者提供一些有关未来研究方向的见解。

最近，立体声匹配基准的记录由端到端视差网络不断破碎。但是，这些深层模型的域适应能力非常有限。解决此类问题，我们提出了一种名为ADASTEREO的新型域自适应方法，该方法旨在对准深度立体声匹配网络的多级表示。与以前的方法相比，我们的ADASTEREO实现了更标准，完整有效的域适应管道。首先，我们提出了一种用于输入图像级对准的非对抗渐进颜色传输算法。其次，我们设计一个有效的无参数成本归一化层，用于内部特征级别对齐。最后，提出了一种高效的辅助任务，自我监督的遮挡感知重建以缩小输出空间中的间隙。我们进行密集的消融研究和分解比较，以验证每个提出的模块的有效性。没有额外推断开销，只有略微增加训练复杂性，我们的Adastereo模型在多个基准上实现了最先进的跨领域性能，包括Kitti，Middrbury，Eth3D和驾驶员，甚至优于一些状态 - 与目标域的地面真相Fineetuned的差异网络。此外，基于两个额外的评估指标，从更多的观点进一步揭示了我们域 - 自适应立体声匹配管道的优越性。最后，我们证明我们的方法对各种域适配设置具有强大，并且可以轻松地集成到快速适应应用方案和现实世界部署中。

虽然卷积神经网络的立体声匹配架构取得了令人印象深刻的成就，但仍有一些限制：1）卷积特征（CF）倾向于捕获外观信息，这对于准确匹配是不充分的。 2）由于静态过滤器，基于电流的卷积差异细化模块通常会产生过平滑的结果。在本文中，我们提出了两种计划来解决这些问题，其中一些传统智慧是整合的。首先，我们为Deep Stereo匹配网络引入了一个成对的特征，命名为LSP（本地相似性模式）。通过显式揭示邻居关系，LSP包含丰富的结构信息，可以利用，以帮助CF以获得更多辨别特征描述。其次，我们设计了动态自我重新组装的细化策略，并分别将其应用于成本分布和视差地图。前者可以配备单峰分布约束来缓解过度平滑的问题，后者更加实用。通过将它们纳入两个众所周知的基本架构，GWCNET和Ganet-Deep，通过将其进行证明，证明了所提出的方法的有效性。 SceneFlow和Kitti基准的实验结果表明，我们的模块显着提高了模型的性能。

Learning based methods have shown very promising results for the task of depth estimation in single images. However, most existing approaches treat depth prediction as a supervised regression problem and as a result, require vast quantities of corresponding ground truth depth data for training. Just recording quality depth data in a range of environments is a challenging problem. In this paper, we innovate beyond existing approaches, replacing the use of explicit depth data during training with easier-to-obtain binocular stereo footage.We propose a novel training objective that enables our convolutional neural network to learn to perform single image depth estimation, despite the absence of ground truth depth data. Exploiting epipolar geometry constraints, we generate disparity images by training our network with an image reconstruction loss. We show that solving for image reconstruction alone results in poor quality depth images. To overcome this problem, we propose a novel training loss that enforces consistency between the disparities produced relative to both the left and right images, leading to improved performance and robustness compared to existing approaches. Our method produces state of the art results for monocular depth estimation on the KITTI driving dataset, even outperforming supervised methods that have been trained with ground truth depth.

3D场景流动表征了当前时间的点如何流到3D欧几里得空间中的下一次，该空间具有自主推断场景中所有对象的非刚性运动的能力。从图像估算场景流的先前方法具有局限性，该方法通过分别估计光流和差异来划分3D场景流的整体性质。学习3D场景从点云流动也面临着综合数据和真实数据与LIDAR点云的稀疏性之间差距的困难。在本文中，利用生成的密集深度图来获得显式的3D坐标，该坐标可直接从2D图像中学习3D场景流。通过将2D像素的密度性质引入3D空间，可以改善预测场景流的稳定性。通过统计方法消除了生成的3D点云中的离群值，以削弱噪声点对3D场景流估计任务的影响。提出了差异一致性损失，以实现3D场景流的更有效的无监督学习。比较了现实世界图像上3D场景流的自我监督学习方法与在综合数据集中学习的多种方法和在LIDAR点云上学习的方法。显示多个场景流量指标的比较可以证明引入伪LIDAR点云到场景流量估计的有效性和优势。

In this paper, we present a learning-based approach for multi-view stereo (MVS), i.e., estimate the depth map of a reference frame using posed multi-view images. Our core idea lies in leveraging a "learning-to-optimize" paradigm to iteratively index a plane-sweeping cost volume and regress the depth map via a convolutional Gated Recurrent Unit (GRU). Since the cost volume plays a paramount role in encoding the multi-view geometry, we aim to improve its construction both in pixel- and frame- levels. In the pixel level, we propose to break the symmetry of the Siamese network (which is typically used in MVS to extract image features) by introducing a transformer block to the reference image (but not to the source images). Such an asymmetric volume allows the network to extract global features from the reference image to predict its depth map. In view of the inaccuracy of poses between reference and source images, we propose to incorporate a residual pose network to make corrections to the relative poses, which essentially rectifies the cost volume in the frame-level. We conduct extensive experiments on real-world MVS datasets and show that our method achieves state-of-the-art performance in terms of both within-dataset evaluation and cross-dataset generalization.

Several leading methods on public benchmarks for depth-from-stereo rely on memory-demanding 4D cost volumes and computationally intensive 3D convolutions for feature matching. We suggest a new way to process the 4D cost volume where we merge two different concepts in one deeply integrated framework to achieve a symbiotic relationship. A feature matching part is responsible for identifying matching pixels pairs along the baseline while a concurrent image volume part is inspired by depth-from-mono CNNs. However, instead of predicting depth directly from image features, it provides additional context to resolve ambiguities during pixel matching. More technically, the processing of the 4D cost volume is separated into a 2D propagation and a 3D propagation part. Starting from feature maps of the left image, the 2D propagation assists the 3D propagation part of the cost volume at different layers by adding visual features to the geometric context. By combining both parts, we can safely reduce the scale of 3D convolution layers in the matching part without sacrificing accuracy. Experiments demonstrate that our end-to-end trained CNN is ranked 2nd on KITTI2012 and ETH3D benchmarks while being significantly faster than the 1st-ranked method. Furthermore, we notice that the coupling of image and matching-volume improves fine-scale details as demonstrated by our qualitative analysis.

We present a unified formulation and model for three motion and 3D perception tasks: optical flow, rectified stereo matching and unrectified stereo depth estimation from posed images. Unlike previous specialized architectures for each specific task, we formulate all three tasks as a unified dense correspondence matching problem, which can be solved with a single model by directly comparing feature similarities. Such a formulation calls for discriminative feature representations, which we achieve using a Transformer, in particular the cross-attention mechanism. We demonstrate that cross-attention enables integration of knowledge from another image via cross-view interactions, which greatly improves the quality of the extracted features. Our unified model naturally enables cross-task transfer since the model architecture and parameters are shared across tasks. We outperform RAFT with our unified model on the challenging Sintel dataset, and our final model that uses a few additional task-specific refinement steps outperforms or compares favorably to recent state-of-the-art methods on 10 popular flow, stereo and depth datasets, while being simpler and more efficient in terms of model design and inference speed.

自我监督的单眼深度估计使机器人能够从原始视频流中学习3D感知。假设世界主要是静态的，这种可扩展的方法利用了投射的几何形状和自我运动来通过视图综合学习。在自主驾驶和人类机器人相互作用中常见的动态场景违反了这一假设。因此，它们需要明确建模动态对象，例如通过估计像素3D运动，即场景流。但是，同时对深度和场景流的自我监督学习是不适合的，因为有许多无限的组合导致相同的3D点。在本文中，我们提出了一种草稿，这是一种通过将合成数据与几何自学意识相结合的新方法，能够共同学习深度，光流和场景流。在木筏架构的基础上，我们将光流作为中间任务，以通过三角剖分来引导深度和场景流量学习。我们的算法还利用任务之间的时间和几何一致性损失来改善多任务学习。我们的草案在标准Kitti基准的自我监督的单眼环境中，同时在所有三个任务中建立了新的最新技术状态。项目页面：https：//sites.google.com/tri.global/draft。

Complete depth information and efficient estimators have become vital ingredients in scene understanding for automated driving tasks. A major problem for LiDAR-based depth completion is the inefficient utilization of convolutions due to the lack of coherent information as provided by the sparse nature of uncorrelated LiDAR point clouds, which often leads to complex and resource-demanding networks. The problem is reinforced by the expensive aquisition of depth data for supervised training. In this work, we propose an efficient depth completion model based on a vgg05-like CNN architecture and propose a semi-supervised domain adaptation approach to transfer knowledge from synthetic to real world data to improve data-efficiency and reduce the need for a large database. In order to boost spatial coherence, we guide the learning process using segmentations as additional source of information. The efficiency and accuracy of our approach is evaluated on the KITTI dataset. Our approach improves on previous efficient and low parameter state of the art approaches while having a noticeably lower computational footprint.

We propose a novel deep learning architecture for regressing disparity from a rectified pair of stereo images. We leverage knowledge of the problem's geometry to form a cost volume using deep feature representations. We learn to incorporate contextual information using 3-D convolutions over this volume. Disparity values are regressed from the cost volume using a proposed differentiable soft argmin operation, which allows us to train our method end-to-end to sub-pixel accuracy without any additional post-processing or regularization. We evaluate our method on the Scene Flow and KITTI datasets and on KITTI we set a new stateof-the-art benchmark, while being significantly faster than competing approaches.