鸟瞰图(BEV)地图已成为现场理解最强大的表达之一,因为他们能够提供丰富的空间上下文,同时容易解释和处理。此类地图已在许多实际任务中发现,广泛地依赖于准确的场景分段以及在BEV空间中的对象实例标识以进行操作。然而,现有的分段算法仅预测BEV空间中的语义,这限制了它们在对象实例概念也是关键的应用中的应用。在这项工作中,给出了前面视图(FV)中的单眼图像,前往直接预测BEV中的密集Panoptic分段图的第一个BEV Panoptic分割方法。我们的架构遵循自上而下的范式,并采用了一种新型密集变压器模块,包括两个不同的变压器,该模块包括从FV到BEV的输入图像中独立地将垂直和平坦区域映射到BEV的不同变压器。另外,我们推导出用于FV-BEV变换的灵敏度的数学制定,其允许我们智能地重量BEV空间中的像素,以考虑在FV图像上的变化描述。关于基提-360和NUSCENES数据集的广泛评估表明,我们的方法分别超过了PQ度量的最先进的3.61 pp和4.93 pp。
translated by 谷歌翻译
点云的Panoptic分割是一种重要的任务,使自动车辆能够使用高精度可靠的激光雷达传感器来理解其附近。现有的自上而下方法通过将独立的任务特定网络或转换方法从图像域转换为忽略激光雷达数据的复杂性,因此通常会导致次优性性能来解决这个问题。在本文中,我们提出了新的自上而下的高效激光乐光线分割(有效的LID)架构,该架构解决了分段激光雷达云中的多种挑战,包括距离依赖性稀疏性,严重的闭塞,大规模变化和重新投影误差。高效地板包括一种新型共享骨干,可以通过加强的几何变换建模容量进行编码,并聚合语义丰富的范围感知多尺度特征。它结合了新的不变语义和实例分段头以及由我们提出的Panoptic外围损耗功能监督的Panoptic Fusion模块。此外,我们制定了正则化的伪标签框架,通过对未标记数据的培训进行进一步提高高效性的性能。我们在两个大型LIDAR数据集中建议模型基准:NUSCENES,我们还提供了地面真相注释和Semantickitti。值得注意的是,高效地将在两个数据集上设置新的最先进状态。
translated by 谷歌翻译
Panoptic现场了解和跟踪动态代理对于机器人和自动化车辆至关重要,以在城市环境中导航。由于LiDAR提供了方案的精确照明和几何描绘,使用LIDAR点云执行这些任务提供可靠的预测。然而,现有数据集缺乏城市场景类型的多样性,并且具有有限数量的动态对象实例,其阻碍了这些任务的学习以及开发方法的可信基准。在本文中,我们介绍了大规模的Panoptic Nuscenes基准数据集,它扩展了我们流行的NUSCENES DataSet,具有用于语义分割,Panoptic分段和Panoptic跟踪任务的Pock-Wise Trountruth annotations。为了便于比较,我们为我们提出的数据集提供了几个任务的强大基线。此外,我们分析了Panoptic跟踪的现有度量标准的缺点,并提出了一种解决问题的小说实例的Pat度量。我们提供详尽的实验,展示了Panoptic Nuscenes与现有数据集相比的效用,并在Nuscenes.org提供的在线评估服务器。我们认为,此扩展将加快新颖的现场了解动态城市环境的新方法研究。
translated by 谷歌翻译
以视觉为中心的BEV感知由于其固有的优点,最近受到行业和学术界的关注,包括展示世界自然代表和融合友好。随着深度学习的快速发展,已经提出了许多方法来解决以视觉为中心的BEV感知。但是,最近没有针对这个小说和不断发展的研究领域的调查。为了刺激其未来的研究,本文对以视觉为中心的BEV感知及其扩展进行了全面调查。它收集并组织了最近的知识,并对常用算法进行了系统的综述和摘要。它还为几项BEV感知任务提供了深入的分析和比较结果,从而促进了未来作品的比较并激发了未来的研究方向。此外,还讨论了经验实现细节并证明有利于相关算法的开发。
translated by 谷歌翻译
在鸟眼中学习强大的表现(BEV),以进行感知任务,这是趋势和吸引行业和学术界的广泛关注。大多数自动驾驶算法的常规方法在正面或透视视图中执行检测,细分,跟踪等。随着传感器配置变得越来越复杂,从不同的传感器中集成了多源信息,并在统一视图中代表功能至关重要。 BEV感知继承了几个优势,因为代表BEV中的周围场景是直观和融合友好的。对于BEV中的代表对象,对于随后的模块,如计划和/或控制是最可取的。 BEV感知的核心问题在于(a)如何通过从透视视图到BEV来通过视图转换来重建丢失的3D信息; (b)如何在BEV网格中获取地面真理注释; (c)如何制定管道以合并来自不同来源和视图的特征; (d)如何适应和概括算法作为传感器配置在不同情况下各不相同。在这项调查中,我们回顾了有关BEV感知的最新工作,并对不同解决方案进行了深入的分析。此外,还描述了该行业的BEV方法的几种系统设计。此外,我们推出了一套完整的实用指南,以提高BEV感知任务的性能,包括相机,激光雷达和融合输入。最后,我们指出了该领域的未来研究指示。我们希望该报告能阐明社区,并鼓励对BEV感知的更多研究。我们保留一个活跃的存储库来收集最新的工作,并在https://github.com/openperceptionx/bevperception-survey-recipe上提供一包技巧的工具箱。
translated by 谷歌翻译
鸟眼视图(BEV)语义分割对于具有强大的空间表示能力的自动驾驶至关重要。由于空间间隙而从单眼图像中估算BEV语义图是一项挑战,因为这是隐含的,以实现均可实现透视到bev-bev的转换和分割。我们提出了一个新型的两阶段几何形状的基于GITNET的基于基于的转换框架,由(i)几何引导的预先对准和(ii)基于射线的变压器组成。在第一阶段,我们将BEV分割分解为透视图的图像分割和基于几何的基于几何映射,并通过将BEV语义标签投影到图像平面上,以明确的监督,以学习可见性吸引的特征和可学习的几何形状,以转化为BEV空间。其次,基于射线的变压器将预先一致的粗细BEV特征进一步变形,以考虑可见性知识。 Gitnet在具有挑战性的Nuscenes和Argoverse数据集上实现了领先的表现。
translated by 谷歌翻译
用于LIDAR点云的快速准确的Panoptic分割系统对于自主驾驶车辆来了解周围物体和场景至关重要。现有方法通常依赖于提案或聚类到分段前景实例。结果,他们努力实现实时性能。在本文中,我们提出了一种用于LIDAR点云的新型实时端到端Panoptic分段网络,称为CPSEG。特别地,CPSEG包括共享编码器,双解码器,任务感知注意模块(TAM)和无簇实例分段头。 TAM旨在强制执行这两个解码器以学习用于语义和实例嵌入的丰富的任务感知功能。此外,CPSEG包含一个新的无簇实例分割头,以根据学习嵌入的嵌入动态占据前景点。然后,它通过找到具有成对嵌入比较的连接的柱子来获取实例标签。因此,将传统的基于提议的或基于聚类的实例分段转换为对成对嵌入比较矩阵的二进制分段问题。为了帮助网络回归实例嵌入,提出了一种快速和确定的深度完成算法,以实时计算每个点云的表面法线。该方法在两个大型自主驾驶数据集中基准测试,即Semantickitti和Nuscenes。值得注意的是,广泛的实验结果表明,CPSEG在两个数据集的实时方法中实现了最先进的结果。
translated by 谷歌翻译
近年来,自主驾驶LIDAR数据的3D对象检测一直在迈出卓越的进展。在最先进的方法中,已经证明了将点云进行编码为鸟瞰图(BEV)是有效且有效的。与透视图不同,BEV在物体之间保留丰富的空间和距离信息;虽然在BEV中相同类型的更远物体不会较小,但它们包含稀疏点云特征。这一事实使用共享卷积神经网络削弱了BEV特征提取。为了解决这一挑战,我们提出了范围感知注意网络(RAANET),提取更强大的BEV功能并产生卓越的3D对象检测。范围感知的注意力(RAA)卷曲显着改善了近距离的特征提取。此外,我们提出了一种新的辅助损耗,用于密度估计,以进一步增强覆盖物体的Raanet的检测精度。值得注意的是,我们提出的RAA卷积轻量级,并兼容,以集成到用于BEV检测的任何CNN架构中。 Nuscenes DataSet上的广泛实验表明,我们的提出方法优于基于LIDAR的3D对象检测的最先进的方法,具有16 Hz的实时推断速度,为LITE版本为22 Hz。该代码在匿名GitHub存储库HTTPS://github.com/Anonymous0522 / ange上公开提供。
translated by 谷歌翻译
我们介绍了MGNET,这是一个多任务框架,用于单眼几何场景。我们将单眼几何场景的理解定义为两个已知任务的组合:全景分割和自我监管的单眼深度估计。全景分段不仅在语义上,而且在实例的基础上捕获完整场景。自我监督的单眼深度估计使用摄像机测量模型得出的几何约束,以便从单眼视频序列中测量深度。据我们所知,我们是第一个在一个模型中提出这两个任务的组合的人。我们的模型专注于低潜伏期,以实时在单个消费级GPU上实时提供快速推断。在部署过程中,我们的模型将产生密集的3D点云,其中具有来自单个高分辨率摄像头图像的实例意识到语义标签。我们对两个流行的自动驾驶基准(即CityScapes and Kitti)评估了模型,并在其他能够实时的方法中表现出竞争性能。源代码可从https://github.com/markusschoen/mgnet获得。
translated by 谷歌翻译
在过去的几年中,自动驾驶的感知系统在其表现方面取得了重大进步。但是,这些系统在极端天气条件下努力表现出稳健性,因为在这些条件下,传感器和相机等传感器套件中的主要传感器都会下降。为了解决此问题,摄像机雷达融合系统为所有可靠的高质量感知提供了独特的机会。相机提供丰富的语义信息,而雷达可以通过遮挡和在所有天气条件下工作。在这项工作中,我们表明,当摄像机输入降解时,最新的融合方法的性能很差,这实际上导致失去了他们设定的全天可靠性。与这些方法相反,我们提出了一种新方法RadSegnet,该方法使用了独立信息提取的新设计理念,并在所有情况下都可以在所有情况下真正实现可靠性,包括遮挡和不利天气。我们在基准ASTYX数据集上开发并验证了我们的系统,并在辐射数据集上进一步验证了这些结果。与最先进的方法相比,Radsegnet在ASTYX上提高了27%,辐射增长了41.46%,平均精度得分,并且在不利天气条件下的性能明显更好
translated by 谷歌翻译
Point cloud learning has lately attracted increasing attention due to its wide applications in many areas, such as computer vision, autonomous driving, and robotics. As a dominating technique in AI, deep learning has been successfully used to solve various 2D vision problems. However, deep learning on point clouds is still in its infancy due to the unique challenges faced by the processing of point clouds with deep neural networks. Recently, deep learning on point clouds has become even thriving, with numerous methods being proposed to address different problems in this area. To stimulate future research, this paper presents a comprehensive review of recent progress in deep learning methods for point clouds. It covers three major tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation. It also presents comparative results on several publicly available datasets, together with insightful observations and inspiring future research directions.
translated by 谷歌翻译
We present AVOD, an Aggregate View Object Detection network for autonomous driving scenarios. The proposed neural network architecture uses LIDAR point clouds and RGB images to generate features that are shared by two subnetworks: a region proposal network (RPN) and a second stage detector network. The proposed RPN uses a novel architecture capable of performing multimodal feature fusion on high resolution feature maps to generate reliable 3D object proposals for multiple object classes in road scenes. Using these proposals, the second stage detection network performs accurate oriented 3D bounding box regression and category classification to predict the extents, orientation, and classification of objects in 3D space. Our proposed architecture is shown to produce state of the art results on the KITTI 3D object detection benchmark [1] while running in real time with a low memory footprint, making it a suitable candidate for deployment on autonomous vehicles. Code is at: https://github.com/kujason/avod
translated by 谷歌翻译
In this paper we propose to exploit multiple related tasks for accurate multi-sensor 3D object detection. Towards this goal we present an end-to-end learnable architecture that reasons about 2D and 3D object detection as well as ground estimation and depth completion. Our experiments show that all these tasks are complementary and help the network learn better representations by fusing information at various levels. Importantly, our approach leads the KITTI benchmark on 2D, 3D and bird's eye view object detection, while being real-time. * Equal contribution.† Work done as part of Uber AI Residency program.
translated by 谷歌翻译
Camera and lidar are important sensor modalities for robotics in general and self-driving cars in particular. The sensors provide complementary information offering an opportunity for tight sensor-fusion. Surprisingly, lidar-only methods outperform fusion methods on the main benchmark datasets, suggesting a gap in the literature. In this work, we propose PointPainting: a sequential fusion method to fill this gap. PointPainting works by projecting lidar points into the output of an image-only semantic segmentation network and appending the class scores to each point. The appended (painted) point cloud can then be fed to any lidaronly method. Experiments show large improvements on three different state-of-the art methods, Point-RCNN, Vox-elNet and PointPillars on the KITTI and nuScenes datasets. The painted version of PointRCNN represents a new state of the art on the KITTI leaderboard for the bird's-eye view detection task. In ablation, we study how the effects of Painting depends on the quality and format of the semantic segmentation output, and demonstrate how latency can be minimized through pipelining.
translated by 谷歌翻译
3D object detection from LiDAR point cloud is a challenging problem in 3D scene understanding and has many practical applications. In this paper, we extend our preliminary work PointRCNN to a novel and strong point-cloud-based 3D object detection framework, the part-aware and aggregation neural network (Part-A 2 net). The whole framework consists of the part-aware stage and the part-aggregation stage. Firstly, the part-aware stage for the first time fully utilizes free-of-charge part supervisions derived from 3D ground-truth boxes to simultaneously predict high quality 3D proposals and accurate intra-object part locations. The predicted intra-object part locations within the same proposal are grouped by our new-designed RoI-aware point cloud pooling module, which results in an effective representation to encode the geometry-specific features of each 3D proposal. Then the part-aggregation stage learns to re-score the box and refine the box location by exploring the spatial relationship of the pooled intra-object part locations. Extensive experiments are conducted to demonstrate the performance improvements from each component of our proposed framework. Our Part-A 2 net outperforms all existing 3D detection methods and achieves new state-of-the-art on KITTI 3D object detection dataset by utilizing only the LiDAR point cloud data. Code is available at https://github.com/sshaoshuai/PointCloudDet3D.
translated by 谷歌翻译
Compared to typical multi-sensor systems, monocular 3D object detection has attracted much attention due to its simple configuration. However, there is still a significant gap between LiDAR-based and monocular-based methods. In this paper, we find that the ill-posed nature of monocular imagery can lead to depth ambiguity. Specifically, objects with different depths can appear with the same bounding boxes and similar visual features in the 2D image. Unfortunately, the network cannot accurately distinguish different depths from such non-discriminative visual features, resulting in unstable depth training. To facilitate depth learning, we propose a simple yet effective plug-and-play module, One Bounding Box Multiple Objects (OBMO). Concretely, we add a set of suitable pseudo labels by shifting the 3D bounding box along the viewing frustum. To constrain the pseudo-3D labels to be reasonable, we carefully design two label scoring strategies to represent their quality. In contrast to the original hard depth labels, such soft pseudo labels with quality scores allow the network to learn a reasonable depth range, boosting training stability and thus improving final performance. Extensive experiments on KITTI and Waymo benchmarks show that our method significantly improves state-of-the-art monocular 3D detectors by a significant margin (The improvements under the moderate setting on KITTI validation set are $\mathbf{1.82\sim 10.91\%}$ mAP in BEV and $\mathbf{1.18\sim 9.36\%}$ mAP in 3D}. Codes have been released at https://github.com/mrsempress/OBMO.
translated by 谷歌翻译
High-definition (HD) semantic map generation of the environment is an essential component of autonomous driving. Existing methods have achieved good performance in this task by fusing different sensor modalities, such as LiDAR and camera. However, current works are based on raw data or network feature-level fusion and only consider short-range HD map generation, limiting their deployment to realistic autonomous driving applications. In this paper, we focus on the task of building the HD maps in both short ranges, i.e., within 30 m, and also predicting long-range HD maps up to 90 m, which is required by downstream path planning and control tasks to improve the smoothness and safety of autonomous driving. To this end, we propose a novel network named SuperFusion, exploiting the fusion of LiDAR and camera data at multiple levels. We benchmark our SuperFusion on the nuScenes dataset and a self-recorded dataset and show that it outperforms the state-of-the-art baseline methods with large margins. Furthermore, we propose a new metric to evaluate the long-range HD map prediction and apply the generated HD map to a downstream path planning task. The results show that by using the long-range HD maps predicted by our method, we can make better path planning for autonomous vehicles. The code will be available at https://github.com/haomo-ai/SuperFusion.
translated by 谷歌翻译
In this paper, we propose a novel 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization. Towards this goal, we design an end-to-end learnable architecture that exploits continuous convolutions to fuse image and LIDAR feature maps at different levels of resolution. Our proposed continuous fusion layer encode both discrete-state image features as well as continuous geometric information. This enables us to design a novel, reliable and efficient end-to-end learnable 3D object detector based on multiple sensors. Our experimental evaluation on both KITTI as well as a large scale 3D object detection benchmark shows significant improvements over the state of the art.
translated by 谷歌翻译
Bird's Eye View(BEV)语义分割在自动驾驶的空间传感中起着至关重要的作用。尽管最近的文献在BEV MAP的理解上取得了重大进展,但它们都是基于基于摄像头的系统,这些系统难以处理遮挡并检测复杂的交通场景中的遥远对象。车辆到车辆(V2V)通信技术使自动驾驶汽车能够共享感应信息,与单代理系统相比,可以显着改善感知性能和范围。在本文中,我们提出了Cobevt,这是可以合作生成BEV MAP预测的第一个通用多代理多机构感知框架。为了有效地从基础变压器体系结构中的多视图和多代理数据融合相机功能,我们设计了融合的轴向注意力或传真模块,可以捕获跨视图和代理的局部和全局空间交互。 V2V感知数据集OPV2V的广泛实验表明,COBEVT实现了合作BEV语义分段的最新性能。此外,COBEVT被证明可以推广到其他任务,包括1)具有单代理多摄像机的BEV分割和2)具有多代理激光雷达系统的3D对象检测,并实现具有实时性能的最新性能时间推理速度。
translated by 谷歌翻译
在这项工作中,我们为基于视觉的不均衡的BEV表示学习提出了PolarBev。为了适应摄像机成像的预先处理效果,我们将BEV空间横向和辐射上栅格化,并引入极性嵌入分解,以模拟极性网格之间的关联。极性网格被重新排列到类似阵列的常规表示,以进行有效处理。此外,为了确定2到3D对应关系,我们根据假设平面迭代更新BEV表面,并采用基于高度的特征转换。PolarBev在单个2080TI GPU上保持实时推理速度,并且在BEV语义分割和BEV实例分割方面都优于其他方法。展示彻底消融以验证设计。该代码将在\ url {https://github.com/superz-liu/polarbev}上发布。
translated by 谷歌翻译