合作感允许连接的自动驾驶汽车(CAV)与附近的其他骑士相互作用,以增强对周围物体的感知以提高安全性和可靠性。它可以弥补常规车辆感知的局限性,例如盲点,低分辨率和天气影响。合作感知中间融合方法的有效特征融合模型可以改善特征选择和信息聚集,以进一步提高感知精度。我们建议具有可训练的特征选择模块的自适应特征融合模型。我们提出的模型之一是通过空间自适应特征融合(S-Adafusion)在OPV2V数据集的两个子集上的所有其他最先进的模型:默认的Carla Towns用于车辆检测和用于域适应的Culver City。此外,先前的研究仅测试了合作感的车辆检测。但是,行人在交通事故中更有可能受到重伤。我们使用CODD数据集评估了车辆和行人检测的合作感的性能。与CODD数据集中的车辆和行人检测相比,我们的架构达到的平均精度(AP)高。实验表明,与常规感知过程相比,合作感也可以提高行人检测准确性。
translated by 谷歌翻译
Deep learning has been widely used in the perception (e.g., 3D object detection) of intelligent vehicle driving. Due to the beneficial Vehicle-to-Vehicle (V2V) communication, the deep learning based features from other agents can be shared to the ego vehicle so as to improve the perception of the ego vehicle. It is named as Cooperative Perception in the V2V research, whose algorithms have been dramatically advanced recently. However, all the existing cooperative perception algorithms assume the ideal V2V communication without considering the possible lossy shared features because of the Lossy Communication (LC) which is common in the complex real-world driving scenarios. In this paper, we first study the side effect (e.g., detection performance drop) by the lossy communication in the V2V Cooperative Perception, and then we propose a novel intermediate LC-aware feature fusion method to relieve the side effect of lossy communication by a LC-aware Repair Network (LCRN) and enhance the interaction between the ego vehicle and other vehicles by a specially designed V2V Attention Module (V2VAM) including intra-vehicle attention of ego vehicle and uncertainty-aware inter-vehicle attention. The extensive experiment on the public cooperative perception dataset OPV2V (based on digital-twin CARLA simulator) demonstrates that the proposed method is quite effective for the cooperative point cloud based 3D object detection under lossy V2V communication.
translated by 谷歌翻译
采用车辆到车辆通信以提高自动驾驶技术中的感知性能,最近引起了相当大的关注;然而,对于基准测试算法的合适开放数据集已经难以开发和评估合作感知技术。为此,我们介绍了用于车辆到车辆的第一个大型开放模拟数据集。它包含超过70个有趣的场景,11,464帧和232,913帧的注释3D车辆边界盒,从卡拉的8个城镇和洛杉矶的数码镇。然后,我们构建了一个全面的基准,共有16种实施模型来评估若干信息融合策略〜(即早期,晚期和中间融合),最先进的激光雷达检测算法。此外,我们提出了一种新的细心中间融合管线,以从多个连接的车辆汇总信息。我们的实验表明,拟议的管道可以很容易地与现有的3D LIDAR探测器集成,即使具有大的压缩速率也可以实现出色的性能。为了鼓励更多的研究人员来调查车辆到车辆的感知,我们将释放数据集,基准方法以及HTTPS://mobility-lab.seas.ucla.edu/opv2v2v/中的所有相关代码。
translated by 谷歌翻译
Utilizing the latest advances in Artificial Intelligence (AI), the computer vision community is now witnessing an unprecedented evolution in all kinds of perception tasks, particularly in object detection. Based on multiple spatially separated perception nodes, Cooperative Perception (CP) has emerged to significantly advance the perception of automated driving. However, current cooperative object detection methods mainly focus on ego-vehicle efficiency without considering the practical issues of system-wide costs. In this paper, we introduce VINet, a unified deep learning-based CP network for scalable, lightweight, and heterogeneous cooperative 3D object detection. VINet is the first CP method designed from the standpoint of large-scale system-level implementation and can be divided into three main phases: 1) Global Pre-Processing and Lightweight Feature Extraction which prepare the data into global style and extract features for cooperation in a lightweight manner; 2) Two-Stream Fusion which fuses the features from scalable and heterogeneous perception nodes; and 3) Central Feature Backbone and 3D Detection Head which further process the fused features and generate cooperative detection results. A cooperative perception platform is designed and developed for CP dataset acquisition and several baselines are compared during the experiments. The experimental analysis shows that VINet can achieve remarkable improvements for pedestrians and cars with 2x less system-wide computational costs and 12x less system-wide communicational costs.
translated by 谷歌翻译
近年来,自主驾驶LIDAR数据的3D对象检测一直在迈出卓越的进展。在最先进的方法中,已经证明了将点云进行编码为鸟瞰图(BEV)是有效且有效的。与透视图不同,BEV在物体之间保留丰富的空间和距离信息;虽然在BEV中相同类型的更远物体不会较小,但它们包含稀疏点云特征。这一事实使用共享卷积神经网络削弱了BEV特征提取。为了解决这一挑战,我们提出了范围感知注意网络(RAANET),提取更强大的BEV功能并产生卓越的3D对象检测。范围感知的注意力(RAA)卷曲显着改善了近距离的特征提取。此外,我们提出了一种新的辅助损耗,用于密度估计,以进一步增强覆盖物体的Raanet的检测精度。值得注意的是,我们提出的RAA卷积轻量级,并兼容,以集成到用于BEV检测的任何CNN架构中。 Nuscenes DataSet上的广泛实验表明,我们的提出方法优于基于LIDAR的3D对象检测的最先进的方法,具有16 Hz的实时推断速度,为LITE版本为22 Hz。该代码在匿名GitHub存储库HTTPS://github.com/Anonymous0522 / ange上公开提供。
translated by 谷歌翻译
我们呈现PIFENET,一种高效准确的实时3D探测器,用于点云的行人检测。我们解决了在检测行人时遇到的3D对象检测框架的两个挑战:Partion云中的柱特征的表达力量和小型行人的小占领区。首先,我们引入了一个可堆叠的柱子感知注意力(PAA)模块,用于增强的柱子特征提取,同时抑制点云中的噪声。通过将多点感知池,点亮,通道和任务感知注意与到一个简单的模块集成到一个简单的模块,在需要几乎额外的计算资源的同时提高表示功能。我们还存在Mini-Bifpn,一个小而有效的特征网络,创建双向信息流和多级串尺度特征融合,以更好地集成多分辨率功能。我们的方法在Kitti Peistrian Bev和3D排行榜中排名第一,同时以每秒26帧(FPS)运行,并在Nuscenes检测基准上实现最先进的性能。
translated by 谷歌翻译
在本文中,我们调查了车辆到所有(V2X)通信的应用,以提高自动驾驶汽车的感知性能。我们使用新型视觉变压器提供了一个与V2X通信的强大合作感知框架。具体而言,我们建立了一个整体关注模型,即V2X-VIT,以有效地融合跨道路代理(即车辆和基础设施)的信息。 V2X-VIT由异质多代理自我注意和多尺度窗口自我注意的交替层组成,该层捕获了代理间的相互作用和全面的空间关系。这些关键模块在统一的变压器体系结构中设计,以应对常见的V2X挑战,包括异步信息共享,姿势错误和V2X组件的异质性。为了验证我们的方法,我们使用Carla和OpenCDA创建了一个大规模的V2X感知数据集。广泛的实验结果表明,V2X-VIT设置了3D对象检测的新最先进的性能,即使在恶劣的嘈杂环境下,也可以实现强大的性能。该代码可在https://github.com/derrickxunu/v2x-vit上获得。
translated by 谷歌翻译
In this paper, we propose a novel 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization. Towards this goal, we design an end-to-end learnable architecture that exploits continuous convolutions to fuse image and LIDAR feature maps at different levels of resolution. Our proposed continuous fusion layer encode both discrete-state image features as well as continuous geometric information. This enables us to design a novel, reliable and efficient end-to-end learnable 3D object detector based on multiple sensors. Our experimental evaluation on both KITTI as well as a large scale 3D object detection benchmark shows significant improvements over the state of the art.
translated by 谷歌翻译
我们提出了DeepFusion,这是一种模块化的多模式结构,可在不同组合中以3D对象检测为融合激光雷达,相机和雷达。专门的功能提取器可以利用每种模式,并且可以轻松交换,从而使该方法变得简单而灵活。提取的特征被转化为鸟眼视图,作为融合的共同表示。在特征空间中融合方式之前,先进行空间和语义对齐。最后,检测头利用丰富的多模式特征,以改善3D检测性能。 LIDAR相机,激光摄像头雷达和摄像头融合的实验结果显示了我们融合方法的灵活性和有效性。在此过程中,我们研究了高达225米的遥远汽车检测的很大程度上未开发的任务,显示了激光摄像机融合的好处。此外,我们研究了3D对象检测的LIDAR点所需的密度,并在对不利天气条件的鲁棒性示例中说明了含义。此外,对我们的摄像头融合的消融研究突出了准确深度估计的重要性。
translated by 谷歌翻译
这项工作旨在通过使用路边激光射击环境的3D感知来应对自动驾驶的挑战。我们设计了一个3D对象检测模型,该模型可以实时检测路边激光雷达的交通参与者。我们的模型使用现有的3D检测器作为基线并提高其准确性。为了证明我们提出的模块的有效性,我们在三个不同的车辆和基础设施数据集上训练和评估模型。为了显示我们探测器的域适应能力,我们在来自中国的基础架构数据集上训练它,并在德国记录的其他数据集上进行转移学习。我们为检测器中每个模块进行几套实验和消融研究,这些实验表明我们的模型的表现优于基线,而推理速度为45 Hz(22 ms)。我们对基于激光雷达的3D探测器做出了重大贡献,可用于智能城市应用程序,以提供连接和自动化的车辆具有深远的视野。连接到路边传感器的车辆可以获取有关拐角处其他车辆的信息,以改善其道路和操纵计划并提高道路交通安全性。
translated by 谷歌翻译
感知环境是实现合作驾驶自动化(CDA)的最基本关键之一,该关键被认为是解决当代运输系统的安全性,流动性和可持续性问题的革命性解决方案。尽管目前在计算机视觉的物体感知领域正在发生前所未有的进化,但由于不可避免的物理遮挡和单辆车的接受程度有限,最先进的感知方法仍在与复杂的现实世界流量环境中挣扎系统。基于多个空间分离的感知节点,合作感知(CP)诞生是为了解锁驱动自动化的感知瓶颈。在本文中,我们全面审查和分析了CP的研究进度,据我们所知,这是第一次提出统一的CP框架。审查了基于不同类型的传感器的CP系统的体系结构和分类学,以显示对CP系统的工作流程和不同结构的高级描述。对节点结构,传感器模式和融合方案进行了审查和分析,并使用全面的文献进行了详细的解释。提出了分层CP框架,然后对现有数据集和模拟器进行审查,以勾勒出CP的整体景观。讨论重点介绍了当前的机会,开放挑战和预期的未来趋势。
translated by 谷歌翻译
人的大脑可以毫不费力地识别和定位对象,而基于激光雷达点云的当前3D对象检测方法仍然报告了较低的性能,以检测闭塞和远处的对象:点云的外观由于遮挡而变化很大,并且在沿线的固有差异沿点固有差异变化。传感器的距离。因此,设计功能表示对此类点云至关重要。受到人类联想识别的启发,我们提出了一个新颖的3D检测框架,该框架通过域的适应来使对象完整特征。我们弥合感知域之间的差距,其中特征是从具有亚最佳表示的真实场景中得出的,以及概念域,其中功能是从由不批准对象组成的增强场景中提取的,并具有丰富的详细信息。研究了一种可行的方法,可以在没有外部数据集的情况下构建概念场景。我们进一步介绍了一个基于注意力的重新加权模块,该模块可适应地增强更翔实区域的特征。该网络的功能增强能力将被利用,而无需在推理过程中引入额外的成本,这是各种3D检测框架中的插件。我们以准确性和速度都在Kitti 3D检测基准上实现了新的最先进性能。关于Nuscenes和Waymo数据集的实验也验证了我们方法的多功能性。
translated by 谷歌翻译
In this paper we propose to exploit multiple related tasks for accurate multi-sensor 3D object detection. Towards this goal we present an end-to-end learnable architecture that reasons about 2D and 3D object detection as well as ground estimation and depth completion. Our experiments show that all these tasks are complementary and help the network learn better representations by fusing information at various levels. Importantly, our approach leads the KITTI benchmark on 2D, 3D and bird's eye view object detection, while being real-time. * Equal contribution.† Work done as part of Uber AI Residency program.
translated by 谷歌翻译
Fusing the camera and LiDAR information has become a de-facto standard for 3D object detection tasks. Current methods rely on point clouds from the LiDAR sensor as queries to leverage the feature from the image space. However, people discovered that this underlying assumption makes the current fusion framework infeasible to produce any prediction when there is a LiDAR malfunction, regardless of minor or major. This fundamentally limits the deployment capability to realistic autonomous driving scenarios. In contrast, we propose a surprisingly simple yet novel fusion framework, dubbed BEVFusion, whose camera stream does not depend on the input of LiDAR data, thus addressing the downside of previous methods. We empirically show that our framework surpasses the state-of-the-art methods under the normal training settings. Under the robustness training settings that simulate various LiDAR malfunctions, our framework significantly surpasses the state-of-the-art methods by 15.7% to 28.9% mAP. To the best of our knowledge, we are the first to handle realistic LiDAR malfunction and can be deployed to realistic scenarios without any post-processing procedure. The code is available at https://github.com/ADLab-AutoDrive/BEVFusion.
translated by 谷歌翻译
在鸟眼中学习强大的表现(BEV),以进行感知任务,这是趋势和吸引行业和学术界的广泛关注。大多数自动驾驶算法的常规方法在正面或透视视图中执行检测,细分,跟踪等。随着传感器配置变得越来越复杂,从不同的传感器中集成了多源信息,并在统一视图中代表功能至关重要。 BEV感知继承了几个优势,因为代表BEV中的周围场景是直观和融合友好的。对于BEV中的代表对象,对于随后的模块,如计划和/或控制是最可取的。 BEV感知的核心问题在于(a)如何通过从透视视图到BEV来通过视图转换来重建丢失的3D信息; (b)如何在BEV网格中获取地面真理注释; (c)如何制定管道以合并来自不同来源和视图的特征; (d)如何适应和概括算法作为传感器配置在不同情况下各不相同。在这项调查中,我们回顾了有关BEV感知的最新工作,并对不同解决方案进行了深入的分析。此外,还描述了该行业的BEV方法的几种系统设计。此外,我们推出了一套完整的实用指南,以提高BEV感知任务的性能,包括相机,激光雷达和融合输入。最后,我们指出了该领域的未来研究指示。我们希望该报告能阐明社区,并鼓励对BEV感知的更多研究。我们保留一个活跃的存储库来收集最新的工作,并在https://github.com/openperceptionx/bevperception-survey-recipe上提供一包技巧的工具箱。
translated by 谷歌翻译
Aiming at highly accurate object detection for connected and automated vehicles (CAVs), this paper presents a Deep Neural Network based 3D object detection model that leverages a three-stage feature extractor by developing a novel LIDAR-Camera fusion scheme. The proposed feature extractor extracts high-level features from two input sensory modalities and recovers the important features discarded during the convolutional process. The novel fusion scheme effectively fuses features across sensory modalities and convolutional layers to find the best representative global features. The fused features are shared by a two-stage network: the region proposal network (RPN) and the detection head (DH). The RPN generates high-recall proposals, and the DH produces final detection results. The experimental results show the proposed model outperforms more recent research on the KITTI 2D and 3D detection benchmark, particularly for distant and highly occluded instances.
translated by 谷歌翻译
Bird's Eye View(BEV)语义分割在自动驾驶的空间传感中起着至关重要的作用。尽管最近的文献在BEV MAP的理解上取得了重大进展,但它们都是基于基于摄像头的系统,这些系统难以处理遮挡并检测复杂的交通场景中的遥远对象。车辆到车辆(V2V)通信技术使自动驾驶汽车能够共享感应信息,与单代理系统相比,可以显着改善感知性能和范围。在本文中,我们提出了Cobevt,这是可以合作生成BEV MAP预测的第一个通用多代理多机构感知框架。为了有效地从基础变压器体系结构中的多视图和多代理数据融合相机功能,我们设计了融合的轴向注意力或传真模块,可以捕获跨视图和代理的局部和全局空间交互。 V2V感知数据集OPV2V的广泛实验表明,COBEVT实现了合作BEV语义分段的最新性能。此外,COBEVT被证明可以推广到其他任务,包括1)具有单代理多摄像机的BEV分割和2)具有多代理激光雷达系统的3D对象检测,并实现具有实时性能的最新性能时间推理速度。
translated by 谷歌翻译
近年来,由于深度学习技术的发展,LiDar Point Clouds的3D对象检测取得了长足的进步。尽管基于体素或基于点的方法在3D对象检测中很受欢迎,但它们通常涉及耗时的操作,例如有关体素的3D卷积或点之间的球查询,从而使所得网络不适合时间关键应用程序。另一方面,基于2D视图的方法具有较高的计算效率,而通常比基于体素或基于点的方法获得的性能低。在这项工作中,我们提出了一个基于实时视图的单阶段3D对象检测器,即CVFNET完成此任务。为了在苛刻的效率条件下加强跨视图的学习,我们的框架提取了不同视图的特征,并以有效的渐进式方式融合了它们。我们首先提出了一个新颖的点范围特征融合模块,该模块在多个阶段深入整合点和范围视图特征。然后,当将所获得的深点视图转换为鸟类视图时,特殊的切片柱旨在很好地维护3D几何形状。为了更好地平衡样品比率,提出了一个稀疏的柱子检测头,将检测集中在非空网上。我们对流行的Kitti和Nuscenes基准进行了实验,并以准确性和速度来实现最先进的性能。
translated by 谷歌翻译
车辆到所有(V2X)通信技术使车辆与附近环境中许多其他实体之间的协作可以从根本上改善自动驾驶的感知系统。但是,缺乏公共数据集极大地限制了协作感知的研究进度。为了填补这一空白,我们提出了V2X-SIM,这是一个针对V2X辅助自动驾驶的全面模拟多代理感知数据集。 V2X-SIM提供:(1)\ hl {Multi-Agent}传感器记录来自路边单元(RSU)和多种能够协作感知的车辆,(2)多模式传感器流,可促进多模式感知和多模式感知和(3)支持各种感知任务的各种基础真理。同时,我们在三个任务(包括检测,跟踪和细分)上为最先进的协作感知算法提供了一个开源测试台,并为最先进的协作感知算法提供了基准。 V2X-SIM试图在现实数据集广泛使用之前刺激自动驾驶的协作感知研究。我们的数据集和代码可在\ url {https://ai4ce.github.io/v2x-sim/}上获得。
translated by 谷歌翻译
Object detection in point clouds is an important aspect of many robotics applications such as autonomous driving. In this paper we consider the problem of encoding a point cloud into a format appropriate for a downstream detection pipeline. Recent literature suggests two types of encoders; fixed encoders tend to be fast but sacrifice accuracy, while encoders that are learned from data are more accurate, but slower. In this work we propose PointPillars, a novel encoder which utilizes PointNets to learn a representation of point clouds organized in vertical columns (pillars). While the encoded features can be used with any standard 2D convolutional detection architecture, we further propose a lean downstream network. Extensive experimentation shows that PointPillars outperforms previous encoders with respect to both speed and accuracy by a large margin. Despite only using lidar, our full detection pipeline significantly outperforms the state of the art, even among fusion methods, with respect to both the 3D and bird's eye view KITTI benchmarks. This detection performance is achieved while running at 62 Hz: a 2 -4 fold runtime improvement. A faster version of our method matches the state of the art at 105 Hz. These benchmarks suggest that PointPillars is an appropriate encoding for object detection in point clouds.
translated by 谷歌翻译