低成本单眼的3D对象检测在自主驾驶中起着基本作用,而其精度仍然远非令人满意。在本文中,我们挖掘了3D对象检测任务,并将其重构为对象本地化和外观感知的子任务,这有​​利于整个任务的互惠信息的深度挖掘。我们介绍了一个名为DFR-Net的动态特征反射网络,其中包含两种新的独立模块:(i)首先将任务特征分开的外观定位特征反射模块(ALFR),然后自相互反映互核特征; (ii)通过自学习方式自适应地重建各个子任务的培训过程的动态内部交易模块(DIT)。关于挑战基蒂数据集的广泛实验证明了DFR网的有效性和泛化。我们在基蒂测试集中的所有单眼3D对象探测器中排名第一(直到2021年3月16日)。所提出的方法在许多尖端的3D检测框架中也容易在较忽略的成本下以忽略的成本来播放。该代码将公开可用。
translated by 谷歌翻译
由于LIDAR传感器捕获的精确深度信息缺乏准确的深度信息,单眼3D对象检测是一个关键而挑战的自主驾驶任务。在本文中,我们提出了一种立体引导的单目3D对象检测网络,称为SGM3D,其利用立体图像提取的鲁棒3D特征来增强从单眼图像中学到的特征。我们创新地研究了多粒度域适配模块(MG-DA)以利用网络的能力,以便仅基于单手套提示产生立体模拟功能。利用粗均衡特征级以及精细锚级域适配,以引导单眼分支。我们介绍了一个基于IOO匹配的对齐模块(iou-ma),用于立体声和单眼域之间的对象级域适应,以减轻先前阶段中的不匹配。我们对最具挑战性的基蒂和Lyft数据集进行了广泛的实验,并实现了新的最先进的性能。此外,我们的方法可以集成到许多其他单眼的方法中以提高性能而不引入任何额外的计算成本。
translated by 谷歌翻译
Compared to typical multi-sensor systems, monocular 3D object detection has attracted much attention due to its simple configuration. However, there is still a significant gap between LiDAR-based and monocular-based methods. In this paper, we find that the ill-posed nature of monocular imagery can lead to depth ambiguity. Specifically, objects with different depths can appear with the same bounding boxes and similar visual features in the 2D image. Unfortunately, the network cannot accurately distinguish different depths from such non-discriminative visual features, resulting in unstable depth training. To facilitate depth learning, we propose a simple yet effective plug-and-play module, One Bounding Box Multiple Objects (OBMO). Concretely, we add a set of suitable pseudo labels by shifting the 3D bounding box along the viewing frustum. To constrain the pseudo-3D labels to be reasonable, we carefully design two label scoring strategies to represent their quality. In contrast to the original hard depth labels, such soft pseudo labels with quality scores allow the network to learn a reasonable depth range, boosting training stability and thus improving final performance. Extensive experiments on KITTI and Waymo benchmarks show that our method significantly improves state-of-the-art monocular 3D detectors by a significant margin (The improvements under the moderate setting on KITTI validation set are $\mathbf{1.82\sim 10.91\%}$ mAP in BEV and $\mathbf{1.18\sim 9.36\%}$ mAP in 3D}. Codes have been released at https://github.com/mrsempress/OBMO.
translated by 谷歌翻译
人的大脑可以毫不费力地识别和定位对象,而基于激光雷达点云的当前3D对象检测方法仍然报告了较低的性能,以检测闭塞和远处的对象:点云的外观由于遮挡而变化很大,并且在沿线的固有差异沿点固有差异变化。传感器的距离。因此,设计功能表示对此类点云至关重要。受到人类联想识别的启发,我们提出了一个新颖的3D检测框架,该框架通过域的适应来使对象完整特征。我们弥合感知域之间的差距,其中特征是从具有亚最佳表示的真实场景中得出的,以及概念域,其中功能是从由不批准对象组成的增强场景中提取的,并具有丰富的详细信息。研究了一种可行的方法,可以在没有外部数据集的情况下构建概念场景。我们进一步介绍了一个基于注意力的重新加权模块,该模块可适应地增强更翔实区域的特征。该网络的功能增强能力将被利用,而无需在推理过程中引入额外的成本,这是各种3D检测框架中的插件。我们以准确性和速度都在Kitti 3D检测基准上实现了新的最先进性能。关于Nuscenes和Waymo数据集的实验也验证了我们方法的多功能性。
translated by 谷歌翻译
来自LIDAR或相机传感器的3D对象检测任务对于自动驾驶至关重要。先锋尝试多模式融合的尝试补充了稀疏的激光雷达点云,其中包括图像的丰富语义纹理信息,以额外的网络设计和开销为代价。在这项工作中,我们提出了一个名为SPNET的新型语义传递框架,以通过丰富的上下文绘画的指导来提高现有基于激光雷达的3D检测模型的性能,在推理过程中没有额外的计算成本。我们的关键设计是首先通过训练语义绘制的教师模型来利用地面真实标签中潜在的指导性语义知识,然后引导纯LIDAR网络通过不同的粒度传播模块来学习语义绘制的表示:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类别:类:类别:类别:类别:类别:类别:类别:类别: - 通过,像素的传递和实例传递。实验结果表明,所提出的SPNET可以与大多数现有的3D检测框架无缝合作,其中AP增益为1〜5%,甚至在KITTI测试基准上实现了新的最新3D检测性能。代码可在以下网址获得:https://github.com/jb892/sp​​net。
translated by 谷歌翻译
In this paper, we propose a novel 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization. Towards this goal, we design an end-to-end learnable architecture that exploits continuous convolutions to fuse image and LIDAR feature maps at different levels of resolution. Our proposed continuous fusion layer encode both discrete-state image features as well as continuous geometric information. This enables us to design a novel, reliable and efficient end-to-end learnable 3D object detector based on multiple sensors. Our experimental evaluation on both KITTI as well as a large scale 3D object detection benchmark shows significant improvements over the state of the art.
translated by 谷歌翻译
对于许多应用程序,包括自动驾驶,机器人抓握和增强现实,单眼3D对象检测是一项基本但非常重要的任务。现有的领先方法倾向于首先估算输入图像的深度,并基于点云检测3D对象。该例程遭受了深度估计和对象检测之间固有的差距。此外,预测误差积累也会影响性能。在本文中,提出了一种名为MonopCN的新方法。引入单频道的洞察力是,我们建议在训练期间模拟基于点云的探测器的特征学习行为。因此,在推理期间,学习的特征和预测将与基于点云的检测器相似。为了实现这一目标,我们建议一个场景级仿真模块,一个ROI级别的仿真模块和一个响应级仿真模块,这些模块逐渐用于检测器的完整特征学习和预测管道。我们将我们的方法应用于著名的M3D-RPN检测器和CADDN检测器,并在Kitti和Waymo Open数据集上进行了广泛的实验。结果表明,我们的方法始终提高不同边缘的不同单眼探测器的性能,而无需更改网络体系结构。我们的方法最终达到了最先进的性能。
translated by 谷歌翻译
In this paper we propose to exploit multiple related tasks for accurate multi-sensor 3D object detection. Towards this goal we present an end-to-end learnable architecture that reasons about 2D and 3D object detection as well as ground estimation and depth completion. Our experiments show that all these tasks are complementary and help the network learn better representations by fusing information at various levels. Importantly, our approach leads the KITTI benchmark on 2D, 3D and bird's eye view object detection, while being real-time. * Equal contribution.† Work done as part of Uber AI Residency program.
translated by 谷歌翻译
3D对象检测是各种实际应用所需的重要功能,例如驾驶员辅助系统。单眼3D检测作为基于图像的方法的代表性的常规设置,提供比依赖Lidars的传统设置更经济的解决方案,但仍然产生不令人满意的结果。本文首先提出了对这个问题的系统研究。我们观察到,目前的单目3D检测可以简化为实例深度估计问题:不准确的实例深度阻止所有其他3D属性预测改善整体检测性能。此外,最近的方法直接估计基于孤立的实例或像素的深度,同时忽略不同对象的几何关系。为此,我们在跨预测对象构建几何关系图,并使用该图来促进深度估计。随着每个实例的初步深度估计通常在这种不均匀的环境中通常不准确,我们纳入了概率表示以捕获不确定性。它提供了一个重要的指标,以确定自信的预测并进一步引导深度传播。尽管基本思想的简单性,但我们的方法,PGD对基蒂和NUSCENES基准的显着改进,尽管在所有单眼视觉的方法中实现了第1个,同时仍保持实时效率。代码和模型将在https://github.com/open-mmlab/mmdetection3d发布。
translated by 谷歌翻译
3D object detection from LiDAR point cloud is a challenging problem in 3D scene understanding and has many practical applications. In this paper, we extend our preliminary work PointRCNN to a novel and strong point-cloud-based 3D object detection framework, the part-aware and aggregation neural network (Part-A 2 net). The whole framework consists of the part-aware stage and the part-aggregation stage. Firstly, the part-aware stage for the first time fully utilizes free-of-charge part supervisions derived from 3D ground-truth boxes to simultaneously predict high quality 3D proposals and accurate intra-object part locations. The predicted intra-object part locations within the same proposal are grouped by our new-designed RoI-aware point cloud pooling module, which results in an effective representation to encode the geometry-specific features of each 3D proposal. Then the part-aggregation stage learns to re-score the box and refine the box location by exploring the spatial relationship of the pooled intra-object part locations. Extensive experiments are conducted to demonstrate the performance improvements from each component of our proposed framework. Our Part-A 2 net outperforms all existing 3D detection methods and achieves new state-of-the-art on KITTI 3D object detection dataset by utilizing only the LiDAR point cloud data. Code is available at https://github.com/sshaoshuai/PointCloudDet3D.
translated by 谷歌翻译
鉴于其经济性与多传感器设置相比,从单眼输入中感知的3D对象对于机器人系统至关重要。它非常困难,因为单个图像无法提供预测绝对深度值的任何线索。通过双眼方法进行3D对象检测,我们利用了相机自我运动提供的强几何结构来进行准确的对象深度估计和检测。我们首先对此一般的两视案例进行了理论分析,并注意两个挑战:1)来自多个估计的累积错误,这些估计使直接预测棘手; 2)由静态摄像机和歧义匹配引起的固有难题。因此,我们建立了具有几何感知成本量的立体声对应关系,作为深度估计的替代方案,并以单眼理解进一步补偿了它,以解决第二个问题。我们的框架(DFM)命名为深度(DFM),然后使用已建立的几何形状将2D图像特征提升到3D空间并检测到其3D对象。我们还提出了一个无姿势的DFM,以使其在摄像头不可用时可用。我们的框架在Kitti基准测试上的优于最先进的方法。详细的定量和定性分析也验证了我们的理论结论。该代码将在https://github.com/tai-wang/depth-from-motion上发布。
translated by 谷歌翻译
最近,融合了激光雷达点云和相机图像,提高了3D对象检测的性能和稳健性,因为这两种方式自然具有强烈的互补性。在本文中,我们通过引入新型级联双向融合〜(CB融合)模块和多模态一致性〜(MC)损耗来提出用于多模态3D对象检测的EPNet ++。更具体地说,所提出的CB融合模块提高点特征的丰富语义信息,以级联双向交互融合方式具有图像特征,导致更全面且辨别的特征表示。 MC损失明确保证预测分数之间的一致性,以获得更全面且可靠的置信度分数。基蒂,JRDB和Sun-RGBD数据集的实验结果展示了通过最先进的方法的EPNet ++的优越性。此外,我们强调一个关键但很容易被忽视的问题,这是探讨稀疏场景中的3D探测器的性能和鲁棒性。广泛的实验存在,EPNet ++优于现有的SOTA方法,在高稀疏点云壳中具有显着的边距,这可能是降低LIDAR传感器的昂贵成本的可用方向。代码将来会发布。
translated by 谷歌翻译
3D object detection is an essential task in autonomous driving. Recent techniques excel with highly accurate detection rates, provided the 3D input data is obtained from precise but expensive LiDAR technology. Approaches based on cheaper monocular or stereo imagery data have, until now, resulted in drastically lower accuracies -a gap that is commonly attributed to poor image-based depth estimation. However, in this paper we argue that it is not the quality of the data but its representation that accounts for the majority of the difference. Taking the inner workings of convolutional neural networks into consideration, we propose to convert image-based depth maps to pseudo-LiDAR representations -essentially mimicking the LiDAR signal. With this representation we can apply different existing LiDAR-based detection algorithms. On the popular KITTI benchmark, our approach achieves impressive improvements over the existing state-of-the-art in image-based performance -raising the detection accuracy of objects within the 30m range from the previous state-of-the-art of 22% to an unprecedented 74%. At the time of submission our algorithm holds the highest entry on the KITTI 3D object detection leaderboard for stereo-image-based approaches. Our code is publicly available at https: //github.com/mileyan/pseudo_lidar.
translated by 谷歌翻译
LiDAR-based 3D Object detectors have achieved impressive performances in many benchmarks, however, multisensors fusion-based techniques are promising to further improve the results. PointPainting, as a recently proposed framework, can add the semantic information from the 2D image into the 3D LiDAR point by the painting operation to boost the detection performance. However, due to the limited resolution of 2D feature maps, severe boundary-blurring effect happens during re-projection of 2D semantic segmentation into the 3D point clouds. To well handle this limitation, a general multimodal fusion framework MSF has been proposed to fuse the semantic information from both the 2D image and 3D points scene parsing results. Specifically, MSF includes three main modules. First, SOTA off-the-shelf 2D/3D semantic segmentation approaches are employed to generate the parsing results for 2D images and 3D point clouds. The 2D semantic information is further re-projected into the 3D point clouds with calibrated parameters. To handle the misalignment between the 2D and 3D parsing results, an AAF module is proposed to fuse them by learning an adaptive fusion score. Then the point cloud with the fused semantic label is sent to the following 3D object detectors. Furthermore, we propose a DFF module to aggregate deep features in different levels to boost the final detection performance. The effectiveness of the framework has been verified on two public large-scale 3D object detection benchmarks by comparing with different baselines. The experimental results show that the proposed fusion strategies can significantly improve the detection performance compared to the methods using only point clouds and the methods using only 2D semantic information. Most importantly, the proposed approach significantly outperforms other approaches and sets new SOTA results on the nuScenes testing benchmark.
translated by 谷歌翻译
基于摄像头的3D对象探测器由于其更广泛的部署而欢迎其比LIDAR传感器较低。我们首先重新访问先前的立体声检测器DSGN,以表示代表3D几何和语义的立体音量构建方式。我们抛光立体声建模,并提出高级版本DSGN ++,旨在在三个主要方面增强整个2d到3D管道的有效信息流。首先,为了有效地将2D信息提高到立体声音量,我们提出了深度扫地(DPS),以允许较密集的连接并提取深度引导的特征。其次,为了掌握不同间距的功能,我们提出了一个新颖的立体声音量 - 双视立体声卷(DSV),该卷(DSV)集成了前视图和顶部视图功能,并重建了相机frustum中的子素深度。第三,随着前景区域在3D空间中的占主导地位,我们提出了一种多模式数据编辑策略-Stereo-lidar拷贝性 - 可确保跨模式对齐并提高数据效率。没有铃铛和哨子,在流行的Kitti基准测试中的各种模式设置中进行了广泛的实验表明,我们的方法始终优于所有类别的基于相机的3D检测器。代码可从https://github.com/chenyilun95/dsgn2获得。
translated by 谷歌翻译
由于基于相交的联盟(IOU)优化维持最终IOU预测度量和损失的一致性,因此它已被广泛用于单级2D对象检测器的回归和分类分支。最近,几种3D对象检测方法采用了基于IOU的优化,并用3D iou直接替换了2D iou。但是,由于复杂的实施和效率低下的向后操作,3D中的这种直接计算非常昂贵。此外,基于3D IOU的优化是优化的,因为它对旋转很敏感,因此可能导致训练不稳定性和检测性能恶化。在本文中,我们提出了一种新型的旋转旋转iou(RDIOU)方法,该方法可以减轻旋转敏感性问题,并在训练阶段与3D IOU相比产生更有效的优化目标。具体而言,我们的RDIOU通过将旋转变量解耦为独立术语,但保留3D iou的几何形状来简化回归参数的复杂相互作用。通过将RDIOU纳入回归和分类分支,鼓励网络学习更精确的边界框,并同时克服分类和回归之间的错位问题。基准Kitti和Waymo开放数据集的广泛实验验证我们的RDIOU方法可以为单阶段3D对象检测带来实质性改进。
translated by 谷歌翻译
单眼3D对象检测旨在将3D边界框本地化在输入单个2D图像中。这是一个非常具有挑战性的问题并且仍然是开放的,特别是当没有额外的信息时(例如,深度,激光雷达和/或多帧)可以利用训练和/或推理。本文提出了一种对单眼3D对象检测的简单而有效的配方,而无需利用任何额外信息。它介绍了从训练中学习单眼背景的单片方法,以帮助单目3D对象检测。关键的想法是,通过图像中的对象的注释3D边界框,在训练中有一个丰富的良好的投影2D监控信号,例如投影的角键点及其相关联的偏移向量相对于中心在2D边界框中,应该被开发为培训中的辅助任务。拟议的单一的单一的机动在衡量标准理论中的克拉默 - Wold定理在高水平下。在实施中,它利用非常简单的端到端设计来证明学习辅助单眼环境的有效性,它由三个组成组成:基于深度神经网络(DNN)的特征骨干,一些回归头部分支用于学习用于3D边界框预测的基本参数,以及用于学习辅助上下文的许多回归头分支。在训练之后,丢弃辅助上下文回归分支以获得更好的推理效率。在实验中,拟议的单一组在基蒂基准(汽车,Pedestrain和骑自行车的人)中测试。它超越了汽车类别上排行榜中的所有现有技术,并在准确性方面获得了行人和骑自行车者的可比性。由于简单的设计,所提出的单控制方法在比较中获得了38.7 FP的最快推断速度
translated by 谷歌翻译
In this work, we study 3D object detection from RGB-D data in both indoor and outdoor scenes. While previous methods focus on images or 3D voxels, often obscuring natural 3D patterns and invariances of 3D data, we directly operate on raw point clouds by popping up RGB-D scans. However, a key challenge of this approach is how to efficiently localize objects in point clouds of large-scale scenes (region proposal). Instead of solely relying on 3D proposals, our method leverages both mature 2D object detectors and advanced 3D deep learning for object localization, achieving efficiency as well as high recall for even small objects. Benefited from learning directly in raw point clouds, our method is also able to precisely estimate 3D bounding boxes even under strong occlusion or with very sparse points. Evaluated on KITTI and SUN RGB-D 3D detection benchmarks, our method outperforms the state of the art by remarkable margins while having real-time capability. * Majority of the work done as an intern at Nuro, Inc. depth to point cloud 2D region (from CNN) to 3D frustum 3D box (from PointNet)
translated by 谷歌翻译
Monocular 3D object detection is a key problem for autonomous vehicles, as it provides a solution with simple configuration compared to typical multi-sensor systems. The main challenge in monocular 3D detection lies in accurately predicting object depth, which must be inferred from object and scene cues due to the lack of direct range measurement. Many methods attempt to directly estimate depth to assist in 3D detection, but show limited performance as a result of depth inaccuracy. Our proposed solution, Categorical Depth Distribution Network (CaDDN), uses a predicted categorical depth distribution for each pixel to project rich contextual feature information to the appropriate depth interval in 3D space. We then use the computationally efficient bird's-eye-view projection and single-stage detector to produce the final output detections. We design CaDDN as a fully differentiable end-to-end approach for joint depth estimation and object detection. We validate our approach on the KITTI 3D object detection benchmark, where we rank 1 st among published monocular methods. We also provide the first monocular 3D detection results on the newly released Waymo Open Dataset. We provide a code release for CaDDN which is made available here.
translated by 谷歌翻译
以视觉为中心的BEV感知由于其固有的优点,最近受到行业和学术界的关注,包括展示世界自然代表和融合友好。随着深度学习的快速发展,已经提出了许多方法来解决以视觉为中心的BEV感知。但是,最近没有针对这个小说和不断发展的研究领域的调查。为了刺激其未来的研究,本文对以视觉为中心的BEV感知及其扩展进行了全面调查。它收集并组织了最近的知识,并对常用算法进行了系统的综述和摘要。它还为几项BEV感知任务提供了深入的分析和比较结果,从而促进了未来作品的比较并激发了未来的研究方向。此外,还讨论了经验实现细节并证明有利于相关算法的开发。
translated by 谷歌翻译