多任务学习的最新研究揭示了解决单个神经网络中相关问题的好处。 3D对象检测和多对象跟踪（MOT）是两个严重的相互交织的问题，可以预测并关联整个时间的对象实例位置。但是，3D MOT中的大多数先前作品都将检测器视为先前的分离管道，不一致地将检测器的输出作为跟踪器的输入。在这项工作中，我们提出了Minkowski Tracker，这是一种稀疏的时空R-CNN，可以共同解决对象检测和跟踪。受基于区域的CNN（R-CNN）的启发，我们建议将跟踪作为对象检测器R-CNN的第二阶段，该跟踪预测了轨道的分配概率。首先，Minkowski Tracker将4D点云作为输入，以生成时空鸟的视图（BEV）特征通过4D稀疏卷积编码器网络。然后，我们提出的TrackAlign聚集了BEV功能的轨道区域（ROI）功能。最后，Minkowski Tracker根据ROI功能预测的检测到追踪匹配概率更新了跟踪及其置信得分。我们在大规模实验中显示，我们方法的总体性能增益是由于四个因素：1。4D编码器的时间推理提高了检测性能2.对象检测的多任务学习和MOT共同增强了彼此3.检测到轨道比赛得分学习隐式运动模型以增强轨道分配4.检测到轨道匹配分数提高了轨道置信度得分的质量。结果，Minkowski Tracker在没有手工设计的运动模型的情况下实现了Nuscenes数据集跟踪任务上的最新性能。

Three-dimensional objects are commonly represented as 3D boxes in a point-cloud. This representation mimics the well-studied image-based 2D bounding-box detection but comes with additional challenges. Objects in a 3D world do not follow any particular orientation, and box-based detectors have difficulties enumerating all orientations or fitting an axis-aligned bounding box to rotated objects. In this paper, we instead propose to represent, detect, and track 3D objects as points. Our framework, CenterPoint, first detects centers of objects using a keypoint detector and regresses to other attributes, including 3D size, 3D orientation, and velocity. In a second stage, it refines these estimates using additional point features on the object. In CenterPoint, 3D object tracking simplifies to greedy closest-point matching. The resulting detection and tracking algorithm is simple, efficient, and effective. CenterPoint achieved state-of-theart performance on the nuScenes benchmark for both 3D detection and tracking, with 65.5 NDS and 63.8 AMOTA for a single model. On the Waymo Open Dataset, Center-Point outperforms all previous single model methods by a large margin and ranks first among all Lidar-only submissions. The code and pretrained models are available at https://github.com/tianweiy/CenterPoint.

Multi-object tracking is a cornerstone capability of any robotic system. Most approaches follow a tracking-by-detection paradigm. However, within this framework, detectors function in a low precision-high recall regime, ensuring a low number of false-negatives while producing a high rate of false-positives. This can negatively affect the tracking component by making data association and track lifecycle management more challenging. Additionally, false-negative detections due to difficult scenarios like occlusions can negatively affect tracking performance. Thus, we propose a method that learns shape and spatio-temporal affinities between consecutive frames to better distinguish between true-positive and false-positive detections and tracks, while compensating for false-negative detections. Our method provides a probabilistic matching of detections that leads to robust data association and track lifecycle management. We quantitatively evaluate our method through ablative experiments and on the nuScenes tracking benchmark where we achieve state-of-the-art results. Our method not only estimates accurate, high-quality tracks but also decreases the overall number of false-positive and false-negative tracks. Please see our project website for source code and demo videos: sites.google.com/view/shasta-3d-mot/home.

3D多对象跟踪旨在唯一，始终如一地识别所有移动实体。尽管在此设置中提供了丰富的时空信息，但当前的3D跟踪方法主要依赖于抽象的信息和有限的历史记录，例如单帧对象边界框。在这项工作中，我们开发了对交通场景的整体表示，该场景利用了现场演员的空间和时间信息。具体而言，我们通过将跟踪的对象表示为时空点和边界框的序列来重新将跟踪作为时空问题，并在悠久的时间历史上进行重新制定。在每个时间戳上，我们通过对对象历史记录的完整顺序进行的细化来改善跟踪对象的位置和运动估计。通过共同考虑时间和空间，我们的代表自然地编码了基本的物理先验，例如对象持久性和整个时间的一致性。我们的时空跟踪框架在Waymo和Nuscenes基准测试中实现了最先进的性能。

在本文中，我们提出了一种基于相机和激光雷达传感器的3D对象检测和跟踪的新的联合对象检测和跟踪（Jodt）框架。所提出的方法称为3D Depectrack，使得检测器和跟踪器能够协作以产生相机和LIDAR数据的时空表示，然后执行3D对象检测和跟踪。检测器通过通过相机和激光乐融合获得的空间特征的加权时间聚集构建时空特征。然后，检测器使用从Roadklet的信息重新配置初始检测结果，从而保持到先前的时间步长。基于由探测器产生的时空特征，跟踪器使用图形神经网络（GNN）将检测到的对象与先前跟踪的对象相关联。我们通过基于规则的边缘修剪和关注的边缘门控的组合设计了一个完全连接的GNN，它利用空间和时间对象上下文来提高跟踪性能。在基准和NUSCENES基准上进行的实验表明，所提出的3D Depectrack在基线方法上的检测和跟踪性能方面实现了显着的改进，并通过检测器和跟踪器之间的协作实现现有方法的最新性能。

与LIDAR相比，相机和雷达传感器在成本，可靠性和维护方面具有显着优势。现有的融合方法通常融合了结果级别的单个模式的输出，称为后期融合策略。这可以通过使用现成的单传感器检测算法受益，但是晚融合无法完全利用传感器的互补特性，因此尽管相机雷达融合的潜力很大，但性能有限。在这里，我们提出了一种新颖的提案级早期融合方法，该方法有效利用了相机和雷达的空间和上下文特性，用于3D对象检测。我们的融合框架首先将图像建议与极坐标系中的雷达点相关联，以有效处理坐标系和空间性质之间的差异。将其作为第一阶段，遵循连续的基于交叉注意的特征融合层在相机和雷达之间自适应地交换时尚信息，从而导致强大而专心的融合。我们的摄像机雷达融合方法可在Nuscenes测试集上获得最新的41.1％地图，而NDS则达到52.3％，比仅摄像机的基线高8.7和10.8点，并在竞争性能上提高竞争性能LIDAR方法。

In this paper we propose to exploit multiple related tasks for accurate multi-sensor 3D object detection. Towards this goal we present an end-to-end learnable architecture that reasons about 2D and 3D object detection as well as ground estimation and depth completion. Our experiments show that all these tasks are complementary and help the network learn better representations by fusing information at various levels. Importantly, our approach leads the KITTI benchmark on 2D, 3D and bird's eye view object detection, while being real-time. * Equal contribution.† Work done as part of Uber AI Residency program.

To track the 3D locations and trajectories of the other traffic participants at any given time, modern autonomous vehicles are equipped with multiple cameras that cover the vehicle's full surroundings. Yet, camera-based 3D object tracking methods prioritize optimizing the single-camera setup and resort to post-hoc fusion in a multi-camera setup. In this paper, we propose a method for panoramic 3D object tracking, called CC-3DT, that associates and models object trajectories both temporally and across views, and improves the overall tracking consistency. In particular, our method fuses 3D detections from multiple cameras before association, reducing identity switches significantly and improving motion modeling. Our experiments on large-scale driving datasets show that fusion before association leads to a large margin of improvement over post-hoc fusion. We set a new state-of-the-art with 12.6% improvement in average multi-object tracking accuracy (AMOTA) among all camera-based methods on the competitive NuScenes 3D tracking benchmark, outperforming previously published methods by 6.5% in AMOTA with the same 3D detector.

In this paper, we propose a novel 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization. Towards this goal, we design an end-to-end learnable architecture that exploits continuous convolutions to fuse image and LIDAR feature maps at different levels of resolution. Our proposed continuous fusion layer encode both discrete-state image features as well as continuous geometric information. This enables us to design a novel, reliable and efficient end-to-end learnable 3D object detector based on multiple sensors. Our experimental evaluation on both KITTI as well as a large scale 3D object detection benchmark shows significant improvements over the state of the art.

Tracking has traditionally been the art of following interest points through space and time. This changed with the rise of powerful deep networks. Nowadays, tracking is dominated by pipelines that perform object detection followed by temporal association, also known as tracking-by-detection. We present a simultaneous detection and tracking algorithm that is simpler, faster, and more accurate than the state of the art. Our tracker, CenterTrack, applies a detection model to a pair of images and detections from the prior frame. Given this minimal input, CenterTrack localizes objects and predicts their associations with the previous frame. That's it. CenterTrack is simple, online (no peeking into the future), and real-time. It achieves 67.8% MOTA on the MOT17 challenge at 22 FPS and 89.4% MOTA on the KITTI tracking benchmark at 15 FPS, setting a new state of the art on both datasets. CenterTrack is easily extended to monocular 3D tracking by regressing additional 3D attributes. Using monocular video input, it achieves 28.3% AMOTA@0.2 on the newly released nuScenes 3D tracking benchmark, substantially outperforming the monocular baseline on this benchmark while running at 28 FPS.

基于LIDAR的传感驱动器电流自主车辆。尽管进展迅速，但目前的激光雷达传感器在分辨率和成本方面仍然落后于传统彩色相机背后的二十年。对于自主驾驶，这意味着靠近传感器的大物体很容易可见，但远方或小物体仅包括一个测量或两个。这是一个问题，尤其是当这些对象结果驾驶危险时。另一方面，在车载RGB传感器中清晰可见这些相同的对象。在这项工作中，我们提出了一种将RGB传感器无缝熔化成基于LIDAR的3D识别方法。我们的方法采用一组2D检测来生成密集的3D虚拟点，以增加否则稀疏的3D点云。这些虚拟点自然地集成到任何基于标准的LIDAR的3D探测器以及常规激光雷达测量。由此产生的多模态检测器简单且有效。大规模NUSCENES数据集的实验结果表明，我们的框架通过显着的6.6地图改善了强大的中心点基线，并且优于竞争融合方法。代码和更多可视化可在https://tianweiy.github.io/mvp/上获得

以前的在线3D多对象跟踪（3DMOT）方法在与几帧的新检测无关时终止ROCKET。但是如果一个物体刚刚变暗，就像被其他物体暂时封闭或者只是从FOV暂时封闭一样，过早地终止ROCKET将导致身份切换。我们揭示了过早的轨迹终端是现代3DMOT系统中身份开关的主要原因。为了解决这个问题，我们提出了一个不朽的跟踪器，一个简单的跟踪系统，它利用轨迹预测来维护对象变暗的物体的轨迹。我们使用一个简单的卡尔曼滤波器进行轨迹预测，并在目标不可见时通过预测保留轨迹。通过这种方法，我们可以避免由过早托管终止产生的96％的车辆标识开关。如果没有任何学习的参数，我们的方法在Waymo Open DataSet测试集上的车载类别的0.0001级和竞争Mota处实现了不匹配的比率。我们的不匹配比率比任何先前发表的方法低一倍。在NUSCENes上报告了类似的结果。我们相信拟议的不朽追踪器可以为推动3DMOT的极限提供简单而强大的解决方案。我们的代码可在https://github.com/immortaltracker/immortaltracker中找到。

基于激光雷达的3D对象检测，语义分割和全景分段通常在具有独特架构的专业网络中实现，这些网络很难相互适应。本文介绍了Lidarmultinet，这是一个基于激光雷达的多任务网络，该网络统一了这三个主要的激光感知任务。在其许多好处中，多任务网络可以通过在多个任务中分享权重和计算来降低总成本。但是，与独立组合的单任务模型相比，它通常表现不佳。拟议的Lidarmultinet旨在弥合多任务网络和多个单任务网络之间的性能差距。 Lidarmultinet的核心是一个强大的基于3D Voxel的编码器架构，具有全局上下文池（GCP）模块，从激光雷达框架中提取全局上下文特征。特定于任务的头部添加在网络之上，以执行三个激光雷达感知任务。只需添加新的任务特定的头部，可以在引入几乎没有额外成本的同时，就可以实现更多任务。还提出了第二阶段来完善第一阶段的分割并生成准确的全景分割结果。 Lidarmultinet在Waymo Open数据集和Nuscenes数据集上进行了广泛的测试，这首先证明了主要的激光雷达感知任务可以统一在单个强大的网络中，该网络是经过训练的端到端，并实现了最先进的性能。值得注意的是，Lidarmultinet在Waymo Open数据集3D语义分割挑战2022中达到了最高的MIOU和最佳准确性，对于测试集中的22个类中的大多数，仅使用LIDAR点作为输入。它还为Waymo 3D对象检测基准和三个Nuscenes基准测试的单个模型设置了新的最新模型。

Object detection in point clouds is an important aspect of many robotics applications such as autonomous driving. In this paper we consider the problem of encoding a point cloud into a format appropriate for a downstream detection pipeline. Recent literature suggests two types of encoders; fixed encoders tend to be fast but sacrifice accuracy, while encoders that are learned from data are more accurate, but slower. In this work we propose PointPillars, a novel encoder which utilizes PointNets to learn a representation of point clouds organized in vertical columns (pillars). While the encoded features can be used with any standard 2D convolutional detection architecture, we further propose a lean downstream network. Extensive experimentation shows that PointPillars outperforms previous encoders with respect to both speed and accuracy by a large margin. Despite only using lidar, our full detection pipeline significantly outperforms the state of the art, even among fusion methods, with respect to both the 3D and bird's eye view KITTI benchmarks. This detection performance is achieved while running at 62 Hz: a 2 -4 fold runtime improvement. A faster version of our method matches the state of the art at 105 Hz. These benchmarks suggest that PointPillars is an appropriate encoding for object detection in point clouds.

3D多对象跟踪（MOT）近年来目睹了众多新颖的基准和方法，尤其是那些在“逐侦测”范式下的基准。尽管他们的进步和有用，但对他们的优势和劣势的深入分析尚不可用。在本文中，我们通过将它们分解为四个组成部分来总结当前的3D MOL方法：检测，关联，运动模型和生命周期管理的预处理。然后，我们将现有算法的故障情况归因于每个组件并详细研究它们。基于分析，我们提出了相应的改进，导致强大但简单的基线：简单进展。 Waymo Open DataSet和Nuscenes上的综合实验结果表明，我们的最终方法可以通过微小的修改来实现新的最先进的结果。此外，我们采取额外的步骤并重新思考当前的基准面是否真实地反映了真实挑战的算法能力。我们深入了解现有基准的细节，并找到一些有趣的事实。最后，我们分析了\ name \中剩余失败的分布和原因，并提出了3D MOT的未来方向。我们的代码可在https://github.com/tusimple/simpletrack获得。

3D object detection from LiDAR point cloud is a challenging problem in 3D scene understanding and has many practical applications. In this paper, we extend our preliminary work PointRCNN to a novel and strong point-cloud-based 3D object detection framework, the part-aware and aggregation neural network (Part-A 2 net). The whole framework consists of the part-aware stage and the part-aggregation stage. Firstly, the part-aware stage for the first time fully utilizes free-of-charge part supervisions derived from 3D ground-truth boxes to simultaneously predict high quality 3D proposals and accurate intra-object part locations. The predicted intra-object part locations within the same proposal are grouped by our new-designed RoI-aware point cloud pooling module, which results in an effective representation to encode the geometry-specific features of each 3D proposal. Then the part-aggregation stage learns to re-score the box and refine the box location by exploring the spatial relationship of the pooled intra-object part locations. Extensive experiments are conducted to demonstrate the performance improvements from each component of our proposed framework. Our Part-A 2 net outperforms all existing 3D detection methods and achieves new state-of-the-art on KITTI 3D object detection dataset by utilizing only the LiDAR point cloud data. Code is available at https://github.com/sshaoshuai/PointCloudDet3D.

在鸟眼中学习强大的表现（BEV），以进行感知任务，这是趋势和吸引行业和学术界的广泛关注。大多数自动驾驶算法的常规方法在正面或透视视图中执行检测，细分，跟踪等。随着传感器配置变得越来越复杂，从不同的传感器中集成了多源信息，并在统一视图中代表功能至关重要。 BEV感知继承了几个优势，因为代表BEV中的周围场景是直观和融合友好的。对于BEV中的代表对象，对于随后的模块，如计划和/或控制是最可取的。 BEV感知的核心问题在于（a）如何通过从透视视图到BEV来通过视图转换来重建丢失的3D信息； （b）如何在BEV网格中获取地面真理注释； （c）如何制定管道以合并来自不同来源和视图的特征； （d）如何适应和概括算法作为传感器配置在不同情况下各不相同。在这项调查中，我们回顾了有关BEV感知的最新工作，并对不同解决方案进行了深入的分析。此外，还描述了该行业的BEV方法的几种系统设计。此外，我们推出了一套完整的实用指南，以提高BEV感知任务的性能，包括相机，激光雷达和融合输入。最后，我们指出了该领域的未来研究指示。我们希望该报告能阐明社区，并鼓励对BEV感知的更多研究。我们保留一个活跃的存储库来收集最新的工作，并在https://github.com/openperceptionx/bevperception-survey-recipe上提供一包技巧的工具箱。

3D多对象跟踪（MOT）是自动驾驶汽车的关键问题，需要在动态环境中执行信息良好的运动计划。特别是对于密集的占领场景，将现有曲目与新检测相关联仍然具有挑战性，因为现有系统倾向于省略关键的上下文信息。我们提出的解决方案InterTrack引入了3D MOT的相互作用变压器，以生成数据关联的区分对象表示。我们为每个轨道和检测提取状态和形状特征，并通过注意力有效地汇总全局信息。然后，我们对每个轨道/检测功能对进行学习的回归以估计亲和力，并使用强大的两阶段数据关联和轨道管理方法来生成最终轨道。我们在Nuscenes 3D MOT基准上验证了我们的方法，在那里我们观察到了显着的改进，尤其是在物理大小和聚类对象的类别上。从提交开始时，InterTrack在使用CenterPoint检测的方法中排名第1位AMOTA。

Point cloud learning has lately attracted increasing attention due to its wide applications in many areas, such as computer vision, autonomous driving, and robotics. As a dominating technique in AI, deep learning has been successfully used to solve various 2D vision problems. However, deep learning on point clouds is still in its infancy due to the unique challenges faced by the processing of point clouds with deep neural networks. Recently, deep learning on point clouds has become even thriving, with numerous methods being proposed to address different problems in this area. To stimulate future research, this paper presents a comprehensive review of recent progress in deep learning methods for point clouds. It covers three major tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation. It also presents comparative results on several publicly available datasets, together with insightful observations and inspiring future research directions.

由于缺乏深度信息，单眼3D对象检测在自主驾驶中非常具有挑战性。本文提出了一种基于多尺度深度分层的单眼单目眼3D对象检测算法，它使用锚定方法检测每像素预测中的3D对象。在所提出的MDS-Net中，开发了一种新的基于深度的分层结构，以通过在对象的深度和图像尺寸之间建立数学模型来改善网络的深度预测能力。然后开发出新的角度损耗功能，以进一步提高角度预测的精度并提高训练的收敛速度。最终在后处理阶段最终应用优化的软，以调整候选盒的置信度。基蒂基准测试的实验表明，MDS-Net在3D检测中优于现有的单目3D检测方法，并在满足实时要求时进行3D检测和BEV检测任务。