Determining accurate bird's eye view (BEV) positions of objects and tracks in a scene is vital for various perception tasks including object interactions mapping, scenario extraction etc., however, the level of supervision required to accomplish that is extremely challenging to procure. We propose a light-weight, weakly supervised method to estimate 3D position of objects by jointly learning to regress the 2D object detections and scene's depth prediction in a single feed-forward pass of a network. Our proposed method extends a center-point based single-shot object detector \cite{zhou2019objects}, and introduces a novel object representation where each object is modeled as a BEV point spatio-temporally, without the need of any 3D or BEV annotations for training and LiDAR data at query time. The approach leverages readily available 2D object supervision along with LiDAR point clouds (used only during training) to jointly train a single network, that learns to predict 2D object detection alongside the whole scene's depth, to spatio-temporally model object tracks as points in BEV. The proposed method is computationally over $\sim$10x efficient compared to recent SOTA approaches [1, 38] while achieving comparable accuracies on KITTI tracking benchmark.

Tracking has traditionally been the art of following interest points through space and time. This changed with the rise of powerful deep networks. Nowadays, tracking is dominated by pipelines that perform object detection followed by temporal association, also known as tracking-by-detection. We present a simultaneous detection and tracking algorithm that is simpler, faster, and more accurate than the state of the art. Our tracker, CenterTrack, applies a detection model to a pair of images and detections from the prior frame. Given this minimal input, CenterTrack localizes objects and predicts their associations with the previous frame. That's it. CenterTrack is simple, online (no peeking into the future), and real-time. It achieves 67.8% MOTA on the MOT17 challenge at 22 FPS and 89.4% MOTA on the KITTI tracking benchmark at 15 FPS, setting a new state of the art on both datasets. CenterTrack is easily extended to monocular 3D tracking by regressing additional 3D attributes. Using monocular video input, it achieves 28.3% AMOTA@0.2 on the newly released nuScenes 3D tracking benchmark, substantially outperforming the monocular baseline on this benchmark while running at 28 FPS.

Three-dimensional objects are commonly represented as 3D boxes in a point-cloud. This representation mimics the well-studied image-based 2D bounding-box detection but comes with additional challenges. Objects in a 3D world do not follow any particular orientation, and box-based detectors have difficulties enumerating all orientations or fitting an axis-aligned bounding box to rotated objects. In this paper, we instead propose to represent, detect, and track 3D objects as points. Our framework, CenterPoint, first detects centers of objects using a keypoint detector and regresses to other attributes, including 3D size, 3D orientation, and velocity. In a second stage, it refines these estimates using additional point features on the object. In CenterPoint, 3D object tracking simplifies to greedy closest-point matching. The resulting detection and tracking algorithm is simple, efficient, and effective. CenterPoint achieved state-of-theart performance on the nuScenes benchmark for both 3D detection and tracking, with 65.5 NDS and 63.8 AMOTA for a single model. On the Waymo Open Dataset, Center-Point outperforms all previous single model methods by a large margin and ranks first among all Lidar-only submissions. The code and pretrained models are available at https://github.com/tianweiy/CenterPoint.

近年来，自主驾驶LIDAR数据的3D对象检测一直在迈出卓越的进展。在最先进的方法中，已经证明了将点云进行编码为鸟瞰图（BEV）是有效且有效的。与透视图不同，BEV在物体之间保留丰富的空间和距离信息;虽然在BEV中相同类型的更远物体不会较小，但它们包含稀疏点云特征。这一事实使用共享卷积神经网络削弱了BEV特征提取。为了解决这一挑战，我们提出了范围感知注意网络（RAANET），提取更强大的BEV功能并产生卓越的3D对象检测。范围感知的注意力（RAA）卷曲显着改善了近距离的特征提取。此外，我们提出了一种新的辅助损耗，用于密度估计，以进一步增强覆盖物体的Raanet的检测精度。值得注意的是，我们提出的RAA卷积轻量级，并兼容，以集成到用于BEV检测的任何CNN架构中。 Nuscenes DataSet上的广泛实验表明，我们的提出方法优于基于LIDAR的3D对象检测的最先进的方法，具有16 Hz的实时推断速度，为LITE版本为22 Hz。该代码在匿名GitHub存储库HTTPS://github.com/Anonymous0522 / ange上公开提供。

由于其在自主驾驶中的应用，因此基于单眼图像的3D感知已成为一个活跃的研究领域。与基于激光雷达的技术相比，单眼3D感知（包括检测和跟踪）的方法通常会产生较低的性能。通过系统的分析，我们确定了每个对象深度估计精度是界限性能的主要因素。在这种观察过程中，我们提出了一种多级融合方法，该方法将不同的表示（RGB和伪LIDAR）和跨多个对象（Tracklets）的时间信息结合在一起，以增强对目标深度估计。我们提出的融合方法实现了Waymo打开数据集，KITTI检测数据集和Kitti MOT数据集的每个对象深度估计的最新性能。我们进一步证明，通过简单地用融合增强的深度替换估计的深度，我们可以在单眼3D感知任务（包括检测和跟踪）方面取得重大改进。

3D object detection is an essential task in autonomous driving. Recent techniques excel with highly accurate detection rates, provided the 3D input data is obtained from precise but expensive LiDAR technology. Approaches based on cheaper monocular or stereo imagery data have, until now, resulted in drastically lower accuracies -a gap that is commonly attributed to poor image-based depth estimation. However, in this paper we argue that it is not the quality of the data but its representation that accounts for the majority of the difference. Taking the inner workings of convolutional neural networks into consideration, we propose to convert image-based depth maps to pseudo-LiDAR representations -essentially mimicking the LiDAR signal. With this representation we can apply different existing LiDAR-based detection algorithms. On the popular KITTI benchmark, our approach achieves impressive improvements over the existing state-of-the-art in image-based performance -raising the detection accuracy of objects within the 30m range from the previous state-of-the-art of 22% to an unprecedented 74%. At the time of submission our algorithm holds the highest entry on the KITTI 3D object detection leaderboard for stereo-image-based approaches. Our code is publicly available at https: //github.com/mileyan/pseudo_lidar.

Figure 1: We introduce datasets for 3D tracking and motion forecasting with rich maps for autonomous driving. Our 3D tracking dataset contains sequences of LiDAR measurements, 360 • RGB video, front-facing stereo (middle-right), and 6-dof localization. All sequences are aligned with maps containing lane center lines (magenta), driveable region (orange), and ground height. Sequences are annotated with 3D cuboid tracks (green). A wider map view is shown in the bottom-right.

In this paper, we propose a novel 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization. Towards this goal, we design an end-to-end learnable architecture that exploits continuous convolutions to fuse image and LIDAR feature maps at different levels of resolution. Our proposed continuous fusion layer encode both discrete-state image features as well as continuous geometric information. This enables us to design a novel, reliable and efficient end-to-end learnable 3D object detector based on multiple sensors. Our experimental evaluation on both KITTI as well as a large scale 3D object detection benchmark shows significant improvements over the state of the art.

在鸟眼中学习强大的表现（BEV），以进行感知任务，这是趋势和吸引行业和学术界的广泛关注。大多数自动驾驶算法的常规方法在正面或透视视图中执行检测，细分，跟踪等。随着传感器配置变得越来越复杂，从不同的传感器中集成了多源信息，并在统一视图中代表功能至关重要。 BEV感知继承了几个优势，因为代表BEV中的周围场景是直观和融合友好的。对于BEV中的代表对象，对于随后的模块，如计划和/或控制是最可取的。 BEV感知的核心问题在于（a）如何通过从透视视图到BEV来通过视图转换来重建丢失的3D信息； （b）如何在BEV网格中获取地面真理注释； （c）如何制定管道以合并来自不同来源和视图的特征； （d）如何适应和概括算法作为传感器配置在不同情况下各不相同。在这项调查中，我们回顾了有关BEV感知的最新工作，并对不同解决方案进行了深入的分析。此外，还描述了该行业的BEV方法的几种系统设计。此外，我们推出了一套完整的实用指南，以提高BEV感知任务的性能，包括相机，激光雷达和融合输入。最后，我们指出了该领域的未来研究指示。我们希望该报告能阐明社区，并鼓励对BEV感知的更多研究。我们保留一个活跃的存储库来收集最新的工作，并在https://github.com/openperceptionx/bevperception-survey-recipe上提供一包技巧的工具箱。

基于LIDAR的传感驱动器电流自主车辆。尽管进展迅速，但目前的激光雷达传感器在分辨率和成本方面仍然落后于传统彩色相机背后的二十年。对于自主驾驶，这意味着靠近传感器的大物体很容易可见，但远方或小物体仅包括一个测量或两个。这是一个问题，尤其是当这些对象结果驾驶危险时。另一方面，在车载RGB传感器中清晰可见这些相同的对象。在这项工作中，我们提出了一种将RGB传感器无缝熔化成基于LIDAR的3D识别方法。我们的方法采用一组2D检测来生成密集的3D虚拟点，以增加否则稀疏的3D点云。这些虚拟点自然地集成到任何基于标准的LIDAR的3D探测器以及常规激光雷达测量。由此产生的多模态检测器简单且有效。大规模NUSCENES数据集的实验结果表明，我们的框架通过显着的6.6地图改善了强大的中心点基线，并且优于竞争融合方法。代码和更多可视化可在https://tianweiy.github.io/mvp/上获得

一方面，在最近的文献中，许多3D多对象跟踪（MOT）的作品集中在跟踪准确性和被忽视的计算速度上，通常是通过设计相当复杂的成本功能和功能提取器来进行的。另一方面，某些方法以跟踪准确性为代价过多地关注计算速度。鉴于这些问题，本文提出了一种强大而快速的基于相机融合的MOT方法，该方法在准确性和速度之间取决于良好的权衡。依靠相机和激光雷达传感器的特性，设计并嵌入了提出的MOT方法中的有效的深层关联机制。该关联机制在对象远处并仅由摄像机检测到2D域中的对象，并在对象出现在LIDAR的视野中以实现平滑融合时获得的2D轨迹进行更新，并更新2D轨迹。 2D和3D轨迹。基于典型数据集的广泛实验表明，就跟踪准确性和处理速度而言，我们提出的方法在最先进的MOT方法上具有明显的优势。我们的代码可公开用于社区的利益。

Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online 1 .

以视觉为中心的BEV感知由于其固有的优点，最近受到行业和学术界的关注，包括展示世界自然代表和融合友好。随着深度学习的快速发展，已经提出了许多方法来解决以视觉为中心的BEV感知。但是，最近没有针对这个小说和不断发展的研究领域的调查。为了刺激其未来的研究，本文对以视觉为中心的BEV感知及其扩展进行了全面调查。它收集并组织了最近的知识，并对常用算法进行了系统的综述和摘要。它还为几项BEV感知任务提供了深入的分析和比较结果，从而促进了未来作品的比较并激发了未来的研究方向。此外，还讨论了经验实现细节并证明有利于相关算法的开发。

In this paper we propose to exploit multiple related tasks for accurate multi-sensor 3D object detection. Towards this goal we present an end-to-end learnable architecture that reasons about 2D and 3D object detection as well as ground estimation and depth completion. Our experiments show that all these tasks are complementary and help the network learn better representations by fusing information at various levels. Importantly, our approach leads the KITTI benchmark on 2D, 3D and bird's eye view object detection, while being real-time. * Equal contribution.† Work done as part of Uber AI Residency program.

近年来，深度学习导致了在城市驾驶场景中移动（即具有运动能力）物体的检测方面取得的巨大进展。监督方法通常需要大型培训集的注释；因此，人们对利用弱，半或自我监督的方法避免这种情况非常兴趣，并取得了很大的成功。虽然弱和半监督的方法需要一些注释，但自我监督的方法已经使用了诸如运动之类的线索来完全减轻注释的需求。但是，完全没有注释通常会降低其性能，而在运动组进行分组期间出现的歧义可以抑制其找到准确的物体边界的能力。在本文中，我们提出了一种称为SCT的新的自制移动对象检测方法。这同时使用运动提示和预期对象大小来提高检测性能，并预测3D方向边界框的密集网格以改善对象发现。我们在Kitti跟踪基准上的最先进的自我监督的移动对象检测方法TCR极大地超过了，并且实现了全面监督的PV-RCNN ++方法的30％以内IOUS <= 0.5。

We propose a model that, given multi-view camera data (left), infers semantics directly in the bird's-eye-view (BEV) coordinate frame (right). We show vehicle segmentation (blue), drivable area (orange), and lane segmentation (green). These BEV predictions are then projected back onto input images (dots on the left).

我们介绍了MGNET，这是一个多任务框架，用于单眼几何场景。我们将单眼几何场景的理解定义为两个已知任务的组合：全景分割和自我监管的单眼深度估计。全景分段不仅在语义上，而且在实例的基础上捕获完整场景。自我监督的单眼深度估计使用摄像机测量模型得出的几何约束，以便从单眼视频序列中测量深度。据我们所知，我们是第一个在一个模型中提出这两个任务的组合的人。我们的模型专注于低潜伏期，以实时在单个消费级GPU上实时提供快速推断。在部署过程中，我们的模型将产生密集的3D点云，其中具有来自单个高分辨率摄像头图像的实例意识到语义标签。我们对两个流行的自动驾驶基准（即CityScapes and Kitti）评估了模型，并在其他能够实时的方法中表现出竞争性能。源代码可从https://github.com/markusschoen/mgnet获得。

Camera and lidar are important sensor modalities for robotics in general and self-driving cars in particular. The sensors provide complementary information offering an opportunity for tight sensor-fusion. Surprisingly, lidar-only methods outperform fusion methods on the main benchmark datasets, suggesting a gap in the literature. In this work, we propose PointPainting: a sequential fusion method to fill this gap. PointPainting works by projecting lidar points into the output of an image-only semantic segmentation network and appending the class scores to each point. The appended (painted) point cloud can then be fed to any lidaronly method. Experiments show large improvements on three different state-of-the art methods, Point-RCNN, Vox-elNet and PointPillars on the KITTI and nuScenes datasets. The painted version of PointRCNN represents a new state of the art on the KITTI leaderboard for the bird's-eye view detection task. In ablation, we study how the effects of Painting depends on the quality and format of the semantic segmentation output, and demonstrate how latency can be minimized through pipelining.

单眼3D对象检测是自主驾驶中的重要任务。在存在自我汽车姿势改变W.R.T的情况下，它可以很容易难以解决。地平面。由于道路平滑度和斜坡的轻微波动，这很常见。由于工业应用缺乏洞察力，开放数据集的现有方法忽略了相机姿势信息，这不可避免地导致探测器易受相机外在参数的影响。物体的扰动在工业产品最自主驾驶案件中非常受欢迎。为此，我们提出了一种捕获摄像机姿势的新方法，以配制无自脉扰动的检测器。具体地，所提出的框架通过检测消失点和地平线改变来预测相机外在参数。转换器旨在纠正潜在空间中的扰动特征。通过这样做，我们的3D探测器独立于外在参数变化，并在现实情况下产生准确的结果，例如坑道和不均匀的道路，几乎所有现有的单眼检测器都无法处理。实验证明我们的方法与基蒂3D和NUSCENES数据集的大型裕度相比，我们的方法与其他最先进的最先进。

虽然在驾驶场景中自我监督的单眼深度估计已经取得了可比性的性能，但违反了静态世界假设的行为仍然可以导致交通参与者的错误深度预测，造成潜在的安全问题。在本文中，我们呈现R4DYN，这是一种新颖的技术，用于在自我监督深度估计框架之上使用成本高效的雷达数据。特别是，我们展示如何在培训期间使用雷达，以及额外的输入，以增强推理时间的估计稳健性。由于汽车雷达很容易获得，这允许从各种现有车辆中收集培训数据。此外，通过过滤和扩展信号以使其与基于学习的方法兼容，我们地满地雷达固有问题，例如噪声和稀疏性。通过R4DYN，我们能够克服自我监督深度估计的一个主要限制，即交通参与者的预测。我们大大提高了动态物体的估计，例如汽车在挑战的NUSCENES数据集中达到37％，因此证明雷达是用于自主车辆中单眼深度估计的有价值的额外传感器。