自动驾驶的运动预测是一项艰巨的任务，因为复杂的驾驶场景导致静态和动态输入的异质组合。这是一个开放的问题，如何最好地表示和融合有关道路几何，车道连接，时变的交通信号状态以及动态代理的历史及其相互作用的历史。为了模拟这一不同的输入功能集，许多提出的方法旨在设计具有多种模态模块的同样复杂系统。这导致难以按严格的方式进行扩展，扩展或调整的系统以进行质量和效率。在本文中，我们介绍了Wayformer，这是一个基于注意力的运动架构，用于运动预测，简单而均匀。 Wayformer提供了一个紧凑的模型描述，该描述由基于注意力的场景编码器和解码器组成。在场景编码器中，我们研究了输入方式的早期，晚和等级融合的选择。对于每种融合类型，我们通过分解的注意力或潜在的查询关注来探索策略来折衷效率和质量。我们表明，尽管早期融合的结构简单，但不仅是情感不可知论，而且还取得了最先进的结果。

预测道路用户的未来行为是自主驾驶中最具挑战性和最重要的问题之一。应用深度学习对此问题需要以丰富的感知信号和地图信息的形式融合异构世界状态，并在可能的期货上推断出高度多模态分布。在本文中，我们呈现MultiPath ++，这是一个未来的预测模型，实现了在流行的基准上实现最先进的性能。 MultiPath ++通过重新访问许多设计选择来改善多径架构。第一关键设计差异是偏离基于图像的基于输入世界状态的偏离，有利于异构场景元素的稀疏编码：多径++消耗紧凑且有效的折线，直接描述道路特征和原始代理状态信息（例如，位置，速度，加速）。我们提出了一种背景感知这些元素的融合，并开发可重用的多上下文选通融合组件。其次，我们重新考虑了预定义，静态锚点的选择，并开发了一种学习模型端到端的潜在锚嵌入的方法。最后，我们在其他ML域中探索合奏和输出聚合技术 - 常见的常见域 - 并为我们的概率多模式输出表示找到有效的变体。我们对这些设计选择进行了广泛的消融，并表明我们所提出的模型在协会运动预测竞争和Waymo开放数据集运动预测挑战上实现了最先进的性能。

预测交通参与者的多模式未来行为对于机器人车辆做出安全决策至关重要。现有作品探索以直接根据潜在特征预测未来的轨迹，或利用密集的目标候选者来识别代理商的目的地，在这种情况下，由于所有运动模式均来自相同的功能，而后者的策略具有效率问题，因此前者策略的收敛缓慢，因为其性能高度依赖关于候选目标的密度。在本文中，我们提出了运动变压器（MTR）框架，该框架将运动预测模拟为全球意图定位和局部运动改进的联合优化。 MTR不使用目标候选者，而是通过采用一系列可学习的运动查询对来结合空间意图。每个运动查询对负责特定运动模式的轨迹预测和完善，这可以稳定训练过程并促进更好的多模式预测。实验表明，MTR在边际和联合运动预测挑战上都达到了最新的性能，在Waymo Open Motion DataSet排行榜上排名第一。代码将在https://github.com/sshaoshuai/mtr上找到。

Modern autonomous driving system is characterized as modular tasks in sequential order, i.e., perception, prediction and planning. As sensors and hardware get improved, there is trending popularity to devise a system that can perform a wide diversity of tasks to fulfill higher-level intelligence. Contemporary approaches resort to either deploying standalone models for individual tasks, or designing a multi-task paradigm with separate heads. These might suffer from accumulative error or negative transfer effect. Instead, we argue that a favorable algorithm framework should be devised and optimized in pursuit of the ultimate goal, i.e. planning of the self-driving-car. Oriented at this goal, we revisit the key components within perception and prediction. We analyze each module and prioritize the tasks hierarchically, such that all these tasks contribute to planning (the goal). To this end, we introduce Unified Autonomous Driving (UniAD), the first comprehensive framework up-to-date that incorporates full-stack driving tasks in one network. It is exquisitely devised to leverage advantages of each module, and provide complementary feature abstractions for agent interaction from a global perspective. Tasks are communicated with unified query design to facilitate each other toward planning. We instantiate UniAD on the challenging nuScenes benchmark. With extensive ablations, the effectiveness of using such a philosophy is proven to surpass previous state-of-the-arts by a large margin in all aspects. The full suite of codebase and models would be available to facilitate future research in the community.

从社交机器人到自动驾驶汽车，多种代理的运动预测（MP）是任意复杂环境中的至关重要任务。当前方法使用端到端网络解决了此问题，其中输入数据通常是场景的最高视图和所有代理的过去轨迹；利用此信息是获得最佳性能的必不可少的。从这个意义上讲，可靠的自动驾驶（AD）系统必须按时产生合理的预测，但是，尽管其中许多方法使用了简单的Convnets和LSTM，但在使用两个信息源时，模型对于实时应用程序可能不够有效（地图和轨迹历史）。此外，这些模型的性能在很大程度上取决于训练数据的数量，这可能很昂贵（尤其是带注释的HD地图）。在这项工作中，我们探讨了如何使用有效的基于注意力的模型在Argoverse 1.0基准上实现竞争性能，该模型将其作为最小地图信息的过去轨迹和基于地图的功能的输入，以确保有效且可靠的MP。这些功能代表可解释的信息作为可驱动区域和合理的目标点，与基于黑框CNN的地图处理方法相反。

The task of motion forecasting is critical for self-driving vehicles (SDVs) to be able to plan a safe maneuver. Towards this goal, modern approaches reason about the map, the agents' past trajectories and their interactions in order to produce accurate forecasts. The predominant approach has been to encode the map and other agents in the reference frame of each target agent. However, this approach is computationally expensive for multi-agent prediction as inference needs to be run for each agent. To tackle the scaling challenge, the solution thus far has been to encode all agents and the map in a shared coordinate frame (e.g., the SDV frame). However, this is sample inefficient and vulnerable to domain shift (e.g., when the SDV visits uncommon states). In contrast, in this paper, we propose an efficient shared encoding for all agents and the map without sacrificing accuracy or generalization. Towards this goal, we leverage pair-wise relative positional encodings to represent geometric relationships between the agents and the map elements in a heterogeneous spatial graph. This parameterization allows us to be invariant to scene viewpoint, and save online computation by re-using map embeddings computed offline. Our decoder is also viewpoint agnostic, predicting agent goals on the lane graph to enable diverse and context-aware multimodal prediction. We demonstrate the effectiveness of our approach on the urban Argoverse 2 benchmark as well as a novel highway dataset.

Reasoning about human motion is an important prerequisite to safe and socially-aware robotic navigation. As a result, multi-agent behavior prediction has become a core component of modern human-robot interactive systems, such as self-driving cars. While there exist many methods for trajectory forecasting, most do not enforce dynamic constraints and do not account for environmental information (e.g., maps). Towards this end, we present Trajectron++, a modular, graph-structured recurrent model that forecasts the trajectories of a general number of diverse agents while incorporating agent dynamics and heterogeneous data (e.g., semantic maps). Trajectron++ is designed to be tightly integrated with robotic planning and control frameworks; for example, it can produce predictions that are optionally conditioned on ego-agent motion plans. We demonstrate its performance on several challenging real-world trajectory forecasting datasets, outperforming a wide array of state-ofthe-art deterministic and generative methods.

Motion prediction systems aim to capture the future behavior of traffic scenarios enabling autonomous vehicles to perform safe and efficient planning. The evolution of these scenarios is highly uncertain and depends on the interactions of agents with static and dynamic objects in the scene. GNN-based approaches have recently gained attention as they are well suited to naturally model these interactions. However, one of the main challenges that remains unexplored is how to address the complexity and opacity of these models in order to deal with the transparency requirements for autonomous driving systems, which includes aspects such as interpretability and explainability. In this work, we aim to improve the explainability of motion prediction systems by using different approaches. First, we propose a new Explainable Heterogeneous Graph-based Policy (XHGP) model based on an heterograph representation of the traffic scene and lane-graph traversals, which learns interaction behaviors using object-level and type-level attention. This learned attention provides information about the most important agents and interactions in the scene. Second, we explore this same idea with the explanations provided by GNNExplainer. Third, we apply counterfactual reasoning to provide explanations of selected individual scenarios by exploring the sensitivity of the trained model to changes made to the input data, i.e., masking some elements of the scene, modifying trajectories, and adding or removing dynamic agents. The explainability analysis provided in this paper is a first step towards more transparent and reliable motion prediction systems, important from the perspective of the user, developers and regulatory agencies. The code to reproduce this work is publicly available at https://github.com/sancarlim/Explainable-MP/tree/v1.1.

Making safe and human-like decisions is an essential capability of autonomous driving systems and learning-based behavior planning is a promising pathway toward this objective. Distinguished from existing learning-based methods that directly output decisions, this work introduces a predictive behavior planning framework that learns to predict and evaluate from human driving data. Concretely, a behavior generation module first produces a diverse set of candidate behaviors in the form of trajectory proposals. Then the proposed conditional motion prediction network is employed to forecast other agents' future trajectories conditioned on each trajectory proposal. Given the candidate plans and associated prediction results, we learn a scoring module to evaluate the plans using maximum entropy inverse reinforcement learning (IRL). We conduct comprehensive experiments to validate the proposed framework on a large-scale real-world urban driving dataset. The results reveal that the conditional prediction model is able to forecast multiple possible future trajectories given a candidate behavior and the prediction results are reactive to different plans. Moreover, the IRL-based scoring module can properly evaluate the trajectory proposals and select close-to-human ones. The proposed framework outperforms other baseline methods in terms of similarity to human driving trajectories. Moreover, we find that the conditional prediction model can improve both prediction and planning performance compared to the non-conditional model, and learning the scoring module is critical to correctly evaluating the candidate plans to align with human drivers.

近年来，行为预测模型已经激增，尤其是在自动驾驶的流行现实机器人技术应用中，代表移动代理可能未来的分布对于安全舒适的运动计划至关重要。在这些模型中，选择代表输入和输出的坐标框架的选择具有至关重要的交易折扣，这些折扣通常属于两个类别之一。以代理为中心的模型转换输入并在以代理为中心的坐标中执行推断。这些模型在场景元素之间的翻译和旋转上本质上不变，在公共排行榜上表现最好，但与代理和场景元素的数量相互缩小。以场景为中心的模型使用固定的坐标系来处理所有代理。这为他们提供了在所有代理之间共享表示形式的优势，并提供有效的摊销推理计算，该计算与代理数量线性缩放。但是，这些模型必须学习场景元素之间的翻译和旋转的不变性，并且通常以表现为中心的模型。在这项工作中，我们在概率运动预测模型之间开发知识蒸馏技术，并应用这些技术来缩小以代理为中心和以场景为中心的模型之间的性能差距。这将以场景为中心的模型性能提高了13.2％，在公共Argoverse基准中，Waymo Open Datatet的7.8％，在大型内部数据集中最多可达9.4％。这些以场景为中心的改进的模型在公共排行榜中排名很高，在繁忙场景中以代理商为中心的教师的效率高15倍。

随着机器学习模型在自动驾驶汽车（AV）的运动预测系统上变得越来越普遍，至关重要的是，我们必须确保模型预测是安全可靠的。但是，详尽地收集和标记充分测试稀有和挑战性场景的长尾所需的数据是困难且昂贵的。在这项工作中，我们构建了一个新的基准测试，用于通过将扰动应用于现有数据来评估和改善模型鲁棒性。具体而言，我们进行了广泛的标签努力，以识别因果因素，或者在Waymo Open Motion数据集（WOMD）中以任何方式影响人类驾驶员行为的代理，我们使用这些标签来通过删除非carusal剂来扰动数据从现场。然后，我们在我们提出的基准上评估了一套各种最先进的深度学习模型体系结构，并发现所有模型在扰动下均显示出很大的变化。在非作业扰动下，我们观察到$ 25 $ - $ 38 \％$ $相对变化，而与原始相比。然后，我们研究以提高模型鲁棒性的技术，包括增加训练数据集的大小以及使用靶向数据增强，这些数据增加在整个培训过程中都放下了代理。我们计划提供因果代理标签作为womd的附加属性，并释放稳健性基准，以帮助社区建立更可靠和安全的深度学习模型，以进行运动预测。

准确地预测占用和流量对于在复杂的交通情况下为自动驾驶汽车提供更好的安全性和互动至关重要。这项工作提出了Strajnet：一个多模式的SWIN变压框架，用于有效的场景占用和流动预测。我们采用Swin Transformer编码图像和相互作用感知运动表示形式，并提出一个交叉意识模块，以在不同的时间步长跨不同时间步骤将运动意识注入网格单元。然后通过颞膜金字塔解码器来解码流量和占用预测。所提出的方法在Waymo Open数据集基准中显示了竞争性预测准确性和其他评估指标。

为了安全和合理地参与密集和异质的交通，自动驾驶汽车需要充分分析周围交通代理的运动模式，并准确预测其未来的轨迹。这是具有挑战性的，因为交通代理的轨迹不仅受交通代理本身的影响，而且还受到彼此的空间互动的影响。以前的方法通常依赖于长期短期存储网络（LSTMS）的顺序逐步处理，并仅提取单型交通代理之间的空间邻居之间的相互作用。我们提出了时空变压器网络（S2TNET），该网络通过时空变压器对时空相互作用进行建模，并通过时间变压器处理颞序序列。我们将其他类别，形状和标题信息输入到我们的网络中，以处理交通代理的异质性。在Apolloscape轨迹数据集上，所提出的方法在平均值和最终位移误差的加权总和上优于Apolloscape轨迹数据集的最先进方法。我们的代码可在https://github.com/chenghuang66/s2tnet上找到。

仿真是对机器人系统（例如自动驾驶汽车）进行扩展验证和验证的关键。尽管高保真物理和传感器模拟取得了进步，但在模拟道路使用者的现实行为方面仍然存在一个危险的差距。这是因为，与模拟物理和图形不同，设计人类行为的第一个原理模型通常是不可行的。在这项工作中，我们采用了一种数据驱动的方法，并提出了一种可以学会从现实世界驱动日志中产生流量行为的方法。该方法通过将交通仿真问题分解为高级意图推理和低级驾驶行为模仿，通过利用驾驶行为的双层层次结构来实现高样本效率和行为多样性。该方法还结合了一个计划模块，以获得稳定的长马行为。我们从经验上验证了我们的方法，即交通模拟（位）的双层模仿，并具有来自两个大规模驾驶数据集的场景，并表明位表明，在现实主义，多样性和长途稳定性方面可以达到平衡的交通模拟性能。我们还探索了评估行为现实主义的方法，并引入了一套评估指标以进行交通模拟。最后，作为我们的核心贡献的一部分，我们开发和开源一个软件工具，该工具将跨不同驱动数据集的数据格式统一，并将现有数据集将场景转换为交互式仿真环境。有关其他信息和视频，请参见https://sites.google.com/view/nvr-bits2022/home

Accurately predicting interactive road agents' future trajectories and planning a socially compliant and human-like trajectory accordingly are important for autonomous vehicles. In this paper, we propose a planning-centric prediction neural network, which takes surrounding agents' historical states and map context information as input, and outputs the joint multi-modal prediction trajectories for surrounding agents, as well as a sequence of control commands for the ego vehicle by imitation learning. An agent-agent interaction module along the time axis is proposed in our network architecture to better comprehend the relationship among all the other intelligent agents on the road. To incorporate the map's topological information, a Dynamic Graph Convolutional Neural Network (DGCNN) is employed to process the road network topology. Besides, the whole architecture can serve as a backbone for the Differentiable Integrated motion Prediction with Planning (DIPP) method by providing accurate prediction results and initial planning commands. Experiments are conducted on real-world datasets to demonstrate the improvements made by our proposed method in both planning and prediction accuracy compared to the previous state-of-the-art methods.

行为预测在集成自主驾驶软件解决方案中起着重要作用。在行为预测研究中，与单一代理行为预测相比，交互行为预测是一个较小的领域。预测互动剂的运动需要启动新的机制来捕获交互式对的关节行为。在这项工作中，我们将端到端的关节预测问题作为边际学习和车辆行为联合学习的顺序学习过程。我们提出了ProspectNet，这是一个采用加权注意分数的联合学习块，以模拟交互式剂对之间的相互影响。联合学习块首先权衡多模式预测的候选轨迹，然后通过交叉注意更新自我代理的嵌入。此外，我们将每个交互式代理的个人未来预测播放到一个智慧评分模块中，以选择顶部的$ K $预测对。我们表明，ProspectNet优于两个边际预测的笛卡尔产品，并在Waymo交互式运动预测基准上实现了可比的性能。

相应地预测周围交通参与者的未来状态，并计划安全，平稳且符合社会的轨迹对于自动驾驶汽车至关重要。当前的自主驾驶系统有两个主要问题：预测模块通常与计划模块解耦，并且计划的成本功能很难指定和调整。为了解决这些问题，我们提出了一个端到端的可区分框架，该框架集成了预测和计划模块，并能够从数据中学习成本函数。具体而言，我们采用可区分的非线性优化器作为运动计划者，该运动计划将神经网络给出的周围剂的预测轨迹作为输入，并优化了自动驾驶汽车的轨迹，从而使框架中的所有操作都可以在框架中具有可观的成本，包括成本功能权重。提出的框架经过大规模的现实驾驶数据集进行了训练，以模仿整个驾驶场景中的人类驾驶轨迹，并在开环和闭环界面中进行了验证。开环测试结果表明，所提出的方法的表现优于各种指标的基线方法，并提供以计划为中心的预测结果，从而使计划模块能够输出接近人类的轨迹。在闭环测试中，提出的方法表明能够处理复杂的城市驾驶场景和鲁棒性，以抵抗模仿学习方法所遭受的分配转移。重要的是，我们发现计划和预测模块的联合培训比在开环和闭环测试中使用单独的训练有素的预测模块进行计划要比计划更好。此外，消融研究表明，框架中的可学习组件对于确保计划稳定性和性能至关重要。

Figure 1: We introduce datasets for 3D tracking and motion forecasting with rich maps for autonomous driving. Our 3D tracking dataset contains sequences of LiDAR measurements, 360 • RGB video, front-facing stereo (middle-right), and 6-dof localization. All sequences are aligned with maps containing lane center lines (magenta), driveable region (orange), and ground height. Sequences are annotated with 3D cuboid tracks (green). A wider map view is shown in the bottom-right.

Predicting the future motion of dynamic agents is of paramount importance to ensure safety or assess risks in motion planning for autonomous robots. In this paper, we propose a two-stage motion prediction method, referred to as R-Pred, that effectively utilizes both the scene and interaction context using a cascade of the initial trajectory proposal network and the trajectory refinement network. The initial trajectory proposal network produces M trajectory proposals corresponding to M modes of a future trajectory distribution. The trajectory refinement network enhances each of M proposals using 1) the tube-query scene attention (TQSA) and 2) the proposal-level interaction attention (PIA). TQSA uses tube-queries to aggregate the local scene context features pooled from proximity around the trajectory proposals of interest. PIA further enhances the trajectory proposals by modeling inter-agent interactions using a group of trajectory proposals selected based on their distances from neighboring agents. Our experiments conducted on the Argoverse and nuScenes datasets demonstrate that the proposed refinement network provides significant performance improvements compared to the single-stage baseline and that R-Pred achieves state-of-the-art performance in some categories of the benchmark.

We propose JFP, a Joint Future Prediction model that can learn to generate accurate and consistent multi-agent future trajectories. For this task, many different methods have been proposed to capture social interactions in the encoding part of the model, however, considerably less focus has been placed on representing interactions in the decoder and output stages. As a result, the predicted trajectories are not necessarily consistent with each other, and often result in unrealistic trajectory overlaps. In contrast, we propose an end-to-end trainable model that learns directly the interaction between pairs of agents in a structured, graphical model formulation in order to generate consistent future trajectories. It sets new state-of-the-art results on Waymo Open Motion Dataset (WOMD) for the interactive setting. We also investigate a more complex multi-agent setting for both WOMD and a larger internal dataset, where our approach improves significantly on the trajectory overlap metrics while obtaining on-par or better performance on single-agent trajectory metrics.