ML-based motion planning is a promising approach to produce agents that exhibit complex behaviors, and automatically adapt to novel environments. In the context of autonomous driving, it is common to treat all available training data equally. However, this approach produces agents that do not perform robustly in safety-critical settings, an issue that cannot be addressed by simply adding more data to the training set - we show that an agent trained using only a 10% subset of the data performs just as well as an agent trained on the entire dataset. We present a method to predict the inherent difficulty of a driving situation given data collected from a fleet of autonomous vehicles deployed on public roads. We then demonstrate that this difficulty score can be used in a zero-shot transfer to generate curricula for an imitation-learning based planning agent. Compared to training on the entire unbiased training dataset, we show that prioritizing difficult driving scenarios both reduces collisions by 15% and increases route adherence by 14% in closed-loop evaluation, all while using only 10% of the training data.
translated by 谷歌翻译
Imitation learning (IL) is a simple and powerful way to use high-quality human driving data, which can be collected at scale, to identify driving preferences and produce human-like behavior. However, policies based on imitation learning alone often fail to sufficiently account for safety and reliability concerns. In this paper, we show how imitation learning combined with reinforcement learning using simple rewards can substantially improve the safety and reliability of driving policies over those learned from imitation alone. In particular, we use a combination of imitation and reinforcement learning to train a policy on over 100k miles of urban driving data, and measure its effectiveness in test scenarios grouped by different levels of collision risk. To our knowledge, this is the first application of a combined imitation and reinforcement learning approach in autonomous driving that utilizes large amounts of real-world human driving data.
translated by 谷歌翻译
仿真是对机器人系统(例如自动驾驶汽车)进行扩展验证和验证的关键。尽管高保真物理和传感器模拟取得了进步,但在模拟道路使用者的现实行为方面仍然存在一个危险的差距。这是因为,与模拟物理和图形不同,设计人类行为的第一个原理模型通常是不可行的。在这项工作中,我们采用了一种数据驱动的方法,并提出了一种可以学会从现实世界驱动日志中产生流量行为的方法。该方法通过将交通仿真问题分解为高级意图推理和低级驾驶行为模仿,通过利用驾驶行为的双层层次结构来实现高样本效率和行为多样性。该方法还结合了一个计划模块,以获得稳定的长马行为。我们从经验上验证了我们的方法,即交通模拟(位)的双层模仿,并具有来自两个大规模驾驶数据集的场景,并表明位表明,在现实主义,多样性和长途稳定性方面可以达到平衡的交通模拟性能。我们还探索了评估行为现实主义的方法,并引入了一套评估指标以进行交通模拟。最后,作为我们的核心贡献的一部分,我们开发和开源一个软件工具,该工具将跨不同驱动数据集的数据格式统一,并将现有数据集将场景转换为交互式仿真环境。有关其他信息和视频,请参见https://sites.google.com/view/nvr-bits2022/home
translated by 谷歌翻译
With the development of deep representation learning, the domain of reinforcement learning (RL) has become a powerful learning framework now capable of learning complex policies in high dimensional environments. This review summarises deep reinforcement learning (DRL) algorithms and provides a taxonomy of automated driving tasks where (D)RL methods have been employed, while addressing key computational challenges in real world deployment of autonomous driving agents. It also delineates adjacent domains such as behavior cloning, imitation learning, inverse reinforcement learning that are related but are not classical RL algorithms. The role of simulators in training agents, methods to validate, test and robustify existing solutions in RL are discussed.
translated by 谷歌翻译
High-quality traffic flow generation is the core module in building simulators for autonomous driving. However, the majority of available simulators are incapable of replicating traffic patterns that accurately reflect the various features of real-world data while also simulating human-like reactive responses to the tested autopilot driving strategies. Taking one step forward to addressing such a problem, we propose Realistic Interactive TrAffic flow (RITA) as an integrated component of existing driving simulators to provide high-quality traffic flow for the evaluation and optimization of the tested driving strategies. RITA is developed with fidelity, diversity, and controllability in consideration, and consists of two core modules called RITABackend and RITAKit. RITABackend is built to support vehicle-wise control and provide traffic generation models from real-world datasets, while RITAKit is developed with easy-to-use interfaces for controllable traffic generation via RITABackend. We demonstrate RITA's capacity to create diversified and high-fidelity traffic simulations in several highly interactive highway scenarios. The experimental findings demonstrate that our produced RITA traffic flows meet all three design goals, hence enhancing the completeness of driving strategy evaluation. Moreover, we showcase the possibility for further improvement of baseline strategies through online fine-tuning with RITA traffic flows.
translated by 谷歌翻译
Making safe and human-like decisions is an essential capability of autonomous driving systems and learning-based behavior planning is a promising pathway toward this objective. Distinguished from existing learning-based methods that directly output decisions, this work introduces a predictive behavior planning framework that learns to predict and evaluate from human driving data. Concretely, a behavior generation module first produces a diverse set of candidate behaviors in the form of trajectory proposals. Then the proposed conditional motion prediction network is employed to forecast other agents' future trajectories conditioned on each trajectory proposal. Given the candidate plans and associated prediction results, we learn a scoring module to evaluate the plans using maximum entropy inverse reinforcement learning (IRL). We conduct comprehensive experiments to validate the proposed framework on a large-scale real-world urban driving dataset. The results reveal that the conditional prediction model is able to forecast multiple possible future trajectories given a candidate behavior and the prediction results are reactive to different plans. Moreover, the IRL-based scoring module can properly evaluate the trajectory proposals and select close-to-human ones. The proposed framework outperforms other baseline methods in terms of similarity to human driving trajectories. Moreover, we find that the conditional prediction model can improve both prediction and planning performance compared to the non-conditional model, and learning the scoring module is critical to correctly evaluating the candidate plans to align with human drivers.
translated by 谷歌翻译
基于变压器神经网络体系结构的自然语言处理(NLP)的令人印象深刻的结果激发了研究人员探索视线离线增强学习(RL)作为通用序列建模问题。基于此范式的最新著作已获得最新的结果,其中一些主要确定性的离线Atari和D4RL基准。但是,由于这些方法将国家和行动共同模拟单一的测序问题,因此它们努力将政策和世界动态对回报的影响解散。因此,在对抗或随机环境中,这些方法导致过度乐观的行为,在自主驾驶(例如自主驾驶)中可能是危险的。在这项工作中,我们提出了一种通过明确解开政策和世界模型来解决这种乐观偏见的方法,该方法使我们在测试时可以搜索对环境中多个可能的未来的稳健性的策略。我们在模拟中的各种自动驾驶任务上展示了我们的方法的出色性能。
translated by 谷歌翻译
相应地预测周围交通参与者的未来状态,并计划安全,平稳且符合社会的轨迹对于自动驾驶汽车至关重要。当前的自主驾驶系统有两个主要问题:预测模块通常与计划模块解耦,并且计划的成本功能很难指定和调整。为了解决这些问题,我们提出了一个端到端的可区分框架,该框架集成了预测和计划模块,并能够从数据中学习成本函数。具体而言,我们采用可区分的非线性优化器作为运动计划者,该运动计划将神经网络给出的周围剂的预测轨迹作为输入,并优化了自动驾驶汽车的轨迹,从而使框架中的所有操作都可以在框架中具有可观的成本,包括成本功能权重。提出的框架经过大规模的现实驾驶数据集进行了训练,以模仿整个驾驶场景中的人类驾驶轨迹,并在开环和闭环界面中进行了验证。开环测试结果表明,所提出的方法的表现优于各种指标的基线方法,并提供以计划为中心的预测结果,从而使计划模块能够输出接近人类的轨迹。在闭环测试中,提出的方法表明能够处理复杂的城市驾驶场景和鲁棒性,以抵抗模仿学习方法所遭受的分配转移。重要的是,我们发现计划和预测模块的联合培训比在开环和闭环测试中使用单独的训练有素的预测模块进行计划要比计划更好。此外,消融研究表明,框架中的可学习组件对于确保计划稳定性和性能至关重要。
translated by 谷歌翻译
对自动驾驶车辆性能的定量评估,交通模拟引起了很多兴趣。为了使模拟器成为有价值的测试工作台,要求对现场每个交通代理的驾驶策略动画,就像人类在保持最小安全保证的同时一样。从记录的人类驾驶数据或通过强化学习中学习交通代理的驾驶政策似乎是在不受控制的交叉路口或回旋处中产生现实且高度互动的交通状况的有吸引力的解决方案。在这项工作中,我们表明,在学习驾驶政策时模仿人类驾驶与保持安全性之间存在权衡。我们通过比较应用于驾驶任务时的各种模仿学习和强化学习算法的性能来做到这一点。我们还提出了一种多物镜学习算法(MOPPO),可以共同提高两个目标。我们在从交互数据集中提取的高度互动驾驶方案上测试驾驶政策,以评估它们的表现如何。
translated by 谷歌翻译
Autonomous vehicle (AV) stacks are typically built in a modular fashion, with explicit components performing detection, tracking, prediction, planning, control, etc. While modularity improves reusability, interpretability, and generalizability, it also suffers from compounding errors, information bottlenecks, and integration challenges. To overcome these challenges, a prominent approach is to convert the AV stack into an end-to-end neural network and train it with data. While such approaches have achieved impressive results, they typically lack interpretability and reusability, and they eschew principled analytical components, such as planning and control, in favor of deep neural networks. To enable the joint optimization of AV stacks while retaining modularity, we present DiffStack, a differentiable and modular stack for prediction, planning, and control. Crucially, our model-based planning and control algorithms leverage recent advancements in differentiable optimization to produce gradients, enabling optimization of upstream components, such as prediction, via backpropagation through planning and control. Our results on the nuScenes dataset indicate that end-to-end training with DiffStack yields substantial improvements in open-loop and closed-loop planning metrics by, e.g., learning to make fewer prediction errors that would affect planning. Beyond these immediate benefits, DiffStack opens up new opportunities for fully data-driven yet modular and interpretable AV architectures. Project website: https://sites.google.com/view/diffstack
translated by 谷歌翻译
离线强化学习(RL)为从离线数据提供学习决策的框架,因此构成了现实世界应用程序作为自动驾驶的有希望的方法。自动驾驶车辆(SDV)学习策略,这甚至可能甚至优于次优数据集中的行为。特别是在安全关键应用中,作为自动化驾驶,解释性和可转换性是成功的关键。这激发了使用基于模型的离线RL方法,该方法利用规划。然而,目前的最先进的方法往往忽视了多种子体系统随机行为引起的溶液不确定性的影响。这项工作提出了一种新的基于不确定感知模型的离线强化学习利用规划(伞)的新方法,其解决了以可解释的基于学习的方式共同的预测,规划和控制问题。训练有素的动作调节的随机动力学模型捕获了交通场景的独特不同的未来演化。分析为我们在挑战自动化驾驶模拟中的效力和基于现实世界的公共数据集的方法提供了经验证据。
translated by 谷歌翻译
The goal of autonomous vehicles is to navigate public roads safely and comfortably. To enforce safety, traditional planning approaches rely on handcrafted rules to generate trajectories. Machine learning-based systems, on the other hand, scale with data and are able to learn more complex behaviors. However, they often ignore that agents and self-driving vehicle trajectory distributions can be leveraged to improve safety. In this paper, we propose modeling a distribution over multiple future trajectories for both the self-driving vehicle and other road agents, using a unified neural network architecture for prediction and planning. During inference, we select the planning trajectory that minimizes a cost taking into account safety and the predicted probabilities. Our approach does not depend on any rule-based planners for trajectory generation or optimization, improves with more training data and is simple to implement. We extensively evaluate our method through a realistic simulator and show that the predicted trajectory distribution corresponds to different driving profiles. We also successfully deploy it on a self-driving vehicle on urban public roads, confirming that it drives safely without compromising comfort. The code for training and testing our model on a public prediction dataset and the video of the road test are available at https://woven.mobi/safepathnet
translated by 谷歌翻译
Designing a safe and human-like decision-making system for an autonomous vehicle is a challenging task. Generative imitation learning is one possible approach for automating policy-building by leveraging both real-world and simulated decisions. Previous work that applies generative imitation learning to autonomous driving policies focuses on learning a low-level controller for simple settings. However, to scale to complex settings, many autonomous driving systems combine fixed, safe, optimization-based low-level controllers with high-level decision-making logic that selects the appropriate task and associated controller. In this paper, we attempt to bridge this gap in complexity by employing Safety-Aware Hierarchical Adversarial Imitation Learning (SHAIL), a method for learning a high-level policy that selects from a set of low-level controller instances in a way that imitates low-level driving data on-policy. We introduce an urban roundabout simulator that controls non-ego vehicles using real data from the Interaction dataset. We then demonstrate empirically that even with simple controller options, our approach can produce better behavior than previous approaches in driver imitation that have difficulty scaling to complex environments. Our implementation is available at https://github.com/sisl/InteractionImitation.
translated by 谷歌翻译
我们介绍了\ textit {nocturne},这是一种新的2D驾驶模拟器,用于调查部分可观察性下的多代理协调。夜曲的重点是在不具有计算机视觉的计算开销并从图像中提取特征的情况下,在现实世界中的推理和心理理论方面进行研究。该模拟器中的代理只会观察到场景的障碍,模仿人类的视觉传感限制。 Unlike existing benchmarks that are bottlenecked by rendering human-like observations directly using a camera input, Nocturne uses efficient intersection methods to compute a vectorized set of visible features in a C++ back-end, allowing the simulator to run at $2000+$ steps-per -第二。使用开源轨迹和映射数据,我们构建了一个模拟器,以加载和重播来自现实世界驾驶数据的任意轨迹和场景。使用这种环境,我们基准了加强学习和模仿学习剂,并证明这些代理远离人类水平的协调能力,并显着偏离专家轨迹。
translated by 谷歌翻译
在本文中,我们提出了一个系统,以培训不仅从自我车辆收集的经验,而且还观察到的所有车辆的经验。该系统使用其他代理的行为来创建更多样化的驾驶场景,而无需收集其他数据。从其他车辆学习的主要困难是没有传感器信息。我们使用一组监督任务来学习一个中间表示,这是对控制车辆的观点不变的。这不仅在训练时间提供了更丰富的信号,而且还可以在推断过程中进行更复杂的推理。了解所有车辆驾驶如何有助于预测测试时的行为,并避免碰撞。我们在闭环驾驶模拟中评估该系统。我们的系统的表现优于公共卡拉排行榜上的所有先前方法,较大的利润率将驾驶得分提高了25,路线完成率提高了24分。我们的方法赢得了2021年的卡拉自动驾驶挑战。代码和数据可在https://github.com/dotchen/lav上获得。
translated by 谷歌翻译
仅国家模仿学习的最新进展将模仿学习的适用性扩展到现实世界中的范围,从而减轻了观察专家行动的需求。但是,现有的解决方案只学会从数据中提取州对行动映射策略,而无需考虑专家如何计划到目标。这阻碍了利用示威游行并限制政策的灵活性的能力。在本文中,我们介绍了解耦政策优化(DEPO),该策略优化(DEPO)明确将策略脱离为高级状态计划者和逆动力学模型。借助嵌入式的脱钩策略梯度和生成对抗训练,DEPO可以将知识转移到不同的动作空间或状态过渡动态,并可以将规划师推广到无示威的状态区域。我们的深入实验分析表明,DEPO在学习最佳模仿性能的同时学习通用目标状态计划者的有效性。我们证明了DEPO通过预训练跨任务转移的吸引力,以及与各种技能共同培训的潜力。
translated by 谷歌翻译
在多机构动态交通情况下的自主驾驶具有挑战性:道路使用者的行为不确定,很难明确建模,并且自我车辆应与他们应用复杂的谈判技巧,例如屈服,合并和交付,以实现,以实现在各种环境中都有安全有效的驾驶。在这些复杂的动态场景中,传统的计划方法主要基于规则,并且通常会导致反应性甚至过于保守的行为。因此,他们需要乏味的人类努力来维持可行性。最近,基于深度学习的方法显示出令人鼓舞的结果,具有更好的概括能力,但手工工程的工作较少。但是,它们要么是通过有监督的模仿学习(IL)来实施的,该学习遭受了数据集偏见和分配不匹配问题,要么接受了深入强化学习(DRL)的培训,但专注于一种特定的交通情况。在这项工作中,我们建议DQ-GAT实现可扩展和主动的自主驾驶,在这些驾驶中,基于图形注意力的网络用于隐式建模相互作用,并采用了深层Q学习来以无聊的方式训练网络端到端的网络。 。在高保真驾驶模拟器中进行的广泛实验表明,我们的方法比以前的基于学习的方法和传统的基于规则的方法获得了更高的成功率,并且在可见和看不见的情况下都可以更好地摆脱安全性和效率。此外,轨迹数据集的定性结果表明,我们所学的政策可以通过实时速度转移到现实世界中。演示视频可在https://caipeide.github.io/dq-gat/上找到。
translated by 谷歌翻译
自治车辆的评估和改善规划需要可扩展的长尾交通方案。有用的是,这些情景必须是现实的和挑战性的,但不能安全地开车。在这项工作中,我们介绍努力,一种自动生成具有挑战性的场景的方法,导致给定的计划者产生不良行为,如冲突。为了维护情景合理性,关键的想法是利用基于图形的条件VAE的形式利用学习的交通运动模型。方案生成在该流量模型的潜在空间中制定了优化,通过扰乱初始的真实世界的场景来产生与给定计划者碰撞的轨迹。随后的优化用于找到“解决方案”的场景,确保改进给定的计划者是有用的。进一步的分析基于碰撞类型的群集生成的场景。我们攻击两名策划者并展示争取在这两种情况下成功地产生了现实,具有挑战性的情景。我们另外“关闭循环”并使用这些方案优化基于规则的策划器的超参数。
translated by 谷歌翻译
人类行为的不确定性对拥挤的城市环境中的自动驾驶构成了重大挑战。部分可观察到的马尔可夫决策过程(POMDP)为不确定性下的计划提供了一个原则的框架,通常利用蒙特卡洛抽样来实现在线绩效进行复杂的任务。但是,抽样还通过潜在缺失关键事件引起了安全问题。为了解决这个问题,我们提出了一种新的算法,学习对驾驶行为(领导者)的关注,这些算法在计划过程中学习了批判性人类行为。领导者学习了一个神经网络生成器,以实时情况下对人类行为的关注。它将注意力集成到信仰空间计划者中,使用重要性抽样来偏向关键事件。为了训练该算法,我们让注意力生成器和计划者组成了最小游戏。通过解决Min-Max游戏,领导者学会了无需人类标签即可执行风险意识的计划。
translated by 谷歌翻译
自动驾驶汽车的一个主要挑战是安全,平稳地与其他交通参与者进行互动。处理此类交通交互的一种有希望的方法是为自动驾驶汽车配备与感知的控制器(IACS)。这些控制器预测,周围人类驾驶员将如何根据驾驶员模型对自动驾驶汽车的行为做出响应。但是,很少验证IACS中使用的驱动程序模型的预测有效性,这可能会限制IACS在简单的模拟环境之外的交互功能。在本文中,我们认为,除了评估IAC的互动能力外,还应在自然的人类驾驶行为上验证其潜在的驱动器模型。我们为此验证提出了一个工作流程,其中包括基于方案的数据提取和基于人为因素文献的两阶段(战术/操作)评估程序。我们在一项案例研究中证明了该工作流程,该案例研究对现有IAC复制的基于反向的基于学习的驱动程序模型。该模型仅在40%的预测中显示出正确的战术行为。该模型的操作行为与观察到的人类行为不一致。案例研究表明,有原则的评估工作流程是有用和需要的。我们认为,我们的工作流将支持为将来的自动化车辆开发适当的驾驶员模型。
translated by 谷歌翻译