基于学习的导航系统广泛用于自主应用，例如机器人，无人驾驶车辆和无人机。已经提出了专门的硬件加速器，以实现这种导航任务的高性能和能效。然而，硬件系统中的瞬态和永久性故障正在增加，并且可以灾难性地违反任务安全。同时，传统的基于冗余的保护方法挑战，用于部署资源受限的边缘应用。在本文中，我们通过从RL训练和推理的算法，对算法，故障模型和数据类型进行了实验评估导航系统的恢复性。我们进一步提出了两种有效的故障缓解技术，实现了基于学习的导航系统的2倍成功率和39％的飞行质量改进。

Quantum computing (QC) promises significant advantages on certain hard computational tasks over classical computers. However, current quantum hardware, also known as noisy intermediate-scale quantum computers (NISQ), are still unable to carry out computations faithfully mainly because of the lack of quantum error correction (QEC) capability. A significant amount of theoretical studies have provided various types of QEC codes; one of the notable topological codes is the surface code, and its features, such as the requirement of only nearest-neighboring two-qubit control gates and a large error threshold, make it a leading candidate for scalable quantum computation. Recent developments of machine learning (ML)-based techniques especially the reinforcement learning (RL) methods have been applied to the decoding problem and have already made certain progress. Nevertheless, the device noise pattern may change over time, making trained decoder models ineffective. In this paper, we propose a continual reinforcement learning method to address these decoding challenges. Specifically, we implement double deep Q-learning with probabilistic policy reuse (DDQN-PPR) model to learn surface code decoding strategies for quantum environments with varying noise patterns. Through numerical simulations, we show that the proposed DDQN-PPR model can significantly reduce the computational complexity. Moreover, increasing the number of trained policies can further improve the agent's performance. Our results open a way to build more capable RL agents which can leverage previously gained knowledge to tackle QEC challenges.

The high emission and low energy efficiency caused by internal combustion engines (ICE) have become unacceptable under environmental regulations and the energy crisis. As a promising alternative solution, multi-power source electric vehicles (MPS-EVs) introduce different clean energy systems to improve powertrain efficiency. The energy management strategy (EMS) is a critical technology for MPS-EVs to maximize efficiency, fuel economy, and range. Reinforcement learning (RL) has become an effective methodology for the development of EMS. RL has received continuous attention and research, but there is still a lack of systematic analysis of the design elements of RL-based EMS. To this end, this paper presents an in-depth analysis of the current research on RL-based EMS (RL-EMS) and summarizes the design elements of RL-based EMS. This paper first summarizes the previous applications of RL in EMS from five aspects: algorithm, perception scheme, decision scheme, reward function, and innovative training method. The contribution of advanced algorithms to the training effect is shown, the perception and control schemes in the literature are analyzed in detail, different reward function settings are classified, and innovative training methods with their roles are elaborated. Finally, by comparing the development routes of RL and RL-EMS, this paper identifies the gap between advanced RL solutions and existing RL-EMS. Finally, this paper suggests potential development directions for implementing advanced artificial intelligence (AI) solutions in EMS.

在包装交付，交通监控，搜索和救援操作以及军事战斗订婚等不同应用中，对使用无人驾驶汽车（UAV）（无人机）的需求越来越不断增加。在所有这些应用程序中，无人机用于自动导航环境 - 没有人类互动，执行特定任务并避免障碍。自主无人机导航通常是使用强化学习（RL）来完成的，在该学习中，代理在域中充当专家在避免障碍的同时导航环境。了解导航环境和算法限制在选择适当的RL算法以有效解决导航问题方面起着至关重要的作用。因此，本研究首先确定了无人机导航任务，并讨论导航框架和仿真软件。接下来，根据环境，算法特征，能力和不同无人机导航问题的应用程序对RL算法进行分类和讨论，这将帮助从业人员和研究人员为其无人机导航使用情况选择适当的RL算法。此外，确定的差距和机会将推动无人机导航研究。

Development of navigation algorithms is essential for the successful deployment of robots in rapidly changing hazardous environments for which prior knowledge of configuration is often limited or unavailable. Use of traditional path-planning algorithms, which are based on localization and require detailed obstacle maps with goal locations, is not possible. In this regard, vision-based algorithms hold great promise, as visual information can be readily acquired by a robot's onboard sensors and provides a much richer source of information from which deep neural networks can extract complex patterns. Deep reinforcement learning has been used to achieve vision-based robot navigation. However, the efficacy of these algorithms in environments with dynamic obstacles and high variation in the configuration space has not been thoroughly investigated. In this paper, we employ a deep Dyna-Q learning algorithm for room evacuation and obstacle avoidance in partially observable environments based on low-resolution raw image data from an onboard camera. We explore the performance of a robotic agent in environments containing no obstacles, convex obstacles, and concave obstacles, both static and dynamic. Obstacles and the exit are initialized in random positions at the start of each episode of reinforcement learning. Overall, we show that our algorithm and training approach can generalize learning for collision-free evacuation of environments with complex obstacle configurations. It is evident that the agent can navigate to a goal location while avoiding multiple static and dynamic obstacles, and can escape from a concave obstacle while searching for and navigating to the exit.

With the development of deep representation learning, the domain of reinforcement learning (RL) has become a powerful learning framework now capable of learning complex policies in high dimensional environments. This review summarises deep reinforcement learning (DRL) algorithms and provides a taxonomy of automated driving tasks where (D)RL methods have been employed, while addressing key computational challenges in real world deployment of autonomous driving agents. It also delineates adjacent domains such as behavior cloning, imitation learning, inverse reinforcement learning that are related but are not classical RL algorithms. The role of simulators in training agents, methods to validate, test and robustify existing solutions in RL are discussed.

安全是自主系统的关键组成部分，仍然是现实世界中要使用的基于学习的政策的挑战。特别是，由于不安全的行为，使用强化学习学习的政策通常无法推广到新的环境。在本文中，我们提出了SIM到LAB到实验室，以弥合现实差距，并提供概率保证的安全意见政策分配。为了提高安全性，我们采用双重政策设置，其中通过累积任务奖励对绩效政策进行培训，并通过根据汉密尔顿 - 雅各布（Hamilton-Jacobi）（HJ）达到可达性分析来培训备用（安全）政策。在SIM到LAB转移中，我们采用监督控制方案来掩盖探索过程中不安全的行动；在实验室到实验室的转移中，我们利用大约正确的（PAC） - 贝斯框架来提供有关在看不见环境中政策的预期性能和安全性的下限。此外，从HJ可达性分析继承，界限说明了每个环境中最坏情况安全性的期望。我们从经验上研究了两种类型的室内环境中的自我视频导航框架，具有不同程度的光真实性。我们还通过具有四足机器人的真实室内空间中的硬件实验来证明强大的概括性能。有关补充材料，请参见https://sites.google.com/princeton.edu/sim-to-lab-to-real。

基于模型的强化学习有望通过学习环境中的中间模型来预测未来的相互作用，从而从与环境的互动较少的相互作用中学习最佳政策。当预测一系列相互作用时，限制预测范围的推出长度是关键的超参数，因为预测的准确性会降低远离真实体验的区域。结果，从长远来看，从长远来看，总体上更糟糕的政策。因此，超参数提供了质量和效率之间的权衡。在这项工作中，我们将调整推出长度调整为元级的顺序决策问题的问题构成了问题，该问题优化了基于模型的强化学习所学到的最终策略，鉴于环境相互作用的固定预算通过基于反馈动态调整超参数来调整超参数。从学习过程中，例如模型的准确性和互动的其余预算。我们使用无模型的深度强化学习来解决元级决策问题，并证明我们的方法在两个众所周知的强化学习环境上优于共同的启发式基准。

This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results in model-free DRL-based methods for power systems, but model-free methods suffer from poor sample efficiency and training time, both critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. And it is desirable to minimize the direct interaction of the DRL agent with the real-world power grid due to its safety-critical nature. Additionally, state-of-the-art DRL-based policies are mostly trained using a physics-based grid simulator where dynamic simulation is computationally intensive, lowering the training efficiency. We propose a novel model-based-DRL framework where a deep neural network (DNN)-based dynamic surrogate model, instead of a real-world power-grid or physics-based simulation, is utilized with the policy learning framework, making the process faster and sample efficient. However, stabilizing model-based DRL is challenging because of the complex system dynamics of large-scale power systems. We solved these issues by incorporating imitation learning to have a warm start in policy learning, reward-shaping, and multi-step surrogate loss. Finally, we achieved 97.5% sample efficiency and 87.7% training efficiency for an application to the IEEE 300-bus test system.

Compared with model-based control and optimization methods, reinforcement learning (RL) provides a data-driven, learning-based framework to formulate and solve sequential decision-making problems. The RL framework has become promising due to largely improved data availability and computing power in the aviation industry. Many aviation-based applications can be formulated or treated as sequential decision-making problems. Some of them are offline planning problems, while others need to be solved online and are safety-critical. In this survey paper, we first describe standard RL formulations and solutions. Then we survey the landscape of existing RL-based applications in aviation. Finally, we summarize the paper, identify the technical gaps, and suggest future directions of RL research in aviation.

Safety is still one of the major research challenges in reinforcement learning (RL). In this paper, we address the problem of how to avoid safety violations of RL agents during exploration in probabilistic and partially unknown environments. Our approach combines automata learning for Markov Decision Processes (MDPs) and shield synthesis in an iterative approach. Initially, the MDP representing the environment is unknown. The agent starts exploring the environment and collects traces. From the collected traces, we passively learn MDPs that abstractly represent the safety-relevant aspects of the environment. Given a learned MDP and a safety specification, we construct a shield. For each state-action pair within a learned MDP, the shield computes exact probabilities on how likely it is that executing the action results in violating the specification from the current state within the next $k$ steps. After the shield is constructed, the shield is used during runtime and blocks any actions that induce a too large risk from the agent. The shielded agent continues to explore the environment and collects new data on the environment. Iteratively, we use the collected data to learn new MDPs with higher accuracy, resulting in turn in shields able to prevent more safety violations. We implemented our approach and present a detailed case study of a Q-learning agent exploring slippery Gridworlds. In our experiments, we show that as the agent explores more and more of the environment during training, the improved learned models lead to shields that are able to prevent many safety violations.

在过去的十年中，多智能经纪人强化学习（Marl）已经有了重大进展，但仍存在许多挑战，例如高样本复杂性和慢趋同稳定的政策，在广泛的部署之前需要克服，这是可能的。然而，在实践中，许多现实世界的环境已经部署了用于生成策略的次优或启发式方法。一个有趣的问题是如何最好地使用这些方法作为顾问，以帮助改善多代理领域的加强学习。在本文中，我们提供了一个原则的框架，用于将动作建议纳入多代理设置中的在线次优顾问。我们描述了在非传记通用随机游戏环境中提供多种智能强化代理（海军上将）的问题，并提出了两种新的基于Q学习的算法：海军上将决策（海军DM）和海军上将 - 顾问评估（Admiral-AE） ，这使我们能够通过适当地纳入顾问（Admiral-DM）的建议来改善学习，并评估顾问（Admiral-AE）的有效性。我们从理论上分析了算法，并在一般加上随机游戏中提供了关于他们学习的定点保证。此外，广泛的实验说明了这些算法：可以在各种环境中使用，具有对其他相关基线的有利相比的性能，可以扩展到大状态行动空间，并且对来自顾问的不良建议具有稳健性。

数字化和远程连接扩大了攻击面，使网络系统更脆弱。由于攻击者变得越来越复杂和资源丰富，仅仅依赖传统网络保护，如入侵检测，防火墙和加密，不足以保护网络系统。网络弹性提供了一种新的安全范式，可以使用弹性机制来补充保护不足。一种网络弹性机制（CRM）适应了已知的或零日威胁和实际威胁和不确定性，并对他们进行战略性地响应，以便在成功攻击时保持网络系统的关键功能。反馈架构在启用CRM的在线感应，推理和致动过程中发挥关键作用。强化学习（RL）是一个重要的工具，对网络弹性的反馈架构构成。它允许CRM提供有限或没有事先知识和攻击者的有限攻击的顺序响应。在这项工作中，我们审查了Cyber​​恢复力的RL的文献，并讨论了对三种主要类型的漏洞，即姿势有关，与信息相关的脆弱性的网络恢复力。我们介绍了三个CRM的应用领域：移动目标防御，防守网络欺骗和辅助人类安全技术。 RL算法也有漏洞。我们解释了RL的三个漏洞和目前的攻击模型，其中攻击者针对环境与代理商之间交换的信息：奖励，国家观察和行动命令。我们展示攻击者可以通过最低攻击努力来欺骗RL代理商学习邪恶的政策。最后，我们讨论了RL为基于RL的CRM的网络安全和恢复力和新兴应用的未来挑战。

使能够评估风险和做出风险意识的决策的能力对于将强化学习应用于无人机等安全性机器人至关重要。在本文中，我们调查了一种特定情况，即纳米四轮摩托车机器人学会在部分可观察性下浏览杂乱无章的环境。我们提出了一个分配加强学习框架，以生成适应性的风险趋势政策。具体而言，我们建议将学习回报分布的较低尾巴条件差异作为内在的不确定性估计，并使用指数加权的平均预测（EWAF）根据估计的不确定性调整风险趋势。在模拟和现实世界的经验结果中，我们表明（1）（1）最有效的风险趋势在各州各不相同，（2）具有自适应风险趋势的代理人比风险中性政策或避免风险的政策基准相比，其绩效优于绩效。

由于数据量增加，金融业的快速变化已经彻底改变了数据处理和数据分析的技术，并带来了新的理论和计算挑战。与古典随机控制理论和解决财务决策问题的其他分析方法相比，解决模型假设的财务决策问题，强化学习（RL）的新发展能够充分利用具有更少模型假设的大量财务数据并改善复杂的金融环境中的决策。该调查纸目的旨在审查最近的资金途径的发展和使用RL方法。我们介绍了马尔可夫决策过程，这是许多常用的RL方法的设置。然后引入各种算法，重点介绍不需要任何模型假设的基于价值和基于策略的方法。连接是用神经网络进行的，以扩展框架以包含深的RL算法。我们的调查通过讨论了这些RL算法在金融中各种决策问题中的应用，包括最佳执行，投资组合优化，期权定价和对冲，市场制作，智能订单路由和Robo-Awaring。

Safety comes first in many real-world applications involving autonomous agents. Despite a large number of reinforcement learning (RL) methods focusing on safety-critical tasks, there is still a lack of high-quality evaluation of those algorithms that adheres to safety constraints at each decision step under complex and unknown dynamics. In this paper, we revisit prior work in this scope from the perspective of state-wise safe RL and categorize them as projection-based, recovery-based, and optimization-based approaches, respectively. Furthermore, we propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. This novel technique explicitly enforces hard constraints via the deep unrolling architecture and enjoys structural advantages in navigating the trade-off between reward improvement and constraint satisfaction. To facilitate further research in this area, we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit, a toolkit that provides off-the-shelf interfaces and evaluation utilities for safety-critical tasks. We then perform a comparative study of the involved algorithms on six benchmarks ranging from robotic control to autonomous driving. The empirical results provide an insight into their applicability and robustness in learning zero-cost-return policies without task-dependent handcrafting. The project page is available at https://sites.google.com/view/saferlkit.

强化学习（RL）为解决各种复杂的决策任务提供了新的机会。但是，现代的RL算法，例如，深Q学习是基于深层神经网络，在Edge设备上运行时的计算成本很高。在本文中，我们提出了QHD，一种高度增强的学习，它模仿了大脑特性，以实现健壮和实时学习。 QHD依靠轻巧的大脑启发模型来学习未知环境中的最佳政策。我们首先建立一个新颖的数学基础和编码模块，该模块将状态行动空间映射到高维空间中。因此，我们开发了一个高维回归模型，以近似Q值函数。 QHD驱动的代理通过比较每个可能动作的Q值来做出决定。我们评估了不同的RL培训批量和本地记忆能力对QHD学习质量的影响。我们的QHD也能够以微小的本地记忆能力在线学习，这与培训批量大小一样小。 QHD通过进一步降低记忆容量和批处理大小来提供实时学习。这使得QHD适用于在边缘环境中高效的增强学习，这对于支持在线和实时学习至关重要。我们的解决方案还支持少量的重播批量大小，与DQN相比，该批量的速度为12.3倍，同时确保质量损失最小。我们的评估显示了实时学习的QHD能力，比最先进的Deep RL算法提供了34.6倍的速度和更高的学习质量。

无人驾驶汽车（UAV）已被广泛用于军事战。在本文中，我们将自动运动控制（AMC）问题作为马尔可夫决策过程（MDP），并提出了一种先进的深度强化学习（DRL）方法，该方法允许无人机在大型动态三维（3D）中执行复杂的任务）环境。为了克服优先体验重播（PER）算法的局限性并提高性能，拟议的异步课程体验重播（ACER）使用多线程来异步更新优先级，分配了真实优先级，并应用了临时体验池，以使可用的更高体验可用学习质量。还引入了第一个无用的体验池（FIUO）体验池，以确保存储体验的更高使用价值。此外，与课程学习（CL）相结合，从简单到困难的抽样体验进行了更合理的培训范式，设计用于培训无人机。通过在基于真实无人机的参数构建的复杂未知环境中训练，提议的ACER将收敛速度提高24.66 \％，而与最先进的双胞胎延迟的深层确定性相比策略梯度（TD3）算法。在具有不同复杂性的环境中进行的测试实验表明，ACER剂的鲁棒性和泛化能力。

通过直接将感知输入映射到机器人控制命令中，深入的强化学习（DRL）算法已被证明在机器人导航中有效，尤其是在未知环境中。但是，大多数现有方法忽略导航中的局部最小问题，从而无法处理复杂的未知环境。在本文中，我们提出了第一个基于DRL的导航方法，该方法由具有连续动作空间，自适应向前模拟时间（AFST）的SMDP建模，以克服此问题。具体而言，我们通过修改其GAE来更好地估计SMDP中的策略梯度，改善了指定SMDP问题的分布式近端策略优化（DPPO）算法。我们在模拟器和现实世界中评估了我们的方法。

背景信息：在过去几年中，机器学习（ML）一直是许多创新的核心。然而，包括在所谓的“安全关键”系统中，例如汽车或航空的系统已经被证明是非常具有挑战性的，因为ML的范式转变为ML带来完全改变传统认证方法。目的：本文旨在阐明与ML为基础的安全关键系统认证有关的挑战，以及文献中提出的解决方案，以解决它们，回答问题的问题如何证明基于机器学习的安全关键系统？'方法：我们开展2015年至2020年至2020年之间发布的研究论文的系统文献综述（SLR），涵盖了与ML系统认证有关的主题。总共确定了217篇论文涵盖了主题，被认为是ML认证的主要支柱：鲁棒性，不确定性，解释性，验证，安全强化学习和直接认证。我们分析了每个子场的主要趋势和问题，并提取了提取的论文的总结。结果：单反结果突出了社区对该主题的热情，以及在数据集和模型类型方面缺乏多样性。它还强调需要进一步发展学术界和行业之间的联系，以加深域名研究。最后，它还说明了必须在上面提到的主要支柱之间建立连接的必要性，这些主要柱主要主要研究。结论：我们强调了目前部署的努力，以实现ML基于ML的软件系统，并讨论了一些未来的研究方向。