多智能体增强学习（MARL）的最新进展提供了各种工具，支持代理能力适应其环境中的意外变化，并鉴于环境的动态性质（可能会通过其他情况加剧代理商）。在这项工作中，我们强调了集团有效合作的能力与集团的弹性之间的关系，我们衡量了该集团适应环境扰动的能力。为了促进恢复力，我们建议通过新的基于混乱的通信协议进行协作，这是根据其以前经验中未对准的观察结果。我们允许有关代理人自主学习的信息的宽度和频率的决定，这被激活以减少混淆。我们在各种MARL设置中展示了我们的方法的实证评估。

了解强化学习（RL）代理的新兴行为可能很困难，因为这种代理通常使用高度复杂的决策程序在复杂的环境中进行训练。这引起了RL中解释性的多种方法，旨在调和可能在主体行为与观察者预期的行为之间产生的差异。最近的方法取决于域知识，这可能并非总是可用的，分析代理商的策略，或者是对基础环境的特定要素的分析，通常被建模为马尔可夫决策过程（MDP）。我们的主要主张是，即使基本的MDP尚不完全了解（例如，尚未准确地了解过渡概率），也没有由代理商维护（即，在使用无模型方法时），但仍可以利用它为自动生成解释。为此，我们建议使用以前在文献中使用的正式MDP抽象和转换来加快寻找最佳策略的搜索，以自动产生解释。由于这种转换通常基于环境的符号表示，因此它们可能代表了预期和实际代理行为之间差距的有意义的解释。我们正式定义了这个问题，建议一类可用于解释新兴行为的转换，并提出了有效搜索解释的方法。我们演示了一组标准基准测试的方法。

Various types of Multi-Agent Reinforcement Learning (MARL) methods have been developed, assuming that agents' policies are based on true states. Recent works have improved the robustness of MARL under uncertainties from the reward, transition probability, or other partners' policies. However, in real-world multi-agent systems, state estimations may be perturbed by sensor measurement noise or even adversaries. Agents' policies trained with only true state information will deviate from optimal solutions when facing adversarial state perturbations during execution. MARL under adversarial state perturbations has limited study. Hence, in this work, we propose a State-Adversarial Markov Game (SAMG) and make the first attempt to study the fundamental properties of MARL under state uncertainties. We prove that the optimal agent policy and the robust Nash equilibrium do not always exist for an SAMG. Instead, we define the solution concept, robust agent policy, of the proposed SAMG under adversarial state perturbations, where agents want to maximize the worst-case expected state value. We then design a gradient descent ascent-based robust MARL algorithm to learn the robust policies for the MARL agents. Our experiments show that adversarial state perturbations decrease agents' rewards for several baselines from the existing literature, while our algorithm outperforms baselines with state perturbations and significantly improves the robustness of the MARL policies under state uncertainties.

Communication is supposed to improve multi-agent collaboration and overall performance in cooperative Multi-agent reinforcement learning (MARL). However, such improvements are prevalently limited in practice since most existing communication schemes ignore communication overheads (e.g., communication delays). In this paper, we demonstrate that ignoring communication delays has detrimental effects on collaborations, especially in delay-sensitive tasks such as autonomous driving. To mitigate this impact, we design a delay-aware multi-agent communication model (DACOM) to adapt communication to delays. Specifically, DACOM introduces a component, TimeNet, that is responsible for adjusting the waiting time of an agent to receive messages from other agents such that the uncertainty associated with delay can be addressed. Our experiments reveal that DACOM has a non-negligible performance improvement over other mechanisms by making a better trade-off between the benefits of communication and the costs of waiting for messages.

强化学习（RL）在很大程度上依赖于探索以从环境中学习并最大程度地获得观察到的奖励。因此，必须设计一个奖励功能，以确保从收到的经验中获得最佳学习。以前的工作将自动机和基于逻辑的奖励成型与环境假设相结合，以提供自动机制，以根据任务综合奖励功能。但是，关于如何将基于逻辑的奖励塑造扩大到多代理增强学习（MARL）的工作有限。如果任务需要合作，则环境将需要考虑联合状态，以跟踪其他代理，从而遭受对代理数量的维度的诅咒。该项目探讨了如何针对不同场景和任务设计基于逻辑的奖励成型。我们提出了一种针对半偏心逻辑基于逻辑的MARL奖励成型的新方法，该方法在代理数量中是可扩展的，并在多种情况下对其进行了评估。

Reinforcement learning (RL) is one of the most important branches of AI. Due to its capacity for self-adaption and decision-making in dynamic environments, reinforcement learning has been widely applied in multiple areas, such as healthcare, data markets, autonomous driving, and robotics. However, some of these applications and systems have been shown to be vulnerable to security or privacy attacks, resulting in unreliable or unstable services. A large number of studies have focused on these security and privacy problems in reinforcement learning. However, few surveys have provided a systematic review and comparison of existing problems and state-of-the-art solutions to keep up with the pace of emerging threats. Accordingly, we herein present such a comprehensive review to explain and summarize the challenges associated with security and privacy in reinforcement learning from a new perspective, namely that of the Markov Decision Process (MDP). In this survey, we first introduce the key concepts related to this area. Next, we cover the security and privacy issues linked to the state, action, environment, and reward function of the MDP process, respectively. We further highlight the special characteristics of security and privacy methodologies related to reinforcement learning. Finally, we discuss the possible future research directions within this area.

Cooperative multi-agent reinforcement learning (MARL) has achieved significant results, most notably by leveraging the representation-learning abilities of deep neural networks. However, large centralized approaches quickly become infeasible as the number of agents scale, and fully decentralized approaches can miss important opportunities for information sharing and coordination. Furthermore, not all agents are equal -- in some cases, individual agents may not even have the ability to send communication to other agents or explicitly model other agents. This paper considers the case where there is a single, powerful, \emph{central agent} that can observe the entire observation space, and there are multiple, low-powered \emph{local agents} that can only receive local observations and are not able to communicate with each other. The central agent's job is to learn what message needs to be sent to different local agents based on the global observations, not by centrally solving the entire problem and sending action commands, but by determining what additional information an individual agent should receive so that it can make a better decision. In this work we present our MARL algorithm \algo, describe where it would be most applicable, and implement it in the cooperative navigation and multi-agent walker domains. Empirical results show that 1) learned communication does indeed improve system performance, 2) results generalize to heterogeneous local agents, and 3) results generalize to different reward structures.

具有很多玩家的非合作和合作游戏具有许多应用程序，但是当玩家数量增加时，通常仍然很棘手。由Lasry和Lions以及Huang，Caines和Malham \'E引入的，平均野外运动会（MFGS）依靠平均场外近似值，以使玩家数量可以成长为无穷大。解决这些游戏的传统方法通常依赖于以完全了解模型的了解来求解部分或随机微分方程。最近，增强学习（RL）似乎有望解决复杂问题。通过组合MFGS和RL，我们希望在人口规模和环境复杂性方面能够大规模解决游戏。在这项调查中，我们回顾了有关学习MFG中NASH均衡的最新文献。我们首先确定最常见的设置（静态，固定和进化）。然后，我们为经典迭代方法（基于最佳响应计算或策略评估）提供了一个通用框架，以确切的方式解决MFG。在这些算法和与马尔可夫决策过程的联系的基础上，我们解释了如何使用RL以无模型的方式学习MFG解决方案。最后，我们在基准问题上介绍了数值插图，并以某些视角得出结论。

值得信赖的强化学习算法应有能力解决挑战性的现实问题，包括{Robustly}处理不确定性，满足{安全}的限制以避免灾难性的失败，以及在部署过程中{prencepentiming}以避免灾难性的失败}。这项研究旨在概述这些可信赖的强化学习的主要观点，即考虑其在鲁棒性，安全性和概括性上的内在脆弱性。特别是，我们给出严格的表述，对相应的方法进行分类，并讨论每个观点的基准。此外，我们提供了一个前景部分，以刺激有希望的未来方向，并简要讨论考虑人类反馈的外部漏洞。我们希望这项调查可以在统一的框架中将单独的研究汇合在一起，并促进强化学习的可信度。

在过去的十年中，多智能经纪人强化学习（Marl）已经有了重大进展，但仍存在许多挑战，例如高样本复杂性和慢趋同稳定的政策，在广泛的部署之前需要克服，这是可能的。然而，在实践中，许多现实世界的环境已经部署了用于生成策略的次优或启发式方法。一个有趣的问题是如何最好地使用这些方法作为顾问，以帮助改善多代理领域的加强学习。在本文中，我们提供了一个原则的框架，用于将动作建议纳入多代理设置中的在线次优顾问。我们描述了在非传记通用随机游戏环境中提供多种智能强化代理（海军上将）的问题，并提出了两种新的基于Q学习的算法：海军上将决策（海军DM）和海军上将 - 顾问评估（Admiral-AE） ，这使我们能够通过适当地纳入顾问（Admiral-DM）的建议来改善学习，并评估顾问（Admiral-AE）的有效性。我们从理论上分析了算法，并在一般加上随机游戏中提供了关于他们学习的定点保证。此外，广泛的实验说明了这些算法：可以在各种环境中使用，具有对其他相关基线的有利相比的性能，可以扩展到大状态行动空间，并且对来自顾问的不良建议具有稳健性。

信息共享是建立团队认知并实现协调与合作的关键。高性能的人类团队也从战略性地采用迭代沟通和合理性的层次结构级别中受益，这意味着人类代理可以推理队友在决策中的行动。然而，多代理强化学习（MARL）的大多数先前工作不支持迭代的理性性，而只能鼓励跨性别的沟通，从而实现了次优的平衡合作策略。在这项工作中，我们表明，在优化政策梯度（PG）时，将代理商的政策重新制定为有条件依靠其邻近队友的政策，从而固有地提高了相互信息（MI）的最大程度。在有限的理性和认知层次结构理论下的决策观念的基础上，我们表明我们的修改后的PG方法不仅可以最大化本地代理人的奖励，而且还隐含着关于代理之间MI的理由，而无需任何明确的临时正则化术语。我们的方法Infopg在学习新兴的协作行为方面优于基准，并在分散的合作MARL任务中设定了最先进的工作。我们的实验通过在几个复杂的合作多代理域中实现较高的样品效率和更大的累积奖励来验证InfoPG的实用性。

In multi-agent reinforcement learning (MARL), many popular methods, such as VDN and QMIX, are susceptible to a critical multi-agent pathology known as relative overgeneralization (RO), which arises when the optimal joint action's utility falls below that of a sub-optimal joint action in cooperative tasks. RO can cause the agents to get stuck into local optima or fail to solve tasks that require significant coordination between agents within a given timestep. Recent value-based MARL algorithms such as QPLEX and WQMIX can overcome RO to some extent. However, our experimental results show that they can still fail to solve cooperative tasks that exhibit strong RO. In this work, we propose a novel approach called curriculum learning for relative overgeneralization (CURO) to better overcome RO. To solve a target task that exhibits strong RO, in CURO, we first fine-tune the reward function of the target task to generate source tasks that are tailored to the current ability of the learning agent and train the agent on these source tasks first. Then, to effectively transfer the knowledge acquired in one task to the next, we use a novel transfer learning method that combines value function transfer with buffer transfer, which enables more efficient exploration in the target task. We demonstrate that, when applied to QMIX, CURO overcomes severe RO problem and significantly improves performance, yielding state-of-the-art results in a variety of cooperative multi-agent tasks, including the challenging StarCraft II micromanagement benchmarks.

This work considers the problem of learning cooperative policies in complex, partially observable domains without explicit communication. We extend three classes of single-agent deep reinforcement learning algorithms based on policy gradient, temporal-difference error, and actor-critic methods to cooperative multi-agent systems. We introduce a set of cooperative control tasks that includes tasks with discrete and continuous actions, as well as tasks that involve hundreds of agents. The three approaches are evaluated against each other using different neural architectures, training procedures, and reward structures. Using deep reinforcement learning with a curriculum learning scheme, our approach can solve problems that were previously considered intractable by most multi-agent reinforcement learning algorithms. We show that policy gradient methods tend to outperform both temporal-difference and actor-critic methods when using feed-forward neural architectures. We also show that recurrent policies, while more difficult to train, outperform feed-forward policies on our evaluation tasks.

加固学习在机器学习中推动了令人印象深刻的进步。同时，量子增强机学习算法使用量子退火的底层划伤。最近，已经提出了一种组合两个范例的多代理强化学习（MARL）架构。这种新的算法利用Q值近似的量子Boltzmann机器（QBMS）在收敛所需的时间步长方面具有优于常规的深度增强学习。但是，该算法仅限于单代理和小型2x2多代理网格域。在这项工作中，我们提出了对原始概念的延伸，以解决更具挑战性问题。类似于Classic DQN，我们添加了重播缓冲区的体验，并使用不同的网络来估计目标和策略值。实验结果表明，学习变得更加稳定，使代理能够在具有更高复杂性的网格域中找到最佳策略。此外，我们还评估参数共享如何影响多代理域中的代理行为。量子采样证明是一种有希望的加强学习任务的方法，但目前受到QPU尺寸的限制，因此通过输入和Boltzmann机器的大小。

最先进的多机构增强学习（MARL）方法为各种复杂问题提供了有希望的解决方案。然而，这些方法都假定代理执行同步的原始操作执行，因此它们不能真正可扩展到长期胜利的真实世界多代理/机器人任务，这些任务固有地要求代理/机器人以异步的理由，涉及有关高级动作选择的理由。不同的时间。宏观行动分散的部分可观察到的马尔可夫决策过程（MACDEC-POMDP）是在完全合作的多代理任务中不确定的异步决策的一般形式化。在本论文中，我们首先提出了MacDec-Pomdps的一组基于价值的RL方法，其中允许代理在三个范式中使用宏观成果功能执行异步学习和决策：分散学习和控制，集中学习，集中学习和控制，以及分散执行的集中培训（CTDE）。在上述工作的基础上，我们在三个训练范式下制定了一组基于宏观行动的策略梯度算法，在该训练范式下，允许代理以异步方式直接优化其参数化策略。我们在模拟和真实的机器人中评估了我们的方法。经验结果证明了我们在大型多代理问题中的方法的优势，并验证了我们算法在学习具有宏观actions的高质量和异步溶液方面的有效性。

部署到现实世界的自主智能代理必须与对感官输入的对抗性攻击保持强大的态度。在加强学习中的现有工作集中于最小值扰动攻击，这些攻击最初是为了模仿计算机视觉中感知不变性的概念。在本文中，我们注意到，这种最小值扰动攻击可以由受害者琐碎地检测到，因为这些导致观察序列与受害者的行为不符。此外，许多现实世界中的代理商（例如物理机器人）通常在人类主管下运行，这些代理商不容易受到这种扰动攻击的影响。结果，我们建议专注于幻觉攻击，这是一种与受害者的世界模式一致的新型攻击形式。我们为这个新颖的攻击框架提供了正式的定义，在各种条件下探索了其特征，并得出结论，代理必须寻求现实主义反馈以对幻觉攻击具有强大的态度。

Adequately assigning credit to actions for future outcomes based on their contributions is a long-standing open challenge in Reinforcement Learning. The assumptions of the most commonly used credit assignment method are disadvantageous in tasks where the effects of decisions are not immediately evident. Furthermore, this method can only evaluate actions that have been selected by the agent, making it highly inefficient. Still, no alternative methods have been widely adopted in the field. Hindsight Credit Assignment is a promising, but still unexplored candidate, which aims to solve the problems of both long-term and counterfactual credit assignment. In this thesis, we empirically investigate Hindsight Credit Assignment to identify its main benefits, and key points to improve. Then, we apply it to factored state representations, and in particular to state representations based on the causal structure of the environment. In this setting, we propose a variant of Hindsight Credit Assignment that effectively exploits a given causal structure. We show that our modification greatly decreases the workload of Hindsight Credit Assignment, making it more efficient and enabling it to outperform the baseline credit assignment method on various tasks. This opens the way to other methods based on given or learned causal structures.

流动性和流量的许多方案都涉及多种不同的代理，需要合作以找到共同解决方案。行为计划的最新进展使用强化学习以寻找有效和绩效行为策略。但是，随着自动驾驶汽车和车辆对X通信变得越来越成熟，只有使用单身独立代理的解决方案在道路上留下了潜在的性能增长。多代理增强学习（MARL）是一个研究领域，旨在为彼此相互作用的多种代理找到最佳解决方案。这项工作旨在将该领域的概述介绍给研究人员的自主行动能力。我们首先解释Marl并介绍重要的概念。然后，我们讨论基于Marl算法的主要范式，并概述每个范式中最先进的方法和思想。在这种背景下，我们调查了MAL在自动移动性场景中的应用程序，并概述了现有的场景和实现。

We explore deep reinforcement learning methods for multi-agent domains. We begin by analyzing the difficulty of traditional algorithms in the multi-agent case: Q-learning is challenged by an inherent non-stationarity of the environment, while policy gradient suffers from a variance that increases as the number of agents grows. We then present an adaptation of actor-critic methods that considers action policies of other agents and is able to successfully learn policies that require complex multiagent coordination. Additionally, we introduce a training regimen utilizing an ensemble of policies for each agent that leads to more robust multi-agent policies. We show the strength of our approach compared to existing methods in cooperative as well as competitive scenarios, where agent populations are able to discover various physical and informational coordination strategies.

使用深神经网络作为函数近似器导致加强学习算法和应用的罢工进展。然而，我们在决策边界几何和神经政策的损失景观中的知识仍然非常有限。在本文中，我们提出了一个框架来调查各种州和跨MDP的决策边界和损失景观相似之处。我们在街机学习环境中进行各种游戏进行实验，并发现神经政策的高灵敏度方向横跨MDP相关。我们认为，这些高灵敏度方向支持非强大功能在加固学习代理的培训环境中共享非强大功能。我们相信我们的结果揭示了深度加强学习培训中使用的环境的基本属性，并代表了建立强大可靠的深度加固学习代理的有形步骤。