In this work, we propose a self-supervised multi-agent system, termed a memory-like adaptive modeling multi-agent learning system (MAMMALS), that realizes online learning towards behavioral pattern clustering tasks for time series. Encoding the visual behaviors as discrete time series(DTS), and training and modeling them in the multi-agent system with a bio-memory-like form. We finally implemented a fully decentralized multi-agent system design framework and completed its feasibility verification in a surveillance video application scenario on vehicle path clustering. In multi-agent learning, using learning methods designed for individual agents will typically perform poorly globally because of the behavior of ignoring the synergy between agents.

流动性和流量的许多方案都涉及多种不同的代理，需要合作以找到共同解决方案。行为计划的最新进展使用强化学习以寻找有效和绩效行为策略。但是，随着自动驾驶汽车和车辆对X通信变得越来越成熟，只有使用单身独立代理的解决方案在道路上留下了潜在的性能增长。多代理增强学习（MARL）是一个研究领域，旨在为彼此相互作用的多种代理找到最佳解决方案。这项工作旨在将该领域的概述介绍给研究人员的自主行动能力。我们首先解释Marl并介绍重要的概念。然后，我们讨论基于Marl算法的主要范式，并概述每个范式中最先进的方法和思想。在这种背景下，我们调查了MAL在自动移动性场景中的应用程序，并概述了现有的场景和实现。

许多现实世界的应用程序都可以作为多机构合作问题进行配置，例如网络数据包路由和自动驾驶汽车的协调。深入增强学习（DRL）的出现为通过代理和环境的相互作用提供了一种有前途的多代理合作方法。但是，在政策搜索过程中，传统的DRL解决方案遭受了多个代理具有连续动作空间的高维度。此外，代理商政策的动态性使训练非平稳。为了解决这些问题，我们建议采用高级决策和低水平的个人控制，以进行有效的政策搜索，提出一种分层增强学习方法。特别是，可以在高级离散的动作空间中有效地学习多个代理的合作。同时，低水平的个人控制可以减少为单格强化学习。除了分层增强学习外，我们还建议对手建模网络在学习过程中对其他代理的政策进行建模。与端到端的DRL方法相反，我们的方法通过以层次结构将整体任务分解为子任务来降低学习的复杂性。为了评估我们的方法的效率，我们在合作车道变更方案中进行了现实世界中的案例研究。模拟和现实世界实验都表明我们的方法在碰撞速度和收敛速度中的优越性。

Cooperative multi-agent reinforcement learning (MARL) has achieved significant results, most notably by leveraging the representation-learning abilities of deep neural networks. However, large centralized approaches quickly become infeasible as the number of agents scale, and fully decentralized approaches can miss important opportunities for information sharing and coordination. Furthermore, not all agents are equal -- in some cases, individual agents may not even have the ability to send communication to other agents or explicitly model other agents. This paper considers the case where there is a single, powerful, \emph{central agent} that can observe the entire observation space, and there are multiple, low-powered \emph{local agents} that can only receive local observations and are not able to communicate with each other. The central agent's job is to learn what message needs to be sent to different local agents based on the global observations, not by centrally solving the entire problem and sending action commands, but by determining what additional information an individual agent should receive so that it can make a better decision. In this work we present our MARL algorithm \algo, describe where it would be most applicable, and implement it in the cooperative navigation and multi-agent walker domains. Empirical results show that 1) learned communication does indeed improve system performance, 2) results generalize to heterogeneous local agents, and 3) results generalize to different reward structures.

大型人口系统的分析和控制对研究和工程的各个领域引起了极大的兴趣，从机器人群的流行病学到经济学和金融。一种越来越流行和有效的方法来实现多代理系统中的顺序决策，这是通过多机构增强学习，因为它允许对高度复杂的系统进行自动和无模型的分析。但是，可伸缩性的关键问题使控制和增强学习算法的设计变得复杂，尤其是在具有大量代理的系统中。尽管强化学习在许多情况下都发现了经验成功，但许多代理商的问题很快就变得棘手了，需要特别考虑。在这项调查中，我们将阐明当前的方法，以通过多代理强化学习以及通过诸如平均场游戏，集体智能或复杂的网络理论等研究领域进行仔细理解和分析大型人口系统。这些经典独立的主题领域提供了多种理解或建模大型人口系统的方法，这可能非常适合将来的可拖动MARL算法制定。最后，我们调查了大规模控制的潜在应用领域，并确定了实用系统中学习算法的富有成果的未来应用。我们希望我们的调查可以为理论和应用科学的初级和高级研究人员提供洞察力和未来的方向。

几乎所有的多代理强化学习算法没有交流，都遵循分散执行的集中培训原则。在集中培训期间，代理可以以相同的信号为指导，例如全球国家。但是，在分散执行期间，代理缺乏共享信号。受到观点不变性和对比学习的启发，我们在本文中提出了共识学习，以学习合作的多代理增强学习。尽管基于局部观察结果，但不同的代理可以在离散空间中推断出相同的共识。在分散执行期间，我们将推断的共识作为对代理网络的明确输入提供了，从而发展了他们的合作精神。我们提出的方法可以扩展到具有小模型更改的各种多代理增强学习算法。此外，我们执行一些完全合作的任务，并获得令人信服的结果。

最近的自主代理和机器人的应用，如自动驾驶汽车，情景的培训师，勘探机器人和服务机器人带来了关注与当前生成人工智能（AI）系统相关的至关重要的信任相关挑战。尽管取得了巨大的成功，基于连接主义深度学习神经网络方法的神经网络方法缺乏解释他们对他人的决策和行动的能力。没有符号解释能力，它们是黑色盒子，这使得他们的决定或行动不透明，这使得难以信任它们在安全关键的应用中。最近对AI系统解释性的立场目睹了可解释的人工智能（XAI）的几种方法;然而，大多数研究都专注于应用于计算科学中的数据驱动的XAI系统。解决越来越普遍的目标驱动器和机器人的研究仍然缺失。本文评论了可解释的目标驱动智能代理和机器人的方法，重点是解释和沟通代理人感知功能的技术（示例，感官和愿景）和认知推理（例如，信仰，欲望，意图，计划和目标）循环中的人类。审查强调了强调透明度，可辨与和持续学习以获得解释性的关键策略。最后，本文提出了解释性的要求，并提出了用于实现有效目标驱动可解释的代理和机器人的路线图。

临时团队合作是设计可以与新队友合作而无需事先协调的研究问题的研究问题。这项调查做出了两个贡献：首先，它提供了对临时团队工作问题不同方面的结构化描述。其次，它讨论了迄今为止该领域取得的进展，并确定了临时团队工作中需要解决的直接和长期开放问题。

Proper functioning of connected and automated vehicles (CAVs) is crucial for the safety and efficiency of future intelligent transport systems. Meanwhile, transitioning to fully autonomous driving requires a long period of mixed autonomy traffic, including both CAVs and human-driven vehicles. Thus, collaboration decision-making for CAVs is essential to generate appropriate driving behaviors to enhance the safety and efficiency of mixed autonomy traffic. In recent years, deep reinforcement learning (DRL) has been widely used in solving decision-making problems. However, the existing DRL-based methods have been mainly focused on solving the decision-making of a single CAV. Using the existing DRL-based methods in mixed autonomy traffic cannot accurately represent the mutual effects of vehicles and model dynamic traffic environments. To address these shortcomings, this article proposes a graph reinforcement learning (GRL) approach for multi-agent decision-making of CAVs in mixed autonomy traffic. First, a generic and modular GRL framework is designed. Then, a systematic review of DRL and GRL methods is presented, focusing on the problems addressed in recent research. Moreover, a comparative study on different GRL methods is further proposed based on the designed framework to verify the effectiveness of GRL methods. Results show that the GRL methods can well optimize the performance of multi-agent decision-making for CAVs in mixed autonomy traffic compared to the DRL methods. Finally, challenges and future research directions are summarized. This study can provide a valuable research reference for solving the multi-agent decision-making problems of CAVs in mixed autonomy traffic and can promote the implementation of GRL-based methods into intelligent transportation systems. The source code of our work can be found at https://github.com/Jacklinkk/Graph_CAVs.

Imitation learning techniques aim to mimic human behavior in a given task. An agent (a learning machine) is trained to perform a task from demonstrations by learning a mapping between observations and actions. The idea of teaching by imitation has been around for many years, however, the field is gaining attention recently due to advances in computing and sensing as well as rising demand for intelligent applications. The paradigm of learning by imitation is gaining popularity because it facilitates teaching complex tasks with minimal expert knowledge of the tasks. Generic imitation learning methods could potentially reduce the problem of teaching a task to that of providing demonstrations; without the need for explicit programming or designing reward functions specific to the task. Modern sensors are able to collect and transmit high volumes of data rapidly, and processors with high computational power allow fast processing that maps the sensory data to actions in a timely manner. This opens the door for many potential AI applications that require real-time perception and reaction such as humanoid robots, self-driving vehicles, human computer interaction and computer games to name a few. However, specialized algorithms are needed to effectively and robustly learn models as learning by imitation poses its own set of challenges. In this paper, we survey imitation learning methods and present design options in different steps of the learning process. We introduce a background and motivation for the field as well as highlight challenges specific to the imitation problem. Methods for designing and evaluating imitation learning tasks are categorized and reviewed. Special attention is given to learning methods in robotics and games as these domains are the most popular in the literature and provide a wide array of problems and methodologies. We extensively discuss combining imitation learning approaches using different sources and methods, as well as incorporating other motion learning methods to enhance imitation. We also discuss the potential impact on industry, present major applications and highlight current and future research directions.

Starcraft II多代理挑战（SMAC）被创建为合作多代理增强学习（MARL）的具有挑战性的基准问题。 SMAC专注于星际争霸微管理的问题，并假设每个单元都由独立行动并仅具有本地信息的学习代理人单独控制；假定通过分散执行（CTDE）进行集中培训。为了在SMAC中表现良好，MARL算法必须处理多机构信贷分配和联合行动评估的双重问题。本文介绍了一种新的体系结构Transmix，这是一个基于变压器的联合行动值混合网络，与其他最先进的合作MARL解决方案相比，我们显示出高效且可扩展的。 Transmix利用变形金刚学习更丰富的混合功能的能力来结合代理的个人价值函数。它与以前的SMAC场景上的工作相当，并且在困难场景上胜过其他技术，以及被高斯噪音损坏的场景以模拟战争的雾。

Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in singleagent settings. We present an actor-critic algorithm that trains decentralized policies in multiagent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep. This attention mechanism enables more effective and scalable learning in complex multiagent environments, when compared to recent approaches. Our approach is applicable not only to cooperative settings with shared rewards, but also individualized reward settings, including adversarial settings, as well as settings that do not provide global states, and it makes no assumptions about the action spaces of the agents. As such, it is flexible enough to be applied to most multi-agent learning problems.

多目标自组织追求（SOP）问题已广泛应用，并被认为是一个充满挑战的分布式系统的自组织游戏，在该系统中，智能代理在其中合作追求具有部分观察的多个动态目标。这项工作为分散的多机构系统提出了一个框架，以提高智能代理的搜索和追求能力。我们将一个自组织的系统建模为可观察到的马尔可夫游戏（POMG），具有权力下放，部分观察和非通信的特征。然后将拟议的分布式算法：模糊自组织合作协同进化（FSC2）杠杆化，以解决多目标SOP中的三个挑战：分布式自组织搜索（SOS），分布式任务分配和分布式单目标追踪。 FSC2包括一种协调的多代理深钢筋学习方法，该方法使均匀的代理能够学习天然SOS模式。此外，我们提出了一种基于模糊的分布式任务分配方法，该方法将多目标SOP分解为几个单目标追求问题。合作进化原则用于协调每个单一目标问题的分布式追随者。因此，可以缓解POMG中固有的部分观察和分布式决策的不确定性。实验结果表明，在所有三个子任务中，分布式不传动的多机构协调都具有部分观察结果，而2048 FSC2代理可以执行有效的多目标SOP，其捕获率几乎为100％。

在复杂的协作任务上共同努力需要代理商协调他们的行为。在实际交互之前明确或完全执行此操作并不总是可能也不充分。代理人还需要不断了解他人的当前行动，并迅速适应自己的行为。在这里，我们调查我们称之为信仰共鸣的精神状态（意图，目标）的效率，自动协调过程如何导致协作的解决问题。我们为协作剂（HAICA）提出了分层有源推断的模型。它将高效的贝叶斯理论与基于预测处理和主动推断的感知动作系统相结合。通过让一个药物的推断精神状态影响另一个代理人的预测信念来实现信仰共振，从而实现了他自己的目标和意图。这样，推断的精神状态影响了代理人自己的任务行为，没有明确的协作推理。我们在超核域中实施和评估此模型，其中两个代理具有不同程度的信仰共振组合，以满足膳食订单。我们的结果表明，基于HAICA的代理商实现了与最近现有技术方法相当的团队表现，同时产生了更低的计算成本。我们还表明，信仰共振在环境中特别有益，代理商是对环境的不对称知识。结果表明，信仰共振和有效推断允许快速高效的代理协调，因此可以用作合作认知剂的结构块。

未来的互联网涉及几种新兴技术，例如5G和5G网络，车辆网络，无人机（UAV）网络和物联网（IOT）。此外，未来的互联网变得异质并分散了许多相关网络实体。每个实体可能需要做出本地决定，以在动态和不确定的网络环境下改善网络性能。最近使用标准学习算法，例如单药强化学习（RL）或深入强化学习（DRL），以使每个网络实体作为代理人通过与未知环境进行互动来自适应地学习最佳决策策略。但是，这种算法未能对网络实体之间的合作或竞争进行建模，而只是将其他实体视为可能导致非平稳性问题的环境的一部分。多机构增强学习（MARL）允许每个网络实体不仅观察环境，还可以观察其他实体的政策来学习其最佳政策。结果，MAL可以显着提高网络实体的学习效率，并且最近已用于解决新兴网络中的各种问题。在本文中，我们因此回顾了MAL在新兴网络中的应用。特别是，我们提供了MARL的教程，以及对MARL在下一代互联网中的应用进行全面调查。特别是，我们首先介绍单代机Agent RL和MARL。然后，我们回顾了MAL在未来互联网中解决新兴问题的许多应用程序。这些问题包括网络访问，传输电源控制，计算卸载，内容缓存，数据包路由，无人机网络的轨迹设计以及网络安全问题。

尖峰神经网络（SNN）引起了脑启发的人工智能和计算神经科学的广泛关注。它们可用于在多个尺度上模拟大脑中的生物信息处理。更重要的是，SNN是适当的抽象水平，可以将大脑和认知的灵感带入人工智能。在本文中，我们介绍了脑启发的认知智力引擎（Braincog），用于创建脑启发的AI和脑模拟模型。 Braincog将不同类型的尖峰神经元模型，学习规则，大脑区域等作为平台提供的重要模块。基于这些易于使用的模块，BrainCog支持各种受脑启发的认知功能，包括感知和学习，决策，知识表示和推理，运动控制和社会认知。这些受脑启发的AI模型已在各种受监督，无监督和强化学习任务上有效验证，并且可以用来使AI模型具有多种受脑启发的认知功能。为了进行大脑模拟，Braincog实现了决策，工作记忆，神经回路的结构模拟以及小鼠大脑，猕猴大脑和人脑的整个大脑结构模拟的功能模拟。一个名为BORN的AI引擎是基于Braincog开发的，它演示了如何将Braincog的组件集成并用于构建AI模型和应用。为了使科学追求解码生物智能的性质并创建AI，Braincog旨在提供必要且易于使用的构件，并提供基础设施支持，以开发基于脑部的尖峰神经网络AI，并模拟认知大脑在多个尺度上。可以在https://github.com/braincog-x上找到Braincog的在线存储库。

在过去的几十年中，多机构增强学习（MARL）一直在学术界和行业受到广泛关注。 MAL中的基本问题之一是如何全面评估不同的方法。在视频游戏或简单的模拟场景中评估了大多数现有的MAL方法。这些方法在实际情况下，尤其是多机器人系统中的性能仍然未知。本文介绍了一个可扩展的仿真平台，用于多机器人增强学习（MRRL），称为SMART，以满足这一需求。确切地说，智能由两个组成部分组成：1）一个模拟环境，该环境为培训提供了各种复杂的交互场景，以及2）现实世界中的多机器人系统，用于现实的性能评估。此外，SMART提供了代理环境API，这些API是算法实现的插件。为了说明我们平台的实用性，我们就合作驾驶车道变更方案进行了案例研究。在案例研究的基础上，我们总结了MRRL的一些独特挑战，这些挑战很少被考虑。最后，我们为鼓励和增强MRRL研究的仿真环境，相关的基准任务和最先进的基线开放。

Reinforcement Learning (RL) is a popular machine learning paradigm where intelligent agents interact with the environment to fulfill a long-term goal. Driven by the resurgence of deep learning, Deep RL (DRL) has witnessed great success over a wide spectrum of complex control tasks. Despite the encouraging results achieved, the deep neural network-based backbone is widely deemed as a black box that impedes practitioners to trust and employ trained agents in realistic scenarios where high security and reliability are essential. To alleviate this issue, a large volume of literature devoted to shedding light on the inner workings of the intelligent agents has been proposed, by constructing intrinsic interpretability or post-hoc explainability. In this survey, we provide a comprehensive review of existing works on eXplainable RL (XRL) and introduce a new taxonomy where prior works are clearly categorized into model-explaining, reward-explaining, state-explaining, and task-explaining methods. We also review and highlight RL methods that conversely leverage human knowledge to promote learning efficiency and performance of agents while this kind of method is often ignored in XRL field. Some challenges and opportunities in XRL are discussed. This survey intends to provide a high-level summarization of XRL and to motivate future research on more effective XRL solutions. Corresponding open source codes are collected and categorized at https://github.com/Plankson/awesome-explainable-reinforcement-learning.

Adequate strategizing of agents behaviors is essential to solving cooperative MARL problems. One intuitively beneficial yet uncommon method in this domain is predicting agents future behaviors and planning accordingly. Leveraging this point, we propose a two-level hierarchical architecture that combines a novel information-theoretic objective with a trajectory prediction model to learn a strategy. To this end, we introduce a latent policy that learns two types of latent strategies: individual $z_A$, and relational $z_R$ using a modified Graph Attention Network module to extract interaction features. We encourage each agent to behave according to the strategy by conditioning its local $Q$ functions on $z_A$, and we further equip agents with a shared $Q$ function that conditions on $z_R$. Additionally, we introduce two regularizers to allow predicted trajectories to be accurate and rewarding. Empirical results on Google Research Football (GRF) and StarCraft (SC) II micromanagement tasks show that our method establishes a new state of the art being, to the best of our knowledge, the first MARL algorithm to solve all super hard SC II scenarios as well as the GRF full game with a win rate higher than $95\%$, thus outperforming all existing methods. Videos and brief overview of the methods and results are available at: https://sites.google.com/view/hier-strats-marl/home.

如今，合作多代理系统用于学习如何在大规模动态环境中实现目标。然而，在这些环境中的学习是具有挑战性的：从搜索空间大小对学习时间的影响，代理商之间的低效合作。此外，增强学习算法可能遭受这种环境的长时间的收敛。本文介绍了通信框架。在拟议的沟通框架中，代理商学会有效地合作，同时通过引入新的状态计算方法，状态空间的大小将大大下降。此外，提出了一种知识传输算法以共享不同代理商之间的获得经验，并制定有效的知识融合机制，以融合利用来自其他团队成员所收到的知识的代理商自己的经验。最后，提供了模拟结果以指示所提出的方法在复杂学习任务中的功效。我们已经评估了我们对牧羊化问题的方法，结果表明，通过利用知识转移机制，学习过程加速了，通过基于状态抽象概念产生类似国家的状态空间的大小均下降。