我们通过集中式培训和分散执行研究多代理增强学习（MARL）。在培训期间，新代理商可能会加入，现有代理商可能会意外离开培训。在这种情况下，必须再次从头开始训练标准的深色MAL模型，这非常耗时。为了解决这个问题，我们提出了一种特殊的网络体系结构，其中包括一些弹跳学习算法，该算法允许在集中式培训期间代理的数量变化。特别是，当新代理参加集中式培训时，我们的几次学习算法使用少量样本训练其政策网络和价值网络；当代理离开训练时，其余代理的训练过程不受影响。我们的实验表明，使用提出的网络体系结构和算法，当新代理连接的速度比基线快100倍以上时，模型适应。我们的工作适用于任何环境，包括合作，竞争和混合。

Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in singleagent settings. We present an actor-critic algorithm that trains decentralized policies in multiagent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep. This attention mechanism enables more effective and scalable learning in complex multiagent environments, when compared to recent approaches. Our approach is applicable not only to cooperative settings with shared rewards, but also individualized reward settings, including adversarial settings, as well as settings that do not provide global states, and it makes no assumptions about the action spaces of the agents. As such, it is flexible enough to be applied to most multi-agent learning problems.

将深度强化学习（DRL）扩展到多代理领域的研究已经解决了许多复杂的问题，并取得了重大成就。但是，几乎所有这些研究都只关注离散或连续的动作空间，而且很少有作品曾经使用过多代理的深度强化学习来实现现实世界中的环境问题，这些问题主要具有混合动作空间。因此，在本文中，我们提出了两种算法：深层混合软性角色批评（MAHSAC）和多代理混合杂种深层确定性政策梯度（MAHDDPG）来填补这一空白。这两种算法遵循集中式培训和分散执行（CTDE）范式，并可以解决混合动作空间问题。我们的经验在多代理粒子环境上运行，这是一个简单的多代理粒子世界，以及一些基本的模拟物理。实验结果表明，这些算法具有良好的性能。

多代理深入的强化学习已应用于解决各种离散或连续动作空间的各种复杂问题，并取得了巨大的成功。但是，大多数实际环境不能仅通过离散的动作空间或连续的动作空间来描述。而且很少有作品曾经利用深入的加固学习（DRL）来解决混合动作空间的多代理问题。因此，我们提出了一种新颖的算法：深层混合软性角色 - 批评（MAHSAC）来填补这一空白。该算法遵循集中式训练但分散执行（CTDE）范式，并扩展软actor-Critic算法（SAC），以根据最大熵在多机构环境中处理混合动作空间问题。我们的经验在一个简单的多代理粒子世界上运行，具有连续的观察和离散的动作空间以及一些基本的模拟物理。实验结果表明，MAHSAC在训练速度，稳定性和抗干扰能力方面具有良好的性能。同时，它在合作场景和竞争性场景中胜过现有的独立深层学习方法。

We explore deep reinforcement learning methods for multi-agent domains. We begin by analyzing the difficulty of traditional algorithms in the multi-agent case: Q-learning is challenged by an inherent non-stationarity of the environment, while policy gradient suffers from a variance that increases as the number of agents grows. We then present an adaptation of actor-critic methods that considers action policies of other agents and is able to successfully learn policies that require complex multiagent coordination. Additionally, we introduce a training regimen utilizing an ensemble of policies for each agent that leads to more robust multi-agent policies. We show the strength of our approach compared to existing methods in cooperative as well as competitive scenarios, where agent populations are able to discover various physical and informational coordination strategies.

Cooperative multi-agent reinforcement learning (MARL) has achieved significant results, most notably by leveraging the representation-learning abilities of deep neural networks. However, large centralized approaches quickly become infeasible as the number of agents scale, and fully decentralized approaches can miss important opportunities for information sharing and coordination. Furthermore, not all agents are equal -- in some cases, individual agents may not even have the ability to send communication to other agents or explicitly model other agents. This paper considers the case where there is a single, powerful, \emph{central agent} that can observe the entire observation space, and there are multiple, low-powered \emph{local agents} that can only receive local observations and are not able to communicate with each other. The central agent's job is to learn what message needs to be sent to different local agents based on the global observations, not by centrally solving the entire problem and sending action commands, but by determining what additional information an individual agent should receive so that it can make a better decision. In this work we present our MARL algorithm \algo, describe where it would be most applicable, and implement it in the cooperative navigation and multi-agent walker domains. Empirical results show that 1) learned communication does indeed improve system performance, 2) results generalize to heterogeneous local agents, and 3) results generalize to different reward structures.

在合作多智能体增强学习（Marl）中的代理商的创造和破坏是一个批判性的研究领域。当前的Marl算法通常认为，在整个实验中，组内的代理数量仍然是固定的。但是，在许多实际问题中，代理人可以在队友之前终止。这次早期终止问题呈现出挑战：终止的代理人必须从本集团的成功或失败中学习，这是超出其自身存在的成败。我们指代薪资奖励的传播价值作为遣返代理商作为追索的奖励作为追索权。当前的MARL方法通过将这些药剂放在吸收状态下，直到整组试剂达到终止条件，通过将这些药剂置于终止状态来处理该问题。虽然吸收状态使现有的算法和API能够在没有修改的情况下处理终止的代理，但存在实际培训效率和资源使用问题。在这项工作中，我们首先表明样本复杂性随着系统监督学习任务中的吸收状态的数量而增加，同时对变量尺寸输入更加强大。然后，我们为现有的最先进的MARL算法提出了一种新颖的架构，它使用注意而不是具有吸收状态的完全连接的层。最后，我们展示了这一新颖架构在剧集中创建或销毁的任务中的标准架构显着优于标准架构以及标准的多代理协调任务。

We consider the problem of multi-agent navigation and collision avoidance when observations are limited to the local neighborhood of each agent. We propose InforMARL, a novel architecture for multi-agent reinforcement learning (MARL) which uses local information intelligently to compute paths for all the agents in a decentralized manner. Specifically, InforMARL aggregates information about the local neighborhood of agents for both the actor and the critic using a graph neural network and can be used in conjunction with any standard MARL algorithm. We show that (1) in training, InforMARL has better sample efficiency and performance than baseline approaches, despite using less information, and (2) in testing, it scales well to environments with arbitrary numbers of agents and obstacles.

分散的学习对合作多代理增强学习（MARL）表现出了巨大的希望。但是，非平稳性仍然是分散学习的重大挑战。在论文中，我们以最简单和基本的方式解决了非平稳性问题，并提出\ textit {多代理替代Q学习}（MA2QL），在那里，代理商轮流通过Q学习来更新其Q-函数。MA2QL是完全分散合作MARL的一种\ Textit {Minimalist}方法，但理论上是基础的。我们证明，当每个代理商在每个回合都保证$ \ varepsilon $ -Convergence时，他们的联合政策会收敛到NASH平衡。实际上，MA2QL仅需要对独立Q学习（IQL）的最小变化。我们经验评估MA2QL对各种合作的多代理任务。结果表明，MA2QL始终胜过IQL，尽管这种变化很小，但它验证了MA2QL的有效性。

政策梯度方法在多智能体增强学习中变得流行，但由于存在环境随机性和探索代理（即非公平性​​），它们遭受了高度的差异，这可能因信用分配难度而受到困扰。结果，需要一种方法，该方法不仅能够有效地解决上述两个问题，而且需要足够强大地解决各种任务。为此，我们提出了一种新的多代理政策梯度方法，称为强大的本地优势（ROLA）演员 - 评论家。 Rola允许每个代理人将个人动作值函数作为当地评论家，以及通过基于集中评论家的新型集中培训方法来改善环境不良。通过使用此本地批评，每个代理都计算基准，以减少对其策略梯度估计的差异，这导致含有其他代理的预期优势动作值，这些选项可以隐式提高信用分配。我们在各种基准测试中评估ROLA，并在许多最先进的多代理政策梯度算法上显示其鲁棒性和有效性。

This work considers the problem of learning cooperative policies in complex, partially observable domains without explicit communication. We extend three classes of single-agent deep reinforcement learning algorithms based on policy gradient, temporal-difference error, and actor-critic methods to cooperative multi-agent systems. We introduce a set of cooperative control tasks that includes tasks with discrete and continuous actions, as well as tasks that involve hundreds of agents. The three approaches are evaluated against each other using different neural architectures, training procedures, and reward structures. Using deep reinforcement learning with a curriculum learning scheme, our approach can solve problems that were previously considered intractable by most multi-agent reinforcement learning algorithms. We show that policy gradient methods tend to outperform both temporal-difference and actor-critic methods when using feed-forward neural architectures. We also show that recurrent policies, while more difficult to train, outperform feed-forward policies on our evaluation tasks.

In multi-agent reinforcement learning (MARL), many popular methods, such as VDN and QMIX, are susceptible to a critical multi-agent pathology known as relative overgeneralization (RO), which arises when the optimal joint action's utility falls below that of a sub-optimal joint action in cooperative tasks. RO can cause the agents to get stuck into local optima or fail to solve tasks that require significant coordination between agents within a given timestep. Recent value-based MARL algorithms such as QPLEX and WQMIX can overcome RO to some extent. However, our experimental results show that they can still fail to solve cooperative tasks that exhibit strong RO. In this work, we propose a novel approach called curriculum learning for relative overgeneralization (CURO) to better overcome RO. To solve a target task that exhibits strong RO, in CURO, we first fine-tune the reward function of the target task to generate source tasks that are tailored to the current ability of the learning agent and train the agent on these source tasks first. Then, to effectively transfer the knowledge acquired in one task to the next, we use a novel transfer learning method that combines value function transfer with buffer transfer, which enables more efficient exploration in the target task. We demonstrate that, when applied to QMIX, CURO overcomes severe RO problem and significantly improves performance, yielding state-of-the-art results in a variety of cooperative multi-agent tasks, including the challenging StarCraft II micromanagement benchmarks.

我们开发了一个多功能辅助救援学习（MARL）方法，以了解目标跟踪的可扩展控制策略。我们的方法可以处理任意数量的追求者和目标;我们显示出现的任务，该任务包括高达1000追踪跟踪1000个目标。我们使用分散的部分可观察的马尔可夫决策过程框架来模拟追求者作为接受偏见观察（范围和轴承）的代理，了解使用固定的未知政策的目标。注意机制用于参数化代理的价值函数;这种机制允许我们处理任意数量的目标。熵 - 正规的脱助政策RL方法用于培训随机政策，我们讨论如何在追求者之间实现对冲行为，尽管有完全分散的控制执行，但仍然导致合作较弱的合作形式。我们进一步开发了一个掩蔽启发式，允许训练较少的问题，少量追求目标和在更大的问题上执行。进行彻底的仿真实验，消融研究和对现有技术算法的比较，以研究对不同数量的代理和目标性能的方法和鲁棒性的可扩展性。

Many real-world problems, such as network packet routing and the coordination of autonomous vehicles, are naturally modelled as cooperative multi-agent systems. There is a great need for new reinforcement learning methods that can efficiently learn decentralised policies for such systems. To this end, we propose a new multi-agent actor-critic method called counterfactual multi-agent (COMA) policy gradients. COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies. In addition, to address the challenges of multi-agent credit assignment, it uses a counterfactual baseline that marginalises out a single agent's action, while keeping the other agents' actions fixed. COMA also uses a critic representation that allows the counterfactual baseline to be computed efficiently in a single forward pass. We evaluate COMA in the testbed of StarCraft unit micromanagement, using a decentralised variant with significant partial observability. COMA significantly improves average performance over other multi-agent actorcritic methods in this setting, and the best performing agents are competitive with state-of-the-art centralised controllers that get access to the full state.

本文考虑了多智能经纪人强化学习（MARL）任务，代理商在集会结束时获得共享全球奖励。这种奖励的延迟性质影响了代理商在中间时间步骤中评估其行动质量的能力。本文侧重于开发学习焦点奖励的时间重新分布的方法，以获得密集奖励信号。解决这些MARL问题需要解决两个挑战：识别（1）沿着集发作（沿时间）的长度相对重要性，以及（2）在任何单一时间步骤（代理商中）的相对重要性。在本文中，我们介绍了奖励中的奖励再分配，在整容多智能体加固学习（Arel）中奖励再分配，以解决这两个挑战。 Arel使用注意机制来表征沿着轨迹（时间关注）对状态转换的动作的影响，以及每个代理在每个时间步骤（代理人注意）的影响。 Arel预测的重新分配奖励是密集的，可以与任何给定的MARL算法集成。我们评估了粒子世界环境的具有挑战性的任务和星际争霸多功能挑战。 arel导致粒子世界的奖励较高，并改善星际争端的胜利率与三个最先进的奖励再分配方法相比。我们的代码可在https://github.com/baicenxiao/arel获得。

Offline multi-agent reinforcement learning (MARL) aims to learn effective multi-agent policies from pre-collected datasets, which is an important step toward the deployment of multi-agent systems in real-world applications. However, in practice, each individual behavior policy that generates multi-agent joint trajectories usually has a different level of how well it performs. e.g., an agent is a random policy while other agents are medium policies. In the cooperative game with global reward, one agent learned by existing offline MARL often inherits this random policy, jeopardizing the performance of the entire team. In this paper, we investigate offline MARL with explicit consideration on the diversity of agent-wise trajectories and propose a novel framework called Shared Individual Trajectories (SIT) to address this problem. Specifically, an attention-based reward decomposition network assigns the credit to each agent through a differentiable key-value memory mechanism in an offline manner. These decomposed credits are then used to reconstruct the joint offline datasets into prioritized experience replay with individual trajectories, thereafter agents can share their good trajectories and conservatively train their policies with a graph attention network (GAT) based critic. We evaluate our method in both discrete control (i.e., StarCraft II and multi-agent particle environment) and continuous control (i.e, multi-agent mujoco). The results indicate that our method achieves significantly better results in complex and mixed offline multi-agent datasets, especially when the difference of data quality between individual trajectories is large.

In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifted. Learning joint actionvalues conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a network that estimates joint action-values as a complex non-linear combination of per-agent values that condition only on local observations. We structurally enforce that the joint-action value is monotonic in the per-agent values, which allows tractable maximisation of the joint action-value in off-policy learning, and guarantees consistency between the centralised and decentralised policies. We evaluate QMIX on a challenging set of StarCraft II micromanagement tasks, and show that QMIX significantly outperforms existing value-based multi-agent reinforcement learning methods.

独立的强化学习算法没有理论保证，用于在多代理设置中找到最佳策略。然而，在实践中，先前的作品报告了在某些域中的独立算法和其他方面的良好性能。此外，文献中缺乏对独立算法的优势和弱点的全面研究。在本文中，我们对四个Pettingzoo环境进行了独立算法的性能的实证比较，这些环境跨越了三种主要类别的多助理环境，即合作，竞争和混合。我们表明，在完全可观察的环境中，独立的算法可以在协作和竞争环境中与多代理算法进行同步。对于混合环境，我们表明通过独立算法培训的代理商学会单独执行，但未能学会与盟友合作并与敌人竞争。我们还表明，添加重复性提高了合作部分可观察环境中独立算法的学习。

协作多代理增强学习（MARL）已在许多实际应用中广泛使用，在许多实际应用中，每个代理商都根据自己的观察做出决定。大多数主流方法在对分散的局部实用程序函数进行建模时，将每个局部观察结果视为完整的。但是，他们忽略了这样一个事实，即可以将局部观察信息进一步分为几个实体，只有一部分实体有助于建模推理。此外，不同实体的重要性可能会随着时间而变化。为了提高分散政策的性能，使用注意机制用于捕获本地信息的特征。然而，现有的注意模型依赖于密集的完全连接的图，并且无法更好地感知重要状态。为此，我们提出了一个稀疏的状态MARL（S2RL）框架，该框架利用稀疏的注意机制将无关的信息丢弃在局部观察中。通过自我注意力和稀疏注意机制估算局部效用函数，然后将其合并为标准的关节价值函数和中央评论家的辅助关节价值函数。我们将S2RL框架设计为即插即用的模块，使其足够一般，可以应用于各种方法。关于Starcraft II的广泛实验表明，S2RL可以显着提高许多最新方法的性能。

可再生能源的增加集成为电源分销网络的运行带来了许多技术挑战。其中，由可再生能源的不稳定性引起的电压波动正在受到越来越多的关注。最近在主动电压控制任务中广泛研究了电网中的多个控制单元（能够处理电源系统快速变化）中的多个控制单元。但是，基于MARL的现有方法忽略了网格的独特性质，并实现有限的性能。在本文中，我们介绍了变压器体系结构，以提取适应电力网络问题的表示形式，并提出基于变压器的多代理参与者 - 批判框架（T-MAAC）以稳定电源分配网络中的电压。此外，我们采用了针对电压控制任务量身定制的新型辅助任务训练过程，从而提高了样品效率并促进基于变压器模型的表示。我们将T-MAAC与不同的多代理 - 参与者批评算法相结合，而主动电压控制任务的一致改进证明了该方法的有效性。