In multi-agent systems with large number of agents, typically the contribution of each agent to the value of other agents is minimal (e.g., aggregation systems such as Uber, Deliveroo). In this paper, we consider such multi-agent systems where each agent is self-interested and takes a sequence of decisions and represent them as a Stochastic Non-atomic Congestion Game (SNCG). We derive key properties for equilibrium solutions in SNCG model with non-atomic and also nearly non-atomic agents. With those key equilibrium properties, we provide a novel Multi-Agent Reinforcement Learning (MARL) mechanism that minimizes variance across values of agents in the same state. To demonstrate the utility of this new mechanism, we provide detailed results on a real-world taxi dataset and also a generic simulator for aggregation systems. We show that our approach reduces the variance in revenues earned by taxi drivers, while still providing higher joint revenues than leading approaches.
translated by 谷歌翻译
在过去的十年中,多智能经纪人强化学习(Marl)已经有了重大进展,但仍存在许多挑战,例如高样本复杂性和慢趋同稳定的政策,在广泛的部署之前需要克服,这是可能的。然而,在实践中,许多现实世界的环境已经部署了用于生成策略的次优或启发式方法。一个有趣的问题是如何最好地使用这些方法作为顾问,以帮助改善多代理领域的加强学习。在本文中,我们提供了一个原则的框架,用于将动作建议纳入多代理设置中的在线次优顾问。我们描述了在非传记通用随机游戏环境中提供多种智能强化代理(海军上将)的问题,并提出了两种新的基于Q学习的算法:海军上将决策(海军DM)和海军上将 - 顾问评估(Admiral-AE) ,这使我们能够通过适当地纳入顾问(Admiral-DM)的建议来改善学习,并评估顾问(Admiral-AE)的有效性。我们从理论上分析了算法,并在一般加上随机游戏中提供了关于他们学习的定点保证。此外,广泛的实验说明了这些算法:可以在各种环境中使用,具有对其他相关基线的有利相比的性能,可以扩展到大状态行动空间,并且对来自顾问的不良建议具有稳健性。
translated by 谷歌翻译
在本文中,我们介绍了有关典型乘车共享系统中决策优化问题的强化学习方法的全面,深入的调查。涵盖了有关乘车匹配,车辆重新定位,乘车,路由和动态定价主题的论文。在过去的几年中,大多数文献都出现了,并且要继续解决一些核心挑战:模型复杂性,代理协调和多个杠杆的联合优化。因此,我们还引入了流行的数据集和开放式仿真环境,以促进进一步的研发。随后,我们讨论了有关该重要领域的强化学习研究的许多挑战和机会。
translated by 谷歌翻译
具有很多玩家的非合作和合作游戏具有许多应用程序,但是当玩家数量增加时,通常仍然很棘手。由Lasry和Lions以及Huang,Caines和Malham \'E引入的,平均野外运动会(MFGS)依靠平均场外近似值,以使玩家数量可以成长为无穷大。解决这些游戏的传统方法通常依赖于以完全了解模型的了解来求解部分或随机微分方程。最近,增强学习(RL)似乎有望解决复杂问题。通过组合MFGS和RL,我们希望在人口规模和环境复杂性方面能够大规模解决游戏。在这项调查中,我们回顾了有关学习MFG中NASH均衡的最新文献。我们首先确定最常见的设置(静态,固定和进化)。然后,我们为经典迭代方法(基于最佳响应计算或策略评估)提供了一个通用框架,以确切的方式解决MFG。在这些算法和与马尔可夫决策过程的联系的基础上,我们解释了如何使用RL以无模型的方式学习MFG解决方案。最后,我们在基准问题上介绍了数值插图,并以某些视角得出结论。
translated by 谷歌翻译
Various types of Multi-Agent Reinforcement Learning (MARL) methods have been developed, assuming that agents' policies are based on true states. Recent works have improved the robustness of MARL under uncertainties from the reward, transition probability, or other partners' policies. However, in real-world multi-agent systems, state estimations may be perturbed by sensor measurement noise or even adversaries. Agents' policies trained with only true state information will deviate from optimal solutions when facing adversarial state perturbations during execution. MARL under adversarial state perturbations has limited study. Hence, in this work, we propose a State-Adversarial Markov Game (SAMG) and make the first attempt to study the fundamental properties of MARL under state uncertainties. We prove that the optimal agent policy and the robust Nash equilibrium do not always exist for an SAMG. Instead, we define the solution concept, robust agent policy, of the proposed SAMG under adversarial state perturbations, where agents want to maximize the worst-case expected state value. We then design a gradient descent ascent-based robust MARL algorithm to learn the robust policies for the MARL agents. Our experiments show that adversarial state perturbations decrease agents' rewards for several baselines from the existing literature, while our algorithm outperforms baselines with state perturbations and significantly improves the robustness of the MARL policies under state uncertainties.
translated by 谷歌翻译
多代理游戏中的均衡选择是指选择帕累托最佳平衡的问题。已经表明,由于每个代理商在训练过程中对其他代理商的政策的不确定性,许多最先进的多机构增强学习(MARL)算法容易融合到帕累托主导的平衡。为了解决次优的平衡选择,我们提出了一种使用无关紧要游戏的简单原则(具有相同奖励的超级合作游戏)的参与者批评算法(PAC):每个代理人都可以假设其他人会选择动作的动作这将导致帕累托最佳平衡。我们评估了PAC在一系列多种多样的游戏中,并表明与替代MARL算法相比,它会收敛到更高的情节回报,并在一系列矩阵游戏中成功收敛到帕累托优势。最后,我们提出了一个图形神经网络扩展,该扩展可以在具有多达15个代理商的游戏中有效地扩展。
translated by 谷歌翻译
平均场理论提供了一种将多基强化学习算法扩展到许多代理可以由虚拟均值代理提取的环境的有效方法。在本文中,我们将平均字段多基因算法扩展到多种类型。这种类型使平均田间强化学习中的核心假设可以放松,即环境中的所有代理都在采用几乎相似的策略,并且具有相同的目标。我们基于标准的魔法框架,对许多代理增强学习领域的三个不同测试床进行实验。我们考虑两种不同类型的平均场环境:a)代理属于预定义类型的游戏,这些类型是先验和b)每个代理的类型未知的游戏,因此必须根据观察结果学习。我们为每种类型的游戏介绍了新的算法,并演示了它们优于最先进的算法,这些算法假定所有代理都属于Magent Framework中的所有代理属于相同类型和其他基线算法。
translated by 谷歌翻译
我们研究了竞争激烈的马尔可夫游戏(MG)环境中的NASH平衡学习,其中多个代理商竞争,并且可以存在多个NASH均衡。特别是,对于寡头的动态定价环境,由于差异性的诅咒,难以获得精确的NASH平衡。我们开发了一种新的无模型方法来找到近似NASH平衡。然后,将无梯度的黑匣子优化应用于估计$ \ epsilon $,这是代理商单方面偏离任何联合政策的最大奖励优势,并估算了任何给定州的$ \ epsilon $降低政策。政策 - $ \ epsilon $通讯和国家对$ \ epsilon $ - 缩小政策的政策由神经网络表示,后者是NASH策略网。在批处理更新期间,我们通过使用NASH策略网调整操作概率在系统上进行NASH Q学习。我们证明可以学习近似的NASH平衡,尤其是在精确溶液通常很棘手的动态定价域中。
translated by 谷歌翻译
Current approaches to multi-agent cooperation rely heavily on centralized mechanisms or explicit communication protocols to ensure convergence. This paper studies the problem of distributed multi-agent learning without resorting to centralized components or explicit communication. It examines the use of distribution matching to facilitate the coordination of independent agents. In the proposed scheme, each agent independently minimizes the distribution mismatch to the corresponding component of a target visitation distribution. The theoretical analysis shows that under certain conditions, each agent minimizing its individual distribution mismatch allows the convergence to the joint policy that generated the target distribution. Further, if the target distribution is from a joint policy that optimizes a cooperative task, the optimal policy for a combination of this task reward and the distribution matching reward is the same joint policy. This insight is used to formulate a practical algorithm (DM$^2$), in which each individual agent matches a target distribution derived from concurrently sampled trajectories from a joint expert policy. Experimental validation on the StarCraft domain shows that combining (1) a task reward, and (2) a distribution matching reward for expert demonstrations for the same task, allows agents to outperform a naive distributed baseline. Additional experiments probe the conditions under which expert demonstrations need to be sampled to obtain the learning benefits.
translated by 谷歌翻译
随机游戏的学习可以说是多功能钢筋学习(MARL)中最标准和最基本的环境。在本文中,我们考虑在非渐近制度的随机游戏中分散的Marl。特别是,我们在大量的一般总和随机游戏(SGS)中建立了完全分散的Q学习算法的有限样本复杂性 - 弱循环SGS,包括对所有代理商的普通合作MARL设置具有相同的奖励(马尔可夫团队问题是一个特例。我们专注于实用的同时具有挑战性地设置完全分散的Marl,既不奖励也没有其他药剂的作用,每个试剂都可以观察到。事实上,每个特工都完全忘记了其他决策者的存在。表格和线性函数近似情况都已考虑。在表格设置中,我们分析了分散的Q学习算法的样本复杂性,以收敛到马尔可夫完美均衡(NASH均衡)。利用线性函数近似,结果用于收敛到线性近似平衡 - 我们提出的均衡的新概念 - 这描述了每个代理的策略是线性空间内的最佳回复(到其他代理)。还提供了数值实验,用于展示结果。
translated by 谷歌翻译
由于共同国家行动空间相对于代理人的数量,多代理强化学习(MARL)中的政策学习(MARL)是具有挑战性的。为了实现更高的可伸缩性,通过分解执行(CTDE)的集中式培训范式被MARL中的分解结构广泛采用。但是,我们观察到,即使在简单的矩阵游戏中,合作MARL中现有的CTDE算法也无法实现最佳性。为了理解这种现象,我们引入了一个具有政策分解(GPF-MAC)的广义多代理参与者批评的框架,该框架的特征是对分解的联合政策的学习,即,每个代理人的政策仅取决于其自己的观察行动历史。我们表明,最受欢迎的CTDE MARL算法是GPF-MAC的特殊实例,可能会陷入次优的联合政策中。为了解决这个问题,我们提出了一个新颖的转型框架,该框架将多代理的MDP重新制定为具有连续结构的特殊“单位代理” MDP,并且可以允许使用现成的单机械加固学习(SARL)算法来有效地学习相应的多代理任务。这种转换保留了SARL算法的最佳保证,以合作MARL。为了实例化此转换框架,我们提出了一个转换的PPO,称为T-PPO,该PPO可以在有限的多代理MDP中进行理论上执行最佳的策略学习,并在一系列合作的多代理任务上显示出明显的超出性能。
translated by 谷歌翻译
交通优化挑战,如负载平衡,流量调度和提高数据包交付时间,是广域网(WAN)中困难的在线决策问题。例如,需要复杂的启发式方法,以找到改善分组输送时间并最小化可能由链接故障或拥塞引起的中断的最佳路径。最近的加强学习(RL)算法的成功可以提供有用的解决方案,以建立更好的鲁棒系统,这些系统从无模式设置中学习。在这项工作中,我们考虑了一条路径优化问题,专门针对数据包路由,在大型复杂网络中。我们开发和评估一种无模型方法,应用多代理元增强学习(MAMRL),可以确定每个数据包的下一跳,以便将其传递到其目的地,最短的时间整体。具体地,我们建议利用和比较深度策略优化RL算法,以便在通信网络中启用分布式无模型控制,并呈现基于新的Meta学习的框架Mamrl,以便快速适应拓扑变化。为了评估所提出的框架,我们用各种WAN拓扑模拟。我们广泛的数据包级仿真结果表明,与古典最短路径和传统的加强学习方法相比,Mamrl即使网络需求增加也显着降低了平均分组交付时间;与非元深策略优化算法相比,我们的结果显示在连杆故障发生的同时出现相当的平均数据包交付时间时减少较少的剧集中的数据包丢失。
translated by 谷歌翻译
流动性和流量的许多方案都涉及多种不同的代理,需要合作以找到共同解决方案。行为计划的最新进展使用强化学习以寻找有效和绩效行为策略。但是,随着自动驾驶汽车和车辆对X通信变得越来越成熟,只有使用单身独立代理的解决方案在道路上留下了潜在的性能增长。多代理增强学习(MARL)是一个研究领域,旨在为彼此相互作用的多种代理找到最佳解决方案。这项工作旨在将该领域的概述介绍给研究人员的自主行动能力。我们首先解释Marl并介绍重要的概念。然后,我们讨论基于Marl算法的主要范式,并概述每个范式中最先进的方法和思想。在这种背景下,我们调查了MAL在自动移动性场景中的应用程序,并概述了现有的场景和实现。
translated by 谷歌翻译
大型人口系统的分析和控制对研究和工程的各个领域引起了极大的兴趣,从机器人群的流行病学到经济学和金融。一种越来越流行和有效的方法来实现多代理系统中的顺序决策,这是通过多机构增强学习,因为它允许对高度复杂的系统进行自动和无模型的分析。但是,可伸缩性的关键问题使控制和增强学习算法的设计变得复杂,尤其是在具有大量代理的系统中。尽管强化学习在许多情况下都发现了经验成功,但许多代理商的问题很快就变得棘手了,需要特别考虑。在这项调查中,我们将阐明当前的方法,以通过多代理强化学习以及通过诸如平均场游戏,集体智能或复杂的网络理论等研究领域进行仔细理解和分析大型人口系统。这些经典独立的主题领域提供了多种理解或建模大型人口系统的方法,这可能非常适合将来的可拖动MARL算法制定。最后,我们调查了大规模控制的潜在应用领域,并确定了实用系统中学习算法的富有成果的未来应用。我们希望我们的调查可以为理论和应用科学的初级和高级研究人员提供洞察力和未来的方向。
translated by 谷歌翻译
This work considers the problem of learning cooperative policies in complex, partially observable domains without explicit communication. We extend three classes of single-agent deep reinforcement learning algorithms based on policy gradient, temporal-difference error, and actor-critic methods to cooperative multi-agent systems. We introduce a set of cooperative control tasks that includes tasks with discrete and continuous actions, as well as tasks that involve hundreds of agents. The three approaches are evaluated against each other using different neural architectures, training procedures, and reward structures. Using deep reinforcement learning with a curriculum learning scheme, our approach can solve problems that were previously considered intractable by most multi-agent reinforcement learning algorithms. We show that policy gradient methods tend to outperform both temporal-difference and actor-critic methods when using feed-forward neural architectures. We also show that recurrent policies, while more difficult to train, outperform feed-forward policies on our evaluation tasks.
translated by 谷歌翻译
用于分散执行的集中培训,其中代理商使用集中信息训练,但在线以分散的方式执行,在多智能体增强学习界中获得了普及。特别是,具有集中评论家和分散的演员的演员 - 批评方法是这个想法的常见实例。然而,即使它是许多算法的标准选择,也没有完全讨论和理解使用集中评论批读的影响。因此,我们正式分析集中和分散的批评批评方法,了解对评论家选择的影响。由于我们的理论使得不切实际的假设,我们还经验化地比较了广泛的环境中集中式和分散的批评方法来验证我们的理论并提供实用建议。我们展示了当前文献中集中评论家存在误解,并表明集中式评论家设计并不是严格用的,而是集中和分散的批评者具有不同的利弊,算法设计人员应该考虑到不同的利弊。
translated by 谷歌翻译
Reinforcement Learning (RL) is currently one of the most commonly used techniques for traffic signal control (TSC), which can adaptively adjusted traffic signal phase and duration according to real-time traffic data. However, a fully centralized RL approach is beset with difficulties in a multi-network scenario because of exponential growth in state-action space with increasing intersections. Multi-agent reinforcement learning (MARL) can overcome the high-dimension problem by employing the global control of each local RL agent, but it also brings new challenges, such as the failure of convergence caused by the non-stationary Markov Decision Process (MDP). In this paper, we introduce an off-policy nash deep Q-Network (OPNDQN) algorithm, which mitigates the weakness of both fully centralized and MARL approaches. The OPNDQN algorithm solves the problem that traditional algorithms cannot be used in large state-action space traffic models by utilizing a fictitious game approach at each iteration to find the nash equilibrium among neighboring intersections, from which no intersection has incentive to unilaterally deviate. One of main advantages of OPNDQN is to mitigate the non-stationarity of multi-agent Markov process because it considers the mutual influence among neighboring intersections by sharing their actions. On the other hand, for training a large traffic network, the convergence rate of OPNDQN is higher than that of existing MARL approaches because it does not incorporate all state information of each agent. We conduct an extensive experiments by using Simulation of Urban MObility simulator (SUMO), and show the dominant superiority of OPNDQN over several existing MARL approaches in terms of average queue length, episode training reward and average waiting time.
translated by 谷歌翻译
We propose a multi-agent reinforcement learning dynamics, and analyze its convergence properties in infinite-horizon discounted Markov potential games. We focus on the independent and decentralized setting, where players can only observe the realized state and their own reward in every stage. Players do not have knowledge of the game model, and cannot coordinate with each other. In each stage of our learning dynamics, players update their estimate of a perturbed Q-function that evaluates their total contingent payoff based on the realized one-stage reward in an asynchronous manner. Then, players independently update their policies by incorporating a smoothed optimal one-stage deviation strategy based on the estimated Q-function. A key feature of the learning dynamics is that the Q-function estimates are updated at a faster timescale than the policies. We prove that the policies induced by our learning dynamics converge to a stationary Nash equilibrium in Markov potential games with probability 1. Our results demonstrate that agents can reach a stationary Nash equilibrium in Markov potential games through simple learning dynamics under the minimum information environment.
translated by 谷歌翻译
Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in singleagent settings. We present an actor-critic algorithm that trains decentralized policies in multiagent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep. This attention mechanism enables more effective and scalable learning in complex multiagent environments, when compared to recent approaches. Our approach is applicable not only to cooperative settings with shared rewards, but also individualized reward settings, including adversarial settings, as well as settings that do not provide global states, and it makes no assumptions about the action spaces of the agents. As such, it is flexible enough to be applied to most multi-agent learning problems.
translated by 谷歌翻译
实际经济体可以被视为一种顺序不完美信息游戏,具有许多异质,互动的各种代理类型的战略代理,例如消费者,公司和政府。动态一般均衡模型是在此类系统中建模经济活动,交互和结果的普通经济工具。然而,当所有代理商是战略和互动时,现有的分析和计算方法努力寻找明确的均衡,而联合学习是不稳定的并且具有挑战性。在其他人中,一个重要的原因是,一个经济代理人的行动可能会改变另一名代理人的奖励职能,例如,当公司更改价格或政府更改税收时,消费者的消费者的消费收入变化。我们表明,多代理深度加强学习(RL)可以发现稳定的解决方案,即通过使用结构的学习课程和高效的GPU,在经济模拟中,在经济仿真中,在经济模拟中,可以发现普遍存器类型的稳定解决方案。仿真和培训。概念上,我们的方法更加灵活,不需要不切实际的假设,例如市场清算,通常用于分析途径。我们的GPU实施使得能够在合理的时间范围内具有大量代理的经济体,例如,在一天内完成培训。我们展示了我们在实际商业周期模型中的方法,这是一个代表性的DGE模型系列,100名工人消费者,10家公司和政府税收和重新分配。我们通过近似最佳响应分析验证了学习的Meta-Game epsilon-Nash均衡,表明RL政策与经济直觉保持一致,我们的方法是建设性的,例如,通过明确地学习Meta-Game epsilon-Nash ePhilia的频谱打开RBC型号。
translated by 谷歌翻译