本文介绍了电力网络的问题，可以为应用多功能增强学习（Marl）创造一个令人兴奋和挑战的现实情景。脱碳的新出现趋势在配电网络上放置过大的压力。主动电压控制被视为有希望的解决方案，以减轻电力拥塞和改善电压质量，无需额外的硬件投资，利用网络中的可控装置，例如屋顶光伏（PVS）和静态VAR补偿器（SVC）。这些可控设备出现在大量广大数字中，并分布在宽的地理区域中，使Marl成为自然候选者。本文在DEC-POMDP框架中制定了主动电压控制问题，并建立了开源环境。它旨在弥合电力社区与马尔社区之间的差距，并成为马尔算法实际应用的驱动力。最后，我们分析了主动电压控制问题的特殊特征，导致最先进的Marl方法挑战，并总结了潜在的方向。

可再生能源的增加集成为电源分销网络的运行带来了许多技术挑战。其中，由可再生能源的不稳定性引起的电压波动正在受到越来越多的关注。最近在主动电压控制任务中广泛研究了电网中的多个控制单元（能够处理电源系统快速变化）中的多个控制单元。但是，基于MARL的现有方法忽略了网格的独特性质，并实现有限的性能。在本文中，我们介绍了变压器体系结构，以提取适应电力网络问题的表示形式，并提出基于变压器的多代理参与者 - 批判框架（T-MAAC）以稳定电源分配网络中的电压。此外，我们采用了针对电压控制任务量身定制的新型辅助任务训练过程，从而提高了样品效率并促进基于变压器模型的表示。我们将T-MAAC与不同的多代理 - 参与者批评算法相结合，而主动电压控制任务的一致改进证明了该方法的有效性。

本文介绍了电力系统运营商的域知识如何集成到强化学习（RL）框架中，以有效学习控制电网拓扑以防止热级联的代理。由于大搜索/优化空间，典型的基于RL的拓扑控制器无法表现良好。在这里，我们提出了一个基于演员 - 评论家的代理，以解决问题的组合性质，并使用由RTE，法国TSO开发的RL环境训练代理。为了解决大型优化空间的挑战，通过使用网络物理修改环境以增强代理学习来纳入训练过程中的基于奖励调整的基于课程的方法。此外，采用多种方案的并行训练方法来避免将代理偏置到几种情况，并使其稳健地对网格操作中的自然变异性。如果没有对培训过程进行这些修改，则RL代理失败了大多数测试场景，说明了正确整合物理系统的域知识以获得真实世界的RL学习的重要性。该代理通过RTE测试2019年学习，以运行电力网络挑战，并以精确度和第1位的速度授予第2位。开发的代码是公共使用开放的。

The energy sector is facing rapid changes in the transition towards clean renewable sources. However, the growing share of volatile, fluctuating renewable generation such as wind or solar energy has already led to an increase in power grid congestion and network security concerns. Grid operators mitigate these by modifying either generation or demand (redispatching, curtailment, flexible loads). Unfortunately, redispatching of fossil generators leads to excessive grid operation costs and higher emissions, which is in direct opposition to the decarbonization of the energy sector. In this paper, we propose an AlphaZero-based grid topology optimization agent as a non-costly, carbon-free congestion management alternative. Our experimental evaluation confirms the potential of topology optimization for power grid operation, achieves a reduction of the average amount of required redispatching by 60%, and shows the interoperability with traditional congestion management methods. Our approach also ranked 1st in the WCCI 2022 Learning to Run a Power Network (L2RPN) competition. Based on our findings, we identify and discuss open research problems as well as technical challenges for a productive system on a real power grid.

Traditional power grid systems have become obsolete under more frequent and extreme natural disasters. Reinforcement learning (RL) has been a promising solution for resilience given its successful history of power grid control. However, most power grid simulators and RL interfaces do not support simulation of power grid under large-scale blackouts or when the network is divided into sub-networks. In this study, we proposed an updated power grid simulator built on Grid2Op, an existing simulator and RL interface, and experimented on limiting the action and observation spaces of Grid2Op. By testing with DDQN and SliceRDQN algorithms, we found that reduced action spaces significantly improve training performance and efficiency. In addition, we investigated a low-rank neural network regularization method for deep Q-learning, one of the most widely used RL algorithms, in this power grid control scenario. As a result, the experiment demonstrated that in the power grid simulation environment, adopting this method will significantly increase the performance of RL agents.

Ongoing risks from climate change have impacted the livelihood of global nomadic communities, and are likely to lead to increased migratory movements in coming years. As a result, mobility considerations are becoming increasingly important in energy systems planning, particularly to achieve energy access in developing countries. Advanced Plug and Play control strategies have been recently developed with such a decentralized framework in mind, more easily allowing for the interconnection of nomadic communities, both to each other and to the main grid. In light of the above, the design and planning strategy of a mobile multi-energy supply system for a nomadic community is investigated in this work. Motivated by the scale and dimensionality of the associated uncertainties, impacting all major design and decision variables over the 30-year planning horizon, Deep Reinforcement Learning (DRL) is implemented for the design and planning problem tackled. DRL based solutions are benchmarked against several rigid baseline design options to compare expected performance under uncertainty. The results on a case study for ger communities in Mongolia suggest that mobile nomadic energy systems can be both technically and economically feasible, particularly when considering flexibility, although the degree of spatial dispersion among households is an important limiting factor. Key economic, sustainability and resilience indicators such as Cost, Equivalent Emissions and Total Unmet Load are measured, suggesting potential improvements compared to available baselines of up to 25%, 67% and 76%, respectively. Finally, the decomposition of values of flexibility and plug and play operation is presented using a variation of real options theory, with important implications for both nomadic communities and policymakers focused on enabling their energy access.

Machine learning frameworks such as Genetic Programming (GP) and Reinforcement Learning (RL) are gaining popularity in flow control. This work presents a comparative analysis of the two, bench-marking some of their most representative algorithms against global optimization techniques such as Bayesian Optimization (BO) and Lipschitz global optimization (LIPO). First, we review the general framework of the model-free control problem, bringing together all methods as black-box optimization problems. Then, we test the control algorithms on three test cases. These are (1) the stabilization of a nonlinear dynamical system featuring frequency cross-talk, (2) the wave cancellation from a Burgers' flow and (3) the drag reduction in a cylinder wake flow. We present a comprehensive comparison to illustrate their differences in exploration versus exploitation and their balance between `model capacity' in the control law definition versus `required complexity'. We believe that such a comparison paves the way toward the hybridization of the various methods, and we offer some perspective on their future development in the literature on flow control problems.

VAR-VAR控制（VVC）是通过控制电源系统中的执行器在健康状态内运行电源分配系统的问题。现有作品主要采用代表电力系统（带有树拓扑的图）作为训练深钢筋学习（RL）策略的向量的常规例程。我们提出了一个将RL与图形神经网络相结合的框架，并研究VVC设置中基于图的策略的好处和局限性。我们的结果表明，与向量表示相比，基于图的策略会渐近地收敛到相同的奖励。我们对观察和行动的影响进行进一步分析：在观察端，我们研究了基于图形的策略对功率系统中两个典型数据采集错误的鲁棒性，即传感器通信失败和测量错误。在动作端，我们表明执行器对系统有各种影响，因此使用由电源系统拓扑引起的图表表示可能不是最佳选择。最后，我们进行了一项案例研究，以证明读取功能架构和图形增强的选择可以进一步提高训练性能和鲁棒性。

Autonomous vehicles are suited for continuous area patrolling problems. However, finding an optimal patrolling strategy can be challenging for many reasons. Firstly, patrolling environments are often complex and can include unknown and evolving environmental factors. Secondly, autonomous vehicles can have failures or hardware constraints such as limited battery lives. Importantly, patrolling large areas often requires multiple agents that need to collectively coordinate their actions. In this work, we consider these limitations and propose an approach based on a distributed, model-free deep reinforcement learning based multi-agent patrolling strategy. In this approach, agents make decisions locally based on their own environmental observations and on shared information. In addition, agents are trained to automatically recharge themselves when required to support continuous collective patrolling. A homogeneous multi-agent architecture is proposed, where all patrolling agents have an identical policy. This architecture provides a robust patrolling system that can tolerate agent failures and allow supplementary agents to be added to replace failed agents or to increase the overall patrol performance. This performance is validated through experiments from multiple perspectives, including the overall patrol performance, the efficiency of the battery recharging strategy, the overall robustness of the system, and the agents' ability to adapt to environment dynamics.

Power grids, across the world, play an important societal and economical role by providing uninterrupted, reliable and transient-free power to several industries, businesses and household consumers. With the advent of renewable power resources and EVs resulting into uncertain generation and highly dynamic load demands, it has become ever so important to ensure robust operation of power networks through suitable management of transient stability issues and localize the events of blackouts. In the light of ever increasing stress on the modern grid infrastructure and the grid operators, this paper presents a reinforcement learning (RL) framework, PowRL, to mitigate the effects of unexpected network events, as well as reliably maintain electricity everywhere on the network at all times. The PowRL leverages a novel heuristic for overload management, along with the RL-guided decision making on optimal topology selection to ensure that the grid is operated safely and reliably (with no overloads). PowRL is benchmarked on a variety of competition datasets hosted by the L2RPN (Learning to Run a Power Network). Even with its reduced action space, PowRL tops the leaderboard in the L2RPN NeurIPS 2020 challenge (Robustness track) at an aggregate level, while also being the top performing agent in the L2RPN WCCI 2020 challenge. Moreover, detailed analysis depicts state-of-the-art performances by the PowRL agent in some of the test scenarios.

近年来，多智能体加固学习（Marl）在各种应用中呈现出令人印象深刻的性能。但是，物理限制，预算限制以及许多其他因素通常会在多代理系统（MAS）上施加\ Texit {约束}，这不能由传统的Marl框架处理。具体而言，本文重点介绍受约束的Mase，其中代理工作\纺织{合作}在各种限制下最大化预期的团队平均成本下的预期团队平均返回，并开发一个名为DECOM的\ TEXTIT {约束合作MARL}框架，名为DECOM这样的苗条。特别是，DECOM将每个代理人的策略分解为两个模块，这使得代理商之间的信息共享，以实现更好的合作。此外，通过这种模块化，DREM的训练算法将原始约束优化分为奖励的无约束优化和成本的约束满足问题。然后，Decom以计算有效的方式迭代地解决这些问题，这使得DECOM高度可扩展。我们还提供了对Decom策略更新算法的融合的理论保障。最后，我们在玩具和大规模（有500个代理）环境中使用各种类型的成本验证了DECOM的有效性。

Value factorisation is a useful technique for multi-agent reinforcement learning (MARL) in global reward game, however its underlying mechanism is not yet fully understood. This paper studies a theoretical framework for value factorisation with interpretability via Shapley value theory. We generalise Shapley value to Markov convex game called Markov Shapley value (MSV) and apply it as a value factorisation method in global reward game, which is obtained by the equivalence between the two games. Based on the properties of MSV, we derive Shapley-Bellman optimality equation (SBOE) to evaluate the optimal MSV, which corresponds to an optimal joint deterministic policy. Furthermore, we propose Shapley-Bellman operator (SBO) that is proved to solve SBOE. With a stochastic approximation and some transformations, a new MARL algorithm called Shapley Q-learning (SHAQ) is established, the implementation of which is guided by the theoretical results of SBO and MSV. We also discuss the relationship between SHAQ and relevant value factorisation methods. In the experiments, SHAQ exhibits not only superior performances on all tasks but also the interpretability that agrees with the theoretical analysis. The implementation of this paper is on https://github.com/hsvgbkhgbv/shapley-q-learning.

在过去的十年中，多智能经纪人强化学习（Marl）已经有了重大进展，但仍存在许多挑战，例如高样本复杂性和慢趋同稳定的政策，在广泛的部署之前需要克服，这是可能的。然而，在实践中，许多现实世界的环境已经部署了用于生成策略的次优或启发式方法。一个有趣的问题是如何最好地使用这些方法作为顾问，以帮助改善多代理领域的加强学习。在本文中，我们提供了一个原则的框架，用于将动作建议纳入多代理设置中的在线次优顾问。我们描述了在非传记通用随机游戏环境中提供多种智能强化代理（海军上将）的问题，并提出了两种新的基于Q学习的算法：海军上将决策（海军DM）和海军上将 - 顾问评估（Admiral-AE） ，这使我们能够通过适当地纳入顾问（Admiral-DM）的建议来改善学习，并评估顾问（Admiral-AE）的有效性。我们从理论上分析了算法，并在一般加上随机游戏中提供了关于他们学习的定点保证。此外，广泛的实验说明了这些算法：可以在各种环境中使用，具有对其他相关基线的有利相比的性能，可以扩展到大状态行动空间，并且对来自顾问的不良建议具有稳健性。

Technology advancements in wireless communications and high-performance Extended Reality (XR) have empowered the developments of the Metaverse. The demand for Metaverse applications and hence, real-time digital twinning of real-world scenes is increasing. Nevertheless, the replication of 2D physical world images into 3D virtual world scenes is computationally intensive and requires computation offloading. The disparity in transmitted scene dimension (2D as opposed to 3D) leads to asymmetric data sizes in uplink (UL) and downlink (DL). To ensure the reliability and low latency of the system, we consider an asynchronous joint UL-DL scenario where in the UL stage, the smaller data size of the physical world scenes captured by multiple extended reality users (XUs) will be uploaded to the Metaverse Console (MC) to be construed and rendered. In the DL stage, the larger-size 3D virtual world scenes need to be transmitted back to the XUs. The decisions pertaining to computation offloading and channel assignment are optimized in the UL stage, and the MC will optimize power allocation for users assigned with a channel in the UL transmission stage. Some problems arise therefrom: (i) interactive multi-process chain, specifically Asynchronous Markov Decision Process (AMDP), (ii) joint optimization in multiple processes, and (iii) high-dimensional objective functions, or hybrid reward scenarios. To ensure the reliability and low latency of the system, we design a novel multi-agent reinforcement learning algorithm structure, namely Asynchronous Actors Hybrid Critic (AAHC). Extensive experiments demonstrate that compared to proposed baselines, AAHC obtains better solutions with preferable training time.

This paper is a technical overview of DeepMind and Google's recent work on reinforcement learning for controlling commercial cooling systems. Building on expertise that began with cooling Google's data centers more efficiently, we recently conducted live experiments on two real-world facilities in partnership with Trane Technologies, a building management system provider. These live experiments had a variety of challenges in areas such as evaluation, learning from offline data, and constraint satisfaction. Our paper describes these challenges in the hope that awareness of them will benefit future applied RL work. We also describe the way we adapted our RL system to deal with these challenges, resulting in energy savings of approximately 9% and 13% respectively at the two live experiment sites.

多代理深度增强学习（Marl）缺乏缺乏共同使用的评估任务和标准，使方法之间的比较困难。在这项工作中，我们提供了一个系统评估，并比较了三种不同类别的Marl算法（独立学习，集中式多代理政策梯度，价值分解）在各种协作多智能经纪人学习任务中。我们的实验是在不同学习任务中作为算法的预期性能的参考，我们为不同学习方法的有效性提供了见解。我们开源EPYMARL，它将Pymarl CodeBase扩展到包括其他算法，并允许灵活地配置算法实现细节，例如参数共享。最后，我们开源两种环境，用于多智能经纪研究，重点关注稀疏奖励下的协调。

A long-standing challenge in artificial intelligence is lifelong learning. In lifelong learning, many tasks are presented in sequence and learners must efficiently transfer knowledge between tasks while avoiding catastrophic forgetting over long lifetimes. On these problems, policy reuse and other multi-policy reinforcement learning techniques can learn many tasks. However, they can generate many temporary or permanent policies, resulting in memory issues. Consequently, there is a need for lifetime-scalable methods that continually refine a policy library of a pre-defined size. This paper presents a first approach to lifetime-scalable policy reuse. To pre-select the number of policies, a notion of task capacity, the maximal number of tasks that a policy can accurately solve, is proposed. To evaluate lifetime policy reuse using this method, two state-of-the-art single-actor base-learners are compared: 1) a value-based reinforcement learner, Deep Q-Network (DQN) or Deep Recurrent Q-Network (DRQN); and 2) an actor-critic reinforcement learner, Proximal Policy Optimisation (PPO) with or without Long Short-Term Memory layer. By selecting the number of policies based on task capacity, D(R)QN achieves near-optimal performance with 6 policies in a 27-task MDP domain and 9 policies in an 18-task POMDP domain; with fewer policies, catastrophic forgetting and negative transfer are observed. Due to slow, monotonic improvement, PPO requires fewer policies, 1 policy for the 27-task domain and 4 policies for the 18-task domain, but it learns the tasks with lower accuracy than D(R)QN. These findings validate lifetime-scalable policy reuse and suggest using D(R)QN for larger and PPO for smaller library sizes.

值得信赖的强化学习算法应有能力解决挑战性的现实问题，包括{Robustly}处理不确定性，满足{安全}的限制以避免灾难性的失败，以及在部署过程中{prencepentiming}以避免灾难性的失败}。这项研究旨在概述这些可信赖的强化学习的主要观点，即考虑其在鲁棒性，安全性和概括性上的内在脆弱性。特别是，我们给出严格的表述，对相应的方法进行分类，并讨论每个观点的基准。此外，我们提供了一个前景部分，以刺激有希望的未来方向，并简要讨论考虑人类反馈的外部漏洞。我们希望这项调查可以在统一的框架中将单独的研究汇合在一起，并促进强化学习的可信度。

我们考虑单个强化学习与基于事件驱动的代理商金融市场模型相互作用时学习最佳执行代理的学习动力。交易在事件时间内通过匹配引擎进行异步进行。最佳执行代理在不同级别的初始订单尺寸和不同尺寸的状态空间上进行考虑。使用校准方法考虑了对基于代理的模型和市场的影响，该方法探讨了经验性风格化事实和价格影响曲线的变化。收敛，音量轨迹和动作痕迹图用于可视化学习动力学。这表明了最佳执行代理如何在模拟的反应性市场框架内学习最佳交易决策，以及如何通过引入战略订单分类来改变模拟市场的反反应。

我们提出了一个混合工业冷却系统模型，该模型将分析解决方案嵌入多物理模拟中。该模型设计用于增强学习（RL）应用程序，并平衡简单性与模拟保真度和解释性。该模型的忠诚度根据大规模冷却系统的现实世界数据进行了评估。接下来是一个案例研究，说明如何将模型用于RL研究。为此，我们开发了一个工业任务套件，该套件允许指定不同的问题设置和复杂性水平，并使用它来评估不同RL算法的性能。