分布式能源的广泛采用以及智能电网技术的出现使传统上被动的电力系统用户得以积极参与能源交易。认识到传统的集中式网格驱动能源市场为这些用户提供最低盈利能力的事实，最近的研究已将重点转移到分散的点对点（P2P）能源市场上。在这些市场中，用户彼此交易能源，比买卖网格的收益更高。但是，假设持续的可用性，参与和完全合规性，大多数P2P能源交易中的研究很大程度上忽略了交易过程中用户的看法。结果，这些方法可能会导致负面态度和随着时间的推移参与度的减少。在本文中，我们设计了一个自动化的P2P能源市场，该市场将用户感知考虑在内。我们采用前景理论来对用户的看法进行建模并制定优化框架，以最大程度地提高买方的感知，同时匹配需求和生产。鉴于优化问题的非线性和非凸性性质，我们提出了基于差分进化的算法，用于交易能源，称为辩论。此外，我们推出了一种具有风险敏感的Q学习算法，该算法名为“ Q-学习和风险敏感性”（PQR），该算法（PQR）得知考虑其感知到的实用程序的卖家的最佳价格。基于真正的能耗和生产的真实痕迹以及现实的前景理论的功能，表明我们的方法可为买家带来26％的感知价值，并为卖方产生7％的奖励，与最新的最新状态相比。

由于数据量增加，金融业的快速变化已经彻底改变了数据处理和数据分析的技术，并带来了新的理论和计算挑战。与古典随机控制理论和解决财务决策问题的其他分析方法相比，解决模型假设的财务决策问题，强化学习（RL）的新发展能够充分利用具有更少模型假设的大量财务数据并改善复杂的金融环境中的决策。该调查纸目的旨在审查最近的资金途径的发展和使用RL方法。我们介绍了马尔可夫决策过程，这是许多常用的RL方法的设置。然后引入各种算法，重点介绍不需要任何模型假设的基于价值和基于策略的方法。连接是用神经网络进行的，以扩展框架以包含深的RL算法。我们的调查通过讨论了这些RL算法在金融中各种决策问题中的应用，包括最佳执行，投资组合优化，期权定价和对冲，市场制作，智能订单路由和Robo-Awaring。

As an efficient way to integrate multiple distributed energy resources and the user side, a microgrid is mainly faced with the problems of small-scale volatility, uncertainty, intermittency and demand-side uncertainty of DERs. The traditional microgrid has a single form and cannot meet the flexible energy dispatch between the complex demand side and the microgrid. In response to this problem, the overall environment of wind power, thermostatically controlled loads, energy storage systems, price-responsive loads and the main grid is proposed. Secondly, the centralized control of the microgrid operation is convenient for the control of the reactive power and voltage of the distributed power supply and the adjustment of the grid frequency. However, there is a problem in that the flexible loads aggregate and generate peaks during the electricity price valley. The existing research takes into account the power constraints of the microgrid and fails to ensure a sufficient supply of electric energy for a single flexible load. This paper considers the response priority of each unit component of TCLs and ESSs on the basis of the overall environment operation of the microgrid so as to ensure the power supply of the flexible load of the microgrid and save the power input cost to the greatest extent. Finally, the simulation optimization of the environment can be expressed as a Markov decision process process. It combines two stages of offline and online operations in the training process. The addition of multiple threads with the lack of historical data learning leads to low learning efficiency. The asynchronous advantage actor-critic with the experience replay pool memory library is added to solve the data correlation and nonstatic distribution problems during training.

联邦学习（FL）变得流行，并在训练大型机器学习（ML）模型的情况下表现出很大的潜力，而不会使所有者的原始数据曝光。在FL中，数据所有者可以根据其本地数据培训ML模型，并且仅将模型更新发送到模型更新，而不是原始数据到模型所有者进行聚合。为了提高模型准确性和培训完成时间的学习绩效，招募足够的参与者至关重要。同时，数据所有者是理性的，可能不愿意由于资源消耗而参与协作学习过程。为了解决这些问题，最近有各种作品旨在激励数据业主贡献其资源。在本文中，我们为文献中提出的经济和游戏理论方法提供了全面的审查，以设计刺激数据业主参加流程培训过程的各种计划。特别是，我们首先在激励机制设计中常用的佛罗里达州的基础和背景，经济理论。然后，我们审查博弈理论和经济方法应用于FL的激励机制的应用。最后，我们突出了一些开放的问题和未来关于FL激励机制设计的研究方向。

This paper presents a multi-agent Deep Reinforcement Learning (DRL) framework for autonomous control and integration of renewable energy resources into smart power grid systems. In particular, the proposed framework jointly considers demand response (DR) and distributed energy management (DEM) for residential end-users. DR has a widely recognized potential for improving power grid stability and reliability, while at the same time reducing end-users energy bills. However, the conventional DR techniques come with several shortcomings, such as the inability to handle operational uncertainties while incurring end-user disutility, which prevents widespread adoption in real-world applications. The proposed framework addresses these shortcomings by implementing DR and DEM based on real-time pricing strategy that is achieved using deep reinforcement learning. Furthermore, this framework enables the power grid service provider to leverage distributed energy resources (i.e., PV rooftop panels and battery storage) as dispatchable assets to support the smart grid during peak hours, thus achieving management of distributed energy resources. Simulation results based on the Deep Q-Network (DQN) demonstrate significant improvements of the 24-hour accumulative profit for both prosumers and the power grid service provider, as well as major reductions in the utilization of the power grid reserve generators.

Driven by the global decarbonization effort, the rapid integration of renewable energy into the conventional electricity grid presents new challenges and opportunities for the battery energy storage system (BESS) participating in the energy market. Energy arbitrage can be a significant source of revenue for the BESS due to the increasing price volatility in the spot market caused by the mismatch between renewable generation and electricity demand. In addition, the Frequency Control Ancillary Services (FCAS) markets established to stabilize the grid can offer higher returns for the BESS due to their capability to respond within milliseconds. Therefore, it is crucial for the BESS to carefully decide how much capacity to assign to each market to maximize the total profit under uncertain market conditions. This paper formulates the bidding problem of the BESS as a Markov Decision Process, which enables the BESS to participate in both the spot market and the FCAS market to maximize profit. Then, Proximal Policy Optimization, a model-free deep reinforcement learning algorithm, is employed to learn the optimal bidding strategy from the dynamic environment of the energy market under a continuous bidding scale. The proposed model is trained and validated using real-world historical data of the Australian National Electricity Market. The results demonstrate that our developed joint bidding strategy in both markets is significantly profitable compared to individual markets.

本文设计了一个由三种类型的代理人：个人，保险公司和政府的微型社会的连续重复游戏。对经济学文献的新生，我们使用加强学习（RL），与多军匪徒问题密切相关，以学习每花费1美元的拟议政策干预措施的福利影响。该论文严格讨论了提议的干预措施的可取性，通过逐案将它们相互比较。本文为校准的理论模型提供了算法政策评估的框架，该模型可以帮助可行性研究。

Ongoing risks from climate change have impacted the livelihood of global nomadic communities, and are likely to lead to increased migratory movements in coming years. As a result, mobility considerations are becoming increasingly important in energy systems planning, particularly to achieve energy access in developing countries. Advanced Plug and Play control strategies have been recently developed with such a decentralized framework in mind, more easily allowing for the interconnection of nomadic communities, both to each other and to the main grid. In light of the above, the design and planning strategy of a mobile multi-energy supply system for a nomadic community is investigated in this work. Motivated by the scale and dimensionality of the associated uncertainties, impacting all major design and decision variables over the 30-year planning horizon, Deep Reinforcement Learning (DRL) is implemented for the design and planning problem tackled. DRL based solutions are benchmarked against several rigid baseline design options to compare expected performance under uncertainty. The results on a case study for ger communities in Mongolia suggest that mobile nomadic energy systems can be both technically and economically feasible, particularly when considering flexibility, although the degree of spatial dispersion among households is an important limiting factor. Key economic, sustainability and resilience indicators such as Cost, Equivalent Emissions and Total Unmet Load are measured, suggesting potential improvements compared to available baselines of up to 25%, 67% and 76%, respectively. Finally, the decomposition of values of flexibility and plug and play operation is presented using a variation of real options theory, with important implications for both nomadic communities and policymakers focused on enabling their energy access.

我们考虑了需求侧能源管理的问题，每个家庭都配备了能够在线安排家用电器的智能电表。目的是最大程度地减少实时定价计划下的整体成本。尽管以前的作品引入了集中式方法，在该方法中，调度算法具有完全可观察的性能，但我们提出了将智能网格环境作为马尔可夫游戏的表述。每个家庭都是具有部分可观察性的去中心化代理，可以在现实环境中进行可扩展性和隐私保护。电网操作员产生的价格信号随能量需求而变化。我们提出了从代理商的角度来解决部分可观察性和环境的局部可观察性的扩展，以解决部分可观察性。该算法学习了一位集中批评者，该批评者协调分散的代理商的培训。因此，我们的方法使用集中学习，但分散执行。仿真结果表明，我们的在线深入强化学习方法可以纯粹基于瞬时观察和价格信号来降低所有消耗的总能量的峰值与平均值和所有家庭的电力。

为了通过使用可再生能源来取代化石燃料，间歇性风能和光伏（PV）功率的资源不平衡是点对点（P2P）功率交易的关键问题。为了解决这个问题，本文介绍了增强学习（RL）技术。对于RL，图形卷积网络（GCN）和双向长期记忆（BI-LSTM）网络由基于合作游戏理论的纳米簇之间的P2P功率交易共同应用于P2P功率交易。柔性且可靠的DC纳米醇适合整合可再生能源以进行分配系统。每个局部纳米粒子群都采用了生产者的位置，同时着重于功率生产和消费。对于纳米级簇的电源管理，使用物联网（IoT）技术将多目标优化应用于每个本地纳米群集群。考虑到风和光伏发电的间歇性特征，进行电动汽车（EV）的充电/排放。 RL算法，例如深Q学习网络（DQN），深度复发Q学习网络（DRQN），BI-DRQN，近端策略优化（PPO），GCN-DQN，GCN-DQN，GCN-DRQN，GCN-DRQN，GCN-BI-DRQN和GCN-PPO用于模拟。因此，合作P2P电力交易系统利用使用时间（TOU）基于关税的电力成本和系统边际价格（SMP）最大化利润，并最大程度地减少电网功耗的量。用P2P电源交易的纳米簇簇的电源管理实时模拟了分配测试馈线，并提议的GCN-PPO技术将纳米糖簇的电量降低了36.7％。

Technology advancements in wireless communications and high-performance Extended Reality (XR) have empowered the developments of the Metaverse. The demand for Metaverse applications and hence, real-time digital twinning of real-world scenes is increasing. Nevertheless, the replication of 2D physical world images into 3D virtual world scenes is computationally intensive and requires computation offloading. The disparity in transmitted scene dimension (2D as opposed to 3D) leads to asymmetric data sizes in uplink (UL) and downlink (DL). To ensure the reliability and low latency of the system, we consider an asynchronous joint UL-DL scenario where in the UL stage, the smaller data size of the physical world scenes captured by multiple extended reality users (XUs) will be uploaded to the Metaverse Console (MC) to be construed and rendered. In the DL stage, the larger-size 3D virtual world scenes need to be transmitted back to the XUs. The decisions pertaining to computation offloading and channel assignment are optimized in the UL stage, and the MC will optimize power allocation for users assigned with a channel in the UL transmission stage. Some problems arise therefrom: (i) interactive multi-process chain, specifically Asynchronous Markov Decision Process (AMDP), (ii) joint optimization in multiple processes, and (iii) high-dimensional objective functions, or hybrid reward scenarios. To ensure the reliability and low latency of the system, we design a novel multi-agent reinforcement learning algorithm structure, namely Asynchronous Actors Hybrid Critic (AAHC). Extensive experiments demonstrate that compared to proposed baselines, AAHC obtains better solutions with preferable training time.

The open-radio access network (O-RAN) embraces cloudification and network function virtualization for base-band function processing by dis-aggregated radio units (RUs), distributed units (DUs), and centralized units (CUs). These enable the cloud-RAN vision in full, where multiple mobile network operators (MNOs) can install their proprietary or open RUs, but lease on-demand computational resources for DU-CU functions from commonly available open-clouds via open x-haul interfaces. In this paper, we propose and compare the performances of min-max fairness and Vickrey-Clarke-Groves (VCG) auction-based x-haul and DU-CU resource allocation mechanisms to create a multi-tenant O-RAN ecosystem that is sustainable for small, medium, and large MNOs. The min-max fair approach minimizes the maximum OPEX of RUs through cost-sharing proportional to their demands, whereas the VCG auction-based approach minimizes the total OPEX for all resources utilized while extracting truthful demands from RUs. We consider time-wavelength division multiplexed (TWDM) passive optical network (PON)-based x-haul interfaces where PON virtualization technique is used to flexibly provide optical connections among RUs and edge-clouds at macro-cell RU locations as well as open-clouds at the central office locations. Moreover, we design efficient heuristics that yield significantly better economic efficiency and network resource utilization than conventional greedy resource allocation algorithms and reinforcement learning-based algorithms.

智能能源网络提供了一种有效的手段，可容纳可变可再生能源（例如太阳能和风能）的高渗透率，这是能源生产深度脱碳的关键。但是，鉴于可再生能源以及能源需求的可变性，必须制定有效的控制和能源存储方案来管理可变的能源产生并实现所需的系统经济学和环境目标。在本文中，我们引入了由电池和氢能存储组成的混合储能系统，以处理与电价，可再生能源生产和消费有关的不确定性。我们旨在提高可再生能源利用率，并最大程度地减少能源成本和碳排放，同时确保网络内的能源可靠性和稳定性。为了实现这一目标，我们提出了一种多代理的深层确定性政策梯度方法，这是一种基于强化的基于强化学习的控制策略，可实时优化混合能源存储系统和能源需求的调度。提出的方法是无模型的，不需要明确的知识和智能能源网络环境的严格数学模型。基于现实世界数据的仿真结果表明：（i）混合储能系统和能源需求的集成和优化操作可将碳排放量减少78.69％，将成本节省的成本储蓄提高23.5％，可续订的能源利用率比13.2％以上。其他基线模型和（ii）所提出的算法优于最先进的自学习算法，例如Deep-Q网络。

电动汽车快速采用（EVS）要求广泛安装EV充电站。为了最大限度地提高充电站的盈利能力，提供充电和电网服务的智能控制器实际上很需要。然而，由于不确定的到达时间和EVS的充电需求，确定最佳充电时间表具有挑战性。在本文中，我们提出了一种新的集中分配和分散执行（CADE）强化学习（RL）框架，以最大限度地提高收费站的利润。在集中分配过程中，EVS被分配给等待或充电点。在分散的执行过程中，每个充电器都在学习来自共享重放内存的动作值函数的同时使其自己的充电/放电决定。该CADE框架显着提高了RL算法的可扩展性和采样效率。数值结果表明，所提出的CADE框架既有计算高效且可扩展，显着优于基线模型预测控制（MPC）。我们还提供了对学习的动作值的深入分析，以解释加强学习代理的内部工作。

Decision-making problems are commonly formulated as optimization problems, which are then solved to make optimal decisions. In this work, we consider the inverse problem where we use prior decision data to uncover the underlying decision-making process in the form of a mathematical optimization model. This statistical learning problem is referred to as data-driven inverse optimization. We focus on problems where the underlying decision-making process is modeled as a convex optimization problem whose parameters are unknown. We formulate the inverse optimization problem as a bilevel program and propose an efficient block coordinate descent-based algorithm to solve large problem instances. Numerical experiments on synthetic datasets demonstrate the computational advantage of our method compared to standard commercial solvers. Moreover, the real-world utility of the proposed approach is highlighted through two realistic case studies in which we consider estimating risk preferences and learning local constraint parameters of agents in a multiplayer Nash bargaining game.

Recent technological advancements in space, air and ground components have made possible a new network paradigm called "space-air-ground integrated network" (SAGIN). Unmanned aerial vehicles (UAVs) play a key role in SAGINs. However, due to UAVs' high dynamics and complexity, the real-world deployment of a SAGIN becomes a major barrier for realizing such SAGINs. Compared to the space and terrestrial components, UAVs are expected to meet performance requirements with high flexibility and dynamics using limited resources. Therefore, employing UAVs in various usage scenarios requires well-designed planning in algorithmic approaches. In this paper, we provide a comprehensive review of recent learning-based algorithmic approaches. We consider possible reward functions and discuss the state-of-the-art algorithms for optimizing the reward functions, including Q-learning, deep Q-learning, multi-armed bandit (MAB), particle swarm optimization (PSO) and satisfaction-based learning algorithms. Unlike other survey papers, we focus on the methodological perspective of the optimization problem, which can be applicable to various UAV-assisted missions on a SAGIN using these algorithms. We simulate users and environments according to real-world scenarios and compare the learning-based and PSO-based methods in terms of throughput, load, fairness, computation time, etc. We also implement and evaluate the 2-dimensional (2D) and 3-dimensional (3D) variations of these algorithms to reflect different deployment cases. Our simulation suggests that the $3$D satisfaction-based learning algorithm outperforms the other approaches for various metrics in most cases. We discuss some open challenges at the end and our findings aim to provide design guidelines for algorithm selections while optimizing the deployment of UAV-assisted SAGINs.

在本文中，我们介绍了有关典型乘车共享系统中决策优化问题的强化学习方法的全面，深入的调查。涵盖了有关乘车匹配，车辆重新定位，乘车，路由和动态定价主题的论文。在过去的几年中，大多数文献都出现了，并且要继续解决一些核心挑战：模型复杂性，代理协调和多个杠杆的联合优化。因此，我们还引入了流行的数据集和开放式仿真环境，以促进进一步的研发。随后，我们讨论了有关该重要领域的强化学习研究的许多挑战和机会。

增强学习（RL）是多能管理系统的有前途的最佳控制技术。它不需要先验模型 - 降低了前期和正在进行的项目特定工程工作，并且能够学习基础系统动力学的更好表示。但是，香草RL不能提供约束满意度的保证 - 导致其在安全至关重要的环境中产生各种不安全的互动。在本文中，我们介绍了两种新颖的安全RL方法，即SafeFallback和Afvafe，其中安全约束配方与RL配方脱钩，并且提供了硬构成满意度，可以保证在培训（探索）和开发过程中（近距离） ）最佳政策。在模拟的多能系统案例研究中，我们已经表明，这两种方法均与香草RL基准相比（94,6％和82,8％，而35.5％）和香草RL基准相比明显更高的效用（即有用的政策）开始。提出的SafeFallback方法甚至可以胜过香草RL基准（102,9％至100％）。我们得出的结论是，这两种方法都是超越RL的安全限制处理技术，正如随机代理所证明的，同时仍提供坚硬的保证。最后，我们向I.A.提出了基本的未来工作。随着更多数据可用，改善约束功能本身。

传感器节点（SNS）的部署总是在无线传感器网络（WSN）的系统性能中起决定性作用。在这项工作中，我们提出了一种实用异构WSN的最佳部署方法，该方法可以深入了解可靠性和部署成本之间的权衡。具体而言，这项工作旨在提供SNS的最佳部署，以最大程度地提高覆盖率和连接学位，同时最大程度地减少整体部署成本。此外，这项工作充分考虑了SNS的异质性（即差异化的传感范围和部署成本）和三维（3-D）部署方案。这是一个多目标优化问题，非凸，多模态和NP-HARD。为了解决它，我们开发了一种新型的基于群体的多目标优化算法，称为竞争性多目标海洋掠食者算法（CMOMPA），其性能通过与十种其他多个多目标优化的全面比较实验验证算法。计算结果表明，在收敛性和准确性方面，CMOMPA优于他人，并且在多模式多目标优化问题上表现出卓越的性能。还进行了足够的模拟来评估基于CMOMPA的最佳SNS部署方法的有效性。结果表明，优化的部署可以平衡部署成本，感知可靠性和网络可靠性之间的权衡平衡。源代码可在https://github.com/inet-wzu/cmompa上找到。

Energy consumption in buildings, both residential and commercial, accounts for approximately 40% of all energy usage in the U.S., and similar numbers are being reported from countries around the world. This significant amount of energy is used to maintain a comfortable, secure, and productive environment for the occupants. So, it is crucial that the energy consumption in buildings must be optimized, all the while maintaining satisfactory levels of occupant comfort, health, and safety. Recently, Machine Learning has been proven to be an invaluable tool in deriving important insights from data and optimizing various systems. In this work, we review the ways in which machine learning has been leveraged to make buildings smart and energy-efficient. For the convenience of readers, we provide a brief introduction of several machine learning paradigms and the components and functioning of each smart building system we cover. Finally, we discuss challenges faced while implementing machine learning algorithms in smart buildings and provide future avenues for research at the intersection of smart buildings and machine learning.