合作的感知在将车辆的感知范围扩展到超出其视线之外至关重要。然而，在有限的通信资源下交换原始感官数据是不可行的。为了实现有效的合作感知，车辆需要解决以下基本问题：需要共享哪些感官数据？，在哪个分辨率？，以及哪个车辆？为了回答这个问题，在本文中，提出了一种新颖的框架来允许加强学习（RL）基于车辆关联，资源块（RB）分配和通过利用基于四叉的点的协作感知消息（CPM）的内容选择云压缩机制。此外，引入了联合的RL方法，以便在跨车辆上加速训练过程。仿真结果表明，RL代理能够有效地学习车辆关联，RB分配和消息内容选择，同时在接收的感官信息方面最大化车辆的满足。结果还表明，与非联邦方法相比，联邦RL改善了培训过程，可以在与非联邦方法相同的时间内实现更好的政策。

未来的互联网涉及几种新兴技术，例如5G和5G网络，车辆网络，无人机（UAV）网络和物联网（IOT）。此外，未来的互联网变得异质并分散了许多相关网络实体。每个实体可能需要做出本地决定，以在动态和不确定的网络环境下改善网络性能。最近使用标准学习算法，例如单药强化学习（RL）或深入强化学习（DRL），以使每个网络实体作为代理人通过与未知环境进行互动来自适应地学习最佳决策策略。但是，这种算法未能对网络实体之间的合作或竞争进行建模，而只是将其他实体视为可能导致非平稳性问题的环境的一部分。多机构增强学习（MARL）允许每个网络实体不仅观察环境，还可以观察其他实体的政策来学习其最佳政策。结果，MAL可以显着提高网络实体的学习效率，并且最近已用于解决新兴网络中的各种问题。在本文中，我们因此回顾了MAL在新兴网络中的应用。特别是，我们提供了MARL的教程，以及对MARL在下一代互联网中的应用进行全面调查。特别是，我们首先介绍单代机Agent RL和MARL。然后，我们回顾了MAL在未来互联网中解决新兴问题的许多应用程序。这些问题包括网络访问，传输电源控制，计算卸载，内容缓存，数据包路由，无人机网络的轨迹设计以及网络安全问题。

In heterogeneous networks (HetNets), the overlap of small cells and the macro cell causes severe cross-tier interference. Although there exist some approaches to address this problem, they usually require global channel state information, which is hard to obtain in practice, and get the sub-optimal power allocation policy with high computational complexity. To overcome these limitations, we propose a multi-agent deep reinforcement learning (MADRL) based power control scheme for the HetNet, where each access point makes power control decisions independently based on local information. To promote cooperation among agents, we develop a penalty-based Q learning (PQL) algorithm for MADRL systems. By introducing regularization terms in the loss function, each agent tends to choose an experienced action with high reward when revisiting a state, and thus the policy updating speed slows down. In this way, an agent's policy can be learned by other agents more easily, resulting in a more efficient collaboration process. We then implement the proposed PQL in the considered HetNet and compare it with other distributed-training-and-execution (DTE) algorithms. Simulation results show that our proposed PQL can learn the desired power control policy from a dynamic environment where the locations of users change episodically and outperform existing DTE MADRL algorithms.

预计下一代（NEVERG）网络将支持苛刻的触觉互联网应用，例如增强现实和连接的自动车辆。虽然最近的创新带来了更大的联系能力的承诺，它们对环境的敏感性以及不稳定的性能无视基于传统的基于模型的控制理由。零触摸数据驱动的方法可以提高网络适应当前操作条件的能力。诸如强化学习（RL）算法等工具可以仅基于观察历史来构建最佳控制策略。具体而言，使用深神经网络（DNN）作为预测器的深RL（DRL）已经被示出，即使在复杂的环境和高维输入中也能够实现良好的性能。但是，DRL模型的培训需要大量数据，这可能会限制其对潜在环境的不断发展统计数据的适应性。此外，无线网络是固有的分布式系统，其中集中式DRL方法需要过多的数据交换，而完全分布的方法可能导致较慢的收敛速率和性能下降。在本文中，为了解决这些挑战，我们向DRL提出了联合学习（FL）方法，我们指的是联邦DRL（F-DRL），其中基站（BS）通过仅共享模型的重量协作培训嵌入式DNN而不是训练数据。我们评估了两个不同版本的F-DRL，价值和策略，并显示出与分布式和集中式DRL相比实现的卓越性能。

自动驾驶汽车（AV）必须在动态环境中安全有效地操作。为此，配备联合雷达通信（JRC）功能的AVS可以通过使用雷达检测和数据通信功能来增强驾驶安全性。但是，在不确定性和周围环境的动态下，通过两种不同功能优化AV系统的性能非常具有挑战性。在这项工作中，我们首先提出一个基于马尔可夫决策过程（MDP）的智能优化框架，以帮助AV在周围环境的动态和不确定性下选择JRC操作功能时做出最佳决策。然后，我们开发了一种有效的学习算法，利用了深度强化学习技术的最新进展，以找到AV的最佳政策，而无需任何有关周围环境的先前信息。此外，为了使我们提出的框架更加可扩展，我们开发了一种转移学习（TL）机制，该机制使AV能够利用有价值的体验来加速培训过程，以加速培训过程。广泛的模拟表明，与其他常规的深钢筋学习方法相比，提议的可转移深钢筋学习框架可将AV的障碍检测概率降低到67％。

车辆到车辆（V2V）通信的性能在很大程度上取决于使用的调度方法。虽然集中式网络调度程序提供高V2V通信可靠性，但它们的操作通常仅限于具有完整的蜂窝网络覆盖范围的区域。相比之下，在细胞外覆盖区域中，使用了相对效率低下的分布式无线电资源管理。为了利用集中式方法的好处来增强V2V通信在缺乏蜂窝覆盖的道路上的可靠性，我们建议使用VRLS（车辆加固学习调度程序），这是一种集中的调度程序，该调度程序主动为覆盖外的V2V Communications主动分配资源，以前}车辆离开蜂窝网络覆盖范围。通过在模拟的车辆环境中进行培训，VRL可以学习一项适应环境变化的调度策略，从而消除了在复杂的现实生活环境中对有针对性（重新）培训的需求。我们评估了在不同的移动性，网络负载，无线通道和资源配置下VRL的性能。 VRL的表现优于最新的区域中最新分布式调度算法，而无需蜂窝网络覆盖，通过在高负载条件下将数据包错误率降低了一半，并在低负载方案中实现了接近最大的可靠性。

With the development of deep representation learning, the domain of reinforcement learning (RL) has become a powerful learning framework now capable of learning complex policies in high dimensional environments. This review summarises deep reinforcement learning (DRL) algorithms and provides a taxonomy of automated driving tasks where (D)RL methods have been employed, while addressing key computational challenges in real world deployment of autonomous driving agents. It also delineates adjacent domains such as behavior cloning, imitation learning, inverse reinforcement learning that are related but are not classical RL algorithms. The role of simulators in training agents, methods to validate, test and robustify existing solutions in RL are discussed.

Recent technological advancements in space, air and ground components have made possible a new network paradigm called "space-air-ground integrated network" (SAGIN). Unmanned aerial vehicles (UAVs) play a key role in SAGINs. However, due to UAVs' high dynamics and complexity, the real-world deployment of a SAGIN becomes a major barrier for realizing such SAGINs. Compared to the space and terrestrial components, UAVs are expected to meet performance requirements with high flexibility and dynamics using limited resources. Therefore, employing UAVs in various usage scenarios requires well-designed planning in algorithmic approaches. In this paper, we provide a comprehensive review of recent learning-based algorithmic approaches. We consider possible reward functions and discuss the state-of-the-art algorithms for optimizing the reward functions, including Q-learning, deep Q-learning, multi-armed bandit (MAB), particle swarm optimization (PSO) and satisfaction-based learning algorithms. Unlike other survey papers, we focus on the methodological perspective of the optimization problem, which can be applicable to various UAV-assisted missions on a SAGIN using these algorithms. We simulate users and environments according to real-world scenarios and compare the learning-based and PSO-based methods in terms of throughput, load, fairness, computation time, etc. We also implement and evaluate the 2-dimensional (2D) and 3-dimensional (3D) variations of these algorithms to reflect different deployment cases. Our simulation suggests that the $3$D satisfaction-based learning algorithm outperforms the other approaches for various metrics in most cases. We discuss some open challenges at the end and our findings aim to provide design guidelines for algorithm selections while optimizing the deployment of UAV-assisted SAGINs.

Unmanned aerial vehicle (UAV) swarms are considered as a promising technique for next-generation communication networks due to their flexibility, mobility, low cost, and the ability to collaboratively and autonomously provide services. Distributed learning (DL) enables UAV swarms to intelligently provide communication services, multi-directional remote surveillance, and target tracking. In this survey, we first introduce several popular DL algorithms such as federated learning (FL), multi-agent Reinforcement Learning (MARL), distributed inference, and split learning, and present a comprehensive overview of their applications for UAV swarms, such as trajectory design, power control, wireless resource allocation, user assignment, perception, and satellite communications. Then, we present several state-of-the-art applications of UAV swarms in wireless communication systems, such us reconfigurable intelligent surface (RIS), virtual reality (VR), semantic communications, and discuss the problems and challenges that DL-enabled UAV swarms can solve in these applications. Finally, we describe open problems of using DL in UAV swarms and future research directions of DL enabled UAV swarms. In summary, this survey provides a comprehensive survey of various DL applications for UAV swarms in extensive scenarios.

许多现实世界的应用程序都可以作为多机构合作问题进行配置，例如网络数据包路由和自动驾驶汽车的协调。深入增强学习（DRL）的出现为通过代理和环境的相互作用提供了一种有前途的多代理合作方法。但是，在政策搜索过程中，传统的DRL解决方案遭受了多个代理具有连续动作空间的高维度。此外，代理商政策的动态性使训练非平稳。为了解决这些问题，我们建议采用高级决策和低水平的个人控制，以进行有效的政策搜索，提出一种分层增强学习方法。特别是，可以在高级离散的动作空间中有效地学习多个代理的合作。同时，低水平的个人控制可以减少为单格强化学习。除了分层增强学习外，我们还建议对手建模网络在学习过程中对其他代理的政策进行建模。与端到端的DRL方法相反，我们的方法通过以层次结构将整体任务分解为子任务来降低学习的复杂性。为了评估我们的方法的效率，我们在合作车道变更方案中进行了现实世界中的案例研究。模拟和现实世界实验都表明我们的方法在碰撞速度和收敛速度中的优越性。

无人驾驶飞行器（UAV）是支持各种服务，包括通信的技术突破之一。UAV将在提高无线网络的物理层安全方面发挥关键作用。本文定义了窃听地面用户与UAV之间的链路的问题，该联接器用作空中基站（ABS）。提出了加强学习算法Q - 学习和深Q网络（DQN），用于优化ABS的位置和传输功率，以增强地面用户的数据速率。如果没有系统了解窃听器的位置，这会增加保密容量。与Q-Learnch和基线方法相比，仿真结果显示了拟议DQN的快速收敛性和最高保密能力。

FOG无线电访问网络（F-RAN）是一项有前途的技术，用户移动设备（MDS）可以将计算任务卸载到附近的FOG接入点（F-APS）。由于F-APS的资源有限，因此设计有效的任务卸载方案很重要。在本文中，通过考虑随时间变化的网络环境，制定了F-RAN中的动态计算卸载和资源分配问题，以最大程度地减少MD的任务执行延迟和能源消耗。为了解决该问题，提出了基于联合的深入强化学习（DRL）算法，其中深层确定性策略梯度（DDPG）算法在每个F-AP中执行计算卸载和资源分配。利用联合学习来培训DDPG代理，以降低培训过程的计算复杂性并保护用户隐私。仿真结果表明，与其他现有策略相比，提议的联合DDPG算法可以更快地实现MDS更快的任务执行延迟和能源消耗。

车辆边缘计算（VEC）可以在网络边缘的不同RSU中缓存内容，以支持实时车辆应用。在VEC中，由于车辆的高运动特性，有必要提前缓存用户数据，并为车辆用户学习最流行和最有趣的内容。由于用户数据通常包含隐私信息，因此用户不愿与他人共享其数据。为了解决这个问题，传统的联合学习（FL）需要通过汇总所有用户的本地模型来保护用户的隐私来同步更新全局模型。但是，车辆可能会在实现本地模型培训之前经常离开VEC的覆盖范围，因此无法按预期上传本地型号，这将降低全球模型的准确性。此外，本地RSU的缓存能力有限，流行内容是多样的，因此预测的流行内容的大小通常超过本地RSU的缓存能力。因此，在考虑内容传输延迟的同时，VEC应在不同的RSU中缓存预测的流行内容。在本文中，我们考虑了车辆的流动性，并提出了基于联合和深度强化学习（CAFR）的VEC中的合作缓存计划。我们首先考虑车辆的移动性，并提出异步FL算法以获得准确的全局模型，然后提出一种算法来预测基于全球模型的流行内容。此外，我们考虑了车辆的移动性，并提出了深入的强化学习算法，以获取预测流行内容的最佳合作缓存位置，以优化内容传输延迟。广泛的实验结果表明，CAFR方案的表现优于其他基线缓存方案。

The connectivity-aware path design is crucial in the effective deployment of autonomous Unmanned Aerial Vehicles (UAVs). Recently, Reinforcement Learning (RL) algorithms have become the popular approach to solving this type of complex problem, but RL algorithms suffer slow convergence. In this paper, we propose a Transfer Learning (TL) approach, where we use a teacher policy previously trained in an old domain to boost the path learning of the agent in the new domain. As the exploration processes and the training continue, the agent refines the path design in the new domain based on the subsequent interactions with the environment. We evaluate our approach considering an old domain at sub-6 GHz and a new domain at millimeter Wave (mmWave). The teacher path policy, previously trained at sub-6 GHz path, is the solution to a connectivity-aware path problem that we formulate as a constrained Markov Decision Process (CMDP). We employ a Lyapunov-based model-free Deep Q-Network (DQN) to solve the path design at sub-6 GHz that guarantees connectivity constraint satisfaction. We empirically demonstrate the effectiveness of our approach for different urban environment scenarios. The results demonstrate that our proposed approach is capable of reducing the training time considerably at mmWave.

我们考虑了在透明的蜂窝车辆到所有物品（C-V2X）系统中的联合渠道分配和电力分配的问题，其中多个车辆到网络（V2N）上行链路共享与多个车辆到车辆的时频资源（ v2v）排，使连接和自动驾驶汽车的团体可以紧密地一起旅行。由于在车辆环境中使用高用户移动性的性质，依赖全球渠道信息的传统集中优化方法在具有大量用户的C-V2X系统中可能不可行。利用多机构增强学习（RL）方法，我们提出了分布式资源分配（RA）算法来克服这一挑战。具体而言，我们将RA问题建模为多代理系统。仅基于本地渠道信息，每个排领导者充当代理，共同相互交互，因此选择了子频段和功率水平的最佳组合来传输其信号。为此，我们利用双重Q学习算法在同时最大化V2N链接的总和率的目标下共同训练代理，并满足所需延迟限制的每个V2V链接的数据包输送概率。仿真结果表明，与众所周知的详尽搜索算法相比，我们提出的基于RL的算法提供了紧密的性能。

协作深度加强学习（CDRL）算法，其中多个代理可以在无线网络上协调是一种有希望的方法，以便在复杂的动态环境中依赖实时决策的未来智能和自主系统。尽管如此，在实际情况下，CDRL由​​于代理的异质性及其学习任务，不同环境，学习时间限制以及无线网络的资源限制，因此CDRL面临着许多挑战。为了解决这些挑战，在本文中，提出了一种新颖的语义感知CDRL方法，以使一组异构未经训练的代理具有语义连接的DRL任务，以在资源受限无线蜂窝网络上有效地协作。为此，提出了一种新的异构联邦DRL（HFDRL）算法，以选择用于协作的语义相关DRL代理的最佳子集。然后，该方法将共同优化合作选定代理的训练损失和无线带宽分配，以便在其实时任务的时间限制内培训每个代理。仿真结果表明，与最先进的基线相比，所提出的算法的卓越性能。

与LTE网络相比，5G的愿景在于提供较高的数据速率，低延迟（为了实现近实时应用程序），大大增加了基站容量以及用户的接近完美服务质量（QoS）。为了提供此类服务，5G系统将支持LTE，NR，NR-U和Wi-Fi等访问技术的各种组合。每种无线电访问技术（RAT）都提供不同类型的访问，这些访问应在用户中对其进行最佳分配和管理。除了资源管理外，5G系统还将支持双重连接服务。因此，网络的编排对于系统经理在旧式访问技术方面来说是一个更困难的问题。在本文中，我们提出了一种基于联合元学习（FML）的大鼠分配算法，该算法使RAN Intelligent Controller（RIC）能够更快地适应动态变化的环境。我们设计了一个包含LTE和5G NR服务技术的模拟环境。在模拟中，我们的目标是在传输的截止日期内满足UE需求，以提供更高的QoS值。我们将提出的算法与单个RL试剂，爬行动物算法和基于规则的启发式方法进行了比较。仿真结果表明，提出的FML方法分别在第一部部署回合21％和12％时达到了较高的缓存率。此外，在比较方法中，提出的方法最快地适应了新任务和环境。

Multi-agent settings remain a fundamental challenge in the reinforcement learning (RL) domain due to the partial observability and the lack of accurate real-time interactions across agents. In this paper, we propose a new method based on local communication learning to tackle the multi-agent RL (MARL) challenge within a large number of agents coexisting. First, we design a new communication protocol that exploits the ability of depthwise convolution to efficiently extract local relations and learn local communication between neighboring agents. To facilitate multi-agent coordination, we explicitly learn the effect of joint actions by taking the policies of neighboring agents as inputs. Second, we introduce the mean-field approximation into our method to reduce the scale of agent interactions. To more effectively coordinate behaviors of neighboring agents, we enhance the mean-field approximation by a supervised policy rectification network (PRN) for rectifying real-time agent interactions and by a learnable compensation term for correcting the approximation bias. The proposed method enables efficient coordination as well as outperforms several baseline approaches on the adaptive traffic signal control (ATSC) task and the StarCraft II multi-agent challenge (SMAC).

Hybrid FSO/RF system requires an efficient FSO and RF link switching mechanism to improve the system capacity by realizing the complementary benefits of both the links. The dynamics of network conditions, such as fog, dust, and sand storms compound the link switching problem and control complexity. To address this problem, we initiate the study of deep reinforcement learning (DRL) for link switching of hybrid FSO/RF systems. Specifically, in this work, we focus on actor-critic called Actor/Critic-FSO/RF and Deep-Q network (DQN) called DQN-FSO/RF for FSO/RF link switching under atmospheric turbulences. To formulate the problem, we define the state, action, and reward function of a hybrid FSO/RF system. DQN-FSO/RF frequently updates the deployed policy that interacts with the environment in a hybrid FSO/RF system, resulting in high switching costs. To overcome this, we lift this problem to ensemble consensus-based representation learning for deep reinforcement called DQNEnsemble-FSO/RF. The proposed novel DQNEnsemble-FSO/RF DRL approach uses consensus learned features representations based on an ensemble of asynchronous threads to update the deployed policy. Experimental results corroborate that the proposed DQNEnsemble-FSO/RF's consensus-learned features switching achieves better performance than Actor/Critic-FSO/RF, DQN-FSO/RF, and MyOpic for FSO/RF link switching while keeping the switching cost significantly low.

The explosive growth of dynamic and heterogeneous data traffic brings great challenges for 5G and beyond mobile networks. To enhance the network capacity and reliability, we propose a learning-based dynamic time-frequency division duplexing (D-TFDD) scheme that adaptively allocates the uplink and downlink time-frequency resources of base stations (BSs) to meet the asymmetric and heterogeneous traffic demands while alleviating the inter-cell interference. We formulate the problem as a decentralized partially observable Markov decision process (Dec-POMDP) that maximizes the long-term expected sum rate under the users' packet dropping ratio constraints. In order to jointly optimize the global resources in a decentralized manner, we propose a federated reinforcement learning (RL) algorithm named federated Wolpertinger deep deterministic policy gradient (FWDDPG) algorithm. The BSs decide their local time-frequency configurations through RL algorithms and achieve global training via exchanging local RL models with their neighbors under a decentralized federated learning framework. Specifically, to deal with the large-scale discrete action space of each BS, we adopt a DDPG-based algorithm to generate actions in a continuous space, and then utilize Wolpertinger policy to reduce the mapping errors from continuous action space back to discrete action space. Simulation results demonstrate the superiority of our proposed algorithm to benchmark algorithms with respect to system sum rate.