在这项工作中，我们提出并评估了一种新的增强学习方法，紧凑体验重放（编者），它使用基于相似转换集的复发的预测目标值的时间差异学习，以及基于两个转换的经验重放的新方法记忆。我们的目标是减少在长期累计累计奖励的经纪人培训所需的经验。它与强化学习的相关性与少量观察结果有关，即它需要实现类似于文献中的相关方法获得的结果，这通常需要数百万视频框架来培训ATARI 2600游戏。我们举报了在八个挑战街机学习环境（ALE）挑战游戏中，为仅10万帧的培训试验和大约25,000次迭代的培训试验中报告了培训试验。我们还在与基线的同一游戏中具有相同的实验协议的DQN代理呈现结果。为了验证从较少数量的观察结果近似于良好的政策，我们还将其结果与从啤酒的基准上呈现的数百万帧中获得的结果进行比较。

We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards. We apply our method to seven Atari 2600 games from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm. We find that it outperforms all previous approaches on six of the games and surpasses a human expert on three of them.

The deep reinforcement learning community has made several independent improvements to the DQN algorithm. However, it is unclear which of these extensions are complementary and can be fruitfully combined. This paper examines six extensions to the DQN algorithm and empirically studies their combination. Our experiments show that the combination provides state-of-the-art performance on the Atari 2600 benchmark, both in terms of data efficiency and final performance. We also provide results from a detailed ablation study that shows the contribution of each component to overall performance.

Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their significance. In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently. We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level performance across many Atari games. DQN with prioritized experience replay achieves a new stateof-the-art, outperforming DQN with uniform replay on 41 out of 49 games.

自成立以来，建立在广泛任务中表现出色的普通代理的任务一直是强化学习的重要目标。这个问题一直是对Alarge工作体系的研究的主题，并且经常通过观察Atari 57基准中包含的广泛范围环境的分数来衡量的性能。 Agent57是所有57场比赛中第一个超过人类基准的代理商，但这是以数据效率差的代价，需要实现近800亿帧的经验。以Agent57为起点，我们采用了各种各样的形式，以降低超过人类基线所需的经验200倍。在减少数据制度和Propose有效的解决方案时，我们遇到了一系列不稳定性和瓶颈，以构建更强大，更有效的代理。我们还使用诸如Muesli和Muzero之类的高性能方法证明了竞争性的性能。 TOOUR方法的四个关键组成部分是（1）近似信任区域方法，该方法可以从TheOnline网络中稳定引导，（2）损失和优先级的归一化方案，在学习具有广泛量表的一组值函数时，可以提高鲁棒性， （3）改进的体系结构采用了NFNET的技术技术来利用更深的网络而无需标准化层，并且（4）政策蒸馏方法可使瞬时贪婪的策略加班。

机器学习算法中多个超参数的最佳设置是发出大多数可用数据的关键。为此目的，已经提出了几种方法，例如进化策略，随机搜索，贝叶斯优化和启发式拇指规则。在钢筋学习（RL）中，学习代理在与其环境交互时收集的数据的信息内容严重依赖于许多超参数的设置。因此，RL算法的用户必须依赖于基于搜索的优化方法，例如网格搜索或Nelder-Mead单简单算法，这对于大多数R1任务来说是非常效率的，显着减慢学习曲线和离开用户的速度有目的地偏见数据收集的负担。在这项工作中，为了使RL算法更加用户独立，提出了一种使用贝叶斯优化的自主超参数设置的新方法。来自过去剧集和不同的超参数值的数据通过执行行为克隆在元学习水平上使用，这有助于提高最大化获取功能的加强学习变体的有效性。此外，通过紧密地整合在加强学习代理设计中的贝叶斯优化，还减少了收敛到给定任务的最佳策略所需的状态转换的数量。与其他手动调整和基于优化的方法相比，计算实验显示了有希望的结果，这突出了改变算法超级参数来增加所生成数据的信息内容的好处。

Deep reinforcement learning is poised to revolutionise the field of AI and represents a step towards building autonomous systems with a higher level understanding of the visual world. Currently, deep learning is enabling reinforcement learning to scale to problems that were previously intractable, such as learning to play video games directly from pixels. Deep reinforcement learning algorithms are also applied to robotics, allowing control policies for robots to be learned directly from camera inputs in the real world. In this survey, we begin with an introduction to the general field of reinforcement learning, then progress to the main streams of value-based and policybased methods. Our survey will cover central algorithms in deep reinforcement learning, including the deep Q-network, trust region policy optimisation, and asynchronous advantage actor-critic. In parallel, we highlight the unique advantages of deep neural networks, focusing on visual understanding via reinforcement learning. To conclude, we describe several current areas of research within the field.

深度强化学习（RL）导致了许多最近和开创性的进步。但是，这些进步通常以培训的基础体系结构的规模增加以及用于训练它们的RL算法的复杂性提高，而均以增加规模的成本。这些增长反过来又使研究人员更难迅速原型新想法或复制已发表的RL算法。为了解决这些问题，这项工作描述了ACME，这是一个用于构建新型RL算法的框架，这些框架是专门设计的，用于启用使用简单的模块化组件构建的代理，这些组件可以在各种执行范围内使用。尽管ACME的主要目标是为算法开发提供一个框架，但第二个目标是提供重要或最先进算法的简单参考实现。这些实现既是对我们的设计决策的验证，也是对RL研究中可重复性的重要贡献。在这项工作中，我们描述了ACME内部做出的主要设计决策，并提供了有关如何使用其组件来实施各种算法的进一步详细信息。我们的实验为许多常见和最先进的算法提供了基准，并显示了如何为更大且更复杂的环境扩展这些算法。这突出了ACME的主要优点之一，即它可用于实现大型，分布式的RL算法，这些算法可以以较大的尺度运行，同时仍保持该实现的固有可读性。这项工作提出了第二篇文章的版本，恰好与模块化的增加相吻合，对离线，模仿和从演示算法学习以及作为ACME的一部分实现的各种新代理。

Efficient exploration remains a major challenge for reinforcement learning (RL). Common dithering strategies for exploration, such as -greedy, do not carry out temporally-extended (or deep) exploration; this can lead to exponentially larger data requirements. However, most algorithms for statistically efficient RL are not computationally tractable in complex environments. Randomized value functions offer a promising approach to efficient exploration with generalization, but existing algorithms are not compatible with nonlinearly parameterized value functions. As a first step towards addressing such contexts we develop bootstrapped DQN. We demonstrate that bootstrapped DQN can combine deep exploration with deep neural networks for exponentially faster learning than any dithering strategy. In the Arcade Learning Environment bootstrapped DQN substantially improves learning speed and cumulative performance across most games.

在探索中，由于当前的低效率而引起的强化学习领域，具有较大动作空间的学习控制政策是一个具有挑战性的问题。在这项工作中，我们介绍了深入的强化学习（DRL）算法呼叫多动作网络（MAN）学习，以应对大型离散动作空间的挑战。我们建议将动作空间分为两个组件，从而为每个子行动创建一个值神经网络。然后，人使用时间差异学习来同步训练网络，这比训练直接动作输出的单个网络要简单。为了评估所提出的方法，我们在块堆叠任务上测试了人，然后扩展了人类从Atari Arcade学习环境中使用18个动作空间的12个游戏。我们的结果表明，人的学习速度比深Q学习和双重Q学习更快，这意味着我们的方法比当前可用于大型动作空间的方法更好地执行同步时间差异算法。

In recent years there have been many successes of using deep representations in reinforcement learning. Still, many of these applications use conventional architectures, such as convolutional networks, LSTMs, or auto-encoders. In this paper, we present a new neural network architecture for model-free reinforcement learning. Our dueling network represents two separate estimators: one for the state value function and one for the state-dependent action advantage function. The main benefit of this factoring is to generalize learning across actions without imposing any change to the underlying reinforcement learning algorithm. Our results show that this architecture leads to better policy evaluation in the presence of many similar-valued actions. Moreover, the dueling architecture enables our RL agent to outperform the state-of-the-art on the Atari 2600 domain.

Atari games have been a long-standing benchmark in the reinforcement learning (RL) community for the past decade. This benchmark was proposed to test general competency of RL algorithms. Previous work has achieved good average performance by doing outstandingly well on many games of the set, but very poorly in several of the most challenging games. We propose Agent57, the first deep RL agent that outperforms the standard human benchmark on all 57 Atari games. To achieve this result, we train a neural network which parameterizes a family of policies ranging from very exploratory to purely exploitative. We propose an adaptive mechanism to choose which policy to prioritize throughout the training process. Additionally, we utilize a novel parameterization of the architecture that allows for more consistent and stable learning.

A long-standing challenge in artificial intelligence is lifelong learning. In lifelong learning, many tasks are presented in sequence and learners must efficiently transfer knowledge between tasks while avoiding catastrophic forgetting over long lifetimes. On these problems, policy reuse and other multi-policy reinforcement learning techniques can learn many tasks. However, they can generate many temporary or permanent policies, resulting in memory issues. Consequently, there is a need for lifetime-scalable methods that continually refine a policy library of a pre-defined size. This paper presents a first approach to lifetime-scalable policy reuse. To pre-select the number of policies, a notion of task capacity, the maximal number of tasks that a policy can accurately solve, is proposed. To evaluate lifetime policy reuse using this method, two state-of-the-art single-actor base-learners are compared: 1) a value-based reinforcement learner, Deep Q-Network (DQN) or Deep Recurrent Q-Network (DRQN); and 2) an actor-critic reinforcement learner, Proximal Policy Optimisation (PPO) with or without Long Short-Term Memory layer. By selecting the number of policies based on task capacity, D(R)QN achieves near-optimal performance with 6 policies in a 27-task MDP domain and 9 policies in an 18-task POMDP domain; with fewer policies, catastrophic forgetting and negative transfer are observed. Due to slow, monotonic improvement, PPO requires fewer policies, 1 policy for the 27-task domain and 4 policies for the 18-task domain, but it learns the tasks with lower accuracy than D(R)QN. These findings validate lifetime-scalable policy reuse and suggest using D(R)QN for larger and PPO for smaller library sizes.

由于数据量增加，金融业的快速变化已经彻底改变了数据处理和数据分析的技术，并带来了新的理论和计算挑战。与古典随机控制理论和解决财务决策问题的其他分析方法相比，解决模型假设的财务决策问题，强化学习（RL）的新发展能够充分利用具有更少模型假设的大量财务数据并改善复杂的金融环境中的决策。该调查纸目的旨在审查最近的资金途径的发展和使用RL方法。我们介绍了马尔可夫决策过程，这是许多常用的RL方法的设置。然后引入各种算法，重点介绍不需要任何模型假设的基于价值和基于策略的方法。连接是用神经网络进行的，以扩展框架以包含深的RL算法。我们的调查通过讨论了这些RL算法在金融中各种决策问题中的应用，包括最佳执行，投资组合优化，期权定价和对冲，市场制作，智能订单路由和Robo-Awaring。

Deep Reinforcement Learning has yielded proficient controllers for complex tasks. However, these controllers have limited memory and rely on being able to perceive the complete game screen at each decision point. To address these shortcomings, this article investigates the effects of adding recurrency to a Deep Q-Network (DQN) by replacing the first post-convolutional fully-connected layer with a recurrent LSTM. The resulting Deep Recurrent Q-Network (DRQN), although capable of seeing only a single frame at each timestep, successfully integrates information through time and replicates DQN's performance on standard Atari games and partially observed equivalents featuring flickering game screens. Additionally, when trained with partial observations and evaluated with incrementally more complete observations, DRQN's performance scales as a function of observability. Conversely, when trained with full observations and evaluated with partial observations, DRQN's performance degrades less than DQN's. Thus, given the same length of history, recurrency is a viable alternative to stacking a history of frames in the DQN's input layer and while recurrency confers no systematic advantage when learning to play the game, the recurrent net can better adapt at evaluation time if the quality of observations changes.

Off-policy reinforcement learning (RL) using a fixed offline dataset of logged interactions is an important consideration in real world applications. This paper studies offline RL using the DQN Replay Dataset comprising the entire replay experience of a DQN agent on 60 Atari 2600 games. We demonstrate that recent off-policy deep RL algorithms, even when trained solely on this fixed dataset, outperform the fully-trained DQN agent. To enhance generalization in the offline setting, we present Random Ensemble Mixture (REM), a robust Q-learning algorithm that enforces optimal Bellman consistency on random convex combinations of multiple Q-value estimates. Offline REM trained on the DQN Replay Dataset surpasses strong RL baselines. Ablation studies highlight the role of offline dataset size and diversity as well as the algorithm choice in our positive results. Overall, the results here present an optimistic view that robust RL algorithms used on sufficiently large and diverse offline datasets can lead to high quality policies. To provide a testbed for offline RL and reproduce our results, the DQN Replay Dataset is released at offline-rl.github.io.

深度神经网络的强大学习能力使强化学习者能够直接从连续环境中学习有效的控制政策。从理论上讲，为了实现稳定的性能，神经网络假设I.I.D.不幸的是，在训练数据在时间上相关且非平稳的一般强化学习范式中，输入不存在。这个问题可能导致“灾难性干扰”和性能崩溃的现象。在本文中，我们提出智商，即干涉意识深度Q学习，以减轻单任务深度加固学习中的灾难性干扰。具体来说，我们求助于在线聚类，以实现在线上下文部门，以及一个多头网络和一个知识蒸馏正规化术语，用于保留学习上下文的政策。与现有方法相比，智商基于深Q网络，始终如一地提高稳定性和性能，并通过对经典控制和ATARI任务进行了广泛的实验。该代码可在以下网址公开获取：https：//github.com/sweety-dm/interference-aware-ware-deep-q-learning。

资产分配（或投资组合管理）是确定如何最佳将有限预算的资金分配给一系列金融工具/资产（例如股票）的任务。这项研究调查了使用无模型的深RL代理应用于投资组合管理的增强学习（RL）的性能。我们培训了几个RL代理商的现实股票价格，以学习如何执行资产分配。我们比较了这些RL剂与某些基线剂的性能。我们还比较了RL代理，以了解哪些类别的代理表现更好。从我们的分析中，RL代理可以执行投资组合管理的任务，因为它们的表现明显优于基线代理（随机分配和均匀分配）。四个RL代理（A2C，SAC，PPO和TRPO）总体上优于最佳基线MPT。这显示了RL代理商发现更有利可图的交易策略的能力。此外，基于价值和基于策略的RL代理之间没有显着的性能差异。演员批评者的表现比其他类型的药物更好。同样，在政策代理商方面的表现要好，因为它们在政策评估方面更好，样品效率在投资组合管理中并不是一个重大问题。这项研究表明，RL代理可以大大改善资产分配，因为它们的表现优于强基础。基于我们的分析，在政策上，参与者批评的RL药物显示出最大的希望。

横跨街机学习环境，彩虹实现了对人类和现代RL算法的竞争程度。然而，获得这种性能水平需要大量的数据和硬件资源，在该区域进行研究计算地昂贵并且在实际应用中使用通常是不可行的。本文的贡献是三倍：我们（1）提出了一种改进的彩虹版本，寻求大大减少彩虹的数据，培训时间和计算要求，同时保持其竞争性能; （2）我们通过实验通过对街机学习环境的实验来证明我们的方法的有效性，以及（3）我们进行了许多消融研究，以研究个体提出的修改的效果。我们改进的Rainbow版本达到了靠近经典彩虹的中位数的人为规范化分数，而使用20倍的数据，只需要7.5小时的单个GPU培训时间。我们还提供了我们的全部实施，包括预先训练的型号。