Constructing agents with planning capabilities has long been one of the main challenges in the pursuit of artificial intelligence. Tree-based planning methods have enjoyed huge success in challenging domains, such as chess and Go, where a perfect simulator is available. However, in real-world problems the dynamics governing the environment are often complex and unknown. In this work we present the MuZero algorithm which, by combining a tree-based search with a learned model, achieves superhuman performance in a range of challenging and visually complex domains, without any knowledge of their underlying dynamics. MuZero learns a model that, when applied iteratively, predicts the quantities most directly relevant to planning: the reward, the action-selection policy, and the value function. When evaluated on 57 different Atari games -the canonical video game environment for testing AI techniques, in which model-based planning approaches have historically struggled -our new algorithm achieved a new state of the art. When evaluated on Go, chess and shogi, without any knowledge of the game rules, MuZero matched the superhuman performance of the AlphaZero algorithm that was supplied with the game rules.

强化学习在许多应用中取得了巨大的成功。然而，样本效率仍然是一个关键挑战，突出的方法需要训练数百万（甚至数十亿）的环境步骤。最近，基于样本的基于图像的RL算法存在显着进展;然而，Atari游戏基准上的一致人级表现仍然是一个难以捉摸的目标。我们提出了一种在Muzero上建立了基于模式的基于模型的Visual RL算法，我们名称为高效零。我们的方法达到了194.3％的人类性能和Atari 100K基准的109.0％的中位数，只有两个小时的实时游戏体验，并且在DMControl 100k基准测试中的某些任务中优于状态萨克。这是第一次算法在atari游戏中实现超级人类性能，具有如此少的数据。高效零的性能也在2亿帧的比赛中靠近DQN的性能，而我们使用的数据减少了500倍。高效零的低样本复杂性和高性能可以使RL更接近现实世界的适用性。我们以易于理解的方式实现我们的算法，它可以在https://github.com/yewr/effionszero中获得。我们希望它将加速更广泛社区中MCT的RL算法的研究。

使用规划算法和神经网络模型的基于模型的强化学习范例最近在不同的应用中实现了前所未有的结果，导致现在被称为深度增强学习的内容。这些代理非常复杂，涉及多个组件，可能会为研究产生挑战的因素。在这项工作中，我们提出了一个适用于这些类型代理的新模块化软件架构，以及一组建筑块，可以轻松重复使用和组装，以构建基于模型的增强学习代理。这些构建块包括规划算法，策略和丢失功能。我们通过将多个这些构建块组合实现和测试经过针对三种不同的测试环境的代理来说明这种架构的使用：Cartpole，Minigrid和Tictactoe。在我们的实施中提供的一个特定的规划算法，并且以前没有用于加强学习，我们称之为Imperage Minimax，在三个测试环境中取得了良好的效果。用这种架构进行的实验表明，规划算法，政策和损失函数的最佳组合依赖性严重问题。该结果提供了证据表明，拟议的架构是模块化和可重复使用的，对想要研究新环境和技术的强化学习研究人员有用。

We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards. We apply our method to seven Atari 2600 games from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm. We find that it outperforms all previous approaches on six of the games and surpasses a human expert on three of them.

A long-standing goal of artificial intelligence is an algorithm that learns, tabula rasa, superhuman proficiency in challenging domains. Recently, AlphaGo became the first program to defeat a world champion in the game of Go. The tree search in AlphaGo evaluated positions and selected moves using deep neural networks. These neural networks were trained by supervised learning from human expert moves, and by reinforcement learning from selfplay. Here, we introduce an algorithm based solely on reinforcement learning, without human data, guidance, or domain knowledge beyond game rules. AlphaGo becomes its own teacher: a neural network is trained to predict AlphaGo's own move selections and also the winner of AlphaGo's games. This neural network improves the strength of tree search, resulting in higher quality move selection and stronger self-play in the next iteration. Starting tabula rasa, our new program AlphaGo Zero achieved superhuman performance, winning 100-0 against the previously published, champion-defeating AlphaGo.Much progress towards artificial intelligence has been made using supervised learning systems that are trained to replicate the decisions of human experts 1-4 . However, expert data is often expensive, unreliable, or simply unavailable. Even when reliable data is available it may impose a ceiling on the performance of systems trained in this manner 5 . In contrast, reinforcement learning systems are trained from their own experience, in principle allowing them to exceed human capabilities, and to operate in domains where human expertise is lacking. Recently, there has been rapid progress towards this goal, using deep neural networks trained by reinforcement learning.These systems have outperformed humans in computer games such as Atari 6, 7 and 3D virtual environments [8][9][10] . However, the most challenging domains in terms of human intellect -such as the

The game of chess is the longest-studied domain in the history of artificial intelligence. The strongest programs are based on a combination of sophisticated search techniques, domain-specific adaptations, and handcrafted evaluation functions that have been refined by human experts over several decades. By contrast, the AlphaGo Zero program recently achieved superhuman performance in the game of Go by reinforcement learning from selfplay. In this paper, we generalize this approach into a single AlphaZero algorithm that can achieve superhuman performance in many challenging games. Starting from random play and given no domain knowledge except the game rules, AlphaZero convincingly defeated a world champion program in the games of chess and shogi (Japanese chess) as well as Go.The study of computer chess is as old as computer science itself. Charles Babbage, Alan Turing, Claude Shannon, and John von Neumann devised hardware, algorithms and theory to analyse and play the game of chess. Chess subsequently became a grand challenge task for a generation of artificial intelligence researchers, culminating in high-performance computer chess programs that play at a super-human level (1,2). However, these systems are highly tuned to their domain, and cannot be generalized to other games without substantial human effort, whereas general game-playing systems (3, 4) remain comparatively weak.A long-standing ambition of artificial intelligence has been to create programs that can instead learn for themselves from first principles (5, 6). Recently, the AlphaGo Zero algorithm achieved superhuman performance in the game of Go, by representing Go knowledge using deep convolutional neural networks (7, 8), trained solely by reinforcement learning from games

2048 is a single-player stochastic puzzle game. This intriguing and addictive game has been popular worldwide and has attracted researchers to develop game-playing programs. Due to its simplicity and complexity, 2048 has become an interesting and challenging platform for evaluating the effectiveness of machine learning methods. This dissertation conducts comprehensive research on reinforcement learning and computer game algorithms for 2048. First, this dissertation proposes optimistic temporal difference learning, which significantly improves the quality of learning by employing optimistic initialization to encourage exploration for 2048. Furthermore, based on this approach, a state-of-the-art program for 2048 is developed, which achieves the highest performance among all learning-based programs, namely an average score of 625377 points and a rate of 72% for reaching 32768-tiles. Second, this dissertation investigates several techniques related to 2048, including the n-tuple network ensemble learning, Monte Carlo tree search, and deep reinforcement learning. These techniques are promising for further improving the performance of the current state-of-the-art program. Finally, this dissertation discusses pedagogical applications related to 2048 by proposing course designs and summarizing the teaching experience. The proposed course designs use 2048-like games as materials for beginners to learn reinforcement learning and computer game algorithms. The courses have been successfully applied to graduate-level students and received well by student feedback.

In this article we introduce the Arcade Learning Environment (ALE): both a challenge problem and a platform and methodology for evaluating the development of general, domain-independent AI technology. ALE provides an interface to hundreds of Atari 2600 game environments, each one different, interesting, and designed to be a challenge for human players. ALE presents significant research challenges for reinforcement learning, model learning, model-based planning, imitation learning, transfer learning, and intrinsic motivation. Most importantly, it provides a rigorous testbed for evaluating and comparing approaches to these problems. We illustrate the promise of ALE by developing and benchmarking domain-independent agents designed using well-established AI techniques for both reinforcement learning and planning. In doing so, we also propose an evaluation methodology made possible by ALE, reporting empirical results on over 55 different games. All of the software, including the benchmark agents, is publicly available.

Deep reinforcement learning is poised to revolutionise the field of AI and represents a step towards building autonomous systems with a higher level understanding of the visual world. Currently, deep learning is enabling reinforcement learning to scale to problems that were previously intractable, such as learning to play video games directly from pixels. Deep reinforcement learning algorithms are also applied to robotics, allowing control policies for robots to be learned directly from camera inputs in the real world. In this survey, we begin with an introduction to the general field of reinforcement learning, then progress to the main streams of value-based and policybased methods. Our survey will cover central algorithms in deep reinforcement learning, including the deep Q-network, trust region policy optimisation, and asynchronous advantage actor-critic. In parallel, we highlight the unique advantages of deep neural networks, focusing on visual understanding via reinforcement learning. To conclude, we describe several current areas of research within the field.

Efficient exploration remains a major challenge for reinforcement learning (RL). Common dithering strategies for exploration, such as -greedy, do not carry out temporally-extended (or deep) exploration; this can lead to exponentially larger data requirements. However, most algorithms for statistically efficient RL are not computationally tractable in complex environments. Randomized value functions offer a promising approach to efficient exploration with generalization, but existing algorithms are not compatible with nonlinearly parameterized value functions. As a first step towards addressing such contexts we develop bootstrapped DQN. We demonstrate that bootstrapped DQN can combine deep exploration with deep neural networks for exponentially faster learning than any dithering strategy. In the Arcade Learning Environment bootstrapped DQN substantially improves learning speed and cumulative performance across most games.

In recent years there have been many successes of using deep representations in reinforcement learning. Still, many of these applications use conventional architectures, such as convolutional networks, LSTMs, or auto-encoders. In this paper, we present a new neural network architecture for model-free reinforcement learning. Our dueling network represents two separate estimators: one for the state value function and one for the state-dependent action advantage function. The main benefit of this factoring is to generalize learning across actions without imposing any change to the underlying reinforcement learning algorithm. Our results show that this architecture leads to better policy evaluation in the presence of many similar-valued actions. Moreover, the dueling architecture enables our RL agent to outperform the state-of-the-art on the Atari 2600 domain.

游戏历史悠久的历史悠久地作为人工智能进步的基准。最近，使用搜索和学习的方法在一系列完美的信息游戏中表现出强烈的表现，并且使用游戏理论推理和学习的方法对特定的不完美信息扑克变体表示了很强的性能。我们介绍游戏玩家，一个通用算法，统一以前的方法，结合导游搜索，自助学习和游戏理论推理。游戏播放器是实现大型完美和不完美信息游戏中强大实证性能的第一个算法 - 这是一项真正的任意环境算法的重要一步。我们证明了游戏玩家是声音，融合到完美的游戏，因为可用的计算时间和近似容量增加。游戏播放器在国际象棋上达到了强大的表现，然后击败了最强大的公开可用的代理商，在头上没有限制德克萨斯州扑克（Slumbot），击败了苏格兰院子的最先进的代理人，这是一个不完美的信息游戏，说明了引导搜索，学习和游戏理论推理的价值。

蒙特卡洛树搜索（MCT）是设计游戏机器人或解决顺序决策问题的强大方法。该方法依赖于平衡探索和开发的智能树搜索。MCT以模拟的形式进行随机抽样，并存储动作的统计数据，以在每个随后的迭代中做出更有教育的选择。然而，该方法已成为组合游戏的最新技术，但是，在更复杂的游戏（例如那些具有较高的分支因素或实时系列的游戏）以及各种实用领域（例如，运输，日程安排或安全性）有效的MCT应用程序通常需要其与问题有关的修改或与其他技术集成。这种特定领域的修改和混合方法是本调查的主要重点。最后一项主要的MCT调查已于2012年发布。自发布以来出现的贡献特别感兴趣。

自成立以来，建立在广泛任务中表现出色的普通代理的任务一直是强化学习的重要目标。这个问题一直是对Alarge工作体系的研究的主题，并且经常通过观察Atari 57基准中包含的广泛范围环境的分数来衡量的性能。 Agent57是所有57场比赛中第一个超过人类基准的代理商，但这是以数据效率差的代价，需要实现近800亿帧的经验。以Agent57为起点，我们采用了各种各样的形式，以降低超过人类基线所需的经验200倍。在减少数据制度和Propose有效的解决方案时，我们遇到了一系列不稳定性和瓶颈，以构建更强大，更有效的代理。我们还使用诸如Muesli和Muzero之类的高性能方法证明了竞争性的性能。 TOOUR方法的四个关键组成部分是（1）近似信任区域方法，该方法可以从TheOnline网络中稳定引导，（2）损失和优先级的归一化方案，在学习具有广泛量表的一组值函数时，可以提高鲁棒性， （3）改进的体系结构采用了NFNET的技术技术来利用更深的网络而无需标准化层，并且（4）政策蒸馏方法可使瞬时贪婪的策略加班。

强化学习（RL）通过与环境相互作用的试验过程解决顺序决策问题。尽管RL在玩复杂的视频游戏方面取得了巨大的成功，但在现实世界中，犯错误总是不希望的。为了提高样本效率并从而降低错误，据信基于模型的增强学习（MBRL）是一个有前途的方向，它建立了环境模型，在该模型中可以进行反复试验，而无需实际成本。在这项调查中，我们对MBRL进行了审查，重点是Deep RL的最新进展。对于非壮观环境，学到的环境模型与真实环境之间始终存在概括性错误。因此，非常重要的是分析环境模型中的政策培训与实际环境中的差异，这反过来又指导了更好的模型学习，模型使用和政策培训的算法设计。此外，我们还讨论了其他形式的RL，包括离线RL，目标条件RL，多代理RL和Meta-RL的最新进展。此外，我们讨论了MBRL在现实世界任务中的适用性和优势。最后，我们通过讨论MBRL未来发展的前景来结束这项调查。我们认为，MBRL在被忽略的现实应用程序中具有巨大的潜力和优势，我们希望这项调查能够吸引更多关于MBRL的研究。

The deep reinforcement learning community has made several independent improvements to the DQN algorithm. However, it is unclear which of these extensions are complementary and can be fruitfully combined. This paper examines six extensions to the DQN algorithm and empirically studies their combination. Our experiments show that the combination provides state-of-the-art performance on the Atari 2600 benchmark, both in terms of data efficiency and final performance. We also provide results from a detailed ablation study that shows the contribution of each component to overall performance.

Monte Carlo Tree Search (MCTS) is a recently proposed search method that combines the precision of tree search with the generality of random sampling. It has received considerable interest due to its spectacular success in the difficult problem of computer Go, but has also proved beneficial in a range of other domains. This paper is a survey of the literature to date, intended to provide a snapshot of the state of the art after the first five years of MCTS research. We outline the core algorithm's derivation, impart some structure on the many variations and enhancements that have been proposed, and summarise the results from the key game and non-game domains to which MCTS methods have been applied. A number of open research questions indicate that the field is ripe for future work.

在这项工作中，我们提出并评估了一种新的增强学习方法，紧凑体验重放（编者），它使用基于相似转换集的复发的预测目标值的时间差异学习，以及基于两个转换的经验重放的新方法记忆。我们的目标是减少在长期累计累计奖励的经纪人培训所需的经验。它与强化学习的相关性与少量观察结果有关，即它需要实现类似于文献中的相关方法获得的结果，这通常需要数百万视频框架来培训ATARI 2600游戏。我们举报了在八个挑战街机学习环境（ALE）挑战游戏中，为仅10万帧的培训试验和大约25,000次迭代的培训试验中报告了培训试验。我们还在与基线的同一游戏中具有相同的实验协议的DQN代理呈现结果。为了验证从较少数量的观察结果近似于良好的政策，我们还将其结果与从啤酒的基准上呈现的数百万帧中获得的结果进行比较。

当相互作用数据稀缺时，深厚的增强学习（RL）算法遭受了严重的性能下降，这限制了其现实世界的应用。最近，视觉表示学习已被证明是有效的，并且有望提高RL样品效率。这些方法通常依靠对比度学习和数据扩展来训练状态预测的过渡模型，这与在RL中使用模型的方式不同 - 基于价值的计划。因此，学到的模型可能无法与环境保持良好状态并产生一致的价值预测，尤其是当国家过渡不是确定性的情况下。为了解决这个问题，我们提出了一种称为价值一致表示学习（VCR）的新颖方法，以学习与决策直接相关的表示形式。更具体地说，VCR训练一个模型，以预测基于当前的状态（也称为“想象的状态”）和一系列动作。 VCR没有将这个想象中的状态与环境返回的真实状态保持一致，而是在两个状态上应用$ q $ - 价值头，并获得了两个行动值分布。然后将距离计算并最小化以迫使想象的状态产生与真实状态相似的动作值预测。我们为离散和连续的动作空间开发了上述想法的两个实现。我们对Atari 100K和DeepMind Control Suite基准测试进行实验，以验证其提高样品效率的有效性。已经证明，我们的方法实现了无搜索RL算法的新最新性能。