深度强化学习（DRL）在自动游戏测试中引起了很多关注。早期尝试依靠游戏内部信息进行游戏空间探索，因此需要与游戏深入集成，这对于实际应用来说是不便的。在这项工作中，我们建议仅使用屏幕截图/像素作为自动游戏测试的输入，并建立了一般游戏测试代理Inspector，可以轻松地将其应用于不同的游戏，而无需与游戏深入集成。除了覆盖所有游戏测试空间外，我们的代理商还试图采取类似人类的行为与游戏中的关键对象进行交互，因为某些错误通常发生在玩家对象的交互中。检查器基于纯粹的像素输入，包括三个关键模块：游戏空间探索器，关键对象检测器和类似人类的对象研究者。 Game Space Explorer旨在通过使用像素输入的基于好奇心的奖励功能来探索整个游戏空间。关键对象检测器的目的是基于少量标记的屏幕快照在游戏中检测关键对象。类似人类的对象研究者的目标是模仿人类的行为，以通过模仿学习来调查关键对象。我们在两个受欢迎的视频游戏中进行实验：射击游戏和动作RPG游戏。实验结果证明了检查员在探索游戏空间，检测关键对象和调查对象方面的有效性。此外，检查员在这两场比赛中成功发现了两个潜在的错误。检查员的演示视频可从https://github.com/inspector-gametesting/inspector-gametesting获得。

在游戏中，就像在其他许多领域一样，设计验证和测试是一个巨大的挑战，因为系统的大小和手动测试变得不可行。本文提出了一种新方法来自动游戏验证和测试。我们的方法利用了数据驱动的模仿学习技术，这几乎不需要精力和时间，并且对机器学习或编程不了解，设计师可以使用该技术有效地训练游戏测试剂。我们通过与行业专家的用户研究一起研究了方法的有效性。调查结果表明，我们的方法确实是一种有效的游戏验证方法，并且数据驱动的编程将是减少努力和提高现代游戏测试质量的有用帮助。该调查还突出了一些开放挑战。在最新文献的帮助下，我们分析了确定的挑战，并提出了适合支持和最大化我们方法实用性的未来研究方向。

现代AAA视频游戏具有巨大的游戏水平和地图，越来越难以详尽的测试人员覆盖。结果，游戏经常带着灾难性的虫子发货，例如玩家落在地板上或被卡在墙壁上。我们提出了一种基于功能强大的探索算法，Go-explore的模拟3D环境中针对可及性错误的方法，该方法在地图上保存了独特的检查点，然后确定有希望的探索。我们表明，当Go-explore与从游戏的导航网格中得出的简单启发式方法相结合时，发现了挑战性的错误，并全面探索了复杂的环境，而无需人类的演示或游戏动力学知识。探索大大优于更复杂的基线，包括增强学习，并在涵盖了发现的整个地图上的导航网格和独特位置的数量中都具有内在好奇心。最后，由于我们使用并行代理，我们的算法可以在10小时内在10小时内完全覆盖1.5公里x 1.5公里的游戏世界，这对于连续测试套件非常有希望。

在嘈杂的互联网规模数据集上进行了预测，已对具有广泛的文本，图像和其他模式能力的培训模型进行了大量研究。但是，对于许多顺序决策域，例如机器人技术，视频游戏和计算机使用，公开可用的数据不包含以相同方式训练行为先验所需的标签。我们通过半监督的模仿学习将互联网规模的预处理扩展到顺序的决策域，其中代理通过观看在线未标记的视频来学习行动。具体而言，我们表明，使用少量标记的数据，我们可以训练一个足够准确的反向动力学模型，可以标记一个巨大的未标记在线数据来源 - 在这里，在线播放Minecraft的在线视频 - 然后我们可以从中训练一般行为先验。尽管使用了本地人类界面（鼠标和键盘为20Hz），但我们表明，这种行为先验具有非平凡的零射击功能，并且可以通过模仿学习和加强学习，可以对其进行微调，以进行硬探索任务。不可能通过增强学习从头开始学习。对于许多任务，我们的模型都表现出人类水平的性能，我们是第一个报告可以制作钻石工具的计算机代理，这些工具可以花费超过20分钟（24,000个环境动作）的游戏玩法来实现。

本文介绍了一种扮演流行的第一人称射击（FPS）视频游戏的AI代理商的AI代理商;来自像素输入的全球攻势（CSGO）。代理人，一个深度神经网络，符合Deathmatch游戏模式内置AI内置AI的媒体难度的性能，同时采用人类的戏剧风格。与在游戏中的许多事先工作不同，CSGO没有API，因此算法必须培训并实时运行。这限制了可以生成的策略数据的数量，妨碍许多增强学习算法。我们的解决方案使用行为克隆 - 在从在线服务器上的人类播放（400万帧，大小与Imagenet相当的400万帧）上刮出的大型嘈杂数据集的行为克隆训练，以及一个较小的高质量专家演示数据集。这种比例是比FPS游戏中的模仿学习的先前工作的数量级。

软件测试活动旨在找到软件产品的可能缺陷，并确保该产品满足其预期要求。一些软件测试接近的方法缺乏自动化或部分自动化，这增加了测试时间和整体软件测试成本。最近，增强学习（RL）已成功地用于复杂的测试任务中，例如游戏测试，回归测试和测试案例优先级，以自动化该过程并提供持续的适应。从业者可以通过从头开始实现RL算法或使用RL框架来使用RL。开发人员已广泛使用这些框架来解决包括软件测试在内的各个领域中的问题。但是，据我们所知，尚无研究从经验上评估RL框架中实用算法的有效性和性能。在本文中，我们凭经验研究了精心选择的RL算法在两个重要的软件测试任务上的应用：在连续集成（CI）和游戏测试的上下文中测试案例的优先级。对于游戏测试任务，我们在简单游戏上进行实验，并使用RL算法探索游戏以检测错误。结果表明，一些选定的RL框架，例如Tensorforce优于文献的最新方法。为了确定测试用例的优先级，我们在CI环境上运行实验，其中使用来自不同框架的RL算法来对测试用例进行排名。我们的结果表明，在某些情况下，预实算算法之间的性能差异很大，激励了进一步的研究。此外，建议对希望选择RL框架的研究人员进行一些基准问题的经验评估，以确保RL算法按预期执行。

With the breakthrough of AlphaGo, deep reinforcement learning becomes a recognized technique for solving sequential decision-making problems. Despite its reputation, data inefficiency caused by its trial and error learning mechanism makes deep reinforcement learning hard to be practical in a wide range of areas. Plenty of methods have been developed for sample efficient deep reinforcement learning, such as environment modeling, experience transfer, and distributed modifications, amongst which, distributed deep reinforcement learning has shown its potential in various applications, such as human-computer gaming, and intelligent transportation. In this paper, we conclude the state of this exciting field, by comparing the classical distributed deep reinforcement learning methods, and studying important components to achieve efficient distributed learning, covering single player single agent distributed deep reinforcement learning to the most complex multiple players multiple agents distributed deep reinforcement learning. Furthermore, we review recently released toolboxes that help to realize distributed deep reinforcement learning without many modifications of their non-distributed versions. By analyzing their strengths and weaknesses, a multi-player multi-agent distributed deep reinforcement learning toolbox is developed and released, which is further validated on Wargame, a complex environment, showing usability of the proposed toolbox for multiple players and multiple agents distributed deep reinforcement learning under complex games. Finally, we try to point out challenges and future trends, hoping this brief review can provide a guide or a spark for researchers who are interested in distributed deep reinforcement learning.

由于部分可观察性，高维视觉感知和延迟奖励，在MINECRAFT等开放世界游戏中的学习理性行为仍然是挑战，以便对加固学习（RL）研究造成挑战性，高维视觉感知和延迟奖励。为了解决这个问题，我们提出了一种具有代表学习和模仿学习的样本有效的等级RL方法，以应对感知和探索。具体来说，我们的方法包括两个层次结构，其中高级控制器学习控制策略来控制选项，低级工作人员学会解决每个子任务。为了提高子任务的学习，我们提出了一种技术组合，包括1）动作感知表示学习，其捕获了行动和表示之间的基础关系，2）基于鉴别者的自模仿学习，以实现有效的探索，以及3）合奏行为克隆一致性筛选政策鲁棒性。广泛的实验表明，Juewu-MC通过大边缘显着提高了样品效率并优于一组基线。值得注意的是，我们赢得了神经脂溢斯矿业锦标赛2021年研究竞赛的冠军，并实现了最高的绩效评分。

Imitation learning techniques aim to mimic human behavior in a given task. An agent (a learning machine) is trained to perform a task from demonstrations by learning a mapping between observations and actions. The idea of teaching by imitation has been around for many years, however, the field is gaining attention recently due to advances in computing and sensing as well as rising demand for intelligent applications. The paradigm of learning by imitation is gaining popularity because it facilitates teaching complex tasks with minimal expert knowledge of the tasks. Generic imitation learning methods could potentially reduce the problem of teaching a task to that of providing demonstrations; without the need for explicit programming or designing reward functions specific to the task. Modern sensors are able to collect and transmit high volumes of data rapidly, and processors with high computational power allow fast processing that maps the sensory data to actions in a timely manner. This opens the door for many potential AI applications that require real-time perception and reaction such as humanoid robots, self-driving vehicles, human computer interaction and computer games to name a few. However, specialized algorithms are needed to effectively and robustly learn models as learning by imitation poses its own set of challenges. In this paper, we survey imitation learning methods and present design options in different steps of the learning process. We introduce a background and motivation for the field as well as highlight challenges specific to the imitation problem. Methods for designing and evaluating imitation learning tasks are categorized and reviewed. Special attention is given to learning methods in robotics and games as these domains are the most popular in the literature and provide a wide array of problems and methodologies. We extensively discuss combining imitation learning approaches using different sources and methods, as well as incorporating other motion learning methods to enhance imitation. We also discuss the potential impact on industry, present major applications and highlight current and future research directions.

Starcraft II（SC2）对强化学习（RL）提出了巨大的挑战，其中主要困难包括巨大的状态空间，不同的动作空间和长期的视野。在这项工作中，我们研究了《星际争霸II》全长游戏的一系列RL技术。我们研究了涉及提取的宏观活动和神经网络的层次结构的层次RL方法。我们研究了课程转移培训程序，并在具有4个GPU和48个CPU线的单台计算机上训练代理。在64x64地图并使用限制性单元上，我们对内置AI的获胜率达到99％。通过课程转移学习算法和战斗模型的混合物，我们在最困难的非作战水平内置AI（7级）中获得了93％的胜利率。在本文的扩展版本中，我们改进了架构，以针对作弊水平训练代理商，并在8级，9级和10级AIS上达到胜利率，为96％，97％和94 ％， 分别。我们的代码在https://github.com/liuruoze/hiernet-sc2上。为了为我们的工作以及研究和开源社区提供基线，我们将其复制了一个缩放版本的Mini-Alphastar（MAS）。 MAS的最新版本为1.07，可以在具有564个动作的原始动作空间上进行培训。它旨在通过使超参数可调节来在单个普通机器上进行训练。然后，我们使用相同的资源将我们的工作与MAS进行比较，并表明我们的方法更有效。迷你α的代码在https://github.com/liuruoze/mini-alphastar上。我们希望我们的研究能够阐明对SC2和其他大型游戏有效增强学习的未来研究。

注入人类知识是加速加强学习（RL）的有效途径。但是，这些方法是缺乏缺陷的。本文介绍了我们发现的抽象前瞻性模型（思想游戏（TG））与转移学习（TL）相结合是有效的方式。我们将星际争霸II作为我们的学习环境。在设计的TG的帮助下，该代理可以在64x64地图上学习99％的速率，在一个商业机器中仅使用1.08小时的1级内置AI。我们还表明TG方法并不像被认为是限制性的。它可以使用粗略设计的TGS，并且在环境变化时也可以很有用。与以前的基于模型的RL相比，我们显示TG更有效。我们还提出了一种TG假设，其赋予TG不同保真度水平的影响。对于具有不等状态和行动空间的真实游戏，我们提出了一种新颖的XFRNET，其中有用性在验证有用性，同时达到欺骗级别-10 AI的90％的赢利。我们认为TG方法可能会在利用人类知识的进一步研究中进一步研究。

In fighting games, individual players of the same skill level often exhibit distinct strategies from one another through their gameplay. Despite this, the majority of AI agents for fighting games have only a single strategy for each "level" of difficulty. To make AI opponents more human-like, we'd ideally like to see multiple different strategies at each level of difficulty, a concept we refer to as "multidimensional" difficulty. In this paper, we introduce a diversity-based deep reinforcement learning approach for generating a set of agents of similar difficulty that utilize diverse strategies. We find this approach outperforms a baseline trained with specialized, human-authored reward functions in both diversity and performance.

In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life. We formulate curiosity as the error in an agent's ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model. Our formulation scales to high-dimensional continuous state spaces like images, bypasses the difficulties of directly predicting pixels, and, critically, ignores the aspects of the environment that cannot affect the agent. The proposed approach is evaluated in two environments: VizDoom and Super Mario Bros. Three broad settings are investigated: 1) sparse extrinsic reward, where curiosity allows for far fewer interactions with the environment to reach the goal; 2) exploration with no extrinsic reward, where curiosity pushes the agent to explore more efficiently; and 3) generalization to unseen scenarios (e.g. new levels of the same game) where the knowledge gained from earlier experience helps the agent explore new places much faster than starting from scratch.

With the development of deep representation learning, the domain of reinforcement learning (RL) has become a powerful learning framework now capable of learning complex policies in high dimensional environments. This review summarises deep reinforcement learning (DRL) algorithms and provides a taxonomy of automated driving tasks where (D)RL methods have been employed, while addressing key computational challenges in real world deployment of autonomous driving agents. It also delineates adjacent domains such as behavior cloning, imitation learning, inverse reinforcement learning that are related but are not classical RL algorithms. The role of simulators in training agents, methods to validate, test and robustify existing solutions in RL are discussed.

自主代理在Atari Games等专业领域取得了长足的进步。但是，他们通常在具有有限和手动构想的目标的孤立环境中学习Tabula Rasa，因此未能跨越各种任务和能力。受到人类如何不断学习和适应开放世界的启发，我们主张建立通才代理的三位一体：1）一个支持多种任务和目标的环境，2）多模式知识的大规模数据库和3个数据库）灵活且可扩展的代理体系结构。我们介绍了MinedoJo，这是一个建立在流行的Minecraft游戏上的新框架，该游戏具有模拟套件，其中包含数千种不同的开放式任务，以及带有Minecraft视频，教程，Wiki页面和论坛讨论的Internet规模知识库。使用Minedojo的数据，我们提出了一种新型的代理学习算法，该算法利用大型预训练的视频语言模型作为学习的奖励功能。我们的代理商能够解决以自由形式的语言指定的各种开放式任务，而无需任何手动设计的密集塑造奖励。我们开源的仿真套件和知识库（https://minedojo.org），以促进研究的研究，以通常具有能力的体现药物的目标。

The ultimate goal of artificial intelligence is to mimic the human brain to perform decision-making and control directly from high-dimensional sensory input. All-optical diffractive neural networks provide a promising solution for realizing artificial intelligence with high-speed and low-power consumption. To date, most of the reported diffractive neural networks focus on single or multiple tasks that do not involve interaction with the environment, such as object recognition and image classification, while the networks that can perform decision-making and control, to our knowledge, have not been developed yet. Here, we propose to use deep reinforcement learning to realize diffractive neural networks that enable imitating the human-level capability of decision-making and control. Such networks allow for finding optimal control policies through interaction with the environment and can be readily realized with the dielectric metasurfaces. The superior performances of these networks are verified by engaging three types of classic games, Tic-Tac-Toe, Super Mario Bros., and Car Racing, and achieving the same or even higher levels comparable to human players. Our work represents a solid step of advancement in diffractive neural networks, which promises a fundamental shift from the target-driven control of a pre-designed state for simple recognition or classification tasks to the high-level sensory capability of artificial intelligence. It may find exciting applications in autonomous driving, intelligent robots, and intelligent manufacturing.

Deep reinforcement learning is poised to revolutionise the field of AI and represents a step towards building autonomous systems with a higher level understanding of the visual world. Currently, deep learning is enabling reinforcement learning to scale to problems that were previously intractable, such as learning to play video games directly from pixels. Deep reinforcement learning algorithms are also applied to robotics, allowing control policies for robots to be learned directly from camera inputs in the real world. In this survey, we begin with an introduction to the general field of reinforcement learning, then progress to the main streams of value-based and policybased methods. Our survey will cover central algorithms in deep reinforcement learning, including the deep Q-network, trust region policy optimisation, and asynchronous advantage actor-critic. In parallel, we highlight the unique advantages of deep neural networks, focusing on visual understanding via reinforcement learning. To conclude, we describe several current areas of research within the field.

本文通过将影响建模的任务视为强化学习（RL）过程，引入了范式转变。根据拟议的范式，RL代理通过尝试通过其环境（即背景）来最大化一组奖励（即行为和情感模式）来学习政策（即情感互动）。我们的假设是，RL是交织的有效范式影响引起和与行为和情感示威的表现。重要的是，我们对达马西奥的躯体标记假设的第二个假设建设是，情绪可以成为决策的促进者。我们通过训练Go-Blend Agents来对人类的唤醒和行为进行模型来检验赛车游戏中的假设； Go-Blend是Go-explore算法的修改版本，该版本最近在硬探索任务中展示了最高性能。我们首先改变了基于唤醒的奖励功能，并观察可以根据指定的奖励有效地显示情感和行为模式调色板的代理。然后，我们使用基于唤醒的状态选择机制来偏向Go-Blend探索的策略。我们的发现表明，Go-Blend不仅是有效的影响建模范式，而且更重要的是，情感驱动的RL改善了探索并产生更高的性能剂，从而验证了Damasio在游戏领域中的假设。

我们提出了一种新的四管齐下的方法，在文献中首次建立消防员的情境意识。我们构建了一系列深度学习框架，彼此之叠，以提高消防员在紧急首次响应设置中进行的救援任务的安全性，效率和成功完成。首先，我们使用深度卷积神经网络（CNN）系统，以实时地分类和识别来自热图像的感兴趣对象。接下来，我们将此CNN框架扩展了对象检测，跟踪，分割与掩码RCNN框架，以及具有多模级自然语言处理（NLP）框架的场景描述。第三，我们建立了一个深入的Q学习的代理，免受压力引起的迷失方向和焦虑，能够根据现场消防环境中观察和存储的事实来制定明确的导航决策。最后，我们使用了一种低计算无监督的学习技术，称为张量分解，在实时对异常检测进行有意义的特征提取。通过这些临时深度学习结构，我们建立了人工智能系统的骨干，用于消防员的情境意识。要将设计的系统带入消防员的使用，我们设计了一种物理结构，其中处理后的结果被用作创建增强现实的投入，这是一个能够建议他们所在地的消防员和周围的关键特征，这对救援操作至关重要在手头，以及路径规划功能，充当虚拟指南，以帮助迷彩的第一个响应者恢复安全。当组合时，这四种方法呈现了一种新颖的信息理解，转移和综合方法，这可能会大大提高消防员响应和功效，并降低寿命损失。

2048 is a single-player stochastic puzzle game. This intriguing and addictive game has been popular worldwide and has attracted researchers to develop game-playing programs. Due to its simplicity and complexity, 2048 has become an interesting and challenging platform for evaluating the effectiveness of machine learning methods. This dissertation conducts comprehensive research on reinforcement learning and computer game algorithms for 2048. First, this dissertation proposes optimistic temporal difference learning, which significantly improves the quality of learning by employing optimistic initialization to encourage exploration for 2048. Furthermore, based on this approach, a state-of-the-art program for 2048 is developed, which achieves the highest performance among all learning-based programs, namely an average score of 625377 points and a rate of 72% for reaching 32768-tiles. Second, this dissertation investigates several techniques related to 2048, including the n-tuple network ensemble learning, Monte Carlo tree search, and deep reinforcement learning. These techniques are promising for further improving the performance of the current state-of-the-art program. Finally, this dissertation discusses pedagogical applications related to 2048 by proposing course designs and summarizing the teaching experience. The proposed course designs use 2048-like games as materials for beginners to learn reinforcement learning and computer game algorithms. The courses have been successfully applied to graduate-level students and received well by student feedback.