除了独奏游戏外，棋盘游戏至少需要其他玩家才能玩。因此，当对手失踪时，我们创建了人工智能（AI）代理商来对抗我们。这些AI代理是通过多种方式创建的，但是这些代理的一个挑战是，与我们相比，代理可以具有较高的能力。在这项工作中，我们描述了如何创建玩棋盘游戏的较弱的AI代理。我们使用Tic-Tac-toe，九名成员的莫里斯和曼卡拉，我们的技术使用了增强学习模型，代理商使用Q学习算法来学习这些游戏。我们展示了这些代理商如何学会完美地玩棋盘游戏，然后我们描述了制作这些代理商较弱版本的方法。最后，我们提供了比较AI代理的方法。

本文涵盖了基于N组的加强学习（RL）算法。我们为TD-，Sarsa-and Q-Learning提供了新的算法，这些算法在各种游戏中无缝工作，任意数量的玩家。这是通过采用以球员为中心的视图来实现的，其中每个玩家将他/她的奖励传播到以前的轮次。我们将称为最终适应RL（Farl）的新元素添加到所有这些算法。我们的主要贡献是，Farl是一项最重要的成分，可以在各种游戏中以可爱的球员为中心的观点实现成功。我们向七个棋盘游戏报告结果1,2和3名球员，包括奥赛罗，Connectfour和Hex。在大多数情况下，发现Farl非常重要，无法学习近乎完美的竞争策略。所有算法都在GitHub上的GBG框架中提供。

Tic Tac Toe is amongst the most well-known games. It has already been shown that it is a biased game, giving more chances to win for the first player leaving only a draw or a loss as possibilities for the opponent, assuming both the players play optimally. Thus on average majority of the games played result in a draw. The majority of the latest research on how to solve a tic tac toe board state employs strategies such as Genetic Algorithms, Neural Networks, Co-Evolution, and Evolutionary Programming. But these approaches deal with a trivial board state of 3X3 and very little research has been done for a generalized algorithm to solve 4X4,5X5,6X6 and many higher states. Even though an algorithm exists which is Min-Max but it takes a lot of time in coming up with an ideal move due to its recursive nature of implementation. A Sample has been created on this link \url{https://bk-tic-tac-toe.herokuapp.com/} to prove this fact. This is the main problem that this study is aimed at solving i.e providing a generalized algorithm(Approximate method, Learning-Based) for higher board states of tic tac toe to make precise moves in a short period. Also, the code changes needed to accommodate higher board states will be nominal. The idea is to pose the tic tac toe game as a well-posed learning problem. The study and its results are promising, giving a high win to draw ratio with each epoch of training. This study could also be encouraging for other researchers to apply the same algorithm to other similar board games like Minesweeper, Chess, and GO for finding efficient strategies and comparing the results.

传统的增强学习（RL）环境通常在培训和测试阶段都相同。因此，当前的RL方法在很大程度上不能推广到概念上相似但与已训练的方法不同的测试环境，我们将其称为新型测试环境。为了将RL研究推向可以推广到新的测试环境的算法，我们介绍了砖Tic-TAC-TOE（BTTT）测试床，其中在测试环境中的砖位与训练环境中的砖位不同。使用BTTT环境上的圆形锦标赛，我们表明传统的RL国家搜索方法，例如Monte Carlo Tree Search（MCTS）和Minimax，比Alphazero更广泛地对新型测试环境更具概括性。令人惊讶的是，Alphazero已被证明可以在GO，Chess和Shogi等环境中实现超人的性能，这可能会导致人们认为它在新颖的测试环境中的性能很好。我们的结果表明，BTTT虽然很简单，但足够丰富，可以探索Alphazero的普遍性。我们发现，仅增加MCT的lookahead迭代是不足以使Alphazero推广到一些新型的测试环境。相反，增加各种培训环境有助于逐步改善所有可能的起始砖配置中的普遍性。

Monte Carlo Tree Search (MCTS) is a recently proposed search method that combines the precision of tree search with the generality of random sampling. It has received considerable interest due to its spectacular success in the difficult problem of computer Go, but has also proved beneficial in a range of other domains. This paper is a survey of the literature to date, intended to provide a snapshot of the state of the art after the first five years of MCTS research. We outline the core algorithm's derivation, impart some structure on the many variations and enhancements that have been proposed, and summarise the results from the key game and non-game domains to which MCTS methods have been applied. A number of open research questions indicate that the field is ripe for future work.

最近，开创性算法Alphago和Alphazero在游戏学习和深入的强化学习方面开始了一个新时代。尽管Alphago和Alphazero的成就 - 在超级人类层面上玩的GO和其他复杂游戏 - 确实令人印象深刻，但这些架构的缺点是它们需要高度的计算资源。许多研究人员正在寻找类似于alphazero但计算需求较低的方法，因此更容易重现。在本文中，我们选择了Alphazero的重要元素 - 蒙特卡洛树搜索（MCTS）计划阶段 - 并将其与时间差异（TD）学习剂相结合。我们首次将MCT包裹在TD N培训网络上，我们仅在测试时间使用此包装来创建多功能代理，从而使计算需求保持较低。我们将这种新体系结构应用于多个复杂游戏（Othello，Connectfour，Rubik的Cube），并显示了这种受alphazero启发的MCTS包装器所获得的优势。特别是，我们提出的结果是，该代理是第一个在标准硬件（无GPU或TPU）上训练的代理商，击败非常强大的Othello计划EDAX到包括7级（大多数其他学习中的学习中，从而只能失败EDAX至2级）。

深度加强学习（RL）的最新进展导致许多2人零和游戏中的相当大的进展，如去，扑克和星际争霸。这种游戏的纯粹对抗性质允许概念上简单地应用R1方法。然而，现实世界的设置是许多代理商，代理交互是复杂的共同利益和竞争方面的混合物。我们认为外交，一个旨在突出由多种代理交互导致的困境的7人棋盘游戏。它还具有大型组合动作空间和同时移动，这对RL算法具有具有挑战性。我们提出了一个简单但有效的近似最佳响应操作员，旨在处理大型组合动作空间并同时移动。我们还介绍了一系列近似虚构游戏的政策迭代方法。通过这些方法，我们成功地将RL申请到外交：我们认为我们的代理商令人信服地令人信服地表明，游戏理论均衡分析表明新过程产生了一致的改进。

我们介绍了DeepNash，这是一种能够学习从头开始播放不完美的信息游戏策略的自主代理，直到人类的专家级别。 Stratego是人工智能（AI）尚未掌握的少数标志性棋盘游戏之一。这个受欢迎的游戏具有$ 10^{535} $节点的巨大游戏树，即，$ 10^{175} $倍的$倍于GO。它具有在不完美的信息下需要决策的其他复杂性，类似于德克萨斯州Hold'em扑克，该扑克的游戏树较小（以$ 10^{164} $节点为单位）。 Stratego中的决策是在许多离散的动作上做出的，而动作与结果之间没有明显的联系。情节很长，在球员获胜之前经常有数百次动作，而Stratego中的情况则不能像扑克中那样轻松地分解成管理大小的子问题。由于这些原因，Stratego几十年来一直是AI领域的巨大挑战，现有的AI方法几乎没有达到业余比赛水平。 Deepnash使用游戏理论，无模型的深钢筋学习方法，而无需搜索，该方法学会通过自我播放来掌握Stratego。 DeepNash的关键组成部分的正则化NASH Dynamics（R-NAD）算法通过直接修改基础多项式学习动力学来收敛到近似NASH平衡，而不是围绕它“循环”。 Deepnash在Stratego中击败了现有的最先进的AI方法，并在Gravon Games平台上获得了年度（2022年）和历史前3名，并与人类专家竞争。

强化学习最近已成为解决棋盘游戏领域中复杂问题的非常强大的工具，其中通常需要代理来根据其自身的经验和收到的奖励来学习复杂的策略和移动。尽管RL胜过用于玩简单视频游戏和受欢迎的棋盘游戏的现有最新方法，但它尚未证明其在古代游戏中的能力。在这里，我们解决了一个这样的问题，在该问题中，我们使用不同的方法来训练代理商，即蒙特卡洛，Qlearning和Hir Hir Hight Sarsa能够学习最佳政策来发挥战略性的UR皇家游戏。我们游戏的状态空间很复杂，但是我们的代理商在玩游戏和学习重要的战略动作方面表现出令人鼓舞的结果。尽管很难得出结论，当接受有限的资源培训时，算法总体上的表现更好，但预计SARSA在学习最快的学习方面表现出了令人鼓舞的结果。

通过少数院校拥有不懈的努力，最近在设计超人AIS中的重大进展，在无限制的德克萨斯州举行（NLTH）中，是大规模不完美信息游戏研究的主要测试平台。然而，新研究人员对新的研究人员来说仍然有挑战性，因为没有与现有方法相比，这严重阻碍了本研究区域的进一步发展。在这项工作中，我们展示了OpenHoldem，一个用于使用NLTH的大规模不完美信息游戏研究的集成工具包。 OpenHoldem对这一研究方向进行了三个主要贡献：1）用于彻底评估不同NLTH AIS，2）用于NLTH AI的四个公开可用的强大基线的标准化评估方案，以及3）一个在线测试平台，公众易于使用API nlth ai评估。我们在Holdem.Ia.ac.CN发布了OpenHoldem，希望它有助于进一步研究该领域的未解决的理论和计算问题，并培养对手建模和人机互动学习等关键研究问题。

2048 is a single-player stochastic puzzle game. This intriguing and addictive game has been popular worldwide and has attracted researchers to develop game-playing programs. Due to its simplicity and complexity, 2048 has become an interesting and challenging platform for evaluating the effectiveness of machine learning methods. This dissertation conducts comprehensive research on reinforcement learning and computer game algorithms for 2048. First, this dissertation proposes optimistic temporal difference learning, which significantly improves the quality of learning by employing optimistic initialization to encourage exploration for 2048. Furthermore, based on this approach, a state-of-the-art program for 2048 is developed, which achieves the highest performance among all learning-based programs, namely an average score of 625377 points and a rate of 72% for reaching 32768-tiles. Second, this dissertation investigates several techniques related to 2048, including the n-tuple network ensemble learning, Monte Carlo tree search, and deep reinforcement learning. These techniques are promising for further improving the performance of the current state-of-the-art program. Finally, this dissertation discusses pedagogical applications related to 2048 by proposing course designs and summarizing the teaching experience. The proposed course designs use 2048-like games as materials for beginners to learn reinforcement learning and computer game algorithms. The courses have been successfully applied to graduate-level students and received well by student feedback.

游戏历史悠久的历史悠久地作为人工智能进步的基准。最近，使用搜索和学习的方法在一系列完美的信息游戏中表现出强烈的表现，并且使用游戏理论推理和学习的方法对特定的不完美信息扑克变体表示了很强的性能。我们介绍游戏玩家，一个通用算法，统一以前的方法，结合导游搜索，自助学习和游戏理论推理。游戏播放器是实现大型完美和不完美信息游戏中强大实证性能的第一个算法 - 这是一项真正的任意环境算法的重要一步。我们证明了游戏玩家是声音，融合到完美的游戏，因为可用的计算时间和近似容量增加。游戏播放器在国际象棋上达到了强大的表现，然后击败了最强大的公开可用的代理商，在头上没有限制德克萨斯州扑克（Slumbot），击败了苏格兰院子的最先进的代理人，这是一个不完美的信息游戏，说明了引导搜索，学习和游戏理论推理的价值。

This paper surveys the eld of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the eld and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but di ers considerably in the details and in the use of the word \reinforcement." The paper discusses central issues of reinforcement learning, including trading o exploration and exploitation, establishing the foundations of the eld via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.

In the Markov decision process (MDP) formalization of reinforcement learning, a single adaptive agent interacts with an environment defined by a probabilistic transition function. In this solipsistic view, secondary agents can only be part of the environment and are therefore fixed in their behavior. The framework of Markov games allows us to widen this view to include multiple adaptive agents with interacting or competing goals. This paper considers a step in this direction in which exactly two agents with diametrically opposed goals share an environment. It describes a Q-learning-like algorithm for finding optimal policies and demonstrates its application to a simple two-player game in which the optimal policy is probabilistic.

在过去的十年中，多智能经纪人强化学习（Marl）已经有了重大进展，但仍存在许多挑战，例如高样本复杂性和慢趋同稳定的政策，在广泛的部署之前需要克服，这是可能的。然而，在实践中，许多现实世界的环境已经部署了用于生成策略的次优或启发式方法。一个有趣的问题是如何最好地使用这些方法作为顾问，以帮助改善多代理领域的加强学习。在本文中，我们提供了一个原则的框架，用于将动作建议纳入多代理设置中的在线次优顾问。我们描述了在非传记通用随机游戏环境中提供多种智能强化代理（海军上将）的问题，并提出了两种新的基于Q学习的算法：海军上将决策（海军DM）和海军上将 - 顾问评估（Admiral-AE） ，这使我们能够通过适当地纳入顾问（Admiral-DM）的建议来改善学习，并评估顾问（Admiral-AE）的有效性。我们从理论上分析了算法，并在一般加上随机游戏中提供了关于他们学习的定点保证。此外，广泛的实验说明了这些算法：可以在各种环境中使用，具有对其他相关基线的有利相比的性能，可以扩展到大状态行动空间，并且对来自顾问的不良建议具有稳健性。

蒙特卡洛树搜索（MCT）是设计游戏机器人或解决顺序决策问题的强大方法。该方法依赖于平衡探索和开发的智能树搜索。MCT以模拟的形式进行随机抽样，并存储动作的统计数据，以在每个随后的迭代中做出更有教育的选择。然而，该方法已成为组合游戏的最新技术，但是，在更复杂的游戏（例如那些具有较高的分支因素或实时系列的游戏）以及各种实用领域（例如，运输，日程安排或安全性）有效的MCT应用程序通常需要其与问题有关的修改或与其他技术集成。这种特定领域的修改和混合方法是本调查的主要重点。最后一项主要的MCT调查已于2012年发布。自发布以来出现的贡献特别感兴趣。

Modern video games are becoming richer and more complex in terms of game mechanics. This complexity allows for the emergence of a wide variety of ways to play the game across the players. From the point of view of the game designer, this means that one needs to anticipate a lot of different ways the game could be played. Machine Learning (ML) could help address this issue. More precisely, Reinforcement Learning is a promising answer to the need of automating video game testing. In this paper we present a video game environment which lets us define multiple play-styles. We then introduce CARI: a Configurable Agent with Reward as Input. An agent able to simulate a wide continuum range of play-styles. It is not constrained to extreme archetypal behaviors like current methods using reward shaping. In addition it achieves this through a single training loop, instead of the usual one loop per play-style. We compare this novel training approach with the more classic reward shaping approach and conclude that CARI can also outperform the baseline on archetypes generation. This novel agent could be used to investigate behaviors and balancing during the production of a video game with a realistic amount of training time.

我们调查攻击者的效果如何，当它只从受害者的行为中学习时，没有受害者的奖励。在这项工作中，当受害者的动机未知时，我们被攻击者想要行事的情景。我们认为一个启发式方法可以使用攻击者是最大化受害者政策的熵。政策通常不会被滥用，这意味着它可以通过被动地观察受害者来提取。我们以奖励无源勘探算法的形式提供这样的策略，可以在勘探阶段最大化攻击者的熵，然后在规划阶段最大化受害者的经验熵。在我们的实验中，受害者代理商通过政策熵最大化而颠覆，暗示攻击者可能无法访问受害者的奖励成功。因此，仅基于观察行为的无奖励攻击表明，即使受害者的奖励信息受到保护，攻击者的可行性也在不了解受害者的动机。

In recent years, Monte Carlo tree search (MCTS) has achieved widespread adoption within the game community. Its use in conjunction with deep reinforcement learning has produced success stories in many applications. While these approaches have been implemented in various games, from simple board games to more complicated video games such as StarCraft, the use of deep neural networks requires a substantial training period. In this work, we explore on-line adaptivity in MCTS without requiring pre-training. We present MCTS-TD, an adaptive MCTS algorithm improved with temporal difference learning. We demonstrate our new approach on the game miniXCOM, a simplified version of XCOM, a popular commercial franchise consisting of several turn-based tactical games, and show how adaptivity in MCTS-TD allows for improved performances against opponents.

In fighting games, individual players of the same skill level often exhibit distinct strategies from one another through their gameplay. Despite this, the majority of AI agents for fighting games have only a single strategy for each "level" of difficulty. To make AI opponents more human-like, we'd ideally like to see multiple different strategies at each level of difficulty, a concept we refer to as "multidimensional" difficulty. In this paper, we introduce a diversity-based deep reinforcement learning approach for generating a set of agents of similar difficulty that utilize diverse strategies. We find this approach outperforms a baseline trained with specialized, human-authored reward functions in both diversity and performance.