In this paper we address the solution of the popular Wordle puzzle, using new reinforcement learning methods, which apply more generally to adaptive control of dynamic systems and to classes of Partially Observable Markov Decision Process (POMDP) problems. These methods are based on approximation in value space and the rollout approach, admit a straightforward implementation, and provide improved performance over various heuristic approaches. For the Wordle puzzle, they yield on-line solution strategies that are very close to optimal at relatively modest computational cost. Our methods are viable for more complex versions of Wordle and related search problems, for which an optimal strategy would be impossible to compute. They are also applicable to a wide range of adaptive sequential decision problems that involve an unknown or frequently changing environment whose parameters are estimated on-line.

We provide a unifying approximate dynamic programming framework that applies to a broad variety of problems involving sequential estimation. We consider first the construction of surrogate cost functions for the purposes of optimization, and we focus on the special case of Bayesian optimization, using the rollout algorithm and some of its variations. We then discuss the more general case of sequential estimation of a random vector using optimal measurement selection, and its application to problems of stochastic and adaptive control. We finally consider related search and sequential decoding problems, and a rollout algorithm for the approximate solution of the Wordle and Mastermind puzzles, recently developed in the paper [BBB22].

This paper surveys the eld of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the eld and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but di ers considerably in the details and in the use of the word \reinforcement." The paper discusses central issues of reinforcement learning, including trading o exploration and exploitation, establishing the foundations of the eld via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.

蒙特卡洛树搜索（MCT）是设计游戏机器人或解决顺序决策问题的强大方法。该方法依赖于平衡探索和开发的智能树搜索。MCT以模拟的形式进行随机抽样，并存储动作的统计数据，以在每个随后的迭代中做出更有教育的选择。然而，该方法已成为组合游戏的最新技术，但是，在更复杂的游戏（例如那些具有较高的分支因素或实时系列的游戏）以及各种实用领域（例如，运输，日程安排或安全性）有效的MCT应用程序通常需要其与问题有关的修改或与其他技术集成。这种特定领域的修改和混合方法是本调查的主要重点。最后一项主要的MCT调查已于2012年发布。自发布以来出现的贡献特别感兴趣。

Monte Carlo Tree Search (MCTS) is a recently proposed search method that combines the precision of tree search with the generality of random sampling. It has received considerable interest due to its spectacular success in the difficult problem of computer Go, but has also proved beneficial in a range of other domains. This paper is a survey of the literature to date, intended to provide a snapshot of the state of the art after the first five years of MCTS research. We outline the core algorithm's derivation, impart some structure on the many variations and enhancements that have been proposed, and summarise the results from the key game and non-game domains to which MCTS methods have been applied. A number of open research questions indicate that the field is ripe for future work.

由于数据量增加，金融业的快速变化已经彻底改变了数据处理和数据分析的技术，并带来了新的理论和计算挑战。与古典随机控制理论和解决财务决策问题的其他分析方法相比，解决模型假设的财务决策问题，强化学习（RL）的新发展能够充分利用具有更少模型假设的大量财务数据并改善复杂的金融环境中的决策。该调查纸目的旨在审查最近的资金途径的发展和使用RL方法。我们介绍了马尔可夫决策过程，这是许多常用的RL方法的设置。然后引入各种算法，重点介绍不需要任何模型假设的基于价值和基于策略的方法。连接是用神经网络进行的，以扩展框架以包含深的RL算法。我们的调查通过讨论了这些RL算法在金融中各种决策问题中的应用，包括最佳执行，投资组合优化，期权定价和对冲，市场制作，智能订单路由和Robo-Awaring。

最近的文学建立了神经网络可以代表供应链和物流中一系列随机动态模型的良好政策。我们提出了一种结合方差减少技术的新算法，以克服通常在文献中使用的算法的限制，以学习此类神经网络策略。对于古典丢失的销售库存模型，该算法了解到使用无模型算法学习的神经网络策略，同时始于最优于数量级的最佳启发式基准。该算法是一个有趣的候选者，适用于供应链和物流中的其他随机动态问题，因为其开发中的思想是通用的。

信息性测量是获取有关未知状态信息的最有效方法。我们给出了一般目的动态编程算法的第一原理推导，通过顺序地最大化可能的测量结果的熵来返回一系列信息测量。该算法可以由自主代理或机器人使用，以确定最佳测量的位置，规划对应于信息序列的最佳信息序列的路径。该算法适用于具有连续或离散的状态和控制，以及随机或确定性的代理动态;包括马尔可夫决策过程。最近的近似动态规划和强化学习的结果，包括卷展栏和蒙特卡罗树搜索等在线近似，允许代理或机器人实时解决测量任务。由此产生的近最佳溶液包括非近视路径和测量序列，其通常可以优于超过，有时基本上使用的贪婪启发式，例如最大化每个测量结果的熵。这是针对全球搜索问题的说明，其中发现使用扩展本地搜索的在线规划来减少搜索中的测量数。

具有很多玩家的非合作和合作游戏具有许多应用程序，但是当玩家数量增加时，通常仍然很棘手。由Lasry和Lions以及Huang，Caines和Malham \'E引入的，平均野外运动会（MFGS）依靠平均场外近似值，以使玩家数量可以成长为无穷大。解决这些游戏的传统方法通常依赖于以完全了解模型的了解来求解部分或随机微分方程。最近，增强学习（RL）似乎有望解决复杂问题。通过组合MFGS和RL，我们希望在人口规模和环境复杂性方面能够大规模解决游戏。在这项调查中，我们回顾了有关学习MFG中NASH均衡的最新文献。我们首先确定最常见的设置（静态，固定和进化）。然后，我们为经典迭代方法（基于最佳响应计算或策略评估）提供了一个通用框架，以确切的方式解决MFG。在这些算法和与马尔可夫决策过程的联系的基础上，我们解释了如何使用RL以无模型的方式学习MFG解决方案。最后，我们在基准问题上介绍了数值插图，并以某些视角得出结论。

我们考虑了路径计划和网络传输中的一些经典优化问题，并引入了基于拍卖的新算法，以实现其最佳和次优的解决方案。这些算法是基于与对象和随之而来的市场平衡的竞争竞标相关的数学思想，这些算法是拍卖过程的基础。但是，我们算法的起点是不同的，即在有向图中加权和未加权的路径构造，而不是将人分配给对象。新算法比现有方法具有多种潜在的优势：在某些重要情况下，它们在经验上更快，例如Max-Flow，它们非常适合在线重新融合，并且可以适应分布式的异步操作。此外，它们允许任意初始价格，而无需互补的懈怠限制，因此非常适合利用加强学习方法，这些方法将使用数据使用离线培训以及实时操作期间的在线培训。新算法还可以在涉及近似的增强学习环境中找到使用，例如Multistep LookAhead和Tree搜索方案和/或推出算法。

在这项工作中，我们证明了如何通过预期最大化算法来处理随机和风险敏感的最佳控制问题。我们展示了这种处理如何实现为两个独立的迭代程序，每个迭代程序都会产生一个独特但密切相关的密度函数序列。我们激励将这些密度解释为信念，将ERGO作为确定性最佳政策的概率代理。更正式的两个固定点迭代方案是根据代表可靠的期望最大化方法的确定性最佳策略一致的固定点得出的。我们倾向于指出我们的结果与控制范式密切相关。在此推理中的控制是指旨在将最佳控制作为概率推断的实例的方法集合。尽管所说的范式已经导致了几种强大的强化学习算法的发展，但基本问题陈述通常是由目的论论证引入的。我们认为，目前的结果表明，较早的控制作为推理框架实际上将一个步骤与所提出的迭代程序中的一个步骤隔离。在任何情况下，本疗法都为他们提供了有效性的义学论点。通过暴露基本的技术机制，我们旨在为控制作为一种推断为取代当前最佳控制范式的框架的普遍接受。为了激发提出的治疗的普遍相关性，我们在勾勒出未来算法开发的大纲之前，进一步讨论了与路径积分控制和其他研究领域的相似之处。

策略梯度方法适用于复杂的，不理解的，通过对参数化的策略进行随机梯度下降来控制问题。不幸的是，即使对于可以通过标准动态编程技术解决的简单控制问题，策略梯度算法也会面临非凸优化问题，并且被广泛理解为仅收敛到固定点。这项工作确定了结构属性 - 通过几个经典控制问题共享 - 确保策略梯度目标函数尽管是非凸面，但没有次优的固定点。当这些条件得到加强时，该目标满足了产生收敛速率的Polyak-lojasiewicz（梯度优势）条件。当其中一些条件放松时，我们还可以在任何固定点的最佳差距上提供界限。

在桥梁到海上平台和风力涡轮机的公民和海上工程系统必须有效地管理，因为它们在其运行寿命中暴露于劣化机制，例如疲劳或腐蚀。确定最佳检查和维护政策要求在不确定性下解决复杂的连续决策问题，主要目的是有效地控制与结构失败相关的风险。解决这种复杂性，基于风险的检查计划方法，通常由动态贝叶斯网络支持，评估一组预定义的启发式决策规则，以合理简化了决策问题。然而，所产生的政策可能受到决策规则定义中考虑的有限空间的损害。避免这种限制，部分观察到的马尔可夫决策过程（POMDPS）在不确定的动作结果和观察下提供了用于随机最佳控制的原则性的数学方法，其中作为整个动态更新的状态概率分布的函数规定了最佳动作。在本文中，我们将动态贝叶斯网络与POMDPS结合在联合框架中，以获得最佳检查和维护计划，我们提供了在结构可靠性背景下开发无限和有限地平线POMDP的配方。所提出的方法是对结构部件进行疲劳劣化的情况的情况下实施和测试，证明了基于最先进的POMDP求解器的能力，用于解决潜在的规划优化问题。在数值实验中，彻底比较了POMDP和基于启发式的策略，并且结果表明POMDP与对应于传统问题设置相比，POMDP达到了大幅降低的成本。

Batch reinforcement learning is a subfield of dynamic programming-based reinforcement learning. Originally defined as the task of learning the best possible policy from a fixed set of a priori-known transition samples, the (batch) algorithms developed in this field can be easily adapted to the classical online case, where the agent interacts with the environment while learning. Due to the efficient use of collected data and the stability of the learning process, this research area has attracted a lot of attention recently. In this chapter, we introduce the basic principles and the theory behind batch reinforcement learning, describe the most important algorithms, exemplarily discuss ongoing research within this field, and briefly survey real-world applications of batch reinforcement learning.

我们探索了一个新的强盗实验模型，其中潜在的非组织序列会影响武器的性能。上下文 - 统一算法可能会混淆，而那些执行正确的推理面部信息延迟的算法。我们的主要见解是，我们称之为Deconfounst Thompson采样的算法在适应性和健壮性之间取得了微妙的平衡。它的适应性在易于固定实例中带来了最佳效率，但是在硬性非平稳性方面显示出令人惊讶的弹性，这会导致其他自适应算法失败。

Reinforcement learning (RL) gained considerable attention by creating decision-making agents that maximize rewards received from fully observable environments. However, many real-world problems are partially or noisily observable by nature, where agents do not receive the true and complete state of the environment. Such problems are formulated as partially observable Markov decision processes (POMDPs). Some studies applied RL to POMDPs by recalling previous decisions and observations or inferring the true state of the environment from received observations. Nevertheless, aggregating observations and decisions over time is impractical for environments with high-dimensional continuous state and action spaces. Moreover, so-called inference-based RL approaches require large number of samples to perform well since agents eschew uncertainty in the inferred state for the decision-making. Active inference is a framework that is naturally formulated in POMDPs and directs agents to select decisions by minimising expected free energy (EFE). This supplies reward-maximising (exploitative) behaviour in RL, with an information-seeking (exploratory) behaviour. Despite this exploratory behaviour of active inference, its usage is limited to discrete state and action spaces due to the computational difficulty of the EFE. We propose a unified principle for joint information-seeking and reward maximization that clarifies a theoretical connection between active inference and RL, unifies active inference and RL, and overcomes their aforementioned limitations. Our findings are supported by strong theoretical analysis. The proposed framework's superior exploration property is also validated by experimental results on partial observable tasks with high-dimensional continuous state and action spaces. Moreover, the results show that our model solves reward-free problems, making task reward design optional.

This paper surveys the recent attempts, both from the machine learning and operations research communities, at leveraging machine learning to solve combinatorial optimization problems. Given the hard nature of these problems, state-of-the-art algorithms rely on handcrafted heuristics for making decisions that are otherwise too expensive to compute or mathematically not well defined. Thus, machine learning looks like a natural candidate to make such decisions in a more principled and optimized way. We advocate for pushing further the integration of machine learning and combinatorial optimization and detail a methodology to do so. A main point of the paper is seeing generic optimization problems as data points and inquiring what is the relevant distribution of problems to use for learning on a given task.

The cross-entropy (CE) method is a new generic approach to combinatorial and multi-extremal optimization and rare event simulation. The purpose of this tutorial is to give a gentle introduction to the CE method. We present the CE methodology, the basic algorithm and its modifications, and discuss applications in combinatorial optimization and machine learning.

Uncertainty is prevalent in engineering design, statistical learning, and decision making broadly. Due to inherent risk-averseness and ambiguity about assumptions, it is common to address uncertainty by formulating and solving conservative optimization models expressed using measure of risk and related concepts. We survey the rapid development of risk measures over the last quarter century. From its beginning in financial engineering, we recount their spread to nearly all areas of engineering and applied mathematics. Solidly rooted in convex analysis, risk measures furnish a general framework for handling uncertainty with significant computational and theoretical advantages. We describe the key facts, list several concrete algorithms, and provide an extensive list of references for further reading. The survey recalls connections with utility theory and distributionally robust optimization, points to emerging applications areas such as fair machine learning, and defines measures of reliability.

This paper presents a tutorial introduction to the use of variational methods for inference and learning in graphical models (Bayesian networks and Markov random fields). We present a number of examples of graphical models, including the QMR-DT database, the sigmoid belief network, the Boltzmann machine, and several variants of hidden Markov models, in which it is infeasible to run exact inference algorithms. We then introduce variational methods, which exploit laws of large numbers to transform the original graphical model into a simplified graphical model in which inference is efficient. Inference in the simpified model provides bounds on probabilities of interest in the original model. We describe a general framework for generating variational transformations based on convex duality. Finally we return to the examples and demonstrate how variational algorithms can be formulated in each case.