在桥梁到海上平台和风力涡轮机的公民和海上工程系统必须有效地管理,因为它们在其运行寿命中暴露于劣化机制,例如疲劳或腐蚀。确定最佳检查和维护政策要求在不确定性下解决复杂的连续决策问题,主要目的是有效地控制与结构失败相关的风险。解决这种复杂性,基于风险的检查计划方法,通常由动态贝叶斯网络支持,评估一组预定义的启发式决策规则,以合理简化了决策问题。然而,所产生的政策可能受到决策规则定义中考虑的有限空间的损害。避免这种限制,部分观察到的马尔可夫决策过程(POMDPS)在不确定的动作结果和观察下提供了用于随机最佳控制的原则性的数学方法,其中作为整个动态更新的状态概率分布的函数规定了最佳动作。在本文中,我们将动态贝叶斯网络与POMDPS结合在联合框架中,以获得最佳检查和维护计划,我们提供了在结构可靠性背景下开发无限和有限地平线POMDP的配方。所提出的方法是对结构部件进行疲劳劣化的情况的情况下实施和测试,证明了基于最先进的POMDP求解器的能力,用于解决潜在的规划优化问题。在数值实验中,彻底比较了POMDP和基于启发式的策略,并且结果表明POMDP与对应于传统问题设置相比,POMDP达到了大幅降低的成本。
translated by 谷歌翻译
在现代环境和社会问题的背景下,人们对能够识别土木工程系统的管理策略的方法的需求越来越大,最大程度地降低了结构性故障风险,同时最好计划检查和维护(I&M)流程。由于与联合系统级状态描述下的全局优化方法相关的计算复杂性,大多数可用方法将I&M决策问题简化为组件级别。在本文中,我们提出了一个有效的算法框架,用于在暴露于恶化环境的工程系统下进行推理和决策制定,从而直接在系统级别提供最佳的管理策略。在我们的方法中,决策问题被提出为部分可观察到的马尔可夫决策过程,其动态是在贝叶斯网络条件结构中编码的。该方法可以通过高斯层次结构和动态贝叶斯网络在组件之间平等或一般,不平等的恶化相关性下处理环境。在政策优化方面,我们采用了深层分散的多代理参与者 - 批评(DDMAC)强化学习方法,其中政策由批评家网络指导的参与者神经网络近似。通过在模拟环境中包括劣化依赖性,并通过在系统级别制定成本模型,DDMAC策略本质上考虑了基本系统效应。通过对疲劳恶化下的9分和钢架进行的数值实验证明了这一点。结果表明,与最先进的启发式方法相比,DDMAC政策可提供可观的好处。 DDMAC策略对系统效应的固有考虑也可以根据学习的政策来解释。
translated by 谷歌翻译
Structural Health Monitoring (SHM) describes a process for inferring quantifiable metrics of structural condition, which can serve as input to support decisions on the operation and maintenance of infrastructure assets. Given the long lifespan of critical structures, this problem can be cast as a sequential decision making problem over prescribed horizons. Partially Observable Markov Decision Processes (POMDPs) offer a formal framework to solve the underlying optimal planning task. However, two issues can undermine the POMDP solutions. Firstly, the need for a model that can adequately describe the evolution of the structural condition under deterioration or corrective actions and, secondly, the non-trivial task of recovery of the observation process parameters from available monitoring data. Despite these potential challenges, the adopted POMDP models do not typically account for uncertainty on model parameters, leading to solutions which can be unrealistically confident. In this work, we address both key issues. We present a framework to estimate POMDP transition and observation model parameters directly from available data, via Markov Chain Monte Carlo (MCMC) sampling of a Hidden Markov Model (HMM) conditioned on actions. The MCMC inference estimates distributions of the involved model parameters. We then form and solve the POMDP problem by exploiting the inferred distributions, to derive solutions that are robust to model uncertainty. We successfully apply our approach on maintenance planning for railway track assets on the basis of a "fractal value" indicator, which is computed from actual railway monitoring data.
translated by 谷歌翻译
有效计划的能力对于生物体和人造系统都是至关重要的。在认知神经科学和人工智能(AI)中广泛研究了基于模型的计划和假期,但是从不同的角度来看,以及难以调和的考虑(生物现实主义与可伸缩性)的不同意见(生物现实主义与可伸缩性)。在这里,我们介绍了一种新颖的方法来计划大型POMDP(Active Tree search(ACT)),该方法结合了神经科学中领先的计划理论的规范性特征和生物学现实主义(主动推论)和树木搜索方法的可扩展性AI。这种统一对两种方法都是有益的。一方面,使用树搜索可以使生物学接地的第一原理,主动推断的方法可应用于大规模问题。另一方面,主动推理为探索 - 开发困境提供了一种原则性的解决方案,该解决方案通常在树搜索方法中以启发性解决。我们的模拟表明,ACT成功地浏览了对基于抽样的方法,需要自适应探索的问题以及大型POMDP问题“ RockSample”的二进制树,其中ACT近似于最新的POMDP解决方案。此外,我们说明了如何使用ACT来模拟人类和其他解决大型计划问题的人类和其他动物的神经生理反应(例如,在海马和前额叶皮层)。这些数值分析表明,主动树搜索是神经科学和AI计划理论的原则性实现,既具有生物现实主义和可扩展性。
translated by 谷歌翻译
嘈杂的传感,不完美的控制和环境变化是许多现实世界机器人任务的定义特征。部分可观察到的马尔可夫决策过程(POMDP)提供了一个原则上的数学框架,用于建模和解决不确定性下的机器人决策和控制任务。在过去的十年中,它看到了许多成功的应用程序,涵盖了本地化和导航,搜索和跟踪,自动驾驶,多机器人系统,操纵和人类机器人交互。这项调查旨在弥合POMDP模型的开发与算法之间的差距,以及针对另一端的不同机器人决策任务的应用。它分析了这些任务的特征,并将它们与POMDP框架的数学和算法属性联系起来,以进行有效的建模和解决方案。对于从业者来说,调查提供了一些关键任务特征,以决定何时以及如何成功地将POMDP应用于机器人任务。对于POMDP算法设计师,该调查为将POMDP应用于机器人系统的独特挑战提供了新的见解,并指出了有希望的新方向进行进一步研究。
translated by 谷歌翻译
受约束的部分可观察到的马尔可夫决策过程(CPOMDP)已用于模拟各种现实现象。但是,众所周知,它们很难解决最优性,并且只有几种近似方法来获得高质量的解决方案。在这项研究中,我们将基于网格的近似值与线性编程(LP)模型结合使用来生成CPOMDP的近似策略。我们考虑了五个CPOMDP问题实例,并对其有限和无限的地平线配方进行了详细的数值研究。我们首先通过使用精确溶液方法进行比较分析来建立近似无约束的POMDP策略的质量。然后,我们显示了基于LP的CPOMDP解决方案方法的性能,用于不同的问题实例的不同预算水平(即成本限制)。最后,我们通过应用确定性政策约束来展示基于LP的方法的灵活性,并研究这些约束对收集的奖励和CPU运行时间的影响。我们的分析表明,LP模型可以有效地为有限和无限的地平线问题生成近似策略,同时提供了将各种其他约束结合到基础模型中的灵活性。
translated by 谷歌翻译
由于数据量增加,金融业的快速变化已经彻底改变了数据处理和数据分析的技术,并带来了新的理论和计算挑战。与古典随机控制理论和解决财务决策问题的其他分析方法相比,解决模型假设的财务决策问题,强化学习(RL)的新发展能够充分利用具有更少模型假设的大量财务数据并改善复杂的金融环境中的决策。该调查纸目的旨在审查最近的资金途径的发展和使用RL方法。我们介绍了马尔可夫决策过程,这是许多常用的RL方法的设置。然后引入各种算法,重点介绍不需要任何模型假设的基于价值和基于策略的方法。连接是用神经网络进行的,以扩展框架以包含深的RL算法。我们的调查通过讨论了这些RL算法在金融中各种决策问题中的应用,包括最佳执行,投资组合优化,期权定价和对冲,市场制作,智能订单路由和Robo-Awaring。
translated by 谷歌翻译
This paper surveys the eld of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the eld and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but di ers considerably in the details and in the use of the word \reinforcement." The paper discusses central issues of reinforcement learning, including trading o exploration and exploitation, establishing the foundations of the eld via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.
translated by 谷歌翻译
主动推断是建模生物学和人造药物的行为的概率框架,该框架源于最小化自由能的原理。近年来,该框架已成功地应用于各种情况下,其目标是最大程度地提高奖励,提供可比性,有时甚至是卓越的性能与替代方法。在本文中,我们通过演示如何以及何时进行主动推理代理执行最佳奖励的动作来阐明奖励最大化和主动推断之间的联系。确切地说,我们展示了主动推理为Bellman方程提供最佳解决方案的条件 - 这种公式是基于模型的增强学习和控制的几种方法。在部分观察到的马尔可夫决策过程中,标准的主动推理方案可以为计划视野1的最佳动作产生最佳动作,但不能超越。相比之下,最近开发的递归活跃推理方案(复杂的推理)可以在任何有限的颞范围内产生最佳作用。我们通过讨论主动推理和强化学习之间更广泛的关系来补充分析。
translated by 谷歌翻译
蒙特卡洛树搜索(MCT)是设计游戏机器人或解决顺序决策问题的强大方法。该方法依赖于平衡探索和开发的智能树搜索。MCT以模拟的形式进行随机抽样,并存储动作的统计数据,以在每个随后的迭代中做出更有教育的选择。然而,该方法已成为组合游戏的最新技术,但是,在更复杂的游戏(例如那些具有较高的分支因素或实时系列的游戏)以及各种实用领域(例如,运输,日程安排或安全性)有效的MCT应用程序通常需要其与问题有关的修改或与其他技术集成。这种特定领域的修改和混合方法是本调查的主要重点。最后一项主要的MCT调查已于2012年发布。自发布以来出现的贡献特别感兴趣。
translated by 谷歌翻译
Reinforcement learning (RL) gained considerable attention by creating decision-making agents that maximize rewards received from fully observable environments. However, many real-world problems are partially or noisily observable by nature, where agents do not receive the true and complete state of the environment. Such problems are formulated as partially observable Markov decision processes (POMDPs). Some studies applied RL to POMDPs by recalling previous decisions and observations or inferring the true state of the environment from received observations. Nevertheless, aggregating observations and decisions over time is impractical for environments with high-dimensional continuous state and action spaces. Moreover, so-called inference-based RL approaches require large number of samples to perform well since agents eschew uncertainty in the inferred state for the decision-making. Active inference is a framework that is naturally formulated in POMDPs and directs agents to select decisions by minimising expected free energy (EFE). This supplies reward-maximising (exploitative) behaviour in RL, with an information-seeking (exploratory) behaviour. Despite this exploratory behaviour of active inference, its usage is limited to discrete state and action spaces due to the computational difficulty of the EFE. We propose a unified principle for joint information-seeking and reward maximization that clarifies a theoretical connection between active inference and RL, unifies active inference and RL, and overcomes their aforementioned limitations. Our findings are supported by strong theoretical analysis. The proposed framework's superior exploration property is also validated by experimental results on partial observable tasks with high-dimensional continuous state and action spaces. Moreover, the results show that our model solves reward-free problems, making task reward design optional.
translated by 谷歌翻译
Partially observable Markov decision processes (POMDPs) provide a flexible representation for real-world decision and control problems. However, POMDPs are notoriously difficult to solve, especially when the state and observation spaces are continuous or hybrid, which is often the case for physical systems. While recent online sampling-based POMDP algorithms that plan with observation likelihood weighting have shown practical effectiveness, a general theory characterizing the approximation error of the particle filtering techniques that these algorithms use has not previously been proposed. Our main contribution is bounding the error between any POMDP and its corresponding finite sample particle belief MDP (PB-MDP) approximation. This fundamental bridge between PB-MDPs and POMDPs allows us to adapt any sampling-based MDP algorithm to a POMDP by solving the corresponding particle belief MDP, thereby extending the convergence guarantees of the MDP algorithm to the POMDP. Practically, this is implemented by using the particle filter belief transition model as the generative model for the MDP solver. While this requires access to the observation density model from the POMDP, it only increases the transition sampling complexity of the MDP solver by a factor of $\mathcal{O}(C)$, where $C$ is the number of particles. Thus, when combined with sparse sampling MDP algorithms, this approach can yield algorithms for POMDPs that have no direct theoretical dependence on the size of the state and observation spaces. In addition to our theoretical contribution, we perform five numerical experiments on benchmark POMDPs to demonstrate that a simple MDP algorithm adapted using PB-MDP approximation, Sparse-PFT, achieves performance competitive with other leading continuous observation POMDP solvers.
translated by 谷歌翻译
Adequately assigning credit to actions for future outcomes based on their contributions is a long-standing open challenge in Reinforcement Learning. The assumptions of the most commonly used credit assignment method are disadvantageous in tasks where the effects of decisions are not immediately evident. Furthermore, this method can only evaluate actions that have been selected by the agent, making it highly inefficient. Still, no alternative methods have been widely adopted in the field. Hindsight Credit Assignment is a promising, but still unexplored candidate, which aims to solve the problems of both long-term and counterfactual credit assignment. In this thesis, we empirically investigate Hindsight Credit Assignment to identify its main benefits, and key points to improve. Then, we apply it to factored state representations, and in particular to state representations based on the causal structure of the environment. In this setting, we propose a variant of Hindsight Credit Assignment that effectively exploits a given causal structure. We show that our modification greatly decreases the workload of Hindsight Credit Assignment, making it more efficient and enabling it to outperform the baseline credit assignment method on various tasks. This opens the way to other methods based on given or learned causal structures.
translated by 谷歌翻译
具有切换持续时间的轮询系统是具有若干实际应用的有用模型。它被归类为离散事件动态系统(DED),没有人在建模方法中同意的是。此外,DEDS非常复杂。迄今为止,最复杂的兴趣调查系统建模的方法是连续时间马尔可夫决策过程(CTMDP)。本文提出了一个半马尔可夫决策过程(SMDP)轮询系统的制定,以引入额外的建模能力。这种权力以截断误差和昂贵的数值积分为代价,自然导致SMDP政策是否提供有价值的优势。为了进一步添加到此方案,显示CTMDP中可以利用稀疏性以开发计算有效的模型。使用半Markov过程模拟器评估SMDP和CTMDP策略的折扣性能。两项政策伴随着专门为该投票系统开发的启发式政策,作为详尽的服务政策。参数和非参数假设试验用于测试性能差异是否有统计学意义。
translated by 谷歌翻译
在强化学习中,代理成功使用了以马尔可夫决策过程(MDP)建模的环境。但是,在许多问题域中,代理可能会遭受嘈杂的观察或随机时间,直到其随后的决定为止。尽管可观察到的马尔可夫决策过程(POMDP)已经处理了嘈杂的观察,但他们尚未处理未知的时间方面。当然,人们可以离散时间,但这导致了贝尔曼的维度诅咒。为了将连续的寄居时间分布纳入代理商的决策中,我们建议部分可观察到的半马尔可夫决策过程(POSMDP)在这方面有所帮助。我们扩展了\ citet {spaan2005a}基于随机点的值迭代(PBVI)\ textsc {perseus}算法,用于POMDP,通过结合连续的SOJOURN时间分布并使用重要性来减少求解器复杂性。我们称此新的PBVI算法为POSMDPS -\ textsc {ChronoSperSeus},其重要性采样。这进一步允许通过将此信息移至pOMSDP的状态周时间来进行压缩的复杂POMDP,需要时间状态信息。第二个见解是,可以在单个备份中使用一组抽样的时间并通过其可能性加权。这有助于进一步降低算法复杂性。该求解器还针对情节性和非疾病问题。我们以两个示例结束了论文,一个情节的巴士问题和非剧烈的维护问题。
translated by 谷歌翻译
Monte Carlo Tree Search (MCTS) is a recently proposed search method that combines the precision of tree search with the generality of random sampling. It has received considerable interest due to its spectacular success in the difficult problem of computer Go, but has also proved beneficial in a range of other domains. This paper is a survey of the literature to date, intended to provide a snapshot of the state of the art after the first five years of MCTS research. We outline the core algorithm's derivation, impart some structure on the many variations and enhancements that have been proposed, and summarise the results from the key game and non-game domains to which MCTS methods have been applied. A number of open research questions indicate that the field is ripe for future work.
translated by 谷歌翻译
Safe Reinforcement Learning can be defined as the process of learning policies that maximize the expectation of the return in problems in which it is important to ensure reasonable system performance and/or respect safety constraints during the learning and/or deployment processes. We categorize and analyze two approaches of Safe Reinforcement Learning. The first is based on the modification of the optimality criterion, the classic discounted finite/infinite horizon, with a safety factor. The second is based on the modification of the exploration process through the incorporation of external knowledge or the guidance of a risk metric. We use the proposed classification to survey the existing literature, as well as suggesting future directions for Safe Reinforcement Learning.
translated by 谷歌翻译
具有成本效益的资产管理是多个行业的兴趣领域。具体而言,本文开发了深入的加固学习(DRL)解决方案,以自动确定不断恶化的水管的最佳康复政策。我们在在线和离线DRL设置中处理康复计划的问题。在在线DRL中,代理与具有不同长度,材料和故障率特征的多个管道的模拟环境进行交互。我们使用深Q学习(DQN)训练代理商,以最低限度的平均成本和减少故障概率学习最佳政策。在离线学习中,代理使用静态数据,例如DQN重播数据,通过保守的Q学习算法学习最佳策略,而无需与环境进行进一步的交互。我们证明,基于DRL的政策改善了标准预防,纠正和贪婪的计划替代方案。此外,从固定的DQN重播数据集中学习超过在线DQN设置。结果保证,由大型国家和行动轨迹组成的水管的现有恶化概况为在离线环境中学习康复政策提供了宝贵的途径,而无需模拟器。
translated by 谷歌翻译
乳腺癌是一种常见且致命的疾病,但是早期诊断时通常可以治愈。尽管大多数国家都有大规模筛查计划,但就乳腺癌筛查的单一全球公认政策尚无共识。疾病的复杂性;筛查方法的可用性有限,例如乳房X线摄影,磁共振成像(MRI)和超声筛选;公共卫生政策都将筛查政策制定。资源可用性问题需要设计符合预算的政策,该问题可以作为约束的部分可观察到的马尔可夫决策过程(CPOMDP)建模。在这项研究中,我们提出了一个多目标CPOMDP模型,用于乳腺癌筛查两个目标:最大程度地减少因乳腺癌而死亡的终生风险,并最大程度地调整了质量调整后的寿命。此外,我们考虑了扩展的动作空间,该空间允许筛查乳房X线摄影超出筛查方法。每个动作都对质量调整后的终身年份和终身风险以及独特的成本都有独特的影响。我们的结果揭示了针对不同预算水平的平均和高风险患者的最佳解决方案的帕累托前沿,决策者可以将其用于实践制定政策。
translated by 谷歌翻译
强化学习和最近的深度增强学习是解决如Markov决策过程建模的顺序决策问题的流行方法。问题和选择算法和超参数的RL建模需要仔细考虑,因为不同的配置可能需要完全不同的性能。这些考虑因素主要是RL专家的任务;然而,RL在研究人员和系统设计师不是RL专家的其他领域中逐渐变得流行。此外,许多建模决策,例如定义状态和动作空间,批次的大小和批量更新的频率以及时间戳的数量通常是手动进行的。由于这些原因,RL框架的自动化不同组成部分具有重要意义,近年来它引起了很多关注。自动RL提供了一个框架,其中RL的不同组件包括MDP建模,算法选择和超参数优化是自动建模和定义的。在本文中,我们探讨了可以在自动化RL中使用的文献和目前的工作。此外,我们讨论了Autorl中的挑战,打开问题和研究方向。
translated by 谷歌翻译