基于模型的强化学习有望通过学习环境中的中间模型来预测未来的相互作用,从而从与环境的互动较少的相互作用中学习最佳政策。当预测一系列相互作用时,限制预测范围的推出长度是关键的超参数,因为预测的准确性会降低远离真实体验的区域。结果,从长远来看,从长远来看,总体上更糟糕的政策。因此,超参数提供了质量和效率之间的权衡。在这项工作中,我们将调整推出长度调整为元级的顺序决策问题的问题构成了问题,该问题优化了基于模型的强化学习所学到的最终策略,鉴于环境相互作用的固定预算通过基于反馈动态调整超参数来调整超参数。从学习过程中,例如模型的准确性和互动的其余预算。我们使用无模型的深度强化学习来解决元级决策问题,并证明我们的方法在两个众所周知的强化学习环境上优于共同的启发式基准。
translated by 谷歌翻译
具有成本效益的资产管理是多个行业的兴趣领域。具体而言,本文开发了深入的加固学习(DRL)解决方案,以自动确定不断恶化的水管的最佳康复政策。我们在在线和离线DRL设置中处理康复计划的问题。在在线DRL中,代理与具有不同长度,材料和故障率特征的多个管道的模拟环境进行交互。我们使用深Q学习(DQN)训练代理商,以最低限度的平均成本和减少故障概率学习最佳政策。在离线学习中,代理使用静态数据,例如DQN重播数据,通过保守的Q学习算法学习最佳策略,而无需与环境进行进一步的交互。我们证明,基于DRL的政策改善了标准预防,纠正和贪婪的计划替代方案。此外,从固定的DQN重播数据集中学习超过在线DQN设置。结果保证,由大型国家和行动轨迹组成的水管的现有恶化概况为在离线环境中学习康复政策提供了宝贵的途径,而无需模拟器。
translated by 谷歌翻译
机器学习算法中多个超参数的最佳设置是发出大多数可用数据的关键。为此目的,已经提出了几种方法,例如进化策略,随机搜索,贝叶斯优化和启发式拇指规则。在钢筋学习(RL)中,学习代理在与其环境交互时收集的数据的信息内容严重依赖于许多超参数的设置。因此,RL算法的用户必须依赖于基于搜索的优化方法,例如网格搜索或Nelder-Mead单简单算法,这对于大多数R1任务来说是非常效率的,显着减慢学习曲线和离开用户的速度有目的地偏见数据收集的负担。在这项工作中,为了使RL算法更加用户独立,提出了一种使用贝叶斯优化的自主超参数设置的新方法。来自过去剧集和不同的超参数值的数据通过执行行为克隆在元学习水平上使用,这有助于提高最大化获取功能的加强学习变体的有效性。此外,通过紧密地整合在加强学习代理设计中的贝叶斯优化,还减少了收敛到给定任务的最佳策略所需的状态转换的数量。与其他手动调整和基于优化的方法相比,计算实验显示了有希望的结果,这突出了改变算法超级参数来增加所生成数据的信息内容的好处。
translated by 谷歌翻译
强化学习(RL)通过与环境相互作用的试验过程解决顺序决策问题。尽管RL在玩复杂的视频游戏方面取得了巨大的成功,但在现实世界中,犯错误总是不希望的。为了提高样本效率并从而降低错误,据信基于模型的增强学习(MBRL)是一个有前途的方向,它建立了环境模型,在该模型中可以进行反复试验,而无需实际成本。在这项调查中,我们对MBRL进行了审查,重点是Deep RL的最新进展。对于非壮观环境,学到的环境模型与真实环境之间始终存在概括性错误。因此,非常重要的是分析环境模型中的政策培训与实际环境中的差异,这反过来又指导了更好的模型学习,模型使用和政策培训的算法设计。此外,我们还讨论了其他形式的RL,包括离线RL,目标条件RL,多代理RL和Meta-RL的最新进展。此外,我们讨论了MBRL在现实世界任务中的适用性和优势。最后,我们通过讨论MBRL未来发展的前景来结束这项调查。我们认为,MBRL在被忽略的现实应用程序中具有巨大的潜力和优势,我们希望这项调查能够吸引更多关于MBRL的研究。
translated by 谷歌翻译
本文介绍了用于交易单一资产的双重Q网络算法,即E-MINI S&P 500连续期货合约。我们使用经过验证的设置作为我们环境的基础,并具有多个扩展。我们的贸易代理商的功能不断扩展,包括其他资产,例如商品,从而产生了四种型号。我们还应对环境条件,包括成本和危机。我们的贸易代理商首先接受了特定时间段的培训,并根据新数据进行了测试,并将其与长期策略(市场)进行了比较。我们分析了各种模型与样本中/样本外性能之间有关环境的差异。实验结果表明,贸易代理人遵循适当的行为。它可以将其政策调整为不同的情况,例如在存在交易成本时更广泛地使用中性位置。此外,净资产价值超过了基准的净值,代理商在测试集中的市场优于市场。我们使用DDQN算法对代理商在金融领域中的行为提供初步见解。这项研究的结果可用于进一步发展。
translated by 谷歌翻译
在这项工作中,我们提出了一种初步调查一种名为DYNA-T的新算法。在钢筋学习(RL)中,规划代理有自己的环境表示作为模型。要发现与环境互动的最佳政策,代理商会收集试验和错误时尚的经验。经验可用于学习更好的模型或直接改进价值函数和政策。通常是分离的,Dyna-Q是一种混合方法,在每次迭代,利用真实体验更新模型以及值函数,同时使用模拟数据从其模型中的应用程序进行行动。然而,规划过程是计算昂贵的并且强烈取决于国家行动空间的维度。我们建议在模拟体验上构建一个上置信树(UCT),并在在线学习过程中搜索要选择的最佳动作。我们证明了我们提出的方法对来自Open AI的三个测试平台环境的一系列初步测试的有效性。与Dyna-Q相比,Dyna-T通过选择更强大的动作选择策略来优于随机环境中的最先进的RL代理。
translated by 谷歌翻译
Deep reinforcement learning is poised to revolutionise the field of AI and represents a step towards building autonomous systems with a higher level understanding of the visual world. Currently, deep learning is enabling reinforcement learning to scale to problems that were previously intractable, such as learning to play video games directly from pixels. Deep reinforcement learning algorithms are also applied to robotics, allowing control policies for robots to be learned directly from camera inputs in the real world. In this survey, we begin with an introduction to the general field of reinforcement learning, then progress to the main streams of value-based and policybased methods. Our survey will cover central algorithms in deep reinforcement learning, including the deep Q-network, trust region policy optimisation, and asynchronous advantage actor-critic. In parallel, we highlight the unique advantages of deep neural networks, focusing on visual understanding via reinforcement learning. To conclude, we describe several current areas of research within the field.
translated by 谷歌翻译
With the development of deep representation learning, the domain of reinforcement learning (RL) has become a powerful learning framework now capable of learning complex policies in high dimensional environments. This review summarises deep reinforcement learning (DRL) algorithms and provides a taxonomy of automated driving tasks where (D)RL methods have been employed, while addressing key computational challenges in real world deployment of autonomous driving agents. It also delineates adjacent domains such as behavior cloning, imitation learning, inverse reinforcement learning that are related but are not classical RL algorithms. The role of simulators in training agents, methods to validate, test and robustify existing solutions in RL are discussed.
translated by 谷歌翻译
Model-Based Reinforcement Learning (RL) is widely believed to have the potential to improve sample efficiency by allowing an agent to synthesize large amounts of imagined experience. Experience Replay (ER) can be considered a simple kind of model, which has proved extremely effective at improving the stability and efficiency of deep RL. In principle, a learned parametric model could improve on ER by generalizing from real experience to augment the dataset with additional plausible experience. However, owing to the many design choices involved in empirically successful algorithms, it can be very hard to establish where the benefits are actually coming from. Here, we provide theoretical and empirical insight into when, and how, we can expect data generated by a learned model to be useful. First, we provide a general theorem motivating how learning a model as an intermediate step can narrow down the set of possible value functions more than learning a value function directly from data using the Bellman equation. Second, we provide an illustrative example showing empirically how a similar effect occurs in a more concrete setting with neural network function approximation. Finally, we provide extensive experiments showing the benefit of model-based learning for online RL in environments with combinatorial complexity, but factored structure that allows a learned model to generalize. In these experiments, we take care to control for other factors in order to isolate, insofar as possible, the benefit of using experience generated by a learned model relative to ER alone.
translated by 谷歌翻译
设计有效的基于模型的增强学习算法很困难,因为必须对模型生成数据的偏置权衡数据生成的易用性。在本文中,我们研究了模型使用在理论上和经验上的政策优化中的作用。我们首先制定和分析一种基于模型的加强学习算法,并在每个步骤中保证单调改善。在实践中,该分析过于悲观,并表明实际的脱助策略数据总是优选模拟策略数据,但我们表明可以将模型概括的经验估计纳入这样的分析以证明模型使用证明模型使用。通过这种分析的动机,我们证明,使用从真实数据分支的短模型生成的卷展栏的简单过程具有更复杂的基于模型的算法而没有通常的缺陷的效益。特别是,这种方法超越了基于模型的方法的样本效率,匹配了最佳无模型算法的渐近性能,并缩放到导致其他基于模型的方法完全失败的视野。
translated by 谷歌翻译
在本文中,我们提出了一种新的马尔可夫决策过程学习分层表示的方法。我们的方法通过将状态空间划分为子集,并定义用于在分区之间执行转换的子任务。我们制定将状态空间作为优化问题分区的问题,该优化问题可以使用梯度下降给出一组采样的轨迹来解决,使我们的方法适用于大状态空间的高维问题。我们经验验证方法,通过表示它可以成功地在导航域中成功学习有用的分层表示。一旦了解到,分层表示可以用于解决给定域中的不同任务,从而概括跨任务的知识。
translated by 谷歌翻译
当国家行动对具有等效的奖励和过渡动态时,动物能够从有限的经验中迅速推断出来。另一方面,现代的强化学习系统必须通过反复试验进行艰苦的学习,以使国家行动对相当于价值 - 需要从其环境中进行过多的大量样本。已经提出了MDP同态,将观察到的环境的MDP降低到抽象的MDP,这可以实现更有效的样本策略学习。因此,当可以先验地构建合适的MDP同构时,已经实现了样本效率的令人印象深刻的提高 - 通常是通过利用执业者对环境对称性的知识来实现​​的。我们提出了一种在离散作用空间中构建同态的新方法,该方法使用部分环境动力学模型来推断哪种状态作用对导致同一状态 - 将状态行动空间的大小减少了一个等于动作空间的基数。我们称此方法等效效果抽象。在GridWorld环境中,我们从经验上证明了等效效果抽象可以提高基于模型的方法的无模型设置和计划效率的样品效率。此外,我们在Cartpole上表明,我们的方法的表现优于学习同构的现有方法,同时使用33倍的培训数据。
translated by 谷歌翻译
A long-standing challenge in artificial intelligence is lifelong learning. In lifelong learning, many tasks are presented in sequence and learners must efficiently transfer knowledge between tasks while avoiding catastrophic forgetting over long lifetimes. On these problems, policy reuse and other multi-policy reinforcement learning techniques can learn many tasks. However, they can generate many temporary or permanent policies, resulting in memory issues. Consequently, there is a need for lifetime-scalable methods that continually refine a policy library of a pre-defined size. This paper presents a first approach to lifetime-scalable policy reuse. To pre-select the number of policies, a notion of task capacity, the maximal number of tasks that a policy can accurately solve, is proposed. To evaluate lifetime policy reuse using this method, two state-of-the-art single-actor base-learners are compared: 1) a value-based reinforcement learner, Deep Q-Network (DQN) or Deep Recurrent Q-Network (DRQN); and 2) an actor-critic reinforcement learner, Proximal Policy Optimisation (PPO) with or without Long Short-Term Memory layer. By selecting the number of policies based on task capacity, D(R)QN achieves near-optimal performance with 6 policies in a 27-task MDP domain and 9 policies in an 18-task POMDP domain; with fewer policies, catastrophic forgetting and negative transfer are observed. Due to slow, monotonic improvement, PPO requires fewer policies, 1 policy for the 27-task domain and 4 policies for the 18-task domain, but it learns the tasks with lower accuracy than D(R)QN. These findings validate lifetime-scalable policy reuse and suggest using D(R)QN for larger and PPO for smaller library sizes.
translated by 谷歌翻译
在这项工作中,我们提出并评估了一种新的增强学习方法,紧凑体验重放(编者),它使用基于相似转换集的复发的预测目标值的时间差异学习,以及基于两个转换的经验重放的新方法记忆。我们的目标是减少在长期累计累计奖励的经纪人培训所需的经验。它与强化学习的相关性与少量观察结果有关,即它需要实现类似于文献中的相关方法获得的结果,这通常需要数百万视频框架来培训ATARI 2600游戏。我们举报了在八个挑战街机学习环境(ALE)挑战游戏中,为仅10万帧的培训试验和大约25,000次迭代的培训试验中报告了培训试验。我们还在与基线的同一游戏中具有相同的实验协议的DQN代理呈现结果。为了验证从较少数量的观察结果近似于良好的政策,我们还将其结果与从啤酒的基准上呈现的数百万帧中获得的结果进行比较。
translated by 谷歌翻译
Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard offpolicy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning without data correlated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.
translated by 谷歌翻译
Safe Reinforcement Learning can be defined as the process of learning policies that maximize the expectation of the return in problems in which it is important to ensure reasonable system performance and/or respect safety constraints during the learning and/or deployment processes. We categorize and analyze two approaches of Safe Reinforcement Learning. The first is based on the modification of the optimality criterion, the classic discounted finite/infinite horizon, with a safety factor. The second is based on the modification of the exploration process through the incorporation of external knowledge or the guidance of a risk metric. We use the proposed classification to survey the existing literature, as well as suggesting future directions for Safe Reinforcement Learning.
translated by 谷歌翻译
In this paper, we build on advances introduced by the Deep Q-Networks (DQN) approach to extend the multi-objective tabular Reinforcement Learning (RL) algorithm W-learning to large state spaces. W-learning algorithm can naturally solve the competition between multiple single policies in multi-objective environments. However, the tabular version does not scale well to environments with large state spaces. To address this issue, we replace underlying Q-tables with DQN, and propose an addition of W-Networks, as a replacement for tabular weights (W) representations. We evaluate the resulting Deep W-Networks (DWN) approach in two widely-accepted multi-objective RL benchmarks: deep sea treasure and multi-objective mountain car. We show that DWN solves the competition between multiple policies while outperforming the baseline in the form of a DQN solution. Additionally, we demonstrate that the proposed algorithm can find the Pareto front in both tested environments.
translated by 谷歌翻译
基于模型的强化学习引起了广泛的样本效率。尽管到目前为止,它令人印象深刻,但仍然不清楚如何适当安排重要的超参数,以实现足够的性能,例如基于Dyna样式的算法中的政策优化的实际数据比。在本文中,我们首先分析了实际数据在政策培训中的作用,这表明逐渐增加了实际数据的比例会产生更好的性能。灵感来自分析,我们提出了一个名为autombpo的框架,以自动安排真实的数据比以及基于培训模型的策略优化(MBPO)算法的其他超参数,是基于模型的方法的代表性运行情况。在几个连续控制任务上,由AutomBPO安排的HyperParameters培训的MBPO实例可以显着超越原始的,并且AutomBPO找到的真实数据比例计划显示了与我们的理论分析的一致性。
translated by 谷歌翻译
Off-policy reinforcement learning (RL) using a fixed offline dataset of logged interactions is an important consideration in real world applications. This paper studies offline RL using the DQN Replay Dataset comprising the entire replay experience of a DQN agent on 60 Atari 2600 games. We demonstrate that recent off-policy deep RL algorithms, even when trained solely on this fixed dataset, outperform the fully-trained DQN agent. To enhance generalization in the offline setting, we present Random Ensemble Mixture (REM), a robust Q-learning algorithm that enforces optimal Bellman consistency on random convex combinations of multiple Q-value estimates. Offline REM trained on the DQN Replay Dataset surpasses strong RL baselines. Ablation studies highlight the role of offline dataset size and diversity as well as the algorithm choice in our positive results. Overall, the results here present an optimistic view that robust RL algorithms used on sufficiently large and diverse offline datasets can lead to high quality policies. To provide a testbed for offline RL and reproduce our results, the DQN Replay Dataset is released at offline-rl.github.io.
translated by 谷歌翻译
大多数政策评估算法基于Bellman期望和最优性方程的理论,它导出了两个流行的方法 - 政策迭代(PI)和价值迭代(VI)。然而,由于多步骤禁止校正的大方差,多步引导往往是在基于PI的基于PI的方法的交叉目的和禁止策略学习。相比之下,基于VI的方法是自然的违规政策,但受到一步学习的影响。本文通过利用具有最优值函数的多步自举函数的潜在结构来推导新的多步贝尔曼最优性方程。通过这种新的等式,我们推出了一种新的多步值迭代方法,该方法将以指数收缩率$ \ mathcal {o}(\ gamma ^ n)$但仅线性计算复杂度收敛到最佳值函数。此外,它可以自然地推导出一套多步脱离策略算法,可以安全地利用任意策略收集的数据,无需校正。实验表明,所提出的方法是可靠的,易于实施和实现最先进的性能在一系列标准基准数据集上。
translated by 谷歌翻译