动物和人工代理商都受益于支持跨任务的快速学习的国家表示,使他们能够有效地遍历其环境以获得奖励状态。在固定政策下衡量预期累积,贴现国家占用的后续代表(SR),可以在否则的马尔可维亚环境中有效地转移到不同的奖励结构,并假设生物行为和神经活动的基础方面。然而,在现实世界中,奖励可能会移动或仅用于消费一次,可能只是将位置或者代理可以简单地旨在尽可能快地到达目标状态,而不会产生人工强加的任务视野的约束。在这种情况下,最具行为相关的代表将携带有关代理人可能首先达到兴趣国的信息的信息,而不是在可能的无限时间跨度访问它们的频率。为了反映此类需求,我们介绍了第一次占用代表(FR),该代表(FR),该代表(FR)衡量预期的时间折扣首次访问状态。我们证明FR有助于探索,选择有效的路径到所需状态,允许代理在某些条件下规划由一系列子板定义的可透明的最佳轨迹,并引起避免威胁刺激的动物类似的行为。
translated by 谷歌翻译
本文研究了一种使用背景计划的新方法,用于基于模型的增强学习:混合(近似)动态编程更新和无模型更新,类似于DYNA体系结构。通过学习模型的背景计划通常比无模型替代方案(例如Double DQN)差,尽管前者使用了更多的内存和计算。基本问题是,学到的模型可能是不准确的,并且经常会产生无效的状态,尤其是在迭代许多步骤时。在本文中,我们通过将背景规划限制为一组(抽象)子目标并仅学习本地,子观念模型来避免这种限制。这种目标空间计划(GSP)方法更有效地是在计算上,自然地纳入了时间抽象,以进行更快的长胜压计划,并避免完全学习过渡动态。我们表明,在各种情况下,我们的GSP算法比双DQN基线要快得多。
translated by 谷歌翻译
Transfer in Reinforcement Learning aims to improve learning performance on target tasks using knowledge from experienced source tasks. Successor Representations (SR) and their extension Successor Features (SF) are prominent transfer mechanisms in domains where reward functions change between tasks. They reevaluate the expected return of previously learned policies in a new target task to transfer their knowledge. The SF framework extended SR by linearly decomposing rewards into successor features and a reward weight vector allowing their application in high-dimensional tasks. But this came with the cost of having a linear relationship between reward functions and successor features, limiting its application to such tasks. We propose a novel formulation of SR based on learning the cumulative discounted probability of successor features, called Successor Feature Representations (SFR). Crucially, SFR allows to reevaluate the expected return of policies for general reward functions. We introduce different SFR variations, prove its convergence, and provide a guarantee on its transfer performance. Experimental evaluations based on SFR with function approximation demonstrate its advantage over SF not only for general reward functions but also in the case of linearly decomposable reward functions.
translated by 谷歌翻译
我们介绍了一种改进政策改进的方法,该方法在基于价值的强化学习(RL)的贪婪方法与基于模型的RL的典型计划方法之间进行了插值。新方法建立在几何视野模型(GHM,也称为伽马模型)的概念上,该模型对给定策略的折现状态验证分布进行了建模。我们表明,我们可以通过仔细的基本策略GHM的仔细组成,而无需任何其他学习,可以评估任何非马尔科夫策略,以固定的概率在一组基本马尔可夫策略之间切换。然后,我们可以将广义政策改进(GPI)应用于此类非马尔科夫政策的收集,以获得新的马尔可夫政策,通常将其表现优于其先驱。我们对这种方法提供了彻底的理论分析,开发了转移和标准RL的应用,并在经验上证明了其对标准GPI的有效性,对充满挑战的深度RL连续控制任务。我们还提供了GHM培训方法的分析,证明了关于先前提出的方法的新型收敛结果,并显示了如何在深度RL设置中稳定训练这些模型。
translated by 谷歌翻译
This paper surveys the eld of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the eld and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but di ers considerably in the details and in the use of the word \reinforcement." The paper discusses central issues of reinforcement learning, including trading o exploration and exploitation, establishing the foundations of the eld via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.
translated by 谷歌翻译
In this work, we focus on the problem of safe policy transfer in reinforcement learning: we seek to leverage existing policies when learning a new task with specified constraints. This problem is important for safety-critical applications where interactions are costly and unconstrained policies can lead to undesirable or dangerous outcomes, e.g., with physical robots that interact with humans. We propose a Constrained Markov Decision Process (CMDP) formulation that simultaneously enables the transfer of policies and adherence to safety constraints. Our formulation cleanly separates task goals from safety considerations and permits the specification of a wide variety of constraints. Our approach relies on a novel extension of generalized policy improvement to constrained settings via a Lagrangian formulation. We devise a dual optimization algorithm that estimates the optimal dual variable of a target task, thus enabling safe transfer of policies derived from successor features learned on source tasks. Our experiments in simulated domains show that our approach is effective; it visits unsafe states less frequently and outperforms alternative state-of-the-art methods when taking safety constraints into account.
translated by 谷歌翻译
当加强学习以稀疏的奖励应用时,代理必须花费很长时间探索未知环境而没有任何学习信号。抽象是一种为代理提供在潜在空间中过渡的内在奖励的方法。先前的工作着重于密集的连续潜在空间,或要求用户手动提供表示形式。我们的方法是第一个自动学习基础环境的离散抽象的方法。此外,我们的方法使用端到端可训练的正规后继代表模型在任意输入空间上起作用。对于抽象状态之间的过渡,我们以选项的形式训练一组时间扩展的动作,即动作抽象。我们提出的算法,离散的国家行动抽象(DSAA),在训练这些选项之间进行迭代交换,并使用它们有效地探索更多环境以改善状态抽象。结果,我们的模型不仅对转移学习,而且在在线学习环境中有用。我们从经验上表明,与基线加强学习算法相比,我们的代理能够探索环境并更有效地解决任务。我们的代码可在\ url {https://github.com/amnonattali/dsaa}上公开获得。
translated by 谷歌翻译
在许多实际应用程序中,强化学习(RL)代理可能必须解决多个任务,每个任务通常都是通过奖励功能建模的。如果奖励功能是线性表达的,并且代理商以前已经学会了一组针对不同任务的策略,则可以利用后继功能(SFS)来组合此类策略并确定有关新问题的合理解决方案。但是,确定的解决方案不能保证是最佳的。我们介绍了一种解决此限制的新颖算法。它允许RL代理结合现有政策并直接确定任意新问题的最佳政策,而无需与环境进行任何进一步的互动。我们首先(在轻度假设下)表明,SFS解决的转移学习问题等同于学习在RL中优化多个目标的学习问题。然后,我们引入了基于SF的乐观线性支持算法的扩展,以学习一组SFS构成凸面覆盖范围集的策略。我们证明,该集合中的策略可以通过广义策略改进组合,以构建任何可表达的新任务的最佳行为,而无需任何其他培训样本。我们从经验上表明,在价值函数近似下,我们的方法在离散和连续域中优于最先进的竞争算法。
translated by 谷歌翻译
我们提出了连续表示的时间扩展变化,我们称其为t-SR。 T-SR通过在原始动作重复序列上构造后继表示,捕获了时间扩展动作的预期状态过渡动力学。这种时间抽象的这种形式不能学习相关任务结构的自上而下的层次结构,而是对耦合动作和动作重复的自下而上的组成。这减少了在没有学习层次政策的情况下控制中所需的决策数量。因此,T-SR直接考虑了时间扩展的动作序列的时间范围,而无需预定义或域特异性选项。我们表明,在具有动态奖励结构的环境中,T-SR能够利用后继表示的灵活性和时间扩展的动作提供的抽象。因此,在一系列稀疏的网格世界环境中,T-SR最佳地适应策略远比基于可比的无模型的强化学习方法快得多。我们还表明,T-SR学到的解决这些任务的方式要求学习的策略的始终如一的频率比非临时扩展的策略少。
translated by 谷歌翻译
深度加强学习的最近成功的大部分是由正常化的政策优化(RPO)算法驱动,具有跨多个域的强大性能。在这家族的方法中,代理经过培训,以在惩罚某些引用或默认策略的行为中的偏差时最大化累积奖励。除了经验的成功外,还有一个强大的理论基础,了解应用于单一任务的RPO方法,与自然梯度,信任区域和变分方法有关。但是,对于多任务设置中的默认策略,对所需属性的正式理解有限,越来越重要的域作为现场转向培训更有能力的代理商。在这里,我们通过将默认策略的质量与其对优化的影响正式链接到其对其影响的效果方面,进行第一步才能填补这种差距。使用这些结果,我们将获得具有强大性能保证的多任务学习的原则性的RPO算法。
translated by 谷歌翻译
在现实世界中经营通常需要代理商来了解复杂的环境,并应用这种理解以实现一系列目标。这个问题被称为目标有条件的强化学习(GCRL),对长地平线的目标变得特别具有挑战性。目前的方法通过使用基于图形的规划算法增强目标条件的策略来解决这个问题。然而,他们努力缩放到大型高维状态空间,并采用用于有效地收集训练数据的探索机制。在这项工作中,我们介绍了继任者功能标志性(SFL),这是一种探索大型高维环境的框架,以获得熟练的政策熟练的策略。 SFL利用继承特性(SF)来捕获转换动态的能力,通过估计状态新颖性来驱动探索,并通过将状态空间作为基于非参数标志的图形来实现高级规划。我们进一步利用SF直接计算地标遍历的目标条件调节策略,我们用于在探索状态空间边缘执行计划“前沿”地标。我们在我们的Minigrid和VizDoom进行了实验,即SFL可以高效地探索大型高维状态空间和优于长地平线GCRL任务的最先进的基线。
translated by 谷歌翻译
在本文中,我们提出了一种新的马尔可夫决策过程学习分层表示的方法。我们的方法通过将状态空间划分为子集,并定义用于在分区之间执行转换的子任务。我们制定将状态空间作为优化问题分区的问题,该优化问题可以使用梯度下降给出一组采样的轨迹来解决,使我们的方法适用于大状态空间的高维问题。我们经验验证方法,通过表示它可以成功地在导航域中成功学习有用的分层表示。一旦了解到,分层表示可以用于解决给定域中的不同任务,从而概括跨任务的知识。
translated by 谷歌翻译
Adequately assigning credit to actions for future outcomes based on their contributions is a long-standing open challenge in Reinforcement Learning. The assumptions of the most commonly used credit assignment method are disadvantageous in tasks where the effects of decisions are not immediately evident. Furthermore, this method can only evaluate actions that have been selected by the agent, making it highly inefficient. Still, no alternative methods have been widely adopted in the field. Hindsight Credit Assignment is a promising, but still unexplored candidate, which aims to solve the problems of both long-term and counterfactual credit assignment. In this thesis, we empirically investigate Hindsight Credit Assignment to identify its main benefits, and key points to improve. Then, we apply it to factored state representations, and in particular to state representations based on the causal structure of the environment. In this setting, we propose a variant of Hindsight Credit Assignment that effectively exploits a given causal structure. We show that our modification greatly decreases the workload of Hindsight Credit Assignment, making it more efficient and enabling it to outperform the baseline credit assignment method on various tasks. This opens the way to other methods based on given or learned causal structures.
translated by 谷歌翻译
在强化学习(RL)中,目标是获得最佳政策,最佳标准在根本上至关重要。两个主要的最优标准是平均奖励和打折的奖励。虽然后者更受欢迎,但在没有固有折扣概念的情况下,在环境中申请是有问题的。这促使我们重新审视a)动态编程中最佳标准的进步,b)人工折现因子的理由和复杂性,c)直接最大化平均奖励标准的好处,这是无折扣的。我们的贡献包括对平均奖励和打折奖励之间的关系以及对RL中的利弊的讨论之间的关系。我们强调的是,平均奖励RL方法具有将无折扣优化标准(Veinott,1969)应用于RL的成分和机制。
translated by 谷歌翻译
在这项工作中,我们提出了一种初步调查一种名为DYNA-T的新算法。在钢筋学习(RL)中,规划代理有自己的环境表示作为模型。要发现与环境互动的最佳政策,代理商会收集试验和错误时尚的经验。经验可用于学习更好的模型或直接改进价值函数和政策。通常是分离的,Dyna-Q是一种混合方法,在每次迭代,利用真实体验更新模型以及值函数,同时使用模拟数据从其模型中的应用程序进行行动。然而,规划过程是计算昂贵的并且强烈取决于国家行动空间的维度。我们建议在模拟体验上构建一个上置信树(UCT),并在在线学习过程中搜索要选择的最佳动作。我们证明了我们提出的方法对来自Open AI的三个测试平台环境的一系列初步测试的有效性。与Dyna-Q相比,Dyna-T通过选择更强大的动作选择策略来优于随机环境中的最先进的RL代理。
translated by 谷歌翻译
Go-Explore achieved breakthrough performance on challenging reinforcement learning (RL) tasks with sparse rewards. The key insight of Go-Explore was that successful exploration requires an agent to first return to an interesting state ('Go'), and only then explore into unknown terrain ('Explore'). We refer to such exploration after a goal is reached as 'post-exploration'. In this paper, we present a clear ablation study of post-exploration in a general intrinsically motivated goal exploration process (IMGEP) framework, that the Go-Explore paper did not show. We study the isolated potential of post-exploration, by turning it on and off within the same algorithm under both tabular and deep RL settings on both discrete navigation and continuous control tasks. Experiments on a range of MiniGrid and Mujoco environments show that post-exploration indeed helps IMGEP agents reach more diverse states and boosts their performance. In short, our work suggests that RL researchers should consider to use post-exploration in IMGEP when possible since it is effective, method-agnostic and easy to implement.
translated by 谷歌翻译
Model-Based Reinforcement Learning (RL) is widely believed to have the potential to improve sample efficiency by allowing an agent to synthesize large amounts of imagined experience. Experience Replay (ER) can be considered a simple kind of model, which has proved extremely effective at improving the stability and efficiency of deep RL. In principle, a learned parametric model could improve on ER by generalizing from real experience to augment the dataset with additional plausible experience. However, owing to the many design choices involved in empirically successful algorithms, it can be very hard to establish where the benefits are actually coming from. Here, we provide theoretical and empirical insight into when, and how, we can expect data generated by a learned model to be useful. First, we provide a general theorem motivating how learning a model as an intermediate step can narrow down the set of possible value functions more than learning a value function directly from data using the Bellman equation. Second, we provide an illustrative example showing empirically how a similar effect occurs in a more concrete setting with neural network function approximation. Finally, we provide extensive experiments showing the benefit of model-based learning for online RL in environments with combinatorial complexity, but factored structure that allows a learned model to generalize. In these experiments, we take care to control for other factors in order to isolate, insofar as possible, the benefit of using experience generated by a learned model relative to ER alone.
translated by 谷歌翻译
由政策引起的马尔可夫链的混合时间限制了现实世界持续学习场景中的性能。然而,混合时间对持续增强学习学习(RL)的影响仍然是曝光率。在本文中,我们表征了长期兴趣的问题,以通过混合时间调用可扩展的MDP来发展持续的RL。特别是,我们建立可扩展的MDP具有与问题的大小相等的混合时间。我们继续证明,多项式混合时间对现有方法产生显着困难,并提出了一种基于模型的算法,通过新颖的引导程序直接优化平均奖励来加速学习。最后,我们对我们提出的方法进行了实证遗憾分析,展示了对基线的清晰改进,以及如何使用可缩放的MDP来分析RL算法作为混合时间规模。
translated by 谷歌翻译
目标条件层次结构增强学习(HRL)是扩大强化学习(RL)技术的有前途的方法。但是,由于高级的动作空间,即目标空间很大。在大型目标空间中进行搜索对于高级子观念和低级政策学习都构成了困难。在本文中,我们表明,可以使用邻接约束来限制从整个目标空间到当前状态的$ k $步骤相邻区域的高级动作空间,从而有效缓解此问题。从理论上讲,我们证明在确定性的马尔可夫决策过程(MDP)中,所提出的邻接约束保留了最佳的层次结构策略,而在随机MDP中,邻接约束诱导了由MDP的过渡结构确定的有界状态价值次数。我们进一步表明,可以通过培训可以区分邻近和非贴种亚目标的邻接网络来实际实现此约束。对离散和连续控制任务的实验结果,包括挑战性的机器人运动和操纵任务,表明合并邻接性约束可显着提高最先进的目标条件条件的HRL方法的性能。
translated by 谷歌翻译
在线强化学习(RL)中的挑战之一是代理人需要促进对环境的探索和对样品的利用来优化其行为。无论我们是否优化遗憾,采样复杂性,状态空间覆盖范围或模型估计,我们都需要攻击不同的勘探开发权衡。在本文中,我们建议在分离方法组成的探索 - 剥削问题:1)“客观特定”算法(自适应)规定哪些样本以收集到哪些状态,似乎它可以访问a生成模型(即环境的模拟器); 2)负责尽可能快地生成规定样品的“客观无关的”样品收集勘探策略。建立最近在随机最短路径问题中进行探索的方法,我们首先提供一种算法,它给出了每个状态动作对所需的样本$ B(S,a)$的样本数量,需要$ \ tilde {o} (bd + d ^ {3/2} s ^ 2 a)收集$ b = \ sum_ {s,a} b(s,a)$所需样本的$时间步骤,以$ s $各国,$ a $行动和直径$ d $。然后我们展示了这种通用探索算法如何与“客观特定的”策略配对,这些策略规定了解决各种设置的样本要求 - 例如,模型估计,稀疏奖励发现,无需无成本勘探沟通MDP - 我们获得改进或新颖的样本复杂性保证。
translated by 谷歌翻译