事后重新标记已成为多进球增强学习(RL)的基础技术。这个想法非常简单:任何任意轨迹都可以看作是达到轨迹最终状态的专家演示。直观地,此程序训练了一个目标条件政策,以模仿次优的专家。但是,模仿与事后重新标签之间的这种联系尚不清楚。现代模仿学习算法是用Divergence最小化的语言描述的,但仍然是一个开放的问题。在这项工作中,我们开发了一个统一的目标,以解释这种联系,从中我们可以从中获得目标条件的监督学习(GCSL)和奖励功能,并从第一原则中获得了事后见解体验重播(她)。在实验上,我们发现,尽管目标条件行为克隆(BC)最近取得了进步,但多进球Q学习仍然可以超越BC样方法。此外,两者的香草组合实际上都损害了模型性能。在我们的框架下,我们研究何时期望卑诗省提供帮助,并从经验上验证我们的发现。我们的工作进一步桥接了目标的目标和生成建模,说明了将生成模型成功扩展到RL的细微差别和新途径。
translated by 谷歌翻译
在许多顺序决策问题(例如,机器人控制,游戏播放,顺序预测),人类或专家数据可用包含有关任务的有用信息。然而,来自少量专家数据的模仿学习(IL)可能在具有复杂动态的高维环境中具有挑战性。行为克隆是一种简单的方法,由于其简单的实现和稳定的收敛而被广泛使用,但不利用涉及环境动态的任何信息。由于对奖励和政策近似器或偏差,高方差梯度估计器,难以在实践中难以在实践中努力训练的许多现有方法。我们介绍了一种用于动态感知IL的方法,它通过学习单个Q函数来避免对抗训练,隐含地代表奖励和策略。在标准基准测试中,隐式学习的奖励显示与地面真实奖励的高正面相关性,说明我们的方法也可以用于逆钢筋学习(IRL)。我们的方法,逆软Q学习(IQ-Learn)获得了最先进的结果,在离线和在线模仿学习设置中,显着优于现有的现有方法,这些方法都在所需的环境交互和高维空间中的可扩展性中,通常超过3倍。
translated by 谷歌翻译
While reinforcement learning algorithms provide automated acquisition of optimal policies, practical application of such methods requires a number of design decisions, such as manually designing reward functions that not only define the task, but also provide sufficient shaping to accomplish it. In this paper, we view reinforcement learning as inferring policies that achieve desired outcomes, rather than as a problem of maximizing rewards. To solve this inference problem, we establish a novel variational inference formulation that allows us to derive a well-shaped reward function which can be learned directly from environment interactions. From the corresponding variational objective, we also derive a new probabilistic Bellman backup operator and use it to develop an off-policy algorithm to solve goal-directed tasks. We empirically demonstrate that this method eliminates the need to hand-craft reward functions for a suite of diverse manipulation and locomotion tasks and leads to effective goal-directed behaviors.
translated by 谷歌翻译
我们提出了状态匹配的离线分布校正估计(SMODICE),这是一种新颖且基于多功能回归的离线模仿学习(IL)算法,该算法是通过状态占用匹配得出的。我们表明,SMODICE目标通过在表格MDP中的Fenchel二元性和一个分析解决方案的应用来接受一个简单的优化过程。不需要访问专家的行动,可以将Smodice有效地应用于三个离线IL设置:(i)模仿观察值(IFO),(ii)IFO具有动态或形态上不匹配的专家,以及(iii)基于示例的加固学习,这些学习我们表明可以将其公式为州占领的匹配问题。我们在GridWorld环境以及高维离线基准上广泛评估了Smodice。我们的结果表明,Smodice对于所有三个问题设置都有效,并且在前最新情况下均明显胜过。
translated by 谷歌翻译
我们研究了离线模仿学习(IL)的问题,在该问题中,代理商旨在学习最佳的专家行为政策,而无需其他在线环境互动。取而代之的是,该代理来自次优行为的补充离线数据集。解决此问题的先前工作要么要求专家数据占据离线数据集的大部分比例,要么需要学习奖励功能并在以后执行离线加强学习(RL)。在本文中,我们旨在解决问题,而无需进行奖励学习和离线RL培训的其他步骤,当时示范包含大量次优数据。基于行为克隆(BC),我们引入了一个额外的歧视者,以区分专家和非专家数据。我们提出了一个合作框架,以增强这两个任务的学习,基于此框架,我们设计了一种新的IL算法,其中歧视者的输出是BC损失的权重。实验结果表明,与基线算法相比,我们提出的算法可获得更高的回报和更快的训练速度。
translated by 谷歌翻译
增强学习(RL)算法假设用户通过手动编写奖励函数来指定任务。但是,这个过程可能是费力的,需要相当大的技术专长。我们可以设计RL算法,而是通过提供成功结果的示例来支持用户来指定任务吗?在本文中,我们推导了一种控制算法,可以最大化这些成功结果示例的未来概率。在前阶段的工作已经接近了类似的问题,首先学习奖励功能,然后使用另一个RL算法优化此奖励功能。相比之下,我们的方法直接从过渡和成功的结果中学习价值函数,而无需学习此中间奖励功能。因此,我们的方法需要较少的封闭式曲折和调试的代码行。我们表明我们的方法满足了一种新的数据驱动Bellman方程,其中示例取代了典型的奖励函数术语。实验表明,我们的方法优于学习明确奖励功能的先前方法。
translated by 谷歌翻译
在线模仿学习是如何最好地访问环境或准确的模拟器的问题的问题。先前的工作表明,在无限的样本制度中,匹配的确切力矩达到了与专家政策的价值等效性。但是,在有限的样本制度中,即使没有优化错误,经验差异也会导致性能差距,该差距以$ h^2 / n $的行为克隆缩放,在线时刻$ h / \ sqrt {n} $匹配,其中$ h $是地平线,$ n $是专家数据集的大小。我们介绍了重播估算的技术以减少这种经验差异:通过反复在随机模拟器中执行缓存的专家动作,我们计算了一个更平滑的专家访问分布估算以匹配的。在存在一般函数近似的情况下,我们证明了一个元定理,可以减少离线分类参数估计误差的方法差距(即学习专家策略)。在表格设置或使用线性函数近似中,我们的元定理表明,我们方法产生的性能差距达到了最佳$ \ widetilde {o} \ left(\ min(\ min({h^h^{3/2}}}} / {n} ,{h} / {\ sqrt {n}} \ right)$依赖关系,在与先前的工作相比明显弱的假设下。我们在多个连续的控制任务上实施了多个方法的多次实例化,并发现我们能够显着提高策略绩效跨各种数据集尺寸。
translated by 谷歌翻译
Inferring reward functions from human behavior is at the center of value alignment - aligning AI objectives with what we, humans, actually want. But doing so relies on models of how humans behave given their objectives. After decades of research in cognitive science, neuroscience, and behavioral economics, obtaining accurate human models remains an open research topic. This begs the question: how accurate do these models need to be in order for the reward inference to be accurate? On the one hand, if small errors in the model can lead to catastrophic error in inference, the entire framework of reward learning seems ill-fated, as we will never have perfect models of human behavior. On the other hand, if as our models improve, we can have a guarantee that reward accuracy also improves, this would show the benefit of more work on the modeling side. We study this question both theoretically and empirically. We do show that it is unfortunately possible to construct small adversarial biases in behavior that lead to arbitrarily large errors in the inferred reward. However, and arguably more importantly, we are also able to identify reasonable assumptions under which the reward inference error can be bounded linearly in the error in the human model. Finally, we verify our theoretical insights in discrete and continuous control tasks with simulated and human data.
translated by 谷歌翻译
逆强化学习(IRL)试图推断出一种成本函数,以解释专家演示的基本目标和偏好。本文介绍了向后的地平线逆增强学习(RHIRL),这是一种新的IRL算法,用于具有黑盒动态模型的高维,嘈杂,连续的系统。 Rhirl解决了IRL的两个主要挑战:可伸缩性和鲁棒性。为了处理高维的连续系统,Rhirl以退缩的地平线方式与当地的专家演示相匹配,并将其“针迹”一起“缝制”本地解决方案以学习成本;因此,它避免了“维度的诅咒”。这与早期的算法形成鲜明对比,这些算法与在整个高维状态空间中与全球范围内的专家示威相匹配。为了与不完美的专家示范和系统控制噪声保持强大的态度,Rhirl在轻度条件下学习了与系统动力学的状态依赖性成本函数。基准任务的实验表明,在大多数情况下,Rhirl的表现都优于几种领先的IRL算法。我们还证明,Rhirl的累积误差随任务持续时间线性增长。
translated by 谷歌翻译
离线目标条件的强化学习(GCRL)承诺以从纯粹的离线数据集实现各种目标的形式的通用技能学习。我们提出$ \ textbf {go} $ al-al-conditioned $ f $ - $ \ textbf {a} $ dvantage $ \ textbf {r} $ egression(gofar),这是一种基于新颖的回归gcrl gcrl algorithm,它源自州越来越多匹配的视角;关键的直觉是,可以将目标任务提出为守护动态的模仿者和直接传送到目标的专家代理之间的状态占用匹配问题。与先前的方法相反,Gofar不需要任何事后重新标签,并且对其价值和策略网络享有未融合的优化。这些独特的功能允许Gofar具有更好的离线性能和稳定性以及统计性能保证,这对于先前的方法无法实现。此外,我们证明了Gofar的训练目标可以重新使用,以从纯粹的离线源数据域数据中学习独立于代理的目标条件计划的计划者,这可以使零射击传输到新的目标域。通过广泛的实验,我们验证了Gofar在各种问题设置和任务中的有效性,显着超过了先前的先验。值得注意的是,在真正的机器人灵活性操纵任务上,虽然没有其他方法取得了有意义的进步,但Gofar获得了成功实现各种目标的复杂操纵行为。
translated by 谷歌翻译
表示学习通常通过管理维度的诅咒在加强学习中起关键作用。代表性的算法类别利用了随机过渡动力学的光谱分解,以构建在理想化环境中具有强大理论特性的表示。但是,当前的光谱方法的适用性有限,因为它们是用于仅国家的聚合并源自策略依赖性过渡内核的,而无需考虑勘探问题。为了解决这些问题,我们提出了一种替代光谱方法,光谱分解表示(SPEDER),该方法从动力学中提取了国家行动抽象而不诱导虚假依赖数据收集策略,同时还可以平衡探索访问权分析交易 - 在学习过程中关闭。理论分析确定了在线和离线设置中所提出的算法的样本效率。此外,一项实验研究表明,在几个基准测试中,比当前的最新算法表现出色。
translated by 谷歌翻译
在加强学习(RL)中,如果给出良好的表示,则更容易解决任务。尽管Deep RL应该自动获得如此良好的表示形式,但先前的工作经常发现以端到端方式学习表示不稳定,而是为RL算法配备了其他表示零件(例如,辅助损失,数据增强)。我们如何设计直接获得良好表示形式的RL算法?在本文中,我们可以表明(对比)表示方法可以将表示零件添加到现有的RL算法中,而是可以将其作为RL算法施加。为此,我们以先前的工作为基础,并将对比度表示学习应用于行动标记的轨迹,以至于学会表示的(内部产品)完全与目标条件的价值函数相对应。我们使用此想法来重新解释先前的RL方法作为执行对比学习,然后使用该想法提出一种更简单的方法,可以实现相似的性能。在一系列具有目标条件的RL任务中,我们证明了对比的RL方法比以前的非对抗性方法(包括在离线RL设置)中获得更高的成功率。我们还表明,对比度RL在不使用数据增强或辅助目标的情况下优于基于图像的任务的先验方法。
translated by 谷歌翻译
Behavioural cloning (BC) is a commonly used imitation learning method to infer a sequential decision-making policy from expert demonstrations. However, when the quality of the data is not optimal, the resulting behavioural policy also performs sub-optimally once deployed. Recently, there has been a surge in offline reinforcement learning methods that hold the promise to extract high-quality policies from sub-optimal historical data. A common approach is to perform regularisation during training, encouraging updates during policy evaluation and/or policy improvement to stay close to the underlying data. In this work, we investigate whether an offline approach to improving the quality of the existing data can lead to improved behavioural policies without any changes in the BC algorithm. The proposed data improvement approach - Trajectory Stitching (TS) - generates new trajectories (sequences of states and actions) by `stitching' pairs of states that were disconnected in the original data and generating their connecting new action. By construction, these new transitions are guaranteed to be highly plausible according to probabilistic models of the environment, and to improve a state-value function. We demonstrate that the iterative process of replacing old trajectories with new ones incrementally improves the underlying behavioural policy. Extensive experimental results show that significant performance gains can be achieved using TS over BC policies extracted from the original data. Furthermore, using the D4RL benchmarking suite, we demonstrate that state-of-the-art results are obtained by combining TS with two existing offline learning methodologies reliant on BC, model-based offline planning (MBOP) and policy constraint (TD3+BC).
translated by 谷歌翻译
我们研究了离线加强学习(RL)的代表性学习,重点是离线政策评估(OPE)的重要任务。最近的工作表明,与监督的学习相反,Q功能的可实现性不足以学习。样品效率OPE的两个足够条件是Bellman的完整性和覆盖范围。先前的工作通常假设给出满足这些条件的表示形式,结果大多是理论上的。在这项工作中,我们提出了BCRL,该BCRL直接从数据中吸取了近似线性的贝尔曼完整表示,并具有良好的覆盖范围。通过这种学识渊博的表示,我们使用最小平方策略评估(LSPE)执行OPE,并在我们学习的表示中具有线性函数。我们提出了端到端的理论分析,表明我们的两阶段算法享有多项式样本复杂性,该算法在所考虑的丰富类别中提供了一些表示形式,这是线性的贝尔曼完成。从经验上讲,我们广泛评估了我们的DeepMind Control Suite的具有挑战性的基于图像的连续控制任务。我们显示我们的表示能够与针对非政策RL开发的先前表示的学习方法(例如Curl,SPR)相比,可以更好地使用OPE。 BCRL使用最先进的方法拟合Q评估(FQE)实现竞争性OPE误差,并在评估超出初始状态分布的评估时击败FQE。我们的消融表明,我们方法的线性铃铛完整和覆盖范围都至关重要。
translated by 谷歌翻译
Effectively leveraging large, previously collected datasets in reinforcement learning (RL) is a key challenge for large-scale real-world applications. Offline RL algorithms promise to learn effective policies from previously-collected, static datasets without further interaction. However, in practice, offline RL presents a major challenge, and standard off-policy RL methods can fail due to overestimation of values induced by the distributional shift between the dataset and the learned policy, especially when training on complex and multi-modal data distributions. In this paper, we propose conservative Q-learning (CQL), which aims to address these limitations by learning a conservative Q-function such that the expected value of a policy under this Q-function lower-bounds its true value. We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees. In practice, CQL augments the standard Bellman error objective with a simple Q-value regularizer which is straightforward to implement on top of existing deep Q-learning and actor-critic implementations. On both discrete and continuous control domains, we show that CQL substantially outperforms existing offline RL methods, often learning policies that attain 2-5 times higher final return, especially when learning from complex and multi-modal data distributions.Preprint. Under review.
translated by 谷歌翻译
Adversarial Imitation Learning (AIL) is a class of popular state-of-the-art Imitation Learning algorithms commonly used in robotics. In AIL, an artificial adversary's misclassification is used as a reward signal that is optimized by any standard Reinforcement Learning (RL) algorithm. Unlike most RL settings, the reward in AIL is $differentiable$ but current model-free RL algorithms do not make use of this property to train a policy. The reward is AIL is also shaped since it comes from an adversary. We leverage the differentiability property of the shaped AIL reward function and formulate a class of Actor Residual Critic (ARC) RL algorithms. ARC algorithms draw a parallel to the standard Actor-Critic (AC) algorithms in RL literature and uses a residual critic, $C$ function (instead of the standard $Q$ function) to approximate only the discounted future return (excluding the immediate reward). ARC algorithms have similar convergence properties as the standard AC algorithms with the additional advantage that the gradient through the immediate reward is exact. For the discrete (tabular) case with finite states, actions, and known dynamics, we prove that policy iteration with $C$ function converges to an optimal policy. In the continuous case with function approximation and unknown dynamics, we experimentally show that ARC aided AIL outperforms standard AIL in simulated continuous-control and real robotic manipulation tasks. ARC algorithms are simple to implement and can be incorporated into any existing AIL implementation with an AC algorithm. Video and link to code are available at: https://sites.google.com/view/actor-residual-critic.
translated by 谷歌翻译
基于生成的对抗网络用于模仿学习的方法是有希望的,因为它们在专家演示方面是有效的样本。但是,培训生成器需要与实际环境进行许多交互,因为采用了无模型的强化学习来更新策略。为了使用基于模型的增强学习提高样品效率,我们在熵调控的马尔可夫决策过程中提出了基于模型的熵调查模仿学习(MB-eril),以减少与实际环境的相互作用数量。 MB-eril使用两个歧视因子。策略歧视者将机器人与专家的动作区分开来,模型歧视者区分了由模型产生的反事实状态转变与实际模型的转变。我们得出结构化的歧视者,以便学习政策和模型是有效的。计算机模拟和实际机器人实验表明,与基线方法相比,MB-eril实现了竞争性能,并显着提高了样品效率。
translated by 谷歌翻译
Model-based reinforcement learning (RL) methods are appealing in the offline setting because they allow an agent to reason about the consequences of actions without interacting with the environment. Prior methods learn a 1-step dynamics model, which predicts the next state given the current state and action. These models do not immediately tell the agent which actions to take, but must be integrated into a larger RL framework. Can we model the environment dynamics in a different way, such that the learned model does directly indicate the value of each action? In this paper, we propose Contrastive Value Learning (CVL), which learns an implicit, multi-step model of the environment dynamics. This model can be learned without access to reward functions, but nonetheless can be used to directly estimate the value of each action, without requiring any TD learning. Because this model represents the multi-step transitions implicitly, it avoids having to predict high-dimensional observations and thus scales to high-dimensional tasks. Our experiments demonstrate that CVL outperforms prior offline RL methods on complex continuous control benchmarks.
translated by 谷歌翻译
样本效率对于仿制学习方法来说至关重要,以适用于现实世界应用。许多研究通过延长对抗性模仿的违法行为来提高样本效率,无论这些违规延迟是否可以改变原始目标或涉及复杂的优化。我们重新审视对抗性模仿的基础,并提出了一种不需要对抗性培训或最小最大优化的脱营式样本有效方法。我们的配方在两个主要见解中大写:(1)Bellman方程和静止状态 - 动作分配方程之间的相似性使我们能够推导出一种新的时间差异(TD)学习方法; (2)使用确定性政策简化了TD学习。结合,这些见解产生了一种实用的算法,确定性和鉴别的模仿(D2仿真),其通过第一分区样本来分为两个重放缓冲区,然后通过禁止策略加强学习学习确定性政策。我们的经验结果表明,D2模仿在实现良好的样本效率方面有效,表现出对许多控制任务的对抗模仿的几种违规延伸方法。
translated by 谷歌翻译
依赖于太多的实验来学习良好的行动,目前的强化学习(RL)算法在现实世界的环境中具有有限的适用性,这可能太昂贵,无法探索探索。我们提出了一种批量RL算法,其中仅使用固定的脱机数据集来学习有效策略,而不是与环境的在线交互。批量RL中的有限数据产生了在培训数据中不充分表示的状态/行动的价值估计中的固有不确定性。当我们的候选政策从生成数据的候选政策发散时,这导致特别严重的外推。我们建议通过两个直接的惩罚来减轻这个问题:减少这种分歧的政策限制和减少过于乐观估计的价值约束。在全面的32个连续动作批量RL基准测试中,我们的方法对最先进的方法进行了比较,无论如何收集离线数据如何。
translated by 谷歌翻译