大多数前往离线强化学习（RL）的方法都采取了一种迭代演员 - 批评批评，涉及违规评估。在本文中，我们展示了使用行为政策的政策Q估计来令人惊讶地执行一步的Q估计，从而简单地执行一个受限制/正规化的政策改进的步骤。该一步算法在大部分D4RL基准测试中击败了先前报告的迭代算法的结果。一步基线实现了这种强劲的性能，同时对超公数更简单，更强大而不是先前提出的迭代算法。我们认为迭代方法的表现相对较差是在违反政策评估中固有的高方差，并通过对这些估计的重复优化的政策进行放大。此外，我们假设一步算法的强大性能是由于环境和行为政策中有利结构的组合。

我们介绍了Smianile过滤仿制学习（QFIL），这是一种用于离线强化学习的新型政策改进操作员。QFIL通过在脱机数据集的过滤版本上运行模仿学习来执行策略改进。过滤过程删除了$ s，其估计的q值低于给定分位于通过从行为策略采样动作引起的值的推送分布。推轴Q分布和产生的值函数分位数的定义是我们方法的主要贡献。我们证明QFIL为我们提供了一种安全的政策改进步骤，函数近似，分位式的选择提供了自然的超参数，以折衷偏差和改进步骤的差异。凭经验，我们执行一个合成实验，说明QFIL如何有效地进行偏差方差权衡，并且我们看到QFil在D4RL基准上表现良好。

Effectively leveraging large, previously collected datasets in reinforcement learning (RL) is a key challenge for large-scale real-world applications. Offline RL algorithms promise to learn effective policies from previously-collected, static datasets without further interaction. However, in practice, offline RL presents a major challenge, and standard off-policy RL methods can fail due to overestimation of values induced by the distributional shift between the dataset and the learned policy, especially when training on complex and multi-modal data distributions. In this paper, we propose conservative Q-learning (CQL), which aims to address these limitations by learning a conservative Q-function such that the expected value of a policy under this Q-function lower-bounds its true value. We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees. In practice, CQL augments the standard Bellman error objective with a simple Q-value regularizer which is straightforward to implement on top of existing deep Q-learning and actor-critic implementations. On both discrete and continuous control domains, we show that CQL substantially outperforms existing offline RL methods, often learning policies that attain 2-5 times higher final return, especially when learning from complex and multi-modal data distributions.Preprint. Under review.

依赖于太多的实验来学习良好的行动，目前的强化学习（RL）算法在现实世界的环境中具有有限的适用性，这可能太昂贵，无法探索探索。我们提出了一种批量RL算法，其中仅使用固定的脱机数据集来学习有效策略，而不是与环境的在线交互。批量RL中的有限数据产生了在培训数据中不充分表示的状态/行动的价值估计中的固有不确定性。当我们的候选政策从生成数据的候选政策发散时，这导致特别严重的外推。我们建议通过两个直接的惩罚来减轻这个问题：减少这种分歧的政策限制和减少过于乐观估计的价值约束。在全面的32个连续动作批量RL基准测试中，我们的方法对最先进的方法进行了比较，无论如何收集离线数据如何。

Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q-learning and actor-critic methods are highly sensitive to the data distribution, and can make only limited progress without collecting additional on-policy data. As a step towards more robust off-policy algorithms, we study the setting where the off-policy experience is fixed and there is no further interaction with the environment. We identify bootstrapping error as a key source of instability in current methods. Bootstrapping error is due to bootstrapping from actions that lie outside of the training data distribution, and it accumulates via the Bellman backup operator. We theoretically analyze bootstrapping error, and demonstrate how carefully constraining action selection in the backup can mitigate it. Based on our analysis, we propose a practical algorithm, bootstrapping error accumulation reduction (BEAR). We demonstrate that BEAR is able to learn robustly from different off-policy distributions, including random and suboptimal demonstrations, on a range of continuous control tasks.

深度强化学习（RL）导致了许多最近和开创性的进步。但是，这些进步通常以培训的基础体系结构的规模增加以及用于训练它们的RL算法的复杂性提高，而均以增加规模的成本。这些增长反过来又使研究人员更难迅速原型新想法或复制已发表的RL算法。为了解决这些问题，这项工作描述了ACME，这是一个用于构建新型RL算法的框架，这些框架是专门设计的，用于启用使用简单的模块化组件构建的代理，这些组件可以在各种执行范围内使用。尽管ACME的主要目标是为算法开发提供一个框架，但第二个目标是提供重要或最先进算法的简单参考实现。这些实现既是对我们的设计决策的验证，也是对RL研究中可重复性的重要贡献。在这项工作中，我们描述了ACME内部做出的主要设计决策，并提供了有关如何使用其组件来实施各种算法的进一步详细信息。我们的实验为许多常见和最先进的算法提供了基准，并显示了如何为更大且更复杂的环境扩展这些算法。这突出了ACME的主要优点之一，即它可用于实现大型，分布式的RL算法，这些算法可以以较大的尺度运行，同时仍保持该实现的固有可读性。这项工作提出了第二篇文章的版本，恰好与模块化的增加相吻合，对离线，模仿和从演示算法学习以及作为ACME的一部分实现的各种新代理。

离线增强学习（RL）将经典RL算法的范式扩展到纯粹从静态数据集中学习，而无需在学习过程中与基础环境进行交互。离线RL的一个关键挑战是政策培训的不稳定，这是由于离线数据的分布与学习政策的未结束的固定状态分配之间的不匹配引起的。为了避免分配不匹配的有害影响，我们将当前政策的未静置固定分配正规化在政策优化过程中的离线数据。此外，我们训练动力学模型既实施此正规化，又可以更好地估计当前策略的固定分布，从而减少了分布不匹配引起的错误。在各种连续控制的离线RL数据集中，我们的方法表示竞争性能，从而验证了我们的算法。该代码公开可用。

Behavior constrained policy optimization has been demonstrated to be a successful paradigm for tackling Offline Reinforcement Learning. By exploiting historical transitions, a policy is trained to maximize a learned value function while constrained by the behavior policy to avoid a significant distributional shift. In this paper, we propose our closed-form policy improvement operators. We make a novel observation that the behavior constraint naturally motivates the use of first-order Taylor approximation, leading to a linear approximation of the policy objective. Additionally, as practical datasets are usually collected by heterogeneous policies, we model the behavior policies as a Gaussian Mixture and overcome the induced optimization difficulties by leveraging the LogSumExp's lower bound and Jensen's Inequality, giving rise to a closed-form policy improvement operator. We instantiate offline RL algorithms with our novel policy improvement operators and empirically demonstrate their effectiveness over state-of-the-art algorithms on the standard D4RL benchmark.

In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.

我们根据相对悲观主义的概念，在数据覆盖不足的情况下提出了经过对抗训练的演员评论家（ATAC），这是一种新的无模型算法（RL）。 ATAC被设计为两人Stackelberg游戏：政策演员与受对抗训练的价值评论家竞争，后者发现参与者不如数据收集行为策略的数据一致方案。我们证明，当演员在两人游戏中不后悔时，运行ATAC会产生一项政策，证明1）在控制悲观程度的各种超级参数上都超过了行为政策，而2）与最佳竞争。 policy covered by data with appropriately chosen hyperparameters.与现有作品相比，尤其是我们的框架提供了一般函数近似的理论保证，也提供了可扩展到复杂环境和大型数据集的深度RL实现。在D4RL基准测试中，ATAC在一系列连续的控制任务上始终优于最先进的离线RL算法。

Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an offpolicy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.

Offline reinforcement learning (RL) refers to the problem of learning policies entirely from a large batch of previously collected data. This problem setting offers the promise of utilizing such datasets to acquire policies without any costly or dangerous active exploration. However, it is also challenging, due to the distributional shift between the offline training data and those states visited by the learned policy. Despite significant recent progress, the most successful prior methods are model-free and constrain the policy to the support of data, precluding generalization to unseen states. In this paper, we first observe that an existing model-based RL algorithm already produces significant gains in the offline setting compared to model-free approaches. However, standard model-based RL methods, designed for the online setting, do not provide an explicit mechanism to avoid the offline setting's distributional shift issue. Instead, we propose to modify the existing model-based RL methods by applying them with rewards artificially penalized by the uncertainty of the dynamics. We theoretically show that the algorithm maximizes a lower bound of the policy's return under the true MDP. We also characterize the trade-off between the gain and risk of leaving the support of the batch data. Our algorithm, Model-based Offline Policy Optimization (MOPO), outperforms standard model-based RL algorithms and prior state-of-the-art model-free offline RL algorithms on existing offline RL benchmarks and two challenging continuous control tasks that require generalizing from data collected for a different task. * equal contribution. † equal advising. Orders randomized.34th Conference on Neural Information Processing Systems (NeurIPS 2020),

无模型的深度增强学习（RL）已成功应用于挑战连续控制域。然而，较差的样品效率可防止这些方法广泛用于现实世界领域。我们通过提出一种新的无模型算法，现实演员 - 评论家（RAC）来解决这个问题，旨在通过学习关于Q函数的各种信任的政策家庭来解决价值低估和高估之间的权衡。我们构建不确定性惩罚Q-Learning（UPQ），该Q-Learning（UPQ）使用多个批评者的合并来控制Q函数的估计偏差，使Q函数平稳地从低于更高的置信范围偏移。随着这些批评者的指导，RAC采用通用价值函数近似器（UVFA），同时使用相同的神经网络学习许多乐观和悲观的政策。乐观的政策会产生有效的探索行为，而悲观政策会降低价值高估的风险，以确保稳定的策略更新和Q函数。该方法可以包含任何违规的演员 - 评论家RL算法。我们的方法实现了10倍的样本效率和25 \％的性能改进与SAC在最具挑战性的人形环境中，获得了11107美元的集中奖励1107美元，价格为10 ^ 6美元。所有源代码都可以在https://github.com/ihuhuhu/rac获得。

Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications. This is often because off-policy RL algorithms suffer from distributional shift, due to mismatch between dataset and the target policy, leading to high variance and over-estimation of value functions. In this work, we propose variance regularization for offline RL algorithms, using stationary distribution corrections. We show that by using Fenchel duality, we can avoid double sampling issues for computing the gradient of the variance regularizer. The proposed algorithm for offline variance regularization (OVAR) can be used to augment any existing offline policy optimization algorithms. We show that the regularizer leads to a lower bound to the offline policy optimization objective, which can help avoid over-estimation errors, and explains the benefits of our approach across a range of continuous control domains when compared to existing state-of-the-art algorithms.

离线增强学习（RL）定义了从静态记录数据集学习的任务，而无需与环境不断交互。学识渊博的政策与行为政策之间的分配变化使得价值函数必须保持保守，以使分布（OOD）的动作不会被严重高估。但是，现有的方法，对看不见的行为进行惩罚或与行为政策进行正规化，太悲观了，这抑制了价值功能的概括并阻碍了性能的提高。本文探讨了温和但足够的保守主义，可以在线学习，同时不损害概括。我们提出了轻度保守的Q学习（MCQ），其中通过分配了适当的伪Q值来积极训练OOD。从理论上讲，我们表明MCQ诱导了至少与行为策略的行为，并且对OOD行动不会发生错误的高估。 D4RL基准测试的实验结果表明，与先前的工作相比，MCQ取得了出色的性能。此外，MCQ在从离线转移到在线时显示出卓越的概括能力，并明显胜过基准。

离线增强学习（RL）可以从静态数据集中学习控制策略，但是像标准RL方法一样，它需要每个过渡的奖励注释。在许多情况下，将大型数据集标记为奖励可能会很高，尤其是如果人类标签必须提供这些奖励，同时收集多样的未标记数据可能相对便宜。我们如何在离线RL中最好地利用这种未标记的数据？一种自然解决方案是从标记的数据中学习奖励函数，并使用它标记未标记的数据。在本文中，我们发现，也许令人惊讶的是，一种简单得多的方法，它简单地将零奖励应用于未标记的数据可以导致理论和实践中的有效数据共享，而无需学习任何奖励模型。虽然这种方法起初可能看起来很奇怪（并且不正确），但我们提供了广泛的理论和经验分析，说明了它如何摆脱奖励偏见，样本复杂性和分配变化，通常会导致良好的结果。我们表征了这种简单策略有效的条件，并进一步表明，使用简单的重新加权方法扩展它可以进一步缓解通过使用不正确的奖励标签引入的偏见。我们的经验评估证实了模拟机器人运动，导航和操纵设置中的这些发现。

Behavioural cloning (BC) is a commonly used imitation learning method to infer a sequential decision-making policy from expert demonstrations. However, when the quality of the data is not optimal, the resulting behavioural policy also performs sub-optimally once deployed. Recently, there has been a surge in offline reinforcement learning methods that hold the promise to extract high-quality policies from sub-optimal historical data. A common approach is to perform regularisation during training, encouraging updates during policy evaluation and/or policy improvement to stay close to the underlying data. In this work, we investigate whether an offline approach to improving the quality of the existing data can lead to improved behavioural policies without any changes in the BC algorithm. The proposed data improvement approach - Trajectory Stitching (TS) - generates new trajectories (sequences of states and actions) by `stitching' pairs of states that were disconnected in the original data and generating their connecting new action. By construction, these new transitions are guaranteed to be highly plausible according to probabilistic models of the environment, and to improve a state-value function. We demonstrate that the iterative process of replacing old trajectories with new ones incrementally improves the underlying behavioural policy. Extensive experimental results show that significant performance gains can be achieved using TS over BC policies extracted from the original data. Furthermore, using the D4RL benchmarking suite, we demonstrate that state-of-the-art results are obtained by combining TS with two existing offline learning methodologies reliant on BC, model-based offline planning (MBOP) and policy constraint (TD3+BC).

一种被称为优先体验重播（PER）的广泛研究的深钢筋学习（RL）技术使代理可以从与其时间差异（TD）误差成正比的过渡中学习。尽管已经表明，PER是离散作用域中深度RL方法总体性能的最关键组成部分之一，但许多经验研究表明，在连续控制中，它的表现非常低于参与者 - 批评算法。从理论上讲，我们表明，无法有效地通过具有较大TD错误的过渡对演员网络进行训练。结果，在Q网络下计算的近似策略梯度与在最佳Q功能下计算的实际梯度不同。在此激励的基础上，我们引入了一种新颖的经验重播抽样框架，用于演员批评方法，该框架还认为稳定性和最新发现的问题是Per的经验表现不佳。引入的算法提出了对演员和评论家网络的有效和高效培训的改进的新分支。一系列广泛的实验验证了我们的理论主张，并证明了引入的方法显着优于竞争方法，并获得了与标准的非政策参与者 - 批评算法相比，获得最先进的结果。

离线强化学习（RL）任务要求代理从预先收集的数据集中学习，没有与环境进行进一步的交互。尽管有可能超越行为政策，但基于RL的方法通常是不切实际的，因为培训不稳定并引导外推错误，这始终需要通过在线评估进行仔细的超参数调整。相比之下，离线模仿学习（IL）没有这样的问题，因为它直接在不估计值函数的情况下直接了解策略。然而，IL通常限制在行为政策的能力，并且倾向于从政策混合收集的数据集中学习平庸行为。在本文中，我们的目标是利用IL但缓解这种缺点。观察行为克隆能够使用较少的数据模仿邻近的策略，我们提出\ Textit {课程脱机仿制学习（线圈）}，它利用具有更高回报的自适应邻近策略的体验挑选策略，并提高了当前策略沿课程阶段。在连续控制基准测试中，我们将线圈与基于仿制的和基于RL的方法进行比较，表明它不仅避免了在混合数据集上学习平庸行为，而且甚至与最先进的离线RL方法竞争。

脱机强化学习 - 从一批数据中学习策略 - 是难以努力的：如果没有制造强烈的假设，它很容易构建实体算法失败的校长。在这项工作中，我们考虑了某些现实世界问题的财产，其中离线强化学习应该有效：行动仅对一部分产生有限的行动。我们正规化并介绍此动作影响规律（AIR）财产。我们进一步提出了一种算法，该算法假定和利用AIR属性，并在MDP满足空气时绑定输出策略的子优相。最后，我们展示了我们的算法在定期保留的两个模拟环境中跨越不同的数据收集策略占据了现有的离线强度学习算法。