离线增强学习吸引了人们对解决传统强化学习的应用挑战的极大兴趣。离线增强学习使用先前收集的数据集来训练代理而无需任何互动。为了解决对OOD的高估（分布式）动作的高估，保守的估计值对所有输入都具有较低的价值。以前的保守估计方法通常很难避免OOD作用对Q值估计的影响。此外，这些算法通常需要失去一些计算效率，以实现保守估计的目的。在本文中，我们提出了一种简单的保守估计方法，即双重保守估计（DCE），该方法使用两种保守估计方法来限制政策。我们的算法引入了V功能，以避免分发作用的错误，同时隐含得出保守的估计。此外，我们的算法使用可控的罚款术语，改变了培训中保守主义的程度。从理论上讲，我们说明了该方法如何影响OOD动作和分布动作的估计。我们的实验分别表明，两种保守的估计方法影响了所有国家行动的估计。 DCE展示了D4RL的最新性能。

离线增强学习（RL）定义了从静态记录数据集学习的任务，而无需与环境不断交互。学识渊博的政策与行为政策之间的分配变化使得价值函数必须保持保守，以使分布（OOD）的动作不会被严重高估。但是，现有的方法，对看不见的行为进行惩罚或与行为政策进行正规化，太悲观了，这抑制了价值功能的概括并阻碍了性能的提高。本文探讨了温和但足够的保守主义，可以在线学习，同时不损害概括。我们提出了轻度保守的Q学习（MCQ），其中通过分配了适当的伪Q值来积极训练OOD。从理论上讲，我们表明MCQ诱导了至少与行为策略的行为，并且对OOD行动不会发生错误的高估。 D4RL基准测试的实验结果表明，与先前的工作相比，MCQ取得了出色的性能。此外，MCQ在从离线转移到在线时显示出卓越的概括能力，并明显胜过基准。

强化学习（RL）已在域中展示有效，在域名可以通过与其操作环境进行积极互动来学习政策。但是，如果我们将RL方案更改为脱机设置，代理商只能通过静态数据集更新其策略，其中脱机强化学习中的一个主要问题出现，即分配转移。我们提出了一种悲观的离线强化学习（PESSORL）算法，以主动引导代理通过操纵价值函数来恢复熟悉的区域。我们专注于由分销外（OOD）状态引起的问题，并且故意惩罚训练数据集中不存在的状态的高值，以便学习的悲观值函数下限界限状态空间内的任何位置。我们在各种基准任务中评估Pessorl算法，在那里我们表明我们的方法通过明确处理OOD状态，与这些方法仅考虑ood行动时，我们的方法通过明确处理OOD状态。

离线增强学习（RL）提供了一个有希望的方向，可以利用大量离线数据来实现复杂的决策任务。由于分配转移问题，当前的离线RL算法通常被设计为在价值估计和行动选择方面是保守的。但是，这种保守主义在现实情况下遇到观察偏差时，例如传感器错误和对抗性攻击时会损害学习政策的鲁棒性。为了权衡鲁棒性和保守主义，我们通过一种新颖的保守平滑技术提出了强大的离线增强学习（RORL）。在RORL中，我们明确地介绍了数据集附近国家的策略和价值函数的正则化，以及对这些OOD状态的其他保守价值估计。从理论上讲，我们表明RORL比线性MDP中的最新理论结果更紧密地构成。我们证明RORL可以在一般离线RL基准上实现最新性能，并且对对抗性观察的扰动非常强大。

Effectively leveraging large, previously collected datasets in reinforcement learning (RL) is a key challenge for large-scale real-world applications. Offline RL algorithms promise to learn effective policies from previously-collected, static datasets without further interaction. However, in practice, offline RL presents a major challenge, and standard off-policy RL methods can fail due to overestimation of values induced by the distributional shift between the dataset and the learned policy, especially when training on complex and multi-modal data distributions. In this paper, we propose conservative Q-learning (CQL), which aims to address these limitations by learning a conservative Q-function such that the expected value of a policy under this Q-function lower-bounds its true value. We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees. In practice, CQL augments the standard Bellman error objective with a simple Q-value regularizer which is straightforward to implement on top of existing deep Q-learning and actor-critic implementations. On both discrete and continuous control domains, we show that CQL substantially outperforms existing offline RL methods, often learning policies that attain 2-5 times higher final return, especially when learning from complex and multi-modal data distributions.Preprint. Under review.

Recent advances in batch (offline) reinforcement learning have shown promising results in learning from available offline data and proved offline reinforcement learning to be an essential toolkit in learning control policies in a model-free setting. An offline reinforcement learning algorithm applied to a dataset collected by a suboptimal non-learning-based algorithm can result in a policy that outperforms the behavior agent used to collect the data. Such a scenario is frequent in robotics, where existing automation is collecting operational data. Although offline learning techniques can learn from data generated by a sub-optimal behavior agent, there is still an opportunity to improve the sample complexity of existing offline reinforcement learning algorithms by strategically introducing human demonstration data into the training process. To this end, we propose a novel approach that uses uncertainty estimation to trigger the injection of human demonstration data and guide policy training towards optimal behavior while reducing overall sample complexity. Our experiments show that this approach is more sample efficient when compared to a naive way of combining expert data with data collected from a sub-optimal agent. We augmented an existing offline reinforcement learning algorithm Conservative Q-Learning with our approach and performed experiments on data collected from MuJoCo and OffWorld Gym learning environments.

We present state advantage weighting for offline reinforcement learning (RL). In contrast to action advantage $A(s,a)$ that we commonly adopt in QSA learning, we leverage state advantage $A(s,s^\prime)$ and QSS learning for offline RL, hence decoupling the action from values. We expect the agent can get to the high-reward state and the action is determined by how the agent can get to that corresponding state. Experiments on D4RL datasets show that our proposed method can achieve remarkable performance against the common baselines. Furthermore, our method shows good generalization capability when transferring from offline to online.

Offline reinforcement learning (RL) promises the ability to learn effective policies solely using existing, static datasets, without any costly online interaction. To do so, offline RL methods must handle distributional shift between the dataset and the learned policy. The most common approach is to learn conservative, or lower-bound, value functions, which underestimate the return of out-of-distribution (OOD) actions. However, such methods exhibit one notable drawback: policies optimized on such value functions can only behave according to a fixed, possibly suboptimal, degree of conservatism. However, this can be alleviated if we instead are able to learn policies for varying degrees of conservatism at training time and devise a method to dynamically choose one of them during evaluation. To do so, in this work, we propose learning value functions that additionally condition on the degree of conservatism, which we dub confidence-conditioned value functions. We derive a new form of a Bellman backup that simultaneously learns Q-values for any degree of confidence with high probability. By conditioning on confidence, our value functions enable adaptive strategies during online evaluation by controlling for confidence level using the history of observations thus far. This approach can be implemented in practice by conditioning the Q-function from existing conservative algorithms on the confidence. We theoretically show that our learned value functions produce conservative estimates of the true value at any desired confidence. Finally, we empirically show that our algorithm outperforms existing conservative offline RL algorithms on multiple discrete control domains.

Behavioural cloning (BC) is a commonly used imitation learning method to infer a sequential decision-making policy from expert demonstrations. However, when the quality of the data is not optimal, the resulting behavioural policy also performs sub-optimally once deployed. Recently, there has been a surge in offline reinforcement learning methods that hold the promise to extract high-quality policies from sub-optimal historical data. A common approach is to perform regularisation during training, encouraging updates during policy evaluation and/or policy improvement to stay close to the underlying data. In this work, we investigate whether an offline approach to improving the quality of the existing data can lead to improved behavioural policies without any changes in the BC algorithm. The proposed data improvement approach - Trajectory Stitching (TS) - generates new trajectories (sequences of states and actions) by `stitching' pairs of states that were disconnected in the original data and generating their connecting new action. By construction, these new transitions are guaranteed to be highly plausible according to probabilistic models of the environment, and to improve a state-value function. We demonstrate that the iterative process of replacing old trajectories with new ones incrementally improves the underlying behavioural policy. Extensive experimental results show that significant performance gains can be achieved using TS over BC policies extracted from the original data. Furthermore, using the D4RL benchmarking suite, we demonstrate that state-of-the-art results are obtained by combining TS with two existing offline learning methodologies reliant on BC, model-based offline planning (MBOP) and policy constraint (TD3+BC).

依赖于太多的实验来学习良好的行动，目前的强化学习（RL）算法在现实世界的环境中具有有限的适用性，这可能太昂贵，无法探索探索。我们提出了一种批量RL算法，其中仅使用固定的脱机数据集来学习有效策略，而不是与环境的在线交互。批量RL中的有限数据产生了在培训数据中不充分表示的状态/行动的价值估计中的固有不确定性。当我们的候选政策从生成数据的候选政策发散时，这导致特别严重的外推。我们建议通过两个直接的惩罚来减轻这个问题：减少这种分歧的政策限制和减少过于乐观估计的价值约束。在全面的32个连续动作批量RL基准测试中，我们的方法对最先进的方法进行了比较，无论如何收集离线数据如何。

Deep reinforcement learning (DRL) provides a new way to generate robot control policy. However, the process of training control policy requires lengthy exploration, resulting in a low sample efficiency of reinforcement learning (RL) in real-world tasks. Both imitation learning (IL) and learning from demonstrations (LfD) improve the training process by using expert demonstrations, but imperfect expert demonstrations can mislead policy improvement. Offline to Online reinforcement learning requires a lot of offline data to initialize the policy, and distribution shift can easily lead to performance degradation during online fine-tuning. To solve the above problems, we propose a learning from demonstrations method named A-SILfD, which treats expert demonstrations as the agent's successful experiences and uses experiences to constrain policy improvement. Furthermore, we prevent performance degradation due to large estimation errors in the Q-function by the ensemble Q-functions. Our experiments show that A-SILfD can significantly improve sample efficiency using a small number of different quality expert demonstrations. In four Mujoco continuous control tasks, A-SILfD can significantly outperform baseline methods after 150,000 steps of online training and is not misled by imperfect expert demonstrations during training.

Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q-learning and actor-critic methods are highly sensitive to the data distribution, and can make only limited progress without collecting additional on-policy data. As a step towards more robust off-policy algorithms, we study the setting where the off-policy experience is fixed and there is no further interaction with the environment. We identify bootstrapping error as a key source of instability in current methods. Bootstrapping error is due to bootstrapping from actions that lie outside of the training data distribution, and it accumulates via the Bellman backup operator. We theoretically analyze bootstrapping error, and demonstrate how carefully constraining action selection in the backup can mitigate it. Based on our analysis, we propose a practical algorithm, bootstrapping error accumulation reduction (BEAR). We demonstrate that BEAR is able to learn robustly from different off-policy distributions, including random and suboptimal demonstrations, on a range of continuous control tasks.

Offline reinforcement learning (RL) refers to the problem of learning policies entirely from a large batch of previously collected data. This problem setting offers the promise of utilizing such datasets to acquire policies without any costly or dangerous active exploration. However, it is also challenging, due to the distributional shift between the offline training data and those states visited by the learned policy. Despite significant recent progress, the most successful prior methods are model-free and constrain the policy to the support of data, precluding generalization to unseen states. In this paper, we first observe that an existing model-based RL algorithm already produces significant gains in the offline setting compared to model-free approaches. However, standard model-based RL methods, designed for the online setting, do not provide an explicit mechanism to avoid the offline setting's distributional shift issue. Instead, we propose to modify the existing model-based RL methods by applying them with rewards artificially penalized by the uncertainty of the dynamics. We theoretically show that the algorithm maximizes a lower bound of the policy's return under the true MDP. We also characterize the trade-off between the gain and risk of leaving the support of the batch data. Our algorithm, Model-based Offline Policy Optimization (MOPO), outperforms standard model-based RL algorithms and prior state-of-the-art model-free offline RL algorithms on existing offline RL benchmarks and two challenging continuous control tasks that require generalizing from data collected for a different task. * equal contribution. † equal advising. Orders randomized.34th Conference on Neural Information Processing Systems (NeurIPS 2020),

离线强化学习用于在实时访问环境昂贵或不可能的情况下培训策略。作为这些恶劣条件的自然后果，在采取行动之前，代理商可能缺乏完全遵守在线环境的资源。我们配备了这种情况资源受限的设置。这导致脱机数据集（可用于培训）的情况可以包含完全处理的功能（使用功能强大的语言模型，图像模型，复杂传感器等）在实际在线时不可用。此断开连接导致离线RL中的有趣和未开发的问题：是否可以使用丰富地处理的脱机数据集来培训可访问在线环境中的更少功能的策略？在这项工作中，我们介绍并正式化这一新颖的资源受限的问题设置。我们突出了使用有限功能培训的完整脱机数据集和策略培训的策略之间的性能差距。我们通过策略传输算法解决了这种性能缺口，该策略传输算法首先使用功能完全可用的脱机数据集列举教师代理，然后将此知识传输到仅使用资源约束功能的学生代理。为了更好地捕获此设置的挑战，我们提出了一个数据收集过程：RL（RC-D4RL）的资源受限数据集。我们在RC-D4RL和流行的D4RL基准测试中评估传输算法，并观察到基线上的一致性改进（无需传输）。实验的代码在https://github.com/jayanthrr /rc-offlinerl上获得。

在没有高保真模拟环境的情况下，学习有效的加强学习（RL）政策可以解决现实世界中的复杂任务。在大多数情况下，我们只有具有简化动力学的不完善的模拟器，这不可避免地导致RL策略学习中的SIM到巨大差距。最近出现的离线RL领域为直接从预先收集的历史数据中学习政策提供了另一种可能性。但是，为了达到合理的性能，现有的离线RL算法需要不切实际的离线数据，并具有足够的州行动空间覆盖范围进行培训。这提出了一个新问题：是否有可能通过在线RL中的不完美模拟器中的离线RL中的有限数据中的学习结合到无限制的探索，以解决两种方法的缺点？在这项研究中，我们提出了动态感知的混合离线和对线增强学习（H2O）框架，以为这个问题提供肯定的答案。 H2O引入了动态感知的政策评估方案，该方案可以自适应地惩罚Q函数在模拟的状态行动对上具有较大的动态差距，同时也允许从固定的现实世界数据集中学习。通过广泛的模拟和现实世界任务以及理论分析，我们证明了H2O与其他跨域在线和离线RL算法相对于其他跨域的表现。 H2O提供了全新的脱机脱机RL范式，该范式可能会阐明未来的RL算法设计，以解决实用的现实世界任务。

在离线强化学习（离线RL）中，主要挑战之一是处理学习策略与给定数据集之间的分布转变。为了解决这个问题，最近的离线RL方法试图引入保守主义偏见，以鼓励在高信心地区学习。无模型方法使用保守的正常化或特殊网络结构直接对策略或价值函数学习进行这样的偏见，但它们约束的策略搜索限制了脱机数据集之外的泛化。基于模型的方法使用保守量量化学习前瞻性动态模型，然后生成虚构的轨迹以扩展脱机数据集。然而，由于离线数据集中的有限样本，保守率量化通常在支撑区域内遭受全面化。不可靠的保守措施将误导基于模型的想象力，以不受欢迎的地区，导致过多的行为。为了鼓励更多的保守主义，我们提出了一种基于模型的离线RL框架，称为反向离线模型的想象（ROMI）。我们与新颖的反向策略结合使用逆向动力学模型，该模型可以生成导致脱机数据集中的目标目标状态的卷展栏。这些反向的想象力提供了无通知的数据增强，以便无模型策略学习，并使远程数据集的保守概括。 ROMI可以有效地与现成的无模型算法组合，以实现基于模型的概括，具有适当的保守主义。经验结果表明，我们的方法可以在离线RL基准任务中产生更保守的行为并实现最先进的性能。

离线强化学习（RL）定义了从固定批次的数据学习的任务。由于来自分发超出操作的值估计的错误，大多数脱机RL算法采用数据集中包含的动作来计算或正规化策略的方法。构建在预先存在的RL算法上，修改使RL算法正常工作的额外复杂性的成本为代价。离线RL算法引入了新的超参数，通常利用辅助组件，例如生成模型，同时调整底层RL算法。在本文中，我们的目标是在实现最小变化的同时进行深度RL算法。我们发现我们可以通过简单地将行为克隆术语添加到在线RL算法的策略更新并归一化数据的策略更新来匹配最先进的离线RL算法的性能。生成的算法是一种简单的实现和曲线基线，而通过去除先前方法的附加计算开销来大于缩短整个运行时间。

不确定性在游戏中无处不在，无论是在玩游戏的代理商还是在游戏本身中。因此，不确定性是成功深入强化学习剂的重要组成部分。尽管在理解和处理监督学习的不确定性方面已经做出了巨大的努力和进展，但不确定性的文献意识到深度强化学习的发展却较少。尽管有关监督学习的神经网络中的不确定性的许多相同问题仍然用于强化学习，但由于可相互作用的环境的性质，还有其他不确定性来源。在这项工作中，我们提供了一个激励和介绍不确定性深入强化学习的现有技术的概述。这些作品在各种强化学习任务上显示出经验益处。这项工作有助于集中不同的结果并促进该领域的未来研究。

离线增强学习（RL）将经典RL算法的范式扩展到纯粹从静态数据集中学习，而无需在学习过程中与基础环境进行交互。离线RL的一个关键挑战是政策培训的不稳定，这是由于离线数据的分布与学习政策的未结束的固定状态分配之间的不匹配引起的。为了避免分配不匹配的有害影响，我们将当前政策的未静置固定分配正规化在政策优化过程中的离线数据。此外，我们训练动力学模型既实施此正规化，又可以更好地估计当前策略的固定分布，从而减少了分布不匹配引起的错误。在各种连续控制的离线RL数据集中，我们的方法表示竞争性能，从而验证了我们的算法。该代码公开可用。

推荐系统（RS）是一个重要的在线应用程序，每天都会影响数十亿个用户。主流RS排名框架由两个部分组成：多任务学习模型（MTL），该模型可预测各种用户反馈，即点击，喜欢，分享和多任务融合模型（MTF），该模型（MTF）结合了多任务就用户满意度而言，输出分为最终排名得分。关于融合模型的研究并不多，尽管它对最终建议作为排名的最后一个关键过程有很大的影响。为了优化长期用户满意度，而不是贪婪地获得即时回报，我们将MTF任务作为Markov决策过程（MDP），并在推荐会话中提出，并建议基于批处理加固学习（RL）基于多任务融合框架（BATCHRL-MTF）包括批处理RL框架和在线探索。前者利用批处理RL从固定的批处理数据离线学习最佳推荐政策，以达到长期用户满意度，而后者则探索了潜在的高价值动作在线，以突破本地最佳难题。通过对用户行为的全面调查，我们通过从用户粘性和用户活动性的两个方面的微妙启发式方法对用户满意度进行了建模。最后，我们对十亿个样本级别的现实数据集进行了广泛的实验，以显示模型的有效性。我们建议保守的离线政策估计器（保守 - 访问器）来测试我们的模型离线。此外，我们在真实推荐环境中进行在线实验，以比较不同模型的性能。作为成功在MTF任务中应用的少数批次RL研究之一，我们的模型也已部署在一个大规模的工业短视频平台上，为数亿用户提供服务。