许多连续的决策问题是使用使用其他一些策略收集的历史数据,需要使用历史数据的高赌注并要求新策略(OPE)。提供无偏估计的最常见的OPE技术之一是基于轨迹的重要性采样(是)。但是,由于轨迹的高方差是估计,最近通过了基于国家行动探索分布(SIS)的重要性采样方法。不幸的是,虽然SIS经常为长视野提供较低的方差估计,但估算状态行动分配比可能是具有挑战性的并且导致偏差估计。在本文中,我们对该偏差差异进行了新的视角,并显示了存在终点是SIS的估计频谱的存在。此外,我们还建立了这些估算器的双重强大和加权版本的频谱。我们提供了经验证据,即该频谱中的估计值可用于在IS和SIS的偏差和方差之间进行折衷,并且可以实现比两者和SIS更低的平均平方误差。
translated by 谷歌翻译
In this paper we present a new way of predicting the performance of a reinforcement learning policy given historical data that may have been generated by a different policy. The ability to evaluate a policy from historical data is important for applications where the deployment of a bad policy can be dangerous or costly. We show empirically that our algorithm produces estimates that often have orders of magnitude lower mean squared error than existing methods-it makes more efficient use of the available data. Our new estimator is based on two advances: an extension of the doubly robust estimator (Jiang & Li, 2015), and a new way to mix between model based estimates and importance sampling based estimates.
translated by 谷歌翻译
我们为加强学习提供了实验基准和实验研究,以便在加固学习中进行违规政策评估(OPE),这是许多安全关键申请中的关键问题。鉴于部署基于学习的方法的兴趣日益越来越令人兴趣,最近的OPE方法提出了势头,导致需要标准化的经验分析。我们的工作强烈关注实验设计的多样性,以实现OPE方法的压力测试。我们提供了一个全面的基准测试套件,以研究不同属性对方法性能的相互作用。我们在实践中将结果蒸煮为OPE的概要指南。我们的软件包,Caltech Ope基准套件(COB),是开放的,我们邀请有兴趣的研究人员进一步贡献基准。
translated by 谷歌翻译
由于策略梯度定理导致的策略设置存在各种理论上 - 声音策略梯度算法,其为梯度提供了简化的形式。然而,由于存在多重目标和缺乏明确的脱助政策政策梯度定理,截止策略设置不太明确。在这项工作中,我们将这些目标统一到一个违规目标,并为此统一目标提供了政策梯度定理。推导涉及强调的权重和利息职能。我们显示多种策略来近似梯度,以识别权重(ACE)称为Actor评论家的算法。我们证明了以前(半梯度)脱离政策演员 - 评论家 - 特别是offpac和DPG - 收敛到错误的解决方案,而Ace找到最佳解决方案。我们还强调为什么这些半梯度方法仍然可以在实践中表现良好,表明ace中的方差策略。我们经验研究了两个经典控制环境的若干ACE变体和基于图像的环境,旨在说明每个梯度近似的权衡。我们发现,通过直接逼近强调权重,ACE在所有测试的所有设置中执行或优于offpac。
translated by 谷歌翻译
Reinforcement learning (RL) is one of the most vibrant research frontiers in machine learning and has been recently applied to solve a number of challenging problems. In this paper, we primarily focus on off-policy evaluation (OPE), one of the most fundamental topics in RL. In recent years, a number of OPE methods have been developed in the statistics and computer science literature. We provide a discussion on the efficiency bound of OPE, some of the existing state-of-the-art OPE methods, their statistical properties and some other related research directions that are currently actively explored.
translated by 谷歌翻译
We study the problem of off-policy value evaluation in reinforcement learning (RL), where one aims to estimate the value of a new policy based on data collected by a different policy. This problem is often a critical step when applying RL to real-world problems. Despite its importance, existing general methods either have uncontrolled bias or suffer high variance. In this work, we extend the doubly robust estimator for bandits to sequential decision-making problems, which gets the best of both worlds: it is guaranteed to be unbiased and can have a much lower variance than the popular importance sampling estimators. We demonstrate the estimator's accuracy in several benchmark problems, and illustrate its use as a subroutine in safe policy improvement. We also provide theoretical results on the inherent hardness of the problem, and show that our estimator can match the lower bound in certain scenarios.
translated by 谷歌翻译
We consider the problem of off-policy evaluation (OPE) in reinforcement learning (RL), where the goal is to estimate the performance of an evaluation policy, $\pi_e$, using a fixed dataset, $\mathcal{D}$, collected by one or more policies that may be different from $\pi_e$. Current OPE algorithms may produce poor OPE estimates under policy distribution shift i.e., when the probability of a particular state-action pair occurring under $\pi_e$ is very different from the probability of that same pair occurring in $\mathcal{D}$ (Voloshin et al. 2021, Fu et al. 2021). In this work, we propose to improve the accuracy of OPE estimators by projecting the high-dimensional state-space into a low-dimensional state-space using concepts from the state abstraction literature. Specifically, we consider marginalized importance sampling (MIS) OPE algorithms which compute state-action distribution correction ratios to produce their OPE estimate. In the original ground state-space, these ratios may have high variance which may lead to high variance OPE. However, we prove that in the lower-dimensional abstract state-space the ratios can have lower variance resulting in lower variance OPE. We then highlight the challenges that arise when estimating the abstract ratios from data, identify sufficient conditions to overcome these issues, and present a minimax optimization problem whose solution yields these abstract ratios. Finally, our empirical evaluation on difficult, high-dimensional state-space OPE tasks shows that the abstract ratios can make MIS OPE estimators achieve lower mean-squared error and more robust to hyperparameter tuning than the ground ratios.
translated by 谷歌翻译
我们考虑在部分可观察到的马尔可夫决策过程(POMDP)中的违法评估(OPE),其中评估策略仅取决于可观察变量,并且行为策略取决于不可观察的潜在变量。现有的作品无论是假设未测量的混乱,还是专注于观察和状态空间都是表格的设置。因此,这些方法在存在未测量的混淆器的情况下遭受大偏差,或者在具有连续或大观察/状态空间的设置中的大方差。在这项工作中,通过引入将目标策略的价值和观察到的数据分布联系起来,提出了具有潜在混淆的POMDPS的新识别方法。在完全可观察到的MDP中,这些桥接功能将熟悉的值函数和评估与行为策略之间的边际密度比减少。我们接下来提出了用于学习这些桥接功能的最小值估计方法。我们的提案允许一般函数近似,因此适用于具有连续或大观察/状态空间的设置。最后,我们基于这些估计的桥梁功能构建了三种估计,对应于基于价值函数的估计器,边缘化重要性采样估计器和双重稳健的估计器。他们的掺入无血症和渐近性质进行了详细研究。
translated by 谷歌翻译
本文考虑如何为策略评估任务提供额外的数据收集,如何补充脱机强化学习(RL)数据。在政策评估中,任务是估计对兴趣环境的评估政策的预期回报。在离线策略评估上的事先工作通常仅考虑静态数据集。我们考虑一个设置,我们可以收集少量附加数据,以与潜在的更大的离线RL数据集组合。我们展示只需运行评估政策 - 策略数据收集 - 此设置是子最优。然后,我们介绍了两个新的数据收集策略进行策略评估,两者都考虑在收集未来数据时考虑先前收集的数据,以便在收集的整个数据集中减少分发班次(或采样错误)。我们的经验结果表明,与政策采样相比,我们的策略产生了具有较低采样误差的数据,并且通常导致任何总数据集大小的策略评估中的较低平均平方误差。我们还表明,这些策略可以从初始禁止策略数据开始,收集其他数据,然后使用初始和新数据来产生低均衡的错误策略评估,而无需使用脱策校正。
translated by 谷歌翻译
面对顺序决策问题时,能够预测如果使用新策略进行决策会发生什么会发生什么。这些预测通常必须基于在一些先前使用的决策规则下收集的数据。许多以前的方法使得这种违规(或反事实)估计的性能测量值的预期值称为返回。在本文中,我们采取了迈向普遍违规估算机(UNO)的第一步 - 为返回分配的任何参数提供截止政策估计和高信任界限。我们使用UNO来估计和同时限制均值,方差,量级/中位数,分位式范围,CVAR和返回的整个累积分布。最后,我们还在各种环境中讨论了UNO的适用性,包括完全可观察,部分可观察的(即,与未观察到的混乱),马尔可夫,非马尔可瓦尔,静止,平稳的非稳定性和离散分布转移。
translated by 谷歌翻译
In off-policy reinforcement learning, a behaviour policy performs exploratory interactions with the environment to obtain state-action-reward samples which are then used to learn a target policy that optimises the expected return. This leads to a problem of off-policy evaluation, where one needs to evaluate the target policy from samples collected by the often unrelated behaviour policy. Importance sampling is a traditional statistical technique that is often applied to off-policy evaluation. While importance sampling estimators are unbiased, their variance increases exponentially with the horizon of the decision process due to computing the importance weight as a product of action probability ratios, yielding estimates with low accuracy for domains involving long-term planning. This paper proposes state-based importance sampling (SIS), which drops the action probability ratios of sub-trajectories with "neglible states" -- roughly speaking, those for which the chosen actions have no impact on the return estimate -- from the computation of the importance weight. Theoretical results show that this results in a reduction of the exponent in the variance upper bound as well as improving the mean squared error. An automated search algorithm based on covariance testing is proposed to identify a negligible state set which has minimal MSE when performing state-based importance sampling. Experiments are conducted on a lift domain, which include "lift states" where the action has no impact on the following state and reward. The results demonstrate that using the search algorithm, SIS yields reduced variance and improved accuracy compared to traditional importance sampling, per-decision importance sampling, and incremental importance sampling.
translated by 谷歌翻译
Off-policy evaluation (OPE) is a method for estimating the return of a target policy using some pre-collected observational data generated by a potentially different behavior policy. In some cases, there may be unmeasured variables that can confound the action-reward or action-next-state relationships, rendering many existing OPE approaches ineffective. This paper develops an instrumental variable (IV)-based method for consistent OPE in confounded Markov decision processes (MDPs). Similar to single-stage decision making, we show that IV enables us to correctly identify the target policy's value in infinite horizon settings as well. Furthermore, we propose an efficient and robust value estimator and illustrate its effectiveness through extensive simulations and analysis of real data from a world-leading short-video platform.
translated by 谷歌翻译
我们介绍了一种改进政策改进的方法,该方法在基于价值的强化学习(RL)的贪婪方法与基于模型的RL的典型计划方法之间进行了插值。新方法建立在几何视野模型(GHM,也称为伽马模型)的概念上,该模型对给定策略的折现状态验证分布进行了建模。我们表明,我们可以通过仔细的基本策略GHM的仔细组成,而无需任何其他学习,可以评估任何非马尔科夫策略,以固定的概率在一组基本马尔可夫策略之间切换。然后,我们可以将广义政策改进(GPI)应用于此类非马尔科夫政策的收集,以获得新的马尔可夫政策,通常将其表现优于其先驱。我们对这种方法提供了彻底的理论分析,开发了转移和标准RL的应用,并在经验上证明了其对标准GPI的有效性,对充满挑战的深度RL连续控制任务。我们还提供了GHM培训方法的分析,证明了关于先前提出的方法的新型收敛结果,并显示了如何在深度RL设置中稳定训练这些模型。
translated by 谷歌翻译
Adequately assigning credit to actions for future outcomes based on their contributions is a long-standing open challenge in Reinforcement Learning. The assumptions of the most commonly used credit assignment method are disadvantageous in tasks where the effects of decisions are not immediately evident. Furthermore, this method can only evaluate actions that have been selected by the agent, making it highly inefficient. Still, no alternative methods have been widely adopted in the field. Hindsight Credit Assignment is a promising, but still unexplored candidate, which aims to solve the problems of both long-term and counterfactual credit assignment. In this thesis, we empirically investigate Hindsight Credit Assignment to identify its main benefits, and key points to improve. Then, we apply it to factored state representations, and in particular to state representations based on the causal structure of the environment. In this setting, we propose a variant of Hindsight Credit Assignment that effectively exploits a given causal structure. We show that our modification greatly decreases the workload of Hindsight Credit Assignment, making it more efficient and enabling it to outperform the baseline credit assignment method on various tasks. This opens the way to other methods based on given or learned causal structures.
translated by 谷歌翻译
Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications. This is often because off-policy RL algorithms suffer from distributional shift, due to mismatch between dataset and the target policy, leading to high variance and over-estimation of value functions. In this work, we propose variance regularization for offline RL algorithms, using stationary distribution corrections. We show that by using Fenchel duality, we can avoid double sampling issues for computing the gradient of the variance regularizer. The proposed algorithm for offline variance regularization (OVAR) can be used to augment any existing offline policy optimization algorithms. We show that the regularizer leads to a lower bound to the offline policy optimization objective, which can help avoid over-estimation errors, and explains the benefits of our approach across a range of continuous control domains when compared to existing state-of-the-art algorithms.
translated by 谷歌翻译
政策梯度定理(Sutton等,2000)规定了目标政策下的累积折扣国家分配以近似梯度。实际上,基于该定理的大多数算法都打破了这一假设,引入了分布转移,该分配转移可能导致逆转溶液的收敛性。在本文中,我们提出了一种新的方法,可以从开始状态重建政策梯度,而无需采取特定的采样策略。可以根据梯度评论家来简化此形式的策略梯度计算,由于梯度的新钟声方程式,可以递归估算。通过使用来自差异数据流的梯度评论家的时间差异更新,我们开发了第一个以无模型方式避开分布变化问题的估计器。我们证明,在某些可实现的条件下,无论采样策略如何,我们的估计器都是公正的。我们从经验上表明,我们的技术在存在非政策样品的情况下实现了卓越的偏见变化权衡和性能。
translated by 谷歌翻译
我们在无限地平线马尔可夫决策过程中考虑批量(离线)策略学习问题。通过移动健康应用程序的推动,我们专注于学习最大化长期平均奖励的政策。我们为平均奖励提出了一款双重强大估算器,并表明它实现了半导体效率。此外,我们开发了一种优化算法来计算参数化随机策略类中的最佳策略。估计政策的履行是通过政策阶级的最佳平均奖励与估计政策的平均奖励之间的差异来衡量,我们建立了有限样本的遗憾保证。通过模拟研究和促进体育活动的移动健康研究的分析来说明该方法的性能。
translated by 谷歌翻译
本文关注的是,基于无限视野设置中预采用的观察数据,为目标策略的价值离线构建置信区间。大多数现有作品都假定不存在混淆观察到的动作的未测量变量。但是,在医疗保健和技术行业等实际应用中,这种假设可能会违反。在本文中,我们表明,使用一些辅助变量介导动作对系统动态的影响,目标策略的价值在混杂的马尔可夫决策过程中可以识别。基于此结果,我们开发了一个有效的非政策值估计器,该估计值可用于潜在模型错误指定并提供严格的不确定性定量。我们的方法是通过理论结果,从乘车共享公司获得的模拟和真实数据集证明的。python实施了建议的过程,请访问https://github.com/mamba413/cope。
translated by 谷歌翻译
解决了与人类偏好的安全一致性以及学习效率之类的各种目的,越来越多的强化学习研究集中在依赖整个收益分配的风险功能上。关于\ emph {Oplicy风险评估}(OPRA)的最新工作,针对上下文匪徒引入了目标策略的收益率以及有限样本保证的一致估计量,并保证了(并同时保留所有风险)。在本文中,我们将OPRA提升到马尔可夫决策过程(MDPS),其中重要性采样(IS)CDF估计量由于有效样本量较小而遭受较长轨迹的较大差异。为了减轻这些问题,我们合并了基于模型的估计,以开发MDPS回报的CDF的第一个双重鲁棒(DR)估计器。该估计器的差异明显较小,并且在指定模型时,可以实现Cramer-Rao方差下限。此外,对于许多风险功能,下游估计值同时享有较低的偏差和较低的差异。此外,我们得出了非政策CDF和风险估计的第一个Minimax下限,这与我们的误差界限到恒定因子。最后,我们在几种不同的环境上实验表明了DR CDF估计的精度。
translated by 谷歌翻译
在上下文土匪中,非政策评估(OPE)已在现实世界中迅速采用,因为它仅使用历史日志数据就可以离线评估新政策。不幸的是,当动作数量较大时,现有的OPE估计器(其中大多数是基于反相反的得分加权)会严重降解,并且可能会遭受极端偏见和差异。这挫败了从推荐系统到语言模型的许多应用程序中使用OPE。为了克服这个问题,我们提出了一个新的OPE估计器,即当动作嵌入在动作空间中提供结构时,利用边缘化的重要性权重。我们表征了所提出的估计器的偏差,方差和平方平方误差,并分析了动作嵌入提供了比常规估计器提供统计益处的条件。除了理论分析外,我们还发现,即使由于大量作用,现有估计量崩溃,经验性绩效的改善也可以实现可靠的OPE。
translated by 谷歌翻译