我们考虑在离线增强学习中有一个具有挑战性的理论问题(RL):仅在功能近似器的可靠性型假设下,通过缺乏足够覆盖的数据集获得样本效率保证。尽管现有的理论已经在可实现性和非探索数据下分别解决了学习,但没有工作能够同时解决这两者(除了我们对详细比较的并发工作除外)。在额外的差距假设下,我们根据边缘化重要性采样(MIS)形成的版本空间(MIS)为简单的悲观算法提供保证,并且保证只需要数据来涵盖最佳策略和功能类,以实现最佳价值和最佳价值和密度比函数。尽管在RL理论的其他领域中使用了类似的差距假设,但我们的工作是第一个识别离线RL中差距假设的实用性和新型机制,其功能近似较弱。
translated by 谷歌翻译
离线增强学习(RL)的样本效率保证通常依赖于对功能类别(例如Bellman-Completeness)和数据覆盖范围(例如,全政策浓缩性)的强有力的假设。尽管最近在放松这些假设方面做出了努力,但现有作品只能放松这两个因素之一,从而使另一个因素的强烈假设完好无损。作为一个重要的开放问题,我们是否可以实现对这两个因素的假设较弱的样本效率离线RL?在本文中,我们以积极的态度回答了这个问题。我们基于MDP的原始偶对偶进行分析了一种简单的算法,其中双重变量(打折占用)是使用密度比函数对离线数据进行建模的。通过适当的正则化,我们表明该算法仅在可变性和单极浓缩性下具有多项式样品的复杂性。我们还基于不同的假设提供了替代分析,以阐明离线RL原始二算法的性质。
translated by 谷歌翻译
我们在一般的非线性函数近似下研究无奖励增强学习(RL),并在各种标准结构假设下建立样品效率和硬度结果。从积极的一面来看,我们提出了在最小的结构假设下进行样品有效奖励探索的Rfolive(无奖励橄榄)算法,该假设涵盖了先前研究的线性MDPS的设置(Jin等,2020b),线性完整性(线性完整性)( Zanette等人,2020b)和低级MDP,具有未知的表示(Modi等,2021)。我们的分析表明,以前针对后两个设置的易学性或可及性假设在统计上对于无奖励探索而言并不是必需的。在负面方面,我们为在线性完整性假设下的无奖励和奖励意识探索提供统计硬度结果时,当基础特征未知时,显示了低级别和线性完整性设置之间的指数分离。
translated by 谷歌翻译
Offline reinforcement learning (RL) concerns pursuing an optimal policy for sequential decision-making from a pre-collected dataset, without further interaction with the environment. Recent theoretical progress has focused on developing sample-efficient offline RL algorithms with various relaxed assumptions on data coverage and function approximators, especially to handle the case with excessively large state-action spaces. Among them, the framework based on the linear-programming (LP) reformulation of Markov decision processes has shown promise: it enables sample-efficient offline RL with function approximation, under only partial data coverage and realizability assumptions on the function classes, with favorable computational tractability. In this work, we revisit the LP framework for offline RL, and advance the existing results in several aspects, relaxing certain assumptions and achieving optimal statistical rates in terms of sample size. Our key enabler is to introduce proper constraints in the reformulation, instead of using any regularization as in the literature, sometimes also with careful choices of the function classes and initial state distributions. We hope our insights further advocate the study of the LP framework, as well as the induced primal-dual minimax optimization, in offline RL.
translated by 谷歌翻译
使用悲观,推理缺乏详尽的勘探数据集时的脱机强化学习最近颇具知名度。尽管它增加了算法的鲁棒性,过于悲观的推理可以在排除利好政策的发现,这是流行的基于红利悲观的问题同样有害。在本文中,我们介绍一般函数近似的Bellman-一致悲观的概念:不是计算逐点下界的值的功能,我们在超过设定的与贝尔曼方程一致的功能的初始状态实现悲观。我们的理论保证只需要贝尔曼封闭性作为探索性的设置标准,其中基于奖金的情况下的悲观情绪未能提供担保。即使在线性函数逼近的特殊情况下更强的表现力假设成立,我们的结果由$ \ mathcal {}Ø(d)在其样品的复杂$在最近的基于奖金的方法改善的时候,动作的空间是有限的。值得注意的是,我们的算法,能够自动适应事后最好的偏差 - 方差折中,而大多数现有的方法中需要调整的额外超参数的先验。
translated by 谷歌翻译
低级MDP已成为研究强化学习中的表示和探索的重要模型。有了已知的代表,存在几种无模型的探索策略。相反,未知表示设置的所有算法都是基于模型的,因此需要对完整动力学进行建模。在这项工作中,我们介绍了低级MDP的第一个无模型表示学习算法。关键的算法贡献是一个新的Minimax表示学习目标,我们为其提供具有不同权衡的变体,其统计和计算属性不同。我们将这一表示的学习步骤与探索策略交织在一起,以无奖励的方式覆盖状态空间。所得算法可证明样品有效,并且可以适应一般函数近似以扩展到复杂的环境。
translated by 谷歌翻译
本文介绍了一项有关离线增强学习中依赖间隙依赖样品复杂性的系统研究。先前的工作显示了何时最佳策略和行为策略之间的密度比上限(最佳策略覆盖范围假设),则代理可以实现$ o \ left(\ frac {1} {\ epsilon^2} \ right)$ rate,这也是最小值的最佳。我们在最佳策略覆盖范围假设下显示,当在最佳$ q $ unction中存在积极的子临时差距时,可以将费率提高到$ o \ left(\ frac {1} {\ epsilon} \ right)$。。此外,我们显示了行为策略的访问概率何时在最佳策略的访问概率为正(统一的最佳策略覆盖范围假设)的状态下,均匀下降,识别最佳政策的样本复杂性独立于$ \ frac {1} {\ epsilon} $。最后,我们呈现几乎匹配的下限,以补充我们的间隙依赖性上限。
translated by 谷歌翻译
We study time-inhomogeneous episodic reinforcement learning (RL) under general function approximation and sparse rewards. We design a new algorithm, Variance-weighted Optimistic $Q$-Learning (VO$Q$L), based on $Q$-learning and bound its regret assuming completeness and bounded Eluder dimension for the regression function class. As a special case, VO$Q$L achieves $\tilde{O}(d\sqrt{HT}+d^6H^{5})$ regret over $T$ episodes for a horizon $H$ MDP under ($d$-dimensional) linear function approximation, which is asymptotically optimal. Our algorithm incorporates weighted regression-based upper and lower bounds on the optimal value function to obtain this improved regret. The algorithm is computationally efficient given a regression oracle over the function class, making this the first computationally tractable and statistically optimal approach for linear MDPs.
translated by 谷歌翻译
This paper studies systematic exploration for reinforcement learning with rich observations and function approximation. We introduce a new model called contextual decision processes, that unifies and generalizes most prior settings. Our first contribution is a complexity measure, the Bellman rank , that we show enables tractable learning of near-optimal behavior in these processes and is naturally small for many well-studied reinforcement learning settings. Our second contribution is a new reinforcement learning algorithm that engages in systematic exploration to learn contextual decision processes with low Bellman rank. Our algorithm provably learns near-optimal behavior with a number of samples that is polynomial in all relevant parameters but independent of the number of unique observations. The approach uses Bellman error minimization with optimistic exploration and provides new insights into efficient exploration for reinforcement learning with function approximation.
translated by 谷歌翻译
Value-function approximation methods that operate in batch mode have foundational importance to reinforcement learning (RL). Finite sample guarantees for these methods often crucially rely on two types of assumptions: (1) mild distribution shift, and (2) representation conditions that are stronger than realizability. However, the necessity ("why do we need them?") and the naturalness ("when do they hold?") of such assumptions have largely eluded the literature. In this paper, we revisit these assumptions and provide theoretical results towards answering the above questions, and make steps towards a deeper understanding of value-function approximation.
translated by 谷歌翻译
Modern Reinforcement Learning (RL) is commonly applied to practical problems with an enormous number of states, where function approximation must be deployed to approximate either the value function or the policy. The introduction of function approximation raises a fundamental set of challenges involving computational and statistical efficiency, especially given the need to manage the exploration/exploitation tradeoff. As a result, a core RL question remains open: how can we design provably efficient RL algorithms that incorporate function approximation? This question persists even in a basic setting with linear dynamics and linear rewards, for which only linear function approximation is needed.This paper presents the first provable RL algorithm with both polynomial runtime and polynomial sample complexity in this linear setting, without requiring a "simulator" or additional assumptions. Concretely, we prove that an optimistic modification of Least-Squares Value Iteration (LSVI)-a classical algorithm frequently studied in the linear setting-achieves O( √ d 3 H 3 T ) regret, where d is the ambient dimension of feature space, H is the length of each episode, and T is the total number of steps. Importantly, such regret is independent of the number of states and actions.
translated by 谷歌翻译
在现实世界的强化学习应用中,学习者的观察空间无处不在,有关手头任务的相关信息和无关紧要。从高维观察中学习一直是监督学习和统计数据(例如,通过稀疏性)进行广泛研究的主题,但是即使在有限的状态/行动(表格)领域,也不能很好地理解强化学习中的类似问题。我们引入了一个新的问题设置,用于增强学习,即马尔可夫决策过程(EXOMDP),其中状态空间将(未知)分解成一个小的(或内源性)组件,并且很大的无关(或外源)组件;外源成分独立于学习者的行为,但以任意的,时间相关的方式演变。我们提供了一种新的算法Exorl,该算法学习了一种近乎最佳的政策,其样品复杂性在内源性组件的大小中多项式,几乎独立于外源成分的大小,从而提供了一个双重指数的改进算法。我们的结果首次突出了在存在外源信息的情况下首次可以进行样品高效的增强学习,并为未来的调查提供了简单,用户友好的基准。
translated by 谷歌翻译
Epsilon-Greedy,SoftMax或Gaussian噪声等近视探索政策在某些强化学习任务中无法有效探索,但是在许多其他方面,它们的表现都很好。实际上,实际上,由于简单性,它们通常被选为最佳选择。但是,对于哪些任务执行此类政策成功?我们可以为他们的有利表现提供理论保证吗?尽管这些政策具有显着的实际重要性,但这些关键问题几乎没有得到研究。本文介绍了对此类政策的理论分析,并为通过近视探索提供了对增强学习的首次遗憾和样本复杂性。我们的结果适用于具有有限的Bellman Eluder维度的情节MDP中的基于价值功能的算法。我们提出了一种新的复杂度度量,称为近视探索差距,用Alpha表示,该差距捕获了MDP的结构属性,勘探策略和给定的值函数类别。我们表明,近视探索的样品复杂性与该数量的倒数1 / alpha^2二次地量表。我们通过具体的例子进一步证明,由于相应的动态和奖励结构,在近视探索成功的几项任务中,近视探索差距确实是有利的。
translated by 谷歌翻译
In offline reinforcement learning (RL), a learner leverages prior logged data to learn a good policy without interacting with the environment. A major challenge in applying such methods in practice is the lack of both theoretically principled and practical tools for model selection and evaluation. To address this, we study the problem of model selection in offline RL with value function approximation. The learner is given a nested sequence of model classes to minimize squared Bellman error and must select among these to achieve a balance between approximation and estimation error of the classes. We propose the first model selection algorithm for offline RL that achieves minimax rate-optimal oracle inequalities up to logarithmic factors. The algorithm, ModBE, takes as input a collection of candidate model classes and a generic base offline RL algorithm. By successively eliminating model classes using a novel one-sided generalization test, ModBE returns a policy with regret scaling with the complexity of the minimally complete model class. In addition to its theoretical guarantees, it is conceptually simple and computationally efficient, amounting to solving a series of square loss regression problems and then comparing relative square loss between classes. We conclude with several numerical simulations showing it is capable of reliably selecting a good model class.
translated by 谷歌翻译
我们研究了具有线性函数近似增强学习中的随机最短路径(SSP)问题,其中过渡内核表示为未知模型的线性混合物。我们将此类别的SSP问题称为线性混合物SSP。我们提出了一种具有Hoeffding-type置信度的新型算法,用于学习线性混合物SSP,可以获得$ \ tilde {\ Mathcal {o}}}}(d B _ {\ star}^{1.5} \ sqrt {k/c_ {k/c_ {k/c_ {k/c_ { \ min}})$遗憾。这里$ k $是情节的数量,$ d $是混合模型中功能映射的维度,$ b _ {\ star} $限制了最佳策略的预期累积成本,$ c _ {\ min}>> 0 $是成本函数的下限。当$ c _ {\ min} = 0 $和$ \ tilde {\ mathcal {o}}}(k^{2/3})$遗憾时,我们的算法也适用于情况。据我们所知,这是第一个具有sublrinear遗憾保证线性混合物SSP的算法。此外,我们设计了精致的伯恩斯坦型信心集并提出了改进的算法,该算法可实现$ \ tilde {\ Mathcal {o}}}(d b _ {\ star} \ sqrt {k/c/c/c {k/c _ {\ min}}) $遗憾。为了补充遗憾的上限,我们还证明了$ \ omega(db _ {\ star} \ sqrt {k})$的下限。因此,我们的改进算法将下限匹配到$ 1/\ sqrt {c _ {\ min}} $ factor和poly-logarithmic因素,从而实现了近乎最佳的遗憾保证。
translated by 谷歌翻译
The offline reinforcement learning (RL) problem is often motivated by the need to learn data-driven decision policies in financial, legal and healthcare applications. However, the learned policy could retain sensitive information of individuals in the training data (e.g., treatment and outcome of patients), thus susceptible to various privacy risks. We design offline RL algorithms with differential privacy guarantees which provably prevent such risks. These algorithms also enjoy strong instance-dependent learning bounds under both tabular and linear Markov decision process (MDP) settings. Our theory and simulation suggest that the privacy guarantee comes at (almost) no drop in utility comparing to the non-private counterpart for a medium-size dataset.
translated by 谷歌翻译
强化学习算法的实用性由于相对于问题大小的规模差而受到限制,因为学习$ \ epsilon $ -optimal策略的样本复杂性为$ \ tilde {\ omega} \ left(| s | s || a || a || a || a | h^3 / \ eps^2 \ right)$在MDP的最坏情况下,带有状态空间$ S $,ACTION SPACE $ A $和HORIZON $ H $。我们考虑一类显示出低级结构的MDP,其中潜在特征未知。我们认为,价值迭代和低级别矩阵估计的自然组合导致估计误差在地平线上呈指数增长。然后,我们提供了一种新算法以及统计保证,即有效利用了对生成模型的访问,实现了$ \ tilde {o} \ left的样本复杂度(d^5(d^5(| s |+| a |)\),我们有效利用低级结构。对于等级$ d $设置的Mathrm {Poly}(h)/\ EPS^2 \ right)$,相对于$ | s |,| a | $和$ \ eps $的缩放,这是最小值的最佳。与线性和低级别MDP的文献相反,我们不需要已知的功能映射,我们的算法在计算上很简单,并且我们的结果长期存在。我们的结果提供了有关MDP对过渡内核与最佳动作值函数所需的最小低级结构假设的见解。
translated by 谷歌翻译
我们建议和分析一个强化学习原理,该原理仅在测试功能的用户定义空间沿使用它们的有效性来近似钟声方程。我们专注于使用功能近似的无模型离线RL应用程序,我们利用这一原理来得出置信区间以进行非政策评估,并在规定的策略类别中优化了对策略的优化。我们证明了关于我们的政策优化程序的甲骨文不平等,就任意比较策略的价值和不确定性之间的权衡而言。测试功能空间的不同选择使我们能够解决共同框架中的不同问题。我们表征了使用我们的程序从政策转移到政策数据的效率的丧失,并建立了与过去工作中研究的浓缩性系数的连接。我们深入研究了具有线性函数近似的方法的实施,即使贝尔曼关闭不结束,也可以通过多项式时间实现提供理论保证。
translated by 谷歌翻译
本文涉及增强学习的样本效率,假设进入生成模型(或模拟器)。我们首先考虑$ \ gamma $ -discounted infinite-horizo​​ n markov决策过程(mdps)与状态空间$ \ mathcal {s} $和动作空间$ \ mathcal {a} $。尽管有许多先前的作品解决这个问题,但尚未确定样本复杂性和统计准确性之间的权衡的完整图像。特别地,所有事先结果都遭受严重的样本大小屏障,因为只有在样本量超过$ \ FRAC {| \ Mathcal {S} || \ Mathcal {A} |} {(1- \ gamma)^ 2} $。目前的论文通过认证了两种算法的最小值 - 基于模型的算法和基于保守模型的算法的最小值,克服了该障碍 - 一旦样本大小超过$ \ FRAC {| \ Mathcal {s } || mathcal {a} |} {1- \ gamma} $(modulo一些日志系数)。超越无限地平线MDP,我们进一步研究了时代的有限情况MDP,并证明了一种基于普通模型的规划算法足以实现任何目标精度水平的最佳样本复杂性。据我们所知,这项工作提供了第一个最低限度的最佳保证,可容纳全部样本尺寸(超出哪个发现有意义的政策是理论上不可行的信息)。
translated by 谷歌翻译
我们根据相对悲观主义的概念,在数据覆盖不足的情况下提出了经过对抗训练的演员评论家(ATAC),这是一种新的无模型算法(RL)。 ATAC被设计为两人Stackelberg游戏:政策演员与受对抗训练的价值评论家竞争,后者发现参与者不如数据收集行为策略的数据一致方案。我们证明,当演员在两人游戏中不后悔时,运行ATAC会产生一项政策,证明1)在控制悲观程度的各种超级参数上都超过了行为政策,而2)与最佳竞争。 policy covered by data with appropriately chosen hyperparameters.与现有作品相比,尤其是我们的框架提供了一般函数近似的理论保证,也提供了可扩展到复杂环境和大型数据集的深度RL实现。在D4RL基准测试中,ATAC在一系列连续的控制任务上始终优于最先进的离线RL算法。
translated by 谷歌翻译