This paper studies offline policy learning, which aims at utilizing observations collected a priori (from either fixed or adaptively evolving behavior policies) to learn an optimal individualized decision rule that achieves the best overall outcomes for a given population. Existing policy learning methods rely on a uniform overlap assumption, i.e., the propensities of exploring all actions for all individual characteristics are lower bounded in the offline dataset; put differently, the performance of the existing methods depends on the worst-case propensity in the offline dataset. As one has no control over the data collection process, this assumption can be unrealistic in many situations, especially when the behavior policies are allowed to evolve over time with diminishing propensities for certain actions. In this paper, we propose a new algorithm that optimizes lower confidence bounds (LCBs) -- instead of point estimates -- of the policy values. The LCBs are constructed using knowledge of the behavior policies for collecting the offline data. Without assuming any uniform overlap condition, we establish a data-dependent upper bound for the suboptimality of our algorithm, which only depends on (i) the overlap for the optimal policy, and (ii) the complexity of the policy class we optimize over. As an implication, for adaptively collected data, we ensure efficient policy learning as long as the propensities for optimal actions are lower bounded over time, while those for suboptimal ones are allowed to diminish arbitrarily fast. In our theoretical analysis, we develop a new self-normalized type concentration inequality for inverse-propensity-weighting estimators, generalizing the well-known empirical Bernstein's inequality to unbounded and non-i.i.d. data.
translated by 谷歌翻译
在因果推理和强盗文献中,基于观察数据的线性功能估算线性功能的问题是规范的。我们分析了首先估计治疗效果函数的广泛的两阶段程序,然后使用该数量来估计线性功能。我们证明了此类过程的均方误差上的非反应性上限:这些边界表明,为了获得非反应性最佳程序,应在特定加权$ l^2 $中最大程度地估算治疗效果的误差。 -规范。我们根据该加权规范的约束回归分析了两阶段的程序,并通过匹配非轴突局部局部最小值下限,在有限样品中建立了实例依赖性最优性。这些结果表明,除了取决于渐近效率方差之外,最佳的非质子风险除了取决于样本量支持的最富有函数类别的真实结果函数与其近似类别之间的加权规范距离。
translated by 谷歌翻译
We study time-inhomogeneous episodic reinforcement learning (RL) under general function approximation and sparse rewards. We design a new algorithm, Variance-weighted Optimistic $Q$-Learning (VO$Q$L), based on $Q$-learning and bound its regret assuming completeness and bounded Eluder dimension for the regression function class. As a special case, VO$Q$L achieves $\tilde{O}(d\sqrt{HT}+d^6H^{5})$ regret over $T$ episodes for a horizon $H$ MDP under ($d$-dimensional) linear function approximation, which is asymptotically optimal. Our algorithm incorporates weighted regression-based upper and lower bounds on the optimal value function to obtain this improved regret. The algorithm is computationally efficient given a regression oracle over the function class, making this the first computationally tractable and statistically optimal approach for linear MDPs.
translated by 谷歌翻译
我们在面对未衡量的混杂因素时研究离线增强学习(RL)。由于缺乏与环境的在线互动,离线RL面临以下两个重大挑战:(i)代理可能会被未观察到的状态变量混淆; (ii)提前收集的离线数据不能为环境提供足够的覆盖范围。为了应对上述挑战,我们借助工具变量研究了混杂的MDP中的政策学习。具体而言,我们首先建立了基于和边缘化的重要性采样(MIS)的识别结果,以确定混杂的MDP中的预期总奖励结果。然后,通过利用悲观主义和我们的认同结果,我们提出了各种政策学习方法,并具有有限样本的次级临时性保证,可以在最小的数据覆盖范围和建模假设下找到最佳的课堂政策。最后,我们广泛的理论研究和一项由肾脏移植动机的数值研究证明了该方法的有希望的表现。
translated by 谷歌翻译
我们研究了用线性函数近似的加固学习中的违规评估(OPE)问题,旨在根据行为策略收集的脱机数据来估计目标策略的价值函数。我们建议纳入价值函数的方差信息以提高ope的样本效率。更具体地说,对于时间不均匀的epiSodic线性马尔可夫决策过程(MDP),我们提出了一种算法VA-OPE,它使用价值函数的估计方差重新重量拟合Q迭代中的Bellman残差。我们表明我们的算法达到了比最着名的结果绑定的更紧密的误差。我们还提供了行为政策与目标政策之间的分布转移的细粒度。广泛的数值实验证实了我们的理论。
translated by 谷歌翻译
我们在随机多臂匪徒问题中使用固定预算和上下文(协变)信息研究最佳武器识别。在观察上下文信息之后,在每一轮中,我们使用过去的观察和当前上下文选择一个治疗臂。我们的目标是确定最好的治疗组,这是一个在上下文分布中被边缘化的最大预期奖励的治疗组,而错误识别的可能性最小。首先,我们为此问题得出半参数的下限,在这里我们将最佳和次优的治疗臂的预期奖励之间的差距视为感兴趣的参数,以及所有其他参数,例如在上下文中的预期奖励,作为滋扰参数。然后,我们开发“上下文RS-AIPW策略”,该策略由随机采样(RS)规则组成,跟踪目标分配比和使用增强反向概率加权(AIPW)估算器的建议规则。我们提出的上下文RS-AIPW策略是最佳的,因为错误识别概率的上限与预算到Infinity时的半参数下限相匹配,并且差距趋于零。
translated by 谷歌翻译
我们研究了在高维稀疏线性上下文匪徒中动态批处理学习的问题,在给定的最大批量约束下,决策者在每个批次结束时只能观察奖励,可以动态地决定如何进行奖励。许多人将包括在下一批中(在当前批次结束时)以及每批采用哪些个性化行动选择方案。在各种实际情况下,这种批处理的限制无处不在,包括在临床试验中的营销和医疗选择中的个性化产品。我们通过后悔的下限表征了此问题中的基本学习限制,并提供了匹配的上限(直至日志因素),从而为此问题开了最佳方案。据我们所知,我们的工作为在高维稀疏线性上下文匪徒中对动态批处理学习的理论理解提供了第一个侵入。值得注意的是,即使我们的结果的一种特殊情况 - 当不存在批处理约束时 - 都会产生简单的无探索算法使用Lasso估算器,已经达到了在高维线性匪板中为标准在线学习的最小值最佳遗憾(对于No-Cargin情况),在高维上下文Bandits的新兴文献中似乎未知。
translated by 谷歌翻译
非政策评估和学习(OPE/L)使用离线观察数据来做出更好的决策,这对于在线实验有限的应用至关重要。但是,完全取决于记录的数据,OPE/L对环境分布的变化很敏感 - 数据生成环境和部署策略的差异。 \ citet {si2020distributional}提议的分布在稳健的OPE/L(Drope/L)解决此问题,但该提案依赖于逆向权重,如果估计错误和遗憾,如果倾向是非参数估计的,即使其差异是次级估计,即使是次级估计的,其估计错误和遗憾将降低。对于标准的,非体,OPE/L,这是通过双重鲁棒(DR)方法来解决的,但它们并不自然地扩展到更复杂的drop/l,涉及最糟糕的期望。在本文中,我们提出了具有KL-Divergence不确定性集的DROPE/L的第一个DR算法。为了进行评估,我们提出了局部双重稳健的drope(LDR $^2 $ ope),并表明它在弱产品速率条件下实现了半摩托效率。多亏了本地化技术,LDR $^2 $ OPE仅需要安装少量回归,就像标准OPE的DR方法一样。为了学习,我们提出了连续的双重稳健下降(CDR $^2 $ opl),并表明,在涉及连续回归的产品速率条件下,它具有$ \ Mathcal {o} \ left的快速后悔率(n^) {-1/2} \ right)$即使未知的倾向是非参数估计的。我们从经验上验证了模拟中的算法,并将结果进一步扩展到一般$ f $ divergence的不确定性集。
translated by 谷歌翻译
我们研究了情节块MDP中模型估计和无奖励学习的问题。在这些MDP中,决策者可以访问少数潜在状态产生的丰富观察或上下文。我们首先对基于固定行为策略生成的数据估算潜在状态解码功能(从观测到潜在状态的映射)感兴趣。我们在估计此功能的错误率上得出了信息理论的下限,并提出了接近此基本限制的算法。反过来,我们的算法还提供了MDP的所有组件的估计值。然后,我们研究在无奖励框架中学习近乎最佳政策的问题。根据我们有效的模型估计算法,我们表明我们可以以最佳的速度推断出策略(随着收集样品的数量增长大)的最佳策略。有趣的是,我们的分析提供了必要和充分的条件,在这些条件下,利用块结构可以改善样本复杂性,以识别近乎最佳的策略。当满足这些条件时,Minimax无奖励设置中的样本复杂性将通过乘法因子$ n $提高,其中$ n $是可能的上下文数量。
translated by 谷歌翻译
我们探索了一个新的强盗实验模型,其中潜在的非组织序列会影响武器的性能。上下文 - 统一算法可能会混淆,而那些执行正确的推理面部信息延迟的算法。我们的主要见解是,我们称之为Deconfounst Thompson采样的算法在适应性和健壮性之间取得了微妙的平衡。它的适应性在易于固定实例中带来了最佳效率,但是在硬性非平稳性方面显示出令人惊讶的弹性,这会导致其他自适应算法失败。
translated by 谷歌翻译
We study sample efficient reinforcement learning (RL) under the general framework of interactive decision making, which includes Markov decision process (MDP), partially observable Markov decision process (POMDP), and predictive state representation (PSR) as special cases. Toward finding the minimum assumption that empowers sample efficient learning, we propose a novel complexity measure, generalized eluder coefficient (GEC), which characterizes the fundamental tradeoff between exploration and exploitation in online interactive decision making. In specific, GEC captures the hardness of exploration by comparing the error of predicting the performance of the updated policy with the in-sample training error evaluated on the historical data. We show that RL problems with low GEC form a remarkably rich class, which subsumes low Bellman eluder dimension problems, bilinear class, low witness rank problems, PO-bilinear class, and generalized regular PSR, where generalized regular PSR, a new tractable PSR class identified by us, includes nearly all known tractable POMDPs. Furthermore, in terms of algorithm design, we propose a generic posterior sampling algorithm, which can be implemented in both model-free and model-based fashion, under both fully observable and partially observable settings. The proposed algorithm modifies the standard posterior sampling algorithm in two aspects: (i) we use an optimistic prior distribution that biases towards hypotheses with higher values and (ii) a loglikelihood function is set to be the empirical loss evaluated on the historical data, where the choice of loss function supports both model-free and model-based learning. We prove that the proposed algorithm is sample efficient by establishing a sublinear regret upper bound in terms of GEC. In summary, we provide a new and unified understanding of both fully observable and partially observable RL.
translated by 谷歌翻译
本文在动态定价的背景下调查预先存在的离线数据对在线学习的影响。我们在$ t $期间的销售地平线上研究单一产品动态定价问题。每个时段的需求由产品价格根据具有未知参数的线性需求模型确定。我们假设在销售地平线开始之前,卖方已经有一些预先存在的离线数据。离线数据集包含$ N $示例,其中每个标准是由历史价格和相关的需求观察组成的输入输出对。卖方希望利用预先存在的离线数据和顺序在线数据来最大限度地减少在线学习过程的遗憾。我们的特征在于在线学习过程的最佳遗憾的脱机数据的大小,位置和分散的联合效果。具体而言,离线数据的大小,位置和色散由历史样本数量为$ n $,平均历史价格与最佳价格$ \ delta $之间的距离以及历史价格的标准差价Sigma $分别。我们表明最佳遗憾是$ \ widetilde \ theta \ left(\ sqrt {t} \ wedge \ frac {t} {(n \ wedge t)\ delta ^ 2 + n \ sigma ^ 2} \右)$,基于“面对不确定性”原则的“乐观主义”的学习算法,其遗憾是最佳的对数因子。我们的结果揭示了对脱机数据的大小的最佳遗憾率的惊人变换,我们称之为阶段转型。此外,我们的结果表明,离线数据的位置和分散也对最佳遗憾具有内在效果,我们通过逆平面法量化了这种效果。
translated by 谷歌翻译
Two central paradigms have emerged in the reinforcement learning (RL) community: online RL and offline RL. In the online RL setting, the agent has no prior knowledge of the environment, and must interact with it in order to find an $\epsilon$-optimal policy. In the offline RL setting, the learner instead has access to a fixed dataset to learn from, but is unable to otherwise interact with the environment, and must obtain the best policy it can from this offline data. Practical scenarios often motivate an intermediate setting: if we have some set of offline data and, in addition, may also interact with the environment, how can we best use the offline data to minimize the number of online interactions necessary to learn an $\epsilon$-optimal policy? In this work, we consider this setting, which we call the \textsf{FineTuneRL} setting, for MDPs with linear structure. We characterize the necessary number of online samples needed in this setting given access to some offline dataset, and develop an algorithm, \textsc{FTPedel}, which is provably optimal. We show through an explicit example that combining offline data with online interactions can lead to a provable improvement over either purely offline or purely online RL. Finally, our results illustrate the distinction between \emph{verifiable} learning, the typical setting considered in online RL, and \emph{unverifiable} learning, the setting often considered in offline RL, and show that there is a formal separation between these regimes.
translated by 谷歌翻译
We study the problem of estimating the fixed point of a contractive operator defined on a separable Banach space. Focusing on a stochastic query model that provides noisy evaluations of the operator, we analyze a variance-reduced stochastic approximation scheme, and establish non-asymptotic bounds for both the operator defect and the estimation error, measured in an arbitrary semi-norm. In contrast to worst-case guarantees, our bounds are instance-dependent, and achieve the local asymptotic minimax risk non-asymptotically. For linear operators, contractivity can be relaxed to multi-step contractivity, so that the theory can be applied to problems like average reward policy evaluation problem in reinforcement learning. We illustrate the theory via applications to stochastic shortest path problems, two-player zero-sum Markov games, as well as policy evaluation and $Q$-learning for tabular Markov decision processes.
translated by 谷歌翻译
我们研究马尔可夫决策过程(MDP)框架中的离线数据驱动的顺序决策问题。为了提高学习政策的概括性和适应性,我们建议通过一套关于在政策诱导的固定分配所在的分发的一套平均奖励来评估每项政策。给定由某些行为策略生成的多个轨迹的预收集数据集,我们的目标是在预先指定的策略类中学习一个强大的策略,可以最大化此集的最小值。利用半参数统计的理论,我们开发了一种统计上有效的策略学习方法,用于估算DE NED强大的最佳政策。在数据集中的总决策点方面建立了达到对数因子的速率最佳遗憾。
translated by 谷歌翻译
在线学习和决策中的一个核心问题 - 从土匪到强化学习 - 是要了解哪种建模假设会导致样本有效的学习保证。我们考虑了一个普遍的对抗性决策框架,该框架涵盖了(结构化的)匪徒问题,这些问题与对抗性动力学有关。我们的主要结果是通过新的上限和下限显示决策估计系数,这是Foster等人引入的复杂度度量。在与我们环境的随机对应物中,对于对抗性决策而言是必要和足够的遗憾。但是,与随机设置相比,必须将决策估计系数应用于所考虑的模型类(或假设)的凸壳。这就确定了容纳对抗奖励或动态的价格受凸层化模型类的行为的约束,并恢复了许多现有结果 - 既积极又负面。在获得这些保证的途径中,我们提供了新的结构结果,将决策估计系数与其他众所周知的复杂性度量的变体联系起来,包括Russo和Van Roy的信息比以及Lattimore和Gy的探索目标\“ {o} rgy。
translated by 谷歌翻译
获取一阶遗憾界限 - 遗憾的界限不是作为最坏情况,但有一些衡量给定实例的最佳政策的性能 - 是连续决策的核心问题。虽然这种界限存在于许多设置中,但它们在具有大状态空间的钢筋学习中被证明是难以捉摸的。在这项工作中,我们解决了这个差距,并表明可以将遗憾的缩放作为$ \ mathcal {o}(\ sqrt {v_1 ^ \ star})$中的钢筋学习,即用大状态空间,即线性MDP设置。这里$ v_1 ^ \ star $是最佳政策的价值,$ k $是剧集的数量。我们证明基于最小二乘估计的现有技术不足以获得该结果,而是基于强大的Catoni平均估计器制定一种新的稳健自归一化浓度,其可能具有独立兴趣。
translated by 谷歌翻译
We study non-parametric estimation of the value function of an infinite-horizon $\gamma$-discounted Markov reward process (MRP) using observations from a single trajectory. We provide non-asymptotic guarantees for a general family of kernel-based multi-step temporal difference (TD) estimates, including canonical $K$-step look-ahead TD for $K = 1, 2, \ldots$ and the TD$(\lambda)$ family for $\lambda \in [0,1)$ as special cases. Our bounds capture its dependence on Bellman fluctuations, mixing time of the Markov chain, any mis-specification in the model, as well as the choice of weight function defining the estimator itself, and reveal some delicate interactions between mixing time and model mis-specification. For a given TD method applied to a well-specified model, its statistical error under trajectory data is similar to that of i.i.d. sample transition pairs, whereas under mis-specification, temporal dependence in data inflates the statistical error. However, any such deterioration can be mitigated by increased look-ahead. We complement our upper bounds by proving minimax lower bounds that establish optimality of TD-based methods with appropriately chosen look-ahead and weighting, and reveal some fundamental differences between value function estimation and ordinary non-parametric regression.
translated by 谷歌翻译
使用历史观察数据的政策学习是发现广泛应用程序的重要问题。示例包括选择优惠,价格,要发送给客户的广告,以及选择要开出患者的药物。但是,现有的文献取决于这样一个关键假设,即将在未来部署学习策略的未来环境与生成数据的过去环境相同 - 这个假设通常是错误或太粗糙的近似值。在本文中,我们提高了这一假设,并旨在通过不完整的观察数据来学习一项稳健的策略。我们首先提出了一个政策评估程序,该程序使我们能够评估政策在最坏情况下的转变下的表现。然后,我们为此建议的政策评估计划建立了中心限制定理类型保证。利用这种评估方案,我们进一步提出了一种新颖的学习算法,该算法能够学习一项对对抗性扰动和未知协变量转移的策略,并根据统一收敛理论的性能保证进行了绩效保证。最后,我们从经验上测试了合成数据集中提出的算法的有效性,并证明它提供了使用标准策略学习算法缺失的鲁棒性。我们通过在现实世界投票数据集的背景下提供了我们方法的全面应用来结束本文。
translated by 谷歌翻译
尽管在理解增强学习的最小样本复杂性(RL)(在“最坏情况”的实例上学习的复杂性)方面已经取得了很多进展,但这种复杂性的衡量标准通常不会捕捉到真正的学习困难。在实践中,在“简单”的情况下,我们可能希望获得比最糟糕的实例可以实现的要好得多。在这项工作中,我们试图理解在具有线性函数近似的RL设置中学习近乎最佳策略(PAC RL)的“实例依赖性”复杂性。我们提出了一种算法,\ textsc {pedel},该算法实现了依赖于实例的复杂性的量度,这是RL中的第一个具有功能近似设置,从而捕获了每个特定问题实例的学习难度。通过一个明确的示例,我们表明\ textsc {pedel}可以在低重晶,最小值 - 最佳算法上获得可证明的收益,并且这种算法无法达到实例 - 最佳速率。我们的方法取决于基于设计的新型实验程序,该程序将勘探预算重点放在与学习近乎最佳政策最相关的“方向”上,并且可能具有独立的兴趣。
translated by 谷歌翻译