We consider local kernel metric learning for off-policy evaluation (OPE) of deterministic policies in contextual bandits with continuous action spaces. Our work is motivated by practical scenarios where the target policy needs to be deterministic due to domain requirements, such as prescription of treatment dosage and duration in medicine. Although importance sampling (IS) provides a basic principle for OPE, it is ill-posed for the deterministic target policy with continuous actions. Our main idea is to relax the target policy and pose the problem as kernel-based estimation, where we learn the kernel metric in order to minimize the overall mean squared error (MSE). We present an analytic solution for the optimal metric, based on the analysis of bias and variance. Whereas prior work has been limited to scalar action spaces or kernel bandwidth selection, our work takes a step further being capable of vector action spaces and metric optimization. We show that our estimator is consistent, and significantly reduces the MSE compared to baseline OPE methods through experiments on various domains.

在上下文土匪中，非政策评估（OPE）已在现实世界中迅速采用，因为它仅使用历史日志数据就可以离线评估新政策。不幸的是，当动作数量较大时，现有的OPE估计器（其中大多数是基于反相反的得分加权）会严重降解，并且可能会遭受极端偏见和差异。这挫败了从推荐系统到语言模型的许多应用程序中使用OPE。为了克服这个问题，我们提出了一个新的OPE估计器，即当动作嵌入在动作空间中提供结构时，利用边缘化的重要性权重。我们表征了所提出的估计器的偏差，方差和平方平方误差，并分析了动作嵌入提供了比常规估计器提供统计益处的条件。除了理论分析外，我们还发现，即使由于大量作用，现有估计量崩溃，经验性绩效的改善也可以实现可靠的OPE。

Counterfactual reasoning from logged data has become increasingly important for many applications such as web advertising or healthcare. In this paper, we address the problem of learning stochastic policies with continuous actions from the viewpoint of counterfactual risk minimization (CRM). While the CRM framework is appealing and well studied for discrete actions, the continuous action case raises new challenges about modelization, optimization, and~offline model selection with real data which turns out to be particularly challenging. Our paper contributes to these three aspects of the CRM estimation pipeline. First, we introduce a modelling strategy based on a joint kernel embedding of contexts and actions, which overcomes the shortcomings of previous discretization approaches. Second, we empirically show that the optimization aspect of counterfactual learning is important, and we demonstrate the benefits of proximal point algorithms and differentiable estimators. Finally, we propose an evaluation protocol for offline policies in real-world logged systems, which is challenging since policies cannot be replayed on test data, and we release a new large-scale dataset along with multiple synthetic, yet realistic, evaluation setups.

我们考虑了上下文匪徒的违规评估（OPE）问题，其中目标是使用日志记录策略收集的数据估计目标策略的值。 ope的最流行方法是通过组合直接方法（DM）估计和涉及逆倾向得分（IP）的校正项而获得的双重稳健（DR）估计器的变型。现有算法主要关注降低大型IPS引起的博士估算器方差的策略。我们提出了一种称为双重强大的新方法，具有信息借用和基于上下文的交换（DR-IC）估计，专注于减少偏差和方差。 DR-IC估计器用参数奖励模型替换标准DM估计器，该参数奖励模型通过依赖于IPS的相关结构从“更近的”上下文中借用信息。 DR-IC估计器还基于特定于上下文的切换规则在该修改的DM估计器和修改的DR估计器之间自适应地插值。我们对DR-IC估算员的表现提供了可证明的保证。我们还展示了DR-IC估计的卓越性能与艺术最先进的OPE算法相比，在许多基准问题上的算法相比。

离线政策优化可能会对许多现实世界的决策问题产生重大影响，因为在线学习在许多应用中可能是不可行的。重要性采样及其变体是离线策略评估中一种常用的估计器类型，此类估计器通常不需要关于价值函数或决策过程模型功能类的属性和代表性能力的假设。在本文中，我们确定了一种重要的过度拟合现象，以优化重要性加权收益，在这种情况下，学到的政策可以基本上避免在最初的状态空间的一部分中做出一致的决策。我们提出了一种算法，以避免通过新的每个国家 - 邻居标准化约束过度拟合，并提供对拟议算法的理论理由。我们还显示了以前尝试这种方法的局限性。我们在以医疗风格的模拟器为中测试算法，该模拟器是从真实医院收集的记录数据集和连续的控制任务。这些实验表明，与最先进的批处理学习算法相比，所提出的方法的过度拟合和更好的测试性能。

我们认为离政策在连续处理设置，如个性化的剂量调查评价（OPE）。在OPE，一个目标来估算下使用不同的决策规则产生的历史数据的新的治疗决策规则中的平均结果。离散处理设置上OPE焦点大多数现有的作品。为了应对持续的治疗，我们开发使用OPE深跳学习一种新的估计方法。我们的方法在于在使用深离散化，通过利用深度学习和多尺度变化点检测自适应离散化治疗领域的主要成分。这使我们能够应用在离散处理现有OPE方法来处理连续治疗。我们的方法是通过理论计算结果，模拟和实际应用程序，以华法林给药进一步合理的。

Off-policy evaluation methods are important in recommendation systems and search engines, where data collected under an existing logging policy is used to estimate the performance of a new proposed policy. A common approach to this problem is weighting, where data is weighted by a density ratio between the probability of actions given contexts in the target and logged policies. In practice, two issues often arise. First, many problems have very large action spaces and we may not observe rewards for most actions, and so in finite samples we may encounter a positivity violation. Second, many recommendation systems are not probabilistic and so having access to logging and target policy densities may not be feasible. To address these issues, we introduce the featurized embedded permutation weighting estimator. The estimator computes the density ratio in an action embedding space, which reduces the possibility of positivity violations. The density ratio is computed leveraging recent advances in normalizing flows and density ratio estimation as a classification problem, in order to obtain estimates which are feasible in practice.

强化学习的主要方法是根据预期的回报将信贷分配给行动。但是，我们表明回报可能取决于政策，这可能会导致价值估计的过度差异和减慢学习的速度。取而代之的是，我们证明了优势函数可以解释为因果效应，并与因果关系共享相似的属性。基于此洞察力，我们提出了直接优势估计（DAE），这是一种可以对优势函数进行建模并直接从政策数据进行估算的新方法，同时同时最大程度地减少了返回的方差而无需（操作 - ）值函数。我们还通过显示如何无缝整合到DAE中来将我们的方法与时间差异方法联系起来。所提出的方法易于实施，并且可以通过现代参与者批评的方法很容易适应。我们对三个离散控制域进行经验评估DAE，并表明它可以超过广义优势估计（GAE），这是优势估计的强大基线，当将大多数环境应用于策略优化时。

In this paper we present a new way of predicting the performance of a reinforcement learning policy given historical data that may have been generated by a different policy. The ability to evaluate a policy from historical data is important for applications where the deployment of a bad policy can be dangerous or costly. We show empirically that our algorithm produces estimates that often have orders of magnitude lower mean squared error than existing methods-it makes more efficient use of the available data. Our new estimator is based on two advances: an extension of the doubly robust estimator (Jiang & Li, 2015), and a new way to mix between model based estimates and importance sampling based estimates.

Reinforcement learning (RL) is one of the most vibrant research frontiers in machine learning and has been recently applied to solve a number of challenging problems. In this paper, we primarily focus on off-policy evaluation (OPE), one of the most fundamental topics in RL. In recent years, a number of OPE methods have been developed in the statistics and computer science literature. We provide a discussion on the efficiency bound of OPE, some of the existing state-of-the-art OPE methods, their statistical properties and some other related research directions that are currently actively explored.

Off-policy evaluation (OPE) attempts to predict the performance of counterfactual policies using log data from a different policy. We extend its applicability by developing an OPE method for a class of both full support and deficient support logging policies in contextual-bandit settings. This class includes deterministic bandit (such as Upper Confidence Bound) as well as deterministic decision-making based on supervised and unsupervised learning. We prove that our method's prediction converges in probability to the true performance of a counterfactual policy as the sample size increases. We validate our method with experiments on partly and entirely deterministic logging policies. Finally, we apply it to evaluate coupon targeting policies by a major online platform and show how to improve the existing policy.

为了在许多因素动态影响输出轨迹的复杂随机系统上学习，希望有效利用从以前迭代中收集的历史样本中的信息来加速策略优化。经典的经验重播使代理商可以通过重复使用历史观察来记住。但是，处理所有观察结果的统一重复使用策略均忽略了不同样本的相对重要性。为了克服这一限制，我们提出了一个基于一般差异的经验重播（VRER）框架，该框架可以选择性地重复使用最相关的样本以改善策略梯度估计。这种选择性机制可以自适应地对过去的样品增加重量，这些样本更可能由当前目标分布产生。我们的理论和实证研究表明，提议的VRER可以加速学习最佳政策，并增强最先进的政策优化方法的性能。

为了在许多因素动态影响输出轨迹的复杂随机系统上学习，希望有效利用从以前迭代中收集的历史样本中的信息来加速策略优化。经典的经验重播使代理商可以通过重复使用历史观察来记住。但是，处理所有观察结果的统一重复使用策略均忽略了不同样本的相对重要性。为了克服这一限制，我们提出了一个基于一般差异的经验重播（VRER）框架，该框架可以选择性地重复使用最相关的样本以改善策略梯度估计。这种选择性机制可以自适应地对过去的样品增加重量，这些样本更可能由当前目标分布产生。我们的理论和实证研究表明，提议的VRER可以加速学习最佳政策，并增强最先进的政策优化方法的性能。

Off-Policy evaluation (OPE) is concerned with evaluating a new target policy using offline data generated by a potentially different behavior policy. It is critical in a number of sequential decision making problems ranging from healthcare to technology industries. Most of the work in existing literature is focused on evaluating the mean outcome of a given policy, and ignores the variability of the outcome. However, in a variety of applications, criteria other than the mean may be more sensible. For example, when the reward distribution is skewed and asymmetric, quantile-based metrics are often preferred for their robustness. In this paper, we propose a doubly-robust inference procedure for quantile OPE in sequential decision making and study its asymptotic properties. In particular, we propose utilizing state-of-the-art deep conditional generative learning methods to handle parameter-dependent nuisance function estimation. We demonstrate the advantages of this proposed estimator through both simulations and a real-world dataset from a short-video platform. In particular, we find that our proposed estimator outperforms classical OPE estimators for the mean in settings with heavy-tailed reward distributions.

We consider the problem of off-policy evaluation (OPE) in reinforcement learning (RL), where the goal is to estimate the performance of an evaluation policy, $\pi_e$, using a fixed dataset, $\mathcal{D}$, collected by one or more policies that may be different from $\pi_e$. Current OPE algorithms may produce poor OPE estimates under policy distribution shift i.e., when the probability of a particular state-action pair occurring under $\pi_e$ is very different from the probability of that same pair occurring in $\mathcal{D}$ (Voloshin et al. 2021, Fu et al. 2021). In this work, we propose to improve the accuracy of OPE estimators by projecting the high-dimensional state-space into a low-dimensional state-space using concepts from the state abstraction literature. Specifically, we consider marginalized importance sampling (MIS) OPE algorithms which compute state-action distribution correction ratios to produce their OPE estimate. In the original ground state-space, these ratios may have high variance which may lead to high variance OPE. However, we prove that in the lower-dimensional abstract state-space the ratios can have lower variance resulting in lower variance OPE. We then highlight the challenges that arise when estimating the abstract ratios from data, identify sufficient conditions to overcome these issues, and present a minimax optimization problem whose solution yields these abstract ratios. Finally, our empirical evaluation on difficult, high-dimensional state-space OPE tasks shows that the abstract ratios can make MIS OPE estimators achieve lower mean-squared error and more robust to hyperparameter tuning than the ground ratios.

我们在无限地平线马尔可夫决策过程中考虑批量（离线）策略学习问题。通过移动健康应用程序的推动，我们专注于学习最大化长期平均奖励的政策。我们为平均奖励提出了一款双重强大估算器，并表明它实现了半导体效率。此外，我们开发了一种优化算法来计算参数化随机策略类中的最佳策略。估计政策的履行是通过政策阶级的最佳平均奖励与估计政策的平均奖励之间的差异来衡量，我们建立了有限样本的遗憾保证。通过模拟研究和促进体育活动的移动健康研究的分析来说明该方法的性能。

非政策评估和学习（OPE/L）使用离线观察数据来做出更好的决策，这对于在线实验有限的应用至关重要。但是，完全取决于记录的数据，OPE/L对环境分布的变化很敏感 - 数据生成环境和部署策略的差异。 \ citet {si2020distributional}提议的分布在稳健的OPE/L（Drope/L）解决此问题，但该提案依赖于逆向权重，如果估计错误和遗憾，如果倾向是非参数估计的，即使其差异是次级估计，即使是次级估计的，其估计错误和遗憾将降低。对于标准的，非体，OPE/L，这是通过双重鲁棒（DR）方法来解决的，但它们并不自然地扩展到更复杂的drop/l，涉及最糟糕的期望。在本文中，我们提出了具有KL-Divergence不确定性集的DROPE/L的第一个DR算法。为了进行评估，我们提出了局部双重稳健的drope（LDR $^2 $ ope），并表明它在弱产品速率条件下实现了半摩托效率。多亏了本地化技术，LDR $^2 $ OPE仅需要安装少量回归，就像标准OPE的DR方法一样。为了学习，我们提出了连续的双重稳健下降（CDR $^2 $ opl），并表明，在涉及连续回归的产品速率条件下，它具有$ \ Mathcal {o} \ left的快速后悔率（n^） {-1/2} \ right）$即使未知的倾向是非参数估计的。我们从经验上验证了模拟中的算法，并将结果进一步扩展到一般$ f $ divergence的不确定性集。

许多深厚的增强学习算法依赖于简单的探索形式，例如经常在连续控制域中使用的加性动作噪声。通常，该动作噪声的缩放因子被选为高参数，并在训练过程中保持恒定。在本文中，我们分析了学到的政策如何受到噪声类型，比例和缩放系数的影响。我们考虑了两种最突出的动作类型：高斯和ornstein-uhlenbeck噪声，并通过系统地改变噪声类型和规模参数以及测量感兴趣的变量（例如预期的政策回报和策略回报）来执行巨大的实验活动。探索期间的状态空间覆盖范围。对于后者，我们提出了一个新颖的状态空间覆盖量$ \ operatatorName {x} _ {\ Mathcal {u} \ text {rel}} $，对边界人工制品比以前提出的措施更强大。较大的噪声尺度通常会增加状态空间覆盖率。但是，我们发现使用较大的噪声量表增加空间覆盖范围通常是无益的。相反，在训练过程中降低噪声量表可以减少差异并通常改善学习绩效。我们得出的结论是，最好的噪声类型和尺度是环境取决于的，并且根据我们的观察结果，得出了指导选择动作噪声作为进一步优化的起点的启发式规则。

我们考虑在部分可观察到的马尔可夫决策过程（POMDP）中的违法评估（OPE），其中评估策略仅取决于可观察变量，并且行为策略取决于不可观察的潜在变量。现有的作品无论是假设未测量的混乱，还是专注于观察和状态空间都是表格的设置。因此，这些方法在存在未测量的混淆器的情况下遭受大偏差，或者在具有连续或大观察/状态空间的设置中的大方差。在这项工作中，通过引入将目标策略的价值和观察到的数据分布联系起来，提出了具有潜在混淆的POMDPS的新识别方法。在完全可观察到的MDP中，这些桥接功能将熟悉的值函数和评估与行为策略之间的边际密度比减少。我们接下来提出了用于学习这些桥接功能的最小值估计方法。我们的提案允许一般函数近似，因此适用于具有连续或大观察/状态空间的设置。最后，我们基于这些估计的桥梁功能构建了三种估计，对应于基于价值函数的估计器，边缘化重要性采样估计器和双重稳健的估计器。他们的掺入无血症和渐近性质进行了详细研究。

我们研究了从记录的匪徒反馈中进行额外学习的增强合奏模型。为了实现这一目标，我们提出了一种新的增强算法，该算法直接优化了对政策预期奖励的估计。我们分析了该算法，并证明，只要满足“弱”的学习条件，每轮增强的经验风险会随着每一轮增强而降低（可能是指数迅速）。我们进一步展示了基础学习者如何减少标准监督学习问题。实验表明，我们的算法可以胜过仅在观察到的奖励上回归的深层外部学习和方法，从而证明了增强和选择正确的学习目标的好处。