我们考虑基于嘈杂的强盗反馈优化黑盒功能的问题。内核强盗算法为此问题显示了强大的实证和理论表现。然而,它们严重依赖于模型所指定的模型,并且没有它可能会失败。相反,我们介绍了一个\ emph {isspecified}内塞的强盗设置,其中未知函数可以是$ \ epsilon $ - 在一些再现内核希尔伯特空间(RKHS)中具有界限范数的函数均匀近似。我们设计高效实用的算法,其性能在模型误操作的存在下最微小地降低。具体而言,我们提出了一种基于高斯过程(GP)方法的两种算法:一种乐观的EC-GP-UCB算法,需要了解误操作误差,并相断的GP不确定性采样,消除型算法,可以适应未知模型拼盘。我们在$ \ epsilon $,时间范围和底层内核方面提供累积遗憾的上限,我们表明我们的算法达到了$ \ epsilon $的最佳依赖性,而没有明确的误解知识。此外,在一个随机的上下文设置中,我们表明EC-GP-UCB可以有效地与遗憾的平衡策略有效地结合,尽管不知道$ \ epsilon $尽管不知道,但仍然可以获得类似的遗憾范围。
translated by 谷歌翻译
高赌注应用中产生的许多黑匣子优化任务需要风险厌恶的决策。但标准贝叶斯优化(BO)范式仅优化了预期值。我们概括了博的商业卑鄙和输入依赖性方差,我们认为我们认为是未知的先验。特别是,我们提出了一种新的风险厌恶异源贝类贝叶斯优化算法(Rahbo),其旨在识别具有高回报和低噪声方差的解决方案,同时在飞行时学习噪声分布。为此,我们将期望和方差模拟(未知)RKHS函数,并提出了一种新的风险感知获取功能。我们对我们的方法绑定了遗憾,并提供了一个强大的规则,以报告必须识别单个解决方案的应用程序的最终决策点。我们展示了Rahbo对合成基准函数和超参数调整任务的有效性。
translated by 谷歌翻译
Many applications require optimizing an unknown, noisy function that is expensive to evaluate. We formalize this task as a multiarmed bandit problem, where the payoff function is either sampled from a Gaussian process (GP) or has low RKHS norm. We resolve the important open problem of deriving regret bounds for this setting, which imply novel convergence rates for GP optimization. We analyze GP-UCB, an intuitive upper-confidence based algorithm, and bound its cumulative regret in terms of maximal information gain, establishing a novel connection between GP optimization and experimental design. Moreover, by bounding the latter in terms of operator spectra, we obtain explicit sublinear regret bounds for many commonly used covariance functions. In some important cases, our bounds have surprisingly weak dependence on the dimensionality. In our experiments on real sensor data, GP-UCB compares favorably with other heuristical GP optimization approaches.
translated by 谷歌翻译
We study bandit model selection in stochastic environments. Our approach relies on a meta-algorithm that selects between candidate base algorithms. We develop a meta-algorithm-base algorithm abstraction that can work with general classes of base algorithms and different type of adversarial meta-algorithms. Our methods rely on a novel and generic smoothing transformation for bandit algorithms that permits us to obtain optimal $O(\sqrt{T})$ model selection guarantees for stochastic contextual bandit problems as long as the optimal base algorithm satisfies a high probability regret guarantee. We show through a lower bound that even when one of the base algorithms has $O(\log T)$ regret, in general it is impossible to get better than $\Omega(\sqrt{T})$ regret in model selection, even asymptotically. Using our techniques, we address model selection in a variety of problems such as misspecified linear contextual bandits, linear bandit with unknown dimension and reinforcement learning with unknown feature maps. Our algorithm requires the knowledge of the optimal base regret to adjust the meta-algorithm learning rate. We show that without such prior knowledge any meta-algorithm can suffer a regret larger than the optimal base regret.
translated by 谷歌翻译
基于内核的模型,例如内核脊回归和高斯工艺在机器学习应用程序中无处不在,用于回归和优化。众所周知,基于内核的模型的主要缺点是高计算成本。给定$ n $样本的数据集,成本增长为$ \ Mathcal {o}(n^3)$。在某些情况下,现有的稀疏近似方法可以大大降低计算成本,从而有效地将实际成本降低到$ \ natercal {o}(n)$。尽管取得了显着的经验成功,但由于近似值而导致的误差的分析范围的现有结果仍然存在显着差距。在这项工作中,我们为NyStr \“ Om方法和稀疏变分高斯过程近似方法提供新颖的置信区间,我们使用模型的近似(代理)后差解释来建立这些方法。我们的置信区间可改善性能。回归和优化问题的界限。
translated by 谷歌翻译
Many real-world reinforcement learning tasks require control of complex dynamical systems that involve both costly data acquisition processes and large state spaces. In cases where the transition dynamics can be readily evaluated at specified states (e.g., via a simulator), agents can operate in what is often referred to as planning with a \emph{generative model}. We propose the AE-LSVI algorithm for best-policy identification, a novel variant of the kernelized least-squares value iteration (LSVI) algorithm that combines optimism with pessimism for active exploration (AE). AE-LSVI provably identifies a near-optimal policy \emph{uniformly} over an entire state space and achieves polynomial sample complexity guarantees that are independent of the number of states. When specialized to the recently introduced offline contextual Bayesian optimization setting, our algorithm achieves improved sample complexity bounds. Experimentally, we demonstrate that AE-LSVI outperforms other RL algorithms in a variety of environments when robustness to the initial state is required.
translated by 谷歌翻译
Despite the significant interest and progress in reinforcement learning (RL) problems with adversarial corruption, current works are either confined to the linear setting or lead to an undesired $\tilde{O}(\sqrt{T}\zeta)$ regret bound, where $T$ is the number of rounds and $\zeta$ is the total amount of corruption. In this paper, we consider the contextual bandit with general function approximation and propose a computationally efficient algorithm to achieve a regret of $\tilde{O}(\sqrt{T}+\zeta)$. The proposed algorithm relies on the recently developed uncertainty-weighted least-squares regression from linear contextual bandit \citep{he2022nearly} and a new weighted estimator of uncertainty for the general function class. In contrast to the existing analysis that heavily relies on the linear structure, we develop a novel technique to control the sum of weighted uncertainty, thus establishing the final regret bounds. We then generalize our algorithm to the episodic MDP setting and first achieve an additive dependence on the corruption level $\zeta$ in the scenario of general function approximation. Notably, our algorithms achieve regret bounds either nearly match the performance lower bound or improve the existing methods for all the corruption levels and in both known and unknown $\zeta$ cases.
translated by 谷歌翻译
我们考虑使用个性化的联合学习,除了全球目标外,每个客户还对最大化个性化的本地目标感兴趣。我们认为,在一般连续的动作空间设置下,目标函数属于繁殖的内核希尔伯特空间。我们提出了基于替代高斯工艺(GP)模型的算法,该算法达到了最佳的遗憾顺序(要归结为各种因素)。此外,我们表明,GP模型的稀疏近似显着降低了客户之间的沟通成本。
translated by 谷歌翻译
我们考虑使用图形结构数据定义的奖励函数的强盗优化问题。这个问题在分子设计和药物发现中具有重要的应用,在图形排列中,奖励自然不变。这种设置的主要挑战是扩展到大型域,以及带有许多节点的图形。我们通过将置换不变性嵌入我们的模型来解决这些挑战。特别是,我们表明图形神经网络(GNN)可用于估计奖励函数,假设它位于置换不变的加性核的再现内核希尔伯特空间。通过在此类内核与图形神经切线内核(GNTK)之间建立新的联系,我们介绍了第一个GNN信心绑定,并使用它来设计一个带有sublinear遗憾的相位脱口算法。我们的遗憾约束取决于GNTK的最大信息增益,我们也为此提供了界限。虽然奖励功能取决于所有$ n $节点功能,但我们的保证与图形节点$ n $的数量无关。从经验上讲,我们的方法在图形结构域上表现出竞争性能,并表现得很好。
translated by 谷歌翻译
在预测功能(假设)中获得可靠的自适应置信度集是顺序决策任务的核心挑战,例如土匪和基于模型的强化学习。这些置信度集合通常依赖于对假设空间的先前假设,例如,繁殖核Hilbert Space(RKHS)的已知核。手动设计此类内核是容易发生的,错误指定可能导致性能差或不安全。在这项工作中,我们建议从离线数据(meta-kel)中进行元学习核。对于未知核是已知碱基核的组合的情况,我们基于结构化的稀疏性开发估计量。在温和的条件下,我们保证我们的估计RKHS会产生有效的置信度集,随着越来越多的离线数据的量,它变得与鉴于真正未知内核的置信度一样紧。我们展示了我们关于内核化强盗问题(又称贝叶斯优化)的方法,我们在其中建立了遗憾的界限,与鉴于真正的内核的人竞争。我们还经验评估方法对贝叶斯优化任务的有效性。
translated by 谷歌翻译
我们在存在对抗性腐败的情况下研究线性上下文的强盗问题,在场,每回合的奖励都被对手损坏,腐败级别(即,地平线上的腐败总数)为$ c \ geq 0 $。在这种情况下,最著名的算法受到限制,因为它们要么在计算效率低下,要么需要对腐败做出强烈的假设,或者他们的遗憾至少比没有腐败的遗憾差的$ C $倍。在本文中,为了克服这些局限性,我们提出了一种基于不确定性的乐观原则的新算法。我们算法的核心是加权山脊回归,每个选择动作的重量都取决于其置信度,直到一定的阈值。 We show that for both known $C$ and unknown $C$ cases, our algorithm with proper choice of hyperparameter achieves a regret that nearly matches the lower bounds.因此,我们的算法几乎是两种情况的对数因素的最佳选择。值得注意的是,我们的算法同时对腐败和未腐败的案件($ c = 0 $)实现了近乎最理想的遗憾。
translated by 谷歌翻译
我们在嵌套政策类别的存在下研究匪徒场景中的模型选择问题,目的是获得同时的对抗和随机性(“两全其美”)高概率的遗憾保证。我们的方法要求每个基础学习者都有一个候选人的遗憾约束,可能会或可能不会举行,而我们的元算法按照一定时间表来扮演每个基础学习者,该时间表使基础学习者的候选人后悔的界限保持平衡,直到被发现违反他们的保证为止。我们开发了专门设计的仔细的错误指定测试,以将上述模型选择标准与利用环境的(潜在良性)性质的能力相结合。我们在对抗环境中恢复畜栏算法的模型选择保证,但是在实现高概率后悔界限的附加益处,特别是在嵌套对抗性线性斑块的情况下。更重要的是,我们的模型选择结果也同时在差距假设​​下的随机环境中同时保持。这些是在(线性)匪徒场景中执行模型选择时,可以达到世界上最好的(随机和对抗性)保证的第一个理论结果。
translated by 谷歌翻译
We consider optimizing a function network in the noise-free grey-box setting with RKHS function classes, where the exact intermediate results are observable. We assume that the structure of the network is known (but not the underlying functions comprising it), and we study three types of structures: (1) chain: a cascade of scalar-valued functions, (2) multi-output chain: a cascade of vector-valued functions, and (3) feed-forward network: a fully connected feed-forward network of scalar-valued functions. We propose a sequential upper confidence bound based algorithm GPN-UCB along with a general theoretical upper bound on the cumulative regret. For the Mat\'ern kernel, we additionally propose a non-adaptive sampling based method along with its theoretical upper bound on the simple regret. We also provide algorithm-independent lower bounds on the simple regret and cumulative regret, showing that GPN-UCB is near-optimal for chains and multi-output chains in broad cases of interest.
translated by 谷歌翻译
In this paper, we address the stochastic contextual linear bandit problem, where a decision maker is provided a context (a random set of actions drawn from a distribution). The expected reward of each action is specified by the inner product of the action and an unknown parameter. The goal is to design an algorithm that learns to play as close as possible to the unknown optimal policy after a number of action plays. This problem is considered more challenging than the linear bandit problem, which can be viewed as a contextual bandit problem with a \emph{fixed} context. Surprisingly, in this paper, we show that the stochastic contextual problem can be solved as if it is a linear bandit problem. In particular, we establish a novel reduction framework that converts every stochastic contextual linear bandit instance to a linear bandit instance, when the context distribution is known. When the context distribution is unknown, we establish an algorithm that reduces the stochastic contextual instance to a sequence of linear bandit instances with small misspecifications and achieves nearly the same worst-case regret bound as the algorithm that solves the misspecified linear bandit instances. As a consequence, our results imply a $O(d\sqrt{T\log T})$ high-probability regret bound for contextual linear bandits, making progress in resolving an open problem in (Li et al., 2019), (Li et al., 2021). Our reduction framework opens up a new way to approach stochastic contextual linear bandit problems, and enables improved regret bounds in a number of instances including the batch setting, contextual bandits with misspecifications, contextual bandits with sparse unknown parameters, and contextual bandits with adversarial corruption.
translated by 谷歌翻译
级别设置估计问题旨在查找域$ {\ cal x} $的所有点,其中一个未知函数$ f:{\ cal x} \ lightarrow \ mathbb {r} $超过阈值$ \ alpha $ 。估计基于可以在$ {\ cal x} $中顺序和自适应地选择的位置获取的嘈杂函数评估。阈值$ \ alpha $可以是\弹性{显式},并提供先验,或\ \ ich {隐式},相对于最佳函数值定义,即$ \ alpha =(1- \ epsilon)f(x_ \ AST)$关于给定$ \ epsilon> 0 $ why $ f(x_ \ ist)$是最大函数值,并且未知。在这项工作中,我们通过将其与最近的自适应实验设计方法相关联,为近期自适应实验设计方法提供了一种新的再现内核盗窃空间(RKHS)设置。我们假设可以通过RKHS中的函数近似于未知的拼写,并为此设置中隐含和显式案件提供新的算法,具有很强的理论保证。此外,在线性(内核)设置中,我们表明我们的界限几乎是最佳的,即,我们的上限与阈值线性匪徒的现有下限匹配。据我们所知,这项工作提供了第一个实例依赖性非渐近的上限,就匹配信息理论下限的水平设定估计的样本复杂性。
translated by 谷歌翻译
Authors are encouraged to submit new papers to INFORMS journals by means of a style file template, which includes the journal title. However, use of a template does not certify that the paper has been accepted for publication in the named journal. INFORMS journal templates are for the exclusive purpose of submitting to an INFORMS journal and should not be used to distribute the papers in print or online or to submit the papers to another publication.
translated by 谷歌翻译
我们考虑具有未知实用程序参数的多项式logit模型(MNL)下的动态分类优化问题。本文研究的主要问题是$ \ varepsilon $ - 污染模型下的模型错误指定,该模型是强大统计和机器学习中的基本模型。特别是,在整个长度$ t $的销售范围内,我们假设客户根据$(1- \ varepsilon)$ - 时间段的$(1- \ varepsilon)的基础多项式logit选择模型进行购买,并进行任意购买取而代之的是在剩余的$ \ varepsilon $ - 分数中的决策。在此模型中,我们通过主动淘汰策略制定了新的强大在线分类优化政策。我们对遗憾建立上限和下界,并表明当分类能力恒定时,我们的政策是$ t $的最佳对数因素。分类能力具有恒定的上限。我们进一步制定了一种完全自适应策略,该政策不需要任何先验知识,即污染参数$ \ varepsilon $。如果存在最佳和亚最佳产品之间存在的亚临时差距,我们还建立了依赖差距的对数遗憾上限和已知的 - $ \ VAREPSILON $和UNKNOWER-$ \ \ VAREPSILON $案例。我们的仿真研究表明,我们的政策表现优于基于上置信度范围(UCB)和汤普森采样的现有政策。
translated by 谷歌翻译
上下文匪徒问题是一个理论上合理的框架,在各个领域都有广泛的应用程序。虽然先前关于此问题的研究通常需要噪声和上下文之间的独立性,但我们的工作考虑了一个更明智的环境,其中噪声成为影响背景和奖励的潜在混杂因素。这样的混杂设置更现实,可以扩展到更广泛的应用程序。但是,未解决的混杂因素将导致奖励功能估计的偏见,从而导致极大的遗憾。为了应对混杂因素带来的挑战,我们应用了双工具变量回归,该回归可以正确识别真正的奖励功能。我们证明,在两种广泛使用的繁殖核希尔伯特空间中,该方法的收敛速率几乎是最佳的。因此,我们可以根据混杂的匪徒问题的理论保证来设计计算高效和遗憾的算法。数值结果说明了我们提出的算法在混杂的匪徒设置中的功效。
translated by 谷歌翻译
我们考虑一个多武装的强盗设置,在每一轮的开始时,学习者接收嘈杂的独立,并且可能偏见,\ emph {评估}每个臂的真正奖励,它选择$ k $武器的目标累积尽可能多的奖励超过$ $ rounds。在假设每轮在每个臂的真正奖励从固定分发中汲取的,我们得出了不同的算法方法和理论保证,具体取决于评估的生成方式。首先,在观察功能是真正奖励的遗传化线性函数时,我们在一般情况下展示$ \ widetilde {o}(t ^ {2/3})$后悔。另一方面,当观察功能是真正奖励的嘈杂线性函数时,我们就可以派生改进的$ \ widetilde {o}(\ sqrt {t})$后悔。最后,我们报告了一个实证验证,确认我们的理论发现,与替代方法进行了彻底的比较,并进一步支持在实践中实现这一环境的兴趣。
translated by 谷歌翻译
我们研究$ k $ used的上下文决斗强盗问题,一个顺序决策制定设置,其中学习者使用上下文信息来制作两个决定,但只观察到\ emph {基于优先级的反馈}建议一个决定比另一个决定更好。我们专注于可实现的遗憾最小化问题,其中反馈由一个由给定函数类$ \ mathcal f $规定的成对偏好矩阵生成。我们提供了一种新的算法,实现了最佳反应遗憾的新概念的最佳遗憾,这是一个严格更强烈的性能测量,而不是先前作品所考虑的绩效衡量标准。该算法还在计算上有效,在多项式时间中运行,假设访问在线丢失回归超过$ \ mathcal f $。这可以解决dud \'ik等人的开放问题。[2015]关于Oracle高效,后悔 - 用于上下文决斗匪徒的最佳算法。
translated by 谷歌翻译