We study a natural extension of classical empirical risk minimization, where the hypothesis space is a random subspace of a given space. In particular, we consider possibly data dependent subspaces spanned by a random subset of the data, recovering as a special case Nystrom approaches for kernel methods. Considering random subspaces naturally leads to computational savings, but the question is whether the corresponding learning accuracy is degraded. These statistical-computational tradeoffs have been recently explored for the least squares loss and self-concordant loss functions, such as the logistic loss. Here, we work to extend these results to convex Lipschitz loss functions, that might not be smooth, such as the hinge loss used in support vector machines. This unified analysis requires developing new proofs, that use different technical tools, such as sub-gaussian inputs, to achieve fast rates. Our main results show the existence of different settings, depending on how hard the learning problem is, for which computational efficiency can be improved with no loss in performance.
translated by 谷歌翻译
我们在对数损失下引入条件密度估计的过程,我们调用SMP(样本Minmax预测器)。该估算器最大限度地减少了统计学习的新一般过度风险。在标准示例中,此绑定量表为$ d / n $,$ d $ d $模型维度和$ n $ sample大小,并在模型拼写条目下批判性仍然有效。作为一个不当(超出型号)的程序,SMP在模型内估算器(如最大似然估计)的内部估算器上,其风险过高的风险降低。相比,与顺序问题的方法相比,我们的界限删除了SubOltimal $ \ log n $因子,可以处理无限的类。对于高斯线性模型,SMP的预测和风险受到协变量的杠杆分数,几乎匹配了在没有条件的线性模型的噪声方差或近似误差的条件下匹配的最佳风险。对于Logistic回归,SMP提供了一种非贝叶斯方法来校准依赖于虚拟样本的概率预测,并且可以通过解决两个逻辑回归来计算。它达到了$ O的非渐近风险((d + b ^ 2r ^ 2)/ n)$,其中$ r $绑定了特征的规范和比较参数的$ B $。相比之下,在模型内估计器内没有比$ \ min达到更好的速率({b r} / {\ sqrt {n}},{d e ^ {br} / {n})$。这为贝叶斯方法提供了更实用的替代方法,这需要近似的后部采样,从而部分地解决了Foster等人提出的问题。 (2018)。
translated by 谷歌翻译
光谱滤波理论是一个显着的工具,可以了解用核心学习的统计特性。对于最小二乘来,它允许导出各种正则化方案,其产生的速度超越风险的收敛率比Tikhonov正规化更快。这通常通过利用称为源和容量条件的经典假设来实现,这表征了学习任务的难度。为了了解来自其他损失功能的估计,Marteau-Ferey等。已经将Tikhonov正规化理论扩展到广义自助损失功能(GSC),其包含例如物流损失。在本文中,我们进一步逐步,并表明通过使用迭代的Tikhonov正规方案,可以实现快速和最佳的速率,该计划与优化中的近端点方法有本质相关,并克服了古典Tikhonov规范化的限制。
translated by 谷歌翻译
我们研究了称为“乐观速率”(Panchenko 2002; Srebro等,2010)的统一收敛概念,用于与高斯数据的线性回归。我们的精致分析避免了现有结果中的隐藏常量和对数因子,这已知在高维设置中至关重要,特别是用于了解插值学习。作为一个特殊情况,我们的分析恢复了Koehler等人的保证。(2021年),在良性过度的过度条件下,严格地表征了低规范内插器的人口风险。但是,我们的乐观速度绑定还分析了具有任意训练错误的预测因子。这使我们能够在随机设计下恢复脊和套索回归的一些经典统计保障,并有助于我们在过度参数化制度中获得精确了解近端器的过度风险。
translated by 谷歌翻译
内核平均值嵌入是一种强大的工具,可以代表任意空间上的概率分布作为希尔伯特空间中的单个点。然而,计算和存储此类嵌入的成本禁止其在大规模设置中的直接使用。我们提出了一个基于NyStr \“ Om方法的有效近似过程,该过程利用了数据集的一个小随机子集。我们的主要结果是该过程的近似误差的上限。它在子样本大小上产生足够的条件以获得足够的条件。降低计算成本的同时,标准的$ n^{ - 1/2} $。我们讨论了此结果的应用,以近似的最大平均差异和正交规则,并通过数值实验说明了我们的理论发现。
translated by 谷歌翻译
在这项工作中,我们考虑线性逆问题$ y = ax + \ epsilon $,其中$ a \ colon x \ to y $是可分离的hilbert spaces $ x $和$ y $之间的已知线性运算符,$ x $。 $ x $和$ \ epsilon $中的随机变量是$ y $的零平均随机过程。该设置涵盖成像中的几个逆问题,包括去噪,去束和X射线层析造影。在古典正规框架内,我们专注于正则化功能的情况下未能先验,而是从数据中学习。我们的第一个结果是关于均方误差的最佳广义Tikhonov规则器的表征。我们发现它完全独立于前向操作员$ a $,并仅取决于$ x $的平均值和协方差。然后,我们考虑从两个不同框架中设置的有限训练中学习常规程序的问题:一个监督,根据$ x $和$ y $的样本,只有一个无人监督,只基于$ x $的样本。在这两种情况下,我们证明了泛化界限,在X $和$ \ epsilon $的分发的一些弱假设下,包括子高斯变量的情况。我们的界限保持在无限尺寸的空间中,从而表明更精细和更细的离散化不会使这个学习问题更加困难。结果通过数值模拟验证。
translated by 谷歌翻译
We consider the problem of estimating the optimal transport map between a (fixed) source distribution $P$ and an unknown target distribution $Q$, based on samples from $Q$. The estimation of such optimal transport maps has become increasingly relevant in modern statistical applications, such as generative modeling. At present, estimation rates are only known in a few settings (e.g. when $P$ and $Q$ have densities bounded above and below and when the transport map lies in a H\"older class), which are often not reflected in practice. We present a unified methodology for obtaining rates of estimation of optimal transport maps in general function spaces. Our assumptions are significantly weaker than those appearing in the literature: we require only that the source measure $P$ satisfies a Poincar\'e inequality and that the optimal map be the gradient of a smooth convex function that lies in a space whose metric entropy can be controlled. As a special case, we recover known estimation rates for bounded densities and H\"older transport maps, but also obtain nearly sharp results in many settings not covered by prior work. For example, we provide the first statistical rates of estimation when $P$ is the normal distribution and the transport map is given by an infinite-width shallow neural network.
translated by 谷歌翻译
我们研究了非参数脊的最小二乘的学习属性。特别是,我们考虑常见的估计人的估计案例,由比例依赖性内核定义,并专注于规模的作用。这些估计器内插数据,可以显示规模来通过条件号控制其稳定性。我们的分析表明,这是不同的制度,具体取决于样本大小,其尺寸与问题的平滑度之间的相互作用。实际上,当样本大小小于数据维度中的指数时,可以选择比例,以便学习错误减少。随着样本尺寸变大,总体错误停止减小但有趣地可以选择规模,使得噪声引起的差异仍然存在界线。我们的分析结合了概率,具有来自插值理论的许多分析技术。
translated by 谷歌翻译
我们考虑与高斯数据的高维线性回归中的插值学习,并在类高斯宽度方面证明了任意假设类别中的内插器的泛化误差。将通用绑定到欧几里德常规球恢复了Bartlett等人的一致性结果。(2020)对于最小规范内插器,并确认周等人的预测。(2020)在高斯数据的特殊情况下,对于近乎最小常态的内插器。我们通过将其应用于单位来证明所界限的一般性,从而获得最小L1-NORM Interpoolator(基础追踪)的新型一致性结果。我们的结果表明,基于规范的泛化界限如何解释并用于分析良性过度装备,至少在某些设置中。
translated by 谷歌翻译
In many modern applications of deep learning the neural network has many more parameters than the data points used for its training. Motivated by those practices, a large body of recent theoretical research has been devoted to studying overparameterized models. One of the central phenomena in this regime is the ability of the model to interpolate noisy data, but still have test error lower than the amount of noise in that data. arXiv:1906.11300 characterized for which covariance structure of the data such a phenomenon can happen in linear regression if one considers the interpolating solution with minimum $\ell_2$-norm and the data has independent components: they gave a sharp bound on the variance term and showed that it can be small if and only if the data covariance has high effective rank in a subspace of small co-dimension. We strengthen and complete their results by eliminating the independence assumption and providing sharp bounds for the bias term. Thus, our results apply in a much more general setting than those of arXiv:1906.11300, e.g., kernel regression, and not only characterize how the noise is damped but also which part of the true signal is learned. Moreover, we extend the result to the setting of ridge regression, which allows us to explain another interesting phenomenon: we give general sufficient conditions under which the optimal regularization is negative.
translated by 谷歌翻译
在机器学习通常与优化通过训练数据定义实证目标的最小化交易。然而,学习的最终目的是尽量减少对未来的数据错误(测试误差),为此,训练数据只提供部分信息。这种观点认为,是实际可行的优化问题是基于不准确的数量在本质上是随机的。在本文中,我们显示了如何概率的结果,特别是浓度梯度,可以用来自不精确优化结果来导出尖锐测试误差保证组合。通过考虑无约束的目标,我们强调优化隐含正规化性学习。
translated by 谷歌翻译
我们提出和研究内核偶联梯度方法(KCGM),并在可分离的希尔伯特空间上进行最小二乘回归的随机投影。考虑两种类型的随机草图和nyStr \“ {o} m子采样产生的随机投影,我们在适当的停止规则下证明了有关算法的规范变体的最佳统计结果。尤其是我们的结果表明,如果投影维度显示了投影维度与问题的有效维度成正比,带有随机草图的KCGM可以最佳地概括,同时获得计算优势。作为推论,我们在良好条件方面的经典KCGM得出了最佳的经典KCGM,因为目标函数可能不会不会在假设空间中。
translated by 谷歌翻译
Influence diagnostics such as influence functions and approximate maximum influence perturbations are popular in machine learning and in AI domain applications. Influence diagnostics are powerful statistical tools to identify influential datapoints or subsets of datapoints. We establish finite-sample statistical bounds, as well as computational complexity bounds, for influence functions and approximate maximum influence perturbations using efficient inverse-Hessian-vector product implementations. We illustrate our results with generalized linear models and large attention based models on synthetic and real data.
translated by 谷歌翻译
在本文中,我们研究了可分离的希尔伯特空间的回归问题,并涵盖了繁殖核希尔伯特空间的非参数回归。我们研究了一类光谱/正则化算法,包括脊回归,主成分回归和梯度方法。我们证明了最佳,高概率的收敛性在研究算法的规范变体方面,考虑到对假设空间的能力假设以及目标函数的一般源条件。因此,我们以最佳速率获得了几乎确定的收敛结果。我们的结果改善并推广了先前的结果,以填补了无法实现的情况的理论差距。
translated by 谷歌翻译
我们解决了条件平均嵌入(CME)的内核脊回归估算的一致性,这是给定$ y $ x $的条件分布的嵌入到目标重现内核hilbert space $ hilbert space $ hilbert Space $ \ Mathcal {H} _y $ $ $ $ 。 CME允许我们对目标RKHS功能的有条件期望,并已在非参数因果和贝叶斯推论中使用。我们解决了错误指定的设置,其中目标CME位于Hilbert-Schmidt操作员的空间中,该操作员从$ \ Mathcal {H} _X _x $和$ L_2 $和$ \ MATHCAL {H} _Y $ $之间的输入插值空间起作用。该操作员的空间被证明是新定义的矢量值插值空间的同构。使用这种同构,我们在未指定的设置下为经验CME估计量提供了一种新颖的自适应统计学习率。我们的分析表明,我们的费率与最佳$ o(\ log n / n)$速率匹配,而无需假设$ \ Mathcal {h} _y $是有限维度。我们进一步建立了学习率的下限,这表明所获得的上限是最佳的。
translated by 谷歌翻译
比较概率分布是许多机器学习算法的关键。最大平均差异(MMD)和最佳运输距离(OT)是在过去几年吸引丰富的关注的概率措施之间的两类距离。本文建立了一些条件,可以通过MMD规范控制Wassersein距离。我们的作品受到压缩统计学习(CSL)理论的推动,资源有效的大规模学习的一般框架,其中训练数据总结在单个向量(称为草图)中,该训练数据捕获与所考虑的学习任务相关的信息。在CSL中的现有结果启发,我们介绍了H \“较旧的较低限制的等距属性(H \”较旧的LRIP)并表明这家属性具有有趣的保证对压缩统计学习。基于MMD与Wassersein距离之间的关系,我们通过引入和研究学习任务的Wassersein可读性的概念来提供压缩统计学习的保证,即概率分布之间的某些特定于特定的特定度量,可以由Wassersein界定距离。
translated by 谷歌翻译
Iterative regularization is a classic idea in regularization theory, that has recently become popular in machine learning. On the one hand, it allows to design efficient algorithms controlling at the same time numerical and statistical accuracy. On the other hand it allows to shed light on the learning curves observed while training neural networks. In this paper, we focus on iterative regularization in the context of classification. After contrasting this setting with that of regression and inverse problems, we develop an iterative regularization approach based on the use of the hinge loss function. More precisely we consider a diagonal approach for a family of algorithms for which we prove convergence as well as rates of convergence. Our approach compares favorably with other alternatives, as confirmed also in numerical simulations.
translated by 谷歌翻译
尽管U统计量在现代概率和统计学中存在着无处不在的,但其在依赖框架中的非反应分析可能被忽略了。在最近的一项工作中,已经证明了对统一的马尔可夫链的U级统计数据的新浓度不平等。在本文中,我们通过在三个不同的研究领域中进一步推动了当前知识状态,将这一理论突破付诸实践。首先,我们为使用MCMC方法估算痕量类积分运算符光谱的新指数不平等。新颖的是,这种结果适用于具有正征和负征值的内核,据我们所知,这是新的。此外,我们研究了使用成对损失函数和马尔可夫链样品的在线算法的概括性能。我们通过展示如何从任何在线学习者产生的假设序列中提取低风险假设来提供在线到批量转换结果。我们最终对马尔可夫链的不变度度量的密度进行了拟合优度测试的非反应分析。我们确定了一些类别的替代方案,基于$ L_2 $距离的测试具有规定的功率。
translated by 谷歌翻译
内核方法是学习算法,这些算法享有坚实的理论基础,同时遭受了重要的计算局限性。素描包括在缩小尺寸的子空间中寻找解决方案,是一种经过广泛研究的方法来减轻这种数值负担。但是,快速的草图策略(例如非自适应子采样)大大降低了算法的保证,而理论上准确的草图(例如高斯曲线)在实践中的实践相对较慢。在本文中,我们介绍了$ p $ -sparsified的草图,这些草图结合了两种方法的好处,以实现统计准确性和计算效率之间的良好权衡。为了支持我们的方法,我们在单个和多个输出问题上得出了多余的风险范围,并具有通用Lipschitz损失,从可靠的回归到多个分位数回归为广泛的应用提供了新的保证。我们还提供了草图优于最近SOTA方法的优势的经验证据。
translated by 谷歌翻译
Many problems in causal inference and economics can be formulated in the framework of conditional moment models, which characterize the target function through a collection of conditional moment restrictions. For nonparametric conditional moment models, efficient estimation often relies on preimposed conditions on various measures of ill-posedness of the hypothesis space, which are hard to validate when flexible models are used. In this work, we address this issue by proposing a procedure that automatically learns representations with controlled measures of ill-posedness. Our method approximates a linear representation defined by the spectral decomposition of a conditional expectation operator, which can be used for kernelized estimators and is known to facilitate minimax optimal estimation in certain settings. We show this representation can be efficiently estimated from data, and establish L2 consistency for the resulting estimator. We evaluate the proposed method on proximal causal inference tasks, exhibiting promising performance on high-dimensional, semi-synthetic data.
translated by 谷歌翻译