本文证明了鲁棒性意味着通过数据依赖性的概括界限进行概括。结果,鲁棒性和概括被证明是以数据依赖性方式紧密连接的。我们的界限改善了以前的两个方向的界限,以解决自2010年以来几乎没有发展的开放问题。第一个是减少对覆盖码的依赖。第二个是消除对假设空间的依赖性。我们提供了几个示例,包括套索和深度学习的例子,其中我们的界限被证明是可取的。关于现实世界数据和理论模型的实验表明,在各种情况下的近乎指数改进。为了实现这些改进,我们不需要关于未知分布的其他假设。取而代之的是,我们仅包含训练样本的可观察到的可计算特性。一个关键的技术创新是对多项式随机变量的改善浓度,它超出了鲁棒性和泛化。
translated by 谷歌翻译
This paper provides theoretical insights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness, and sharp minima, responding to an open question in the literature. We also discuss approaches to provide non-vacuous generalization guarantees for deep learning. Based on theoretical observations, we propose new open problems and discuss the limitations of our results.
translated by 谷歌翻译
所有著名的机器学习算法构成了受监督和半监督的学习工作,只有在一个共同的假设下:培训和测试数据遵循相同的分布。当分布变化时,大多数统计模型必须从新收集的数据中重建,对于某些应用程序,这些数据可能是昂贵或无法获得的。因此,有必要开发方法,以减少在相关领域中可用的数据并在相似领域中进一步使用这些数据,从而减少需求和努力获得新的标签样品。这引起了一个新的机器学习框架,称为转移学习:一种受人类在跨任务中推断知识以更有效学习的知识能力的学习环境。尽管有大量不同的转移学习方案,但本调查的主要目的是在特定的,可以说是最受欢迎的转移学习中最受欢迎的次级领域,概述最先进的理论结果,称为域适应。在此子场中,假定数据分布在整个培训和测试数据中发生变化,而学习任务保持不变。我们提供了与域适应性问题有关的现有结果的首次最新描述,该结果涵盖了基于不同统计学习框架的学习界限。
translated by 谷歌翻译
我们重新审视耐受分发测试的问题。也就是说,给出来自未知分发$ P $超过$ \ {1,\ dots,n \} $的样本,它是$ \ varepsilon_1 $ -close到或$ \ varepsilon_2 $ -far从引用分发$ q $(总变化距离)?尽管过去十年来兴趣,但在极端情况下,这个问题很好。在无噪声设置(即,$ \ varepsilon_1 = 0 $)中,样本复杂性是$ \ theta(\ sqrt {n})$,强大的域大小。在频谱的另一端时,当$ \ varepsilon_1 = \ varepsilon_2 / 2 $时,样本复杂性跳转到勉强su​​blinear $ \ theta(n / \ log n)$。然而,非常少于中级制度。我们充分地表征了分发测试中的公差价格,作为$ N $,$ varepsilon_1 $,$ \ varepsilon_2 $,最多一个$ \ log n $ factor。具体来说,我们显示了\ [\ tilde \ theta \ left的样本复杂性(\ frac {\ sqrt {n}} {\ varepsilon_2 ^ {2}} + \ frac {n} {\ log n} \ cdot \ max \左\ {\ frac {\ varepsilon_1} {\ varepsilon_2 ^ 2},\ left(\ frac {\ varepsilon_1} {\ varepsilon_2 ^ 2} \右)^ {\!\!\!2} \ \ \} \右) ,\]提供两个先前已知的案例之间的顺利折衷。我们还为宽容的等价测试问题提供了类似的表征,其中$ p $和$ q $均未赘述。令人惊讶的是,在这两种情况下,对样本复杂性的主数量是比率$ \ varepsilon_1 / varepsilon_2 ^ 2 $,而不是更直观的$ \ varepsilon_1 / \ varepsilon_2 $。特别是技术兴趣是我们的下限框架,这涉及在以往的工作中处理不对称所需的新颖近似性理论工具,从而缺乏以前的作品。
translated by 谷歌翻译
训练神经网络的一种常见方法是将所有权重初始化为独立的高斯向量。我们观察到,通过将权重初始化为独立对,每对由两个相同的高斯向量组成,我们可以显着改善收敛分析。虽然已经研究了类似的技术来进行随机输入[Daniely,Neurips 2020],但尚未使用任意输入进行分析。使用此技术,我们展示了如何显着减少两层relu网络所需的神经元数量,均在逻辑损失的参数化设置不足的情况下,大约$ \ gamma^{ - 8} $ [Ji and telgarsky,ICLR, 2020]至$ \ gamma^{ - 2} $,其中$ \ gamma $表示带有神经切线内核的分离边距,以及在与平方损失的过度参数化设置中,从大约$ n^4 $ [song [song]和Yang,2019年]至$ n^2 $,隐含地改善了[Brand,Peng,Song和Weinstein,ITCS 2021]的近期运行时间。对于参数不足的设置,我们还证明了在先前工作时改善的新下限,并且在某些假设下是最好的。
translated by 谷歌翻译
We define notions of stability for learning algorithms and show how to use these notions to derive generalization error bounds based on the empirical error and the leave-one-out error. The methods we use can be applied in the regression framework as well as in the classification one when the classifier is obtained by thresholding a real-valued function. We study the stability properties of large classes of learning algorithms such as regularization based algorithms. In particular we focus on Hilbert space regularization and Kullback-Leibler regularization. We demonstrate how to apply the results to SVM for regression and classification.1. For a qualitative discussion about sensitivity analysis with links to other resources see e.g. http://sensitivity-analysis.jrc.cec.eu.int/
translated by 谷歌翻译
This paper presents a margin-based multiclass generalization bound for neural networks that scales with their margin-normalized spectral complexity: their Lipschitz constant, meaning the product of the spectral norms of the weight matrices, times a certain correction factor. This bound is empirically investigated for a standard AlexNet network trained with SGD on the mnist and cifar10 datasets, with both original and random labels; the bound, the Lipschitz constants, and the excess risks are all in direct correlation, suggesting both that SGD selects predictors whose complexity scales with the difficulty of the learning task, and secondly that the presented bound is sensitive to this complexity.
translated by 谷歌翻译
State-of-the-art results on image recognition tasks are achieved using over-parameterized learning algorithms that (nearly) perfectly fit the training set and are known to fit well even random labels. This tendency to memorize the labels of the training data is not explained by existing theoretical analyses. Memorization of the training data also presents significant privacy risks when the training data contains sensitive personal information and thus it is important to understand whether such memorization is necessary for accurate learning.We provide the first conceptual explanation and a theoretical model for this phenomenon. Specifically, we demonstrate that for natural data distributions memorization of labels is necessary for achieving closeto-optimal generalization error. Crucially, even labels of outliers and noisy labels need to be memorized. The model is motivated and supported by the results of several recent empirical works. In our model, data is sampled from a mixture of subpopulations and our results show that memorization is necessary whenever the distribution of subpopulation frequencies is long-tailed. Image and text data is known to be long-tailed and therefore our results establish a formal link between these empirical phenomena. Our results allow to quantify the cost of limiting memorization in learning and explain the disparate effects that privacy and model compression have on different subgroups.
translated by 谷歌翻译
这项工作确立了梯度流量(GF)和随机梯度下降(SGD)的低测试误差(SGD)在具有标准初始化的两层relu网络上,在三个方案中,关键的重量集很少旋转(自然要么是由于GF和SGD,要么是由于GF和SGD,或达到人为的约束),并利用边缘作为核心分析技术。第一个制度几乎是初始化的,特别是直到权重以$ \ mathcal {o}(\ sqrt m)$移动为止,其中$ m $表示网络宽度,这与$ \ mathcal {o}(O}(O}(O})形成鲜明对比) 1)神经切线内核(NTK)允许的重量运动;在这里显示,GF和SGD仅需要网络宽度和样本数量与NTK边缘成反比,此外,GF至少达到了NTK保证金本身,这足以建立避免距离范围目标的不良KKT点的逃脱,该点的距离逃脱了。而先前的工作只能确定不折扣但任意的边缘。第二个制度是神经塌陷(NC)设置,其中数据在于极度隔离的组中,样品复杂性尺度与组数。在这里,先前工作的贡献是对初始化的整个GF轨迹的分析。最后,如果内层的权重限制为仅在规范中变化并且无法旋转,则具有较大宽度的GF达到了全球最大边缘,并且其样品复杂度与它们的逆尺度相比;这与先前的工作相反,后者需要无限的宽度和一个棘手的双收敛假设。作为纯粹的技术贡献,这项工作开发了各种潜在功能和其他工具,希望有助于未来的工作。
translated by 谷歌翻译
Machine learning models are often susceptible to adversarial perturbations of their inputs. Even small perturbations can cause state-of-the-art classifiers with high "standard" accuracy to produce an incorrect prediction with high confidence. To better understand this phenomenon, we study adversarially robust learning from the viewpoint of generalization. We show that already in a simple natural data model, the sample complexity of robust learning can be significantly larger than that of "standard" learning. This gap is information theoretic and holds irrespective of the training algorithm or the model family. We complement our theoretical results with experiments on popular image classification datasets and show that a similar gap exists here as well. We postulate that the difficulty of training robust classifiers stems, at least partially, from this inherently larger sample complexity.
translated by 谷歌翻译
本文为信号去噪提供了一般交叉验证框架。然后将一般框架应用于非参数回归方法,例如趋势过滤和二元推车。然后显示所得到的交叉验证版本以获得最佳调谐的类似物所熟知的几乎相同的收敛速度。没有任何先前的趋势过滤或二元推车的理论分析。为了说明框架的一般性,我们还提出并研究了两个基本估算器的交叉验证版本;套索用于高维线性回归和矩阵估计的奇异值阈值阈值。我们的一般框架是由Chatterjee和Jafarov(2015)的想法的启发,并且可能适用于使用调整参数的广泛估算方法。
translated by 谷歌翻译
PAC-Bayes has recently re-emerged as an effective theory with which one can derive principled learning algorithms with tight performance guarantees. However, applications of PAC-Bayes to bandit problems are relatively rare, which is a great misfortune. Many decision-making problems in healthcare, finance and natural sciences can be modelled as bandit problems. In many of these applications, principled algorithms with strong performance guarantees would be very much appreciated. This survey provides an overview of PAC-Bayes performance bounds for bandit problems and an experimental comparison of these bounds. Our experimental comparison has revealed that available PAC-Bayes upper bounds on the cumulative regret are loose, whereas available PAC-Bayes lower bounds on the expected reward can be surprisingly tight. We found that an offline contextual bandit algorithm that learns a policy by optimising a PAC-Bayes bound was able to learn randomised neural network polices with competitive expected reward and non-vacuous performance guarantees.
translated by 谷歌翻译
算法高斯化是一种现象,当使用随机素描或采样方法生成较小的大数据集的较小表示时,可能会出现的现象:对于某些任务,已经观察到这些草图表示表现出许多可靠的性能特征,这些性能是在数据样本中出现的,这些性能来自次高斯随机设计,是一个强大的数据分布统计模型。但是,这种现象仅研究了特定的任务和指标,或依靠计算昂贵的方法。我们通过为平均值提供用于高斯数据分布的算法框架来解决这一问题,并证明可以有效构建几乎无法区分的数据草图(与亚高斯随机设计有关的总变化距离)。特别是,依靠最近引入的素描技术称为杠杆得分稀疏(少)嵌入,我们表明一个人可以构造$ n \ times d $矩阵$ a $的$ n \ times d $ sketch of $ n \ times d $ n \ ll n $,几乎与次高斯设计几乎没有区别$ a $中的非零条目的数量。结果,可以直接适用于我们的草图框架,可直接适用于我们的草图框架。我们通过对草图最小二乘正方形的新近似保证进行了说明。
translated by 谷歌翻译
套索是一种高维回归的方法,当时,当协变量$ p $的订单数量或大于观测值$ n $时,通常使用它。由于两个基本原因,经典的渐近态性理论不适用于该模型:$(1)$正规风险是非平滑的; $(2)$估算器$ \ wideHat {\ boldsymbol {\ theta}} $与true参数vector $ \ boldsymbol {\ theta}^*$无法忽略。结果,标准的扰动论点是渐近正态性的传统基础。另一方面,套索估计器可以精确地以$ n $和$ p $大,$ n/p $的订单为一。这种表征首先是在使用I.I.D的高斯设计的情况下获得的。协变量:在这里,我们将其推广到具有非偏差协方差结构的高斯相关设计。这是根据更简单的``固定设计''模型表示的。我们在两个模型中各种数量的分布之间的距离上建立了非反应界限,它们在合适的稀疏类别中均匀地固定在信号上$ \ boldsymbol {\ theta}^*$。作为应用程序,我们研究了借助拉索的分布,并表明需要校正程度对于计算有效的置信区间是必要的。
translated by 谷歌翻译
收购数据是机器学习的许多应用中的一项艰巨任务,只有一个人希望并且预期人口风险在单调上汇率增加(更好的性能)。事实证明,甚至对于最小化经验风险的最大限度的算法,甚至不令人惊讶的情况。在训练中的风险和不稳定的非单调行为表现出并出现在双重血统描述中的流行深度学习范式中。这些问题突出了目前对学习算法和泛化的理解缺乏了解。因此,追求这种行为的表征是至关重要的,这是至关重要的。在本文中,我们在弱假设下获得了一致和风险的单调算法,从而解决了一个打开问题Viering等。 2019关于如何避免风险曲线的非单调行为。我们进一步表明,风险单调性不一定以更糟糕的风险率的价格出现。为实现这一目标,我们推出了持有某些非I.I.D的独立利益的新经验伯恩斯坦的浓度不等式。鞅差异序列等进程。
translated by 谷歌翻译
深度分离结果提出了对深度神经网络过较浅的架构的好处的理论解释,建立前者具有卓越的近似能力。然而,没有已知的结果,其中更深的架构利用这种优势成为可提供的优化保证。我们证明,当数据由具有满足某些温和假设的径向对称的分布产生的数据时,梯度下降可以使用具有两层S形激活的深度2神经网络有效地学习球指示器功能,并且隐藏层固定在一起训练。由于众所周知,当使用用单层非线性的深度2网络(Safran和Shamir,2017)使用深度2网络时,球指示器难以近似于一定的重型分配,这建立了我们最好的知识,基于第一优化的分离结果,其中近似架构的近似效益在实践中可怕的。我们的证明技术依赖于随机特征方法,该方法减少了用单个神经元学习的问题,其中新工具需要在数据分布重尾时显示梯度下降的收敛。
translated by 谷歌翻译
我们在决策边界是一定规律的假设下,研究从无噪声训练样本的学习分类功能的问题。我们为这一估计问题建立了普遍的下限,对于连续决策边界的一般阶级。对于本地禁区的类别,我们发现最佳估计率基本上独立于底层维度,并且可以通过在适当类的深神经网络上通过经验风险最小化方法实现。这些结果基于$ l ^ 1 $和$ l ^ \ infty $ intty $ inthty $ off的禁区常规职能的新颖估计数。
translated by 谷歌翻译
The multi-armed bandit problem is a popular model for studying exploration/exploitation trade-off in sequential decision problems. Many algorithms are now available for this well-studied problem. One of the earliest algorithms, given by W. R. Thompson, dates back to 1933. This algorithm, referred to as Thompson Sampling, is a natural Bayesian algorithm. The basic idea is to choose an arm to play according to its probability of being the best arm. Thompson Sampling algorithm has experimentally been shown to be close to optimal. In addition, it is efficient to implement and exhibits several desirable properties such as small regret for delayed feedback. However, theoretical understanding of this algorithm was quite limited. In this paper, for the first time, we show that Thompson Sampling algorithm achieves logarithmic expected regret for the stochastic multi-armed bandit problem. More precisely, for the stochastic two-armed bandit problem, the expected regret in time T is O( ln T ∆ + 1 ∆ 3 ). And, for the stochastic N -armed bandit problem, the expected regret in time) 2 ln T ). Our bounds are optimal but for the dependence on ∆i and the constant factors in big-Oh.
translated by 谷歌翻译
我们使用对单个的,相同的$ d $维状态的相同副本进行的测量来研究量子断层扫描和阴影断层扫描的问题。我们首先因Haah等人而重新审视已知的下限。 (2017年)在痕量距离上具有准确性$ \ epsilon $的量子断层扫描,当测量选择与先前观察到的结果无关(即它们是非适应性的)时。我们简要地证明了这一结果。当学习者使用具有恒定结果数量的测量值时,这会导致更强的下限。特别是,这严格确定了民间传说的最佳性``Pauli phymography''算法的样本复杂性。我们还得出了$ \ omega(r^2 d/\ epsilon^2)$和$ \ omega(r^2 d/\ epsilon^2)的新颖界限( R^2 d^2/\ epsilon^2)$用于学习排名$ r $状态,分别使用任意和恒定的结果测量,在非适应性情况下。除了样本复杂性,对于学习量子的实际意义,是一种实际意义的资源状态是算法使用的不同测量值的数量。我们将下限扩展到学习者从固定的$ \ exp(o(d))$测量的情况下进行自适应测量的情况。这特别意味着适应性。没有使用可有效实现的单拷贝测量结果给我们任何优势。在目标是预测给定的可观察到给定序列的期望值的情况下,我们还获得了类似的界限,该任务被称为阴影层析成像。在适应性的情况下单拷贝测量可通过多项式大小的电路实现,我们证明了基于计算给定可观察物的样本平均值的直接策略是最佳的。
translated by 谷歌翻译
随机一阶方法是训练大规模机器学习模型的标准。随机行为可能导致算法的特定运行导​​致高度次优的目标值,而通常证明理论保证是出于目标值的期望。因此,从理论上保证算法具有很高的可能性,这一点至关重要。非平滑随机凸优化的现有方法具有复杂的界限,其依赖性对置信度或对数为负功率,但在额外的假设下是高斯(轻尾)噪声分布的额外假设,这些噪声分布在实践中可能不存在。在我们的论文中,我们解决了这个问题,并得出了第一个高概率收敛的结果,并以对数依赖性对非平滑凸的随机优化问题的置信度依赖,并带有非Sub-Gaussian(重尾)噪声。为了得出我们的结果,我们建议针对两种随机方法进行梯度剪辑的新步骤规则。此外,我们的分析适用于使用H \“较旧连续梯度的通用平滑目标,对于这两种方法,我们都为强烈凸出问题提供了扩展。最后,我们的结果暗示我们认为的第一种(加速)方法也具有最佳的迭代。在所有制度中,Oracle的复杂性,第二个机制在非平滑设置中都是最佳的。
translated by 谷歌翻译