机器学习模型的概括对数据,模型和学习算法具有复杂的依赖性。我们研究训练和测试性能,以及它们在不同数据集样本上的差异给出的概括差距,以理解其``典型''行为。我们得出了差距的表达式,作为模型之间协方差的函数参数分布和列车损耗以及平均测试性能的另一种表达,显示了测试概括仅取决于数据平均参数分布和数据平均损失。我们显示,对于大型模型参数分布,修改的概括差距为始终是非负的。通过进一步专门针对由随机梯度下降(SGD)产生的参数分布,以及一些近似值和建模考虑,我们能够预测有关通用差距和模型训练和测试性能如何变化为一个方面的一些方面SGD噪声的功能。我们基于RESNET体系结构对CIFAR10分类任务进行经验评估这些预测。
translated by 谷歌翻译
长期存在的辩论围绕着相关的假设,即低曲率的最小值更好地推广,而SGD则不鼓励曲率。我们提供更完整和细微的观点,以支持两者。首先,我们表明曲率通过两种新机制损害了测试性能,除了已知的参数搭配机制外,弯曲和偏置曲线除了偏置和偏置。尽管曲率不是,但对测试性能的三个曲率介导的贡献是重复的,尽管曲率不是。移位横向的变化是连接列车和测试局部最小值的线路,由于数据集采样或分布位移而差异。尽管在训练时间的转移尚不清楚,但仍可以通过最大程度地减少总体曲率来减轻横向横向。其次,我们得出了一种新的,明确的SGD稳态分布,表明SGD优化了与火车损失相关的有效潜力,并且SGD噪声介导了这种有效潜力的深层与低外生区域之间的权衡。第三,将我们的测试性能分析与SGD稳态相结合,表明,对于小的SGD噪声,移位膜可能是三种机制中最重要的。我们的实验证实了狂热对测试损失的影响,并进一步探索了SGD噪声与曲率之间的关系。
translated by 谷歌翻译
我们研究了使用尖刺,现场依赖的随机矩阵理论研究迷你批次对深神经网络损失景观的影响。我们表明,批量黑森州的极值值的大小大于经验丰富的黑森州。我们还获得了类似的结果对Hessian的概括高斯牛顿矩阵近似。由于我们的定理,我们推导出作为批量大小的最大学习速率的分析表达式,为随机梯度下降(线性缩放)和自适应算法(例如ADAM(Square Root Scaling)提供了通知实际培训方案,例如光滑,非凸深神经网络。虽然随机梯度下降的线性缩放是在我们概括的更多限制性条件下导出的,但是适应优化者的平方根缩放规则是我们的知识,完全小说。随机二阶方法和自适应方法的百分比,我们得出了最小阻尼系数与学习率与批量尺寸的比率成比例。我们在Cifar-$ 100 $和ImageNet数据集上验证了我们的VGG / WimerEsnet架构上的索赔。根据我们对象检的调查,我们基于飞行学习率和动量学习者开发了一个随机兰齐齐竞争,这避免了对这些关键的超参数进行昂贵的多重评估的需求,并在预残留的情况下显示出良好的初步结果Cifar的architecure - $ 100 $。
translated by 谷歌翻译
我们使用高斯过程扰动模型在高维二次上的真实和批量风险表面之间的高斯过程扰动模型分析和解释迭代平均的泛化性能。我们从我们的理论结果中获得了三个现象\姓名:}(1)将迭代平均值(ia)与大型学习率和正则化进行了改进的正规化的重要性。 (2)对较少频繁平均的理由。 (3)我们预计自适应梯度方法同样地工作,或者更好,而不是其非自适应对应物的迭代平均值。灵感来自这些结果\姓据{,一起与}对迭代解决方案多样性的适当正则化的重要性,我们提出了两个具有迭代平均的自适应算法。与随机梯度下降(SGD)相比,这些结果具有明显更好的结果,需要较少调谐并且不需要早期停止或验证设定监视。我们在各种现代和古典网络架构上展示了我们对CiFar-10/100,Imagenet和Penn TreeBank数据集的方法的疗效。
translated by 谷歌翻译
在这项工作中,我们探讨了随机梯度下降(SGD)训练的深神经网络的限制动态。如前所述,长时间的性能融合,网络继续通过参数空间通过一个异常扩散的过程,其中距离在具有非活动指数的梯度更新的数量中增加距离。我们揭示了优化的超公数,梯度噪声结构之间的复杂相互作用,以及在训练结束时解释这种异常扩散的Hessian矩阵。为了构建这种理解,我们首先为SGD推导出一个连续时间模型,具有有限的学习速率和批量尺寸,作为欠下的Langevin方程。我们在线性回归中研究了这个方程,我们可以为参数的相位空间动态和它们的瞬时速度来得出精确的分析表达式,从初始化到实用性。使用Fokker-Planck方程,我们表明驾驶这些动态的关键成分不是原始的训练损失,而是修改的损失的组合,其隐含地规则地规范速度和概率电流,这导致相位空间中的振荡。我们在ImageNet培训的Reset-18模型的动态中确定了这种理论的定性和定量预测。通过统计物理的镜头,我们揭示了SGD培训的深神经网络的异常限制动态的机制来源。
translated by 谷歌翻译
深度学习的概括分析通常假定训练会收敛到固定点。但是,最近的结果表明,实际上,用随机梯度下降优化的深神经网络的权重通常无限期振荡。为了减少理论和实践之间的这种差异,本文着重于神经网络的概括,其训练动力不一定会融合到固定点。我们的主要贡献是提出一个统计算法稳定性(SAS)的概念,该算法将经典算法稳定性扩展到非convergergent算法并研究其与泛化的联系。与传统的优化和学习理论观点相比,这种崇高的理论方法可导致新的见解。我们证明,学习算法的时间复杂行为的稳定性与其泛化有关,并在经验上证明了损失动力学如何为概括性能提供线索。我们的发现提供了证据表明,即使训练无限期继续并且权重也不会融合,即使训练持续进行训练,训练更好地概括”的网络也是如此。
translated by 谷歌翻译
We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases is completely non-sparse. It is derived by approximating various large blocks of the Fisher (corresponding to entire layers) as being the Kronecker product of two much smaller matrices. While only several times more expensive to compute than the plain stochastic gradient, the updates produced by K-FAC make much more progress optimizing the objective, which results in an algorithm that can be much faster than stochastic gradient descent with momentum in practice. And unlike some previously proposed approximate natural-gradient/Newton methods which use high-quality non-diagonal curvature matrices (such as Hessian-free optimization), K-FAC works very well in highly stochastic optimization regimes. This is because the cost of storing and inverting K-FAC's approximation to the curvature matrix does not depend on the amount of data used to estimate it, which is a feature typically associated only with diagonal or low-rank approximations to the curvature matrix.
translated by 谷歌翻译
Learning curves provide insight into the dependence of a learner's generalization performance on the training set size. This important tool can be used for model selection, to predict the effect of more training data, and to reduce the computational complexity of model training and hyperparameter tuning. This review recounts the origins of the term, provides a formal definition of the learning curve, and briefly covers basics such as its estimation. Our main contribution is a comprehensive overview of the literature regarding the shape of learning curves. We discuss empirical and theoretical evidence that supports well-behaved curves that often have the shape of a power law or an exponential. We consider the learning curves of Gaussian processes, the complex shapes they can display, and the factors influencing them. We draw specific attention to examples of learning curves that are ill-behaved, showing worse learning performance with more training data. To wrap up, we point out various open problems that warrant deeper empirical and theoretical investigation. All in all, our review underscores that learning curves are surprisingly diverse and no universal model can be identified.
translated by 谷歌翻译
尽管过度参数过多,但人们认为,通过随机梯度下降(SGD)训练的深度神经网络令人惊讶地概括了。基于预先指定的假设集的Rademacher复杂性,已经开发出不同的基于规范的泛化界限来解释这种现象。但是,最近的研究表明,这些界限可能会随着训练集的规模而增加,这与经验证据相反。在这项研究中,我们认为假设集SGD探索是轨迹依赖性的,因此可能在其Rademacher复杂性上提供更严格的结合。为此,我们通过假设发生的随机梯度噪声遵循分数的布朗运动,通过随机微分方程来表征SGD递归。然后,我们根据覆盖数字识别Rademacher的复杂性,并将其与优化轨迹的Hausdorff维度相关联。通过调用假设集稳定性,我们得出了针对深神经网络的新型概括。广泛的实验表明,它可以很好地预测几种常见的实验干预措施的概括差距。我们进一步表明,分数布朗运动的HURST参数比现有的概括指标(例如幂律指数和上blumenthal-getoor索引)更具信息性。
translated by 谷歌翻译
随机梯度算法在大规模学习和推理问题中广泛用于优化和采样。但是,实际上,调整这些算法通常是使用启发式和反复试验而不是严格的,可概括的理论来完成的。为了解决理论和实践之间的这一差距,我们通过表征具有固定步长的非常通用的预处理随机梯度算法的迭代术的大样本行为来对调整参数的效果进行新的见解。在优化设置中,我们的结果表明,具有较大固定步长的迭代平均值可能会导致(局部)M-静态器的统计效率近似。在抽样环境中,我们的结果表明,通过适当的调整参数选择,限制固定协方差可以与Bernstein匹配 - 后验的von Mises限制,对模型错误指定后验的调整或MLE的渐近分布;而幼稚的调整极限与这些都不相对应。此外,我们认为可以在数据集对固定数量的通行证后获得基本独立的样本。我们使用模拟和真实数据通过多个实验来验证渐近样结果。总体而言,我们证明具有恒定步长的正确调整的随机梯度算法为获得点估计或后部样品提供了计算上有效且统计上健壮的方法。
translated by 谷歌翻译
强大的机器学习模型的开发中的一个重要障碍是协变量的转变,当训练和测试集的输入分布时发生的分配换档形式在条件标签分布保持不变时发生。尽管现实世界应用的协变量转变普遍存在,但在现代机器学习背景下的理论理解仍然缺乏。在这项工作中,我们检查协变量的随机特征回归的精确高尺度渐近性,并在该设置中提出了限制测试误差,偏差和方差的精确表征。我们的结果激发了一种自然部分秩序,通过协变速转移,提供足够的条件来确定何时何时损害(甚至有助于)测试性能。我们发现,过度分辨率模型表现出增强的协会转变的鲁棒性,为这种有趣现象提供了第一个理论解释之一。此外,我们的分析揭示了分销和分发外概率性能之间的精确线性关系,为这一令人惊讶的近期实证观察提供了解释。
translated by 谷歌翻译
这项正在进行的工作旨在为统计学习提供统一的介绍,从诸如GMM和HMM等经典模型到现代神经网络(如VAE和扩散模型)缓慢地构建。如今,有许多互联网资源可以孤立地解释这一点或新的机器学习算法,但是它们并没有(也不能在如此简短的空间中)将这些算法彼此连接起来,或者与统计模型的经典文献相连现代算法出现了。同样明显缺乏的是一个单一的符号系统,尽管对那些已经熟悉材料的人(如这些帖子的作者)不满意,但对新手的入境造成了重大障碍。同样,我的目的是将各种模型(尽可能)吸收到一个用于推理和学习的框架上,表明(以及为什么)如何以最小的变化将一个模型更改为另一个模型(其中一些是新颖的,另一些是文献中的)。某些背景当然是必要的。我以为读者熟悉基本的多变量计算,概率和统计以及线性代数。这本书的目标当然不是​​完整性,而是从基本知识到过去十年中极强大的新模型的直线路径或多或少。然后,目标是补充而不是替换,诸如Bishop的\ emph {模式识别和机器学习}之类的综合文本,该文本现在已经15岁了。
translated by 谷歌翻译
二阶优化器被认为具有加快神经网络训练的潜力,但是由于曲率矩阵的尺寸巨大,它们通常需要近似值才能计算。最成功的近似家庭是Kronecker因块状曲率估计值(KFAC)。在这里,我们结合了先前工作的工具,以评估确切的二阶更新和仔细消融以建立令人惊讶的结果:由于其近似值,KFAC与二阶更新无关,尤其是,它极大地胜过真实的第二阶段更新。订单更新。这一挑战广泛地相信,并立即提出了为什么KFAC表现如此出色的问题。为了回答这个问题,我们提出了强烈的证据,表明KFAC近似于一阶算法,该算法在神经元上执行梯度下降而不是权重。最后,我们表明,这种优化器通常会在计算成本和数据效率方面改善KFAC。
translated by 谷歌翻译
The vast majority of successful deep neural networks are trained using variants of stochastic gradient descent (SGD) algorithms. Recent attempts to improve SGD can be broadly categorized into two approaches: (1) adaptive learning rate schemes, such as AdaGrad and Adam, and (2) accelerated schemes, such as heavy-ball and Nesterov momentum. In this paper, we propose a new optimization algorithm, Lookahead, that is orthogonal to these previous approaches and iteratively updates two sets of weights. Intuitively, the algorithm chooses a search direction by looking ahead at the sequence of "fast weights" generated by another optimizer. We show that Lookahead improves the learning stability and lowers the variance of its inner optimizer with negligible computation and memory cost. We empirically demonstrate Lookahead can significantly improve the performance of SGD and Adam, even with their default hyperparameter settings on ImageNet, CIFAR-10/100, neural machine translation, and Penn Treebank.
translated by 谷歌翻译
贝叶斯推理允许在贝叶斯神经网络的上下文中获取有关模型参数的有用信息,或者在贝叶斯神经网络的背景下。通常的Monte Carlo方法的计算成本,用于在贝叶斯推理中对贝叶斯推理的后验法律进行线性点的数量与数据点的数量进行线性。将其降低到这一成本的一小部分的一种选择是使用Langevin动态的未经调整的离散化来诉诸Mini-Batching,在这种情况下,只使用数据的随机分数来估计梯度。然而,这导致动态中的额外噪声,因此在马尔可夫链采样的不变度量上的偏差。我们倡导使用所谓的自适应Langevin动态,这是一种改进标准惯性Langevin动态,其动态摩擦力,可自动校正迷你批次引起的增加的噪声。我们调查假设适应性Langevin的假设(恒定协方差估计梯度的恒定协方差),这在贝叶斯推理的典型模型中不满足,并在这种情况下量化小型匹配诱导的偏差。我们还展示了如何扩展ADL,以便通过考虑根据参数的当前值来系统地减少后部分布的偏置。
translated by 谷歌翻译
In this thesis, we consider two simple but typical control problems and apply deep reinforcement learning to them, i.e., to cool and control a particle which is subject to continuous position measurement in a one-dimensional quadratic potential or in a quartic potential. We compare the performance of reinforcement learning control and conventional control strategies on the two problems, and show that the reinforcement learning achieves a performance comparable to the optimal control for the quadratic case, and outperforms conventional control strategies for the quartic case for which the optimal control strategy is unknown. To our knowledge, this is the first time deep reinforcement learning is applied to quantum control problems in continuous real space. Our research demonstrates that deep reinforcement learning can be used to control a stochastic quantum system in real space effectively as a measurement-feedback closed-loop controller, and our research also shows the ability of AI to discover new control strategies and properties of the quantum systems that are not well understood, and we can gain insights into these problems by learning from the AI, which opens up a new regime for scientific research.
translated by 谷歌翻译
本文评价用机器学习问题的数值优化方法。由于机器学习模型是高度参数化的,我们专注于适合高维优化的方法。我们在二次模型上构建直觉,以确定哪种方法适用于非凸优化,并在凸函数上开发用于这种方法的凸起函数。随着随机梯度下降和动量方法的这种理论基础,我们试图解释为什么机器学习领域通常使用的方法非常成功。除了解释成功的启发式之外,最后一章还提供了对更多理论方法的广泛审查,这在实践中并不像惯例。所以在某些情况下,这项工作试图回答这个问题:为什么默认值中包含的默认TensorFlow优化器?
translated by 谷歌翻译
深神经网络(DNN)是用于压缩和蒸馏信息的强大工具。由于它们的规模和复杂性,通常涉及数十亿间相互作用的内部自由度,精确分析方法通常会缩短。这种情况下的共同策略是识别平均潜在的快速微观变量的不稳定行为的缓慢自由度。在这里,我们在训练结束时识别在过度参数化的深卷积神经网络(CNNS)中发生的尺度的分离。它意味着神经元预激活与几乎高斯的方式与确定性潜在内核一起波动。在对于具有无限许多频道的CNN来说,这些内核是惰性的,对于有限的CNNS,它们以分析的方式通过数据适应和学习数据。由此产生的深度学习的热力学理论产生了几种深度非线性CNN玩具模型的准确预测。此外,它还提供了新的分析和理解CNN的方法。
translated by 谷歌翻译
We propose SWA-Gaussian (SWAG), a simple, scalable, and general purpose approach for uncertainty representation and calibration in deep learning. Stochastic Weight Averaging (SWA), which computes the first moment of stochastic gradient descent (SGD) iterates with a modified learning rate schedule, has recently been shown to improve generalization in deep learning. With SWAG, we fit a Gaussian using the SWA solution as the first moment and a low rank plus diagonal covariance also derived from the SGD iterates, forming an approximate posterior distribution over neural network weights; we then sample from this Gaussian distribution to perform Bayesian model averaging. We empirically find that SWAG approximates the shape of the true posterior, in accordance with results describing the stationary distribution of SGD iterates. Moreover, we demonstrate that SWAG performs well on a wide variety of tasks, including out of sample detection, calibration, and transfer learning, in comparison to many popular alternatives including MC dropout, KFAC Laplace, SGLD, and temperature scaling.
translated by 谷歌翻译
Classical asymptotic theory for statistical inference usually involves calibrating a statistic by fixing the dimension $d$ while letting the sample size $n$ increase to infinity. Recently, much effort has been dedicated towards understanding how these methods behave in high-dimensional settings, where $d$ and $n$ both increase to infinity together. This often leads to different inference procedures, depending on the assumptions about the dimensionality, leaving the practitioner in a bind: given a dataset with 100 samples in 20 dimensions, should they calibrate by assuming $n \gg d$, or $d/n \approx 0.2$? This paper considers the goal of dimension-agnostic inference; developing methods whose validity does not depend on any assumption on $d$ versus $n$. We introduce an approach that uses variational representations of existing test statistics along with sample splitting and self-normalization to produce a new test statistic with a Gaussian limiting distribution, regardless of how $d$ scales with $n$. The resulting statistic can be viewed as a careful modification of degenerate U-statistics, dropping diagonal blocks and retaining off-diagonal blocks. We exemplify our technique for some classical problems including one-sample mean and covariance testing, and show that our tests have minimax rate-optimal power against appropriate local alternatives. In most settings, our cross U-statistic matches the high-dimensional power of the corresponding (degenerate) U-statistic up to a $\sqrt{2}$ factor.
translated by 谷歌翻译