Deep and recurrent neural networks (DNNs and RNNs respectively) are powerful models that were considered to be almost impossible to train using stochastic gradient descent with momentum. In this paper, we show that when stochastic gradient descent with momentum uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter, it can train both DNNs and RNNs (on datasets with long-term dependencies) to levels of performance that were previously achievable only with Hessian-Free optimization. We find that both the initialization and the momentum are crucial since poorly initialized networks cannot be trained with momentum and well-initialized networks perform markedly worse when the momentum is absent or poorly tuned.Our success training these models suggests that previous attempts to train deep and recurrent neural networks from random initializations have likely failed due to poor initialization schemes. Furthermore, carefully tuned momentum methods su ce for dealing with the curvature issues in deep and recurrent network training objectives without the need for sophisticated second-order methods.

Deep Learning optimization involves minimizing a high-dimensional loss function in the weight space which is often perceived as difficult due to its inherent difficulties such as saddle points, local minima, ill-conditioning of the Hessian and limited compute resources. In this paper, we provide a comprehensive review of 12 standard optimization methods successfully used in deep learning research and a theoretical assessment of the difficulties in numerical optimization from the optimization literature.

We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases is completely non-sparse. It is derived by approximating various large blocks of the Fisher (corresponding to entire layers) as being the Kronecker product of two much smaller matrices. While only several times more expensive to compute than the plain stochastic gradient, the updates produced by K-FAC make much more progress optimizing the objective, which results in an algorithm that can be much faster than stochastic gradient descent with momentum in practice. And unlike some previously proposed approximate natural-gradient/Newton methods which use high-quality non-diagonal curvature matrices (such as Hessian-free optimization), K-FAC works very well in highly stochastic optimization regimes. This is because the cost of storing and inverting K-FAC's approximation to the curvature matrix does not depend on the amount of data used to estimate it, which is a feature typically associated only with diagonal or low-rank approximations to the curvature matrix.

深度学习在广泛的AI应用方面取得了有希望的结果。较大的数据集和模型一致地产生更好的性能。但是，我们一般花费更长的培训时间，以更多的计算和沟通。在本调查中，我们的目标是在模型精度和模型效率方面提供关于大规模深度学习优化的清晰草图。我们调查最常用于优化的算法，详细阐述了大批量培训中出现的泛化差距的可辩论主题，并审查了解决通信开销并减少内存足迹的SOTA策略。

本文评价用机器学习问题的数值优化方法。由于机器学习模型是高度参数化的，我们专注于适合高维优化的方法。我们在二次模型上构建直觉，以确定哪种方法适用于非凸优化，并在凸函数上开发用于这种方法的凸起函数。随着随机梯度下降和动量方法的这种理论基础，我们试图解释为什么机器学习领域通常使用的方法非常成功。除了解释成功的启发式之外，最后一章还提供了对更多理论方法的广泛审查，这在实践中并不像惯例。所以在某些情况下，这项工作试图回答这个问题：为什么默认值中包含的默认TensorFlow优化器？

近期在应用于培训深度神经网络和数据分析中的其他优化问题中的非凸优化的优化算法的兴趣增加，我们概述了最近对非凸优化优化算法的全球性能保证的理论结果。我们从古典参数开始，显示一般非凸面问题无法在合理的时间内有效地解决。然后，我们提供了一个问题列表，可以通过利用问题的结构来有效地找到全球最小化器，因为可能的问题。处理非凸性的另一种方法是放宽目标，从找到全局最小，以找到静止点或局部最小值。对于该设置，我们首先为确定性一阶方法的收敛速率提出了已知结果，然后是最佳随机和随机梯度方案的一般理论分析，以及随机第一阶方法的概述。之后，我们讨论了非常一般的非凸面问题，例如最小化$ \ alpha $ -weakly-are-convex功能和满足Polyak-lojasiewicz条件的功能，这仍然允许获得一阶的理论融合保证方法。然后，我们考虑更高阶和零序/衍生物的方法及其收敛速率，以获得非凸优化问题。

我们介绍了SubGD，这是一种新颖的几声学习方法，基于最近的发现，即随机梯度下降更新往往生活在低维参数子空间中。在实验和理论分析中，我们表明模型局限于合适的预定义子空间，可以很好地推广用于几次学习。合适的子空间符合给定任务的三个标准：IT（a）允许通过梯度流量减少训练误差，（b）导致模型良好的模型，并且（c）可以通过随机梯度下降来识别。 SUBGD从不同任务的更新说明的自动相关矩阵的特征组合中标识了这些子空间。明确的是，我们可以识别出低维合适的子空间，用于对动态系统的几次学习，而动态系统具有不同的属性，这些属性由分析系统描述的一个或几个参数描述。这种系统在科学和工程领域的现实应用程序中无处不在。我们在实验中证实了SubGD在三个不同的动态系统问题设置上的优势，在样本效率和性能方面，均超过了流行的几次学习方法。

The vast majority of successful deep neural networks are trained using variants of stochastic gradient descent (SGD) algorithms. Recent attempts to improve SGD can be broadly categorized into two approaches: (1) adaptive learning rate schemes, such as AdaGrad and Adam, and (2) accelerated schemes, such as heavy-ball and Nesterov momentum. In this paper, we propose a new optimization algorithm, Lookahead, that is orthogonal to these previous approaches and iteratively updates two sets of weights. Intuitively, the algorithm chooses a search direction by looking ahead at the sequence of "fast weights" generated by another optimizer. We show that Lookahead improves the learning stability and lowers the variance of its inner optimizer with negligible computation and memory cost. We empirically demonstrate Lookahead can significantly improve the performance of SGD and Adam, even with their default hyperparameter settings on ImageNet, CIFAR-10/100, neural machine translation, and Penn Treebank.

我们使用高斯过程扰动模型在高维二次上的真实和批量风险表面之间的高斯过程扰动模型分析和解释迭代平均的泛化性能。我们从我们的理论结果中获得了三个现象\姓名：}（1）将迭代平均值（ia）与大型学习率和正则化进行了改进的正规化的重要性。 （2）对较少频繁平均的理由。 （3）我们预计自适应梯度方法同样地工作，或者更好，而不是其非自适应对应物的迭代平均值。灵感来自这些结果\姓据{，一起与}对迭代解决方案多样性的适当正则化的重要性，我们提出了两个具有迭代平均的自适应算法。与随机梯度下降（SGD）相比，这些结果具有明显更好的结果，需要较少调谐并且不需要早期停止或验证设定监视。我们在各种现代和古典网络架构上展示了我们对CiFar-10/100，Imagenet和Penn TreeBank数据集的方法的疗效。

我们研究了使用尖刺，现场依赖的随机矩阵理论研究迷你批次对深神经网络损失景观的影响。我们表明，批量黑森州的极值值的大小大于经验丰富的黑森州。我们还获得了类似的结果对Hessian的概括高斯牛顿矩阵近似。由于我们的定理，我们推导出作为批量大小的最大学习速率的分析表达式，为随机梯度下降（线性缩放）和自适应算法（例如ADAM（Square Root Scaling）提供了通知实际培训方案，例如光滑，非凸深神经网络。虽然随机梯度下降的线性缩放是在我们概括的更多限制性条件下导出的，但是适应优化者的平方根缩放规则是我们的知识，完全小说。随机二阶方法和自适应方法的百分比，我们得出了最小阻尼系数与学习率与批量尺寸的比率成比例。我们在Cifar-$ 100 $和ImageNet数据集上验证了我们的VGG / WimerEsnet架构上的索赔。根据我们对象检的调查，我们基于飞行学习率和动量学习者开发了一个随机兰齐齐竞争，这避免了对这些关键的超参数进行昂贵的多重评估的需求，并在预残留的情况下显示出良好的初步结果Cifar的architecure - $ 100 $。

A central challenge to many fields of science and engineering involves minimizing non-convex error functions over continuous, high dimensional spaces. Gradient descent or quasi-Newton methods are almost ubiquitously used to perform such minimizations, and it is often thought that a main source of difficulty for these local methods to find the global minimum is the proliferation of local minima with much higher error than the global minimum. Here we argue, based on results from statistical physics, random matrix theory, neural network theory, and empirical evidence, that a deeper and more profound difficulty originates from the proliferation of saddle points, not local minima, especially in high dimensional problems of practical interest. Such saddle points are surrounded by high error plateaus that can dramatically slow down learning, and give the illusory impression of the existence of a local minimum. Motivated by these arguments, we propose a new approach to second-order optimization, the saddle-free Newton method, that can rapidly escape high dimensional saddle points, unlike gradient descent and quasi-Newton methods. We apply this algorithm to deep or recurrent neural network training, and provide numerical evidence for its superior optimization performance. This work extends the results of .

We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm. * Equal contribution. Author ordering determined by coin flip over a Google Hangout.

学习的优化器是可以训练解决优化问题的算法。与使用从理论原则派生的简单更新规则的基线优化器（例如势头或亚当）相比，学习的优化器使用灵活，高维，非线性参数化。虽然这可能导致某些设置中的更好性能，但他们的内部工作仍然是一个谜。学习优化器如何优于一个良好的调整基线？它是否学习了现有优化技术的复杂组合，或者是实现全新的行为吗？在这项工作中，我们通过仔细分析和可视化的学习优化器来解决这些问题。我们研究了从三个不同的任务中从头开始培训的优化器，并发现他们已经了解了可解释的机制，包括：势头，渐变剪辑，学习率计划以及新形式的学习率适应形式。此外，我们展示了学习优化器的动态如何实现这些行为。我们的结果帮助阐明了对学习优化器的工作原理的先前密切了解，并建立了解释未来学习优化器的工具。

Multilayer Neural Networks trained with the backpropagation algorithm constitute the best example of a successful Gradient-Based Learning technique. Given an appropriate network architecture, Gradient-Based Learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional Neural Networks, that are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques.Real-life document recognition systems are composed of multiple modules including eld extraction, segmentation, recognition, and language modeling. A new learning paradigm, called Graph Transformer Networks (GTN), allows such multi-module systems to be trained globally using Gradient-Based methods so as to minimize an overall performance measure.Two systems for on-line handwriting recognition are described. Experiments demonstrate the advantage of global training, and the exibility of Graph Transformer Networks.A Graph Transformer Network for reading bank check is also described. It uses Convolutional Neural Network character recognizers combined with global training techniques to provides record accuracy on business and personal checks. It is deployed commercially and reads several million checks per day.

Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We

我们引入了一种降低尺寸的二阶方法（DRSOM），用于凸和非凸的不受约束优化。在类似信任区域的框架下，我们的方法保留了二阶方法的收敛性，同时仅在两个方向上使用Hessian-Vector产品。此外，计算开销仍然与一阶相当，例如梯度下降方法。我们证明该方法的复杂性为$ O（\ epsilon^{ - 3/2}）$，以满足子空间中的一阶和二阶条件。DRSOM的适用性和性能通过逻辑回归，$ L_2-L_P $最小化，传感器网络定位和神经网络培训的各种计算实验展示。对于神经网络，我们的初步实施似乎在训练准确性和迭代复杂性方面与包括SGD和ADAM在内的最先进的一阶方法获得了计算优势。

简单的随机动量方法被广泛用于机器学习优化，但它们的良好实践表现与文献中没有理论保证的理论保证相矛盾。在这项工作中，我们的目标是通过表明随机重球动量来弥合理论和实践之间的差距，该动力可以解释为具有动量的随机kaczmarz算法，保留了二次优化问题（确定性）重球动量的快速线性速率，至少在使用足够大的批次大小的小型匹配时。该分析依赖于仔细分解动量过渡矩阵，并使用新的光谱范围浓度界限来进行独立随机矩阵的产物。我们提供数值实验，以证明我们的边界相当锐利。

最近，在学习没有更换SGD的收敛率的情况下，有很多兴趣，并证明它在最坏情况下比更换SGD更快。然而，已知的下限忽略了问题的几何形状，包括其条件号，而上限明确取决于它。也许令人惊讶的是，我们证明，当考虑条件号时，没有替换SGD \ EMPH {没有}在最坏情况下，除非是时期的数量（通过数据来说）大于条件号。由于机器学习和其他领域的许多问题都没有条件并涉及大型数据集，这表明没有替换不一定改善用于现实迭代预算的更换采样。我们通过提供具有紧密（最多日志因子）的新下限和上限来展示这一点，用于致通二次术语的二次问题，精确地量化了对问题参数的依赖性。

已知生物制剂在他们的生活过程中学习许多不同的任务，并且能够重新审视以前的任务和行为，而没有表现不损失。相比之下，人工代理容易出于“灾难性遗忘”，在以前任务上的性能随着所获取的新的任务而恶化。最近使用该方法通过鼓励参数保持接近以前任务的方法来解决此缺点。这可以通过（i）使用特定的参数正常数来完成，该参数正常数是在参数空间中映射合适的目的地，或（ii）通过将渐变投影到不会干扰先前任务的子空间来指导优化旅程。然而，这些方法通常在前馈和经常性神经网络中表现出子分子表现，并且经常性网络对支持生物持续学习的神经动力学研究感兴趣。在这项工作中，我们提出了自然的持续学习（NCL），一种统一重量正则化和预测梯度下降的新方法。 NCL使用贝叶斯重量正常化来鼓励在收敛的所有任务上进行良好的性能，并将其与梯度投影结合使用先前的精度，这可以防止在优化期间陷入灾难性遗忘。当应用于前馈和经常性网络中的连续学习问题时，我们的方法占据了标准重量正则化技术和投影的方法。最后，训练有素的网络演变了特定于任务特定的动态，这些动态被认为是学习的新任务，类似于生物电路中的实验结果。

近年来，在诸如denoing，压缩感应，介入和超分辨率等反问题中使用深度学习方法的使用取得了重大进展。尽管这种作品主要是由实践算法和实验驱动的，但它也引起了各种有趣的理论问题。在本文中，我们调查了这一作品中一些突出的理论发展，尤其是生成先验，未经训练的神经网络先验和展开算法。除了总结这些主题中的现有结果外，我们还强调了一些持续的挑战和开放问题。