In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these perparameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-row and percolumn sums of these moving averages, and estimating the per-parameter second moments based on these sums. We demonstrate empirically that this method produces similar results to the baseline. Secondly, we show that adaptive methods can produce larger-than-desired updates when the decay rate of the second moment accumulator is too slow. We propose update clipping and a gradually increasing decay rate scheme as remedies. Combining these methods and dropping momentum, we achieve comparable results to the published Adam regime in training the Transformer model on the WMT 2014 English-German machine translation task, while using very little auxiliary storage in the optimizer. Finally, we propose scaling the parameter updates based on the scale of the parameters themselves.

深度学习在广泛的AI应用方面取得了有希望的结果。较大的数据集和模型一致地产生更好的性能。但是，我们一般花费更长的培训时间，以更多的计算和沟通。在本调查中，我们的目标是在模型精度和模型效率方面提供关于大规模深度学习优化的清晰草图。我们调查最常用于优化的算法，详细阐述了大批量培训中出现的泛化差距的可辩论主题，并审查了解决通信开销并减少内存足迹的SOTA策略。

We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm. * Equal contribution. Author ordering determined by coin flip over a Google Hangout.

Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSPROP, ADAM, ADADELTA, NADAM are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings). We show that one cause for such failures is the exponential moving average used in the algorithms. We provide an explicit example of a simple convex optimization setting where ADAM does not converge to the optimal solution, and describe the precise problems with the previous analysis of ADAM algorithm. Our analysis suggests that the convergence issues can be fixed by endowing such algorithms with "long-term memory" of past gradients, and propose new variants of the ADAM algorithm which not only fix the convergence issues but often also lead to improved empirical performance.

与SGD相比，Adam等自适应梯度方法允许对现代深层网络（尤其是大型语言模型）进行强有力的培训。但是，适应性的使用不仅是为了额外的记忆，而且还提出了一个基本问题：SGD等非自适应方法可以享受类似的好处吗？在本文中，我们通过提议通过以下一般配方提议实现健壮和记忆效率的培训来为这个问题提供肯定的答案：（1）修改体系结构并使IT规模不变，即参数规模不影响。网络的输出，（2）使用SGD和重量衰减的训练，以及（3）剪辑全局梯度标准与重量标准成比例成正比，乘以$ \ sqrt {\ tfrac {\ tfrac {2 \ lambda} {\ eta}} {\ eta}}} $， $ \ eta $是学习率，而$ \ lambda $是权重腐烂。我们表明，这种一般方法是通过证明其收敛性仅取决于初始化和损失的规模来重新恢复参数和丢失的强大，而标准SGD甚至可能不会收敛许多初始化。在我们的食谱之后，我们设计了一个名为Sibert的Bert版本的比例不变版本，该版本仅由Vanilla SGD进行训练时，可以实现与Bert在下游任务中受过自适应方法训练的BERT相当的性能。

We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases is completely non-sparse. It is derived by approximating various large blocks of the Fisher (corresponding to entire layers) as being the Kronecker product of two much smaller matrices. While only several times more expensive to compute than the plain stochastic gradient, the updates produced by K-FAC make much more progress optimizing the objective, which results in an algorithm that can be much faster than stochastic gradient descent with momentum in practice. And unlike some previously proposed approximate natural-gradient/Newton methods which use high-quality non-diagonal curvature matrices (such as Hessian-free optimization), K-FAC works very well in highly stochastic optimization regimes. This is because the cost of storing and inverting K-FAC's approximation to the curvature matrix does not depend on the amount of data used to estimate it, which is a feature typically associated only with diagonal or low-rank approximations to the curvature matrix.

关于自适应梯度方法等自适应梯度方法等训练动力的知之甚少。在本文中，我们阐明了这些算法在全批处理和足够大的批处理设置中的行为。具体而言，我们从经验上证明，在全批训练中，预处理的Hessian的最大特征值通常在某个数值下平衡 - 梯度下降算法的稳定性阈值。对于带有步长$ \ eta $和$ \ beta_1 = 0.9 $的Adam，此稳定性阈值为$ 38/\ eta $。在Minibatch培训期间发生了类似的影响，尤其是随着批处理大小的增长。然而，即使自适应方法在``稳定性的自适应边缘''（AEOS）上训练，但它们在该制度中的行为与EOS的非自适应方法的行为有很大不同。 EOS处的非自适应算法被阻止进入损失景观的高曲率区域，而AEOS的自适应梯度方法可以继续前进到高外观区域，同时适应预先调节器以补偿。我们的发现可以成为社区对深度学习中适应性梯度方法的未来理解的基础。

The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate -its variance is problematically large in the early stage, and presume warmup works as a variance reduction technique. We provide both empirical and theoretical evidence to verify our hypothesis. We further propose Rectified Adam (RAdam), a novel variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate. Experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the efficacy and robustness of RAdam. 1 * Work was done during an internship at Microsoft Dynamics 365 AI. † Work was done during an internship at Microsoft Dynamics 365 AI.

非凸优化的传统分析通常取决于平滑度的假设，即要求梯度为Lipschitz。但是，最近的证据表明，这种平滑度条件并未捕获一些深度学习目标功能的特性，包括涉及复发性神经网络和LSTM的函数。取而代之的是，他们满足了更轻松的状况，并具有潜在的无界光滑度。在这个轻松的假设下，从理论和经验上表明，倾斜的SGD比香草具有优势。在本文中，我们表明，在解决此类情况时，剪辑对于ADAM型算法是不可或缺的：从理论上讲，我们证明了广义标志GD算法可以获得与带有剪辑的SGD相似的收敛速率，但根本不需要显式剪辑。一端的这个算法家族恢复了符号，另一端与受欢迎的亚当算法非常相似。我们的分析强调了动量在分析符号类型和ADAM型算法中发挥作用的关键作用：它不仅降低了噪声的影响，因此在先前的符号分析中消除了大型迷你批次的需求显着降低了无界平滑度和梯度规范的影响。我们还将这些算法与流行的优化器进行了比较，在一组深度学习任务上，观察到我们可以在击败其他人的同时匹配亚当的性能。

The vast majority of successful deep neural networks are trained using variants of stochastic gradient descent (SGD) algorithms. Recent attempts to improve SGD can be broadly categorized into two approaches: (1) adaptive learning rate schemes, such as AdaGrad and Adam, and (2) accelerated schemes, such as heavy-ball and Nesterov momentum. In this paper, we propose a new optimization algorithm, Lookahead, that is orthogonal to these previous approaches and iteratively updates two sets of weights. Intuitively, the algorithm chooses a search direction by looking ahead at the sequence of "fast weights" generated by another optimizer. We show that Lookahead improves the learning stability and lowers the variance of its inner optimizer with negligible computation and memory cost. We empirically demonstrate Lookahead can significantly improve the performance of SGD and Adam, even with their default hyperparameter settings on ImageNet, CIFAR-10/100, neural machine translation, and Penn Treebank.

引入了归一化层（例如，批处理归一化，层归一化），以帮助在非常深的网中获得优化困难，但它们显然也有助于概括，即使在不太深入的网中也是如此。由于长期以来的信念，即最小的最小值导致更好的概括，本文提供了数学分析和支持实验，这表明归一化（与伴随的重量赛一起）鼓励GD降低损失表面的清晰度。鉴于损失是标准不变的，这是标准化的已知结果，因此仔细地定义了“清晰度”。具体而言，对于具有归一化的相当广泛的神经网类，我们的理论解释了有限学习率的GD如何进入所谓的稳定边缘（EOS）制度，并通过连续的清晰度来表征GD的轨迹 - 还原流。

我们提出ACPROP（异步 - 居中 - PROP），一个适应优化器，它结合了第二次动量和异步更新的居中（例如，用于$ T $ -Th更新，分母使用信息最多为步骤$ T-1 $，而Dumerator使用梯度$ t-the step）。 ACPROP具有强大的理论特性和经验性能。用reddi等人的例子。 （2018），我们表明异步优化器（例如Adashift，ACProp）的收敛条件较弱，而不是同步优化器（例如ADAM，RMSPROP，Adabelief）;在异步优化器中，我们表明，第二次势头的中心进一步削弱了收敛条件。我们展示了随机非凸面的$ O（\ FRAC {1} {\ SQRT {}）$的收敛速度，它与ORACLE率和优于$ O（\ FRAC {logt}相匹配{\ sqrt {t}}）$ rmsprop和adam的$率。我们在广泛的实证研究中验证了ACPROP：ACPRAC在使用CNN的图像分类中表现出SGD和其他自适应优化器，并且在各种GAN模型，加固学习和变压器的培训中优于良好调整的自适应优化器。总而言之，ACPROP具有良好的理论特性，包括弱收敛条件和最佳收敛速度，以及强的经验性能，包括SGD等良好普遍性，如亚当等训练稳定性。我们在https://github.com/juntang-zhuang/acprop-optimizer提供实现。

亚当是训练深神经网络的最具影响力的自适应随机算法之一，即使在简单的凸面设置中，它也被指出是不同的。许多尝试，例如降低自适应学习率，采用较大的批量大小，结合了时间去相关技术，寻求类似的替代物，\ textit {etc。}，以促进Adam-type算法融合。与现有方法相反，我们引入了另一种易于检查的替代条件，这仅取决于基础学习率的参数和历史二阶时刻的组合，以确保通用ADAM的全球融合以解决大型融合。缩放非凸随机优化。这种观察结果以及这种足够的条件，对亚当的差异产生了更深刻的解释。另一方面，在实践中，无需任何理论保证，广泛使用了迷你ADAM和分布式ADAM。我们进一步分析了分布式系统中的批次大小或节点的数量如何影响亚当的收敛性，从理论上讲，这表明迷你批次和分布式亚当可以通过使用较大的迷你批量或较大的大小来线性地加速节点的数量。最后，我们应用了通用的Adam和Mini Batch Adam，具有足够条件来求解反例并在各种真实世界数据集上训练多个神经网络。实验结果完全符合我们的理论分析。

随机梯度体面（SGD）是深神经网络成功背后的核心技术之一。梯度提供有关功能具有最陡变化率的方向的信息。基本SGD的主要问题是通过梯度行为而对所有参数的相等大小的步骤进行更改。因此，深度网络优化的有效方式是为每个参数进行自适应步骤尺寸。最近，已经进行了几次尝试，以改善梯度下降方法，例如Adagrad，Adadelta，RMSProp和Adam。这些方法依赖于平方过去梯度的指数移动平均线的平方根。因此，这些方法不利用梯度的局部变化。在本文中，基于当前和立即梯度（即，差异）之间的差异提出了一种新颖的优化器。在所提出的差异优化技术中，以这样的方式调整步长，使得它应该具有更大的梯度改变参数的较大步长，以及用于较低梯度改变参数的较低步长。收敛分析是使用在线学习框架的遗憾方法完成。在本文中进行严格的分析超过三种合成复合的非凸功能。图像分类实验也在CiFar10和CiFAR100数据集上进行，以观察漫反射的性能，相对于最先进的优化器，例如SGDM，Adagrad，Adadelta，RMSProp，Amsgrad和Adam。基于基于单元（Reset）的基于卷积神经网络（CNN）架构用于实验中。实验表明，Diffgrad优于其他优化器。此外，我们表明差异对使用不同的激活功能训练CNN的均匀良好。源代码在https://github.com/shivram1987/diffgrad公开使用。

我们使用高斯过程扰动模型在高维二次上的真实和批量风险表面之间的高斯过程扰动模型分析和解释迭代平均的泛化性能。我们从我们的理论结果中获得了三个现象\姓名：}（1）将迭代平均值（ia）与大型学习率和正则化进行了改进的正规化的重要性。 （2）对较少频繁平均的理由。 （3）我们预计自适应梯度方法同样地工作，或者更好，而不是其非自适应对应物的迭代平均值。灵感来自这些结果\姓据{，一起与}对迭代解决方案多样性的适当正则化的重要性，我们提出了两个具有迭代平均的自适应算法。与随机梯度下降（SGD）相比，这些结果具有明显更好的结果，需要较少调谐并且不需要早期停止或验证设定监视。我们在各种现代和古典网络架构上展示了我们对CiFar-10/100，Imagenet和Penn TreeBank数据集的方法的疗效。

Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low-rank matrix approximation. These techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets.This paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions. These methods use random sampling to identify a subspace that captures most of the action of a matrix. The input matrix is then compressed-either explicitly or implicitly-to this subspace, and the reduced matrix is manipulated deterministically to obtain the desired low-rank factorization. In many cases, this approach beats its classical competitors in terms of accuracy, speed, and robustness. These claims are supported by extensive numerical experiments and a detailed error analysis.The specific benefits of randomized techniques depend on the computational environment. Consider the model problem of finding the k dominant components of the singular value decomposition of an m × n matrix. (i) For a dense input matrix, randomized algorithms require O(mn log(k)) floating-point operations (flops) in contrast with O(mnk) for classical algorithms. (ii) For a sparse input matrix, the flop count matches classical Krylov subspace methods, but the randomized approach is more robust and can easily be reorganized to exploit multi-processor architectures. (iii) For a matrix that is too large to fit in fast memory, the randomized techniques require only a constant number of passes over the data, as opposed to O(k) passes for classical algorithms. In fact, it is sometimes possible to perform matrix approximation with a single pass over the data.

本文评价用机器学习问题的数值优化方法。由于机器学习模型是高度参数化的，我们专注于适合高维优化的方法。我们在二次模型上构建直觉，以确定哪种方法适用于非凸优化，并在凸函数上开发用于这种方法的凸起函数。随着随机梯度下降和动量方法的这种理论基础，我们试图解释为什么机器学习领域通常使用的方法非常成功。除了解释成功的启发式之外，最后一章还提供了对更多理论方法的广泛审查，这在实践中并不像惯例。所以在某些情况下，这项工作试图回答这个问题：为什么默认值中包含的默认TensorFlow优化器？

Deep Learning optimization involves minimizing a high-dimensional loss function in the weight space which is often perceived as difficult due to its inherent difficulties such as saddle points, local minima, ill-conditioning of the Hessian and limited compute resources. In this paper, we provide a comprehensive review of 12 standard optimization methods successfully used in deep learning research and a theoretical assessment of the difficulties in numerical optimization from the optimization literature.

神经架构的创新促进了语言建模和计算机视觉中的重大突破。不幸的是，如果网络参数未正确初始化，新颖的架构通常会导致挑战超参数选择和培训不稳定。已经提出了许多架构特定的初始化方案，但这些方案并不总是可移植到新体系结构。本文介绍了毕业，一种用于初始化神经网络的自动化和架构不可知论由方法。毕业基础是一个简单的启发式;调整每个网络层的规范，使得具有规定的超参数的SGD或ADAM的单个步骤导致可能的损耗值最小。通过在每个参数块前面引入标量乘数变量，然后使用简单的数字方案优化这些变量来完成此调整。 GradInit加速了许多卷积架构的收敛性和测试性能，无论是否有跳过连接，甚至没有归一化层。它还提高了机器翻译的原始变压器架构的稳定性，使得在广泛的学习速率和动量系数下使用ADAM或SGD来训练它而无需学习速率预热。代码可在https://github.com/zhuchen03/gradinit上获得。

近期在应用于培训深度神经网络和数据分析中的其他优化问题中的非凸优化的优化算法的兴趣增加，我们概述了最近对非凸优化优化算法的全球性能保证的理论结果。我们从古典参数开始，显示一般非凸面问题无法在合理的时间内有效地解决。然后，我们提供了一个问题列表，可以通过利用问题的结构来有效地找到全球最小化器，因为可能的问题。处理非凸性的另一种方法是放宽目标，从找到全局最小，以找到静止点或局部最小值。对于该设置，我们首先为确定性一阶方法的收敛速率提出了已知结果，然后是最佳随机和随机梯度方案的一般理论分析，以及随机第一阶方法的概述。之后，我们讨论了非常一般的非凸面问题，例如最小化$ \ alpha $ -weakly-are-convex功能和满足Polyak-lojasiewicz条件的功能，这仍然允许获得一阶的理论融合保证方法。然后，我们考虑更高阶和零序/衍生物的方法及其收敛速率，以获得非凸优化问题。