We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results on the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the proposed approach to train modern network architectures with millions of weights and millions of hyperparameters. We learn a data-augmentation networkwhere every weight is a hyperparameter tuned for validation performance-that outputs augmented training examples; we learn a distilled dataset where each feature in each datapoint is a hyperparameter; and we tune millions of regularization hyperparameters. Jointly tuning weights and hyperparameters with our approach is only a few times more costly in memory and compute than standard training.• We scale IFT-based hyperparameter optimization to modern, large neural architectures, including AlexNet and LSTM-based language models.• We demonstrate several uses for fitting hyperparameters almost as easily as weights, including perparameter regularization, data distillation, and learned-from-scratch data augmentation methods.• We explore how training-validation splits should change when tuning many hyperparameters.

Many problems in machine learning involve bilevel optimization (BLO), including hyperparameter optimization, meta-learning, and dataset distillation. Bilevel problems consist of two nested sub-problems, called the outer and inner problems, respectively. In practice, often at least one of these sub-problems is overparameterized. In this case, there are many ways to choose among optima that achieve equivalent objective values. Inspired by recent studies of the implicit bias induced by optimization algorithms in single-level optimization, we investigate the implicit bias of gradient-based algorithms for bilevel optimization. We delineate two standard BLO methods -- cold-start and warm-start -- and show that the converged solution or long-run behavior depends to a large degree on these and other algorithmic choices, such as the hypergradient approximation. We also show that the inner solutions obtained by warm-start BLO can encode a surprising amount of information about the outer objective, even when the outer parameters are low-dimensional. We believe that implicit bias deserves as central a role in the study of bilevel optimization as it has attained in the study of single-level neural net optimization.

通过强制了解输入中某些转换保留输出的知识，通常应用数据增强来提高深度学习的性能。当前，使用的数据扩大是通过人类的努力和昂贵的交叉验证来选择的，这使得应用于新数据集很麻烦。我们开发了一种基于梯度的方便方法，用于在没有验证数据的情况下和在深度神经网络的培训期间选择数据增强。我们的方法依赖于措辞增强作为先前分布的不变性，并使用贝叶斯模型选择学习，该模型已被证明在高斯过程中起作用，但尚未用于深神经网络。我们提出了一个可区分的Kronecker因拉普拉斯（Laplace）近似与边际可能性的近似，作为我们的目标，可以在没有人类监督或验证数据的情况下优化。我们表明，我们的方法可以成功地恢复数据中存在的不断增长，这提高了图像数据集的概括和数据效率。

预训练（PT），然后进行微调（FT）是培训神经网络的有效方法，并导致许多域中的显着性能改进。 PT可以包含各种设计选择，如任务和数据重新免除策略，增强政策和噪声模型，所有这些都可以显着影响所学到的陈述的质量。因此，必须适当地调整这些策略引入的超级参数。但是，设置这些超参数的值是具有挑战性的。大多数现有方法都努力缩放到高维度，太慢和内存密集，或者不能直接应用于两级PT和FT学习过程。在这项工作中，我们提出了一种基于渐变的梯度的算法，以Meta-Learn PT HyperParameters。我们将PT HyperParameter优化问题正式化，并提出了一种通过展开优化结合隐式分化和反向来获得PT超级参数梯度的新方法。我们展示了我们的方法可以提高两个真实域的预测性能。首先，我们优化高维任务加权超参数，用于多任务对蛋白质 - 蛋白质相互作用图进行培训，并将Auroc提高至3.9％。其次，我们在心电图数据上优化用于SIMCLR的SIMCLR的数据增强神经网络，并将Auroc提高到1.9％。

A core capability of intelligent systems is the ability to quickly learn new tasks by drawing on prior experience. Gradient (or optimization) based meta-learning has recently emerged as an effective approach for few-shot learning. In this formulation, meta-parameters are learned in the outer loop, while task-specific models are learned in the inner-loop, by using only a small amount of data from the current task. A key challenge in scaling these approaches is the need to differentiate through the inner loop learning process, which can impose considerable computational and memory burdens. By drawing upon implicit differentiation, we develop the implicit MAML algorithm, which depends only on the solution to the inner level optimization and not the path taken by the inner loop optimizer. This effectively decouples the meta-gradient computation from the choice of inner loop optimizer. As a result, our approach is agnostic to the choice of inner loop optimizer and can gracefully handle many gradient steps without vanishing gradients or memory constraints. Theoretically, we prove that implicit MAML can compute accurate meta-gradients with a memory footprint no more than that which is required to compute a single inner loop gradient and at no overall increase in the total computational cost. Experimentally, we show that these benefits of implicit MAML translate into empirical gains on few-shot image recognition benchmarks.

影响功能有效地估计了删除单个训练数据点对模型学习参数的影响。尽管影响估计值与线性模型的剩余重新进行了良好的重新对齐，但最近的作品表明，在神经网络中，这种比对通常很差。在这项工作中，我们通过将其分解为五个单独的术语来研究导致这种差异的特定因素。我们研究每个术语对各种架构和数据集的贡献，以及它们如何随网络宽度和培训时间等因素而变化。尽管实际影响函数估计值可能是非线性网络中保留对方的重新培训的差异，但我们表明它们通常是对不同对象的良好近似值，我们称其为近端Bregman响应函数（PBRF）。由于PBRF仍然可以用来回答许多激励影响功能的问题，例如识别有影响力或标记的示例，因此我们的结果表明，影响功能估计的当前算法比以前的错误分析所暗示的更有用的结果。

二阶优化器被认为具有加快神经网络训练的潜力，但是由于曲率矩阵的尺寸巨大，它们通常需要近似值才能计算。最成功的近似家庭是Kronecker因块状曲率估计值（KFAC）。在这里，我们结合了先前工作的工具，以评估确切的二阶更新和仔细消融以建立令人惊讶的结果：由于其近似值，KFAC与二阶更新无关，尤其是，它极大地胜过真实的第二阶段更新。订单更新。这一挑战广泛地相信，并立即提出了为什么KFAC表现如此出色的问题。为了回答这个问题，我们提出了强烈的证据，表明KFAC近似于一阶算法，该算法在神经元上执行梯度下降而不是权重。最后，我们表明，这种优化器通常会在计算成本和数据效率方面改善KFAC。

深度学习在广泛的AI应用方面取得了有希望的结果。较大的数据集和模型一致地产生更好的性能。但是，我们一般花费更长的培训时间，以更多的计算和沟通。在本调查中，我们的目标是在模型精度和模型效率方面提供关于大规模深度学习优化的清晰草图。我们调查最常用于优化的算法，详细阐述了大批量培训中出现的泛化差距的可辩论主题，并审查了解决通信开销并减少内存足迹的SOTA策略。

Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural network architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum.

We introduce a framework based on bilevel programming that unifies gradient-based hyperparameter optimization and meta-learning. We show that an approximate version of the bilevel problem can be solved by taking into explicit account the optimization dynamics for the inner objective. Depending on the specific setting, the outer variables take either the meaning of hyperparameters in a supervised learning problem or parameters of a meta-learner. We provide sufficient conditions under which solutions of the approximate problem converge to those of the exact problem. We instantiate our approach for meta-learning in the case of deep learning where representation layers are treated as hyperparameters shared across a set of training episodes. In experiments, we confirm our theoretical findings, present encouraging results for few-shot learning and contrast the bilevel approach against classical approaches for learning-to-learn.

本文侧重于培训无限层的隐含模型。具体而言，以前的作品采用隐式差分，并解决后向传播的精确梯度。但是，是否有必要计算训练的这种精确但昂贵的渐变？在这项工作中，我们提出了一种新颖的梯度估计，用于隐式模型，命名为Phantom梯度，1）用于精确梯度的昂贵计算; 2）提供了对隐式模型培训的凭经质优选的更新方向。理论上，理论上可以分析可以找到损失景观的上升方向的条件，并基于阻尼展开和Neumann系列提供幻象梯度的两个特定实例化。大规模任务的实验表明，这些轻质幻像梯度大大加快了培训隐式模型中的后向往大约1.7倍，甚至基于想象成上的精确渐变来提高对方法的性能。

The vast majority of successful deep neural networks are trained using variants of stochastic gradient descent (SGD) algorithms. Recent attempts to improve SGD can be broadly categorized into two approaches: (1) adaptive learning rate schemes, such as AdaGrad and Adam, and (2) accelerated schemes, such as heavy-ball and Nesterov momentum. In this paper, we propose a new optimization algorithm, Lookahead, that is orthogonal to these previous approaches and iteratively updates two sets of weights. Intuitively, the algorithm chooses a search direction by looking ahead at the sequence of "fast weights" generated by another optimizer. We show that Lookahead improves the learning stability and lowers the variance of its inner optimizer with negligible computation and memory cost. We empirically demonstrate Lookahead can significantly improve the performance of SGD and Adam, even with their default hyperparameter settings on ImageNet, CIFAR-10/100, neural machine translation, and Penn Treebank.

我们研究了使用尖刺，现场依赖的随机矩阵理论研究迷你批次对深神经网络损失景观的影响。我们表明，批量黑森州的极值值的大小大于经验丰富的黑森州。我们还获得了类似的结果对Hessian的概括高斯牛顿矩阵近似。由于我们的定理，我们推导出作为批量大小的最大学习速率的分析表达式，为随机梯度下降（线性缩放）和自适应算法（例如ADAM（Square Root Scaling）提供了通知实际培训方案，例如光滑，非凸深神经网络。虽然随机梯度下降的线性缩放是在我们概括的更多限制性条件下导出的，但是适应优化者的平方根缩放规则是我们的知识，完全小说。随机二阶方法和自适应方法的百分比，我们得出了最小阻尼系数与学习率与批量尺寸的比率成比例。我们在Cifar-$ 100 $和ImageNet数据集上验证了我们的VGG / WimerEsnet架构上的索赔。根据我们对象检的调查，我们基于飞行学习率和动量学习者开发了一个随机兰齐齐竞争，这避免了对这些关键的超参数进行昂贵的多重评估的需求，并在预残留的情况下显示出良好的初步结果Cifar的architecure - $ 100 $。

We introduce SketchySGD, a stochastic quasi-Newton method that uses sketching to approximate the curvature of the loss function. Quasi-Newton methods are among the most effective algorithms in traditional optimization, where they converge much faster than first-order methods such as SGD. However, for contemporary deep learning, quasi-Newton methods are considered inferior to first-order methods like SGD and Adam owing to higher per-iteration complexity and fragility due to inexact gradients. SketchySGD circumvents these issues by a novel combination of subsampling, randomized low-rank approximation, and dynamic regularization. In the convex case, we show SketchySGD with a fixed stepsize converges to a small ball around the optimum at a faster rate than SGD for ill-conditioned problems. In the non-convex case, SketchySGD converges linearly under two additional assumptions, interpolation and the Polyak-Lojaciewicz condition, the latter of which holds with high probability for wide neural networks. Numerical experiments on image and tabular data demonstrate the improved reliability and speed of SketchySGD for deep learning, compared to standard optimizers such as SGD and Adam and existing quasi-Newton methods.

平衡系统是表达神经计算的有力方法。作为特殊情况，它们包括对神经科学和机器学习的最新兴趣模型，例如平衡复发性神经网络，深度平衡模型或元学习。在这里，我们提出了一个新的原则，用于学习具有时间和空间本地规则的此类系统。我们的原理将学习作为一个最不控制的问题，我们首先引入一个最佳控制器，以将系统带入解决方案状态，然后将学习定义为减少达到这种状态所需的控制量。我们表明，将学习信号纳入动力学作为最佳控制可以以先前未知的方式传输信用分配信息，避免将中间状态存储在内存中，并且不依赖无穷小的学习信号。在实践中，我们的原理可以使基于梯度的学习方法的强大绩效匹配，该方法应用于涉及复发性神经网络和元学习的一系列问题。我们的结果阐明了大脑如何学习并提供解决广泛的机器学习问题的新方法。

二重优化（BO）可用于解决各种重要的机器学习问题，包括但不限于超参数优化，元学习，持续学习和增强学习。常规的BO方法需要通过与隐式分化的低级优化过程进行区分，这需要与Hessian矩阵相关的昂贵计算。最近，人们一直在寻求BO的一阶方法，但是迄今为止提出的方法对于大规模的深度学习应用程序往往是复杂且不切实际的。在这项工作中，我们提出了一种简单的一阶BO算法，仅取决于一阶梯度信息，不需要隐含的区别，并且对于大规模的非凸函数而言是实用和有效的。我们为提出的方法提供了非注重方法分析非凸目标的固定点，并提出了表明其出色实践绩效的经验结果。

找到模型的最佳超参数可以作为双重优化问题，通常使用零级技术解决。在这项工作中，当内部优化问题是凸但不平滑时，我们研究一阶方法。我们表明，近端梯度下降和近端坐标下降序列序列的前向模式分化，雅各比人会收敛到精确的雅各布式。使用隐式差异化，我们表明可以利用内部问题的非平滑度来加快计算。最后，当内部优化问题大约解决时，我们对高度降低的误差提供了限制。关于回归和分类问题的结果揭示了高参数优化的计算益处，尤其是在需要多个超参数时。

Bilevel优化是在机器学习的许多领域中最小化涉及另一个功能的价值函数的问题。在大规模的经验风险最小化设置中，样品数量很大，开发随机方法至关重要，而随机方法只能一次使用一些样品进行进展。但是，计算值函数的梯度涉及求解线性系统，这使得很难得出无偏的随机估计。为了克服这个问题，我们引入了一个新颖的框架，其中内部问题的解决方案，线性系统的解和主要变量同时发展。这些方向是作为总和写成的，使其直接得出无偏估计。我们方法的简单性使我们能够开发全球差异算法，其中所有变量的动力学都会降低差异。我们证明，萨巴（Saba）是我们框架中著名的传奇算法的改编，具有$ o（\ frac1t）$收敛速度，并且在polyak-lojasciewicz的假设下实现了线性收敛。这是验证这些属性之一的双光线优化的第一种随机算法。数值实验验证了我们方法的实用性。

用于估计模型不确定性的线性拉普拉斯方法在贝叶斯深度学习社区中引起了人们的重新关注。该方法提供了可靠的误差线，并接受模型证据的封闭式表达式，从而可以选择模型超参数。在这项工作中，我们检查了这种方法背后的假设，尤其是与模型选择结合在一起。我们表明，这些与一些深度学习的标准工具（构成近似方法和归一化层）相互作用，并为如何更好地适应这种经典方法对现代环境提出建议。我们为我们的建议提供理论支持，并在MLP，经典CNN，具有正常化层，生成性自动编码器和变压器的剩余网络上进行经验验证它们。

We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases is completely non-sparse. It is derived by approximating various large blocks of the Fisher (corresponding to entire layers) as being the Kronecker product of two much smaller matrices. While only several times more expensive to compute than the plain stochastic gradient, the updates produced by K-FAC make much more progress optimizing the objective, which results in an algorithm that can be much faster than stochastic gradient descent with momentum in practice. And unlike some previously proposed approximate natural-gradient/Newton methods which use high-quality non-diagonal curvature matrices (such as Hessian-free optimization), K-FAC works very well in highly stochastic optimization regimes. This is because the cost of storing and inverting K-FAC's approximation to the curvature matrix does not depend on the amount of data used to estimate it, which is a feature typically associated only with diagonal or low-rank approximations to the curvature matrix.