The loss functions of deep neural networks are complex and their geometric properties are not well understood. We show that the optima of these complex loss functions are in fact connected by simple curves over which training and test accuracy are nearly constant. We introduce a training procedure to discover these high-accuracy pathways between modes. Inspired by this new geometric insight, we also propose a new ensembling method entitled Fast Geometric Ensembling (FGE). Using FGE we can train high-performing ensembles in the time required to train a single model. We achieve improved performance compared to the recent state-of-the-art Snapshot Ensembles, on CIFAR-10, CIFAR-100, and ImageNet. * Equal contribution. 1 Suppose we have three weight vectors w1, w2, w3. We set u = (w2 − w1), v = (w3 − w1) − w3 − w1, w2 − w1 / w2 − w1 2 • (w2 − w1). Then the normalized vectors û = u/ u , v = v/ v form an orthonormal basis in the plane containing w1, w2, w3. To visualize the loss in this plane, we define a Cartesian grid in the basis û, v and evaluate the networks corresponding to each of the points in the grid. A point P with coordinates (x, y) in the plane would then be given by P = w1 + x • û + y • v.

Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) procedure finds much flatter solutions than SGD, and approximates the recent Fast Geometric Ensembling (FGE) approach with a single model. Using SWA we achieve notable improvement in test accuracy over conventional SGD training on a range of state-of-the-art residual networks, PyramidNets, DenseNets, and Shake-Shake networks on CIFAR-10, CIFAR-100, and ImageNet. In short, SWA is extremely easy to implement, improves generalization, and has almost no computational overhead.

通过更好地了解多层网络的损失表面，我们可以构建更强大和准确的培训程序。最近发现，独立训练的SGD解决方案可以沿近持续训练损失的一维路径连接。在本文中，我们表明存在模式连接的单纯复合物，形成低损耗的多维歧管，连接许多独立培训的型号。灵感来自这一发现，我们展示了如何有效地建立快速合奏的单纯性复杂，表现优于准确性，校准和对数据集移位的鲁棒性的独立培训的深度集合。值得注意的是，我们的方法只需要几个训练时期来发现低损失单纯乳，从预先接受训练的解决方案开始。代码可在https://github.com/g-benton/loss-surface-simplexes中获得。

We propose SWA-Gaussian (SWAG), a simple, scalable, and general purpose approach for uncertainty representation and calibration in deep learning. Stochastic Weight Averaging (SWA), which computes the first moment of stochastic gradient descent (SGD) iterates with a modified learning rate schedule, has recently been shown to improve generalization in deep learning. With SWAG, we fit a Gaussian using the SWA solution as the first moment and a low rank plus diagonal covariance also derived from the SGD iterates, forming an approximate posterior distribution over neural network weights; we then sample from this Gaussian distribution to perform Bayesian model averaging. We empirically find that SWAG approximates the shape of the true posterior, in accordance with results describing the stationary distribution of SGD iterates. Moreover, we demonstrate that SWAG performs well on a wide variety of tasks, including out of sample detection, calibration, and transfer learning, in comparison to many popular alternatives including MC dropout, KFAC Laplace, SGLD, and temperature scaling.

在神经网络的经验风险景观中扁平最小值的性质已经讨论了一段时间。越来越多的证据表明他们对尖锐物质具有更好的泛化能力。首先，我们讨论高斯混合分类模型，并分析显示存在贝叶斯最佳点估算器，其对应于属于宽平区域的最小值。可以通过直接在分类器（通常是独立的）或学习中使用的可分解损耗函数上应用最大平坦度算法来找到这些估计器。接下来，我们通过广泛的数值验证将分析扩展到深度学习场景。使用两种算法，熵-SGD和复制-SGD，明确地包括在优化目标中，所谓的非局部平整度措施称为本地熵，我们一直提高常见架构的泛化误差（例如Resnet，CeffectnNet）。易于计算的平坦度测量显示与测试精度明确的相关性。

Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradientbased optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at https://github.com/loshchil/SGDR

我们通过将其基于实现功能空间而不是参数空间的几何形状来系统地研究深度神经网络景观的方法。将分类器分组到等效类中，我们开发了一个标准化的参数化，其中所有对称性都被删除，从而导致环形拓扑。在这个空间上，我们探讨了误差景观而不是损失。这使我们能够得出有意义的概念，即最小化器的平坦度和连接它们的地球通道的概念。使用不同的优化算法，这些算法采样具有不同平坦度的最小化器，我们研究模式连接性和相对距离。测试各种最先进的体系结构和基准数据集，我们确认了平面度和泛化性能之间的相关性；我们进一步表明，在功能空间中，minima彼此更近，并且连接它们的大地测量学的屏障很小。我们还发现，通过梯度下降的变体发现的最小化器可以通过由参数空间中的两个直线组成的零误差路径连接，即带有单个弯曲的多边形链。我们观察到具有二进制权重和激活的神经网络中相似的定性结果，这为在这种情况下的连通性提供了第一个结果之一。我们的结果取决于对称性的去除，并且与对简单浅层模型进行的一些分析研究所描述的丰富现象学非常吻合。

在本文中，我们推测，如果考虑到神经网络的置换不变性，SGD解决方案可能不会在它们之间的线性插值中没有障碍。尽管这是一个大胆的猜想，但我们展示了广泛的经验尝试却没有反驳。我们进一步提供了初步的理论结果来支持我们的猜想。我们的猜想对彩票票证假设，分布式培训和合奏方法有影响。

Many applications require sparse neural networks due to space or inference time restrictions. There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-tosparse training methods. Our method updates the topology of the sparse network during training by using parameter magnitudes and infrequent gradient calculations. We show that this approach requires fewer floating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques. We demonstrate state-of-the-art sparse training results on a variety of networks and datasets, including ResNet-50, MobileNets on Imagenet-2012, and RNNs on WikiText-103. Finally, we provide some insights into why allowing the topology to change during the optimization can overcome local minima encountered when the topology remains static * .

关于稀疏神经网络训练（稀疏训练）的最新研究表明，通过从头开始训练本质上稀疏的神经网络可以实现绩效和效率之间的令人信服的权衡。现有的稀疏训练方法通常努力在一次跑步中找到最佳的稀疏子网，而无需涉及任何昂贵的密集或预训练步骤。例如，作为最突出的方向之一，动态稀疏训练（DST）能够通过在训练过程中迭代发展稀疏拓扑来实现竞争性训练的竞争性能。在本文中，我们认为最好分配有限的资源来创建多个低损失的稀疏子网并将其超级置于更强的基因，而不是完全分配所有资源以找到单个子网络。为了实现这一目标，需要两个Desiderata：（1）在一个培训过程中有效生产许多低损失的子网，即所谓的廉价门票，仅限于用于密集培训的标准培训时间； （2）将这些廉价的门票有效地超级为一个更强的子网，而无需超越约束参数预算。为了证实我们的猜想，我们提出了一种新颖的稀疏训练方法，称为\ textbf {sup-tickets}，可以在单个稀疏到较小的训练过程中同时满足上述两个desiderata。在CIFAR-10/100和Imagenet上的各种现代体系结构中，我们表明，SUP-Tickets与现有的稀疏训练方法无缝集成，并显示出一致的性能提高。

This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenvalues. We leverage upon this observation to construct a local-entropy-based objective function that favors well-generalizable solutions lying in large flat regions of the energy landscape, while avoiding poorly-generalizable solutions located in the sharp valleys. Conceptually, our algorithm resembles two nested loops of SGD where we use Langevin dynamics in the inner loop to compute the gradient of the local entropy before each update of the weights. We show that the new objective has a smoother energy landscape and show improved generalization over SGD using uniform stability, under certain assumptions. Our experiments on convolutional and recurrent networks demonstrate that Entropy-SGD compares favorably to state-of-the-art techniques in terms of generalization error and training time.

随机重量平均（SWA）被认为是一种简单的，而一种有效的方法来改善随机梯度下降（SGD）的推广，用于训练深层神经网络（DNN）。解释其成功的常见见解是，在配备周期性或高常数学习率的SGD过程之后的平均权重可以发现更广泛的Optima，然后导致更好的泛化。我们给出了一个不同意上述内容的新洞察力。我们的表征，SWA的性能高度依赖于SWA收敛前运行的SGD进程的程度，并且权重平均的操作仅有助于减少方差。这种新的Insight表明了更好的算法设计上的实用指南。作为一个实例化，我们表明，随着收敛不足的SGD过程，运行SWA更多次导致泛化方面的持续增量益处。我们的发现在不同网络架构上的广泛实验得到了证实，包括基线CNN，PRERESNET-164，WieresNetNet-28-10，VGG16，Resnet-50，Reset-152，DenSenet-161和不同的数据集，包括CiFar- {10,100}和想象因。

提高黑箱逃避攻击的可转移性的一种既定方法是在基于合奏的替代物上制作对抗性例子，以提高多样性。我们认为可转移性与不确定性根本相关。基于一种最先进的贝叶斯深度学习技术，我们提出了一种新方法，通过大约从神经网络权重的后验分布进行采样来有效地构建代理，这代表了每个参数的价值的信念。我们对Imagenet，CIFAR-10和MNIST进行的广泛实验表明，在内部结构和结构转移性中，我们的方法显着提高了四个最新攻击的成功率（高达83.2个百分点）。在Imagenet上，与经过独立训练的DNN合奏相比，我们的方法可以达到成功率的94％，同时将训练计算从11.6降低到2.4个Exaflops。与为此目的设计的三种测试时间技术相比，我们的香草代理人的可传递性高87.5％。我们的工作表明，训练代理人的方法被忽略了，尽管这是基于转移攻击的重要组成部分。因此，我们是第一个回顾几种培训方法在提高可传递性方面的有效性的。我们提供了新的方向，以更好地了解可转移性现象，并为将来的工作提供简单但强大的基线。

利用深度神经网络在监督学习设置中产生校准预测概率的多种技术已经出现了利用在多个随机起点（深坐标）的循环训练或培训期间发现的集合不同解决方案的方法。但是，只有有限的工作已经调查了探索各种解决方案（后模式）探索本地区域的效用。在CIFAR-10数据集上使用三种众所周知的深层架构，我们评估了几种简单的方法，用于探索重量空间的局部区域，相对于BRICR得分，准确性和预期的校准误差。我们考虑贝叶斯推理技术（变分推理和汉密尔顿蒙特卡罗施加到Softmax输出层）以及利用Optima附近的随机梯度下降轨迹。在将单独模式添加到合奏中均匀提高性能时，我们表明，这里考虑的简单模式探索方法在没有模式探索的情况下对整体产生的简单模式勘探方法很少。

我们研究了使用尖刺，现场依赖的随机矩阵理论研究迷你批次对深神经网络损失景观的影响。我们表明，批量黑森州的极值值的大小大于经验丰富的黑森州。我们还获得了类似的结果对Hessian的概括高斯牛顿矩阵近似。由于我们的定理，我们推导出作为批量大小的最大学习速率的分析表达式，为随机梯度下降（线性缩放）和自适应算法（例如ADAM（Square Root Scaling）提供了通知实际培训方案，例如光滑，非凸深神经网络。虽然随机梯度下降的线性缩放是在我们概括的更多限制性条件下导出的，但是适应优化者的平方根缩放规则是我们的知识，完全小说。随机二阶方法和自适应方法的百分比，我们得出了最小阻尼系数与学习率与批量尺寸的比率成比例。我们在Cifar-$ 100 $和ImageNet数据集上验证了我们的VGG / WimerEsnet架构上的索赔。根据我们对象检的调查，我们基于飞行学习率和动量学习者开发了一个随机兰齐齐竞争，这避免了对这些关键的超参数进行昂贵的多重评估的需求，并在预残留的情况下显示出良好的初步结果Cifar的architecure - $ 100 $。

我们使用高斯过程扰动模型在高维二次上的真实和批量风险表面之间的高斯过程扰动模型分析和解释迭代平均的泛化性能。我们从我们的理论结果中获得了三个现象\姓名：}（1）将迭代平均值（ia）与大型学习率和正则化进行了改进的正规化的重要性。 （2）对较少频繁平均的理由。 （3）我们预计自适应梯度方法同样地工作，或者更好，而不是其非自适应对应物的迭代平均值。灵感来自这些结果\姓据{，一起与}对迭代解决方案多样性的适当正则化的重要性，我们提出了两个具有迭代平均的自适应算法。与随机梯度下降（SGD）相比，这些结果具有明显更好的结果，需要较少调谐并且不需要早期停止或验证设定监视。我们在各种现代和古典网络架构上展示了我们对CiFar-10/100，Imagenet和Penn TreeBank数据集的方法的疗效。

差异隐私（DP）提供了正式的隐私保证，以防止对手可以访问机器学习模型，从而从提取有关单个培训点的信息。最受欢迎的DP训练方法是差异私有随机梯度下降（DP-SGD），它通过在训练过程中注入噪声来实现这种保护。然而，以前的工作发现，DP-SGD通常会导致标准图像分类基准的性能显着降解。此外，一些作者假设DP-SGD在大型模型上固有地表现不佳，因为保留隐私所需的噪声规范与模型维度成正比。相反，我们证明了过度参数化模型上的DP-SGD可以比以前想象的要好得多。将仔细的超参数调整与简单技术结合起来，以确保信号传播并提高收敛速率，我们获得了新的SOTA，而没有额外数据的CIFAR-10，在81.4％的81.4％下（8，10^{ - 5}） - 使用40 -layer wide-Resnet，比以前的SOTA提高了71.7％。当对预训练的NFNET-F3进行微调时，我们在ImageNet（0.5，8*10^{ - 7}）下达到了83.8％的TOP-1精度。此外，我们还在（8，8 \ cdot 10^{ - 7}）下达到了86.7％的TOP-1精度，DP仅比当前的非私人SOTA仅4.3％。我们认为，我们的结果是缩小私人图像分类和非私有图像分类之间准确性差距的重要一步。

Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and wellchosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effects on the underlying loss landscape, are not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple "filter normalization" method that helps us visualize loss function curvature and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture affects the loss landscape, and how training parameters affect the shape of minimizers.

The ability to estimate epistemic uncertainty is often crucial when deploying machine learning in the real world, but modern methods often produce overconfident, uncalibrated uncertainty predictions. A common approach to quantify epistemic uncertainty, usable across a wide class of prediction models, is to train a model ensemble. In a naive implementation, the ensemble approach has high computational cost and high memory demand. This challenges in particular modern deep learning, where even a single deep network is already demanding in terms of compute and memory, and has given rise to a number of attempts to emulate the model ensemble without actually instantiating separate ensemble members. We introduce FiLM-Ensemble, a deep, implicit ensemble method based on the concept of Feature-wise Linear Modulation (FiLM). That technique was originally developed for multi-task learning, with the aim of decoupling different tasks. We show that the idea can be extended to uncertainty quantification: by modulating the network activations of a single deep network with FiLM, one obtains a model ensemble with high diversity, and consequently well-calibrated estimates of epistemic uncertainty, with low computational overhead in comparison. Empirically, FiLM-Ensemble outperforms other implicit ensemble methods, and it and comes very close to the upper bound of an explicit ensemble of networks (sometimes even beating it), at a fraction of the memory cost.

最近对稀疏神经网络的作品已经证明了独立从头开始训练稀疏子网，以匹配其相应密集网络的性能。然而，识别这种稀疏的子网（获奖票）涉及昂贵的迭代火车 - 培训 - 培训过程（例如，彩票票证假设）或过度扩展的训练时间（例如，动态稀疏训练）。在这项工作中，我们在稀疏神经网络训练和深度合并技术之间汲取了独特的联系，产生了一个名为FreeTickets的新型集合学习框架。 FreeTickets而不是从密集的网络开始，随机初始化稀疏的子网，然后在动态调整其稀疏掩码的同时列举子网，从而在整个训练过程中产生许多不同的稀疏子网。 FreeTickets被定义为这些稀疏子网的集合，在这种单次通过，稀疏稀疏训练中自由获得，其仅使用Vanilla密集培训所需的计算资源的一小部分。此外，尽管是模型的集合，但与单一密集模型相比，FreeTickets的参数和训练拖鞋更少：这种看似反向直观的结果是由于每个子网的高稀疏性。与标准致密基线相比，观察到惯性基因术，以预测准确性，不确定度估计，鲁棒性和效率相比表现出显着的全面改进。 FreeTickets在ImageNet上只使用后者所需的四分之一的培训拖鞋，可以轻松地表达Naive Deep EndleBe。我们的结果提供了对稀疏神经网络的强度的见解，并表明稀疏性的好处超出了通常预期的推理效率。