众所周知，过度参数化的深网能够完全拟合训练数据，同时显示出良好的概括性能。从线性回归上的直觉中得出的常见范式表明，大型网络甚至可以插入嘈杂的数据，而不会显着偏离地面真相信号。目前，缺少这种现象的精确表征。在这项工作中，我们介绍了深网的损失景观清晰度的实证研究，因为我们系统地控制了模型参数和训练时期的数量。我们将研究扩展到培训数据的街区以及清洁和嘈杂标记的样本。我们的发现表明，输入空间中的损失清晰度均遵循模型和时期的双重下降，在嘈杂的标签周围观察到了较差的峰值。与现有直觉相比，小型插值模型尤其适合干净和嘈杂的数据，但大型模型表达了平稳而平坦的损失景观。

在他们的损失景观方面观看神经网络模型在学习的统计力学方法方面具有悠久的历史，并且近年来它在机器学习中得到了关注。除此之外，已显示局部度量（例如损失景观的平滑度）与模型的全局性质（例如良好的泛化性能）相关联。在这里，我们对数千个神经网络模型的损失景观结构进行了详细的实证分析，系统地改变了学习任务，模型架构和/或数据数量/质量。通过考虑试图捕获损失景观的不同方面的一系列指标，我们证明了最佳的测试精度是如下：损失景观在全球连接;训练型模型的集合彼此更像;而模型会聚到局部平滑的地区。我们还表明，当模型很小或培训以较低质量数据时，可以出现全球相连的景观景观;而且，如果损失景观全球相连，则培训零损失实际上可以导致更糟糕的测试精度。我们详细的经验结果阐明了学习阶段的阶段（以及后续双重行为），基本与偶然的决定因素良好的概括决定因素，负载样和温度相同的参数在学习过程中，不同的影响对模型的损失景观的影响不同和数据，以及地方和全球度量之间的关系，近期兴趣的所有主题。

在许多情况下，更简单的模型比更复杂的模型更可取，并且该模型复杂性的控制是机器学习中许多方法的目标，例如正则化，高参数调整和体系结构设计。在深度学习中，很难理解复杂性控制的潜在机制，因为许多传统措施并不适合深度神经网络。在这里，我们开发了几何复杂性的概念，该概念是使用离散的dirichlet能量计算的模型函数变异性的量度。使用理论论据和经验结果的结合，我们表明，许多常见的训练启发式方法，例如参数规范正规化，光谱规范正则化，平稳性正则化，隐式梯度正则化，噪声正则化和参数初始化的选择，都可以控制几何学复杂性，并提供一个统一的框架，以表征深度学习模型的行为。

We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effective model complexity and conjecture a generalized double descent with respect to this measure. Furthermore, our notion of model complexity allows us to identify certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance. * Work performed in part while Preetum Nakkiran was interning at OpenAI, with Ilya Sutskever. We especially thank Mikhail Belkin and Christopher Olah for helpful discussions throughout this work.

尽管他们能够代表高度表现力的功能，但深度学习模型似乎找到了简单的解决方案，这些解决方案令人惊讶地概括了。光谱偏见 - 神经网络优先学习低频功能的趋势 - 是对此现象的一种可能解释，但是到目前为止，在理论模型和简化实验中，主要观察到了光谱偏差。在这项工作中，我们提出了用于测量CIFAR-10和Imagenet上现代图像分类网络中光谱偏差的方法。我们发现这些网络确实表现出光谱偏差，并且提高CIFAR-10测试准确性的干预措施往往会产生学到的功能，这些功能总体上具有较高的频率，但在每个类别的示例附近频率较低。这种趋势在培训时间，模型架构，培训示例的数量，数据增强和自我介绍的变化之间存在。我们还探索了功能频率和图像频率之间的连接，并发现光谱偏置对自然图像中普遍存在的低频敏感。在Imagenet上，我们发现学习的功能频率也随内部类别的多样性而变化，并且在更多样化的类别上具有较高的频率。我们的工作使测量并最终影响用于图像分类的神经网络的光谱行为，并且是理解为什么深层模型良好概述的一步。

在最近的几项研究中已经显示了过度参数化在实现卓越概括性能方面的好处，证明了在实践中使用较大模型的趋势。然而，在强大的学习背景下，神经网络大小的影响尚未得到很好的研究。在这项工作中，我们发现，在大量错误标记的示例的存在下，将网络大小的增加超出某个点可能是有害的。特别是，当标签噪声增加时，最初是单调或“双重下降”测试损失曲线（W.R.T.网络宽度）变成U形或双U形曲线，这表明某些模型具有中等大小的模型实现了最佳的概括。我们观察到，当通过随机修剪通过密度控制网络大小时，观察到相似的测试损失行为。我们还通过偏置变化分解和理论上表征标签噪声塑造方差项的方式来仔细研究现象。即使采用最新的鲁棒方法，也可以观察到测试损失的类似行为，这表明限制网络大小可以进一步提高现有方法。最后，我们从经验上检查网络大小对学习函数平稳性的影响，并发现最初的大小和平滑度之间的负相关性是由标签噪声翻转的。

In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality. Motivated by prior work connecting the geometry of the loss landscape and generalization, we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulation results in a minmax optimization problem on which gradient descent can be performed efficiently. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets (e.g., CIFAR-{10, 100}, Ima-geNet, finetuning tasks) and models, yielding novel state-of-the-art performance for several. Additionally, we find that SAM natively provides robustness to label noise on par with that provided by state-of-the-art procedures that specifically target learning with noisy labels. We open source our code at https: //github.com/google-research/sam. * Work done as part of the Google AI Residency program.

我们研究了使用尖刺，现场依赖的随机矩阵理论研究迷你批次对深神经网络损失景观的影响。我们表明，批量黑森州的极值值的大小大于经验丰富的黑森州。我们还获得了类似的结果对Hessian的概括高斯牛顿矩阵近似。由于我们的定理，我们推导出作为批量大小的最大学习速率的分析表达式，为随机梯度下降（线性缩放）和自适应算法（例如ADAM（Square Root Scaling）提供了通知实际培训方案，例如光滑，非凸深神经网络。虽然随机梯度下降的线性缩放是在我们概括的更多限制性条件下导出的，但是适应优化者的平方根缩放规则是我们的知识，完全小说。随机二阶方法和自适应方法的百分比，我们得出了最小阻尼系数与学习率与批量尺寸的比率成比例。我们在Cifar-$ 100 $和ImageNet数据集上验证了我们的VGG / WimerEsnet架构上的索赔。根据我们对象检的调查，我们基于飞行学习率和动量学习者开发了一个随机兰齐齐竞争，这避免了对这些关键的超参数进行昂贵的多重评估的需求，并在预残留的情况下显示出良好的初步结果Cifar的architecure - $ 100 $。

Power等人报道的\ emph {grokking现象} {power2021grokking}是指一个长期过度拟合之后，似乎突然过渡到完美的概括。在本文中，我们试图通过一系列经验研究来揭示Grokking的基础。具体而言，我们在极端的训练阶段（称为\ emph {slingshot机构）发现了一个优化的异常缺陷自适应优化器。可以通过稳定和不稳定的训练方案之间的循环过渡来测量弹弓机制的突出伪像，并且可以通过最后一层重量的规范的循环行为轻松监测。我们从经验上观察到，在\ cite {power2021grokking}中报道的无明确正规化，几乎完全发生在\ emph {slingshots}的开始时，并且没有它。虽然在更一般的环境中常见且容易复制，但弹弓机制并不遵循我们所知道的任何已知优化理论，并且可以轻松地忽略而无需深入研究。我们的工作表明，在培训的后期阶段，适应性梯度优化器的令人惊讶且有用的归纳偏见，要求对其起源进行修订。

This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenvalues. We leverage upon this observation to construct a local-entropy-based objective function that favors well-generalizable solutions lying in large flat regions of the energy landscape, while avoiding poorly-generalizable solutions located in the sharp valleys. Conceptually, our algorithm resembles two nested loops of SGD where we use Langevin dynamics in the inner loop to compute the gradient of the local entropy before each update of the weights. We show that the new objective has a smoother energy landscape and show improved generalization over SGD using uniform stability, under certain assumptions. Our experiments on convolutional and recurrent networks demonstrate that Entropy-SGD compares favorably to state-of-the-art techniques in terms of generalization error and training time.

深度学习归一化技术的基本特性，例如批准归一化，正在使范围前的参数量表不变。此类参数的固有域是单位球，因此可以通过球形优化的梯度优化动力学以不同的有效学习率（ELR）来表示，这是先前研究的。在这项工作中，我们使用固定的ELR直接研究了训练量表不变的神经网络的特性。我们根据ELR值发现了这种训练的三个方案：收敛，混乱平衡和差异。我们详细研究了这些制度示例的理论检查，以及对真实规模不变深度学习模型的彻底经验分析。每个制度都有独特的特征，并反映了内在损失格局的特定特性，其中一些与先前对常规和规模不变的神经网络培训的研究相似。最后，我们证明了如何在归一化网络的常规培训以及如何利用它们以实现更好的Optima中反映发现的制度。

Neural networks are known to be a class of highly expressive functions able to fit even random inputoutput mappings with 100% accuracy. In this work we present properties of neural networks that complement this aspect of expressivity. By using tools from Fourier analysis, we highlight a learning bias of deep networks towards low frequency functions -i.e. functions that vary globally without local fluctuations -which manifests itself as a frequency-dependent learning speed. Intuitively, this property is in line with the observation that over-parameterized networks prioritize learning simple patterns that generalize across data samples. We also investigate the role of the shape of the data manifold by presenting empirical and theoretical evidence that, somewhat counter-intuitively, learning higher frequencies gets easier with increasing manifold complexity.

人们通常认为，修剪网络不仅会降低深网的计算成本，而且还可以通过降低模型容量来防止过度拟合。但是，我们的工作令人惊讶地发现，网络修剪有时甚至会加剧过度拟合。我们报告了出乎意料的稀疏双后裔现象，随着我们通过网络修剪增加模型稀疏性，首先测试性能变得更糟（由于过度拟合），然后变得更好（由于过度舒适），并且终于变得更糟（由于忘记了有用的有用信息）。尽管最近的研究集中在模型过度参数化方面，但他们未能意识到稀疏性也可能导致双重下降。在本文中，我们有三个主要贡献。首先，我们通过广泛的实验报告了新型的稀疏双重下降现象。其次，对于这种现象，我们提出了一种新颖的学习距离解释，即$ \ ell_ {2} $稀疏模型的学习距离（从初始化参数到最终参数）可能与稀疏的双重下降曲线良好相关，并更好地反映概括比最小平坦。第三，在稀疏的双重下降的背景下，彩票票假设中的获胜票令人惊讶地并不总是赢。

Deep neural networks excel at learning the training data, but often provide incorrect and confident predictions when evaluated on slightly different test examples. This includes distribution shifts, outliers, and adversarial examples. To address these issues, we propose Manifold Mixup, a simple regularizer that encourages neural networks to predict less confidently on interpolations of hidden representations. Manifold Mixup leverages semantic interpolations as additional training signal, obtaining neural networks with smoother decision boundaries at multiple levels of representation. As a result, neural networks trained with Manifold Mixup learn class-representations with fewer directions of variance. We prove theory on why this flattening happens under ideal conditions, validate it on practical situations, and connect it to previous works on information theory and generalization. In spite of incurring no significant computation and being implemented in a few lines of code, Manifold Mixup improves strong baselines in supervised learning, robustness to single-step adversarial attacks, and test log-likelihood.

Deep neural networks may easily memorize noisy labels present in real-world data, which degrades their ability to generalize. It is therefore important to track and evaluate the robustness of models against noisy label memorization. We propose a metric, called susceptibility, to gauge such memorization for neural networks. Susceptibility is simple and easy to compute during training. Moreover, it does not require access to ground-truth labels and it only uses unlabeled data. We empirically show the effectiveness of our metric in tracking memorization on various architectures and datasets and provide theoretical insights into the design of the susceptibility metric. Finally, we show through extensive experiments on datasets with synthetic and real-world label noise that one can utilize susceptibility and the overall training accuracy to distinguish models that maintain a low memorization on the training set and generalize well to unseen clean data.

我们研究了回归中神经网络（NNS）的模型不确定性的方法。为了隔离模型不确定性的效果，我们专注于稀缺训练数据的无噪声环境。我们介绍了关于任何方法都应满足的模型不确定性的五个重要的逃亡者。但是，我们发现，建立的基准通常无法可靠地捕获其中一些逃避者，即使是贝叶斯理论要求的基准。为了解决这个问题，我们介绍了一种新方法来捕获NNS的模型不确定性，我们称之为基于神经优化的模型不确定性（NOMU）。 NOMU的主要思想是设计一个由两个连接的子NN组成的网络体系结构，一个用于模型预测，一个用于模型不确定性，并使用精心设计的损耗函数进行训练。重要的是，我们的设计执行NOMU满足我们的五个Desiderata。由于其模块化体系结构，NOMU可以为任何给定（先前训练）NN提供模型不确定性，如果访问其培训数据。我们在各种回归任务和无嘈杂的贝叶斯优化（BO）中评估NOMU，并具有昂贵的评估。在回归中，NOMU至少和最先进的方法。在BO中，Nomu甚至胜过所有考虑的基准。

梯度下降可能令人惊讶地擅长优化深层神经网络，而不会过度拟合并且没有明确的正则化。我们发现，梯度下降的离散步骤通过惩罚具有较大损耗梯度的梯度下降轨迹来隐式化模型。我们称之为隐式梯度正则化（IGR），并使用向后错误分析来计算此正则化的大小。我们从经验上确认，隐式梯度正则化偏向梯度下降到平面最小值，在该较小情况下，测试误差很小，溶液对嘈杂的参数扰动是可靠的。此外，我们证明了隐式梯度正规化项可以用作显式正常化程序，从而使我们能够直接控制此梯度正则化。从更广泛的角度来看，我们的工作表明，向后错误分析是一种有用的理论方法，即对学习率，模型大小和参数正则化如何相互作用以确定用梯度下降优化的过度参数化模型的属性。

Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and wellchosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effects on the underlying loss landscape, are not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple "filter normalization" method that helps us visualize loss function curvature and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture affects the loss landscape, and how training parameters affect the shape of minimizers.

Accurate uncertainty quantification is a major challenge in deep learning, as neural networks can make overconfident errors and assign high confidence predictions to out-of-distribution (OOD) inputs. The most popular approaches to estimate predictive uncertainty in deep learning are methods that combine predictions from multiple neural networks, such as Bayesian neural networks (BNNs) and deep ensembles. However their practicality in real-time, industrial-scale applications are limited due to the high memory and computational cost. Furthermore, ensembles and BNNs do not necessarily fix all the issues with the underlying member networks. In this work, we study principled approaches to improve uncertainty property of a single network, based on a single, deterministic representation. By formalizing the uncertainty quantification as a minimax learning problem, we first identify distance awareness, i.e., the model's ability to quantify the distance of a testing example from the training data, as a necessary condition for a DNN to achieve high-quality (i.e., minimax optimal) uncertainty estimation. We then propose Spectral-normalized Neural Gaussian Process (SNGP), a simple method that improves the distance-awareness ability of modern DNNs with two simple changes: (1) applying spectral normalization to hidden weights to enforce bi-Lipschitz smoothness in representations and (2) replacing the last output layer with a Gaussian process layer. On a suite of vision and language understanding benchmarks, SNGP outperforms other single-model approaches in prediction, calibration and out-of-domain detection. Furthermore, SNGP provides complementary benefits to popular techniques such as deep ensembles and data augmentation, making it a simple and scalable building block for probabilistic deep learning. Code is open-sourced at https://github.com/google/uncertainty-baselines

与损耗值相关的权重集的体积可能是由于合同体积的现象而导致的隐式正则化的来源，而高脑证明的几何图的尺寸增加了。我们通过考虑因沿训练路径可用的潜在重量设置更新的分布而产生的相似属性，将几何正规化猜想和提取物引入了双重下降现象的解释，如果该分布在遍历卷中缩回量的维度曲线曲线，则该分布的分布。接近全球最小值时，我们可以预期重新出现几何正则化。我们说明数据保真性表示复杂性如何影响模型容量双重下降插值阈值。源自不同几何形式的时期和模型容量的双重下降曲线的存在可能意味着具有尺寸调整的n-sphere体积对应关系的封闭n个manifolds的通用性。