Recent works have cast some light on the mystery of why deep nets fit any data and generalize despite being very overparametrized. This paper analyzes training and generalization for a simple 2-layer ReLU net with random initialization, and provides the following improvements over recent works: (i) Using a tighter characterization of training speed than recent papers, an explanation for why training a neural net with random labels leads to slower training, as originally observed in [Zhang et al. ICLR'17]. (ii) Generalization bound independent of network size, using a data-dependent complexity measure. Our measure distinguishes clearly between random labels and true labels on MNIST and CIFAR, as shown by experiments. Moreover, recent papers require sample complexity to increase (slowly) with the size, while our sample complexity is completely independent of the network size. (iii) Learnability of a broad class of smooth functions by 2-layer ReLU nets trained via gradient descent.The key idea is to track dynamics of training and generalization via properties of a related kernel.

Gradient descent finds a global minimum in training deep neural networks despite the objective function being non-convex. The current paper proves gradient descent achieves zero training loss in polynomial time for a deep overparameterized neural network with residual connections (ResNet). Our analysis relies on the particular structure of the Gram matrix induced by the neural network architecture. This structure allows us to show the Gram matrix is stable throughout the training process and this stability implies the global optimality of the gradient descent algorithm. We further extend our analysis to deep residual convolutional neural networks and obtain a similar convergence result.

Deep neural networks (DNNs) have demonstrated dominating performance in many fields; since AlexNet, networks used in practice are going wider and deeper. On the theoretical side, a long line of works have been focusing on why we can train neural networks when there is only one hidden layer. The theory of multi-layer networks remains unsettled. In this work, we prove simple algorithms such as stochastic gradient descent (SGD) can find global minima on the training objective of DNNs in polynomial time. We only make two assumptions: the inputs do not degenerate and the network is over-parameterized. The latter means the number of hidden neurons is sufficiently large: polynomial in L, the number of DNN layers and in n, the number of training samples. As concrete examples, starting from randomly initialized weights, we show that SGD attains 100% training accuracy in classification tasks, or minimizes regression loss in linear convergence speed ε ∝ e −Ω(T ) , with running time polynomial in n and L. Our theory applies to the widely-used but non-smooth ReLU activation, and to any smooth and possibly non-convex loss functions. In terms of network architectures, our theory at least applies to fully-connected neural networks, convolutional neural networks (CNN), and residual neural networks (ResNet).* Equal contribution . Full version and future updates are available at https://arxiv.org/abs/1811.03962.This paper is a follow up to the recurrent neural network (RNN) paper (Allen-Zhu et al., 2018b) by the same set of authors. Most of the techniques used in this paper were already discovered in the RNN paper, and this paper can be viewed as a simplification (or to some extent a special case) of the RNN setting in order to reach out to a wider audience. We compare the difference and mention our additional contribution in Section 1.2.

微调是深度学习的常见做法，使用相对较少的训练数据来实现卓越的普遍性导致下游任务。虽然在实践中广泛使用，但它缺乏强烈的理论理解。我们分析了若干架构中线性教师的回归的本方案的样本复杂性。直观地，微调的成功取决于源任务与目标任务之间的相似性，但是测量它是非微不足道的。我们表明相关措施考虑了源任务，目标任务和目标数据的协方差结构之间的关系。在线性回归的设置中，我们表明，在现实的情况下，当上述措施低时，在实际设置下，显着的样本复杂性降低是合理的。对于深线性回归，我们在用预制权重初始化网络时，我们提出了关于基于梯度训练的感应偏差的新颖结果。使用此结果，我们显示此设置的相似度量也受网络深度的影响。我们进一步在浅relu模型上显示结果，并分析了在源和目标任务中的样本复杂性的依赖性。我们经验证明了我们对合成和现实数据的结果。

深度学习理论的最新目标是确定神经网络如何逃脱“懒惰训练”或神经切线内核（NTK）制度，在该制度中，网络与初始化时的一阶泰勒扩展相结合。尽管NTK是最大程度地用于学习密集多项式的最佳选择（Ghorbani等，2021），但它无法学习特征，因此对于学习包括稀疏多项式（稀疏多项式）的许多类别的功能的样本复杂性较差。因此，最近的工作旨在确定基于梯度的算法比NTK更好地概括的设置。一个这样的例子是Bai和Lee（2020）的“ Quadntk”方法，该方法分析了泰勒膨胀中的二阶项。 Bai和Lee（2020）表明，二阶项可以有效地学习稀疏的多项式。但是，它牺牲了学习一般密集多项式的能力。在本文中，我们分析了两层神经网络上的梯度下降如何通过利用NTK（Montanari和Zhong，2020）的光谱表征并在Quadntk方法上构建来逃脱NTK制度。我们首先扩展了光谱分析，以确定参数空间中的“良好”方向，在该空间中我们可以在不损害概括的情况下移动。接下来，我们表明一个宽的两层神经网络可以共同使用NTK和QUADNTK来适合由密集的低度项和稀疏高度术语组成的目标功能 - NTK和Quadntk无法在他们自己的。最后，我们构建了一个正常化程序，该正规化器鼓励我们的参数向量以“良好”的方向移动，并表明正规化损失上的梯度下降将融合到全局最小化器，这也有较低的测试误差。这产生了端到端的融合和概括保证，并自行对NTK和Quadntk进行了可证明的样本复杂性的改善。

Cohen等人的深度学习实验。 [2021]使用确定性梯度下降（GD）显示学习率（LR）和清晰度（即Hessian最大的特征值）的稳定边缘（EOS）阶段不再像传统优化一样行为。清晰度稳定在$ 2/$ LR的左右，并且在迭代中损失不断上下，但仍有整体下降趋势。当前的论文数学分析了EOS阶段中隐式正则化的新机制，因此，由于非平滑损失景观而导致的GD更新沿着最小损失的多种流量进行了一些确定性流程发展。这与许多先前关于隐式偏差依靠无限更新或梯度中的噪声的结果相反。正式地，对于具有某些规律性条件的任何平滑函数$ l $，对于（1）标准化的GD，即具有不同的lr $ \ eta_t = \ frac {\ eta} {||的GD证明了此效果。 \ nabla l（x（t））||} $和损失$ l $; （2）具有常数LR和损失$ \ sqrt {l- \ min_x l（x）} $的GD。两者都可以证明进入稳定性的边缘，在歧管上相关的流量最小化$ \ lambda_ {1}（\ nabla^2 l）$。一项实验研究证实了上述理论结果。

神经网络在许多领域取得了巨大的经验成功。已经观察到，通过一阶方法训练的随机初始化的神经网络能够实现接近零的训练损失，尽管其损失景观是非凸的并且不平滑的。这种现象很少有理论解释。最近，通过分析过参数化制度中的梯度下降〜（GD）和重球方法〜（HB）的梯度来弥合实践和理论之间的这种差距。在这项工作中，通过考虑Nesterov的加速梯度方法〜（nag），我们通过恒定的动量参数进行进一步进展。我们通过Relu激活分析其用于过度参数化的双层完全连接神经网络的收敛性。具体而言，我们证明了NAG的训练误差以非渐近线性收敛率$（1- \θ（1 / \ sqrt {\ kappa}））收敛到零（1 / \ sqrt {\ kappa}）^ t $ the $ t $迭代，其中$ \ Kappa> 1 $由神经网络的初始化和架构决定。此外，我们在NAG和GD和HB的现有收敛结果之间提供了比较。我们的理论结果表明，NAG实现了GD的加速度，其会聚率与HB相当。此外，数值实验验证了我们理论分析的正确性。

过度分化的深网络的泛化神秘具有有动力的努力，了解梯度下降（GD）如何收敛到概括井的低损耗解决方案。现实生活中的神经网络从小随机值初始化，并以分类的“懒惰”或“懒惰”或“NTK”的训练训练，分析更成功，以及最近的结果序列（Lyu和Li ，2020年; Chizat和Bach，2020; Ji和Telgarsky，2020）提供了理论证据，即GD可以收敛到“Max-ramin”解决方案，其零损失可能呈现良好。但是，仅在某些环境中证明了余量的全球最优性，其中神经网络无限或呈指数级宽。目前的纸张能够为具有梯度流动训练的两层泄漏的Relu网，无论宽度如何，都能为具有梯度流动的双层泄漏的Relu网建立这种全局最优性。分析还为最近的经验研究结果（Kalimeris等，2019）给出了一些理论上的理由，就GD的所谓简单的偏见为线性或其他“简单”的解决方案，特别是在训练中。在悲观方面，该论文表明这种结果是脆弱的。简单的数据操作可以使梯度流量会聚到具有次优裕度的线性分类器。

The fundamental learning theory behind neural networks remains largely open. What classes of functions can neural networks actually learn? Why doesn't the trained network overfit when it is overparameterized?In this work, we prove that overparameterized neural networks can learn some notable concept classes, including two and three-layer networks with fewer parameters and smooth activations. Moreover, the learning can be simply done by SGD (stochastic gradient descent) or its variants in polynomial time using polynomially many samples. The sample complexity can also be almost independent of the number of parameters in the network.On the technique side, our analysis goes beyond the so-called NTK (neural tangent kernel) linearization of neural networks in prior works. We establish a new notion of quadratic approximation of the neural network (that can be viewed as a second-order variant of NTK), and connect it to the SGD theory of escaping saddle points.

过度分辨率是指选择神经网络的宽度，使得学习算法可以在非凸训练中可被估计零损失的重要现象。现有理论建立了各种初始化策略，培训修改和宽度缩放等全局融合。特别地，最先进的结果要求宽度以二次逐步缩放，并在实践中使用的标准初始化策略下进行培训数据的数量，以获得最佳泛化性能。相比之下，最新的结果可以获得线性缩放，需要导致导致“懒惰训练”的初始化，或者仅训练单层。在这项工作中，我们提供了一个分析框架，使我们能够采用标准的初始化策略，可能避免懒惰的训练，并在基本浅色神经网络中同时培训所有层，同时获得网络宽度的理想子标缩放。我们通过Polyak-Lojasiewicz条件，平滑度和数据标准假设实现了Desiderata，并使用随机矩阵理论的工具。

We study the training and generalization of deep neural networks (DNNs) in the overparameterized regime, where the network width (i.e., number of hidden nodes per layer) is much larger than the number of training data points. We show that, the expected 0-1 loss of a wide enough ReLU network trained with stochastic gradient descent (SGD) and random initialization can be bounded by the training loss of a random feature model induced by the network gradient at initialization, which we call a neural tangent random feature (NTRF) model. For data distributions that can be classified by NTRF model with sufficiently small error, our result yields a generalization error bound in the order of r Opn ´1{2 q that is independent of the network width. Our result is more general and sharper than many existing generalization error bounds for over-parameterized neural networks. In addition, we establish a strong connection between our generalization error bound and the neural tangent kernel (NTK) proposed in recent work.

This work studies training one-hidden-layer overparameterized ReLU networks via gradient descent in the neural tangent kernel (NTK) regime, where, differently from the previous works, the networks' biases are trainable and are initialized to some constant rather than zero. The first set of results of this work characterize the convergence of the network's gradient descent dynamics. Surprisingly, it is shown that the network after sparsification can achieve as fast convergence as the original network. The contribution over previous work is that not only the bias is allowed to be updated by gradient descent under our setting but also a finer analysis is given such that the required width to ensure the network's closeness to its NTK is improved. Secondly, the networks' generalization bound after training is provided. A width-sparsity dependence is presented which yields sparsity-dependent localized Rademacher complexity and a generalization bound matching previous analysis (up to logarithmic factors). As a by-product, if the bias initialization is chosen to be zero, the width requirement improves the previous bound for the shallow networks' generalization. Lastly, since the generalization bound has dependence on the smallest eigenvalue of the limiting NTK and the bounds from previous works yield vacuous generalization, this work further studies the least eigenvalue of the limiting NTK. Surprisingly, while it is not shown that trainable biases are necessary, trainable bias helps to identify a nice data-dependent region where a much finer analysis of the NTK's smallest eigenvalue can be conducted, which leads to a much sharper lower bound than the previously known worst-case bound and, consequently, a non-vacuous generalization bound.

重要的理论工作已经确定，在特定的制度中，通过梯度下降训练的神经网络像内核方法一样行为。但是，在实践中，众所周知，神经网络非常优于其相关内核。在这项工作中，我们通过证明有一大批功能可以通过内核方法有效地学习，但是可以通过学习表示与相关的学习表示，可以轻松地学习这一差距。到目标任务。我们还证明了这些表示允许有效的转移学习，这在内核制度中是不可能的。具体而言，我们考虑学习多项式的问题，该问题仅取决于少数相关的方向，即$ f^\ star（x）= g（ux）$ withy $ u：\ r^d \ to \ r^r $ d \ gg r $。当$ f^\ star $的度数为$ p $时，众所周知，在内核制度中学习$ f^\ star $是必要的。我们的主要结果是，梯度下降学会了数据的表示，这仅取决于与$ f^\ star $相关的指示。这导致改进的样本复杂性为$ n \ asymp d^2 r + dr^p $。此外，在转移学习设置中，源和目标域中的数据分布共享相同的表示$ u $，但具有不同的多项式头部，我们表明，转移学习的流行启发式启发式启发式具有目标样本复杂性，独立于$ d $。

最近的发现（例如ARXIV：2103.00065）表明，通过全批梯度下降训练的现代神经网络通常进入一个称为稳定边缘（EOS）的政权。在此制度中，清晰度（即最大的Hessian特征值）首先增加到值2/（步长尺寸）（渐进锐化阶段），然后在该值（EOS相）周围振荡。本文旨在分析沿优化轨迹的GD动力学和清晰度。我们的分析自然将GD轨迹分为四个阶段，具体取决于清晰度的变化。从经验上，我们将输出层重量的规范视为清晰动力学的有趣指标。基于这一经验观察，我们尝试从理论和经验上解释导致EOS每个阶段清晰度变化的各种关键量的动力学。此外，基于某些假设，我们提供了两层完全连接的线性神经网络中EOS制度的清晰度行为的理论证明。我们还讨论了其他一些经验发现以及我们的理论结果的局限性。

最近关于深度学习的研究侧重于极端过度参数化的设置，并表明，当网络宽度大于训练样本大小的高度多项式$ N $和目标错误$ \ epsilon ^ {-1} $，由（随机）梯度下降学习的深度神经网络享受很好的优化和泛化保证。最近，表明，在训练数据的某些边缘假设下，PolyGarithic宽度条件足以使两层Relu网络收敛和概括（Ji和Telgarsky，2019）。但是，是否可以通过这种轻度过度参数化学习深度神经网络仍然是一个开放的问题。在这项工作中，我们肯定地回答了这个问题，并建立了由（随机）梯度下降所培训的深度Relu网络的更尖锐的学习保证。具体而言，在以前的工作中的某些假设下，我们的优化和泛化保证以$ N $和$ \ epsilon ^ { - 1} $持有网络宽度波动力算法。我们的结果推动了对更实际的环境的过度参数化深神经网络的研究。

我们证明了由例如He等人提出的广泛使用的方法。（2015年）并使用梯度下降对最小二乘损失进行训练并不普遍。具体而言，我们描述了一大批一维数据生成分布，较高的概率下降只会发现优化景观的局部最小值不好，因为它无法将其偏离偏差远离其初始化，以零移动。。事实证明，在这些情况下，即使目标函数是非线性的，发现的网络也基本执行线性回归。我们进一步提供了数值证据，表明在实际情况下，对于某些多维分布而发生这种情况，并且随机梯度下降表现出相似的行为。我们还提供了有关初始化和优化器的选择如何影响这种行为的经验结果。

We explore the ability of overparameterized shallow ReLU neural networks to learn Lipschitz, non-differentiable, bounded functions with additive noise when trained by Gradient Descent (GD). To avoid the problem that in the presence of noise, neural networks trained to nearly zero training error are inconsistent in this class, we focus on the early-stopped GD which allows us to show consistency and optimal rates. In particular, we explore this problem from the viewpoint of the Neural Tangent Kernel (NTK) approximation of a GD-trained finite-width neural network. We show that whenever some early stopping rule is guaranteed to give an optimal rate (of excess risk) on the Hilbert space of the kernel induced by the ReLU activation function, the same rule can be used to achieve minimax optimal rate for learning on the class of considered Lipschitz functions by neural networks. We discuss several data-free and data-dependent practically appealing stopping rules that yield optimal rates.

鉴于密集的浅色神经网络，我们专注于迭代创建，培训和组合随机选择的子网（代理函数），以训练完整模型。通过仔细分析$ i）$ Subnetworks的神经切线内核，II美元）$代理职能'梯度，以及$ iii）$我们如何对替代品函数进行采样并结合训练错误的线性收敛速度 - 内部一个错误区域 - 对于带有回归任务的Relu激活的过度参数化单隐藏层Perceptron。我们的结果意味着，对于固定的神经元选择概率，当我们增加代理模型的数量时，误差项会减少，并且随着我们增加每个所选子网的本地训练步骤的数量而增加。考虑的框架概括并提供了关于辍学培训，多样化辍学培训以及独立的子网培训的新见解;对于每种情况，我们提供相应的收敛结果，作为我们主要定理的冠状动脉。

我们重新审视GD的平均算法稳定性，用于训练过度的浅色神经网络，并证明没有NTK或PL假设的新的泛化和过度的风险范围。特别是，我们显示Oracle类型的界限，揭示了GD的泛化和过度风险由具有最短GD路径的插值网络从初始化（从某种意义上是具有最小相对规范的内插网络）来控制。虽然这是封闭式嵌入式嵌入式的，但我们的证据直接适用于GD培训的网络，而无需中间结石。与此同时，通过在这里开发的放松Oracle不等式，我们以简单的方式恢复基于NTK的风险范围，这表明我们的分析更加紧张。最后，与大多数基于NTK的分析不同，我们专注于带标签噪声的回归，并显示早期停止的GD是一致的。

How well does a classic deep net architecture like AlexNet or VGG19 classify on a standard dataset such as CIFAR-10 when its "width"-namely, number of channels in convolutional layers, and number of nodes in fully-connected internal layers -is allowed to increase to infinity? Such questions have come to the forefront in the quest to theoretically understand deep learning and its mysteries about optimization and generalization. They also connect deep learning to notions such as Gaussian processes and kernels. A recent paper [Jacot et al., 2018] introduced the Neural Tangent Kernel (NTK) which captures the behavior of fully-connected deep nets in the infinite width limit trained by gradient descent; this object was implicit in some other recent papers. An attraction of such ideas is that a pure kernel-based method is used to capture the power of a fully-trained deep net of infinite width. The current paper gives the first efficient exact algorithm for computing the extension of NTK to convolutional neural nets, which we call Convolutional NTK (CNTK), as well as an efficient GPU implementation of this algorithm. This results in a significant new benchmark for performance of a pure kernel-based method on CIFAR-10, being 10% higher than the methods reported in [Novak et al., 2019], and only 6% lower than the performance of the corresponding finite deep net architecture (once batch normalization etc. are turned off). Theoretically, we also give the first non-asymptotic proof showing that a fully-trained sufficiently wide net is indeed equivalent to the kernel regression predictor using NTK.