神经网络的架构和参数通常独立优化，这需要每当修改体系结构时对参数的昂贵再次再次再次进行验证。在这项工作中，我们专注于在不需要昂贵的再培训的情况下越来越多。我们提出了一种在训练期间添加新神经元的方法，而不会影响已经学到的内容，同时改善了培训动态。我们通过最大化新重量的梯度来实现后者，并通过奇异值分解（SVD）有效地找到最佳初始化。我们称这种技术渐变最大化增长（Gradmax），并展示其各种视觉任务和架构的效力。

Many applications require sparse neural networks due to space or inference time restrictions. There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-tosparse training methods. Our method updates the topology of the sparse network during training by using parameter magnitudes and infrequent gradient calculations. We show that this approach requires fewer floating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques. We demonstrate state-of-the-art sparse training results on a variety of networks and datasets, including ResNet-50, MobileNets on Imagenet-2012, and RNNs on WikiText-103. Finally, we provide some insights into why allowing the topology to change during the optimization can overcome local minima encountered when the topology remains static * .

神经架构的创新促进了语言建模和计算机视觉中的重大突破。不幸的是，如果网络参数未正确初始化，新颖的架构通常会导致挑战超参数选择和培训不稳定。已经提出了许多架构特定的初始化方案，但这些方案并不总是可移植到新体系结构。本文介绍了毕业，一种用于初始化神经网络的自动化和架构不可知论由方法。毕业基础是一个简单的启发式;调整每个网络层的规范，使得具有规定的超参数的SGD或ADAM的单个步骤导致可能的损耗值最小。通过在每个参数块前面引入标量乘数变量，然后使用简单的数字方案优化这些变量来完成此调整。 GradInit加速了许多卷积架构的收敛性和测试性能，无论是否有跳过连接，甚至没有归一化层。它还提高了机器翻译的原始变压器架构的稳定性，使得在广泛的学习速率和动量系数下使用ADAM或SGD来训练它而无需学习速率预热。代码可在https://github.com/zhuchen03/gradinit上获得。

Structural pruning of neural network parameters reduces computation, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the first and secondorder Taylor expansions to approximate a filter's contribution. Both methods scale consistently across any network layer without requiring per-layer sensitivity analysis and can be applied to any kind of layer, including skip connections. For modern networks trained on ImageNet, we measured experimentally a high (>93%) correlation between the contribution computed by our methods and a reliable estimate of the true importance. Pruning with the proposed methods leads to an improvement over state-ofthe-art in terms of accuracy, FLOPs, and parameter reduction. On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0.02% in the top-1 accuracy on ImageNet. Code is available at https://github.com/NVlabs/Taylor_pruning.

深度学习使用由其重量进行参数化的神经网络。通常通过调谐重量来直接最小化给定损耗功能来训练神经网络。在本文中，我们建议将权重重新参数转化为网络中各个节点的触发强度的目标。给定一组目标，可以计算使得发射强度最佳地满足这些目标的权重。有人认为，通过我们称之为级联解压缩的过程，使用培训的目标解决爆炸梯度的问题，并使损失功能表面更加光滑，因此导致更容易，培训更快，以及潜在的概括，神经网络。它还允许更容易地学习更深层次和经常性的网络结构。目标对重量的必要转换有额外的计算费用，这是在许多情况下可管理的。在目标空间中学习可以与现有的神经网络优化器相结合，以额外收益。实验结果表明了使用目标空间的速度，以及改进的泛化的示例，用于全连接的网络和卷积网络，以及调用和处理长时间序列的能力，并使用经常性网络进行自然语言处理。

由于稀疏神经网络通常包含许多零权重，因此可以在不降低网络性能的情况下潜在地消除这些不必要的网络连接。因此，设计良好的稀疏神经网络具有显着降低拖鞋和计算资源的潜力。在这项工作中，我们提出了一种新的自动修剪方法 - 稀疏连接学习（SCL）。具体地，重量被重新参数化为可培训权重变量和二进制掩模的元素方向乘法。因此，由二进制掩模完全描述网络连接，其由单位步进函数调制。理论上，从理论上证明了使用直通估计器（STE）进行网络修剪的基本原理。这一原则是STE的代理梯度应该是积极的，确保掩模变量在其最小值处收敛。在找到泄漏的Relu后，SoftPlus和Identity Stes可以满足这个原理，我们建议采用SCL的身份STE以进行离散面膜松弛。我们发现不同特征的面具梯度非常不平衡，因此，我们建议将每个特征的掩模梯度标准化以优化掩码变量训练。为了自动训练稀疏掩码，我们将网络连接总数作为我们的客观函数中的正则化术语。由于SCL不需要由网络层设计人员定义的修剪标准或超级参数，因此在更大的假设空间中探讨了网络，以实现最佳性能的优化稀疏连接。 SCL克服了现有自动修剪方法的局限性。实验结果表明，SCL可以自动学习并选择各种基线网络结构的重要网络连接。 SCL培训的深度学习模型以稀疏性，精度和减少脚波特的SOTA人类设计和自动修剪方法训练。

While machine learning is traditionally a resource intensive task, embedded systems, autonomous navigation, and the vision of the Internet of Things fuel the interest in resource-efficient approaches. These approaches aim for a carefully chosen trade-off between performance and resource consumption in terms of computation and energy. The development of such approaches is among the major challenges in current machine learning research and key to ensure a smooth transition of machine learning technology from a scientific environment with virtually unlimited computing resources into everyday's applications. In this article, we provide an overview of the current state of the art of machine learning techniques facilitating these real-world requirements. In particular, we focus on deep neural networks (DNNs), the predominant machine learning models of the past decade. We give a comprehensive overview of the vast literature that can be mainly split into three non-mutually exclusive categories: (i) quantized neural networks, (ii) network pruning, and (iii) structural efficiency. These techniques can be applied during training or as post-processing, and they are widely used to reduce the computational demands in terms of memory footprint, inference speed, and energy efficiency. We also briefly discuss different concepts of embedded hardware for DNNs and their compatibility with machine learning techniques as well as potential for energy and latency reduction. We substantiate our discussion with experiments on well-known benchmark datasets using compression techniques (quantization, pruning) for a set of resource-constrained embedded systems, such as CPUs, GPUs and FPGAs. The obtained results highlight the difficulty of finding good trade-offs between resource efficiency and predictive performance.

Low-rankness plays an important role in traditional machine learning, but is not so popular in deep learning. Most previous low-rank network compression methods compress the networks by approximating pre-trained models and re-training. However, the optimal solution in the Euclidean space may be quite different from the one in the low-rank manifold. A well-pre-trained model is not a good initialization for the model with low-rank constraints. Thus, the performance of a low-rank compressed network degrades significantly. Compared to other network compression methods such as pruning, low-rank methods attracts less attention in recent years. In this paper, we devise a new training method, low-rank projection with energy transfer (LRPET), that trains low-rank compressed networks from scratch and achieves competitive performance. First, we propose to alternately perform stochastic gradient descent training and projection onto the low-rank manifold. Compared to re-training on the compact model, this enables full utilization of model capacity since solution space is relaxed back to Euclidean space after projection. Second, the matrix energy (the sum of squares of singular values) reduction caused by projection is compensated by energy transfer. We uniformly transfer the energy of the pruned singular values to the remaining ones. We theoretically show that energy transfer eases the trend of gradient vanishing caused by projection. Third, we propose batch normalization (BN) rectification to cut off its effect on the optimal low-rank approximation of the weight matrix, which further improves the performance. Comprehensive experiments on CIFAR-10 and ImageNet have justified that our method is superior to other low-rank compression methods and also outperforms recent state-of-the-art pruning methods. Our code is available at https://github.com/BZQLin/LRPET.

使用DataSet的真实标签培训而不是随机标签导致更快的优化和更好的泛化。这种差异归因于自然数据集中的输入和标签之间的对齐概念。我们发现，随机或真正标签上的具有不同架构和优化器的培训神经网络在隐藏的表示和训练标签之间强制执行相同的关系，阐明为什么神经网络表示为转移如此成功。我们首先突出显示为什么对齐的特征在经典的合成转移问题中促进转移和展示，即对齐是对相似和不同意任务的正负传输的确定因素。然后我们调查各种神经网络架构，并发现（a）在各种不同的架构和优化器中出现的对齐，并且从深度（b）对准产生的更多对准对于更接近输出的层和（c）现有的性能深度CNN表现出高级别的对准。

Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the lottery ticket hypothesis: dense, randomly-initialized, feed-forward networks contain subnetworks (winning tickets) that-when trained in isolationreach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

培训深度神经网络是一项非常苛刻的任务，尤其是具有挑战性的是如何适应体系结构以提高训练有素的模型的性能。我们可以发现，有时，浅网络比深网概括得更好，并且增加更多层会导致更高的培训和测试错误。深层残留学习框架通过将跳过连接添加到几个神经网络层来解决此降解问题。最初，需要这种跳过连接才能成功地训练深层网络，因为网络的表达性会随着深度的指数增长而成功。在本文中，我们首先通过神经网络分析信息流。我们介绍和评估批处理循环，该批处理通过神经网络的每一层量化信息流。我们从经验和理论上证明，基于梯度下降的训练方法需要正面批处理融合，以成功地优化给定的损失功能。基于这些见解，我们引入了批处理凝聚正则化，以使基于梯度下降的训练算法能够单独通过每个隐藏层来优化信息流。借助批处理正则化，梯度下降优化器可以将不可吸引的网络转换为可训练的网络。我们从经验上表明，因此我们可以训练“香草”完全连接的网络和卷积神经网络 - 没有跳过连接，批处理标准化，辍学或任何其他建筑调整 - 只需将批处理 - 凝集正则术语添加到500层中损失功能。批处理 - 注入正则化的效果不仅在香草神经网络上评估，还评估了在各种计算机视觉以及自然语言处理任务上的剩余网络，自动编码器以及变压器模型上。

深度学习归一化技术的基本特性，例如批准归一化，正在使范围前的参数量表不变。此类参数的固有域是单位球，因此可以通过球形优化的梯度优化动力学以不同的有效学习率（ELR）来表示，这是先前研究的。在这项工作中，我们使用固定的ELR直接研究了训练量表不变的神经网络的特性。我们根据ELR值发现了这种训练的三个方案：收敛，混乱平衡和差异。我们详细研究了这些制度示例的理论检查，以及对真实规模不变深度学习模型的彻底经验分析。每个制度都有独特的特征，并反映了内在损失格局的特定特性，其中一些与先前对常规和规模不变的神经网络培训的研究相似。最后，我们证明了如何在归一化网络的常规培训以及如何利用它们以实现更好的Optima中反映发现的制度。

神经网络修剪对于在预训练的密集网络架构中发现有效，高性能的子网有用。然而，更常见的是，它涉及三步过程 - 预先训练，修剪和重新训练 - 这是计算昂贵的，因为必须完全预先训练的密集模型。幸运的是，已经经过了多种作品，证明可以通过修剪发现高性能的子网，而无需完全预先训练密集网络。旨在理论上分析修剪网络表现良好的密集网络预培训量，我们发现在两层全连接网络上的SGD预训练迭代数量中发现了一个理论界限，超出了由此进行修剪贪婪的前瞻性选择产生了一个达到良好训练错误的子网。该阈值显示在对数上依赖于数据集的大小，这意味着具有较大数据集的实验需要更好地训练通过修剪以执行良好执行的子网。我们经验展示了我们在各种架构和数据集中的理论结果的有效性，包括在Mnist上培训的全连接网络以及在CIFAR10和ImageNet上培训的几个深度卷积神经网络（CNN）架构。

基于网络形态的神经体系结构搜索（NAS）是最有效的方法之一，但是，知道何时何地添加新的神经元或删除非功能功能的方法通常留给黑盒增强学习模型。在本文中，我们提出了一种新的基于网络形态的NAS，称为NOISY启发式NAS，该NAS使用了从手动开发神经网络模型中学到的启发式方法，并受到生物神经元动力学的启发。首先，我们随机添加新的神经元，并修剪一些神经元，以选择最佳的合身神经元。其次，我们使用隐藏单元与输入输出连接数的关系控制网络中的层数。我们的方法可以在线增加或降低模型的容量或非线性，该模型由用户指定了一些元参数。我们的方法在玩具数据集以及MNIST，CIFAR-10和CIFAR-100等实际数据集上概括了。性能与具有相似参数的手工设计架构Resnet-18相当。

Batch Normalization (BatchNorm) is a widely adopted technique that enables faster and more stable training of deep neural networks (DNNs). Despite its pervasiveness, the exact reasons for BatchNorm's effectiveness are still poorly understood. The popular belief is that this effectiveness stems from controlling the change of the layers' input distributions during training to reduce the so-called "internal covariate shift". In this work, we demonstrate that such distributional stability of layer inputs has little to do with the success of BatchNorm. Instead, we uncover a more fundamental impact of BatchNorm on the training process: it makes the optimization landscape significantly smoother. This smoothness induces a more predictive and stable behavior of the gradients, allowing for faster training.

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batchnormalized networks, we improve upon the best published result on ImageNet classification: reaching 4.9% top-5 validation error (and 4.8% test error), exceeding the accuracy of human raters.

深度学习文献通过新的架构和培训技术不断更新。然而，尽管有一些关于随机权重的发现，但最近的研究却忽略了重量初始化。另一方面，最近的作品一直在接近网络科学，以了解训练后人工神经网络（ANN）的结构和动态。因此，在这项工作中，我们分析了随机初始化网络中神经元的中心性。我们表明，较高的神经元强度方差可能会降低性能，而较低的神经元强度方差通常会改善它。然后，提出了一种新方法，根据其强度根据优先附着（PA）规则重新连接神经元连接，从而大大降低了通过常见方法初始化的层的强度方差。从这个意义上讲，重新布线仅重新组织连接，同时保留权重的大小和分布。我们通过对图像分类进行的广泛统计分析表明，在使用简单和复杂的体系结构和学习时间表时，在大多数情况下，在培训和测试过程中，性能都会提高。我们的结果表明，除了规模外，权重的组织也与更好的初始化初始化有关。

最近的作品研究了在神经切线内核（NTK）制度中训练的广泛神经网络的理论和经验特性。鉴于生物神经网络比其人工对应物宽得多，因此我们认为NTK范围广泛的神经网络是生物神经网络的可能模型。利用NTK理论，我们从理论上说明梯度下降驱动层的重量更新与其输入活动相关性一致，并通过误差加权，并从经验上证明了结果在有限宽度的宽网络中也存在。对齐结果使我们能够制定一个生物动机的，无反向传播的学习规则，理论上等同于无限宽度网络中的反向传播。我们测试了馈电和经常性神经网络中基准问题的这些学习规则，并在宽网络中证明了与反向传播相当的性能。所提出的规则在低数据制度中特别有效，这在生物学习环境中很常见。

在许多情况下，更简单的模型比更复杂的模型更可取，并且该模型复杂性的控制是机器学习中许多方法的目标，例如正则化，高参数调整和体系结构设计。在深度学习中，很难理解复杂性控制的潜在机制，因为许多传统措施并不适合深度神经网络。在这里，我们开发了几何复杂性的概念，该概念是使用离散的dirichlet能量计算的模型函数变异性的量度。使用理论论据和经验结果的结合，我们表明，许多常见的训练启发式方法，例如参数规范正规化，光谱规范正则化，平稳性正则化，隐式梯度正则化，噪声正则化和参数初始化的选择，都可以控制几何学复杂性，并提供一个统一的框架，以表征深度学习模型的行为。

我们研究了使用尖刺，现场依赖的随机矩阵理论研究迷你批次对深神经网络损失景观的影响。我们表明，批量黑森州的极值值的大小大于经验丰富的黑森州。我们还获得了类似的结果对Hessian的概括高斯牛顿矩阵近似。由于我们的定理，我们推导出作为批量大小的最大学习速率的分析表达式，为随机梯度下降（线性缩放）和自适应算法（例如ADAM（Square Root Scaling）提供了通知实际培训方案，例如光滑，非凸深神经网络。虽然随机梯度下降的线性缩放是在我们概括的更多限制性条件下导出的，但是适应优化者的平方根缩放规则是我们的知识，完全小说。随机二阶方法和自适应方法的百分比，我们得出了最小阻尼系数与学习率与批量尺寸的比率成比例。我们在Cifar-$ 100 $和ImageNet数据集上验证了我们的VGG / WimerEsnet架构上的索赔。根据我们对象检的调查，我们基于飞行学习率和动量学习者开发了一个随机兰齐齐竞争，这避免了对这些关键的超参数进行昂贵的多重评估的需求，并在预残留的情况下显示出良好的初步结果Cifar的architecure - $ 100 $。