基于网络形态的神经体系结构搜索（NAS）是最有效的方法之一，但是，知道何时何地添加新的神经元或删除非功能功能的方法通常留给黑盒增强学习模型。在本文中，我们提出了一种新的基于网络形态的NAS，称为NOISY启发式NAS，该NAS使用了从手动开发神经网络模型中学到的启发式方法，并受到生物神经元动力学的启发。首先，我们随机添加新的神经元，并修剪一些神经元，以选择最佳的合身神经元。其次，我们使用隐藏单元与输入输出连接数的关系控制网络中的层数。我们的方法可以在线增加或降低模型的容量或非线性，该模型由用户指定了一些元参数。我们的方法在玩具数据集以及MNIST，CIFAR-10和CIFAR-100等实际数据集上概括了。性能与具有相似参数的手工设计架构Resnet-18相当。

神经网络的架构和参数通常独立优化，这需要每当修改体系结构时对参数的昂贵再次再次再次进行验证。在这项工作中，我们专注于在不需要昂贵的再培训的情况下越来越多。我们提出了一种在训练期间添加新神经元的方法，而不会影响已经学到的内容，同时改善了培训动态。我们通过最大化新重量的梯度来实现后者，并通过奇异值分解（SVD）有效地找到最佳初始化。我们称这种技术渐变最大化增长（Gradmax），并展示其各种视觉任务和架构的效力。

人们通常认为，修剪网络不仅会降低深网的计算成本，而且还可以通过降低模型容量来防止过度拟合。但是，我们的工作令人惊讶地发现，网络修剪有时甚至会加剧过度拟合。我们报告了出乎意料的稀疏双后裔现象，随着我们通过网络修剪增加模型稀疏性，首先测试性能变得更糟（由于过度拟合），然后变得更好（由于过度舒适），并且终于变得更糟（由于忘记了有用的有用信息）。尽管最近的研究集中在模型过度参数化方面，但他们未能意识到稀疏性也可能导致双重下降。在本文中，我们有三个主要贡献。首先，我们通过广泛的实验报告了新型的稀疏双重下降现象。其次，对于这种现象，我们提出了一种新颖的学习距离解释，即$ \ ell_ {2} $稀疏模型的学习距离（从初始化参数到最终参数）可能与稀疏的双重下降曲线良好相关，并更好地反映概括比最小平坦。第三，在稀疏的双重下降的背景下，彩票票假设中的获胜票令人惊讶地并不总是赢。

Structural pruning of neural network parameters reduces computation, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the first and secondorder Taylor expansions to approximate a filter's contribution. Both methods scale consistently across any network layer without requiring per-layer sensitivity analysis and can be applied to any kind of layer, including skip connections. For modern networks trained on ImageNet, we measured experimentally a high (>93%) correlation between the contribution computed by our methods and a reliable estimate of the true importance. Pruning with the proposed methods leads to an improvement over state-ofthe-art in terms of accuracy, FLOPs, and parameter reduction. On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0.02% in the top-1 accuracy on ImageNet. Code is available at https://github.com/NVlabs/Taylor_pruning.

私人推论（PI）可以直接对密码安全的数据进行推断。虽然有望解决许多隐私问题，但由于极端的运行时间，它的使用有限。与明文推断不同，在PI非线性函数（即relu）中，延迟是由拖曳支配的，即瓶颈。因此，实用的PI需要新颖的恢复优化。为了减少PI潜伏期，我们提出了一种基于梯度的算法，该算法在维持预测准确性的同时选择性地线性地线性性地线性性地线性性性地线性性地线性性地线性性性地线性性性地线性化。我们评估了几种标准PI基准测试的算法。结果表明，比目前的最新水平（70 \％的ISO-ACCURACY \％），最高$ 4.25 \％$的准确性（ISO-RELU计数为50K）或$ 2.2 \ tims $少于$ $ $ $。 - 准确空间。为了补充经验结果，我们提出了一个“无免费午餐”定理，该定理阐明了如何以及何时进行网络线性化，同时保持预测准确性。公共代码可在\ url {https://github.com/nyu-dice-lab/selective_network_linearization}获得。

Many applications require sparse neural networks due to space or inference time restrictions. There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-tosparse training methods. Our method updates the topology of the sparse network during training by using parameter magnitudes and infrequent gradient calculations. We show that this approach requires fewer floating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques. We demonstrate state-of-the-art sparse training results on a variety of networks and datasets, including ResNet-50, MobileNets on Imagenet-2012, and RNNs on WikiText-103. Finally, we provide some insights into why allowing the topology to change during the optimization can overcome local minima encountered when the topology remains static * .

Pruning refers to the elimination of trivial weights from neural networks. The sub-networks within an overparameterized model produced after pruning are often called Lottery tickets. This research aims to generate winning lottery tickets from a set of lottery tickets that can achieve similar accuracy to the original unpruned network. We introduce a novel winning ticket called Cyclic Overlapping Lottery Ticket (COLT) by data splitting and cyclic retraining of the pruned network from scratch. We apply a cyclic pruning algorithm that keeps only the overlapping weights of different pruned models trained on different data segments. Our results demonstrate that COLT can achieve similar accuracies (obtained by the unpruned model) while maintaining high sparsities. We show that the accuracy of COLT is on par with the winning tickets of Lottery Ticket Hypothesis (LTH) and, at times, is better. Moreover, COLTs can be generated using fewer iterations than tickets generated by the popular Iterative Magnitude Pruning (IMP) method. In addition, we also notice COLTs generated on large datasets can be transferred to small ones without compromising performance, demonstrating its generalizing capability. We conduct all our experiments on Cifar-10, Cifar-100 & TinyImageNet datasets and report superior performance than the state-of-the-art methods.

Sparse neural networks attract increasing interest as they exhibit comparable performance to their dense counterparts while being computationally efficient. Pruning the dense neural networks is among the most widely used methods to obtain a sparse neural network. Driven by the high training cost of such methods that can be unaffordable for a low-resource device, training sparse neural networks sparsely from scratch has recently gained attention. However, existing sparse training algorithms suffer from various issues, including poor performance in high sparsity scenarios, computing dense gradient information during training, or pure random topology search. In this paper, inspired by the evolution of the biological brain and the Hebbian learning theory, we present a new sparse training approach that evolves sparse neural networks according to the behavior of neurons in the network. Concretely, by exploiting the cosine similarity metric to measure the importance of the connections, our proposed method, Cosine similarity-based and Random Topology Exploration (CTRE), evolves the topology of sparse neural networks by adding the most important connections to the network without calculating dense gradient in the backward. We carried out different experiments on eight datasets, including tabular, image, and text datasets, and demonstrate that our proposed method outperforms several state-of-the-art sparse training algorithms in extremely sparse neural networks by a large gap. The implementation code is available on https://github.com/zahraatashgahi/CTRE

Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy. In this work, we make several surprising observations which contradict common beliefs. For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights. For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch. Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods. We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin, 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization. * Equal contribution. † Work done while visiting UC Berkeley.

修剪深度神经网络的现有方法专注于去除训练有素的网络的不必要参数，然后微调模型，找到恢复训练模型的初始性能的良好解决方案。与其他作品不同，我们的方法特别注意通过修剪神经元的压缩模型和推理计算时间的解决方案的质量。通过探索Hessian的光谱半径，所提出的算法通过探索Hessian的光谱半径来指示压缩模型的参数，这导致了更好地推广了未经看涨的数据。此外，该方法不适用于预先训练的网络，并同时执行训练和修剪。我们的结果表明，它改善了神经元压缩的最先进的结果。该方法能够在不同神经网络模型上实现具有小精度下降的非常小的网络。

当前的深神经网络（DNN）被过度参数化，并在推断每个任务期间使用其大多数神经元连接。然而，人的大脑开发了针对不同任务的专门区域，并通过其神经元连接的一小部分进行推断。我们提出了一种迭代修剪策略，引入了一个简单的重要性评分度量度量，该指标可以停用不重要的连接，解决DNN中的过度参数化并调节射击模式。目的是找到仍然能够以可比精度解决给定任务的最小连接，即更简单的子网。我们在MNIST上实现了LENET体系结构的可比性能，并且与CIFAR-10/100和Tiny-ImageNet上的VGG和Resnet架构的最先进算法相比，参数压缩的性能明显更高。我们的方法对于考虑到ADAM和SGD的两个不同优化器也表现良好。该算法并非旨在在考虑当前的硬件和软件实现时最小化失败，尽管与最新技术相比，该算法的性能合理。

我们为神经网络提出了一种新颖，结构化修剪算法 - 迭代，稀疏结构修剪算法，称为I-Spasp。从稀疏信号恢复的思想启发，I-Spasp通过迭代地识别网络内的较大的重要参数组（例如，滤波器或神经元），这些参数组大多数对修剪和密集网络输出之间的残差贡献，然后基于这些组阈值以较小的预定定义修剪比率。对于具有Relu激活的双层和多层网络架构，我们展示了通过多项式修剪修剪诱导的错误，该衰减是基于密集网络隐藏表示的稀疏性任意大的。在我们的实验中，I-Spasp在各种数据集（即MNIST和ImageNet）和架构（即馈送前向网络，Resnet34和MobileNetv2）中进行评估，其中显示用于发现高性能的子网和改进经过几种数量级的可提供基线方法的修剪效率。简而言之，I-Spasp很容易通过自动分化实现，实现强大的经验结果，具有理论收敛保证，并且是高效的，因此将自己区分开作为少数几个计算有效，实用，实用，实用，实用，实用，实用，实用，实用和可提供的修剪算法之一。

Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task.

We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms. Our approach uses a sequential model-based optimization (SMBO) strategy, in which we search for structures in order of increasing complexity, while simultaneously learning a surrogate model to guide the search through structure space. Direct comparison under the same search space shows that our method is up to 5 times more efficient than the RL method of Zoph et al. (2018) in terms of number of models evaluated, and 8 times faster in terms of total compute. The structures we discover in this way achieve state of the art classification accuracies on CIFAR-10 and ImageNet.

The success of CNNs in various applications is accompanied by a significant increase in the computation and parameter storage costs. Recent efforts toward reducing these overheads involve pruning and compressing the weights of various layers without hurting original accuracy. However, magnitude-based pruning of weights reduces a significant number of parameters from the fully connected layers and may not adequately reduce the computation costs in the convolutional layers due to irregular sparsity in the pruned networks. We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications. We show that even simple filter pruning techniques can reduce inference costs for VGG-16 by up to 34% and ResNet-110 by up to 38% on CIFAR10 while regaining close to the original accuracy by retraining the networks.

网络修剪是一种广泛使用的技术，用于有效地压缩深神经网络，几乎没有在推理期间在性能下降低。迭代幅度修剪（IMP）是由几种迭代训练和修剪步骤组成的网络修剪的最熟悉的方法之一，其中在修剪后丢失了大量网络的性能，然后在随后的再培训阶段中恢复。虽然常用为基准参考，但经常认为a）通过不将稀疏纳入训练阶段来达到次优状态，b）其全球选择标准未能正确地确定最佳层面修剪速率和c）其迭代性质使它变得缓慢和不竞争。根据最近提出的再培训技术，我们通过严格和一致的实验来调查这些索赔，我们将Impr到培训期间的训练算法进行比较，评估其选择标准的建议修改，并研究实际需要的迭代次数和总培训时间。我们发现IMP与SLR进行再培训，可以优于最先进的修剪期间，没有或仅具有很少的计算开销，即全局幅度选择标准在很大程度上具有更复杂的方法，并且只有几个刷新时期在实践中需要达到大部分稀疏性与IMP的诽谤 - 与性能权衡。我们的目标既可以证明基本的进攻已经可以提供最先进的修剪结果，甚至优于更加复杂或大量参数化方法，也可以为未来的研究建立更加现实但易于可实现的基线。

现代深度神经网络往往太大而无法在许多实际情况下使用。神经网络修剪是降低这种模型的大小的重要技术和加速推断。Gibbs修剪是一种表达和设计神经网络修剪方法的新框架。结合统计物理和随机正则化方法的方法，它可以同时培训和修剪网络，使得学习的权重和修剪面膜彼此很好地适应。它可用于结构化或非结构化修剪，我们为每个提出了许多特定方法。我们将拟议的方法与许多当代神经网络修剪方法进行比较，发现Gibbs修剪优于它们。特别是，我们通过CIFAR-10数据集来实现修剪Reset-56的新型最先进的结果。

This work targets designing a principled and unified training-free framework for Neural Architecture Search (NAS), with high performance, low cost, and in-depth interpretation. NAS has been explosively studied to automate the discovery of top-performer neural networks, but suffers from heavy resource consumption and often incurs search bias due to truncated training or approximations. Recent NAS works start to explore indicators that can predict a network's performance without training. However, they either leveraged limited properties of deep networks, or the benefits of their training-free indicators are not applied to more extensive search methods. By rigorous correlation analysis, we present a unified framework to understand and accelerate NAS, by disentangling "TEG" characteristics of searched networks - Trainability, Expressivity, Generalization - all assessed in a training-free manner. The TEG indicators could be scaled up and integrated with various NAS search methods, including both supernet and single-path approaches. Extensive studies validate the effective and efficient guidance from our TEG-NAS framework, leading to both improved search accuracy and over 56% reduction in search time cost. Moreover, we visualize search trajectories on three landscapes of "TEG" characteristics, observing that while a good local minimum is easier to find on NAS-Bench-201 given its simple topology, balancing "TEG" characteristics is much harder on the DARTS search space due to its complex landscape geometry. Our code is available at https://github.com/VITA-Group/TEGNAS.

深度学习技术在各种任务中都表现出了出色的有效性，并且深度学习具有推进多种应用程序（包括在边缘计算中）的潜力，其中将深层模型部署在边缘设备上，以实现即时的数据处理和响应。一个关键的挑战是，虽然深层模型的应用通常会产生大量的内存和计算成本，但Edge设备通常只提供非常有限的存储和计算功能，这些功能可能会在各个设备之间差异很大。这些特征使得难以构建深度学习解决方案，以释放边缘设备的潜力，同时遵守其约束。应对这一挑战的一种有希望的方法是自动化有效的深度学习模型的设计，这些模型轻巧，仅需少量存储，并且仅产生低计算开销。该调查提供了针对边缘计算的深度学习模型设计自动化技术的全面覆盖。它提供了关键指标的概述和比较，这些指标通常用于量化模型在有效性，轻度和计算成本方面的水平。然后，该调查涵盖了深层设计自动化技术的三类最新技术：自动化神经体系结构搜索，自动化模型压缩以及联合自动化设计和压缩。最后，调查涵盖了未来研究的开放问题和方向。

我们提出了三种新型的修剪技术，以提高推理意识到的可区分神经结构搜索（DNAS）的成本和结果。首先，我们介绍了DNA的随机双路构建块，它可以通过内存和计算复杂性在内部隐藏尺寸上进行搜索。其次，我们在搜索过程中提出了一种在超级网的随机层中修剪块的算法。第三，我们描述了一种在搜索过程中修剪不必要的随机层的新技术。由搜索产生的优化模型称为Prunet，并在Imagenet Top-1图像分类精度的推理潜伏期中为NVIDIA V100建立了新的最先进的Pareto边界。将Prunet作为骨架还优于COCO对象检测任务的GPUNET和EFIDENENET，相对于平均平均精度（MAP）。