Structured channel pruning has been shown to significantly accelerate inference time for convolution neural networks (CNNs) on modern hardware, with a relatively minor loss of network accuracy. Recent works permanently zero these channels during training, which we observe to significantly hamper final accuracy, particularly as the fraction of the network being pruned increases. We propose Soft Masking for cost-constrained Channel Pruning (SMCP) to allow pruned channels to adaptively return to the network while simultaneously pruning towards a target cost constraint. By adding a soft mask re-parameterization of the weights and channel pruning from the perspective of removing input channels, we allow gradient updates to previously pruned channels and the opportunity for the channels to later return to the network. We then formulate input channel pruning as a global resource allocation problem. Our method outperforms prior works on both the ImageNet classification and PASCAL VOC detection datasets.

Structural pruning of neural network parameters reduces computation, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the first and secondorder Taylor expansions to approximate a filter's contribution. Both methods scale consistently across any network layer without requiring per-layer sensitivity analysis and can be applied to any kind of layer, including skip connections. For modern networks trained on ImageNet, we measured experimentally a high (>93%) correlation between the contribution computed by our methods and a reliable estimate of the true importance. Pruning with the proposed methods leads to an improvement over state-ofthe-art in terms of accuracy, FLOPs, and parameter reduction. On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0.02% in the top-1 accuracy on ImageNet. Code is available at https://github.com/NVlabs/Taylor_pruning.

最近对深神经网络（DNN）效率的重点已导致了模型压缩方法的重要工作，其中重量修剪是最受欢迎的方法之一。同时，有快速增长的计算支持，以有效地执行通过修剪获得的非结构化模型。但是，大多数现有的修剪方法最小化仅剩余权重的数量，即模型的大小，而不是针对推理时间进行优化。我们通过引入SPDY来解决这一差距，SPDY是一种新的压缩方法，该方法会自动确定层次的稀疏性目标，可以在给定系统上实现所需的推理速度，同时最大程度地减少准确性损失。 SPDY由两种新技术组成：第一个是一种有效的动态编程算法，用于求解一组给定的层敏感性得分，以解决加速约束的层压缩问题；第二个是一个局部搜索程序，用于确定准确的层敏感性得分。跨流行视觉和语言模型的实验表明，SPDY可以保证相对于现有策略的恢复较高的准确性，无论是一次性和逐步修剪方案，并且与大多数现有的修剪方法兼容。我们还将方法扩展到了最近实施的修剪任务，几乎没有数据，在该数据中，我们在修剪GPU支持的2：4稀疏模式时实现了最著名的准确性恢复。

我们考虑在具有挑战性的训练后环境中，深度神经网络（DNN）的模型压缩问题，在该设置中，我们将获得精确的训练模型，并且必须仅基于少量校准输入数据而无需任何重新培训即可压缩它。鉴于新兴软件和硬件支持通过加速修剪和/或量化压缩的模型，并且已经针对两种压缩方法独立提出了良好的表现解决方案，因此该问题已变得流行。在本文中，我们引入了一个新的压缩框架，该框架涵盖了统一环境中的重量修剪和量化，时间和空间效率高，并且在现有的后训练方法的实际性能上大大改善。在技​​术层面上，我们的方法基于[Lecun，Denker和Solla，1990年]在现代DNN的规模上的经典最佳脑外科医生（OBS）框架的第一个精确实现，我们进一步扩展到覆盖范围。重量量化。这是通过一系列可能具有独立利益的算法开发来实现的。从实际的角度来看，我们的实验结果表明，它可以在现有后训练方法的压缩 - 准确性权衡方面显着改善，并且甚至可以在训练后进行修剪和量化的准确共同应用。

Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy. In this work, we make several surprising observations which contradict common beliefs. For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights. For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch. Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods. We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin, 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization. * Equal contribution. † Work done while visiting UC Berkeley.

The deployment of deep convolutional neural networks (CNNs) in many real world applications is largely hindered by their high computational cost. In this paper, we propose a novel learning scheme for CNNs to simultaneously 1) reduce the model size; 2) decrease the run-time memory footprint; and 3) lower the number of computing operations, without compromising accuracy. This is achieved by enforcing channel-level sparsity in the network in a simple but effective way. Different from many existing approaches, the proposed method directly applies to modern CNN architectures, introduces minimum overhead to the training process, and requires no special software/hardware accelerators for the resulting models. We call our approach network slimming, which takes wide and large networks as input models, but during training insignificant channels are automatically identified and pruned afterwards, yielding thin and compact models with comparable accuracy. We empirically demonstrate the effectiveness of our approach with several state-of-the-art CNN models, including VGGNet, ResNet and DenseNet, on various image classification datasets. For VGGNet, a multi-pass version of network slimming gives a 20× reduction in model size and a 5× reduction in computing operations.

由于稀疏神经网络通常包含许多零权重，因此可以在不降低网络性能的情况下潜在地消除这些不必要的网络连接。因此，设计良好的稀疏神经网络具有显着降低拖鞋和计算资源的潜力。在这项工作中，我们提出了一种新的自动修剪方法 - 稀疏连接学习（SCL）。具体地，重量被重新参数化为可培训权重变量和二进制掩模的元素方向乘法。因此，由二进制掩模完全描述网络连接，其由单位步进函数调制。理论上，从理论上证明了使用直通估计器（STE）进行网络修剪的基本原理。这一原则是STE的代理梯度应该是积极的，确保掩模变量在其最小值处收敛。在找到泄漏的Relu后，SoftPlus和Identity Stes可以满足这个原理，我们建议采用SCL的身份STE以进行离散面膜松弛。我们发现不同特征的面具梯度非常不平衡，因此，我们建议将每个特征的掩模梯度标准化以优化掩码变量训练。为了自动训练稀疏掩码，我们将网络连接总数作为我们的客观函数中的正则化术语。由于SCL不需要由网络层设计人员定义的修剪标准或超级参数，因此在更大的假设空间中探讨了网络，以实现最佳性能的优化稀疏连接。 SCL克服了现有自动修剪方法的局限性。实验结果表明，SCL可以自动学习并选择各种基线网络结构的重要网络连接。 SCL培训的深度学习模型以稀疏性，精度和减少脚波特的SOTA人类设计和自动修剪方法训练。

深度学习技术在各种任务中都表现出了出色的有效性，并且深度学习具有推进多种应用程序（包括在边缘计算中）的潜力，其中将深层模型部署在边缘设备上，以实现即时的数据处理和响应。一个关键的挑战是，虽然深层模型的应用通常会产生大量的内存和计算成本，但Edge设备通常只提供非常有限的存储和计算功能，这些功能可能会在各个设备之间差异很大。这些特征使得难以构建深度学习解决方案，以释放边缘设备的潜力，同时遵守其约束。应对这一挑战的一种有希望的方法是自动化有效的深度学习模型的设计，这些模型轻巧，仅需少量存储，并且仅产生低计算开销。该调查提供了针对边缘计算的深度学习模型设计自动化技术的全面覆盖。它提供了关键指标的概述和比较，这些指标通常用于量化模型在有效性，轻度和计算成本方面的水平。然后，该调查涵盖了深层设计自动化技术的三类最新技术：自动化神经体系结构搜索，自动化模型压缩以及联合自动化设计和压缩。最后，调查涵盖了未来研究的开放问题和方向。

在过去几年中，神经网络的性能在越来越多的浮点操作（拖鞋）的成本上显着提高。但是，当计算资源有限时，更多的拖鞋可能是一个问题。作为解决这个问题的尝试，修剪过滤器是一种常见的解决方案，但大多数现有的修剪方法不有效地保持模型精度，因此需要大量的芬降时期。在本文中，我们提出了一种自动修剪方法，该方法学习保存的神经元以保持模型精度，同时将絮凝到预定目标。为了完成这项任务，我们介绍了一种可训练的瓶颈，只需要一个单一的单一时期，只需要一个数据集的25.6％（Cifar-10）或7.49％（ILSVRC2012）来了解哪些过滤器。在各种架构和数据集上的实验表明，该方法不仅可以在修剪后保持精度，而且在FineTuning之后也优越现有方法。我们在Reset-50上达到了52.00％的拖鞋，在ILSVRC2012上的灌溉后的前1个精度为47.51％，最先进的（SOTA）精度为76.63％。代码可用（链接匿名审核）。

Many applications require sparse neural networks due to space or inference time restrictions. There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-tosparse training methods. Our method updates the topology of the sparse network during training by using parameter magnitudes and infrequent gradient calculations. We show that this approach requires fewer floating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques. We demonstrate state-of-the-art sparse training results on a variety of networks and datasets, including ResNet-50, MobileNets on Imagenet-2012, and RNNs on WikiText-103. Finally, we provide some insights into why allowing the topology to change during the optimization can overcome local minima encountered when the topology remains static * .

While machine learning is traditionally a resource intensive task, embedded systems, autonomous navigation, and the vision of the Internet of Things fuel the interest in resource-efficient approaches. These approaches aim for a carefully chosen trade-off between performance and resource consumption in terms of computation and energy. The development of such approaches is among the major challenges in current machine learning research and key to ensure a smooth transition of machine learning technology from a scientific environment with virtually unlimited computing resources into everyday's applications. In this article, we provide an overview of the current state of the art of machine learning techniques facilitating these real-world requirements. In particular, we focus on deep neural networks (DNNs), the predominant machine learning models of the past decade. We give a comprehensive overview of the vast literature that can be mainly split into three non-mutually exclusive categories: (i) quantized neural networks, (ii) network pruning, and (iii) structural efficiency. These techniques can be applied during training or as post-processing, and they are widely used to reduce the computational demands in terms of memory footprint, inference speed, and energy efficiency. We also briefly discuss different concepts of embedded hardware for DNNs and their compatibility with machine learning techniques as well as potential for energy and latency reduction. We substantiate our discussion with experiments on well-known benchmark datasets using compression techniques (quantization, pruning) for a set of resource-constrained embedded systems, such as CPUs, GPUs and FPGAs. The obtained results highlight the difficulty of finding good trade-offs between resource efficiency and predictive performance.

We propose a new formulation for pruning convolutional kernels in neural networks to enable efficient inference. We interleave greedy criteria-based pruning with finetuning by backpropagation-a computationally efficient procedure that maintains good generalization in the pruned network. We propose a new criterion based on Taylor expansion that approximates the change in the cost function induced by pruning network parameters. We focus on transfer learning, where large pretrained networks are adapted to specialized tasks. The proposed criterion demonstrates superior performance compared to other criteria, e.g. the norm of kernel weights or feature map activation, for pruning large CNNs after adaptation to fine-grained classification tasks (Birds-200 and Flowers-102) relaying only on the first order gradient information. We also show that pruning can lead to more than 10× theoretical reduction in adapted 3D-convolutional filters with a small drop in accuracy in a recurrent gesture classifier. Finally, we show results for the largescale ImageNet dataset to emphasize the flexibility of our approach.

嵌入式和个人物联网设备由微控制器单元（MCU）供电，其极端资源稀缺是依赖于设备的深度学习推断的应用的主要障碍。与通常需要执行神经网络的内容相比，存储器，内存和计算能力较少的秩序，对网络架构上的严格结构约束并呼叫专业模型压缩方法。在这项工作中，我们为卷积神经网络提出了可分散的结构化网络修剪方法，它集成了模型的MCU特定的资源使用和参数重要性反馈，以获得高度压缩但准确的分类模型。我们的方法（a）提高了高达80倍的模型的关键资源使用; （b）在培训型号的同时迭代地修剪，导致没有开销甚至改善培训时间; （c）与现有MCU的特定方法相比，在比较多的时间内生产具有匹配或改进的资源使用的压缩模型。压缩模型可供下载。

动态模型修剪是最近的方向，其允许不同的子网络中的部署过程中每个输入采样的推断。然而，当前的动态方法依赖于学习的连续通道通过诱导稀疏性损失通过正则化门控。这一提法介绍了平衡不同损失的复杂性（如任务的损失，正规化损失）。此外，基于正则化方法缺乏透明的折衷选择超参数，实现计算的预算。我们的贡献是双重的：1）分离任务和修剪培训。 2）简单的超参数选择，使训练前FLOPS减少估计。在神经科学的赫布理论的启发：“神经元一起火一起丝”，我们提出来预测基于其上一层的活化层口罩方法K过滤器。我们提出的问题，因为自监督二元分类问题。每个掩模预测模块被训练以预测，如果对数似然在当前层中的每个过滤器属于前k激活的过滤器。值k被动态地估计基于使用热图的质量的新颖标准每个输入。我们发现在几个神经结构，如VGG，RESNET和MobileNet上CIFAR和ImageNet数据集实验。在CIFAR，我们得出了类似的精度SOTA方法有15％和24％以上FLOPS减少。同样，在ImageNet，我们达到的精度低下降高达13％的改善FLOPS减少。

The mainstream approach for filter pruning is usually either to force a hard-coded importance estimation upon a computation-heavy pretrained model to select "important" filters, or to impose a hyperparameter-sensitive sparse constraint on the loss objective to regularize the network training. In this paper, we present a novel filter pruning method, dubbed dynamic-coded filter fusion (DCFF), to derive compact CNNs in a computation-economical and regularization-free manner for efficient image classification. Each filter in our DCFF is firstly given an inter-similarity distribution with a temperature parameter as a filter proxy, on top of which, a fresh Kullback-Leibler divergence based dynamic-coded criterion is proposed to evaluate the filter importance. In contrast to simply keeping high-score filters in other methods, we propose the concept of filter fusion, i.e., the weighted averages using the assigned proxies, as our preserved filters. We obtain a one-hot inter-similarity distribution as the temperature parameter approaches infinity. Thus, the relative importance of each filter can vary along with the training of the compact CNN, leading to dynamically changeable fused filters without both the dependency on the pretrained model and the introduction of sparse constraints. Extensive experiments on classification benchmarks demonstrate the superiority of our DCFF over the compared counterparts. For example, our DCFF derives a compact VGGNet-16 with only 72.77M FLOPs and 1.06M parameters while reaching top-1 accuracy of 93.47% on CIFAR-10. A compact ResNet-50 is obtained with 63.8% FLOPs and 58.6% parameter reductions, retaining 75.60% top-1 accuracy on ILSVRC-2012. Our code, narrower models and training logs are available at https://github.com/lmbxmu/DCFF.

This paper proposed a Soft Filter Pruning (SFP) method to accelerate the inference procedure of deep Convolutional Neural Networks (CNNs). Specifically, the proposed SFP enables the pruned filters to be updated when training the model after pruning. SFP has two advantages over previous works: (1) Larger model capacity. Updating previously pruned filters provides our approach with larger optimization space than fixing the filters to zero. Therefore, the network trained by our method has a larger model capacity to learn from the training data. (2) Less dependence on the pretrained model. Large capacity enables SFP to train from scratch and prune the model simultaneously. In contrast, previous filter pruning methods should be conducted on the basis of the pre-trained model to guarantee their performance. Empirically, SFP from scratch outperforms the previous filter pruning methods. Moreover, our approach has been demonstrated effective for many advanced CNN architectures. Notably, on ILSCRC-2012, SFP reduces more than 42% FLOPs on ResNet-101 with even 0.2% top-5 accuracy improvement, which has advanced the state-of-the-art. Code is publicly available on GitHub: https://github.com/he-y/softfilter-pruning

During the deployment of deep neural networks (DNNs) on edge devices, many research efforts are devoted to the limited hardware resource. However, little attention is paid to the influence of dynamic power management. As edge devices typically only have a budget of energy with batteries (rather than almost unlimited energy support on servers or workstations), their dynamic power management often changes the execution frequency as in the widely-used dynamic voltage and frequency scaling (DVFS) technique. This leads to highly unstable inference speed performance, especially for computation-intensive DNN models, which can harm user experience and waste hardware resources. We firstly identify this problem and then propose All-in-One, a highly representative pruning framework to work with dynamic power management using DVFS. The framework can use only one set of model weights and soft masks (together with other auxiliary parameters of negligible storage) to represent multiple models of various pruning ratios. By re-configuring the model to the corresponding pruning ratio for a specific execution frequency (and voltage), we are able to achieve stable inference speed, i.e., keeping the difference in speed performance under various execution frequencies as small as possible. Our experiments demonstrate that our method not only achieves high accuracy for multiple models of different pruning ratios, but also reduces their variance of inference latency for various frequencies, with minimal memory consumption of only one model and one soft mask.

混合精确的深神经网络达到了硬件部署所需的能源效率和吞吐量，尤其是在资源有限的情况下，而无需牺牲准确性。但是，不容易找到保留精度的最佳每层钻头精度，尤其是在创建巨大搜索空间的大量模型，数据集和量化技术中。为了解决这一困难，最近出现了一系列文献，并且已经提出了一些实现有希望的准确性结果的框架。在本文中，我们首先总结了文献中通常使用的量化技术。然后，我们对混合精液框架进行了彻底的调查，该调查是根据其优化技术进行分类的，例如增强学习和量化技术，例如确定性舍入。此外，讨论了每个框架的优势和缺点，我们在其中呈现并列。我们最终为未来的混合精液框架提供了指南。

深度神经网络（DNN）的计算要求增加导致获得稀疏，且准确的DNN模型的兴趣。最近的工作已经调查了稀疏训练的更加困难的情况，其中DNN重量尽可能稀少，以减少训练期间的计算成本。现有的稀疏训练方法通常是经验的，并且可以具有相对于致密基线的准确性较低。在本文中，我们介绍了一种称为交替压缩/解压缩（AC / DC）训练DNN的一般方法，证明了算法变体的收敛，并表明AC / DC在类似的计算预算中准确地表现出现有的稀疏训练方法;在高稀疏水平下，AC / DC甚至优于现有的现有方法，依赖于准确的预训练密集模型。 AC / DC的一个重要属性是它允许联合培训密集和稀疏的型号，在训练过程结束时产生精确的稀疏密集模型对。这在实践中是有用的，其中压缩变体可能是为了在资源受限的设置中进行部署而不重新执行整个训练流，并且还为我们提供了深入和压缩模型之间的精度差距的见解。代码可在：https://github.com/ist-daslab/acdc。

To reduce the significant redundancy in deep Convolutional Neural Networks (CNNs), most existing methods prune neurons by only considering statistics of an individual layer or two consecutive layers (e.g., prune one layer to minimize the reconstruction error of the next layer), ignoring the effect of error propagation in deep networks. In contrast, we argue that it is essential to prune neurons in the entire neuron network jointly based on a unified goal: minimizing the reconstruction error of important responses in the "final response layer" (FRL), which is the secondto-last layer before classification, for a pruned network to retrain its predictive power. Specifically, we apply feature ranking techniques to measure the importance of each neuron in the FRL, and formulate network pruning as a binary integer optimization problem and derive a closed-form solution to it for pruning neurons in earlier layers. Based on our theoretical analysis, we propose the Neuron Importance Score Propagation (NISP) algorithm to propagate the importance scores of final responses to every neuron in the network. The CNN is pruned by removing neurons with least importance, and then fine-tuned to retain its predictive power. NISP is evaluated on several datasets with multiple CNN models and demonstrated to achieve significant acceleration and compression with negligible accuracy loss.