由于深度学习模型通常包含数百万可培训的权重，因此对更有效的网络结构具有越来越高的存储空间和提高的运行时效率。修剪是最受欢迎的网络压缩技术之一。在本文中，我们提出了一种新颖的非结构化修剪管线，基于关注的同时稀疏结构和体重学习（ASWL）。与传统的频道和体重注意机制不同，ASWL提出了一种有效的算法来计算每层的层次引起的修剪比率，并且跟踪密度网络和稀疏网络的两种权重，以便修剪结构是同时从随机初始化的权重学习。我们在Mnist，CiFar10和Imagenet上的实验表明，与最先进的网络修剪方法相比，ASWL在准确性，修剪比率和操作效率方面取得了卓越的修剪。

The deployment of deep convolutional neural networks (CNNs) in many real world applications is largely hindered by their high computational cost. In this paper, we propose a novel learning scheme for CNNs to simultaneously 1) reduce the model size; 2) decrease the run-time memory footprint; and 3) lower the number of computing operations, without compromising accuracy. This is achieved by enforcing channel-level sparsity in the network in a simple but effective way. Different from many existing approaches, the proposed method directly applies to modern CNN architectures, introduces minimum overhead to the training process, and requires no special software/hardware accelerators for the resulting models. We call our approach network slimming, which takes wide and large networks as input models, but during training insignificant channels are automatically identified and pruned afterwards, yielding thin and compact models with comparable accuracy. We empirically demonstrate the effectiveness of our approach with several state-of-the-art CNN models, including VGGNet, ResNet and DenseNet, on various image classification datasets. For VGGNet, a multi-pass version of network slimming gives a 20× reduction in model size and a 5× reduction in computing operations.

Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy. In this work, we make several surprising observations which contradict common beliefs. For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights. For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch. Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods. We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin, 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization. * Equal contribution. † Work done while visiting UC Berkeley.

Low-rankness plays an important role in traditional machine learning, but is not so popular in deep learning. Most previous low-rank network compression methods compress the networks by approximating pre-trained models and re-training. However, the optimal solution in the Euclidean space may be quite different from the one in the low-rank manifold. A well-pre-trained model is not a good initialization for the model with low-rank constraints. Thus, the performance of a low-rank compressed network degrades significantly. Compared to other network compression methods such as pruning, low-rank methods attracts less attention in recent years. In this paper, we devise a new training method, low-rank projection with energy transfer (LRPET), that trains low-rank compressed networks from scratch and achieves competitive performance. First, we propose to alternately perform stochastic gradient descent training and projection onto the low-rank manifold. Compared to re-training on the compact model, this enables full utilization of model capacity since solution space is relaxed back to Euclidean space after projection. Second, the matrix energy (the sum of squares of singular values) reduction caused by projection is compensated by energy transfer. We uniformly transfer the energy of the pruned singular values to the remaining ones. We theoretically show that energy transfer eases the trend of gradient vanishing caused by projection. Third, we propose batch normalization (BN) rectification to cut off its effect on the optimal low-rank approximation of the weight matrix, which further improves the performance. Comprehensive experiments on CIFAR-10 and ImageNet have justified that our method is superior to other low-rank compression methods and also outperforms recent state-of-the-art pruning methods. Our code is available at https://github.com/BZQLin/LRPET.

在过去几年中，神经网络的性能在越来越多的浮点操作（拖鞋）的成本上显着提高。但是，当计算资源有限时，更多的拖鞋可能是一个问题。作为解决这个问题的尝试，修剪过滤器是一种常见的解决方案，但大多数现有的修剪方法不有效地保持模型精度，因此需要大量的芬降时期。在本文中，我们提出了一种自动修剪方法，该方法学习保存的神经元以保持模型精度，同时将絮凝到预定目标。为了完成这项任务，我们介绍了一种可训练的瓶颈，只需要一个单一的单一时期，只需要一个数据集的25.6％（Cifar-10）或7.49％（ILSVRC2012）来了解哪些过滤器。在各种架构和数据集上的实验表明，该方法不仅可以在修剪后保持精度，而且在FineTuning之后也优越现有方法。我们在Reset-50上达到了52.00％的拖鞋，在ILSVRC2012上的灌溉后的前1个精度为47.51％，最先进的（SOTA）精度为76.63％。代码可用（链接匿名审核）。

This paper proposed a Soft Filter Pruning (SFP) method to accelerate the inference procedure of deep Convolutional Neural Networks (CNNs). Specifically, the proposed SFP enables the pruned filters to be updated when training the model after pruning. SFP has two advantages over previous works: (1) Larger model capacity. Updating previously pruned filters provides our approach with larger optimization space than fixing the filters to zero. Therefore, the network trained by our method has a larger model capacity to learn from the training data. (2) Less dependence on the pretrained model. Large capacity enables SFP to train from scratch and prune the model simultaneously. In contrast, previous filter pruning methods should be conducted on the basis of the pre-trained model to guarantee their performance. Empirically, SFP from scratch outperforms the previous filter pruning methods. Moreover, our approach has been demonstrated effective for many advanced CNN architectures. Notably, on ILSCRC-2012, SFP reduces more than 42% FLOPs on ResNet-101 with even 0.2% top-5 accuracy improvement, which has advanced the state-of-the-art. Code is publicly available on GitHub: https://github.com/he-y/softfilter-pruning

由于稀疏神经网络通常包含许多零权重，因此可以在不降低网络性能的情况下潜在地消除这些不必要的网络连接。因此，设计良好的稀疏神经网络具有显着降低拖鞋和计算资源的潜力。在这项工作中，我们提出了一种新的自动修剪方法 - 稀疏连接学习（SCL）。具体地，重量被重新参数化为可培训权重变量和二进制掩模的元素方向乘法。因此，由二进制掩模完全描述网络连接，其由单位步进函数调制。理论上，从理论上证明了使用直通估计器（STE）进行网络修剪的基本原理。这一原则是STE的代理梯度应该是积极的，确保掩模变量在其最小值处收敛。在找到泄漏的Relu后，SoftPlus和Identity Stes可以满足这个原理，我们建议采用SCL的身份STE以进行离散面膜松弛。我们发现不同特征的面具梯度非常不平衡，因此，我们建议将每个特征的掩模梯度标准化以优化掩码变量训练。为了自动训练稀疏掩码，我们将网络连接总数作为我们的客观函数中的正则化术语。由于SCL不需要由网络层设计人员定义的修剪标准或超级参数，因此在更大的假设空间中探讨了网络，以实现最佳性能的优化稀疏连接。 SCL克服了现有自动修剪方法的局限性。实验结果表明，SCL可以自动学习并选择各种基线网络结构的重要网络连接。 SCL培训的深度学习模型以稀疏性，精度和减少脚波特的SOTA人类设计和自动修剪方法训练。

过滤器修剪方法通过去除选定的过滤器来引入结构稀疏性，因此对于降低复杂性特别有效。先前的作品从验证较小规范的过滤器的角度从经验修剪网络中造成了较小的最终结果贡献。但是，此类标准已被证明对过滤器的分布敏感，并且由于修剪后的容量差距是固定的，因此准确性可能很难恢复。在本文中，我们提出了一种称为渐近软簇修剪（ASCP）的新型过滤器修剪方法，以根据过滤器的相似性来识别网络的冗余。首先通过聚类来区分来自参数过度的网络的每个过滤器，然后重建以手动将冗余引入其中。提出了一些聚类指南，以更好地保留特征提取能力。重建后，允许更新过滤器，以消除错误选择的效果。此外，还采用了各种修剪率的衰减策略来稳定修剪过程并改善最终性能。通过逐渐在每个群集中生成更相同的过滤器，ASCP可以通过通道添加操作将其删除，几乎没有准确性下降。 CIFAR-10和Imagenet数据集的广泛实验表明，与许多最新算法相比，我们的方法可以取得竞争性结果。

现有的可区分通道修剪方法通常将缩放因子或掩模在通道后面的掩盖范围内，以减少重要性的修剪过滤器，并假设输入样品统一贡献以过滤重要性。具体而言，实例复杂性对修剪性能的影响尚未得到充分研究。在本文中，我们提出了一个基于实例复杂性滤波器的重要性得分的简单而有效的可区分网络修剪方法上限。我们通过给硬样品给出更高的权重来定义每个样品的实例复杂性与重量相关的重量，并测量样品特异性软膜的加权总和，以模拟不同输入的非均匀贡献，这鼓励硬样品主导修剪过程和模型性能保存完好。此外，我们还引入了一个新的正规器，以鼓励面具两极分化，以便很容易找到甜蜜的位置以识别要修剪的过滤器。各种网络体系结构和数据集的性能评估表明，CAP在修剪大型网络方面具有优势。例如，CAP在删除65.64％的拖鞋后，CAP在CIFAR-10数据集上的RESNET56的准确性提高了0.33％，而Prunes在ImagEnet数据集上的RESNET50的PRUNES 87.75％，只有0.89％的TOP-1精度损失。

Most existing pruning works are resource-intensive, requiring retraining or fine-tuning of the pruned models for accuracy. We propose a retraining-free pruning method based on hyperspherical learning and loss penalty terms. The proposed loss penalty term pushes some of the model weights far from zero, while the rest weight values are pushed near zero and can be safely pruned with no need for retraining and a negligible accuracy drop. In addition, our proposed method can instantly recover the accuracy of a pruned model by replacing the pruned values with their mean value. Our method obtains state-of-the-art results in retraining-free pruning and is evaluated on ResNet-18/50 and MobileNetV2 with ImageNet dataset. One can easily get a 50\% pruned ResNet18 model with a 0.47\% accuracy drop. With fine-tuning, the experiment results show that our method can significantly boost the accuracy of the pruned models compared with existing works. For example, the accuracy of a 70\% pruned (except the first convolutional layer) MobileNetV2 model only drops 3.5\%, much less than the 7\% $\sim$ 10\% accuracy drop with conventional methods.

To reduce the significant redundancy in deep Convolutional Neural Networks (CNNs), most existing methods prune neurons by only considering statistics of an individual layer or two consecutive layers (e.g., prune one layer to minimize the reconstruction error of the next layer), ignoring the effect of error propagation in deep networks. In contrast, we argue that it is essential to prune neurons in the entire neuron network jointly based on a unified goal: minimizing the reconstruction error of important responses in the "final response layer" (FRL), which is the secondto-last layer before classification, for a pruned network to retrain its predictive power. Specifically, we apply feature ranking techniques to measure the importance of each neuron in the FRL, and formulate network pruning as a binary integer optimization problem and derive a closed-form solution to it for pruning neurons in earlier layers. Based on our theoretical analysis, we propose the Neuron Importance Score Propagation (NISP) algorithm to propagate the importance scores of final responses to every neuron in the network. The CNN is pruned by removing neurons with least importance, and then fine-tuned to retain its predictive power. NISP is evaluated on several datasets with multiple CNN models and demonstrated to achieve significant acceleration and compression with negligible accuracy loss.

The success of CNNs in various applications is accompanied by a significant increase in the computation and parameter storage costs. Recent efforts toward reducing these overheads involve pruning and compressing the weights of various layers without hurting original accuracy. However, magnitude-based pruning of weights reduces a significant number of parameters from the fully connected layers and may not adequately reduce the computation costs in the convolutional layers due to irregular sparsity in the pruned networks. We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications. We show that even simple filter pruning techniques can reduce inference costs for VGG-16 by up to 34% and ResNet-110 by up to 38% on CIFAR10 while regaining close to the original accuracy by retraining the networks.

We propose an efficient and unified framework, namely ThiNet, to simultaneously accelerate and compress CNN models in both training and inference stages. We focus on the filter level pruning, i.e., the whole filter would be discarded if it is less important. Our method does not change the original network structure, thus it can be perfectly supported by any off-the-shelf deep learning libraries. We formally establish filter pruning as an optimization problem, and reveal that we need to prune filters based on statistics information computed from its next layer, not the current layer, which differentiates ThiNet from existing methods. Experimental results demonstrate the effectiveness of this strategy, which has advanced the state-of-the-art. We also show the performance of ThiNet on ILSVRC-12 benchmark. ThiNet achieves 3.31× FLOPs reduction and 16.63× compression on VGG-16, with only 0.52% top-5 accuracy drop. Similar experiments with ResNet-50 reveal that even for a compact network, ThiNet can also reduce more than half of the parameters and FLOPs, at the cost of roughly 1% top-5 accuracy drop. Moreover, the original VGG-16 model can be further pruned into a very small model with only 5.05MB model size, preserving AlexNet level accuracy but showing much stronger generalization ability.

Many applications require sparse neural networks due to space or inference time restrictions. There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-tosparse training methods. Our method updates the topology of the sparse network during training by using parameter magnitudes and infrequent gradient calculations. We show that this approach requires fewer floating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques. We demonstrate state-of-the-art sparse training results on a variety of networks and datasets, including ResNet-50, MobileNets on Imagenet-2012, and RNNs on WikiText-103. Finally, we provide some insights into why allowing the topology to change during the optimization can overcome local minima encountered when the topology remains static * .

In this paper, we propose a novel meta learning approach for automatic channel pruning of very deep neural networks. We first train a PruningNet, a kind of meta network, which is able to generate weight parameters for any pruned structure given the target network. We use a simple stochastic structure sampling method for training the PruningNet. Then, we apply an evolutionary procedure to search for good-performing pruned networks. The search is highly efficient because the weights are directly generated by the trained PruningNet and we do not need any finetuning at search time. With a single PruningNet trained for the target network, we can search for various Pruned Networks under different constraints with little human participation. Compared to the state-of-the-art pruning methods, we have demonstrated superior performances on Mo-bileNet V1/V2 and ResNet. Codes are available on https: //github.com/liuzechun/MetaPruning. This work is done when Zechun Liu and Haoyuan Mu are interns at Megvii Technology.

卷积神经网络（CNN）具有一定量的参数冗余，滤波器修剪旨在去除冗余滤波器，并提供在终端设备上应用CNN的可能性。但是，以前的作品更加注重设计了滤波器重要性的评估标准，然后缩短了具有固定修剪率的重要滤波器或固定数量，以减少卷积神经网络的冗余。它不考虑为每层预留有多少筛选器是最合理的选择。从这个角度来看，我们通过搜索适当的过滤器（SNF）来提出新的过滤器修剪方法。 SNF专用于搜索每层的最合理的保留过滤器，然后是具有特定标准的修剪过滤器。它可以根据不同的拖鞋定制最合适的网络结构。通过我们的方法进行过滤器修剪导致CIFAR-10的最先进（SOTA）精度，并在Imagenet ILSVRC-2012上实现了竞争性能。基于Reset-56网络，在Top-中增加了0.14％的增加0.14％ 1对CIFAR-10拖出的52.94％的精度为52.94％。在减少68.68％拖鞋时，CiFar-10上的修剪Resnet-110还提高了0.03％的1 0.03％的精度。对于Imagenet，我们将修剪速率设置为52.10％的拖鞋，前1个精度只有0.74％。该代码可以在https://github.com/pk-l/snf上获得。

近年来，深度神经网络在各种应用领域中都有广泛的成功。但是，它们需要重要的计算和内存资源，严重阻碍其部署，特别是在移动设备上或实时应用程序。神经网络通常涉及大量参数，该参数对应于网络的权重。在培训过程中获得的这种参数是用于网络性能的决定因素。但是，它们也非常冗余。修剪方法尤其试图通过识别和移除不相关的重量来减小参数集的大小。在本文中，我们研究了培训策略对修剪效率的影响。考虑和比较了两种培训方式：（1）微调和（2）从头开始。在四个数据集（CIFAR10，CiFAR100，SVHN和CALTECH101）上获得的实验结果和两个不同的CNNS（VGG16和MOBILENET）证明已经在大语料库（例如想象成）上预先培训的网络，然后进行微调特定数据集可以更有效地修剪（高达80％的参数减少），而不是从头开始培训的相同网络。

网络压缩对于使深网的效率更高，更快且可推广到低端硬件至关重要。当前的网络压缩方法有两个开放问题：首先，缺乏理论框架来估计最大压缩率；其次，有些层可能会过多地进行，从而导致网络性能大幅下降。为了解决这两个问题，这项研究提出了一种基于梯度矩阵分析方法，以估计最大网络冗余。在最大速率的指导下，开发了一种新颖而有效的层次网络修剪算法，以最大程度地凝结神经元网络结构而无需牺牲网络性能。进行实质性实验以证明新方法修剪几个高级卷积神经网络（CNN）体系结构的功效。与现有的修剪方法相比，拟议的修剪算法实现了最先进的性能。与其他方法相比，在相同或相似的压缩比下，新方法提供了最高的网络预测准确性。

修剪技术可全面使用图像分类压缩卷积神经网络（CNN）。但是，大多数修剪方法需要一个经过良好训练的模型，以提供有用的支持参数，例如C1-核心，批处理值和梯度信息，如果预训练的模型的参数为，这可能会导致过滤器评估的不一致性不太优化。因此，我们提出了一种基于敏感性的方法，可以通过为原始模型增加额外的损害来评估每一层的重要性。由于准确性的性能取决于参数在所有层而不是单个参数中的分布，因此基于灵敏度的方法将对参数的更新具有鲁棒性。也就是说，我们可以获得对不完美训练和完全训练的模型之间每个卷积层的相似重要性评估。对于CIFAR-10上的VGG-16，即使原始模型仅接受50个时期训练，我们也可以对层的重要性进行相同的评估，并在对模型进行充分训练时的结果。然后，我们将通过量化的灵敏度从每一层中删除过滤器。我们基于敏感性的修剪框架在VGG-16，分别具有CIFAR-10，MNIST和CIFAR-100的VGG-16上有效验证。

深度神经网络（DNN）在解决许多真实问题方面都有效。较大的DNN模型通常表现出更好的质量（例如，精度，精度），但它们的过度计算会导致长期推理时间。模型稀疏可以降低计算和内存成本，同时保持模型质量。大多数现有的稀疏算法是单向移除的重量，而其他人则随机或贪婪地探索每层进行修剪的小权重子集。这些算法的局限性降低了可实现的稀疏性水平。此外，许多算法仍然需要预先训练的密集模型，因此遭受大的内存占地面积。在本文中，我们提出了一种新颖的预定生长和修剪（间隙）方法，而无需预先培训密集模型。它通过反复生长一个层次的层来解决以前的作品的缺点，然后在一些训练后修剪回到稀疏。实验表明，使用所提出的方法修剪模型匹配或击败高度优化的密集模型的质量，在各种任务中以80％的稀疏度，例如图像分类，客观检测，3D对象分段和翻译。它们还优于模型稀疏的其他最先进的（SOTA）方法。作为一个例子，通过间隙获得的90％不均匀的稀疏resnet-50模型在想象中实现了77.9％的前1个精度，提高了先前的SOTA结果1.5％。所有代码将公开发布。