表现良好的深度学习模型通常具有很高的计算成本。在本文中，我们结合了两种试图降低计算成本的方法，同时保持模型性能很高：修剪和提早出口网络。我们评估了修剪早期出口网络的两种方法：（1）立即修剪整个网络，（2）以有序的方式修剪基本网络和其他线性分类器。实验结果表明，一般而言，立即修剪整个网络是更好的策略。但是，以高精度的速度，这两种方法具有相似的性能，这意味着可以将修剪和提早出口的过程分开而不会丧失最佳性。

Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy. In this work, we make several surprising observations which contradict common beliefs. For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights. For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch. Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods. We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin, 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization. * Equal contribution. † Work done while visiting UC Berkeley.

Most existing pruning works are resource-intensive, requiring retraining or fine-tuning of the pruned models for accuracy. We propose a retraining-free pruning method based on hyperspherical learning and loss penalty terms. The proposed loss penalty term pushes some of the model weights far from zero, while the rest weight values are pushed near zero and can be safely pruned with no need for retraining and a negligible accuracy drop. In addition, our proposed method can instantly recover the accuracy of a pruned model by replacing the pruned values with their mean value. Our method obtains state-of-the-art results in retraining-free pruning and is evaluated on ResNet-18/50 and MobileNetV2 with ImageNet dataset. One can easily get a 50\% pruned ResNet18 model with a 0.47\% accuracy drop. With fine-tuning, the experiment results show that our method can significantly boost the accuracy of the pruned models compared with existing works. For example, the accuracy of a 70\% pruned (except the first convolutional layer) MobileNetV2 model only drops 3.5\%, much less than the 7\% $\sim$ 10\% accuracy drop with conventional methods.

Slimmable Neural Networks (S-Net) is a novel network which enabled to select one of the predefined proportions of channels (sub-network) dynamically depending on the current computational resource availability. The accuracy of each sub-network on S-Net, however, is inferior to that of individually trained networks of the same size due to its difficulty of simultaneous optimization on different sub-networks. In this paper, we propose Slimmable Pruned Neural Networks (SP-Net), which has sub-network structures learned by pruning instead of adopting structures with the same proportion of channels in each layer (width multiplier) like S-Net, and we also propose new pruning procedures: multi-base pruning instead of one-shot or iterative pruning to realize high accuracy and huge training time saving. We also introduced slimmable channel sorting (scs) to achieve calculation as fast as S-Net and zero padding match (zpm) pruning to prune residual structure in more efficient way. SP-Net can be combined with any kind of channel pruning methods and does not require any complicated processing or time-consuming architecture search like NAS models. Compared with each sub-network of the same FLOPs on S-Net, SP-Net improves accuracy by 1.2-1.5% for ResNet-50, 0.9-4.4% for VGGNet, 1.3-2.7% for MobileNetV1, 1.4-3.1% for MobileNetV2 on ImageNet. Furthermore, our methods outperform other SOTA pruning methods and are on par with various NAS models according to our experimental results on ImageNet. The code is available at https://github.com/hideakikuratsu/SP-Net.

This paper proposed a Soft Filter Pruning (SFP) method to accelerate the inference procedure of deep Convolutional Neural Networks (CNNs). Specifically, the proposed SFP enables the pruned filters to be updated when training the model after pruning. SFP has two advantages over previous works: (1) Larger model capacity. Updating previously pruned filters provides our approach with larger optimization space than fixing the filters to zero. Therefore, the network trained by our method has a larger model capacity to learn from the training data. (2) Less dependence on the pretrained model. Large capacity enables SFP to train from scratch and prune the model simultaneously. In contrast, previous filter pruning methods should be conducted on the basis of the pre-trained model to guarantee their performance. Empirically, SFP from scratch outperforms the previous filter pruning methods. Moreover, our approach has been demonstrated effective for many advanced CNN architectures. Notably, on ILSCRC-2012, SFP reduces more than 42% FLOPs on ResNet-101 with even 0.2% top-5 accuracy improvement, which has advanced the state-of-the-art. Code is publicly available on GitHub: https://github.com/he-y/softfilter-pruning

由于深度学习模型通常包含数百万可培训的权重，因此对更有效的网络结构具有越来越高的存储空间和提高的运行时效率。修剪是最受欢迎的网络压缩技术之一。在本文中，我们提出了一种新颖的非结构化修剪管线，基于关注的同时稀疏结构和体重学习（ASWL）。与传统的频道和体重注意机制不同，ASWL提出了一种有效的算法来计算每层的层次引起的修剪比率，并且跟踪密度网络和稀疏网络的两种权重，以便修剪结构是同时从随机初始化的权重学习。我们在Mnist，CiFar10和Imagenet上的实验表明，与最先进的网络修剪方法相比，ASWL在准确性，修剪比率和操作效率方面取得了卓越的修剪。

少量样本压缩旨在将大冗余模型压缩成一个小型紧凑型，只有少量样品。如果我们的微调模型直接具有这些限制的样本，模型将容易受到过度装备，并且几乎没有学习。因此，先前的方法优化压缩模型逐层，并尝试使每个层具有与教师模型中的相应层相同的输出，这是麻烦的。在本文中，我们提出了一个名为mimicking的新框架，然后替换（mir），以实现几个样本压缩，这首先促使修剪模型输出与教师在倒数第二层中的相同功能，然后在倒数第二个之前替换教师的图层调整良好的紧凑型。与以前的层面重建方法不同，我们的MIR完全优化整个网络，这不仅简单而有效，而且还无人驾驶和一般。MIR优于以前的余量。代码即将推出。

网络修剪是一种广泛使用的技术，用于有效地压缩深神经网络，几乎没有在推理期间在性能下降低。迭代幅度修剪（IMP）是由几种迭代训练和修剪步骤组成的网络修剪的最熟悉的方法之一，其中在修剪后丢失了大量网络的性能，然后在随后的再培训阶段中恢复。虽然常用为基准参考，但经常认为a）通过不将稀疏纳入训练阶段来达到次优状态，b）其全球选择标准未能正确地确定最佳层面修剪速率和c）其迭代性质使它变得缓慢和不竞争。根据最近提出的再培训技术，我们通过严格和一致的实验来调查这些索赔，我们将Impr到培训期间的训练算法进行比较，评估其选择标准的建议修改，并研究实际需要的迭代次数和总培训时间。我们发现IMP与SLR进行再培训，可以优于最先进的修剪期间，没有或仅具有很少的计算开销，即全局幅度选择标准在很大程度上具有更复杂的方法，并且只有几个刷新时期在实践中需要达到大部分稀疏性与IMP的诽谤 - 与性能权衡。我们的目标既可以证明基本的进攻已经可以提供最先进的修剪结果，甚至优于更加复杂或大量参数化方法，也可以为未来的研究建立更加现实但易于可实现的基线。

卷积神经网络（CNN）具有一定量的参数冗余，滤波器修剪旨在去除冗余滤波器，并提供在终端设备上应用CNN的可能性。但是，以前的作品更加注重设计了滤波器重要性的评估标准，然后缩短了具有固定修剪率的重要滤波器或固定数量，以减少卷积神经网络的冗余。它不考虑为每层预留有多少筛选器是最合理的选择。从这个角度来看，我们通过搜索适当的过滤器（SNF）来提出新的过滤器修剪方法。 SNF专用于搜索每层的最合理的保留过滤器，然后是具有特定标准的修剪过滤器。它可以根据不同的拖鞋定制最合适的网络结构。通过我们的方法进行过滤器修剪导致CIFAR-10的最先进（SOTA）精度，并在Imagenet ILSVRC-2012上实现了竞争性能。基于Reset-56网络，在Top-中增加了0.14％的增加0.14％ 1对CIFAR-10拖出的52.94％的精度为52.94％。在减少68.68％拖鞋时，CiFar-10上的修剪Resnet-110还提高了0.03％的1 0.03％的精度。对于Imagenet，我们将修剪速率设置为52.10％的拖鞋，前1个精度只有0.74％。该代码可以在https://github.com/pk-l/snf上获得。

Previous works utilized "smaller-norm-less-important" criterion to prune filters with smaller norm values in a convolutional neural network. In this paper, we analyze this norm-based criterion and point out that its effectiveness depends on two requirements that are not always met: (1) the norm deviation of the filters should be large; (2) the minimum norm of the filters should be small. To solve this problem, we propose a novel filter pruning method, namely Filter Pruning via Geometric Median (FPGM), to compress the model regardless of those two requirements. Unlike previous methods, FPGM compresses CNN models by pruning filters with redundancy, rather than those with "relatively less" importance. When applied to two image classification benchmarks, our method validates its usefulness and strengths. Notably, on CIFAR-10, FPGM reduces more than 52% FLOPs on ResNet-110 with even 2.69% relative accuracy improvement. Moreover, on ILSVRC-2012, FPGM reduces more than 42% FLOPs on ResNet-101 without top-5 accuracy drop, which has advanced the state-of-the-art. Code is publicly available on GitHub: https://github.com/he-y/filter-pruning-geometric-median * Corresponding Author. Part of this work was done when Yi Yang was visiting Baidu Research during his Professional Experience Program.

现有的可区分通道修剪方法通常将缩放因子或掩模在通道后面的掩盖范围内，以减少重要性的修剪过滤器，并假设输入样品统一贡献以过滤重要性。具体而言，实例复杂性对修剪性能的影响尚未得到充分研究。在本文中，我们提出了一个基于实例复杂性滤波器的重要性得分的简单而有效的可区分网络修剪方法上限。我们通过给硬样品给出更高的权重来定义每个样品的实例复杂性与重量相关的重量，并测量样品特异性软膜的加权总和，以模拟不同输入的非均匀贡献，这鼓励硬样品主导修剪过程和模型性能保存完好。此外，我们还引入了一个新的正规器，以鼓励面具两极分化，以便很容易找到甜蜜的位置以识别要修剪的过滤器。各种网络体系结构和数据集的性能评估表明，CAP在修剪大型网络方面具有优势。例如，CAP在删除65.64％的拖鞋后，CAP在CIFAR-10数据集上的RESNET56的准确性提高了0.33％，而Prunes在ImagEnet数据集上的RESNET50的PRUNES 87.75％，只有0.89％的TOP-1精度损失。

过滤器修剪方法通过去除选定的过滤器来引入结构稀疏性，因此对于降低复杂性特别有效。先前的作品从验证较小规范的过滤器的角度从经验修剪网络中造成了较小的最终结果贡献。但是，此类标准已被证明对过滤器的分布敏感，并且由于修剪后的容量差距是固定的，因此准确性可能很难恢复。在本文中，我们提出了一种称为渐近软簇修剪（ASCP）的新型过滤器修剪方法，以根据过滤器的相似性来识别网络的冗余。首先通过聚类来区分来自参数过度的网络的每个过滤器，然后重建以手动将冗余引入其中。提出了一些聚类指南，以更好地保留特征提取能力。重建后，允许更新过滤器，以消除错误选择的效果。此外，还采用了各种修剪率的衰减策略来稳定修剪过程并改善最终性能。通过逐渐在每个群集中生成更相同的过滤器，ASCP可以通过通道添加操作将其删除，几乎没有准确性下降。 CIFAR-10和Imagenet数据集的广泛实验表明，与许多最新算法相比，我们的方法可以取得竞争性结果。

Neural network pruning offers a promising prospect to facilitate deploying deep neural networks on resourcelimited devices. However, existing methods are still challenged by the training inefficiency and labor cost in pruning designs, due to missing theoretical guidance of non-salient network components. In this paper, we propose a novel filter pruning method by exploring the High Rank of feature maps (HRank). Our HRank is inspired by the discovery that the average rank of multiple feature maps generated by a single filter is always the same, regardless of the number of image batches CNNs receive. Based on HRank, we develop a method that is mathematically formulated to prune filters with low-rank feature maps. The principle behind our pruning is that low-rank feature maps contain less information, and thus pruned results can be easily reproduced. Besides, we experimentally show that weights with high-rank feature maps contain more important information, such that even when a portion is not updated, very little damage would be done to the model performance. Without introducing any additional constraints, HRank leads to significant improvements over the state-of-the-arts in terms of FLOPs and parameters reduction, with similar accuracies. For example, with ResNet-110, we achieve a 58.2%-FLOPs reduction by removing 59.2% of the parameters, with only a small loss of 0.14% in top-1 accuracy on CIFAR-10. With Res-50, we achieve a 43.8%-FLOPs reduction by removing 36.7% of the parameters, with only a loss of 1.17% in the top-1 accuracy on ImageNet. The codes can be available at https://github.com/lmbxmu/HRank.

The deployment of deep convolutional neural networks (CNNs) in many real world applications is largely hindered by their high computational cost. In this paper, we propose a novel learning scheme for CNNs to simultaneously 1) reduce the model size; 2) decrease the run-time memory footprint; and 3) lower the number of computing operations, without compromising accuracy. This is achieved by enforcing channel-level sparsity in the network in a simple but effective way. Different from many existing approaches, the proposed method directly applies to modern CNN architectures, introduces minimum overhead to the training process, and requires no special software/hardware accelerators for the resulting models. We call our approach network slimming, which takes wide and large networks as input models, but during training insignificant channels are automatically identified and pruned afterwards, yielding thin and compact models with comparable accuracy. We empirically demonstrate the effectiveness of our approach with several state-of-the-art CNN models, including VGGNet, ResNet and DenseNet, on various image classification datasets. For VGGNet, a multi-pass version of network slimming gives a 20× reduction in model size and a 5× reduction in computing operations.

神经网络修剪具有显着性能，可以降低深网络模型的复杂性。最近的网络修剪方法通常集中在网络中删除不重要或冗余过滤器。在本文中，通过探索特征图之间的相似性，我们提出了一种新颖的滤波器修剪方法，中央滤波器（CF），这表明在适当的调整之后滤波器大致等于一组其他滤波器。我们的方法基于发现特征贴图之间的平均相似性的发现，而不管输入图像的数量如何，都会很少变化。基于此发现，我们在特征映射上建立相似性图，并计算每个节点的近密中心以选择中央滤波器。此外，我们设计一种方法，可以在与中央滤波器对应的下一层中直接调整权重，有效地最小化由修剪引起的误差。通过对各种基准网络和数据集的实验，CF产生最先进的性能。例如，对于Reset-56，CF通过去除47.1％的参数来减少约39.7％的絮凝物，甚至在CiFar-10上的精度改善0.33％。通过Googlenet，CF通过去除55.6％的参数来减少大约63.2％的拖鞋，仅在CIFAR-10上的前1个精度下降0.35％的损失。通过resnet-50，CF通过去除36.9％的参数减少约47.9％的拖鞋，仅在Imagenet上的前1个精度下降1.07％。该代码可以在https://github.com/8ubpshlr23/centrter上获得。

The success of CNNs in various applications is accompanied by a significant increase in the computation and parameter storage costs. Recent efforts toward reducing these overheads involve pruning and compressing the weights of various layers without hurting original accuracy. However, magnitude-based pruning of weights reduces a significant number of parameters from the fully connected layers and may not adequately reduce the computation costs in the convolutional layers due to irregular sparsity in the pruned networks. We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications. We show that even simple filter pruning techniques can reduce inference costs for VGG-16 by up to 34% and ResNet-110 by up to 38% on CIFAR10 while regaining close to the original accuracy by retraining the networks.

近年来，深度神经网络在各种应用领域中都有广泛的成功。但是，它们需要重要的计算和内存资源，严重阻碍其部署，特别是在移动设备上或实时应用程序。神经网络通常涉及大量参数，该参数对应于网络的权重。在培训过程中获得的这种参数是用于网络性能的决定因素。但是，它们也非常冗余。修剪方法尤其试图通过识别和移除不相关的重量来减小参数集的大小。在本文中，我们研究了培训策略对修剪效率的影响。考虑和比较了两种培训方式：（1）微调和（2）从头开始。在四个数据集（CIFAR10，CiFAR100，SVHN和CALTECH101）上获得的实验结果和两个不同的CNNS（VGG16和MOBILENET）证明已经在大语料库（例如想象成）上预先培训的网络，然后进行微调特定数据集可以更有效地修剪（高达80％的参数减少），而不是从头开始培训的相同网络。

过滤器修剪的目标是搜索不重要的过滤器以删除以便使卷积神经网络（CNNS）有效而不牺牲过程中的性能。挑战在于找到可以帮助确定每个过滤器关于神经网络的最终输出的重要或相关的信息的信息。在这项工作中，我们分享了我们的观察说，预先训练的CNN的批量标准化（BN）参数可用于估计激活输出的特征分布，而无需处理训练数据。在观察时，我们通过基于预先训练的CNN的BN参数评估每个滤波器的重要性来提出简单而有效的滤波修剪方法。 CiFar-10和Imagenet的实验结果表明，该方法可以在准确性下降和计算复杂性的计算复杂性和降低的折衷方面具有和不进行微调的卓越性能。

我们提出了一种用于相干光子神经网络的新型硬件感知幅度修剪技术。该技术可以将99.45％的网络参数进行99.45％，并将静态功耗降低98.23％，精度损失可忽略不计。

Neural network pruning-the task of reducing the size of a network by removing parameters-has been the subject of a great deal of work in recent years. We provide a meta-analysis of the literature, including an overview of approaches to pruning and consistent findings in the literature. After aggregating results across 81 papers and pruning hundreds of models in controlled conditions, our clearest finding is that the community suffers from a lack of standardized benchmarks and metrics. This deficiency is substantial enough that it is hard to compare pruning techniques to one another or determine how much progress the field has made over the past three decades. To address this situation, we identify issues with current practices, suggest concrete remedies, and introduce ShrinkBench, an open-source framework to facilitate standardized evaluations of pruning methods. We use ShrinkBench to compare various pruning techniques and show that its comprehensive evaluation can prevent common pitfalls when comparing pruning methods.