卷积神经网络（CNN）已在许多物联网（IoT）设备中应用于多种下游任务。但是，随着边缘设备上的数据量的增加，CNN几乎无法及时完成某些任务，而计算和存储资源有限。最近，过滤器修剪被认为是压缩和加速CNN的有效技术，但是从压缩高维张量的角度来看，现有的方法很少是修剪CNN。在本文中，我们提出了一种新颖的理论，可以在三维张量中找到冗余信息，即量化特征图（QSFM）之间的相似性，并利用该理论来指导滤波器修剪过程。我们在数据集（CIFAR-10，CIFAR-100和ILSVRC-12）上执行QSFM和Edge设备，证明所提出的方法可以在神经网络中找到冗余信息，具有可比的压缩和可耐受的准确性下降。没有任何微调操作，QSFM可以显着压缩CIFAR-56（48.7％的Flops和57.9％的参数），而TOP-1的准确性仅损失0.54％。对于边缘设备的实际应用，QSFM可以将Mobilenet-V2推理速度加速1.53倍，而ILSVRC-12 TOP-1的精度仅损失1.23％。

Neural network pruning offers a promising prospect to facilitate deploying deep neural networks on resourcelimited devices. However, existing methods are still challenged by the training inefficiency and labor cost in pruning designs, due to missing theoretical guidance of non-salient network components. In this paper, we propose a novel filter pruning method by exploring the High Rank of feature maps (HRank). Our HRank is inspired by the discovery that the average rank of multiple feature maps generated by a single filter is always the same, regardless of the number of image batches CNNs receive. Based on HRank, we develop a method that is mathematically formulated to prune filters with low-rank feature maps. The principle behind our pruning is that low-rank feature maps contain less information, and thus pruned results can be easily reproduced. Besides, we experimentally show that weights with high-rank feature maps contain more important information, such that even when a portion is not updated, very little damage would be done to the model performance. Without introducing any additional constraints, HRank leads to significant improvements over the state-of-the-arts in terms of FLOPs and parameters reduction, with similar accuracies. For example, with ResNet-110, we achieve a 58.2%-FLOPs reduction by removing 59.2% of the parameters, with only a small loss of 0.14% in top-1 accuracy on CIFAR-10. With Res-50, we achieve a 43.8%-FLOPs reduction by removing 36.7% of the parameters, with only a loss of 1.17% in the top-1 accuracy on ImageNet. The codes can be available at https://github.com/lmbxmu/HRank.

神经网络修剪具有显着性能，可以降低深网络模型的复杂性。最近的网络修剪方法通常集中在网络中删除不重要或冗余过滤器。在本文中，通过探索特征图之间的相似性，我们提出了一种新颖的滤波器修剪方法，中央滤波器（CF），这表明在适当的调整之后滤波器大致等于一组其他滤波器。我们的方法基于发现特征贴图之间的平均相似性的发现，而不管输入图像的数量如何，都会很少变化。基于此发现，我们在特征映射上建立相似性图，并计算每个节点的近密中心以选择中央滤波器。此外，我们设计一种方法，可以在与中央滤波器对应的下一层中直接调整权重，有效地最小化由修剪引起的误差。通过对各种基准网络和数据集的实验，CF产生最先进的性能。例如，对于Reset-56，CF通过去除47.1％的参数来减少约39.7％的絮凝物，甚至在CiFar-10上的精度改善0.33％。通过Googlenet，CF通过去除55.6％的参数来减少大约63.2％的拖鞋，仅在CIFAR-10上的前1个精度下降0.35％的损失。通过resnet-50，CF通过去除36.9％的参数减少约47.9％的拖鞋，仅在Imagenet上的前1个精度下降1.07％。该代码可以在https://github.com/8ubpshlr23/centrter上获得。

This paper proposed a Soft Filter Pruning (SFP) method to accelerate the inference procedure of deep Convolutional Neural Networks (CNNs). Specifically, the proposed SFP enables the pruned filters to be updated when training the model after pruning. SFP has two advantages over previous works: (1) Larger model capacity. Updating previously pruned filters provides our approach with larger optimization space than fixing the filters to zero. Therefore, the network trained by our method has a larger model capacity to learn from the training data. (2) Less dependence on the pretrained model. Large capacity enables SFP to train from scratch and prune the model simultaneously. In contrast, previous filter pruning methods should be conducted on the basis of the pre-trained model to guarantee their performance. Empirically, SFP from scratch outperforms the previous filter pruning methods. Moreover, our approach has been demonstrated effective for many advanced CNN architectures. Notably, on ILSCRC-2012, SFP reduces more than 42% FLOPs on ResNet-101 with even 0.2% top-5 accuracy improvement, which has advanced the state-of-the-art. Code is publicly available on GitHub: https://github.com/he-y/softfilter-pruning

在物联网（IoT）支持的网络边缘（IOT）上的人工智能（AI）的最新进展已通过启用低延期性和计算效率来实现多种应用程序（例如智能农业，智能医院和智能工厂）的优势情报。但是，部署最先进的卷积神经网络（CNN），例如VGG-16和在资源约束的边缘设备上的重新连接，由于其大量参数和浮点操作（Flops），因此实际上是不可行的。因此，将网络修剪作为一种模型压缩的概念正在引起注意在低功率设备上加速CNN。结构化或非结构化的最先进的修剪方法都不认为卷积层表现出的复杂性的不同基本性质，并遵循训练放回训练的管道，从而导致其他计算开销。在这项工作中，我们通过利用CNN的固有层层级复杂性来提出一种新颖和计算高效的修剪管道。与典型的方法不同，我们提出的复杂性驱动算法根据其对整体网络复杂性的贡献选择了特定层用于滤波器。我们遵循一个直接训练修剪模型并避免计算复杂排名和微调步骤的过程。此外，我们定义了修剪的三种模式，即参数感知（PA），拖网（FA）和内存感知（MA），以引入CNN的多功能压缩。我们的结果表明，我们的方法在准确性和加速方面的竞争性能。最后，我们提出了不同资源和准确性之间的权衡取舍，这对于开发人员在资源受限的物联网环境中做出正确的决策可能会有所帮助。

Previous works utilized "smaller-norm-less-important" criterion to prune filters with smaller norm values in a convolutional neural network. In this paper, we analyze this norm-based criterion and point out that its effectiveness depends on two requirements that are not always met: (1) the norm deviation of the filters should be large; (2) the minimum norm of the filters should be small. To solve this problem, we propose a novel filter pruning method, namely Filter Pruning via Geometric Median (FPGM), to compress the model regardless of those two requirements. Unlike previous methods, FPGM compresses CNN models by pruning filters with redundancy, rather than those with "relatively less" importance. When applied to two image classification benchmarks, our method validates its usefulness and strengths. Notably, on CIFAR-10, FPGM reduces more than 52% FLOPs on ResNet-110 with even 2.69% relative accuracy improvement. Moreover, on ILSVRC-2012, FPGM reduces more than 42% FLOPs on ResNet-101 without top-5 accuracy drop, which has advanced the state-of-the-art. Code is publicly available on GitHub: https://github.com/he-y/filter-pruning-geometric-median * Corresponding Author. Part of this work was done when Yi Yang was visiting Baidu Research during his Professional Experience Program.

过滤器修剪方法通过去除选定的过滤器来引入结构稀疏性，因此对于降低复杂性特别有效。先前的作品从验证较小规范的过滤器的角度从经验修剪网络中造成了较小的最终结果贡献。但是，此类标准已被证明对过滤器的分布敏感，并且由于修剪后的容量差距是固定的，因此准确性可能很难恢复。在本文中，我们提出了一种称为渐近软簇修剪（ASCP）的新型过滤器修剪方法，以根据过滤器的相似性来识别网络的冗余。首先通过聚类来区分来自参数过度的网络的每个过滤器，然后重建以手动将冗余引入其中。提出了一些聚类指南，以更好地保留特征提取能力。重建后，允许更新过滤器，以消除错误选择的效果。此外，还采用了各种修剪率的衰减策略来稳定修剪过程并改善最终性能。通过逐渐在每个群集中生成更相同的过滤器，ASCP可以通过通道添加操作将其删除，几乎没有准确性下降。 CIFAR-10和Imagenet数据集的广泛实验表明，与许多最新算法相比，我们的方法可以取得竞争性结果。

卷积神经网络（CNN）具有一定量的参数冗余，滤波器修剪旨在去除冗余滤波器，并提供在终端设备上应用CNN的可能性。但是，以前的作品更加注重设计了滤波器重要性的评估标准，然后缩短了具有固定修剪率的重要滤波器或固定数量，以减少卷积神经网络的冗余。它不考虑为每层预留有多少筛选器是最合理的选择。从这个角度来看，我们通过搜索适当的过滤器（SNF）来提出新的过滤器修剪方法。 SNF专用于搜索每层的最合理的保留过滤器，然后是具有特定标准的修剪过滤器。它可以根据不同的拖鞋定制最合适的网络结构。通过我们的方法进行过滤器修剪导致CIFAR-10的最先进（SOTA）精度，并在Imagenet ILSVRC-2012上实现了竞争性能。基于Reset-56网络，在Top-中增加了0.14％的增加0.14％ 1对CIFAR-10拖出的52.94％的精度为52.94％。在减少68.68％拖鞋时，CiFar-10上的修剪Resnet-110还提高了0.03％的1 0.03％的精度。对于Imagenet，我们将修剪速率设置为52.10％的拖鞋，前1个精度只有0.74％。该代码可以在https://github.com/pk-l/snf上获得。

The success of CNNs in various applications is accompanied by a significant increase in the computation and parameter storage costs. Recent efforts toward reducing these overheads involve pruning and compressing the weights of various layers without hurting original accuracy. However, magnitude-based pruning of weights reduces a significant number of parameters from the fully connected layers and may not adequately reduce the computation costs in the convolutional layers due to irregular sparsity in the pruned networks. We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications. We show that even simple filter pruning techniques can reduce inference costs for VGG-16 by up to 34% and ResNet-110 by up to 38% on CIFAR10 while regaining close to the original accuracy by retraining the networks.

Low-rankness plays an important role in traditional machine learning, but is not so popular in deep learning. Most previous low-rank network compression methods compress the networks by approximating pre-trained models and re-training. However, the optimal solution in the Euclidean space may be quite different from the one in the low-rank manifold. A well-pre-trained model is not a good initialization for the model with low-rank constraints. Thus, the performance of a low-rank compressed network degrades significantly. Compared to other network compression methods such as pruning, low-rank methods attracts less attention in recent years. In this paper, we devise a new training method, low-rank projection with energy transfer (LRPET), that trains low-rank compressed networks from scratch and achieves competitive performance. First, we propose to alternately perform stochastic gradient descent training and projection onto the low-rank manifold. Compared to re-training on the compact model, this enables full utilization of model capacity since solution space is relaxed back to Euclidean space after projection. Second, the matrix energy (the sum of squares of singular values) reduction caused by projection is compensated by energy transfer. We uniformly transfer the energy of the pruned singular values to the remaining ones. We theoretically show that energy transfer eases the trend of gradient vanishing caused by projection. Third, we propose batch normalization (BN) rectification to cut off its effect on the optimal low-rank approximation of the weight matrix, which further improves the performance. Comprehensive experiments on CIFAR-10 and ImageNet have justified that our method is superior to other low-rank compression methods and also outperforms recent state-of-the-art pruning methods. Our code is available at https://github.com/BZQLin/LRPET.

Deploying convolutional neural networks (CNNs) on embedded devices is difficult due to the limited memory and computation resources. The redundancy in feature maps is an important characteristic of those successful CNNs, but has rarely been investigated in neural architecture design. This paper proposes a novel Ghost module to generate more feature maps from cheap operations. Based on a set of intrinsic feature maps, we apply a series of linear transformations with cheap cost to generate many ghost feature maps that could fully reveal information underlying intrinsic features. The proposed Ghost module can be taken as a plug-and-play component to upgrade existing convolutional neural networks. Ghost bottlenecks are designed to stack Ghost modules, and then the lightweight Ghost-Net can be easily established. Experiments conducted on benchmarks demonstrate that the proposed Ghost module is an impressive alternative of convolution layers in baseline models, and our GhostNet can achieve higher recognition performance (e.g. 75.7% top-1 accuracy) than MobileNetV3 with similar computational cost on the ImageNet ILSVRC-2012 classification dataset. Code is available at https: //github.com/huawei-noah/ghostnet.

We propose an efficient and unified framework, namely ThiNet, to simultaneously accelerate and compress CNN models in both training and inference stages. We focus on the filter level pruning, i.e., the whole filter would be discarded if it is less important. Our method does not change the original network structure, thus it can be perfectly supported by any off-the-shelf deep learning libraries. We formally establish filter pruning as an optimization problem, and reveal that we need to prune filters based on statistics information computed from its next layer, not the current layer, which differentiates ThiNet from existing methods. Experimental results demonstrate the effectiveness of this strategy, which has advanced the state-of-the-art. We also show the performance of ThiNet on ILSVRC-12 benchmark. ThiNet achieves 3.31× FLOPs reduction and 16.63× compression on VGG-16, with only 0.52% top-5 accuracy drop. Similar experiments with ResNet-50 reveal that even for a compact network, ThiNet can also reduce more than half of the parameters and FLOPs, at the cost of roughly 1% top-5 accuracy drop. Moreover, the original VGG-16 model can be further pruned into a very small model with only 5.05MB model size, preserving AlexNet level accuracy but showing much stronger generalization ability.

深度学习技术在各种任务中都表现出了出色的有效性，并且深度学习具有推进多种应用程序（包括在边缘计算中）的潜力，其中将深层模型部署在边缘设备上，以实现即时的数据处理和响应。一个关键的挑战是，虽然深层模型的应用通常会产生大量的内存和计算成本，但Edge设备通常只提供非常有限的存储和计算功能，这些功能可能会在各个设备之间差异很大。这些特征使得难以构建深度学习解决方案，以释放边缘设备的潜力，同时遵守其约束。应对这一挑战的一种有希望的方法是自动化有效的深度学习模型的设计，这些模型轻巧，仅需少量存储，并且仅产生低计算开销。该调查提供了针对边缘计算的深度学习模型设计自动化技术的全面覆盖。它提供了关键指标的概述和比较，这些指标通常用于量化模型在有效性，轻度和计算成本方面的水平。然后，该调查涵盖了深层设计自动化技术的三类最新技术：自动化神经体系结构搜索，自动化模型压缩以及联合自动化设计和压缩。最后，调查涵盖了未来研究的开放问题和方向。

In this paper, we introduce a new channel pruning method to accelerate very deep convolutional neural networks. Given a trained CNN model, we propose an iterative two-step algorithm to effectively prune each layer, by a LASSO regression based channel selection and least square reconstruction. We further generalize this algorithm to multi-layer and multi-branch cases. Our method reduces the accumulated error and enhance the compatibility with various architectures. Our pruned VGG-16 achieves the state-of-the-art results by 5× speed-up along with only 0.3% increase of error. More importantly, our method is able to accelerate modern networks like ResNet, Xception and suffers only 1.4%, 1.0% accuracy loss under 2× speedup respectively, which is significant. Code has been made publicly available 1 .

由于其实现的实际加速，过滤器修剪已广泛用于神经网络压缩。迄今为止，大多数现有滤波器修剪工作探索过滤器通过使用通道内信息的重要性。在本文中，从频道间透视开始，我们建议使用信道独立性进行有效的滤波器修剪，该指标测量不同特征映射之间的相关性。较少独立的特征映射被解释为包含较少有用的信息$ / $知识，因此可以修剪其相应的滤波器而不会影响模型容量。我们在过滤器修剪的背景下系统地调查了渠道独立性的量化度量，测量方案和敏感性$ / $可靠性。我们对各种数据集不同模型的评估结果显示了我们方法的卓越性能。值得注意的是，在CIFAR-10数据集上，我们的解决方案可以分别为基线Resnet-56和Resnet-110型号的0.75 \％$ 0.94 \％$ 0.94 \％。模型大小和拖鞋减少了42.8 \％$和$ 47.4 \％$（for Resnet-56）和48.3 \％$ 48.3 \％$ 52.1 \％$（for resnet-110）。在ImageNet DataSet上，我们的方法可以分别达到40.8 \％$ 44.8 \％$ 74.8 \％$ 0.15 \％$ 0.15 \％$ 0.15美元的准确性。该代码可在https://github.com/eclipsess/chip_neurivs2021上获得。

网络压缩对于使深网的效率更高，更快且可推广到低端硬件至关重要。当前的网络压缩方法有两个开放问题：首先，缺乏理论框架来估计最大压缩率；其次，有些层可能会过多地进行，从而导致网络性能大幅下降。为了解决这两个问题，这项研究提出了一种基于梯度矩阵分析方法，以估计最大网络冗余。在最大速率的指导下，开发了一种新颖而有效的层次网络修剪算法，以最大程度地凝结神经元网络结构而无需牺牲网络性能。进行实质性实验以证明新方法修剪几个高级卷积神经网络（CNN）体系结构的功效。与现有的修剪方法相比，拟议的修剪算法实现了最先进的性能。与其他方法相比，在相同或相似的压缩比下，新方法提供了最高的网络预测准确性。

由于存储器和计算资源有限，部署在移动设备上的卷积神经网络（CNNS）是困难的。我们的目标是通过利用特征图中的冗余来设计包括CPU和GPU的异构设备的高效神经网络，这很少在神经结构设计中进行了研究。对于类似CPU的设备，我们提出了一种新颖的CPU高效的Ghost（C-Ghost）模块，以生成从廉价操作的更多特征映射。基于一组内在的特征映射，我们使用廉价的成本应用一系列线性变换，以生成许多幽灵特征图，可以完全揭示内在特征的信息。所提出的C-Ghost模块可以作为即插即用组件，以升级现有的卷积神经网络。 C-Ghost瓶颈旨在堆叠C-Ghost模块，然后可以轻松建立轻量级的C-Ghostnet。我们进一步考虑GPU设备的有效网络。在建筑阶段的情况下，不涉及太多的GPU效率（例如，深度明智的卷积），我们建议利用阶段明智的特征冗余来制定GPU高效的幽灵（G-GHOST）阶段结构。舞台中的特征被分成两个部分，其中使用具有较少输出通道的原始块处理第一部分，用于生成内在特征，另一个通过利用阶段明智的冗余来生成廉价的操作。在基准测试上进行的实验证明了所提出的C-Ghost模块和G-Ghost阶段的有效性。 C-Ghostnet和G-Ghostnet分别可以分别实现CPU和GPU的准确性和延迟的最佳权衡。代码可在https://github.com/huawei-noah/cv-backbones获得。

The mainstream approach for filter pruning is usually either to force a hard-coded importance estimation upon a computation-heavy pretrained model to select "important" filters, or to impose a hyperparameter-sensitive sparse constraint on the loss objective to regularize the network training. In this paper, we present a novel filter pruning method, dubbed dynamic-coded filter fusion (DCFF), to derive compact CNNs in a computation-economical and regularization-free manner for efficient image classification. Each filter in our DCFF is firstly given an inter-similarity distribution with a temperature parameter as a filter proxy, on top of which, a fresh Kullback-Leibler divergence based dynamic-coded criterion is proposed to evaluate the filter importance. In contrast to simply keeping high-score filters in other methods, we propose the concept of filter fusion, i.e., the weighted averages using the assigned proxies, as our preserved filters. We obtain a one-hot inter-similarity distribution as the temperature parameter approaches infinity. Thus, the relative importance of each filter can vary along with the training of the compact CNN, leading to dynamically changeable fused filters without both the dependency on the pretrained model and the introduction of sparse constraints. Extensive experiments on classification benchmarks demonstrate the superiority of our DCFF over the compared counterparts. For example, our DCFF derives a compact VGGNet-16 with only 72.77M FLOPs and 1.06M parameters while reaching top-1 accuracy of 93.47% on CIFAR-10. A compact ResNet-50 is obtained with 63.8% FLOPs and 58.6% parameter reductions, retaining 75.60% top-1 accuracy on ILSVRC-2012. Our code, narrower models and training logs are available at https://github.com/lmbxmu/DCFF.

在过去几年中，神经网络的性能在越来越多的浮点操作（拖鞋）的成本上显着提高。但是，当计算资源有限时，更多的拖鞋可能是一个问题。作为解决这个问题的尝试，修剪过滤器是一种常见的解决方案，但大多数现有的修剪方法不有效地保持模型精度，因此需要大量的芬降时期。在本文中，我们提出了一种自动修剪方法，该方法学习保存的神经元以保持模型精度，同时将絮凝到预定目标。为了完成这项任务，我们介绍了一种可训练的瓶颈，只需要一个单一的单一时期，只需要一个数据集的25.6％（Cifar-10）或7.49％（ILSVRC2012）来了解哪些过滤器。在各种架构和数据集上的实验表明，该方法不仅可以在修剪后保持精度，而且在FineTuning之后也优越现有方法。我们在Reset-50上达到了52.00％的拖鞋，在ILSVRC2012上的灌溉后的前1个精度为47.51％，最先进的（SOTA）精度为76.63％。代码可用（链接匿名审核）。

在本文中，我们提出了用于卷积神经网络的可分散的信道稀疏性搜索（DCS）。与需要用户手动设置每个卷积层的紫星比的传统信道修剪算法不同，DCSS自动搜索稀疏的最佳组合。灵感来自可怜的架构搜索（飞镖），我们从连续放松中汲取课程，并利用梯度信息来平衡计算成本和指标。由于直接应用飞镖方案引起形状不匹配和过度的记忆消耗，因此在过滤器内引入一种名为重量共享的新技术。这种技术优雅地消除了具有可忽略额外资源的形状不匹配的问题。我们不仅开展全面的实验，不仅是图像分类，还可以找到包括语义分割和图像超分辨率的粒度任务，以验证DCSS的有效性。与以前的网络修剪方法相比，DCSS实现了图像分类的最先进结果。语义分割和图像超分辨率的实验结果表明，特定于任务特定搜索的性能比转移超薄模型实现了更好的性能，展示了广泛的适用性和高效率的DCSS。