在训练之前对神经网络进行修剪不仅会压缩原始模型，还可以加速其具有实质性应用值的网络培训阶段。当前的工作着重于细粒修剪，该修剪使用指标来计算重量筛查的重量评分，并从初始的单阶修剪到迭代修剪。通过这些作品，我们认为可以将网络修剪总结为权重的表达力传递过程，其中保留权重将从被删除的力量中占据表达力，以维持原始网络的性能。为了实现最佳的表达力调度，我们在训练名为神经网络Pannaging之前提出了一种修剪计划，该方案通过多指数和多进程步骤指导表达力转移，并设计一种基于强化学习以自动化过程的平移代理。实验结果表明，平平在训练方法之前的性能优于各种可用的修剪。

深度加强学习（RL）是解决复杂的现实问题的强大框架。在框架中使用的大型神经网络传统上与更好的泛化能力相关，但它们的增加的大小需要广泛的培训持续时间，大量硬件资源和较长推理时间的缺点。解决这个问题的一种方法是修剪神经网络，只留下必要的参数。用于在固定数据分布的应用中施加稀疏性的最先进的并发修剪技术。但是，他们尚未在RL的背景下大大探索。我们缩小了RL和单次修剪技术之间的差距，并将一般修剪方法呈现给离线RL。在RL培训开始之前，我们利用固定数据集进行修剪神经网络。然后，我们运行不同网络稀疏度水平的实验，并评估连续控制任务中的初始化技术修剪的有效性。我们的结果表明，随着95％的网络权重修剪，离线-RL算法仍然可以在我们的大部分实验中保持性能。据我们所知，没有先前的工作，利用在这种高水平的稀疏性的RL保留的性能中进行修剪。此外，在未改变学习目标的情况下，可以在任何现有的离线-RL算法中容易地集成到任何现有的离线RL算法中。

修剪是稀疏深神经网络的任务，最近受到了越来越多的关注。尽管最先进的修剪方法提取了高度稀疏的模型，但它们忽略了两个主要挑战：（1）寻找这些稀疏模型的过程通常非常昂贵； （2）非结构化的修剪在GPU记忆，训练时间或碳排放方面没有提供好处。我们提出了通过梯度流量保存（早期CROP）提出的早期压缩，该压缩在训练挑战（1）的培训（1）中有效提取最先进的稀疏模型，并且可以以结构化的方式应用来应对挑战（2）。这使我们能够在商品GPU上训练稀疏的网络，该商品GPU的密集版本太大，从而节省了成本并减少了硬件要求。我们从经验上表明，早期杂交的表现优于许多任务（包括分类，回归）和域（包括计算机视觉，自然语言处理和增强学习）的丰富基线。早期杂交导致准确性与密集训练相当，同时超过修剪基线。

Pruning refers to the elimination of trivial weights from neural networks. The sub-networks within an overparameterized model produced after pruning are often called Lottery tickets. This research aims to generate winning lottery tickets from a set of lottery tickets that can achieve similar accuracy to the original unpruned network. We introduce a novel winning ticket called Cyclic Overlapping Lottery Ticket (COLT) by data splitting and cyclic retraining of the pruned network from scratch. We apply a cyclic pruning algorithm that keeps only the overlapping weights of different pruned models trained on different data segments. Our results demonstrate that COLT can achieve similar accuracies (obtained by the unpruned model) while maintaining high sparsities. We show that the accuracy of COLT is on par with the winning tickets of Lottery Ticket Hypothesis (LTH) and, at times, is better. Moreover, COLTs can be generated using fewer iterations than tickets generated by the popular Iterative Magnitude Pruning (IMP) method. In addition, we also notice COLTs generated on large datasets can be transferred to small ones without compromising performance, demonstrating its generalizing capability. We conduct all our experiments on Cifar-10, Cifar-100 & TinyImageNet datasets and report superior performance than the state-of-the-art methods.

Model compression is a critical technique to efficiently deploy neural network models on mobile devices which have limited computation resources and tight power budgets. Conventional model compression techniques rely on hand-crafted heuristics and rule-based policies that require domain experts to explore the large design space trading off among model size, speed, and accuracy, which is usually sub-optimal and time-consuming. In this paper, we propose AutoML for Model Compression (AMC) which leverage reinforcement learning to provide the model compression policy. This learning-based compression policy outperforms conventional rule-based compression policy by having higher compression ratio, better preserving the accuracy and freeing human labor. Under 4× FLOPs reduction, we achieved 2.7% better accuracy than the handcrafted model compression policy for VGG-16 on ImageNet. We applied this automated, push-the-button compression pipeline to MobileNet and achieved 1.81× speedup of measured inference latency on an Android phone and 1.43× speedup on the Titan XP GPU, with only 0.1% loss of ImageNet Top-1 accuracy.

信道修剪中最有效的方法之一是根据每个神经元的重要性来修剪。然而，测量每个神经元的重要性是NP难题。以前的作品通过考虑单层或多个连续的神经元层的统计来修剪。这些作品无法消除不同数据对重建错误模型的影响，并且目前没有工作证明参数的绝对值可以直接用作判断权重的重要性的基础。一种更合理的方法是消除准确测量影响力的批量数据之间的差异。在本文中，我们建议使用集合学习来培训不同批量数据的模型，并使用影响功能（来自强大的统计数据的经典技术）来学习算法跟踪模型的预测并返回其训练参数梯度，使其返回其训练参数梯度，使其返回其培训参数梯度，使其返回其培训参数梯度，使其返回其培训参数梯度，使其返回其训练参数梯度我们可以在预测过程中确定我们称之为“影响”的每个参数的责任。此外，我们理论上证明了深度网络的后传播是权重的影响函数的一阶泰勒近似。我们执行广泛的实验，以证明使用集合学习的思想基于影响功能的修剪将比仅关注误差重建更有效。 CIFAR的实验表明，影响修剪达到最先进的结果。

轻量级模型设计已成为应用深度学习技术的重要方向，修剪是实现模型参数和拖鞋的大量减少的有效均值。现有的神经网络修剪方法主要从参数的重要性开始，以及设计参数评估度量来迭代地执行参数修剪。这些方法不是从模型拓扑的角度研究的，可能是有效但不高效的，并且需要完全不同的不同数据集修剪。在本文中，我们研究了神经网络的图形结构，并提出了常规的基于图的修剪（RGP）来执行单次神经网络修剪。我们生成常规图，将图的节点度值设置为满足修剪比率，并通过将边缘交换以获得最佳边缘分布来降低曲线图的平均最短路径长度。最后，将获得的图形映射到神经网络结构中以实现修剪。实验表明，曲线图的平均最短路径长度与相应神经网络的分类精度负相关，所提出的RGP显示出强的精度保持能力，具有极高的参数减少（超过90％）和拖鞋（更多超过90％）。

This paper presents a method for adding multiple tasks to a single deep neural network while avoiding catastrophic forgetting. Inspired by network pruning techniques, we exploit redundancies in large deep networks to free up parameters that can then be employed to learn new tasks. By performing iterative pruning and network re-training, we are able to sequentially "pack" multiple tasks into a single network while ensuring minimal drop in performance and minimal storage overhead. Unlike prior work that uses proxy losses to maintain accuracy on older tasks, we always optimize for the task at hand. We perform extensive experiments on a variety of network architectures and largescale datasets, and observe much better robustness against catastrophic forgetting than prior work. In particular, we are able to add three fine-grained classification tasks to a single ImageNet-trained VGG-16 network and achieve accuracies close to those of separately trained networks for each task. Code available at https://github.com/ arunmallya/packnet

Compressing neural network architectures is important to allow the deployment of models to embedded or mobile devices, and pruning and quantization are the major approaches to compress neural networks nowadays. Both methods benefit when compression parameters are selected specifically for each layer. Finding good combinations of compression parameters, so-called compression policies, is hard as the problem spans an exponentially large search space. Effective compression policies consider the influence of the specific hardware architecture on the used compression methods. We propose an algorithmic framework called Galen to search such policies using reinforcement learning utilizing pruning and quantization, thus providing automatic compression for neural networks. Contrary to other approaches we use inference latency measured on the target hardware device as an optimization goal. With that, the framework supports the compression of models specific to a given hardware target. We validate our approach using three different reinforcement learning agents for pruning, quantization and joint pruning and quantization. Besides proving the functionality of our approach we were able to compress a ResNet18 for CIFAR-10, on an embedded ARM processor, to 20% of the original inference latency without significant loss of accuracy. Moreover, we can demonstrate that a joint search and compression using pruning and quantization is superior to an individual search for policies using a single compression method.

Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy. In this work, we make several surprising observations which contradict common beliefs. For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights. For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch. Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods. We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin, 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization. * Equal contribution. † Work done while visiting UC Berkeley.

近年来，稀疏神经网络的使用迅速增长，尤其是在计算机视觉中。它们的吸引力在很大程度上源于培训和存储所需的参数数量以及学习效率的提高。有些令人惊讶的是，很少有努力探索他们在深度强化学习中的使用（DRL）。在这项工作中，我们进行了系统的调查，以在各种DRL代理和环境上应用许多现有的稀疏培训技术。我们的结果证实了计算机视觉域中稀疏训练的发现 - 稀疏网络在DRL域中对相同的参数计数的稀疏网络表现更好。我们提供了有关DRL中各种组件如何受到稀疏网络的影响的详细分析，并通过建议有希望的途径提高稀疏训练方法的有效性以及推进其在DRL中的使用来结论。

现代深度神经网络往往太大而无法在许多实际情况下使用。神经网络修剪是降低这种模型的大小的重要技术和加速推断。Gibbs修剪是一种表达和设计神经网络修剪方法的新框架。结合统计物理和随机正则化方法的方法，它可以同时培训和修剪网络，使得学习的权重和修剪面膜彼此很好地适应。它可用于结构化或非结构化修剪，我们为每个提出了许多特定方法。我们将拟议的方法与许多当代神经网络修剪方法进行比较，发现Gibbs修剪优于它们。特别是，我们通过CIFAR-10数据集来实现修剪Reset-56的新型最先进的结果。

Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task.

重量修剪是一种有效的模型压缩技术，可以解决在移动设备上实现实时深神经网络（DNN）推断的挑战。然而，由于精度劣化，难以利用硬件加速度，以及某些类型的DNN层的限制，难以降低的应用方案具有有限的应用方案。在本文中，我们提出了一般的细粒度的结构化修剪方案和相应的编译器优化，适用于任何类型的DNN层，同时实现高精度和硬件推理性能。随着使用我们的编译器优化所支持的不同层的灵活性，我们进一步探讨了确定最佳修剪方案的新问题，了解各种修剪方案的不同加速度和精度性能。两个修剪方案映射方法，一个是基于搜索，另一个是基于规则的，建议自动推导出任何给定DNN的每层的最佳修剪规则和块大小。实验结果表明，我们的修剪方案映射方法，以及一般细粒化结构修剪方案，优于最先进的DNN优化框架，最高可达2.48 $ \ times $和1.73 $ \ times $ DNN推理加速在CiFar-10和Imagenet DataSet上没有准确性损失。

Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the lottery ticket hypothesis: dense, randomly-initialized, feed-forward networks contain subnetworks (winning tickets) that-when trained in isolationreach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

深度学习技术在各种任务中都表现出了出色的有效性，并且深度学习具有推进多种应用程序（包括在边缘计算中）的潜力，其中将深层模型部署在边缘设备上，以实现即时的数据处理和响应。一个关键的挑战是，虽然深层模型的应用通常会产生大量的内存和计算成本，但Edge设备通常只提供非常有限的存储和计算功能，这些功能可能会在各个设备之间差异很大。这些特征使得难以构建深度学习解决方案，以释放边缘设备的潜力，同时遵守其约束。应对这一挑战的一种有希望的方法是自动化有效的深度学习模型的设计，这些模型轻巧，仅需少量存储，并且仅产生低计算开销。该调查提供了针对边缘计算的深度学习模型设计自动化技术的全面覆盖。它提供了关键指标的概述和比较，这些指标通常用于量化模型在有效性，轻度和计算成本方面的水平。然后，该调查涵盖了深层设计自动化技术的三类最新技术：自动化神经体系结构搜索，自动化模型压缩以及联合自动化设计和压缩。最后，调查涵盖了未来研究的开放问题和方向。

We propose a new formulation for pruning convolutional kernels in neural networks to enable efficient inference. We interleave greedy criteria-based pruning with finetuning by backpropagation-a computationally efficient procedure that maintains good generalization in the pruned network. We propose a new criterion based on Taylor expansion that approximates the change in the cost function induced by pruning network parameters. We focus on transfer learning, where large pretrained networks are adapted to specialized tasks. The proposed criterion demonstrates superior performance compared to other criteria, e.g. the norm of kernel weights or feature map activation, for pruning large CNNs after adaptation to fine-grained classification tasks (Birds-200 and Flowers-102) relaying only on the first order gradient information. We also show that pruning can lead to more than 10× theoretical reduction in adapted 3D-convolutional filters with a small drop in accuracy in a recurrent gesture classifier. Finally, we show results for the largescale ImageNet dataset to emphasize the flexibility of our approach.

由于稀疏神经网络通常包含许多零权重，因此可以在不降低网络性能的情况下潜在地消除这些不必要的网络连接。因此，设计良好的稀疏神经网络具有显着降低拖鞋和计算资源的潜力。在这项工作中，我们提出了一种新的自动修剪方法 - 稀疏连接学习（SCL）。具体地，重量被重新参数化为可培训权重变量和二进制掩模的元素方向乘法。因此，由二进制掩模完全描述网络连接，其由单位步进函数调制。理论上，从理论上证明了使用直通估计器（STE）进行网络修剪的基本原理。这一原则是STE的代理梯度应该是积极的，确保掩模变量在其最小值处收敛。在找到泄漏的Relu后，SoftPlus和Identity Stes可以满足这个原理，我们建议采用SCL的身份STE以进行离散面膜松弛。我们发现不同特征的面具梯度非常不平衡，因此，我们建议将每个特征的掩模梯度标准化以优化掩码变量训练。为了自动训练稀疏掩码，我们将网络连接总数作为我们的客观函数中的正则化术语。由于SCL不需要由网络层设计人员定义的修剪标准或超级参数，因此在更大的假设空间中探讨了网络，以实现最佳性能的优化稀疏连接。 SCL克服了现有自动修剪方法的局限性。实验结果表明，SCL可以自动学习并选择各种基线网络结构的重要网络连接。 SCL培训的深度学习模型以稀疏性，精度和减少脚波特的SOTA人类设计和自动修剪方法训练。

模型压缩是在功率和内存受限资源上部署深神网络（DNN）的必要技术。但是，现有的模型压缩方法通常依赖于人类的专业知识，并专注于参数的本地重要性，而忽略了DNN中丰富的拓扑信息。在本文中，我们提出了一种基于图神经网络（GNNS）的新型多阶段嵌入技术，以识别DNN拓扑并使用增强学习（RL）以找到合适的压缩策略。我们执行了资源约束（即失败）通道修剪，并将我们的方法与最先进的模型压缩方法进行了比较。我们评估了从典型到移动友好网络的各种模型的方法，例如Resnet家族，VGG-16，Mobilenet-V1/V2和Shufflenet。结果表明，我们的方法可以通过最低的微调成本实现更高的压缩比，但产生了出色和竞争性的表现。

初始化时（OPAI）的一次性网络修剪是降低网络修剪成本的有效方法。最近，人们越来越相信数据在OPAI中是不必要的。但是，我们通过两种代表性的OPAI方法，即剪切和掌握的消融实验获得了相反的结论。具体而言，我们发现信息数据对于增强修剪性能至关重要。在本文中，我们提出了两种新颖的方法，即判别性的单发网络修剪（DOP）和超级缝制，以通过高级视觉判别图像贴片来修剪网络。我们的贡献如下。（1）广泛的实验表明OPAI是数据依赖性的。（2）超级缝线的性能明显优于基准图像网上的原始OPAI方法，尤其是在高度压缩的模型中。