深入学习的成功已归功于大量数据培训大量的过度公正模型。随着这种趋势的继续，模型培训已经过分昂贵，需要获得强大的计算系统来培训最先进的网络。一大堆研究已经致力于通过各种模型压缩技术解决训练的迭代的成本，如修剪和量化。花费较少的努力来定位迭代的数量。以前的工作，例如忘记得分和宏伟/ el2n分数，通过识别完整数据集中的重要样本并修剪剩余的样本来解决这个问题，从而减少每时代的迭代。虽然这些方法降低了训练时间，但它们在训练前使用昂贵的静态评分算法。在计入得分机制时，通常会增加总运行时间。在这项工作中，我们通过动态数据修剪算法解决了这种缺点。令人惊讶的是，我们发现均匀的随机动态修剪可以以积极的修剪速率更优于现有的工作。我们将其归因于存在“有时”样本 - 对学习决策边界很重要的点，只有一些培训时间。为了更好地利用有时样本的微妙性，我们提出了基于加强学习技术的两种算法，以动态修剪样本并实现比随机动态方法更高的准确性。我们针对全数据集基线和CIFAR-10和CIFAR-100上的先前工作测试所有方法，我们可以将培训时间降低到2倍，而无明显的性能损失。我们的结果表明，数据修剪应理解为与模型的训练轨迹密切相关的动态过程，而不是仅基于数据集的静态步骤。

对网络规模数据进行培训可能需要几个月的时间。但是，在已经学习或不可学习的冗余和嘈杂点上浪费了很多计算和时间。为了加速训练，我们引入了可减少的持有损失选择（Rho-loss），这是一种简单但原则上的技术，它大致选择了这些训练点，最大程度地减少了模型的概括损失。结果，Rho-loss减轻了现有数据选择方法的弱点：优化文献中的技术通常选择“硬损失”（例如，高损失），但是这种点通常是嘈杂的（不可学习）或更少的任务与任务相关。相反，课程学习优先考虑“简单”的积分，但是一旦学习，就不必对这些要点进行培训。相比之下，Rho-Loss选择了可以学习的点，值得学习的，尚未学习。与先前的艺术相比，Rho-loss火车的步骤要少得多，可以提高准确性，并加快对广泛的数据集，超参数和体系结构（MLP，CNNS和BERT）的培训。在大型Web绑带图像数据集服装1M上，与统一的数据改组相比，步骤少18倍，最终精度的速度少2％。

主动学习（AL）是一个有希望的ML范式，有可能解析大型未标记数据并有助于降低标记数据可能令人难以置信的域中的注释成本。最近提出的基于神经网络的AL方法使用不同的启发式方法来实现这一目标。在这项研究中，我们证明，在相同的实验环境下，不同类型的AL算法（基于不确定性，基于多样性和委员会）产生了与随机采样基线相比的不一致增长。通过各种实验，控制了随机性来源，我们表明，AL算法实现的性能指标方差可能会导致与先前报道的结果不符的结果。我们还发现，在强烈的正则化下，AL方法在各种实验条件下显示出比随机采样基线的边缘或没有优势。最后，我们以一系列建议进行结论，以了解如何使用新的AL算法评估结果，以确保在实验条件下的变化下结果可再现和健壮。我们共享我们的代码以促进AL评估。我们认为，我们的发现和建议将有助于使用神经网络在AL中进行可重复的研究。我们通过https://github.com/prateekmunjal/torchal开源代码

Many recent works on understanding deep learning try to quantify how much individual data instances influence the optimization and generalization of a model, either by analyzing the behavior of the model during training or by measuring the performance gap of the model when the instance is removed from the dataset. Such approaches reveal characteristics and importance of individual instances, which may provide useful information in diagnosing and improving deep learning. However, most of the existing works on data valuation require actual training of a model, which often demands high-computational cost. In this paper, we provide a training-free data valuation score, called complexity-gap score, which is a data-centric score to quantify the influence of individual instances in generalization of two-layer overparameterized neural networks. The proposed score can quantify irregularity of the instances and measure how much each data instance contributes in the total movement of the network parameters during training. We theoretically analyze and empirically demonstrate the effectiveness of the complexity-gap score in finding 'irregular or mislabeled' data instances, and also provide applications of the score in analyzing datasets and diagnosing training dynamics.

广泛观察到的神经缩放定律，其中错误是训练集大小，模型大小或两者兼而有之的误差，从而促进了深度学习的实质性改进。但是，仅通过缩放来进行这些改进就需要计算和能源成本相当大。在这里，我们专注于数据集大小的错误缩放，并展示在理论和实践中如何超越幂律的扩展，并将其减少到指数缩放，如果我们可以访问高质量的数据修剪指标，以将顺序排名为应该丢弃哪些培训示例以实现任何修剪的数据集大小。然后，我们通过经验修剪的数据集大小来测试这一新的指数缩放预测，并且实际上观察到了在CIFAR-10，SVHN和Imagenet训练的重新NET上的功率定律缩放性能。鉴于找到高质量的修剪指标的重要性，我们对ImageNet上十个不同的数据修剪指标进行了第一个大规模的基准测试研究。我们发现大多数现有的高性能指标尺寸较差，而对于ImageNet来说，最佳尺度是计算密集型的，并且需要为每个图像标签。因此，我们开发了一种新的简单，便宜和可扩展的自我监督的修剪指标，该指标与最佳监督指标相当。总体而言，我们的工作表明，发现良好的数据指标可能会为可行的途径提供可行的途径，从而大大改善神经缩放法律，从而降低现代深度学习的资源成本。

随着深入学习更加标签的目标，越来越多的论文已经研究了深度模型的主动学习（AL）。然而，普遍存在的实验设置中存在许多问题，主要源于缺乏统一的实施和基准。当前文献中的问题包括有时对不同AL算法的性能的矛盾观察，意外排除重要的概括方法，如数据增强和SGD进行优化，缺乏对al的标签效率等评价方面的研究，并且很少或没有在Al优于随机采样（RS）的情况下的清晰度。在这项工作中，我们通过我们的新开源AL Toolkit Distil在图像分类的背景下统一重新实现了最先进的AL算法，我们仔细研究了这些问题作为有效评估的方面。在积极的方面，我们表明AL技术为2美元至4倍以上$ 4 \倍。与使用数据增强相比，与卢比相比，高效。令人惊讶的是，当包括数据增强时，在使用徽章，最先进的方法，在简单的不确定性采样中不再存在一致的增益。然后，我们仔细分析现有方法如何具有不同数量的冗余和每个类的示例。最后，我们为AL从业者提供了几次见解，以考虑在将来的工作中考虑，例如Al批量大小的效果，初始化的效果，在每一轮中再培训模型的重要性以及其他见解。

关于稀疏神经网络训练（稀疏训练）的最新研究表明，通过从头开始训练本质上稀疏的神经网络可以实现绩效和效率之间的令人信服的权衡。现有的稀疏训练方法通常努力在一次跑步中找到最佳的稀疏子网，而无需涉及任何昂贵的密集或预训练步骤。例如，作为最突出的方向之一，动态稀疏训练（DST）能够通过在训练过程中迭代发展稀疏拓扑来实现竞争性训练的竞争性能。在本文中，我们认为最好分配有限的资源来创建多个低损失的稀疏子网并将其超级置于更强的基因，而不是完全分配所有资源以找到单个子网络。为了实现这一目标，需要两个Desiderata：（1）在一个培训过程中有效生产许多低损失的子网，即所谓的廉价门票，仅限于用于密集培训的标准培训时间； （2）将这些廉价的门票有效地超级为一个更强的子网，而无需超越约束参数预算。为了证实我们的猜想，我们提出了一种新颖的稀疏训练方法，称为\ textbf {sup-tickets}，可以在单个稀疏到较小的训练过程中同时满足上述两个desiderata。在CIFAR-10/100和Imagenet上的各种现代体系结构中，我们表明，SUP-Tickets与现有的稀疏训练方法无缝集成，并显示出一致的性能提高。

Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy. In this work, we make several surprising observations which contradict common beliefs. For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights. For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch. Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods. We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin, 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization. * Equal contribution. † Work done while visiting UC Berkeley.

Structural pruning of neural network parameters reduces computation, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the first and secondorder Taylor expansions to approximate a filter's contribution. Both methods scale consistently across any network layer without requiring per-layer sensitivity analysis and can be applied to any kind of layer, including skip connections. For modern networks trained on ImageNet, we measured experimentally a high (>93%) correlation between the contribution computed by our methods and a reliable estimate of the true importance. Pruning with the proposed methods leads to an improvement over state-ofthe-art in terms of accuracy, FLOPs, and parameter reduction. On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0.02% in the top-1 accuracy on ImageNet. Code is available at https://github.com/NVlabs/Taylor_pruning.

Many applications require sparse neural networks due to space or inference time restrictions. There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-tosparse training methods. Our method updates the topology of the sparse network during training by using parameter magnitudes and infrequent gradient calculations. We show that this approach requires fewer floating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques. We demonstrate state-of-the-art sparse training results on a variety of networks and datasets, including ResNet-50, MobileNets on Imagenet-2012, and RNNs on WikiText-103. Finally, we provide some insights into why allowing the topology to change during the optimization can overcome local minima encountered when the topology remains static * .

有效地近似损失函数的局部曲率信息是用于深神经网络的优化和压缩的关键工具。然而，大多数现有方法近似二阶信息具有高计算或存储成本，这可以限制其实用性。在这项工作中，我们调查矩阵，用于估计逆象征的矢量产品（IHVPS）的矩阵线性时间方法，因为当Hessian可以近似为乘语 - 一个矩阵的总和时，如Hessian的经典近似由经验丰富的Fisher矩阵。我们提出了两个新的算法作为称为M-FAC的框架的一部分：第一个算法朝着网络压缩量身定制，如果Hessian给出了M $等级的总和，则可以计算Dimension $ D $的IHVP。 ，使用$ O（DM ^ 2）$预压制，$ O（DM）$代价计算IHVP，并查询逆Hessian的任何单个元素的费用$ O（m）$。第二算法针对优化设置，我们希望在反向Hessian之间计算产品，估计在优化步骤的滑动窗口和给定梯度方向上，根据预先说明的SGD所需的梯度方向。我们为计算IHVP和OHVP和O（DM + M ^ 3）$ of $ o（dm + m ^ 2）$提供算法，以便从滑动窗口添加或删除任何渐变。这两种算法产生最先进的结果，用于网络修剪和相对于现有二阶方法的计算开销的优化。在[9]和[17]可用实现。

In semi-supervised representation learning frameworks, when the number of labelled data is very scarce, the quality and representativeness of these samples become increasingly important. Existing literature on semi-supervised learning randomly sample a limited number of data points for labelling. All these labelled samples are then used along with the unlabelled data throughout the training process. In this work, we ask two important questions in this context: (1) does it matter which samples are selected for labelling? (2) does it matter how the labelled samples are used throughout the training process along with the unlabelled data? To answer the first question, we explore a number of unsupervised methods for selecting specific subsets of data to label (without prior knowledge of their labels), with the goal of maximizing representativeness w.r.t. the unlabelled set. Then, for our second line of inquiry, we define a variety of different label injection strategies in the training process. Extensive experiments on four popular datasets, CIFAR-10, CIFAR-100, SVHN, and STL-10, show that unsupervised selection of samples that are more representative of the entire data improves performance by up to ~2% over the existing semi-supervised frameworks such as MixMatch, ReMixMatch, FixMatch and others with random sample labelling. We show that this boost could even increase to 7.5% for very few-labelled scenarios. However, our study shows that gradually injecting the labels throughout the training procedure does not impact the performance considerably versus when all the existing labels are used throughout the entire training.

Neural network pruning-the task of reducing the size of a network by removing parameters-has been the subject of a great deal of work in recent years. We provide a meta-analysis of the literature, including an overview of approaches to pruning and consistent findings in the literature. After aggregating results across 81 papers and pruning hundreds of models in controlled conditions, our clearest finding is that the community suffers from a lack of standardized benchmarks and metrics. This deficiency is substantial enough that it is hard to compare pruning techniques to one another or determine how much progress the field has made over the past three decades. To address this situation, we identify issues with current practices, suggest concrete remedies, and introduce ShrinkBench, an open-source framework to facilitate standardized evaluations of pruning methods. We use ShrinkBench to compare various pruning techniques and show that its comprehensive evaluation can prevent common pitfalls when comparing pruning methods.

网络修剪是一种广泛使用的技术，用于有效地压缩深神经网络，几乎没有在推理期间在性能下降低。迭代幅度修剪（IMP）是由几种迭代训练和修剪步骤组成的网络修剪的最熟悉的方法之一，其中在修剪后丢失了大量网络的性能，然后在随后的再培训阶段中恢复。虽然常用为基准参考，但经常认为a）通过不将稀疏纳入训练阶段来达到次优状态，b）其全球选择标准未能正确地确定最佳层面修剪速率和c）其迭代性质使它变得缓慢和不竞争。根据最近提出的再培训技术，我们通过严格和一致的实验来调查这些索赔，我们将Impr到培训期间的训练算法进行比较，评估其选择标准的建议修改，并研究实际需要的迭代次数和总培训时间。我们发现IMP与SLR进行再培训，可以优于最先进的修剪期间，没有或仅具有很少的计算开销，即全局幅度选择标准在很大程度上具有更复杂的方法，并且只有几个刷新时期在实践中需要达到大部分稀疏性与IMP的诽谤 - 与性能权衡。我们的目标既可以证明基本的进攻已经可以提供最先进的修剪结果，甚至优于更加复杂或大量参数化方法，也可以为未来的研究建立更加现实但易于可实现的基线。

学习存在于数据的背景下，但信心的概念通常集中在模型预测上，而不是标签质量上。自信学习（CL）是一种替代方法，它通过根据修剪嘈杂数据的原理来表征和识别数据集中的标签错误来重点关注标签质量，并使用概率阈值来估算噪声，并将示例排名以自信。尽管许多研究已经独立开发了这些原理，但在这里，我们将它们结合起来，建立在类似的噪声过程的基础上，以直接估计嘈杂（给定的）标签和未腐败（未知）标签之间的关节分布。这导致了广义的CL，该CL证明是一致且具有实验性能的。我们提供了足够的条件，CL准确地发现标签错误，并且CL性能超过了CIFAR数据集上使用嘈杂标签的七种近期学习方法。独特的是，CL框架不与特定的数据模式或模型耦合（例如，我们使用CL在假定的无错误MNIST数据集中查找几个标签错误，并在亚马逊评论中对文本数据进行改善的情感分类）。我们还使用Imagenet上的CL来量化本体论类重叠（例如，估计645个“导弹”图像被错误标记为其母体类“弹丸”），并通过清洁训练前清洁数据来提高模型准确性（例如，用于RESNET）。使用开源清洁行释放可以复制这些结果。

Pruning refers to the elimination of trivial weights from neural networks. The sub-networks within an overparameterized model produced after pruning are often called Lottery tickets. This research aims to generate winning lottery tickets from a set of lottery tickets that can achieve similar accuracy to the original unpruned network. We introduce a novel winning ticket called Cyclic Overlapping Lottery Ticket (COLT) by data splitting and cyclic retraining of the pruned network from scratch. We apply a cyclic pruning algorithm that keeps only the overlapping weights of different pruned models trained on different data segments. Our results demonstrate that COLT can achieve similar accuracies (obtained by the unpruned model) while maintaining high sparsities. We show that the accuracy of COLT is on par with the winning tickets of Lottery Ticket Hypothesis (LTH) and, at times, is better. Moreover, COLTs can be generated using fewer iterations than tickets generated by the popular Iterative Magnitude Pruning (IMP) method. In addition, we also notice COLTs generated on large datasets can be transferred to small ones without compromising performance, demonstrating its generalizing capability. We conduct all our experiments on Cifar-10, Cifar-100 & TinyImageNet datasets and report superior performance than the state-of-the-art methods.

主动学习（AL）算法旨在识别注释的最佳数据子集，使得深神经网络（DNN）在此标记子集上培训时可以实现更好的性能。 AL特别有影响的工业规模设置，其中数据标签成本高，从业者使用各种工具来处理，以提高模型性能。最近自我监督预测（SSP）的成功突出了利用丰富的未标记数据促进模型性能的重要性。通过将AL与SSP结合起来，我们可以使用未标记的数据，同时标记和培训特别是信息样本。在这项工作中，我们研究了Imagenet上的AL和SSP的组合。我们发现小型玩具数据集上的性能 - 文献中的典型基准设置 - 由于活动学习者选择的类不平衡样本，而不是想象中的性能。在我们测试的现有基线中，各种小型和大规​​模设置的流行AL算法未能以随机抽样优于差异。为了解决类别不平衡问题，我们提出了平衡选择（基础），这是一种简单，可伸缩的AL算法，通过选择比现有方法更加平衡样本来始终如一地始终采样。我们的代码可用于：https://github.com/zeyademam/active_learning。

虽然深度学习（DL）是渴望数据的，并且通常依靠广泛的标记数据来提供良好的性能，但主动学习（AL）通过从未标记的数据中选择一小部分样本进行标签和培训来降低标签成本。因此，近年来，在有限的标签成本/预算下，深入的积极学习（DAL）是可行的解决方案，可在有限的标签成本/预算下最大化模型性能。尽管已经开发了大量的DAL方法并进行了各种文献综述，但在公平比较设置下对DAL方法的性能评估尚未可用。我们的工作打算填补这一空白。在这项工作中，我们通过重新实现19种引用的DAL方法来构建DAL Toolkit，即Deepal+。我们调查和分类与DAL相关的作品，并构建经常使用的数据集和DAL算法的比较实验。此外，我们探讨了影响DAL功效的一些因素（例如，批处理大小，训练过程中的时期数），这些因素为研究人员设计其DAL实验或执行DAL相关应用程序提供了更好的参考。

We introduce a new setting, optimize-and-estimate structured bandits. Here, a policy must select a batch of arms, each characterized by its own context, that would allow it to both maximize reward and maintain an accurate (ideally unbiased) population estimate of the reward. This setting is inherent to many public and private sector applications and often requires handling delayed feedback, small data, and distribution shifts. We demonstrate its importance on real data from the United States Internal Revenue Service (IRS). The IRS performs yearly audits of the tax base. Two of its most important objectives are to identify suspected misreporting and to estimate the "tax gap" -- the global difference between the amount paid and true amount owed. Based on a unique collaboration with the IRS, we cast these two processes as a unified optimize-and-estimate structured bandit. We analyze optimize-and-estimate approaches to the IRS problem and propose a novel mechanism for unbiased population estimation that achieves rewards comparable to baseline approaches. This approach has the potential to improve audit efficacy, while maintaining policy-relevant estimates of the tax gap. This has important social consequences given that the current tax gap is estimated at nearly half a trillion dollars. We suggest that this problem setting is fertile ground for further research and we highlight its interesting challenges. The results of this and related research are currently being incorporated into the continual improvement of the IRS audit selection methods.

动态模型修剪是最近的方向，其允许不同的子网络中的部署过程中每个输入采样的推断。然而，当前的动态方法依赖于学习的连续通道通过诱导稀疏性损失通过正则化门控。这一提法介绍了平衡不同损失的复杂性（如任务的损失，正规化损失）。此外，基于正则化方法缺乏透明的折衷选择超参数，实现计算的预算。我们的贡献是双重的：1）分离任务和修剪培训。 2）简单的超参数选择，使训练前FLOPS减少估计。在神经科学的赫布理论的启发：“神经元一起火一起丝”，我们提出来预测基于其上一层的活化层口罩方法K过滤器。我们提出的问题，因为自监督二元分类问题。每个掩模预测模块被训练以预测，如果对数似然在当前层中的每个过滤器属于前k激活的过滤器。值k被动态地估计基于使用热图的质量的新颖标准每个输入。我们发现在几个神经结构，如VGG，RESNET和MobileNet上CIFAR和ImageNet数据集实验。在CIFAR，我们得出了类似的精度SOTA方法有15％和24％以上FLOPS减少。同样，在ImageNet，我们达到的精度低下降高达13％的改善FLOPS减少。