Mixed-precision quantization has been widely applied on deep neural networks (DNNs) as it leads to significantly better efficiency-accuracy tradeoffs compared to uniform quantization. Meanwhile, determining the exact precision of each layer remains challenging. Previous attempts on bit-level regularization and pruning-based dynamic precision adjustment during training suffer from noisy gradients and unstable convergence. In this work, we propose Continuous Sparsification Quantization (CSQ), a bit-level training method to search for mixed-precision quantization schemes with improved stability. CSQ stabilizes the bit-level mixed-precision training process with a bi-level gradual continuous sparsification on both the bit values of the quantized weights and the bit selection in determining the quantization precision of each layer. The continuous sparsification scheme enables fully-differentiable training without gradient approximation while achieving an exact quantized model in the end.A budget-aware regularization of total model size enables the dynamic growth and pruning of each layer's precision towards a mixed-precision quantization scheme of the desired size. Extensive experiments show CSQ achieves better efficiency-accuracy tradeoff than previous methods on multiple models and datasets.

Model quantization enables the deployment of deep neural networks under resource-constrained devices. Vector quantization aims at reducing the model size by indexing model weights with full-precision embeddings, i.e., codewords, while the index needs to be restored to 32-bit during computation. Binary and other low-precision quantization methods can reduce the model size up to 32$\times$, however, at the cost of a considerable accuracy drop. In this paper, we propose an efficient framework for ternary quantization to produce smaller and more accurate compressed models. By integrating hyperspherical learning, pruning and reinitialization, our proposed Hyperspherical Quantization (HQ) method reduces the cosine distance between the full-precision and ternary weights, thus reducing the bias of the straight-through gradient estimator during ternary quantization. Compared with existing work at similar compression levels ($\sim$30$\times$, $\sim$40$\times$), our method significantly improves the test accuracy and reduces the model size.

混合精确的深神经网络达到了硬件部署所需的能源效率和吞吐量，尤其是在资源有限的情况下，而无需牺牲准确性。但是，不容易找到保留精度的最佳每层钻头精度，尤其是在创建巨大搜索空间的大量模型，数据集和量化技术中。为了解决这一困难，最近出现了一系列文献，并且已经提出了一些实现有希望的准确性结果的框架。在本文中，我们首先总结了文献中通常使用的量化技术。然后，我们对混合精液框架进行了彻底的调查，该调查是根据其优化技术进行分类的，例如增强学习和量化技术，例如确定性舍入。此外，讨论了每个框架的优势和缺点，我们在其中呈现并列。我们最终为未来的混合精液框架提供了指南。

由于神经网络变得更加强大，因此在现实世界中部署它们的愿望是一个上升的愿望;然而，神经网络的功率和准确性主要是由于它们的深度和复杂性，使得它们难以部署，尤其是在资源受限的设备中。最近出现了神经网络量化，以满足这种需求通过降低网络的精度来降低神经网络的大小和复杂性。具有较小和更简单的网络，可以在目标硬件的约束中运行神经网络。本文调查了在过去十年中开发的许多神经网络量化技术。基于该调查和神经网络量化技术的比较，我们提出了该地区的未来研究方向。

深神经网络（DNN）的庞大计算和记忆成本通常排除了它们在资源约束设备中的使用。将参数和操作量化为较低的位精确，为神经网络推断提供了可观的记忆和能量节省，从而促进了在边缘计算平台上使用DNN。量化DNN的最新努力采用了一系列技术，包括渐进式量化，步进尺寸的适应性和梯度缩放。本文提出了一种针对边缘计算的混合精度卷积神经网络（CNN）的新量化方法。我们的方法在模型准确性和内存足迹上建立了一个新的Pareto前沿，展示了一系列量化模型，可提供低于4.3 MB的权重（WGTS。）和激活（ACTS。）。我们的主要贡献是：（i）用张量学的学习精度，（ii）WGTS的靶向梯度修饰，（i）硬件感知的异质可区分量化。和行为。为了减轻量化错误，以及（iii）多相学习时间表，以解决从更新到学习的量化器和模型参数引起的学习不稳定性。我们证明了我们的技术在Imagenet数据集上的有效性，包括高效网络lite0（例如，WGTS。的4.14MB和ACTS。以67.66％的精度）和MobilenEtV2（例如3.51MB WGTS。 ％ 准确性）。

为了以计算有效的方式部署深层模型，经常使用模型量化方法。此外，由于新的硬件支持混合的位算术操作，最近对混合精度量化（MPQ）的研究开始通过搜索网络中不同层和模块的优化位低宽，从而完全利用表示的能力。但是，先前的研究主要是在使用强化学习，神经体系结构搜索等的昂贵方案中搜索MPQ策略，或者简单地利用部分先验知识来进行位于刻度分配，这可能是有偏见和优势的。在这项工作中，我们提出了一种新颖的随机量化量化（SDQ）方法，该方法可以在更灵活，更全球优化的空间中自动学习MPQ策略，并具有更平滑的梯度近似。特别是，可区分的位宽参数（DBP）被用作相邻位意选择之间随机量化的概率因素。在获取最佳MPQ策略之后，我们将进一步训练网络使用熵感知的bin正则化和知识蒸馏。我们广泛评估了不同硬件（GPU和FPGA）和数据集的多个网络的方法。 SDQ的表现优于所有最先进的混合或单个精度量化，甚至比较低的位置量化，甚至比各种重新网络和Mobilenet家族的全精度对应物更好，这表明了我们方法的有效性和优势。

Although considerable progress has been obtained in neural network quantization for efficient inference, existing methods are not scalable to heterogeneous devices as one dedicated model needs to be trained, transmitted, and stored for one specific hardware setting, incurring considerable costs in model training and maintenance. In this paper, we study a new vertical-layered representation of neural network weights for encapsulating all quantized models into a single one. With this representation, we can theoretically achieve any precision network for on-demand service while only needing to train and maintain one model. To this end, we propose a simple once quantization-aware training (QAT) scheme for obtaining high-performance vertical-layered models. Our design incorporates a cascade downsampling mechanism which allows us to obtain multiple quantized networks from one full precision source model by progressively mapping the higher precision weights to their adjacent lower precision counterparts. Then, with networks of different bit-widths from one source model, multi-objective optimization is employed to train the shared source model weights such that they can be updated simultaneously, considering the performance of all networks. By doing this, the shared weights will be optimized to balance the performance of different quantized models, thus making the weights transferable among different bit widths. Experiments show that the proposed vertical-layered representation and developed once QAT scheme are effective in embodying multiple quantized networks into a single one and allow one-time training, and it delivers comparable performance as that of quantized models tailored to any specific bit-width. Code will be available.

Quantization has become a predominant approach for model compression, enabling deployment of large models trained on GPUs onto smaller form-factor devices for inference. Quantization-aware training (QAT) optimizes model parameters with respect to the end task while simulating quantization error, leading to better performance than post-training quantization. Approximation of gradients through the non-differentiable quantization operator is typically achieved using the straight-through estimator (STE) or additive noise. However, STE-based methods suffer from instability due to biased gradients, whereas existing noise-based methods cannot reduce the resulting variance. In this work, we incorporate exponentially decaying quantization-error-aware noise together with a learnable scale of task loss gradient to approximate the effect of a quantization operator. We show this method combines gradient scale and quantization noise in a better optimized way, providing finer-grained estimation of gradients at each weight and activation layer's quantizer bin size. Our controlled noise also contains an implicit curvature term that could encourage flatter minima, which we show is indeed the case in our experiments. Experiments training ResNet architectures on the CIFAR-10, CIFAR-100 and ImageNet benchmarks show that our method obtains state-of-the-art top-1 classification accuracy for uniform (non mixed-precision) quantization, out-performing previous methods by 0.5-1.2% absolute.

具有混合精度量化的大DNN可以实现超高压缩，同时保持高分类性能。但是，由于找到了可以引导优化过程的准确度量的挑战，与32位浮点（FP-32）基线相比，这些方法牺牲了显着性能，或者依赖于计算昂贵的迭代培训政策这需要预先训练的基线的可用性。要解决此问题，本文提出了BMPQ，一种使用位梯度来分析层敏感性的训练方法，并产生混合精度量化模型。 BMPQ需要单一的训练迭代，但不需要预先训练的基线。它使用整数线性程序（ILP）来动态调整培训期间层的精度，但经过固定的硬件预算。为了评估BMPQ的功效，我们对CiFar-10，CiFar-100和微小想象数据集的VGG16和Reset18进行了广泛的实验。与基线FP-32型号相比，BMPQ可以产生具有15.4倍的参数比特的模型，精度可忽略不计。与SOTA“在培训期间”相比，混合精确训练方案，我们的模型分别在CiFar-10，CiFar-100和微小想象中分别为2.1倍，2.2倍2.9倍，具有提高的精度高达14.54％。

随着最近在移动和边缘设备上部署神经网络模型的需求，希望提高模型对看不见的测试数据的普遍性，以及提高模型在固定点量化下的稳健性，以实现有效部署。然而，最大限度地减少培训损失在泛化和量化性能上提供了一些保证。在这项工作中，我们通过在改善模型对界限重量扰动的框架下理论上统一它们的理论上统一并最小化模型权重的稳健性并最小化了模型权重的框架的框架，同时履行泛化和量化性能。因此，我们提出了HESSIAN增强的鲁棒优化方法，以通过基于梯度的训练过程最小化Hessian特征值，同时提高泛化和量化性能。 HERO在测试准确性上高达3.8％，高度高达30％，在80％的培训标签扰动下的准确性高达30％，以及各种精度范围内的最佳训练后量化精度，包括在SGD上的高精度改善> 10％在各种数据集上的共同模型架构培训模型。

网络量化是一种有效的压缩方法，以降低模型大小和计算成本。尽管压缩比高，但训练低精度模型由于量化的离散和不可分散的性质，难以实现相当大的性能下降。最近，提出了清晰度感知最小化（SAM），以通过同时最小化损耗值和损耗曲率来改善模型的泛化性能。在本文中，我们设计了锐度感知量化（SAQ）方法来培训量化模型，从而导致更好的泛化性能。此外，由于每个层与网络的损耗和损耗锐度有不同的贡献，我们进一步设计了一种有效的方法，该方法学习配置生成器以自动确定每层的位宽度配置，鼓励平面区域的较低位，反之亦然尖锐的景观，同时促进最小值的平整度，以实现更积极的量化。对CiFar-100和Imagenet的广泛实验显示了所提出的方法的优越性。例如，我们的量化Reset-18具有55.1X比特操作（BOP）减少甚至在前1个精度方面均匀地优于0.7％。代码可在https://github.com/zhuang-group/saq获得。

Most of the existing works use projection functions for ternary quantization in discrete space. Scaling factors and thresholds are used in some cases to improve the model accuracy. However, the gradients used for optimization are inaccurate and result in a notable accuracy gap between the full precision and ternary models. To get more accurate gradients, some works gradually increase the discrete portion of the full precision weights in the forward propagation pass, e.g., using temperature-based Sigmoid function. Instead of directly performing ternary quantization in discrete space, we push full precision weights close to ternary ones through regularization term prior to ternary quantization. In addition, inspired by the temperature-based method, we introduce a re-scaling factor to obtain more accurate gradients by simulating the derivatives of Sigmoid function. The experimental results show that our method can significantly improve the accuracy of ternary quantization in both image classification and object detection tasks.

Although weight and activation quantization is an effective approach for Deep Neural Network (DNN) compression and has a lot of potentials to increase inference speed leveraging bit-operations, there is still a noticeable gap in terms of prediction accuracy between the quantized model and the full-precision model. To address this gap, we propose to jointly train a quantized, bit-operation-compatible DNN and its associated quantizers, as opposed to using fixed, handcrafted quantization schemes such as uniform or logarithmic quantization. Our method for learning the quantizers applies to both network weights and activations with arbitrary-bit precision, and our quantizers are easy to train. The comprehensive experiments on CIFAR-10 and ImageNet datasets show that our method works consistently well for various network structures such as AlexNet, VGG-Net, GoogLeNet, ResNet, and DenseNet, surpassing previous quantization methods in terms of accuracy by an appreciable margin. Code available at https://github.com/Microsoft/LQ-Nets

Inference time, model size, and accuracy are three key factors in deep model compression. Most of the existing work addresses these three key factors separately as it is difficult to optimize them all at the same time. For example, low-bit quantization aims at obtaining a faster model; weight sharing quantization aims at improving compression ratio and accuracy; and mixed-precision quantization aims at balancing accuracy and inference time. To simultaneously optimize bit-width, model size, and accuracy, we propose pruning ternary quantization (PTQ): a simple, effective, symmetric ternary quantization method. We integrate L2 normalization, pruning, and the weight decay term to reduce the weight discrepancy in the gradient estimator during quantization, thus producing highly compressed ternary weights. Our method brings the highest test accuracy and the highest compression ratio. For example, it produces a 939kb (49$\times$) 2bit ternary ResNet-18 model with only 4\% accuracy drop on the ImageNet dataset. It compresses 170MB Mask R-CNN to 5MB (34$\times$) with only 2.8\% average precision drop. Our method is verified on image classification, object detection/segmentation tasks with different network structures such as ResNet-18, ResNet-50, and MobileNetV2.

诸如BERT的预先接受的语言模型在各种自然语言处理任务中显示出显着的效果。但是，这些模型通常包含数百万个参数，这可以防止它们在资源受限设备上实际部署。已知知识蒸馏，重量修剪和量化是模型压缩中的主要方向。然而，通过知识蒸馏获得的紧凑型模型即使对于相对小的压缩比也可能遭受显着的精度下降。另一方面，只有少数量化尝试专门用于自然语言处理任务。它们患有小的压缩比或较大的错误率，因为需要对超参数的手动设置，并且不支持微粒子组 - 方向量化。在本文中，我们提出了一种自动混合精密量化框架，设计用于伯特，其可以同时在亚组 - 明智的水平中进行量化和修剪。具体而言，我们所提出的方法利用可微分的神经结构搜索，搜索自动地分配每个子组中的参数的比例和精度，同时捕获冗余参数组。对BERT下游任务的广泛评估揭示了我们所提出的方法通过提供相同的模型尺寸来实现相同的性能。我们还通过将我们的解决方案与Ottherbert等正交方法相结合来展示获得极其轻量级模型的可行性。

While machine learning is traditionally a resource intensive task, embedded systems, autonomous navigation, and the vision of the Internet of Things fuel the interest in resource-efficient approaches. These approaches aim for a carefully chosen trade-off between performance and resource consumption in terms of computation and energy. The development of such approaches is among the major challenges in current machine learning research and key to ensure a smooth transition of machine learning technology from a scientific environment with virtually unlimited computing resources into everyday's applications. In this article, we provide an overview of the current state of the art of machine learning techniques facilitating these real-world requirements. In particular, we focus on deep neural networks (DNNs), the predominant machine learning models of the past decade. We give a comprehensive overview of the vast literature that can be mainly split into three non-mutually exclusive categories: (i) quantized neural networks, (ii) network pruning, and (iii) structural efficiency. These techniques can be applied during training or as post-processing, and they are widely used to reduce the computational demands in terms of memory footprint, inference speed, and energy efficiency. We also briefly discuss different concepts of embedded hardware for DNNs and their compatibility with machine learning techniques as well as potential for energy and latency reduction. We substantiate our discussion with experiments on well-known benchmark datasets using compression techniques (quantization, pruning) for a set of resource-constrained embedded systems, such as CPUs, GPUs and FPGAs. The obtained results highlight the difficulty of finding good trade-offs between resource efficiency and predictive performance.

深度神经网络（DNN）的记录断裂性能具有沉重的参数化，导致外部动态随机存取存储器（DRAM）进行存储。 DRAM访问的禁用能量使得在资源受限的设备上部署DNN是不普遍的，呼叫最小化重量和数据移动以提高能量效率。我们呈现SmartDeal（SD），算法框架，以进行更高成本的存储器存储/访问的较低成本计算，以便在推理和培训中积极提高存储和能量效率。 SD的核心是一种具有结构约束的新型重量分解，精心制作以释放硬件效率潜力。具体地，我们将每个重量张量分解为小基矩阵的乘积以及大的结构稀疏系数矩阵，其非零被量化为-2的功率。由此产生的稀疏和量化的DNN致力于为数据移动和重量存储而大大降低的能量，因为由于稀疏的比特 - 操作和成本良好的计算，恢复原始权重的最小开销。除了推理之外，我们采取了另一次飞跃来拥抱节能培训，引入创新技术，以解决培训时出现的独特障碍，同时保留SD结构。我们还设计专用硬件加速器，充分利用SD结构来提高实际能源效率和延迟。我们在不同的设置中对多个任务，模型和数据集进行实验。结果表明：1）应用于推理，SD可实现高达2.44倍的能效，通过实际硬件实现评估; 2）应用于培训，储存能量降低10.56倍，减少了10.56倍和4.48倍，与最先进的训练基线相比，可忽略的准确性损失。我们的源代码在线提供。

通过移除昂贵的乘法操作并将连续权重量化成低比特离散值来减少计算复杂性，与传统的神经网络相比，这是快速且节能的低比特离散值。然而，现有的换档网络对重量初始化敏感，并且还产生由消失梯度和重量率冻结问题引起的降级性能。为了解决这些问题，我们提出了一种低点重新参数化，这是一种用于训练低位换档网络的新技术。我们的方法以符号稀疏偏移3倍的方式分解离散参数。以这种方式，它有效地学习了一个低比特网络，其权重动力学类似于全精密网络并对重量初始化不敏感。我们所提出的培训方法推动移位神经网络的界限，并以在想象中的前1个精度方面显示出3位换档网络。

用于压缩神经网络的非均匀量化策略通常实现的性能比其对应于对应物，即统一的策略，因为其优越的代表性能力。然而，许多非均匀量化方法在实现不均匀量化的权重/激活时忽略了复杂的投影过程，这在硬件部署中引起了不可忽略的时间和空间开销。在这项研究中，我们提出了非均匀致均匀的量化（N2UQ），一种方法，其能够保持非均匀方法的强表示能力，同时硬件友好且有效地作为模型推理的均匀量化。我们通过学习灵活的等距输入阈值来实现这一目标，以更好地拟合潜在的分布，同时将这些实值输入量化为等距输出电平。要使用可学习的输入阈值训练量化网络，我们将广义直通估计器（G-STE）介绍，用于难以应答的后向衍生计算W.r.t.阈值参数。此外，我们考虑熵保持正则化，以进一步降低重量量化的信息损失。即使在这种不利约束的施加均匀量化的重量和激活的情况下，我们的N2UQ也经历了最先进的非均匀量化方法，在想象中达到了0.7〜1.8％，展示了N2UQ设计的贡献。代码将公开可用。

深度学习技术在各种任务中都表现出了出色的有效性，并且深度学习具有推进多种应用程序（包括在边缘计算中）的潜力，其中将深层模型部署在边缘设备上，以实现即时的数据处理和响应。一个关键的挑战是，虽然深层模型的应用通常会产生大量的内存和计算成本，但Edge设备通常只提供非常有限的存储和计算功能，这些功能可能会在各个设备之间差异很大。这些特征使得难以构建深度学习解决方案，以释放边缘设备的潜力，同时遵守其约束。应对这一挑战的一种有希望的方法是自动化有效的深度学习模型的设计，这些模型轻巧，仅需少量存储，并且仅产生低计算开销。该调查提供了针对边缘计算的深度学习模型设计自动化技术的全面覆盖。它提供了关键指标的概述和比较，这些指标通常用于量化模型在有效性，轻度和计算成本方面的水平。然后，该调查涵盖了深层设计自动化技术的三类最新技术：自动化神经体系结构搜索，自动化模型压缩以及联合自动化设计和压缩。最后，调查涵盖了未来研究的开放问题和方向。