卷积一直是现代深层神经网络的核心运作。众所周知，可以在傅立叶变换域中实现卷积。在本文中，我们建议使用二进制块WALSH-HATAMARD变换（WHT）而不是傅里叶变换。我们使用基于WHT的二进制层来替换深度神经网络中的一些常规卷积层。我们本文利用了一维（1-D）和二维（2-D）二进制WHT。在两个1-D和2-D层中，我们计算输入特征图的二进制WHT，并使用非线性去噪该WHT域系数，该非线性通过将软阈值与TanH函数组合而获得的非线性。在去噪后，我们计算反相WHT。我们使用1d-wht来取代$ 1 \ times 1 $卷积层，2d-wht层可以取代3 $ \ times $ 3卷积层和挤压和激发层。具有可培训重量的2D-WHT层也可以在全局平均池（间隙）层之前插入以辅助致密层。通过这种方式，我们可以显着降低可训练参数的衡量参数的数量。在本文中，我们将WHT层实施到MobileNet-V2，MobileNet-V3大，并重新阅读，以显着降低参数的数量，以可忽略不计的精度损失。此外，根据我们的速度测试，2D-FWWHT层的运行大约是常规3美元3美元3美元的速度大约为19.51次较少的RAM使用率在NVIDIA Jetson Nano实验中的使用率。

随着计算机愿景任务中的神经网络的不断发展，越来越多的网络架构取得了突出的成功。作为最先进的神经网络架构之一，DenSenet捷径所有特征映射都可以解决模型深度的问题。虽然这种网络架构在低MAC（乘法和累积）上具有优异的准确性，但它需要过度推理时间。为了解决这个问题，HardNet减少了特征映射之间的连接，使得其余连接类似于谐波。然而，这种压缩方法可能导致模型精度和增加的MAC和模型大小降低。该网络架构仅减少了内存访问时间，需要改进其整体性能。因此，我们提出了一种新的网络架构，使用阈值机制来进一步优化连接方法。丢弃不同卷积层的不同数量的连接以压缩阈值中的特征映射。所提出的网络架构使用了三个数据集，CiFar-10，CiFar-100和SVHN，以评估图像分类的性能。实验结果表明，与DENSENET相比，阈值可降低推理时间高达60％，并且在这些数据集上的硬盘相比，训练速度快高达35％的训练速度和20％的误差率降低。

由于存储器和计算资源有限，部署在移动设备上的卷积神经网络（CNNS）是困难的。我们的目标是通过利用特征图中的冗余来设计包括CPU和GPU的异构设备的高效神经网络，这很少在神经结构设计中进行了研究。对于类似CPU的设备，我们提出了一种新颖的CPU高效的Ghost（C-Ghost）模块，以生成从廉价操作的更多特征映射。基于一组内在的特征映射，我们使用廉价的成本应用一系列线性变换，以生成许多幽灵特征图，可以完全揭示内在特征的信息。所提出的C-Ghost模块可以作为即插即用组件，以升级现有的卷积神经网络。 C-Ghost瓶颈旨在堆叠C-Ghost模块，然后可以轻松建立轻量级的C-Ghostnet。我们进一步考虑GPU设备的有效网络。在建筑阶段的情况下，不涉及太多的GPU效率（例如，深度明智的卷积），我们建议利用阶段明智的特征冗余来制定GPU高效的幽灵（G-GHOST）阶段结构。舞台中的特征被分成两个部分，其中使用具有较少输出通道的原始块处理第一部分，用于生成内在特征，另一个通过利用阶段明智的冗余来生成廉价的操作。在基准测试上进行的实验证明了所提出的C-Ghost模块和G-Ghost阶段的有效性。 C-Ghostnet和G-Ghostnet分别可以分别实现CPU和GPU的准确性和延迟的最佳权衡。代码可在https://github.com/huawei-noah/cv-backbones获得。

In this paper we describe a new mobile architecture, MobileNetV2, that improves the state of the art performance of mobile models on multiple tasks and benchmarks as well as across a spectrum of different model sizes. We also describe efficient ways of applying these mobile models to object detection in a novel framework we call SSDLite. Additionally, we demonstrate how to build mobile semantic segmentation models through a reduced form of DeepLabv3 which we call Mobile DeepLabv3.is based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers. The intermediate expansion layer uses lightweight depthwise convolutions to filter features as a source of non-linearity. Additionally, we find that it is important to remove non-linearities in the narrow layers in order to maintain representational power. We demonstrate that this improves performance and provide an intuition that led to this design.Finally, our approach allows decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis. We measure our performance on ImageNet [1] classification, COCO object detection [2], VOC image segmentation [3]. We evaluate the trade-offs between accuracy, and number of operations measured by multiply-adds (MAdd), as well as actual latency, and the number of parameters.

同态加密（HE），允许对加密数据（Ciphertext）进行计算，而无需首先解密，因此可以实现对云中隐私性的应用程序的安全性缓慢的卷积神经网络（CNN）推断。为了减少推理潜伏期，一种方法是将多个消息打包到单个密文中，以减少密文的数量并支持同型多态多重蓄能（HMA）操作的大量并行性。尽管HECNN的推断速度更快，但主流包装方案密集的包装（密度）和卷积包装（Convpack）仍将昂贵的旋转开销引入了昂贵的旋转开销，这延长了HECNN的推断潜伏期，以实现更深和更广泛的CNN体​​系结构。在本文中，我们提出了一种名为FFCONV的低级分解方法，该方法专门用于有效的密文填料，用于减少旋转台面和HMA操作。 FFCONV近似于低级分解卷积的A D X D卷积层，其中D X D低率卷积具有较少的通道，然后是1 x 1卷积以恢复通道。 D X D低级别卷积带有密度，导致旋转操作显着降低，而1 x 1卷积的旋转开销接近零。据我们所知，FFCONV是能够同时减少densepack和Convpack产生的旋转头顶的第一项工作，而无需将其他特殊块引入HECNN推理管道。与先前的Art Lola和Falcon相比，我们的方法分别将推理潜伏期降低了88％和21％，其精度在MNIST和CIFAR-10上具有可比的精度。

尽管胶囊网络在为视觉识别任务中定义了深度神经网络中的特征之间的位置关系，但它们是计算昂贵的并且不适合于在移动设备上运行的能力。瓶颈处于胶囊之间使用的动态路由机构的计算复杂性。另一方面，诸如Xnor-Net之类的神经网络是快速和计算的高效，但由于其在二值化过程中的信息丢失，具有相对低的精度。本文通过XNorize在CAPSFC层内的动态路由外部或内部的线性投影仪来提出新的完全连接（FC）层。具体而言，我们提出的FC层有两个版本，XNODR（Xnorizing线性投影仪外部动态路由）和XNIDR（动态路由内的xnorizing线性投影仪）。要测试其泛化，我们将它们插入MobileNet V2和Reset-50分别。在三个数据集，Mnist，CiFar-10，多方派的实验验证其有效性。我们的实验结果表明，XNODR和XNIDR都有助于网络具有高精度，具有较低的拖波和更少的参数（例如，95.32 \％的精度，在2.99M参数和311.22M拖薄的CIFAR-10上）。

We present the next generation of MobileNets based on a combination of complementary search techniques as well as a novel architecture design. MobileNetV3 is tuned to mobile phone CPUs through a combination of hardwareaware network architecture search (NAS) complemented by the NetAdapt algorithm and then subsequently improved through novel architecture advances. This paper starts the exploration of how automated search algorithms and network design can work together to harness complementary approaches improving the overall state of the art. Through this process we create two new MobileNet models for release: MobileNetV3-Large and MobileNetV3-Small which are targeted for high and low resource use cases. These models are then adapted and applied to the tasks of object detection and semantic segmentation. For the task of semantic segmentation (or any dense pixel prediction), we propose a new efficient segmentation decoder Lite Reduced Atrous Spatial Pyramid Pooling (LR-ASPP). We achieve new state of the art results for mobile classification, detection and segmentation. MobileNetV3-Large is 3.2% more accurate on ImageNet classification while reducing latency by 20% compared to MobileNetV2. MobileNetV3-Small is 6.6% more accurate compared to a MobileNetV2 model with comparable latency. MobileNetV3-Large detection is over 25% faster at roughly the same accuracy as Mo-bileNetV2 on COCO detection. MobileNetV3-Large LR-ASPP is 34% faster than MobileNetV2 R-ASPP at similar accuracy for Cityscapes segmentation.

We propose two efficient approximations to standard convolutional neural networks: Binary-Weight-Networks and XNOR-Networks. In Binary-Weight-Networks, the filters are approximated with binary values resulting in 32× memory saving. In XNOR-Networks, both the filters and the input to convolutional layers are binary. XNOR-Networks approximate convolutions using primarily binary operations. This results in 58× faster convolutional operations (in terms of number of the high precision operations) and 32× memory savings. XNOR-Nets offer the possibility of running state-of-the-art networks on CPUs (rather than GPUs) in real-time. Our binary networks are simple, accurate, efficient, and work on challenging visual tasks. We evaluate our approach on the ImageNet classification task. The classification accuracy with a Binary-Weight-Network version of AlexNet is the same as the full-precision AlexNet. We compare our method with recent network binarization methods, BinaryConnect and BinaryNets, and outperform these methods by large margins on ImageNet, more than 16% in top-1 accuracy. Our code is available at: http://allenai.org/plato/xnornet.

While machine learning is traditionally a resource intensive task, embedded systems, autonomous navigation, and the vision of the Internet of Things fuel the interest in resource-efficient approaches. These approaches aim for a carefully chosen trade-off between performance and resource consumption in terms of computation and energy. The development of such approaches is among the major challenges in current machine learning research and key to ensure a smooth transition of machine learning technology from a scientific environment with virtually unlimited computing resources into everyday's applications. In this article, we provide an overview of the current state of the art of machine learning techniques facilitating these real-world requirements. In particular, we focus on deep neural networks (DNNs), the predominant machine learning models of the past decade. We give a comprehensive overview of the vast literature that can be mainly split into three non-mutually exclusive categories: (i) quantized neural networks, (ii) network pruning, and (iii) structural efficiency. These techniques can be applied during training or as post-processing, and they are widely used to reduce the computational demands in terms of memory footprint, inference speed, and energy efficiency. We also briefly discuss different concepts of embedded hardware for DNNs and their compatibility with machine learning techniques as well as potential for energy and latency reduction. We substantiate our discussion with experiments on well-known benchmark datasets using compression techniques (quantization, pruning) for a set of resource-constrained embedded systems, such as CPUs, GPUs and FPGAs. The obtained results highlight the difficulty of finding good trade-offs between resource efficiency and predictive performance.

Top-1 ImageNet优化促进了可能在推理设置中不切实际的网络。二元神经网络（BNN）具有显着降低计算强度，但现有模型的质量低。为了克服这种缺陷，我们提出了PokeConv，一个二进制卷积块，这是通过添加多个剩余路径的技术提高BNN的质量，并调整激活函数。我们将其应用于Reset-50并优化Reset的初始卷积层，这很难二向化。我们命名由此产生的网络系列POKBNN。选择这些技术以产生最高1精度和网络成本的良好改进。为了使成本的联合优化以及准确性，我们定义算术计算工作（ACE），用于量化和二值化网络的硬件和能量启发成本度量。我们还确定需要优化控制二值化梯度近似的探索过的超参数。我们在高精度上建立了一种新的，强大的最先进（SOTA），以及常用的CPU64成本，ACE成本和网络大小指标。 ReactNET-ADAM是BNN中的先前SOTA，实现了7.9 ACE的70.5％的前1个精度。一小块的炭达到70.5％的前1个，成本降低超过3倍;一个较大的POKBNN以7.8 ACE获得75.6％的顶级1，在不增加成本的情况下，准确性提高超过5％以上。 JAX /亚麻和再现说明中的POKEBNN实现是开放的。

Deep neural networks (DNNs) are currently widely used for many artificial intelligence (AI) applications including computer vision, speech recognition, and robotics. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Accordingly, techniques that enable efficient processing of DNNs to improve energy efficiency and throughput without sacrificing application accuracy or increasing hardware cost are critical to the wide deployment of DNNs in AI systems.This article aims to provide a comprehensive tutorial and survey about the recent advances towards the goal of enabling efficient processing of DNNs. Specifically, it will provide an overview of DNNs, discuss various hardware platforms and architectures that support DNNs, and highlight key trends in reducing the computation cost of DNNs either solely via hardware design changes or via joint hardware design and DNN algorithm changes. It will also summarize various development resources that enable researchers and practitioners to quickly get started in this field, and highlight important benchmarking metrics and design considerations that should be used for evaluating the rapidly growing number of DNN hardware designs, optionally including algorithmic co-designs, being proposed in academia and industry.The reader will take away the following concepts from this article: understand the key design considerations for DNNs; be able to evaluate different DNN hardware implementations with benchmarks and comparison metrics; understand the trade-offs between various hardware architectures and platforms; be able to evaluate the utility of various DNN design techniques for efficient processing; and understand recent implementation trends and opportunities.

卷积神经网络（CNNS）已被广泛应用。但随着CNN的成长，算术运算和内存占用的数量也增加。此外，典型的非线性激活函数不允许连续层编码的操作的相关性，通过组合它们来防止简化中间步骤。我们提出了一种新的激活函数，允许CNN的顺序层之间的关联性。即使我们的激活函数是非线性的，它也可以通过欧几里德几何形状的共形模型中的一系列线性操作来表示。在此域中，操作，但不限于卷积，平均池和丢失保持线性。我们利用关联性来组合所有的“保形层”并使推理的成本持续，而不管网络的深度如何。

Deploying convolutional neural networks (CNNs) on embedded devices is difficult due to the limited memory and computation resources. The redundancy in feature maps is an important characteristic of those successful CNNs, but has rarely been investigated in neural architecture design. This paper proposes a novel Ghost module to generate more feature maps from cheap operations. Based on a set of intrinsic feature maps, we apply a series of linear transformations with cheap cost to generate many ghost feature maps that could fully reveal information underlying intrinsic features. The proposed Ghost module can be taken as a plug-and-play component to upgrade existing convolutional neural networks. Ghost bottlenecks are designed to stack Ghost modules, and then the lightweight Ghost-Net can be easily established. Experiments conducted on benchmarks demonstrate that the proposed Ghost module is an impressive alternative of convolution layers in baseline models, and our GhostNet can achieve higher recognition performance (e.g. 75.7% top-1 accuracy) than MobileNetV3 with similar computational cost on the ImageNet ILSVRC-2012 classification dataset. Code is available at https: //github.com/huawei-noah/ghostnet.

已证明卷积神经网络中的渠道注意机制在各种计算机视觉任务中有效。但是，性能改进具有额外的模型复杂性和计算成本。在本文中，我们提出了一种被称为信道分流块的轻量级和有效的注意模块，以通过在全球层面建立信道关系来增强全局背景。与其他通道注意机制不同，所提出的模块通过在考虑信道激活时更加关注空间可区分的渠道，专注于最辨别的特征。与其他介绍模块不同的其他中间层之间的其他关注模型不同，所提出的模块嵌入在骨干网络的末尾，使其易于实现。在CiFar-10，SVHN和微型想象中心数据集上进行了广泛的实验表明，所提出的模块平均提高了基线网络的性能3％的余量。

虽然残留连接使训练非常深的神经网络，但由于其多分支拓扑而​​导致在线推断不友好。这鼓励许多研究人员在推动时没有残留连接的情况下设计DNN。例如，repvgg在部署时将多分支拓扑重新参数化为vgg型（单分支）模型，当网络相对较浅时显示出具有很大的性能。但是，RepVGG不能等效地将Reset转换为VGG，因为重新参数化方法只能应用于线性块，并且必须将非线性层（Relu）放在残余连接之外，这导致了有限的表示能力，特别是更深入网络。在本文中，我们的目标是通过在Resblock上的保留和合并（RM）操作等效地纠正此问题，并提出删除Vanilla Reset中的残留连接。具体地，RM操作允许输入特征映射通过块，同时保留其信息，并在每个块的末尾合并所有信息，这可以去除残差而不改变原始输出。作为一个插件方法，RM操作基本上有三个优点：1）其实现使其实现高比率网络修剪。 2）它有助于打破RepVGG的深度限制。 3）与Reset和RepVGG相比，它导致更好的精度速度折衷网络（RMNet）。我们相信RM操作的意识形态可以激发对未来社区的模型设计的许多见解。代码可用：https://github.com/fxmeng/rmnet。

We introduce a method to train Quantized Neural Networks (QNNs) -neural networks with extremely low precision (e.g., 1-bit) weights and activations, at run-time. At traintime the quantized weights and activations are used for computing the parameter gradients. During the forward pass, QNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations. As a result, power consumption is expected to be drastically reduced. We trained QNNs over the MNIST, CIFAR-10, SVHN and ImageNet datasets. The resulting QNNs achieve prediction accuracy comparable to their 32-bit counterparts. For example, our quantized version of AlexNet with 1-bit weights and 2-bit activations achieves 51% top-1 accuracy. Moreover, we quantize the parameter gradients to 6-bits as well which enables gradients computation using only bit-wise operation. Quantized recurrent neural networks were tested over the Penn Treebank dataset, and achieved comparable accuracy as their 32-bit counterparts using only 4-bits. Last but not least, we programmed a binary matrix multiplication GPU kernel with which it is possible to run our MNIST QNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The QNN code is available online.

在本文中，我们提出了解决稳定性和卷积神经网络（CNN）的稳定性和视野的问题的神经网络。作为提高网络深度或宽度以提高性能的替代方案，我们提出了与全球加权拉普拉斯，分数拉普拉斯和逆分数拉普拉斯算子有关的基于积分的空间非识别算子，其在物理科学中的几个问题中出现。这种网络的前向传播由部分积分微分方程（PIDE）启发。我们在自动驾驶中测试基准图像分类数据集和语义分段任务的提出神经架构的有效性。此外，我们调查了这些密集的运营商的额外计算成本以及提出神经网络的前向传播的稳定性。

我们提出了一种多移民通道（MGIC）方法，该方法可以解决参数数量相对于标准卷积神经网络（CNN）中的通道数的二次增长。因此，我们的方法解决了CNN中的冗余，这也被轻量级CNN的成功所揭示。轻巧的CNN可以达到与参数较少的标准CNN的可比精度。但是，权重的数量仍然随CNN的宽度四倍地缩放。我们的MGIC体系结构用MGIC对应物代替了每个CNN块，该块利用了小组大小的嵌套分组卷积的层次结构来解决此问题。因此，我们提出的架构相对于网络的宽度线性扩展，同时保留了通道的完整耦合，如标准CNN中。我们对图像分类，分割和点云分类进行的广泛实验表明，将此策略应用于Resnet和MobilenetV3等不同体系结构，可以减少参数的数量，同时获得相似或更好的准确性。

二进制神经网络（BNN）是卷积神经网络（CNN）的极端量化版本，其所有功能和权重映射到仅1位。尽管BNN节省了大量的内存和计算需求以使CNN适用于边缘或移动设备，但由于二进制后的表示能力降低，BNN遭受了网络性能的下降。在本文中，我们提出了一个新的可更换且易于使用的卷积模块reponv，该模块reponv通过复制输入或沿通道维度的输出来增强特征地图，而不是$ \ beta $ times，而没有额外的参数和卷积计算费用。我们还定义了一组Reptran规则，可以在整个BNN模块中使用Repconv，例如二进制卷积，完全连接的层和批处理归一化。实验表明，在Reptran转换之后，一组高度引用的BNN与原始BNN版本相比，实现了普遍的性能。例如，Rep-Recu-Resnet-20的前1位准确性，即REPBCONV增强的RECU-RESNET-20，在CIFAR-10上达到了88.97％，比原始网络高1.47％。 Rep-Adambnn-Reactnet-A在Imagenet上获得了71.342％的TOP-1精度，这是BNN的最新结果。代码和型号可在以下网址提供：https：//github.com/imfinethanks/rep_adambnn。

卷积神经网络（CNN）已在许多物联网（IoT）设备中应用于多种下游任务。但是，随着边缘设备上的数据量的增加，CNN几乎无法及时完成某些任务，而计算和存储资源有限。最近，过滤器修剪被认为是压缩和加速CNN的有效技术，但是从压缩高维张量的角度来看，现有的方法很少是修剪CNN。在本文中，我们提出了一种新颖的理论，可以在三维张量中找到冗余信息，即量化特征图（QSFM）之间的相似性，并利用该理论来指导滤波器修剪过程。我们在数据集（CIFAR-10，CIFAR-100和ILSVRC-12）上执行QSFM和Edge设备，证明所提出的方法可以在神经网络中找到冗余信息，具有可比的压缩和可耐受的准确性下降。没有任何微调操作，QSFM可以显着压缩CIFAR-56（48.7％的Flops和57.9％的参数），而TOP-1的准确性仅损失0.54％。对于边缘设备的实际应用，QSFM可以将Mobilenet-V2推理速度加速1.53倍，而ILSVRC-12 TOP-1的精度仅损失1.23％。