基于奇异值分解的相干集成光子神经网络（SC-IPNN）具有大的占地面积，遭受高静态功耗进行训练和推理，并且不能使用传统的DNN修剪技术进行修剪。我们利用彩票假设提出了一种用于SC-IPN的第一种硬件感知修剪方法，通过最小化重量参数的数量来缓解这些挑战。我们修剪基于多层的Perceptron的SC-IPN，并显示高达89％的相位角，其对应于SC-IPNN中的重量参数，可以在减少时具有可忽略的精度损失（小于5％）。静电功耗高达86％。

我们提出了一种用于相干光子神经网络的新型硬件感知幅度修剪技术。该技术可以将99.45％的网络参数进行99.45％，并将静态功耗降低98.23％，精度损失可忽略不计。

综合光子神经网络（IPNN）成为常规电子AI加速器的有前途的后继者，因为它们在计算速度和能源效率方面提供了实质性的提高。特别是，相干IPNN使用Mach-Zehnder干涉仪（MZIS）的阵列进行单位转换来执行节能矩阵矢量乘法。然而，IPNN中的基本MZI设备易受光刻变化和热串扰引起的不确定性，并且由于不均匀的MZI插入损失和量化错误而导致不确定的不确定性，这是由于调谐相角的编码较低而导致的。在本文中，我们首次使用自下而上的方法系统地表征了IPNN中这种不确定性和不确定性（共同称为缺陷）的影响。我们表明，它们对IPNN准确性的影响可能会根据受影响组件的调谐参数（例如相角），其物理位置以及缺陷的性质和分布而差异很大。为了提高可靠性措施，我们确定了关键的IPNN构件，在不完美之下，这些基础可能导致分类准确性的灾难性降解。我们表明，在多个同时缺陷下，即使不完美参数限制在较小的范围内，IPNN推断精度也可能会降低46％。我们的结果还表明，推论精度对影响IPNN输入层旁边的线性层中MZI的缺陷敏感。

在本文中，提出了一种新的方法，该方法允许基于神经网络（NN）均衡器的低复杂性发展，以缓解高速相干光学传输系统中的损伤。在这项工作中，我们提供了已应用于馈电和经常性NN设计的各种深层模型压缩方法的全面描述和比较。此外，我们评估了这些策略对每个NN均衡器的性能的影响。考虑量化，重量聚类，修剪和其他用于模型压缩的尖端策略。在这项工作中，我们提出并评估贝叶斯优化辅助压缩，其中选择了压缩的超参数以同时降低复杂性并提高性能。总之，通过使用模拟和实验数据来评估每种压缩方法的复杂性及其性能之间的权衡，以完成分析。通过利用最佳压缩方法，我们表明可以设计基于NN的均衡器，该均衡器比传统的数字背部传播（DBP）均衡器具有更好的性能，并且只有一个步骤。这是通过减少使用加权聚类和修剪算法后在NN均衡器中使用的乘数数量来完成的。此外，我们证明了基于NN的均衡器也可以实现卓越的性能，同时仍然保持与完整的电子色色散补偿块相同的复杂性。我们通过强调开放问题和现有挑战以及未来的研究方向来结束分析。

由于深度学习在许多人工智能应用中显示了革命性的性能，其升级的计算需求需要用于巨大并行性的硬件加速器和改进的吞吐量。光学神经网络（ONN）是下一代神经关键组成的有希望的候选者，由于其高并行，低延迟和低能量消耗。在这里，我们设计了一个硬件高效的光子子空间神经网络（PSNN）架构，其针对具有比具有可比任务性能的前一个ONN架构的光学元件使用，区域成本和能量消耗。此外，提供了一种硬件感知培训框架，以最小化所需的设备编程精度，减少芯片区域，并提高噪声鲁棒性。我们在实验上展示了我们的PSNN在蝴蝶式可编程硅光子集成电路上，并在实用的图像识别任务中显示其实用性。

Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy by learning only the important connections. Our method prunes redundant connections using a three-step method. First, we train the network to learn which connections are important. Next, we prune the unimportant connections. Finally, we retrain the network to fine tune the weights of the remaining connections. On the ImageNet dataset, our method reduced the number of parameters of AlexNet by a factor of 9×, from 61 million to 6.7 million, without incurring accuracy loss. Similar experiments with VGG-16 found that the total number of parameters can be reduced by 13×, from 138 million to 10.3 million, again with no loss of accuracy.

Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy by learning only the important connections. Our method prunes redundant connections using a three-step method. First, we train the network to learn which connections are important. Next, we prune the unimportant connections. Finally, we retrain the network to fine tune the weights of the remaining connections. On the ImageNet dataset, our method reduced the number of parameters of AlexNet by a factor of 9×, from 61 million to 6.7 million, without incurring accuracy loss. Similar experiments with VGG-16 found that the total number of parameters can be reduced by 13×, from 138 million to 10.3 million, again with no loss of accuracy.

深度神经网络（DNN）的记录断裂性能具有沉重的参数化，导致外部动态随机存取存储器（DRAM）进行存储。 DRAM访问的禁用能量使得在资源受限的设备上部署DNN是不普遍的，呼叫最小化重量和数据移动以提高能量效率。我们呈现SmartDeal（SD），算法框架，以进行更高成本的存储器存储/访问的较低成本计算，以便在推理和培训中积极提高存储和能量效率。 SD的核心是一种具有结构约束的新型重量分解，精心制作以释放硬件效率潜力。具体地，我们将每个重量张量分解为小基矩阵的乘积以及大的结构稀疏系数矩阵，其非零被量化为-2的功率。由此产生的稀疏和量化的DNN致力于为数据移动和重量存储而大大降低的能量，因为由于稀疏的比特 - 操作和成本良好的计算，恢复原始权重的最小开销。除了推理之外，我们采取了另一次飞跃来拥抱节能培训，引入创新技术，以解决培训时出现的独特障碍，同时保留SD结构。我们还设计专用硬件加速器，充分利用SD结构来提高实际能源效率和延迟。我们在不同的设置中对多个任务，模型和数据集进行实验。结果表明：1）应用于推理，SD可实现高达2.44倍的能效，通过实际硬件实现评估; 2）应用于培训，储存能量降低10.56倍，减少了10.56倍和4.48倍，与最先进的训练基线相比，可忽略的准确性损失。我们的源代码在线提供。

While machine learning is traditionally a resource intensive task, embedded systems, autonomous navigation, and the vision of the Internet of Things fuel the interest in resource-efficient approaches. These approaches aim for a carefully chosen trade-off between performance and resource consumption in terms of computation and energy. The development of such approaches is among the major challenges in current machine learning research and key to ensure a smooth transition of machine learning technology from a scientific environment with virtually unlimited computing resources into everyday's applications. In this article, we provide an overview of the current state of the art of machine learning techniques facilitating these real-world requirements. In particular, we focus on deep neural networks (DNNs), the predominant machine learning models of the past decade. We give a comprehensive overview of the vast literature that can be mainly split into three non-mutually exclusive categories: (i) quantized neural networks, (ii) network pruning, and (iii) structural efficiency. These techniques can be applied during training or as post-processing, and they are widely used to reduce the computational demands in terms of memory footprint, inference speed, and energy efficiency. We also briefly discuss different concepts of embedded hardware for DNNs and their compatibility with machine learning techniques as well as potential for energy and latency reduction. We substantiate our discussion with experiments on well-known benchmark datasets using compression techniques (quantization, pruning) for a set of resource-constrained embedded systems, such as CPUs, GPUs and FPGAs. The obtained results highlight the difficulty of finding good trade-offs between resource efficiency and predictive performance.

Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the lottery ticket hypothesis: dense, randomly-initialized, feed-forward networks contain subnetworks (winning tickets) that-when trained in isolationreach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

Pruning refers to the elimination of trivial weights from neural networks. The sub-networks within an overparameterized model produced after pruning are often called Lottery tickets. This research aims to generate winning lottery tickets from a set of lottery tickets that can achieve similar accuracy to the original unpruned network. We introduce a novel winning ticket called Cyclic Overlapping Lottery Ticket (COLT) by data splitting and cyclic retraining of the pruned network from scratch. We apply a cyclic pruning algorithm that keeps only the overlapping weights of different pruned models trained on different data segments. Our results demonstrate that COLT can achieve similar accuracies (obtained by the unpruned model) while maintaining high sparsities. We show that the accuracy of COLT is on par with the winning tickets of Lottery Ticket Hypothesis (LTH) and, at times, is better. Moreover, COLTs can be generated using fewer iterations than tickets generated by the popular Iterative Magnitude Pruning (IMP) method. In addition, we also notice COLTs generated on large datasets can be transferred to small ones without compromising performance, demonstrating its generalizing capability. We conduct all our experiments on Cifar-10, Cifar-100 & TinyImageNet datasets and report superior performance than the state-of-the-art methods.

我们考虑在具有挑战性的训练后环境中，深度神经网络（DNN）的模型压缩问题，在该设置中，我们将获得精确的训练模型，并且必须仅基于少量校准输入数据而无需任何重新培训即可压缩它。鉴于新兴软件和硬件支持通过加速修剪和/或量化压缩的模型，并且已经针对两种压缩方法独立提出了良好的表现解决方案，因此该问题已变得流行。在本文中，我们引入了一个新的压缩框架，该框架涵盖了统一环境中的重量修剪和量化，时间和空间效率高，并且在现有的后训练方法的实际性能上大大改善。在技​​术层面上，我们的方法基于[Lecun，Denker和Solla，1990年]在现代DNN的规模上的经典最佳脑外科医生（OBS）框架的第一个精确实现，我们进一步扩展到覆盖范围。重量量化。这是通过一系列可能具有独立利益的算法开发来实现的。从实际的角度来看，我们的实验结果表明，它可以在现有后训练方法的压缩 - 准确性权衡方面显着改善，并且甚至可以在训练后进行修剪和量化的准确共同应用。

Most existing pruning works are resource-intensive, requiring retraining or fine-tuning of the pruned models for accuracy. We propose a retraining-free pruning method based on hyperspherical learning and loss penalty terms. The proposed loss penalty term pushes some of the model weights far from zero, while the rest weight values are pushed near zero and can be safely pruned with no need for retraining and a negligible accuracy drop. In addition, our proposed method can instantly recover the accuracy of a pruned model by replacing the pruned values with their mean value. Our method obtains state-of-the-art results in retraining-free pruning and is evaluated on ResNet-18/50 and MobileNetV2 with ImageNet dataset. One can easily get a 50\% pruned ResNet18 model with a 0.47\% accuracy drop. With fine-tuning, the experiment results show that our method can significantly boost the accuracy of the pruned models compared with existing works. For example, the accuracy of a 70\% pruned (except the first convolutional layer) MobileNetV2 model only drops 3.5\%, much less than the 7\% $\sim$ 10\% accuracy drop with conventional methods.

The deployment of deep convolutional neural networks (CNNs) in many real world applications is largely hindered by their high computational cost. In this paper, we propose a novel learning scheme for CNNs to simultaneously 1) reduce the model size; 2) decrease the run-time memory footprint; and 3) lower the number of computing operations, without compromising accuracy. This is achieved by enforcing channel-level sparsity in the network in a simple but effective way. Different from many existing approaches, the proposed method directly applies to modern CNN architectures, introduces minimum overhead to the training process, and requires no special software/hardware accelerators for the resulting models. We call our approach network slimming, which takes wide and large networks as input models, but during training insignificant channels are automatically identified and pruned afterwards, yielding thin and compact models with comparable accuracy. We empirically demonstrate the effectiveness of our approach with several state-of-the-art CNN models, including VGGNet, ResNet and DenseNet, on various image classification datasets. For VGGNet, a multi-pass version of network slimming gives a 20× reduction in model size and a 5× reduction in computing operations.

The success of CNNs in various applications is accompanied by a significant increase in the computation and parameter storage costs. Recent efforts toward reducing these overheads involve pruning and compressing the weights of various layers without hurting original accuracy. However, magnitude-based pruning of weights reduces a significant number of parameters from the fully connected layers and may not adequately reduce the computation costs in the convolutional layers due to irregular sparsity in the pruned networks. We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications. We show that even simple filter pruning techniques can reduce inference costs for VGG-16 by up to 34% and ResNet-110 by up to 38% on CIFAR10 while regaining close to the original accuracy by retraining the networks.

在神经网络中引入稀疏性是一种有效的方法，可以降低其复杂性，同时保持其性能几乎完好无损。在大多数情况下，使用三阶段管道引入稀疏性：1）训练模型以收敛，2）根据某些标准修剪模型，3）微调修剪模型以恢复性能。最后两个步骤通常是迭代执行的，从而导致合理的结果，但也取得了耗时且复杂的过程。在我们的工作中，我们建议摆脱管道的第一步，并在单个修剪训练周期中结合其他两个步骤，从而使模型在修剪时共同学习最佳权重。我们通过介绍一个名为One Cycle Pruning的小说修剪时间表来做到这一点，该时间表从培训开始就开始修剪，直到最后。采用这样的时间表不仅可以更好地执行修剪模型，而且还大大降低了修剪模型所需的培训预算。实验是在多种架构（VGG-16和RESNET-18）和数据集（CIFAR-10，CIFAR-100和CALTECH-101）上进行的，以及相对较高的稀疏值（80％，90％，95％的权重，删除）。我们的结果表明，按固定的培训预算，一环修剪始终优于通常使用的修剪时间表，例如单发修剪，迭代修剪和自动化逐渐修剪。

Neural network pruning has been a well-established compression technique to enable deep learning models on resource-constrained devices. The pruned model is usually specialized to meet specific hardware platforms and training tasks (defined as deployment scenarios). However, existing pruning approaches rely heavily on training data to trade off model size, efficiency, and accuracy, which becomes ineffective for federated learning (FL) over distributed and confidential datasets. Moreover, the memory- and compute-intensive pruning process of most existing approaches cannot be handled by most FL devices with resource limitations. In this paper, we develop FedTiny, a novel distributed pruning framework for FL, to obtain specialized tiny models for memory- and computing-constrained participating devices with confidential local data. To alleviate biased pruning due to unseen heterogeneous data over devices, FedTiny introduces an adaptive batch normalization (BN) selection module to adaptively obtain an initially pruned model to fit deployment scenarios. Besides, to further improve the initial pruning, FedTiny develops a lightweight progressive pruning module for local finer pruning under tight memory and computational budgets, where the pruning policy for each layer is gradually determined rather than evaluating the overall deep model structure. Extensive experimental results demonstrate the effectiveness of FedTiny, which outperforms state-of-the-art baseline approaches, especially when compressing deep models to extremely sparse tiny models.

网络修剪是一种广泛使用的技术，用于有效地压缩深神经网络，几乎没有在推理期间在性能下降低。迭代幅度修剪（IMP）是由几种迭代训练和修剪步骤组成的网络修剪的最熟悉的方法之一，其中在修剪后丢失了大量网络的性能，然后在随后的再培训阶段中恢复。虽然常用为基准参考，但经常认为a）通过不将稀疏纳入训练阶段来达到次优状态，b）其全球选择标准未能正确地确定最佳层面修剪速率和c）其迭代性质使它变得缓慢和不竞争。根据最近提出的再培训技术，我们通过严格和一致的实验来调查这些索赔，我们将Impr到培训期间的训练算法进行比较，评估其选择标准的建议修改，并研究实际需要的迭代次数和总培训时间。我们发现IMP与SLR进行再培训，可以优于最先进的修剪期间，没有或仅具有很少的计算开销，即全局幅度选择标准在很大程度上具有更复杂的方法，并且只有几个刷新时期在实践中需要达到大部分稀疏性与IMP的诽谤 - 与性能权衡。我们的目标既可以证明基本的进攻已经可以提供最先进的修剪结果，甚至优于更加复杂或大量参数化方法，也可以为未来的研究建立更加现实但易于可实现的基线。

我们日常生活中的深度学习是普遍存在的，包括自驾车，虚拟助理，社交网络服务，医疗服务，面部识别等，但是深度神经网络在训练和推理期间需要大量计算资源。该机器学习界主要集中在模型级优化（如深度学习模型的架构压缩），而系统社区则专注于实施级别优化。在其间，在算术界中提出了各种算术级优化技术。本文在模型，算术和实施级技术方面提供了关于资源有效的深度学习技术的调查，并确定了三种不同级别技术的资源有效的深度学习技术的研究差距。我们的调查基于我们的资源效率度量定义，阐明了较低级别技术的影响，并探讨了资源有效的深度学习研究的未来趋势。

深度加强学习（RL）是解决复杂的现实问题的强大框架。在框架中使用的大型神经网络传统上与更好的泛化能力相关，但它们的增加的大小需要广泛的培训持续时间，大量硬件资源和较长推理时间的缺点。解决这个问题的一种方法是修剪神经网络，只留下必要的参数。用于在固定数据分布的应用中施加稀疏性的最先进的并发修剪技术。但是，他们尚未在RL的背景下大大探索。我们缩小了RL和单次修剪技术之间的差距，并将一般修剪方法呈现给离线RL。在RL培训开始之前，我们利用固定数据集进行修剪神经网络。然后，我们运行不同网络稀疏度水平的实验，并评估连续控制任务中的初始化技术修剪的有效性。我们的结果表明，随着95％的网络权重修剪，离线-RL算法仍然可以在我们的大部分实验中保持性能。据我们所知，没有先前的工作，利用在这种高水平的稀疏性的RL保留的性能中进行修剪。此外，在未改变学习目标的情况下，可以在任何现有的离线-RL算法中容易地集成到任何现有的离线RL算法中。