突起从量子物理学起源的张量网络是古典和量子机学习的有效工具。然而，张量网络与古典机器学习的复杂神经网络模型仍然存在相当大的精度差距。在这项工作中，我们将矩阵产品状态（MPS），最简单的张量网络结构和残差神经网络的思想结合起来，提出了残余矩阵产品状态（Resmps）。 RESMP可以被视为其层，其中其层将“隐藏”特征映射到输出（例如，分类），并且层的变分参数是样本的特征（例如图像的像素）的功能。这与神经网络不同，其中层将向前映射到输出的功能。 RESMP可以用非线性激活和丢弃层配备，并且在效率，稳定性和表达功率方面优于最先进的张量网络模型。此外，Resmps是从多项式扩展的角度解释的，其中分解和指数机器自然出现。我们的工作有助于连接和杂交的神经和张量网络，这对进一步提高了我们了解工作机制并提高两种模型的性能至关重要。

量子多体系统的状态是在高维的希尔伯特空间中定义的，可以对子系统之间的丰富而复杂的相互作用进行建模。在机器学习中，复杂的多个多线性相关性也可能存在于输入功能中。在本文中，我们提出了一个量子启发的多线性模型，称为残留张量列（RESTT），以捕获单个模型中从低阶到高阶的特征的多次多线性相关性。 RESTT能够在高维空间中建立强大的决策边界，以解决拟合和分类任务。特别是，我们证明了完全连接的层和Volterra系列可以将其视为特殊情况。此外，我们得出了根据平均场分析来稳定RESTT训练的权重初始化规则。我们证明，这样的规则比TT的规则放松得多，这意味着休息可以轻松解决现有TT模型中存在的消失和爆炸梯度问题。数值实验表明，RESTT的表现优于最先进的张量网络，并在MNIST和时尚MNIST数据集上进行基准深度学习模型。此外，RESTT在两个实践示例上的统计方法比其他有限数据的统计方法更好，这些方法具有复杂的功能相互作用。

最近的工作已经开始探索参数化量子电路（PQC）作为一般函数近似器的潜力。在这项工作中，我们提出了一种量子古典的深网络结构，以提高经典的CNN模型辨别性。卷积层使用线性滤波器来扫描输入数据。此外，我们构建PQC，这是一种更有效的函数近似器，具有更复杂的结构，以捕获接收领域内的特征。通过以与CNN类似的方式将PQC滑过输入来获得特征图。我们还为所提出的模型提供培训算法。我们设计中使用的混合模型通过数值模拟验证。我们展示了MNIST上合理的分类性能，我们将性能与不同的设置中的模型进行比较。结果揭示了具有高表现性的ANSATZ模型实现了更低的成本和更高的准确性。

卷积神经网络（CNNS）已被广泛应用。但随着CNN的成长，算术运算和内存占用的数量也增加。此外，典型的非线性激活函数不允许连续层编码的操作的相关性，通过组合它们来防止简化中间步骤。我们提出了一种新的激活函数，允许CNN的顺序层之间的关联性。即使我们的激活函数是非线性的，它也可以通过欧几里德几何形状的共形模型中的一系列线性操作来表示。在此域中，操作，但不限于卷积，平均池和丢失保持线性。我们利用关联性来组合所有的“保形层”并使推理的成本持续，而不管网络的深度如何。

We propose to employ the hierarchical coarse-grained structure in the artificial neural networks explicitly to improve the interpretability without degrading performance. The idea has been applied in two situations. One is a neural network called TaylorNet, which aims to approximate the general mapping from input data to output result in terms of Taylor series directly, without resorting to any magic nonlinear activations. The other is a new setup for data distillation, which can perform multi-level abstraction of the input dataset and generate new data that possesses the relevant features of the original dataset and can be used as references for classification. In both cases, the coarse-grained structure plays an important role in simplifying the network and improving both the interpretability and efficiency. The validity has been demonstrated on MNIST and CIFAR-10 datasets. Further improvement and some open questions related are also discussed.

We propose two efficient approximations to standard convolutional neural networks: Binary-Weight-Networks and XNOR-Networks. In Binary-Weight-Networks, the filters are approximated with binary values resulting in 32× memory saving. In XNOR-Networks, both the filters and the input to convolutional layers are binary. XNOR-Networks approximate convolutions using primarily binary operations. This results in 58× faster convolutional operations (in terms of number of the high precision operations) and 32× memory savings. XNOR-Nets offer the possibility of running state-of-the-art networks on CPUs (rather than GPUs) in real-time. Our binary networks are simple, accurate, efficient, and work on challenging visual tasks. We evaluate our approach on the ImageNet classification task. The classification accuracy with a Binary-Weight-Network version of AlexNet is the same as the full-precision AlexNet. We compare our method with recent network binarization methods, BinaryConnect and BinaryNets, and outperform these methods by large margins on ImageNet, more than 16% in top-1 accuracy. Our code is available at: http://allenai.org/plato/xnornet.

这项调查的目的是介绍对深神经网络的近似特性的解释性回顾。具体而言，我们旨在了解深神经网络如何以及为什么要优于其他经典线性和非线性近似方法。这项调查包括三章。在第1章中，我们回顾了深层网络及其组成非线性结构的关键思想和概念。我们通过在解决回归和分类问题时将其作为优化问题来形式化神经网络问题。我们简要讨论用于解决优化问题的随机梯度下降算法以及用于解决优化问题的后传播公式，并解决了与神经网络性能相关的一些问题，包括选择激活功能，成本功能，过度适应问题和正则化。在第2章中，我们将重点转移到神经网络的近似理论上。我们首先介绍多项式近似中的密度概念，尤其是研究实现连续函数的Stone-WeierStrass定理。然后，在线性近似的框架内，我们回顾了馈电网络的密度和收敛速率的一些经典结果，然后在近似Sobolev函数中进行有关深网络复杂性的最新发展。在第3章中，利用非线性近似理论，我们进一步详细介绍了深度和近似网络与其他经典非线性近似方法相比的近似优势。

本文提出了一种新的和富有激光激活方法，被称为FPLUS，其利用具有形式的极性标志的数学功率函数。它是通过常见的逆转操作来启发，同时赋予仿生学的直观含义。制剂在某些先前知识和预期特性的条件下理论上得出，然后通过使用典型的基准数据集通过一系列实验验证其可行性，其结果表明我们的方法在许多激活功能中拥有卓越的竞争力，以及兼容稳定性许多CNN架构。此外，我们将呈现给更广泛类型的功能延伸到称为PFPlus的函数，具有两个可以固定的或学习的参数，以便增加其表现力的容量，并且相同的测试结果验证了这种改进。

While machine learning is traditionally a resource intensive task, embedded systems, autonomous navigation, and the vision of the Internet of Things fuel the interest in resource-efficient approaches. These approaches aim for a carefully chosen trade-off between performance and resource consumption in terms of computation and energy. The development of such approaches is among the major challenges in current machine learning research and key to ensure a smooth transition of machine learning technology from a scientific environment with virtually unlimited computing resources into everyday's applications. In this article, we provide an overview of the current state of the art of machine learning techniques facilitating these real-world requirements. In particular, we focus on deep neural networks (DNNs), the predominant machine learning models of the past decade. We give a comprehensive overview of the vast literature that can be mainly split into three non-mutually exclusive categories: (i) quantized neural networks, (ii) network pruning, and (iii) structural efficiency. These techniques can be applied during training or as post-processing, and they are widely used to reduce the computational demands in terms of memory footprint, inference speed, and energy efficiency. We also briefly discuss different concepts of embedded hardware for DNNs and their compatibility with machine learning techniques as well as potential for energy and latency reduction. We substantiate our discussion with experiments on well-known benchmark datasets using compression techniques (quantization, pruning) for a set of resource-constrained embedded systems, such as CPUs, GPUs and FPGAs. The obtained results highlight the difficulty of finding good trade-offs between resource efficiency and predictive performance.

To reduce the significant redundancy in deep Convolutional Neural Networks (CNNs), most existing methods prune neurons by only considering statistics of an individual layer or two consecutive layers (e.g., prune one layer to minimize the reconstruction error of the next layer), ignoring the effect of error propagation in deep networks. In contrast, we argue that it is essential to prune neurons in the entire neuron network jointly based on a unified goal: minimizing the reconstruction error of important responses in the "final response layer" (FRL), which is the secondto-last layer before classification, for a pruned network to retrain its predictive power. Specifically, we apply feature ranking techniques to measure the importance of each neuron in the FRL, and formulate network pruning as a binary integer optimization problem and derive a closed-form solution to it for pruning neurons in earlier layers. Based on our theoretical analysis, we propose the Neuron Importance Score Propagation (NISP) algorithm to propagate the importance scores of final responses to every neuron in the network. The CNN is pruned by removing neurons with least importance, and then fine-tuned to retain its predictive power. NISP is evaluated on several datasets with multiple CNN models and demonstrated to achieve significant acceleration and compression with negligible accuracy loss.

Understanding the functional principles of information processing in deep neural networks continues to be a challenge, in particular for networks with trained and thus non-random weights. To address this issue, we study the mapping between probability distributions implemented by a deep feed-forward network. We characterize this mapping as an iterated transformation of distributions, where the non-linearity in each layer transfers information between different orders of correlation functions. This allows us to identify essential statistics in the data, as well as different information representations that can be used by neural networks. Applied to an XOR task and to MNIST, we show that correlations up to second order predominantly capture the information processing in the internal layers, while the input layer also extracts higher-order correlations from the data. This analysis provides a quantitative and explainable perspective on classification.

这是一门专门针对STEM学生开发的介绍性机器学习课程。我们的目标是为有兴趣的读者提供基础知识，以在自己的项目中使用机器学习，并将自己熟悉术语作为进一步阅读相关文献的基础。在这些讲义中，我们讨论受监督，无监督和强化学习。注释从没有神经网络的机器学习方法的说明开始，例如原理分析，T-SNE，聚类以及线性回归和线性分类器。我们继续介绍基本和先进的神经网络结构，例如密集的进料和常规神经网络，经常性的神经网络，受限的玻尔兹曼机器，（变性）自动编码器，生成的对抗性网络。讨论了潜在空间表示的解释性问题，并使用梦和对抗性攻击的例子。最后一部分致力于加强学习，我们在其中介绍了价值功能和政策学习的基本概念。

These notes were compiled as lecture notes for a course developed and taught at the University of the Southern California. They should be accessible to a typical engineering graduate student with a strong background in Applied Mathematics. The main objective of these notes is to introduce a student who is familiar with concepts in linear algebra and partial differential equations to select topics in deep learning. These lecture notes exploit the strong connections between deep learning algorithms and the more conventional techniques of computational physics to achieve two goals. First, they use concepts from computational physics to develop an understanding of deep learning algorithms. Not surprisingly, many concepts in deep learning can be connected to similar concepts in computational physics, and one can utilize this connection to better understand these algorithms. Second, several novel deep learning algorithms can be used to solve challenging problems in computational physics. Thus, they offer someone who is interested in modeling a physical phenomena with a complementary set of tools.

We introduce DropConnect, a generalization of Dropout (Hinton et al., 2012), for regularizing large fully-connected layers within neural networks. When training with Dropout, a randomly selected subset of activations are set to zero within each layer. DropConnect instead sets a randomly selected subset of weights within the network to zero. Each unit thus receives input from a random subset of units in the previous layer. We derive a bound on the generalization performance of both Dropout and DropConnect. We then evaluate DropConnect on a range of datasets, comparing to Dropout, and show state-of-the-art results on several image recognition benchmarks by aggregating multiple DropConnect-trained models.

深度学习使用由其重量进行参数化的神经网络。通常通过调谐重量来直接最小化给定损耗功能来训练神经网络。在本文中，我们建议将权重重新参数转化为网络中各个节点的触发强度的目标。给定一组目标，可以计算使得发射强度最佳地满足这些目标的权重。有人认为，通过我们称之为级联解压缩的过程，使用培训的目标解决爆炸梯度的问题，并使损失功能表面更加光滑，因此导致更容易，培训更快，以及潜在的概括，神经网络。它还允许更容易地学习更深层次和经常性的网络结构。目标对重量的必要转换有额外的计算费用，这是在许多情况下可管理的。在目标空间中学习可以与现有的神经网络优化器相结合，以额外收益。实验结果表明了使用目标空间的速度，以及改进的泛化的示例，用于全连接的网络和卷积网络，以及调用和处理长时间序列的能力，并使用经常性网络进行自然语言处理。

作为一种强大的建模方法，分段线性神经网络（PWLNNS）已在各个领域都被证明是成功的，最近在深度学习中。为了应用PWLNN方法，长期以来一直研究了表示和学习。 1977年，规范表示率先通过增量设计学到了浅层PWLNN的作品，但禁止使用大规模数据的应用。 2010年，纠正的线性单元（RELU）提倡在深度学习中PWLNN的患病率。从那以后，PWLNNS已成功地应用于广泛的任务并实现了有利的表现。在本引物中，我们通过将作品分组为浅网络和深层网络来系统地介绍PWLNNS的方法。首先，不同的PWLNN表示模型是由详细示例构建的。使用PWLNNS，提出了学习数据的学习算法的演变，并且基本理论分析遵循深入的理解。然后，将代表性应用与讨论和前景一起引入。

我们介绍了深张量网络，这些网络是基于权重矩阵的张量网络表示的成倍宽的神经网络。我们评估图像分类（MNIST，FashionMnist）和序列预测（蜂窝自动机）任务的建议方法。在图像分类案例中，深度张量网络改善了我们的矩阵产品状态基线，并在MNIST上达到0.49％的错误率，而时尚人士的错误率为8.3％。在序列预测情况下，我们证明了与一层张量网络方法相比，参数数量的指数改善。在这两种情况下，我们都讨论了非均匀和均匀的张量网络模型，并表明后者可以很好地推广到不同的输入尺寸。

由于稀疏神经网络通常包含许多零权重，因此可以在不降低网络性能的情况下潜在地消除这些不必要的网络连接。因此，设计良好的稀疏神经网络具有显着降低拖鞋和计算资源的潜力。在这项工作中，我们提出了一种新的自动修剪方法 - 稀疏连接学习（SCL）。具体地，重量被重新参数化为可培训权重变量和二进制掩模的元素方向乘法。因此，由二进制掩模完全描述网络连接，其由单位步进函数调制。理论上，从理论上证明了使用直通估计器（STE）进行网络修剪的基本原理。这一原则是STE的代理梯度应该是积极的，确保掩模变量在其最小值处收敛。在找到泄漏的Relu后，SoftPlus和Identity Stes可以满足这个原理，我们建议采用SCL的身份STE以进行离散面膜松弛。我们发现不同特征的面具梯度非常不平衡，因此，我们建议将每个特征的掩模梯度标准化以优化掩码变量训练。为了自动训练稀疏掩码，我们将网络连接总数作为我们的客观函数中的正则化术语。由于SCL不需要由网络层设计人员定义的修剪标准或超级参数，因此在更大的假设空间中探讨了网络，以实现最佳性能的优化稀疏连接。 SCL克服了现有自动修剪方法的局限性。实验结果表明，SCL可以自动学习并选择各种基线网络结构的重要网络连接。 SCL培训的深度学习模型以稀疏性，精度和减少脚波特的SOTA人类设计和自动修剪方法训练。

We propose a simultaneous learning and pruning algorithm capable of identifying and eliminating irrelevant structures in a neural network during the early stages of training. Thus, the computational cost of subsequent training iterations, besides that of inference, is considerably reduced. Our method, based on variational inference principles using Gaussian scale mixture priors on neural network weights, learns the variational posterior distribution of Bernoulli random variables multiplying the units/filters similarly to adaptive dropout. Our algorithm, ensures that the Bernoulli parameters practically converge to either 0 or 1, establishing a deterministic final network. We analytically derive a novel hyper-prior distribution over the prior parameters that is crucial for their optimal selection and leads to consistent pruning levels and prediction accuracy regardless of weight initialization or the size of the starting network. We prove the convergence properties of our algorithm establishing theoretical and practical pruning conditions. We evaluate the proposed algorithm on the MNIST and CIFAR-10 data sets and the commonly used fully connected and convolutional LeNet and VGG16 architectures. The simulations show that our method achieves pruning levels on par with state-of the-art methods for structured pruning, while maintaining better test-accuracy and more importantly in a manner robust with respect to network initialization and initial size.

神经切线内核（NTK）是分析神经网络及其泛化界限的训练动力学的强大工具。关于NTK的研究已致力于典型的神经网络体系结构，但对于Hadamard产品（NNS-HP）的神经网络不完整，例如StyleGAN和多项式神经网络。在这项工作中，我们为特殊类别的NNS-HP（即多项式神经网络）得出了有限宽度的NTK公式。我们证明了它们与关联的NTK与内核回归预测变量的等效性，该预测扩大了NTK的应用范围。根据我们的结果，我们阐明了针对外推和光谱偏置，PNN在标准神经网络上的分离。我们的两个关键见解是，与标准神经网络相比，PNN能够在外推方案中拟合更复杂的功能，并承认相应NTK的特征值衰减较慢。此外，我们的理论结果可以扩展到其他类型的NNS-HP，从而扩大了我们工作的范围。我们的经验结果验证了更广泛的NNS-HP类别的分离，这为对神经体系结构有了更深入的理解提供了良好的理由。