在本文中，我们提出了解决稳定性和卷积神经网络（CNN）的稳定性和视野的问题的神经网络。作为提高网络深度或宽度以提高性能的替代方案，我们提出了与全球加权拉普拉斯，分数拉普拉斯和逆分数拉普拉斯算子有关的基于积分的空间非识别算子，其在物理科学中的几个问题中出现。这种网络的前向传播由部分积分微分方程（PIDE）启发。我们在自动驾驶中测试基准图像分类数据集和语义分段任务的提出神经架构的有效性。此外，我们调查了这些密集的运营商的额外计算成本以及提出神经网络的前向传播的稳定性。

These notes were compiled as lecture notes for a course developed and taught at the University of the Southern California. They should be accessible to a typical engineering graduate student with a strong background in Applied Mathematics. The main objective of these notes is to introduce a student who is familiar with concepts in linear algebra and partial differential equations to select topics in deep learning. These lecture notes exploit the strong connections between deep learning algorithms and the more conventional techniques of computational physics to achieve two goals. First, they use concepts from computational physics to develop an understanding of deep learning algorithms. Not surprisingly, many concepts in deep learning can be connected to similar concepts in computational physics, and one can utilize this connection to better understand these algorithms. Second, several novel deep learning algorithms can be used to solve challenging problems in computational physics. Thus, they offer someone who is interested in modeling a physical phenomena with a complementary set of tools.

卷积神经网络（CNN）的量化是缓解CNN部署的计算负担，尤其是在低资源边缘设备上的常见方法。但是，对于神经网络所涉及的计算类型，固定点算术并不是自然的。在这项工作中，我们探索了使用基于PDE的观点和分析来改善量化CNN的方法。首先，我们利用总变化方法（电视）方法将边缘意识平滑应用于整个网络的特征图。这旨在减少值分布的异常值并促进零件恒定图，这更适合量化。其次，我们考虑用于图像分类的常见CNN的对称和稳定变体，以及用于图源分类的图形卷积网络（GCN）。我们通过几个实验证明，正向稳定性的性质保留了在不同量化速率下网络的作用。结果，稳定的量化网络的行为与非量化的网络相似，即使它们依赖于较少的参数。我们还发现，有时，稳定性甚至有助于提高准确性。对于敏感，资源受限，低功率或实时应用（例如自动驾驶），这些属性特别感兴趣。

神经运营商最近成为设计神经网络形式的功能空间之间的解决方案映射的流行工具。不同地，从经典的科学机器学习方法，以固定分辨率为输入参数的单个实例学习参数，神经运算符近似PDE系列的解决方案图。尽管他们取得了成功，但是神经运营商的用途迄今为止仅限于相对浅的神经网络，并限制了学习隐藏的管理法律。在这项工作中，我们提出了一种新颖的非局部神经运营商，我们将其称为非本体内核网络（NKN），即独立的分辨率，其特征在于深度神经网络，并且能够处理各种任务，例如学习管理方程和分类图片。我们的NKN源于神经网络的解释，作为离散的非局部扩散反应方程，在无限层的极限中，相当于抛物线非局部方程，其稳定性通过非本种载体微积分分析。与整体形式的神经运算符相似允许NKN捕获特征空间中的远程依赖性，而节点到节点交互的持续处理使NKNS分辨率独立于NKNS分辨率。与神经杂物中的相似性，在非本体意义上重新解释，并且层之间的稳定网络动态允许NKN的最佳参数从浅到深网络中的概括。这一事实使得能够使用浅层初始化技术。我们的测试表明，NKNS在学习管理方程和图像分类任务中占据基线方法，并概括到不同的分辨率和深度。

我们提出了一种多移民通道（MGIC）方法，该方法可以解决参数数量相对于标准卷积神经网络（CNN）中的通道数的二次增长。因此，我们的方法解决了CNN中的冗余，这也被轻量级CNN的成功所揭示。轻巧的CNN可以达到与参数较少的标准CNN的可比精度。但是，权重的数量仍然随CNN的宽度四倍地缩放。我们的MGIC体系结构用MGIC对应物代替了每个CNN块，该块利用了小组大小的嵌套分组卷积的层次结构来解决此问题。因此，我们提出的架构相对于网络的宽度线性扩展，同时保留了通道的完整耦合，如标准CNN中。我们对图像分类，分割和点云分类进行的广泛实验表明，将此策略应用于Resnet和MobilenetV3等不同体系结构，可以减少参数的数量，同时获得相似或更好的准确性。

While machine learning is traditionally a resource intensive task, embedded systems, autonomous navigation, and the vision of the Internet of Things fuel the interest in resource-efficient approaches. These approaches aim for a carefully chosen trade-off between performance and resource consumption in terms of computation and energy. The development of such approaches is among the major challenges in current machine learning research and key to ensure a smooth transition of machine learning technology from a scientific environment with virtually unlimited computing resources into everyday's applications. In this article, we provide an overview of the current state of the art of machine learning techniques facilitating these real-world requirements. In particular, we focus on deep neural networks (DNNs), the predominant machine learning models of the past decade. We give a comprehensive overview of the vast literature that can be mainly split into three non-mutually exclusive categories: (i) quantized neural networks, (ii) network pruning, and (iii) structural efficiency. These techniques can be applied during training or as post-processing, and they are widely used to reduce the computational demands in terms of memory footprint, inference speed, and energy efficiency. We also briefly discuss different concepts of embedded hardware for DNNs and their compatibility with machine learning techniques as well as potential for energy and latency reduction. We substantiate our discussion with experiments on well-known benchmark datasets using compression techniques (quantization, pruning) for a set of resource-constrained embedded systems, such as CPUs, GPUs and FPGAs. The obtained results highlight the difficulty of finding good trade-offs between resource efficiency and predictive performance.

Deep Neural Networks (DNNs) training can be difficult due to vanishing and exploding gradients during weight optimization through backpropagation. To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stem from the discretization of continuous-time Hamiltonian systems and include several existing DNN architectures based on ordinary differential equations. Our main result is that a broad set of H-DNNs ensures non-vanishing gradients by design for an arbitrary network depth. This is obtained by proving that, using a semi-implicit Euler discretization scheme, the backward sensitivity matrices involved in gradient computations are symplectic. We also provide an upper-bound to the magnitude of sensitivity matrices and show that exploding gradients can be controlled through regularization. Finally, we enable distributed implementations of backward and forward propagation algorithms in H-DNNs by characterizing appropriate sparsity constraints on the weight matrices. The good performance of H-DNNs is demonstrated on benchmark classification problems, including image classification with the MNIST dataset.

神经网络的经典发展主要集中在有限维欧基德空间或有限组之间的学习映射。我们提出了神经网络的概括，以学习映射无限尺寸函数空间之间的运算符。我们通过一类线性积分运算符和非线性激活函数的组成制定运营商的近似，使得组合的操作员可以近似复杂的非线性运算符。我们证明了我们建筑的普遍近似定理。此外，我们介绍了四类运算符参数化：基于图形的运算符，低秩运算符，基于多极图形的运算符和傅里叶运算符，并描述了每个用于用每个计算的高效算法。所提出的神经运营商是决议不变的：它们在底层函数空间的不同离散化之间共享相同的网络参数，并且可以用于零击超分辨率。在数值上，与现有的基于机器学习的方法，达西流程和Navier-Stokes方程相比，所提出的模型显示出卓越的性能，而与传统的PDE求解器相比，与现有的基于机器学习的方法有关的基于机器学习的方法。

Many scientific fields study data with an underlying structure that is a non-Euclidean space. Some examples include social networks in computational social sciences, sensor networks in communications, functional networks in brain imaging, regulatory networks in genetics, and meshed surfaces in computer graphics. In many applications, such geometric data are large and complex (in the case of social networks, on the scale of billions), and are natural targets for machine learning techniques. In particular, we would like to use deep neural networks, which have recently proven to be powerful tools for a broad range of problems from computer vision, natural language processing, and audio analysis. However, these tools have been most successful on data with an underlying Euclidean or grid-like structure, and in cases where the invariances of these structures are built into networks used to model them.Geometric deep learning is an umbrella term for emerging techniques attempting to generalize (structured) deep neural models to non-Euclidean domains such as graphs and manifolds. The purpose of this paper is to overview different examples of geometric deep learning problems and present available solutions, key difficulties, applications, and future research directions in this nascent field.

Recent years have witnessed a growth in mathematics for deep learning--which seeks a deeper understanding of the concepts of deep learning with mathematics, and explores how to make it more robust--and deep learning for mathematics, where deep learning algorithms are used to solve problems in mathematics. The latter has popularised the field of scientific machine learning where deep learning is applied to problems in scientific computing. Specifically, more and more neural network architectures have been developed to solve specific classes of partial differential equations (PDEs). Such methods exploit properties that are inherent to PDEs and thus solve the PDEs better than classical feed-forward neural networks, recurrent neural networks, and convolutional neural networks. This has had a great impact in the area of mathematical modeling where parametric PDEs are widely used to model most natural and physical processes arising in science and engineering, In this work, we review such methods and extend them for parametric studies as well as for solving the related inverse problems. We equally proceed to show their relevance in some industrial applications.

We study characteristics of receptive fields of units in deep convolutional networks. The receptive field size is a crucial issue in many visual tasks, as the output must respond to large enough areas in the image to capture information about large objects. We introduce the notion of an effective receptive field, and show that it both has a Gaussian distribution and only occupies a fraction of the full theoretical receptive field. We analyze the effective receptive field in several architecture designs, and the effect of nonlinear activations, dropout, sub-sampling and skip connections on it. This leads to suggestions for ways to address its tendency to be too small.

我们为深度残留网络（RESNETS）提出了一种全球收敛的多级训练方法。设计的方法可以看作是递归多级信任区域（RMTR）方法的新型变体，该方法通过在训练过程中自适应调节迷你批量，在混合（随机确定性）设置中运行。多级层次结构和传输运算符是通过利用动力学系统的观点来构建的，该观点通过重新连接来解释远期传播作为对初始值问题的正向Euler离散化。与传统的培训方法相反，我们的新型RMTR方法还通过有限的内存SR1方法结合了有关多级层次结构各个级别的曲率信息。使用分类和回归领域的示例，对我们的多级训练方法的总体性能和收敛属性进行了数值研究。

通过卫星摄像机获取关于地球表面的大面积的信息使我们能够看到远远超过我们在地面上看到的更多。这有助于我们在检测和监测土地使用模式，大气条件，森林覆盖和许多非上市方面的地区的物理特征。所获得的图像不仅跟踪连续的自然现象，而且对解决严重森林砍伐的全球挑战也至关重要。其中亚马逊盆地每年占最大份额。适当的数据分析将有助于利用可持续健康的氛围来限制对生态系统和生物多样性的不利影响。本报告旨在通过不同的机器学习和优越的深度学习模型用大气和各种陆地覆盖或土地使用亚马逊雨林的卫星图像芯片。评估是基于F2度量完成的，而用于损耗函数，我们都有S形跨熵以及Softmax交叉熵。在使用预先训练的ImageNet架构中仅提取功能之后，图像被间接馈送到机器学习分类器。鉴于深度学习模型，通过传输学习使用微调Imagenet预训练模型的集合。到目前为止，我们的最佳分数与F2度量为0.927。

物理信息的神经网络（PINN）是神经网络（NNS），它们作为神经网络本身的组成部分编码模型方程，例如部分微分方程（PDE）。如今，PINN是用于求解PDE，分数方程，积分分化方程和随机PDE的。这种新颖的方法已成为一个多任务学习框架，在该框架中，NN必须在减少PDE残差的同时拟合观察到的数据。本文对PINNS的文献进行了全面的综述：虽然该研究的主要目标是表征这些网络及其相关的优势和缺点。该综述还试图将出版物纳入更广泛的基于搭配的物理知识的神经网络，这些神经网络构成了香草·皮恩（Vanilla Pinn）以及许多其他变体，例如物理受限的神经网络（PCNN），各种HP-VPINN，变量HP-VPINN，VPINN，VPINN，变体。和保守的Pinn（CPINN）。该研究表明，大多数研究都集中在通过不同的激活功能，梯度优化技术，神经网络结构和损耗功能结构来定制PINN。尽管使用PINN的应用范围广泛，但通过证明其在某些情况下比有限元方法（FEM）等经典数值技术更可行的能力，但仍有可能的进步，最著名的是尚未解决的理论问题。

这本数字本书包含在物理模拟的背景下与深度学习相关的一切实际和全面的一切。尽可能多，所有主题都带有Jupyter笔记本的形式的动手代码示例，以便快速入门。除了标准的受监督学习的数据中，我们将看看物理丢失约束，更紧密耦合的学习算法，具有可微分的模拟，以及加强学习和不确定性建模。我们生活在令人兴奋的时期：这些方法具有从根本上改变计算机模拟可以实现的巨大潜力。

多层erceptron（MLP），作为出现的第一个神经网络结构，是一个大的击中。但是由硬件计算能力和数据集的大小限制，它一旦沉没了数十年。在此期间，我们目睹了从手动特征提取到带有局部接收领域的CNN的范式转变，以及基于自我关注机制的全球接收领域的变换。今年（2021年），随着MLP混合器的推出，MLP已重新进入敏捷，并吸引了计算机视觉界的广泛研究。与传统的MLP进行比较，它变得更深，但改变了完全扁平化以补丁平整的输入。鉴于其高性能和较少的需求对视觉特定的感应偏见，但社区无法帮助奇迹，将MLP，最简单的结构与全球接受领域，但没有关注，成为一个新的电脑视觉范式吗？为了回答这个问题，本调查旨在全面概述视觉深层MLP模型的最新发展。具体而言，我们从微妙的子模块设计到全局网络结构，我们审查了这些视觉深度MLP。我们比较了不同网络设计的接收领域，计算复杂性和其他特性，以便清楚地了解MLP的开发路径。调查表明，MLPS的分辨率灵敏度和计算密度仍未得到解决，纯MLP逐渐发展朝向CNN样。我们建议，目前的数据量和计算能力尚未准备好接受纯的MLP，并且人工视觉指导仍然很重要。最后，我们提供了开放的研究方向和可能的未来作品的分析。我们希望这项努力能够点燃社区的进一步兴趣，并鼓励目前为神经网络进行更好的视觉量身定制设计。

我们通过应用更为理论证明的操作员来寻求改善神经网络中的汇集操作。我们证明Logsumexp提供了用于登录的自然或操作员。当一个人对池中汇集运算符中的元素数正确时，这将成为$ \ text {logavgexp}：= \ log（\ text {mean}（\ exp（x）））$。通过引入单个温度参数，LogavgeXP将其操作数的最大值平滑地过渡到平均值（在限制性情况下发现$ 0 ^ + $和$ t \ to + \ idty $）。在各种深度神经网络架构中，我们在实验测试的LogavgeXP，无论是没有学习的温度参数，都在电脑视觉中的各种深度神经网络架构中。

这是一门专门针对STEM学生开发的介绍性机器学习课程。我们的目标是为有兴趣的读者提供基础知识，以在自己的项目中使用机器学习，并将自己熟悉术语作为进一步阅读相关文献的基础。在这些讲义中，我们讨论受监督，无监督和强化学习。注释从没有神经网络的机器学习方法的说明开始，例如原理分析，T-SNE，聚类以及线性回归和线性分类器。我们继续介绍基本和先进的神经网络结构，例如密集的进料和常规神经网络，经常性的神经网络，受限的玻尔兹曼机器，（变性）自动编码器，生成的对抗性网络。讨论了潜在空间表示的解释性问题，并使用梦和对抗性攻击的例子。最后一部分致力于加强学习，我们在其中介绍了价值功能和政策学习的基本概念。

Computational units in artificial neural networks follow a simplified model of biological neurons. In the biological model, the output signal of a neuron runs down the axon, splits following the many branches at its end, and passes identically to all the downward neurons of the network. Each of the downward neurons will use their copy of this signal as one of many inputs dendrites, integrate them all and fire an output, if above some threshold. In the artificial neural network, this translates to the fact that the nonlinear filtering of the signal is performed in the upward neuron, meaning that in practice the same activation is shared between all the downward neurons that use that signal as their input. Dendrites thus play a passive role. We propose a slightly more complex model for the biological neuron, where dendrites play an active role: the activation in the output of the upward neuron becomes optional, and instead the signals going through each dendrite undergo independent nonlinear filterings, before the linear combination. We implement this new model into a ReLU computational unit and discuss its biological plausibility. We compare this new computational unit with the standard one and describe it from a geometrical point of view. We provide a Keras implementation of this unit into fully connected and convolutional layers and estimate their FLOPs and weights change. We then use these layers in ResNet architectures on CIFAR-10, CIFAR-100, Imagenette, and Imagewoof, obtaining performance improvements over standard ResNets up to 1.73%. Finally, we prove a universal representation theorem for continuous functions on compact sets and show that this new unit has more representational power than its standard counterpart.

Deep neural networks (DNNs) recently emerged as a promising tool for analyzing and solving complex differential equations arising in science and engineering applications. Alternative to traditional numerical schemes, learning-based solvers utilize the representation power of DNNs to approximate the input-output relations in an automated manner. However, the lack of physics-in-the-loop often makes it difficult to construct a neural network solver that simultaneously achieves high accuracy, low computational burden, and interpretability. In this work, focusing on a class of evolutionary PDEs characterized by having decomposable operators, we show that the classical ``operator splitting'' numerical scheme of solving these equations can be exploited to design neural network architectures. This gives rise to a learning-based PDE solver, which we name Deep Operator-Splitting Network (DOSnet). Such non-black-box network design is constructed from the physical rules and operators governing the underlying dynamics contains learnable parameters, and is thus more flexible than the standard operator splitting scheme. Once trained, it enables the fast solution of the same type of PDEs. To validate the special structure inside DOSnet, we take the linear PDEs as the benchmark and give the mathematical explanation for the weight behavior. Furthermore, to demonstrate the advantages of our new AI-enhanced PDE solver, we train and validate it on several types of operator-decomposable differential equations. We also apply DOSnet to nonlinear Schr\"odinger equations (NLSE) which have important applications in the signal processing for modern optical fiber transmission systems, and experimental results show that our model has better accuracy and lower computational complexity than numerical schemes and the baseline DNNs.