We investigate the parameterization of deep neural networks that by design satisfy the continuity equation, a fundamental conservation law. This is enabled by the observation that any solution of the continuity equation can be represented as a divergence-free vector field. We hence propose building divergence-free neural networks through the concept of differential forms, and with the aid of automatic differentiation, realize two practical constructions. As a result, we can parameterize pairs of densities and vector fields that always exactly satisfy the continuity equation, foregoing the need for extra penalty methods or expensive numerical simulation. Furthermore, we prove these models are universal and so can be used to represent any divergence-free vector field. Finally, we experimentally validate our approaches by computing neural network-based solutions to fluid equations, solving for the Hodge decomposition, and learning dynamical optimal transport maps.

Optimal Transport（OT）提供了一个多功能框架，以几何有意义的方式比较复杂的数据分布。计算Wasserstein距离和概率措施之间的大地测量方法的传统方法需要网络依赖性域离散化，并且受差异性的诅咒。我们提出了Geonet，这是一个网状不变的深神经操作员网络，该网络从输入对的初始和终端分布对到Wasserstein Geodesic连接两个端点分布的非线性映射。在离线训练阶段，Geonet了解了以耦合PDE系统为特征的原始和双空间中OT问题动态提出的鞍点最佳条件。随后的推理阶段是瞬时的，可以在在线学习环境中进行实时预测。我们证明，Geonet在模拟示例和CIFAR-10数据集上达到了与标准OT求解器的可比测试精度，其推断阶段计算成本大大降低了。

Fokker-Planck方程（FPE）是控制IT \^o过程密度演变的部分微分方程，并且对统计物理学和机器学习的文献非常重要。 FPE可以被视为连续性方程，其中密度的变化完全由时间变化的速度场决定。重要的是，此速度场也取决于当前密度函数。结果，可以证明地面真相速度字段是固定点方程的解决方案，即我们称之为自洽的属性。在本文中，我们利用这一概念来设计假设速度字段的潜在功能，并证明，如果在训练过程中这样的功能减少到零，则假设速度场产生的密度轨迹会收敛到解决方案转化为解决方案。 Wasserstein-2的FPE。所提出的潜在函数可与基于神经网络的参数化相提并论，因为可以有效地计算相对于参数的随机梯度。一旦训练了一个参数化模型，例如神经普通微分方程，我们就可以生成FPE的整个轨迹。

Recent years have witnessed a growth in mathematics for deep learning--which seeks a deeper understanding of the concepts of deep learning with mathematics, and explores how to make it more robust--and deep learning for mathematics, where deep learning algorithms are used to solve problems in mathematics. The latter has popularised the field of scientific machine learning where deep learning is applied to problems in scientific computing. Specifically, more and more neural network architectures have been developed to solve specific classes of partial differential equations (PDEs). Such methods exploit properties that are inherent to PDEs and thus solve the PDEs better than classical feed-forward neural networks, recurrent neural networks, and convolutional neural networks. This has had a great impact in the area of mathematical modeling where parametric PDEs are widely used to model most natural and physical processes arising in science and engineering, In this work, we review such methods and extend them for parametric studies as well as for solving the related inverse problems. We equally proceed to show their relevance in some industrial applications.

Harmonic functions are abundant in nature, appearing in limiting cases of Maxwell's, Navier-Stokes equations, the heat and the wave equation. Consequently, there are many applications of harmonic functions, spanning applications from industrial process optimisation to robotic path planning and the calculation of first exit times of random walks. Despite their ubiquity and relevance, there have been few attempts to develop effective means of representing harmonic functions in the context of machine learning architectures, either in machine learning on classical computers, or in the nascent field of quantum machine learning. Architectures which impose or encourage an inductive bias towards harmonic functions would facilitate data-driven modelling and the solution of inverse problems in a range of applications. For classical neural networks, it has already been established how leveraging inductive biases can in general lead to improved performance of learning algorithms. The introduction of such inductive biases within a quantum machine learning setting is instead still in its nascent stages. In this work, we derive exactly-harmonic (conventional- and quantum-) neural networks in two dimensions for simply-connected domains by leveraging the characteristics of holomorphic complex functions. We then demonstrate how these can be approximately extended to multiply-connected two-dimensional domains using techniques inspired by domain decomposition in physics-informed neural networks. We further provide architectures and training protocols to effectively impose approximately harmonic constraints in three dimensions and higher, and as a corollary we report divergence-free network architectures in arbitrary dimensions. Our approaches are demonstrated with applications to heat transfer, electrostatics and robot navigation, with comparisons to physics-informed neural networks included.

物理信息的神经网络（PINN）是神经网络（NNS），它们作为神经网络本身的组成部分编码模型方程，例如部分微分方程（PDE）。如今，PINN是用于求解PDE，分数方程，积分分化方程和随机PDE的。这种新颖的方法已成为一个多任务学习框架，在该框架中，NN必须在减少PDE残差的同时拟合观察到的数据。本文对PINNS的文献进行了全面的综述：虽然该研究的主要目标是表征这些网络及其相关的优势和缺点。该综述还试图将出版物纳入更广泛的基于搭配的物理知识的神经网络，这些神经网络构成了香草·皮恩（Vanilla Pinn）以及许多其他变体，例如物理受限的神经网络（PCNN），各种HP-VPINN，变量HP-VPINN，VPINN，VPINN，变体。和保守的Pinn（CPINN）。该研究表明，大多数研究都集中在通过不同的激活功能，梯度优化技术，神经网络结构和损耗功能结构来定制PINN。尽管使用PINN的应用范围广泛，但通过证明其在某些情况下比有限元方法（FEM）等经典数值技术更可行的能力，但仍有可能的进步，最著名的是尚未解决的理论问题。

神经网络的经典发展主要集中在有限维欧基德空间或有限组之间的学习映射。我们提出了神经网络的概括，以学习映射无限尺寸函数空间之间的运算符。我们通过一类线性积分运算符和非线性激活函数的组成制定运营商的近似，使得组合的操作员可以近似复杂的非线性运算符。我们证明了我们建筑的普遍近似定理。此外，我们介绍了四类运算符参数化：基于图形的运算符，低秩运算符，基于多极图形的运算符和傅里叶运算符，并描述了每个用于用每个计算的高效算法。所提出的神经运营商是决议不变的：它们在底层函数空间的不同离散化之间共享相同的网络参数，并且可以用于零击超分辨率。在数值上，与现有的基于机器学习的方法，达西流程和Navier-Stokes方程相比，所提出的模型显示出卓越的性能，而与传统的PDE求解器相比，与现有的基于机器学习的方法有关的基于机器学习的方法。

在本文中，我们提出了一种无网格的方法来解决完整的Stokes方程，该方程用非线性流变学模拟了冰川运动。我们的方法是受[12]中提出的深里兹方法的启发。我们首先将非牛顿冰流模型的解决方案提出到具有边界约束的变分积分的最小化器中。然后，通过一个深神经网络近似溶液，该网络的损失函数是变异积分加上混合边界条件的软约束。我们的方法不需要引入网格网格或基础函数来评估损失函数，而只需要统一的域和边界采样器。为了解决现实世界缩放中的不稳定性，我们将网络的输入重新归一致，并平衡每个单独边界的正则化因子。最后，我们通过几个数值实验说明了我们方法的性能，包括具有分析解决方案的2D模型，具有真实缩放的Arolla Glacier模型和具有周期性边界条件的3D模型。数值结果表明，我们提出的方法有效地解决了通过非线性流变学引起的冰川建模引起的非牛顿力学。

在高维度中整合时间依赖性的fokker-planck方程的选择方法是通过集成相关的随机微分方程来生成溶液中的样品。在这里，我们介绍了基于整合描述概率流的普通微分方程的替代方案。与随机动力学不同，该方程式在以后的任何时候都会从初始密度将样品从溶液中的样品推到样品。该方法具有直接访问数量的优势，这些数量挑战仅估算仅给定解决方案的样品，例如概率电流，密度本身及其熵。概率流程方程取决于溶液对数的梯度（其“得分”），因此A-Priori未知也是如此。为了解决这种依赖性，我们用一个深神网络对分数进行建模，该网络通过根据瞬时概率电流传播一组粒子来实现，该网络可以在直接学习中学习。我们的方法是基于基于得分的生成建模的最新进展，其重要区别是训练程序是独立的，并且不需要来自目标密度的样本才能事先可用。为了证明该方法的有效性，我们考虑了相互作用粒子系统物理学的几个示例。我们发现该方法可以很好地缩放到高维系统，并准确匹配可用的分析解决方案和通过蒙特卡洛计算的力矩。

We propose characteristic-informed neural networks (CINN), a simple and efficient machine learning approach for solving forward and inverse problems involving hyperbolic PDEs. Like physics-informed neural networks (PINN), CINN is a meshless machine learning solver with universal approximation capabilities. Unlike PINN, which enforces a PDE softly via a multi-part loss function, CINN encodes the characteristics of the PDE in a general-purpose deep neural network trained with the usual MSE data-fitting regression loss and standard deep learning optimization methods. This leads to faster training and can avoid well-known pathologies of gradient descent optimization of multi-part PINN loss functions. If the characteristic ODEs can be solved exactly, which is true in important cases, the output of a CINN is an exact solution of the PDE, even at initialization, preventing the occurrence of non-physical outputs. Otherwise, the ODEs must be solved approximately, but the CINN is still trained only using a data-fitting loss function. The performance of CINN is assessed empirically in forward and inverse linear hyperbolic problems. These preliminary results indicate that CINN is able to improve on the accuracy of the baseline PINN, while being nearly twice as fast to train and avoiding non-physical solutions. Future extensions to hyperbolic PDE systems and nonlinear PDEs are also briefly discussed.

We propose the tensorizing flow method for estimating high-dimensional probability density functions from the observed data. The method is based on tensor-train and flow-based generative modeling. Our method first efficiently constructs an approximate density in the tensor-train form via solving the tensor cores from a linear system based on the kernel density estimators of low-dimensional marginals. We then train a continuous-time flow model from this tensor-train density to the observed empirical distribution by performing a maximum likelihood estimation. The proposed method combines the optimization-less feature of the tensor-train with the flexibility of the flow-based generative models. Numerical results are included to demonstrate the performance of the proposed method.

标准化流量（NF）是基于可能性的强大生成模型，能够在表达性和拖延性之间进行折衷，以模拟复杂的密度。现已建立的研究途径利用了最佳运输（OT），并寻找Monge地图，即源和目标分布之间的努力最小的模型。本文介绍了一种基于Brenier的极性分解定理的方法，该方法将任何受过训练的NF转换为更高效率的版本而不改变最终密度。我们通过学习源（高斯）分布的重新排列来最大程度地减少源和最终密度之间的OT成本。由于Euler的方程式，我们进一步限制了导致估计的Monge图的路径，将估计的Monge地图放在量化量的差异方程的空间中。所提出的方法导致几种现有模型的OT成本降低的平滑流动，而不会影响模型性能。

These notes were compiled as lecture notes for a course developed and taught at the University of the Southern California. They should be accessible to a typical engineering graduate student with a strong background in Applied Mathematics. The main objective of these notes is to introduce a student who is familiar with concepts in linear algebra and partial differential equations to select topics in deep learning. These lecture notes exploit the strong connections between deep learning algorithms and the more conventional techniques of computational physics to achieve two goals. First, they use concepts from computational physics to develop an understanding of deep learning algorithms. Not surprisingly, many concepts in deep learning can be connected to similar concepts in computational physics, and one can utilize this connection to better understand these algorithms. Second, several novel deep learning algorithms can be used to solve challenging problems in computational physics. Thus, they offer someone who is interested in modeling a physical phenomena with a complementary set of tools.

在科学的背景下，众所周知的格言“一张图片胜过千言万语”可能是“一个型号胜过一千个数据集”。在本手稿中，我们将Sciml软件生态系统介绍作为混合物理法律和科学模型的信息，并使用数据驱动的机器学习方法。我们描述了一个数学对象，我们表示通用微分方程（UDE），作为连接生态系统的统一框架。我们展示了各种各样的应用程序，从自动发现解决高维汉密尔顿 - Jacobi-Bellman方程的生物机制，可以通过UDE形式主义和工具进行措辞和有效地处理。我们展示了软件工具的一般性，以处理随机性，延迟和隐式约束。这使得各种SCIML应用程序变为核心训练机构的核心集，这些训练机构高度优化，稳定硬化方程，并与分布式并行性和GPU加速器兼容。

We present a unified hard-constraint framework for solving geometrically complex PDEs with neural networks, where the most commonly used Dirichlet, Neumann, and Robin boundary conditions (BCs) are considered. Specifically, we first introduce the "extra fields" from the mixed finite element method to reformulate the PDEs so as to equivalently transform the three types of BCs into linear forms. Based on the reformulation, we derive the general solutions of the BCs analytically, which are employed to construct an ansatz that automatically satisfies the BCs. With such a framework, we can train the neural networks without adding extra loss terms and thus efficiently handle geometrically complex PDEs, alleviating the unbalanced competition between the loss terms corresponding to the BCs and PDEs. We theoretically demonstrate that the "extra fields" can stabilize the training process. Experimental results on real-world geometrically complex PDEs showcase the effectiveness of our method compared with state-of-the-art baselines.

Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows through the lens of probabilistic modeling and inference. We place special emphasis on the fundamental principles of flow design, and discuss foundational topics such as expressive power and computational trade-offs. We also broaden the conceptual framing of flows by relating them to more general probability transformations. Lastly, we summarize the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.

Deep learning has achieved remarkable success in diverse applications; however, its use in solving partial differential equations (PDEs) has emerged only recently. Here, we present an overview of physics-informed neural networks (PINNs), which embed a PDE into the loss of the neural network using automatic differentiation. The PINN algorithm is simple, and it can be applied to different types of PDEs, including integro-differential equations, fractional PDEs, and stochastic PDEs. Moreover, from the implementation point of view, PINNs solve inverse problems as easily as forward problems. We propose a new residual-based adaptive refinement (RAR) method to improve the training efficiency of PINNs. For pedagogical reasons, we compare the PINN algorithm to a standard finite element method. We also present a Python library for PINNs, DeepXDE, which is designed to serve both as an education tool to be used in the classroom as well as a research tool for solving problems in computational science and engineering. Specifically, DeepXDE can solve forward problems given initial and boundary conditions, as well as inverse problems given some extra measurements. DeepXDE supports complex-geometry domains based on the technique of constructive solid geometry, and enables the user code to be compact, resembling closely the mathematical formulation. We introduce the usage of DeepXDE and its customizability, and we also demonstrate the capability of PINNs and the user-friendliness of DeepXDE for five different examples. More broadly, DeepXDE contributes to the more rapid development of the emerging Scientific Machine Learning field.

在本文中，我们演示并调查了一些挑战，这些挑战阻碍了使用物理知识的神经网络解决复杂问题的方式。特别是，我们可视化受过训练的模型的损失景观，并在存在物理学的情况下对反向传播梯度进行灵敏度分析。我们的发现表明，现有的方法产生了难以导航的高度非凸损失景观。此外，高阶PDE污染了可能阻碍或防止收敛的反向传播梯度。然后，我们提出了一种新的方法，该方法绕过了高阶PDE操作员的计算并减轻反向传播梯度的污染。为此，我们降低了解决方案搜索空间的维度，并通过非平滑解决方案促进学习问题。我们的配方还提供了一种反馈机制，可帮助我们的模型适应地专注于难以学习的领域的复杂区域。然后，我们通过调整Lagrange乘数方法来提出一个无约束的二重问题。我们运用我们的方法来解决由线性和非线性PDE控制的几个具有挑战性的基准问题。

High-dimensional PDEs have been a longstanding computational challenge. We propose to solve highdimensional PDEs by approximating the solution with a deep neural network which is trained to satisfy the differential operator, initial condition, and boundary conditions. Our algorithm is meshfree, which is key since meshes become infeasible in higher dimensions. Instead of forming a mesh, the neural network is trained on batches of randomly sampled time and space points. The algorithm is tested on a class of high-dimensional free boundary PDEs, which we are able to accurately solve in up to 200 dimensions. The algorithm is also tested on a high-dimensional Hamilton-Jacobi-Bellman PDE and Burgers' equation. The deep learning algorithm approximates the general solution to the Burgers' equation for a continuum of different boundary conditions and physical conditions (which can be viewed as a high-dimensional space). We call the algorithm a "Deep Galerkin Method (DGM)" since it is similar in spirit to Galerkin methods, with the solution approximated by a neural network instead of a linear combination of basis functions. In addition, we prove a theorem regarding the approximation power of neural networks for a class of quasilinear parabolic PDEs.

在许多学科中，动态系统的数据信息预测模型的开发引起了广泛的兴趣。我们提出了一个统一的框架，用于混合机械和机器学习方法，以从嘈杂和部分观察到的数据中识别动态系统。我们将纯数据驱动的学习与混合模型进行比较，这些学习结合了不完善的域知识。我们的公式与所选的机器学习模型不可知，在连续和离散的时间设置中都呈现，并且与表现出很大的内存和错误的模型误差兼容。首先，我们从学习理论的角度研究无内存线性（W.R.T.参数依赖性）模型误差，从而定义了过多的风险和概括误差。对于沿阵行的连续时间系统，我们证明，多余的风险和泛化误差都通过与T的正方形介于T的术语（指定训练数据的时间间隔）的术语界定。其次，我们研究了通过记忆建模而受益的方案，证明了两类连续时间复发性神经网络（RNN）的通用近似定理：两者都可以学习与内存有关的模型误差。此外，我们将一类RNN连接到储层计算，从而将学习依赖性错误的学习与使用随机特征在Banach空间之间进行监督学习的最新工作联系起来。给出了数值结果（Lorenz '63，Lorenz '96多尺度系统），以比较纯粹的数据驱动和混合方法，发现混合方法较少，渴望数据较少，并且更有效。最后，我们从数值上证明了如何利用数据同化来从嘈杂，部分观察到的数据中学习隐藏的动态，并说明了通过这种方法和培训此类模型来表示记忆的挑战。