A normalizing flow (NF) is a mapping that transforms a chosen probability distribution to a normal distribution. Such flows are a common technique used for data generation and density estimation in machine learning and data science. The density estimate obtained with a NF requires a change of variables formula that involves the computation of the Jacobian determinant of the NF transformation. In order to tractably compute this determinant, continuous normalizing flows (CNF) estimate the mapping and its Jacobian determinant using a neural ODE. Optimal transport (OT) theory has been successfully used to assist in finding CNFs by formulating them as OT problems with a soft penalty for enforcing the standard normal distribution as a target measure. A drawback of OT-based CNFs is the addition of a hyperparameter, $\alpha$, that controls the strength of the soft penalty and requires significant tuning. We present JKO-Flow, an algorithm to solve OT-based CNF without the need of tuning $\alpha$. This is achieved by integrating the OT CNF framework into a Wasserstein gradient flow framework, also known as the JKO scheme. Instead of tuning $\alpha$, we repeatedly solve the optimization problem for a fixed $\alpha$ effectively performing a JKO update with a time-step $\alpha$. Hence we obtain a "divide and conquer" algorithm by repeatedly solving simpler problems instead of solving a potentially harder problem with large $\alpha$.

平均场游戏（MFGS）是针对具有大量交互代理的系统的建模框架。他们在经济学，金融和游戏理论中有应用。标准化流（NFS）是一个深层生成模型的家族，通过使用可逆映射来计算数据的可能性，该映射通常通过使用神经网络进行参数化。它们对于密度建模和数据生成很有用。尽管对这两种模型进行了积极的研究，但很少有人注意到两者之间的关系。在这项工作中，我们通过将NF的训练视为解决MFG来揭示MFGS和NFS之间的联系。这是通过根据试剂轨迹重新解决MFG问题的实现，并通过流量体系结构对所得MFG的离散化进行参数化。通过这种联系，我们探讨了两个研究方向。首先，我们采用表达的NF体系结构来准确地求解高维MFG，以避开传统数值方法中维度的诅咒。与其他深度学习方法相比，我们的基于轨迹的公式编码神经网络中的连续性方程，从而更好地近似人口动态。其次，我们对NFS进行运输成本的培训正规，并显示了控制模型Lipschitz绑定的有效性，从而获得了更好的概括性能。我们通过对各种合成和现实生活数据集的全面实验来展示数值结果。

Wasserstein-Fisher-Rao（WFR）距离是一个指标家族，用于评估两种ra措施的差异，这同时考虑了运输和重量的变化。球形WFR距离是WFR距离的投影版本，以实现概率措施，因此配备了WFR的ra尺度空间可以在概率测量的空间中，用球形WFR视为公式锥。与Wasserstein距离相比，在球形WFR下对大地测量学的理解尚不清楚，并且仍然是持续的研究重点。在本文中，我们开发了一个深度学习框架，以计算球形WFR指标下的大地测量学，并且可以采用学习的大地测量学来生成加权样品。我们的方法基于球形WFR的Benamou-Brenier型动态配方。为了克服重量变化带来的边界约束的困难，将基于反向映射的kullback-leibler（KL）发散术语引入成本函数。此外，引入了使用粒子速度的新的正则化项，以替代汉密尔顿 - 雅各比方程的动态公式中的潜力。当用于样品生成时，与先前的流量模型相比，与给定加权样品的应用相比，我们的框架可能对具有给定加权样品的应用有益。

Normalizing flow is a class of deep generative models for efficient sampling and density estimation. In practice, the flow often appears as a chain of invertible neural network blocks; to facilitate training, existing works have regularized flow trajectories and designed special network architectures. The current paper develops a neural ODE flow network inspired by the Jordan-Kinderleherer-Otto (JKO) scheme, which allows efficient block-wise training of the residual blocks and avoids inner loops of score matching or variational learning. As the JKO scheme unfolds the dynamic of gradient flow, the proposed model naturally stacks residual network blocks one-by-one, reducing the memory load and difficulty of performing end-to-end training of deep flow networks. We also develop adaptive time reparameterization of the flow network with a progressive refinement of the trajectory in probability space, which improves the model training efficiency and accuracy in practice. Using numerical experiments with synthetic and real data, we show that the proposed JKO-iFlow model achieves similar or better performance in generating new samples compared with existing flow and diffusion models at a significantly reduced computational and memory cost.

标准化流量（NF）是基于可能性的强大生成模型，能够在表达性和拖延性之间进行折衷，以模拟复杂的密度。现已建立的研究途径利用了最佳运输（OT），并寻找Monge地图，即源和目标分布之间的努力最小的模型。本文介绍了一种基于Brenier的极性分解定理的方法，该方法将任何受过训练的NF转换为更高效率的版本而不改变最终密度。我们通过学习源（高斯）分布的重新排列来最大程度地减少源和最终密度之间的OT成本。由于Euler的方程式，我们进一步限制了导致估计的Monge图的路径，将估计的Monge地图放在量化量的差异方程的空间中。所提出的方法导致几种现有模型的OT成本降低的平滑流动，而不会影响模型性能。

在概率密度范围内相对于Wassersein度量的空间的梯度流程通常具有很好的特性，并且已在几种机器学习应用中使用。计算Wasserstein梯度流量的标准方法是有限差异，使网格上的基础空间离散，并且不可扩展。在这项工作中，我们提出了一种可扩展的近端梯度型算法，用于Wassersein梯度流。我们的方法的关键是目标函数的变分形式，这使得可以通过引流 - 双重优化实现JKO近端地图。可以通过替代地更新内部和外环中的参数来有效地解决该原始问题。我们的框架涵盖了包括热方程和多孔介质方程的所有经典Wasserstein梯度流。我们展示了若干数值示例的算法的性能和可扩展性。

渐变流是一种强大的工具，用于优化一般度量空间中的功能，包括赋予WasserseIn度量标准的概率空间。解决这种优化问题的典型方法依赖于它与最佳运输的动态配方的连接和庆祝的Jordan-KinderLehrer-Otto（JKO）方案。然而，该制剂涉及优化凸起功能，这是具有挑战性的，尤其是高维度。在这项工作中，我们提出了一种依赖于最近引入的输入 - 凸神经网络（ICNN）的方法，以参加凸起功能的空间，以便近似JKO方案，以及在享受收敛保证的措施中设计功能。我们推出了这种JKO-ICNN框架的计算上有效的实现，并通过了解具有已知解决方案的低维局部微分方程的近似解的可行性和有效性。我们还通过对分子发现的受控生成的实验展示其在高维应用中的可行性。

Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows through the lens of probabilistic modeling and inference. We place special emphasis on the fundamental principles of flow design, and discuss foundational topics such as expressive power and computational trade-offs. We also broaden the conceptual framing of flows by relating them to more general probability transformations. Lastly, we summarize the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.

最佳运输（OT）提供了比较和映射概率度量的有效工具。我们建议利用神经网络的灵活性学习近似的最佳传输图。更确切地说，我们提出了一种新的原始方法，以解决将有限的样本集与第一个基础未知分布相关的有限样本，向另一个未知分布中绘制的有限样本集有关。我们表明，可逆神经网络的特定实例，即归一化流，可用于近似一对经验分布之间的该OT问题的解决方案。为此，我们建议通过通过最小化相应的瓦斯坦距离来替换推送前措施的相等性约束来放松OT的蒙加公式。然后将要检索的推向运算符被限制为正常化的流，该流程通过优化所得的成本函数来训练。这种方法允许将传输图离散作为函数的组成。这些功能中的每一个都与网络的一个子流有关，其输出提供了原始测量和目标度量之间传输的中间步骤。这种离散化也产生了两种感兴趣量度之间的一组中间重点。在玩具示例上进行的实验以及无监督翻译的具有挑战性的任务证明了该方法的兴趣。最后，一些实验表明，提出的方法导致了真实OT的良好近似值。

标准化流动，扩散归一化流量和变形自动置换器是强大的生成模型。在本文中，我们提供了一个统一的框架来通过马尔可夫链处理这些方法。实际上，我们考虑随机标准化流量作为一对马尔可夫链，满足一些属性，并表明许多用于数据生成的最先进模型适合该框架。马尔可夫链的观点使我们能够将确定性层作为可逆的神经网络和随机层作为大都会加速层，Langevin层和变形自身偏移，以数学上的声音方式。除了具有Langevin层的密度的层，扩散层或变形自身形式，也可以处理与确定性层或大都会加热器层没有密度的层。因此，我们的框架建立了一个有用的数学工具来结合各种方法。

Lipschitz regularized f-divergences are constructed by imposing a bound on the Lipschitz constant of the discriminator in the variational representation. They interpolate between the Wasserstein metric and f-divergences and provide a flexible family of loss functions for non-absolutely continuous (e.g. empirical) distributions, possibly with heavy tails. We construct Lipschitz regularized gradient flows on the space of probability measures based on these divergences. Examples of such gradient flows are Lipschitz regularized Fokker-Planck and porous medium partial differential equations (PDEs) for the Kullback-Leibler and alpha-divergences, respectively. The regularization corresponds to imposing a Courant-Friedrichs-Lewy numerical stability condition on the PDEs. For empirical measures, the Lipschitz regularization on gradient flows induces a numerically stable transporter/discriminator particle algorithm, where the generative particles are transported along the gradient of the discriminator. The gradient structure leads to a regularized Fisher information (particle kinetic energy) used to track the convergence of the algorithm. The Lipschitz regularized discriminator can be implemented via neural network spectral normalization and the particle algorithm generates approximate samples from possibly high-dimensional distributions known only from data. Notably, our particle algorithm can generate synthetic data even in small sample size regimes. A new data processing inequality for the regularized divergence allows us to combine our particle algorithm with representation learning, e.g. autoencoder architectures. The resulting algorithm yields markedly improved generative properties in terms of efficiency and quality of the synthetic samples. From a statistical mechanics perspective the encoding can be interpreted dynamically as learning a better mobility for the generative particles.

Diffusion models have recently outperformed alternative approaches to model the distribution of natural images, such as GANs. Such diffusion models allow for deterministic sampling via the probability flow ODE, giving rise to a latent space and an encoder map. While having important practical applications, such as estimation of the likelihood, the theoretical properties of this map are not yet fully understood. In the present work, we partially address this question for the popular case of the VP SDE (DDPM) approach. We show that, perhaps surprisingly, the DDPM encoder map coincides with the optimal transport map for common distributions; we support this claim theoretically and by extensive numerical experiments.

在本文中，我们提出了一种确定性变分推理方法，通过最小化内核差异来产生低差异点，也称为最大均值差异或MMD。基于Wang Et的一般能量变分推理框架。 al。 （2021），最小化内核差异被转换为通过显式欧拉方案求解动态颂音系统。我们将结果算法EVI-MMD命名，并通过其中统一化常数的常规规定常量规定的实例，并以培训数据的形式明确地已知的示例。与分布近似，数值集成和生成式学习中的应用中的替代方法相比，其性能令人满意。 EVI-MMD算法克服了现有MMD-DESCLITHMS的瓶颈，主要适用于两个样本问题。可以在EVI框架下开发具有更复杂结构和潜在优势的算法。

We investigate the parameterization of deep neural networks that by design satisfy the continuity equation, a fundamental conservation law. This is enabled by the observation that any solution of the continuity equation can be represented as a divergence-free vector field. We hence propose building divergence-free neural networks through the concept of differential forms, and with the aid of automatic differentiation, realize two practical constructions. As a result, we can parameterize pairs of densities and vector fields that always exactly satisfy the continuity equation, foregoing the need for extra penalty methods or expensive numerical simulation. Furthermore, we prove these models are universal and so can be used to represent any divergence-free vector field. Finally, we experimentally validate our approaches by computing neural network-based solutions to fluid equations, solving for the Hodge decomposition, and learning dynamical optimal transport maps.

The modeling of probability distributions, specifically generative modeling and density estimation, has become an immensely popular subject in recent years by virtue of its outstanding performance on sophisticated data such as images and texts. Nevertheless, a theoretical understanding of its success is still incomplete. One mystery is the paradox between memorization and generalization: In theory, the model is trained to be exactly the same as the empirical distribution of the finite samples, whereas in practice, the trained model can generate new samples or estimate the likelihood of unseen samples. Likewise, the overwhelming diversity of distribution learning models calls for a unified perspective on this subject. This paper provides a mathematical framework such that all the well-known models can be derived based on simple principles. To demonstrate its efficacy, we present a survey of our results on the approximation error, training error and generalization error of these models, which can all be established based on this framework. In particular, the aforementioned paradox is resolved by proving that these models enjoy implicit regularization during training, so that the generalization error at early-stopping avoids the curse of dimensionality. Furthermore, we provide some new results on landscape analysis and the mode collapse phenomenon.

Normalizing Flows are generative models which produce tractable distributions where both sampling and density evaluation can be efficient and exact. The goal of this survey article is to give a coherent and comprehensive review of the literature around the construction and use of Normalizing Flows for distribution learning. We aim to provide context and explanation of the models, review current state-of-the-art literature, and identify open questions and promising future directions.

在这里，我们提出了一种称为歧管插值最佳传输流量（MIOFLOW）的方法，该方法从零星时间点上采集的静态快照样品中学习随机，连续的种群动力学。 Mioflow结合了动态模型，流动学习和通过训练神经普通微分方程（神经ode）的最佳运输，以在静态种群快照之间插值，以通过具有歧管地面距离的最佳运输来惩罚。此外，我们通过在自动编码器的潜在空间中运行我们称为Geodesic AutoCododer（GAE）来确保流量遵循几何形状。在GAE中，正规化了点之间的潜在空间距离，以匹配我们定义的数据歧管上的新型多尺度测量距离。我们表明，这种方法优于正常流，Schr \“ Odinger Bridges和其他旨在根据人群之间插值的噪声流向数据的生成模型。从理论上讲，我们将这些轨迹与动态最佳运输联系起来。我们评估了我们的评估使用分叉和合并的模拟数据，以及来自胚胎身体分化和急性髓样白血病的SCRNA-SEQ数据。

通过最小化kullback-leibler（kl）差异，变化推断近似于非差异分布。尽管这种差异对于计算有效，并且已在应用中广泛使用，但它具有一些不合理的属性。例如，它不是一个适当的度量标准，即，它是非对称的，也不保留三角形不等式。另一方面，最近的最佳运输距离显示出比KL差异的一些优势。在这些优势的帮助下，我们通过最大程度地减少切片的瓦斯汀距离，这是一种由最佳运输产生的有效度量，提出了一种新的变异推理方法。仅通过运行MCMC而不能解决任何优化问题，就可以简单地近似切片的Wasserstein距离。我们的近似值也不需要变异分布的易于处理密度函数，因此诸如神经网络之类的发电机可以摊销近似家庭。此外，我们提供了方法的理论特性分析。说明了关于合成和真实数据的实验，以显示提出的方法的性能。

标准化流是构建概率和生成模型的流行方法。但是，由于需要计算雅各布人的计算昂贵决定因素，因此对流量的最大似然训练是具有挑战性的。本文通过引入一种受到两样本测试启发的流动训练的方法来解决这一挑战。我们框架的核心是能源目标，这是适当评分规则的多维扩展，该规则基于随机预测，可以接受有效的估计器，并且超过了一系列可以在我们的框架中得出的替代两样本目标。至关重要的是，能量目标及其替代方案不需要计算决定因素，因此支持不适合最大似然训练的一般流量体系结构（例如，密度连接的网络）。我们从经验上证明，能量流达到竞争性生成建模性能，同时保持快速产生和后部推断。

我们提出了整流的流程，这是一种令人惊讶的简单学习方法（神经）的普通微分方程（ODE）模型，用于在两个经验观察到的分布\ pi_0和\ pi_1之间运输，因此为生成建模和域转移提供了统一的解决方案，以及其他各种任务。涉及分配运输。整流流的想法是学习ode，以遵循尽可能多的连接从\ pi_0和\ pi_1的直径。这是通过解决直接的非线性最小二乘优化问题来实现的，该问题可以轻松地缩放到大型模型，而无需在标准监督学习之外引入额外的参数。直径是特殊的，因此是特殊的，因为它们是两个点之间的最短路径，并且可以精确模拟而无需时间离散，因此可以在计算上产生高效的模型。我们表明，从数据（称为整流）中学习的整流流的过程将\ pi_0和\ pi_1的任意耦合转变为新的确定性耦合，并证明是非侵入的凸面运输成本。此外，递归应用矫正使我们能够获得具有越来越直的路径的流动序列，可以在推理阶段进行粗略的时间离散化来准确地模拟。在实证研究中，我们表明，整流流对图像产生，图像到图像翻译和域的适应性表现出色。特别是，在图像生成和翻译上，我们的方法几乎产生了几乎直流的流，即使是单个Euler离散步骤，也会产生高质量的结果。