在本文中,我们提出了一种确定性变分推理方法,通过最小化内核差异来产生低差异点,也称为最大均值差异或MMD。基于Wang Et的一般能量变分推理框架。 al。 (2021),最小化内核差异被转换为通过显式欧拉方案求解动态颂音系统。我们将结果算法EVI-MMD命名,并通过其中统一化常数的常规规定常量规定的实例,并以培训数据的形式明确地已知的示例。与分布近似,数值集成和生成式学习中的应用中的替代方法相比,其性能令人满意。 EVI-MMD算法克服了现有MMD-DESCLITHMS的瓶颈,主要适用于两个样本问题。可以在EVI框架下开发具有更复杂结构和潜在优势的算法。
translated by 谷歌翻译
We propose a general purpose variational inference algorithm that forms a natural counterpart of gradient descent for optimization. Our method iteratively transports a set of particles to match the target distribution, by applying a form of functional gradient descent that minimizes the KL divergence. Empirical studies are performed on various real world models and datasets, on which our method is competitive with existing state-of-the-art methods. The derivation of our method is based on a new theoretical result that connects the derivative of KL divergence under smooth transforms with Stein's identity and a recently proposed kernelized Stein discrepancy, which is of independent interest.
translated by 谷歌翻译
最近开发的基于粒子的变分推理(PARVI)方法通过迭代更新粒子的位置驱动一组\ EMPH {固定权重}粒子的经验分布。然而,固定的重量限制大大限制了经验分布的近似能力,特别是当粒子数有限时。在本文中,我们提出根据Fisher-Rao反应流动动态调整粒子的重量。我们根据新颖的连续复合流动开发了一种通用的动态重量粒子变分推理(DPVI)框架,其同时演化颗粒的位置和重量。我们表明,我们的复合流的平均场限制实际上是某些不相似函数$ \ mathcal {f} $的Wasserstein-fisher-Rao梯度流量,这导致$ \ mathcal {f} $的速度更快地减少Wassersein梯度流动现有的固定重量帕维。通过在我们的总框架中使用不同的有限粒子近似,我们推出了几种高效的DPVI算法。经验结果展示了我们的固定重量对应物的衍生DPVI算法的优越性。
translated by 谷歌翻译
我们为Nesterov在概率空间中加速的梯度流提供了一个框架,以设计有效的平均田间马尔可夫链蒙特卡洛(MCMC)贝叶斯逆问题算法。在这里,考虑了四个信息指标的示例,包括Fisher-Rao Metric,Wasserstein-2 Metric,Kalman-Wasserstein Metric和Stein Metric。对于Fisher-Rao和Wasserstein-2指标,我们都证明了加速梯度流的收敛性。在实施中,我们建议使用重新启动技术的Wasserstein-2,Kalman-Wasseintein和Stein加速梯度流的抽样效率离散算法。我们还制定了一种内核带宽选择方法,该方法从布朗动物样品中学习了密度对数的梯度。与最先进的算法相比,包括贝叶斯逻辑回归和贝叶斯神经网络在内的数值实验显示了所提出方法的强度。
translated by 谷歌翻译
Lipschitz regularized f-divergences are constructed by imposing a bound on the Lipschitz constant of the discriminator in the variational representation. They interpolate between the Wasserstein metric and f-divergences and provide a flexible family of loss functions for non-absolutely continuous (e.g. empirical) distributions, possibly with heavy tails. We construct Lipschitz regularized gradient flows on the space of probability measures based on these divergences. Examples of such gradient flows are Lipschitz regularized Fokker-Planck and porous medium partial differential equations (PDEs) for the Kullback-Leibler and alpha-divergences, respectively. The regularization corresponds to imposing a Courant-Friedrichs-Lewy numerical stability condition on the PDEs. For empirical measures, the Lipschitz regularization on gradient flows induces a numerically stable transporter/discriminator particle algorithm, where the generative particles are transported along the gradient of the discriminator. The gradient structure leads to a regularized Fisher information (particle kinetic energy) used to track the convergence of the algorithm. The Lipschitz regularized discriminator can be implemented via neural network spectral normalization and the particle algorithm generates approximate samples from possibly high-dimensional distributions known only from data. Notably, our particle algorithm can generate synthetic data even in small sample size regimes. A new data processing inequality for the regularized divergence allows us to combine our particle algorithm with representation learning, e.g. autoencoder architectures. The resulting algorithm yields markedly improved generative properties in terms of efficiency and quality of the synthetic samples. From a statistical mechanics perspective the encoding can be interpreted dynamically as learning a better mobility for the generative particles.
translated by 谷歌翻译
Multilevel Stein variational gradient descent is a method for particle-based variational inference that leverages hierarchies of approximations of target distributions with varying costs and fidelity to computationally speed up inference. This work provides a cost complexity analysis of multilevel Stein variational gradient descent that applies under milder conditions than previous results, especially in discrete-in-time regimes and beyond the limited settings where Stein variational gradient descent achieves exponentially fast convergence. The analysis shows that the convergence rate of Stein variational gradient descent enters only as a constant factor for the cost complexity of the multilevel version, which means that the costs of the multilevel version scale independently of the convergence rate of Stein variational gradient descent on a single level. Numerical experiments with Bayesian inverse problems of inferring discretized basal sliding coefficient fields of the Arolla glacier ice demonstrate that multilevel Stein variational gradient descent achieves orders of magnitude speedups compared to its single-level version.
translated by 谷歌翻译
在概率密度范围内相对于Wassersein度量的空间的梯度流程通常具有很好的特性,并且已在几种机器学习应用中使用。计算Wasserstein梯度流量的标准方法是有限差异,使网格上的基础空间离散,并且不可扩展。在这项工作中,我们提出了一种可扩展的近端梯度型算法,用于Wassersein梯度流。我们的方法的关键是目标函数的变分形式,这使得可以通过引流 - 双重优化实现JKO近端地图。可以通过替代地更新内部和外环中的参数来有效地解决该原始问题。我们的框架涵盖了包括热方程和多孔介质方程的所有经典Wasserstein梯度流。我们展示了若干数值示例的算法的性能和可扩展性。
translated by 谷歌翻译
在本章中,我们确定了基本的几何结构,这些几何结构是采样,优化,推理和自适应决策问题的基础。基于此识别,我们得出了利用这些几何结构来有效解决这些问题的算法。我们表明,在这些领域中自然出现了广泛的几何理论,范围从测量过程,信息差异,泊松几何和几何整合。具体而言,我们解释了(i)如何利用汉密尔顿系统的符合性几何形状,使我们能够构建(加速)采样和优化方法,(ii)希尔伯特亚空间和Stein操作员的理论提供了一种通用方法来获得可靠的估计器,(iii)(iii)(iii)保留决策的信息几何形状会产生执行主动推理的自适应剂。在整个过程中,我们强调了这些领域之间的丰富联系。例如,推论借鉴了抽样和优化,并且自适应决策通过推断其反事实后果来评估决策。我们的博览会提供了基本思想的概念概述,而不是技术讨论,可以在本文中的参考文献中找到。
translated by 谷歌翻译
变性推理(VI)为基于传统的采样方法提供了一种吸引人的替代方法,用于实施贝叶斯推断,因为其概念性的简单性,统计准确性和计算可扩展性。然而,常见的变分近似方案(例如平均场(MF)近似)需要某些共轭结构以促进有效的计算,这可能会增加不必要的限制对可行的先验分布家族,并对变异近似族对差异进行进一步的限制。在这项工作中,我们开发了一个通用计算框架,用于实施MF-VI VIA WASSERSTEIN梯度流(WGF),这是概率度量空间上的梯度流。当专门针对贝叶斯潜在变量模型时,我们将分析基于时间消化的WGF交替最小化方案的算法收敛,用于实现MF近似。特别是,所提出的算法类似于EM算法的分布版本,包括更新潜在变量变异分布的E step以及在参数的变异分布上进行最陡峭下降的m step。我们的理论分析依赖于概率度量空间中的最佳运输理论和细分微积分。我们证明了时间限制的WGF的指数收敛性,以最大程度地减少普通大地测量学严格的凸度的通用物镜功能。我们还提供了通过使用时间限制的WGF的固定点方程从MF近似获得的变异分布的指数收缩的新证明。我们将方法和理论应用于两个经典的贝叶斯潜在变量模型,即高斯混合模型和回归模型的混合物。还进行了数值实验,以补充这两个模型下的理论发现。
translated by 谷歌翻译
Wasserstein-Fisher-Rao(WFR)距离是一个指标家族,用于评估两种ra措施的差异,这同时考虑了运输和重量的变化。球形WFR距离是WFR距离的投影版本,以实现概率措施,因此配备了WFR的ra尺度空间可以在概率测量的空间中,用球形WFR视为公式锥。与Wasserstein距离相比,在球形WFR下对大地测量学的理解尚不清楚,并且仍然是持续的研究重点。在本文中,我们开发了一个深度学习框架,以计算球形WFR指标下的大地测量学,并且可以采用学习的大地测量学来生成加权样品。我们的方法基于球形WFR的Benamou-Brenier型动态配方。为了克服重量变化带来的边界约束的困难,将基于反向映射的kullback-leibler(KL)发散术语引入成本函数。此外,引入了使用粒子速度的新的正则化项,以替代汉密尔顿 - 雅各比方程的动态公式中的潜力。当用于样品生成时,与先前的流量模型相比,与给定加权样品的应用相比,我们的框架可能对具有给定加权样品的应用有益。
translated by 谷歌翻译
The modeling of probability distributions, specifically generative modeling and density estimation, has become an immensely popular subject in recent years by virtue of its outstanding performance on sophisticated data such as images and texts. Nevertheless, a theoretical understanding of its success is still incomplete. One mystery is the paradox between memorization and generalization: In theory, the model is trained to be exactly the same as the empirical distribution of the finite samples, whereas in practice, the trained model can generate new samples or estimate the likelihood of unseen samples. Likewise, the overwhelming diversity of distribution learning models calls for a unified perspective on this subject. This paper provides a mathematical framework such that all the well-known models can be derived based on simple principles. To demonstrate its efficacy, we present a survey of our results on the approximation error, training error and generalization error of these models, which can all be established based on this framework. In particular, the aforementioned paradox is resolved by proving that these models enjoy implicit regularization during training, so that the generalization error at early-stopping avoids the curse of dimensionality. Furthermore, we provide some new results on landscape analysis and the mode collapse phenomenon.
translated by 谷歌翻译
Simulator-based models are models for which the likelihood is intractable but simulation of synthetic data is possible. They are often used to describe complex real-world phenomena, and as such can often be misspecified in practice. Unfortunately, existing Bayesian approaches for simulators are known to perform poorly in those cases. In this paper, we propose a novel algorithm based on the posterior bootstrap and maximum mean discrepancy estimators. This leads to a highly-parallelisable Bayesian inference algorithm with strong robustness properties. This is demonstrated through an in-depth theoretical study which includes generalisation bounds and proofs of frequentist consistency and robustness of our posterior. The approach is then assessed on a range of examples including a g-and-k distribution and a toggle-switch model.
translated by 谷歌翻译
本文介绍了一种新的基于仿真的推理程序,以对访问I.I.D. \ samples的多维概率分布进行建模和样本,从而规避明确建模密度函数或设计Markov Chain Monte Carlo的通常方法。我们提出了一个称为可逆的Gromov-monge(RGM)距离的新概念的距离和同构的动机,并研究了RGM如何用于设计新的转换样本,以执行基于模拟的推断。我们的RGM采样器还可以估计两个异质度量度量空间之间的最佳对齐$(\ cx,\ mu,c _ {\ cx})$和$(\ cy,\ cy,\ nu,c _ {\ cy})$从经验数据集中,估计的地图大约将一个量度$ \ mu $推向另一个$ \ nu $,反之亦然。我们研究了RGM距离的分析特性,并在轻度条件下得出RGM等于经典的Gromov-Wasserstein距离。奇怪的是,与Brenier的两极分解结合了连接,我们表明RGM采样器以$ C _ {\ cx} $和$ C _ {\ cy} $的正确选择诱导了强度同构的偏见。研究了有关诱导采样器的收敛,表示和优化问题的统计率。还展示了展示RGM采样器有效性的合成和现实示例。
translated by 谷歌翻译
通过最小化kullback-leibler(kl)差异,变化推断近似于非差异分布。尽管这种差异对于计算有效,并且已在应用中广泛使用,但它具有一些不合理的属性。例如,它不是一个适当的度量标准,即,它是非对称的,也不保留三角形不等式。另一方面,最近的最佳运输距离显示出比KL差异的一些优势。在这些优势的帮助下,我们通过最大程度地减少切片的瓦斯汀距离,这是一种由最佳运输产生的有效度量,提出了一种新的变异推理方法。仅通过运行MCMC而不能解决任何优化问题,就可以简单地近似切片的Wasserstein距离。我们的近似值也不需要变异分布的易于处理密度函数,因此诸如神经网络之类的发电机可以摊销近似家庭。此外,我们提供了方法的理论特性分析。说明了关于合成和真实数据的实验,以显示提出的方法的性能。
translated by 谷歌翻译
我们考虑了最小化客观功能的优化问题,该问题允许变异形式,并根据\ textIt {约束域}上的概率分布定义,这对理论分析和算法设计构成了挑战。受镜下降算法的启发,我们提出了一种迭代和基于粒子的算法,称为镜像变异传输(\ textbf {mirriryvt})。对于每次迭代,\ textbf {mirrirvt}将粒子映射到由镜像映射引起的无约束的双空间,然后大约在通过推动粒子来定义的分布的歧管上大致执行wasserstein梯度下降。在迭代结束时,将粒子映射回原始的约束空间。通过模拟实验,我们证明了\ textbf {mirrirvt}的有效性,可以最大程度地限制函数,而不是单纯形和欧几里得球受到的域上的概率分布。我们还分析了其理论特性,并将其融合到目标功能的全局最小值。
translated by 谷歌翻译
Normalizing Flows are generative models which produce tractable distributions where both sampling and density evaluation can be efficient and exact. The goal of this survey article is to give a coherent and comprehensive review of the literature around the construction and use of Normalizing Flows for distribution learning. We aim to provide context and explanation of the models, review current state-of-the-art literature, and identify open questions and promising future directions.
translated by 谷歌翻译
A normalizing flow (NF) is a mapping that transforms a chosen probability distribution to a normal distribution. Such flows are a common technique used for data generation and density estimation in machine learning and data science. The density estimate obtained with a NF requires a change of variables formula that involves the computation of the Jacobian determinant of the NF transformation. In order to tractably compute this determinant, continuous normalizing flows (CNF) estimate the mapping and its Jacobian determinant using a neural ODE. Optimal transport (OT) theory has been successfully used to assist in finding CNFs by formulating them as OT problems with a soft penalty for enforcing the standard normal distribution as a target measure. A drawback of OT-based CNFs is the addition of a hyperparameter, $\alpha$, that controls the strength of the soft penalty and requires significant tuning. We present JKO-Flow, an algorithm to solve OT-based CNF without the need of tuning $\alpha$. This is achieved by integrating the OT CNF framework into a Wasserstein gradient flow framework, also known as the JKO scheme. Instead of tuning $\alpha$, we repeatedly solve the optimization problem for a fixed $\alpha$ effectively performing a JKO update with a time-step $\alpha$. Hence we obtain a "divide and conquer" algorithm by repeatedly solving simpler problems instead of solving a potentially harder problem with large $\alpha$.
translated by 谷歌翻译
We develop an online kernel Cumulative Sum (CUSUM) procedure, which consists of a parallel set of kernel statistics with different window sizes to account for the unknown change-point location. Compared with many existing sliding window-based kernel change-point detection procedures, which correspond to the Shewhart chart-type procedure, the proposed procedure is more sensitive to small changes. We further present a recursive computation of detection statistics, which is crucial for online procedures to achieve a constant computational and memory complexity, such that we do not need to calculate and remember the entire Gram matrix, which can be a computational bottleneck otherwise. We obtain precise analytic approximations of the two fundamental performance metrics, the Average Run Length (ARL) and Expected Detection Delay (EDD). Furthermore, we establish the optimal window size on the order of $\log ({\rm ARL})$ such that there is nearly no power loss compared with an oracle procedure, which is analogous to the classic result for window-limited Generalized Likelihood Ratio (GLR) procedure. We present extensive numerical experiments to validate our theoretical results and the competitive performance of the proposed method.
translated by 谷歌翻译
Normalizing flow is a class of deep generative models for efficient sampling and density estimation. In practice, the flow often appears as a chain of invertible neural network blocks; to facilitate training, existing works have regularized flow trajectories and designed special network architectures. The current paper develops a neural ODE flow network inspired by the Jordan-Kinderleherer-Otto (JKO) scheme, which allows efficient block-wise training of the residual blocks and avoids inner loops of score matching or variational learning. As the JKO scheme unfolds the dynamic of gradient flow, the proposed model naturally stacks residual network blocks one-by-one, reducing the memory load and difficulty of performing end-to-end training of deep flow networks. We also develop adaptive time reparameterization of the flow network with a progressive refinement of the trajectory in probability space, which improves the model training efficiency and accuracy in practice. Using numerical experiments with synthetic and real data, we show that the proposed JKO-iFlow model achieves similar or better performance in generating new samples compared with existing flow and diffusion models at a significantly reduced computational and memory cost.
translated by 谷歌翻译
在高维度中整合时间依赖性的fokker-planck方程的选择方法是通过集成相关的随机微分方程来生成溶液中的样品。在这里,我们介绍了基于整合描述概率流的普通微分方程的替代方案。与随机动力学不同,该方程式在以后的任何时候都会从初始密度将样品从溶液中的样品推到样品。该方法具有直接访问数量的优势,这些数量挑战仅估算仅给定解决方案的样品,例如概率电流,密度本身及其熵。概率流程方程取决于溶液对数的梯度(其“得分”),因此A-Priori未知也是如此。为了解决这种依赖性,我们用一个深神网络对分数进行建模,该网络通过根据瞬时概率电流传播一组粒子来实现,该网络可以在直接学习中学习。我们的方法是基于基于得分的生成建模的最新进展,其重要区别是训练程序是独立的,并且不需要来自目标密度的样本才能事先可用。为了证明该方法的有效性,我们考虑了相互作用粒子系统物理学的几个示例。我们发现该方法可以很好地缩放到高维系统,并准确匹配可用的分析解决方案和通过蒙特卡洛计算的力矩。
translated by 谷歌翻译