三角形流量,也称为kn \“{o}的Rosenblatt测量耦合,包括用于生成建模和密度估计的归一化流模型的重要构建块,包括诸如实值的非体积保存变换模型的流行自回归流模型(真实的NVP)。我们提出了三角形流量统计模型的统计保证和样本复杂性界限。特别是,我们建立了KN的统计一致性和kullback-leibler估算器的rospblatt的kullback-leibler估计的有限样本会聚率使用实证过程理论的工具测量耦合。我们的结果突出了三角形流动下播放功能类的各向异性几何形状,优化坐标排序,并导致雅各比比流动的统计保证。我们对合成数据进行数值实验,以说明我们理论发现的实际意义。
translated by 谷歌翻译
我们研究基于度量传输的非参数密度估计器的收敛性和相关距离。这些估计量代表了利息的度量,作为传输图下选择的参考分布的推动力,其中地图是通过最大似然目标选择(等效地,将经验性的kullback-leibler损失)或其受惩罚版本选择。我们通过将M估计的技术与基于运输的密度表示的分析性能相结合,为一般惩罚措施估计量的一般类别的措施运输估计器建立了浓度不平等。然后,我们证明了我们的理论对三角形knothe-rosenblatt(kr)在$ d $维单元方面的运输的含义,并表明该估计器的惩罚和未化的版本都达到了Minimax最佳收敛速率,超过了H \ \ \'“较旧的密度类别。具体来说,我们建立了在有限的h \“较旧型球上,未确定的非参数最大似然估计,然后在某些sobolev-penalate的估计器和筛分的小波估计器中建立了最佳速率。
translated by 谷歌翻译
度量的运输提供了一种用于建模复杂概率分布的多功能方法,并具有密度估计,贝叶斯推理,生成建模及其他方法的应用。单调三角传输地图$ \ unicode {x2014} $近似值$ \ unicode {x2013} $ rosenblatt(kr)重新安排$ \ unicode {x2014} $是这些任务的规范选择。然而,此类地图的表示和参数化对它们的一般性和表现力以及对从数据学习地图学习(例如,通过最大似然估计)出现的优化问题的属性产生了重大影响。我们提出了一个通用框架,用于通过平滑函数的可逆变换来表示单调三角图。我们建立了有关转化的条件,以使相关的无限维度最小化问题没有伪造的局部最小值,即所有局部最小值都是全球最小值。我们展示了满足某些尾巴条件的目标分布,唯一的全局最小化器与KR地图相对应。鉴于来自目标的样品,我们提出了一种自适应算法,该算法估计了基础KR映射的稀疏半参数近似。我们证明了如何将该框架应用于关节和条件密度估计,无可能的推断以及有向图形模型的结构学习,并在一系列样本量之间具有稳定的概括性能。
translated by 谷歌翻译
量化概率分布之间的异化的统计分歧(SDS)是统计推理和机器学习的基本组成部分。用于估计这些分歧的现代方法依赖于通过神经网络(NN)进行参数化经验变化形式并优化参数空间。这种神经估算器在实践中大量使用,但相应的性能保证是部分的,并呼吁进一步探索。特别是,涉及的两个错误源之间存在基本的权衡:近似和经验估计。虽然前者需要NN课程富有富有表现力,但后者依赖于控制复杂性。我们通过非渐近误差界限基于浅NN的基于浅NN的估计的估算权,重点关注四个流行的$ \ mathsf {f} $ - 分离 - kullback-leibler,chi squared,squared hellinger,以及总变异。我们分析依赖于实证过程理论的非渐近功能近似定理和工具。界限揭示了NN尺寸和样品数量之间的张力,并使能够表征其缩放速率,以确保一致性。对于紧凑型支持的分布,我们进一步表明,上述上三次分歧的神经估算器以适当的NN生长速率接近Minimax率 - 最佳,实现了对数因子的参数速率。
translated by 谷歌翻译
在因果推理和强盗文献中,基于观察数据的线性功能估算线性功能的问题是规范的。我们分析了首先估计治疗效果函数的广泛的两阶段程序,然后使用该数量来估计线性功能。我们证明了此类过程的均方误差上的非反应性上限:这些边界表明,为了获得非反应性最佳程序,应在特定加权$ l^2 $中最大程度地估算治疗效果的误差。 -规范。我们根据该加权规范的约束回归分析了两阶段的程序,并通过匹配非轴突局部局部最小值下限,在有限样品中建立了实例依赖性最优性。这些结果表明,除了取决于渐近效率方差之外,最佳的非质子风险除了取决于样本量支持的最富有函数类别的真实结果函数与其近似类别之间的加权规范距离。
translated by 谷歌翻译
对于高维和非参数统计模型,速率最优估计器平衡平方偏差和方差是一种常见的现象。虽然这种平衡被广泛观察到,但很少知道是否存在可以避免偏差和方差之间的权衡的方法。我们提出了一般的策略,以获得对任何估计方差的下限,偏差小于预先限定的界限。这表明偏差差异折衷的程度是不可避免的,并且允许量化不服从其的方法的性能损失。该方法基于许多抽象的下限,用于涉及关于不同概率措施的预期变化以及诸如Kullback-Leibler或Chi-Sque-diversence的信息措施的变化。其中一些不平等依赖于信息矩阵的新概念。在该物品的第二部分中,将抽象的下限应用于几种统计模型,包括高斯白噪声模型,边界估计问题,高斯序列模型和高维线性回归模型。对于这些特定的统计应用,发生不同类型的偏差差异发生,其实力变化很大。对于高斯白噪声模型中集成平方偏置和集成方差之间的权衡,我们将较低界限的一般策略与减少技术相结合。这允许我们将原始问题与估计的估计器中的偏差折衷联动,以更简单的统计模型中具有额外的对称性属性。在高斯序列模型中,发生偏差差异的不同相位转换。虽然偏差和方差之间存在非平凡的相互作用,但是平方偏差的速率和方差不必平衡以实现最小估计速率。
translated by 谷歌翻译
概率分布之间的差异措施,通常被称为统计距离,在概率理论,统计和机器学习中普遍存在。为了在估计这些距离的距离时,对维度的诅咒,最近的工作已经提出了通过带有高斯内核的卷积在测量的分布中平滑局部不规则性。通过该框架的可扩展性至高维度,我们研究了高斯平滑$ P $ -wassersein距离$ \ mathsf {w} _p ^ {(\ sigma)} $的结构和统计行为,用于任意$ p \ GEQ 1 $。在建立$ \ mathsf {w} _p ^ {(\ sigma)} $的基本度量和拓扑属性之后,我们探索$ \ mathsf {w} _p ^ {(\ sigma)}(\ hat {\ mu} _n,\ mu)$,其中$ \ hat {\ mu} _n $是$ n $独立观察的实证分布$ \ mu $。我们证明$ \ mathsf {w} _p ^ {(\ sigma)} $享受$ n ^ { - 1/2} $的参数经验融合速率,这对比$ n ^ { - 1 / d} $率对于未平滑的$ \ mathsf {w} _p $ why $ d \ geq 3 $。我们的证明依赖于控制$ \ mathsf {w} _p ^ {(\ sigma)} $ by $ p $ th-sting spoollow sobolev restion $ \ mathsf {d} _p ^ {(\ sigma)} $并导出限制$ \ sqrt {n} \,\ mathsf {d} _p ^ {(\ sigma)}(\ hat {\ mu} _n,\ mu)$,适用于所有尺寸$ d $。作为应用程序,我们提供了使用$ \ mathsf {w} _p ^ {(\ sigma)} $的两个样本测试和最小距离估计的渐近保证,使用$ p = 2 $的实验使用$ \ mathsf {d} _2 ^ {(\ sigma)} $。
translated by 谷歌翻译
We consider the problem of estimating the optimal transport map between a (fixed) source distribution $P$ and an unknown target distribution $Q$, based on samples from $Q$. The estimation of such optimal transport maps has become increasingly relevant in modern statistical applications, such as generative modeling. At present, estimation rates are only known in a few settings (e.g. when $P$ and $Q$ have densities bounded above and below and when the transport map lies in a H\"older class), which are often not reflected in practice. We present a unified methodology for obtaining rates of estimation of optimal transport maps in general function spaces. Our assumptions are significantly weaker than those appearing in the literature: we require only that the source measure $P$ satisfies a Poincar\'e inequality and that the optimal map be the gradient of a smooth convex function that lies in a space whose metric entropy can be controlled. As a special case, we recover known estimation rates for bounded densities and H\"older transport maps, but also obtain nearly sharp results in many settings not covered by prior work. For example, we provide the first statistical rates of estimation when $P$ is the normal distribution and the transport map is given by an infinite-width shallow neural network.
translated by 谷歌翻译
近年来,生成的对抗性网络(GANS)已经证明了令人印象深刻的实验结果,同时只有一些作品促进了统计学习理论。在这项工作中,我们提出了一种用于生成对抗性学习的无限尺寸理论框架。假设统一界限的$ k $-times $ \ alpha $ -h \“较旧的可分辨率和统一的正密度,我们表明Rosenblatt的转换引起了最佳发电机,可在$ \ alpha $的假设空间中可实现H \“较旧的微分发电机。通过一致的鉴别者假设空间的定义,我们进一步表明,在我们的框架中,由发电机引起的分布与来自对手学习过程的分布之间的jensen-shannon发散,并且数据生成分布会聚到零。在足够严格的规律性假设下对数据产生过程密度的假设,我们还基于浓度和链接提供会聚率。
translated by 谷歌翻译
变性推理(VI)为基于传统的采样方法提供了一种吸引人的替代方法,用于实施贝叶斯推断,因为其概念性的简单性,统计准确性和计算可扩展性。然而,常见的变分近似方案(例如平均场(MF)近似)需要某些共轭结构以促进有效的计算,这可能会增加不必要的限制对可行的先验分布家族,并对变异近似族对差异进行进一步的限制。在这项工作中,我们开发了一个通用计算框架,用于实施MF-VI VIA WASSERSTEIN梯度流(WGF),这是概率度量空间上的梯度流。当专门针对贝叶斯潜在变量模型时,我们将分析基于时间消化的WGF交替最小化方案的算法收敛,用于实现MF近似。特别是,所提出的算法类似于EM算法的分布版本,包括更新潜在变量变异分布的E step以及在参数的变异分布上进行最陡峭下降的m step。我们的理论分析依赖于概率度量空间中的最佳运输理论和细分微积分。我们证明了时间限制的WGF的指数收敛性,以最大程度地减少普通大地测量学严格的凸度的通用物镜功能。我们还提供了通过使用时间限制的WGF的固定点方程从MF近似获得的变异分布的指数收缩的新证明。我们将方法和理论应用于两个经典的贝叶斯潜在变量模型,即高斯混合模型和回归模型的混合物。还进行了数值实验,以补充这两个模型下的理论发现。
translated by 谷歌翻译
鉴于$ n $ i.i.d.从未知的分发$ P $绘制的样本,何时可以生成更大的$ n + m $ samples,这些标题不能与$ n + m $ i.i.d区别区别。从$ p $绘制的样品?(AXELROD等人2019)将该问题正式化为样本放大问题,并为离散分布和高斯位置模型提供了最佳放大程序。然而,这些程序和相关的下限定制到特定分布类,对样本扩增的一般统计理解仍然很大程度上。在这项工作中,我们通过推出通常适用的放大程序,下限技术和与现有统计概念的联系来放置对公司统计基础的样本放大问题。我们的技术适用于一大类分布,包括指数家庭,并在样本放大和分配学习之间建立严格的联系。
translated by 谷歌翻译
Mixtures of regression are a powerful class of models for regression learning with respect to a highly uncertain and heterogeneous response variable of interest. In addition to being a rich predictive model for the response given some covariates, the parameters in this model class provide useful information about the heterogeneity in the data population, which is represented by the conditional distributions for the response given the covariates associated with a number of distinct but latent subpopulations. In this paper, we investigate conditions of strong identifiability, rates of convergence for conditional density and parameter estimation, and the Bayesian posterior contraction behavior arising in finite mixture of regression models, under exact-fitted and over-fitted settings and when the number of components is unknown. This theory is applicable to common choices of link functions and families of conditional distributions employed by practitioners. We provide simulation studies and data illustrations, which shed some light on the parameter learning behavior found in several popular regression mixture models reported in the literature.
translated by 谷歌翻译
生成对抗网络(GAN)在数据生成方面取得了巨大成功。但是,其统计特性尚未完全理解。在本文中,我们考虑了GAN的一般$ f $ divergence公式的统计行为,其中包括Kullback- Leibler Divergence与最大似然原理密切相关。我们表明,对于正确指定的参数生成模型,在适当的规律性条件下,所有具有相同歧视类别类别的$ f $ divergence gans均在渐近上等效。 Moreover, with an appropriately chosen local discriminator, they become equivalent to the maximum likelihood estimate asymptotically.对于被误解的生成模型,具有不同$ f $ -Divergences {收敛到不同估计器}的gan,因此无法直接比较。但是,结果表明,对于某些常用的$ f $ -Diverences,原始的$ f $ gan并不是最佳的,因为当更换原始$ f $ gan配方中的判别器培训时,可以实现较小的渐近方差通过逻辑回归。结果估计方法称为对抗梯度估计(年龄)。提供了实证研究来支持该理论,并证明了年龄的优势,而不是模型错误的原始$ f $ gans。
translated by 谷歌翻译
We consider the problem of estimating a multivariate function $f_0$ of bounded variation (BV), from noisy observations $y_i = f_0(x_i) + z_i$ made at random design points $x_i \in \mathbb{R}^d$, $i=1,\ldots,n$. We study an estimator that forms the Voronoi diagram of the design points, and then solves an optimization problem that regularizes according to a certain discrete notion of total variation (TV): the sum of weighted absolute differences of parameters $\theta_i,\theta_j$ (which estimate the function values $f_0(x_i),f_0(x_j)$) at all neighboring cells $i,j$ in the Voronoi diagram. This is seen to be equivalent to a variational optimization problem that regularizes according to the usual continuum (measure-theoretic) notion of TV, once we restrict the domain to functions that are piecewise constant over the Voronoi diagram. The regression estimator under consideration hence performs (shrunken) local averaging over adaptively formed unions of Voronoi cells, and we refer to it as the Voronoigram, following the ideas in Koenker (2005), and drawing inspiration from Tukey's regressogram (Tukey, 1961). Our contributions in this paper span both the conceptual and theoretical frontiers: we discuss some of the unique properties of the Voronoigram in comparison to TV-regularized estimators that use other graph-based discretizations; we derive the asymptotic limit of the Voronoi TV functional; and we prove that the Voronoigram is minimax rate optimal (up to log factors) for estimating BV functions that are essentially bounded.
translated by 谷歌翻译
The modeling of probability distributions, specifically generative modeling and density estimation, has become an immensely popular subject in recent years by virtue of its outstanding performance on sophisticated data such as images and texts. Nevertheless, a theoretical understanding of its success is still incomplete. One mystery is the paradox between memorization and generalization: In theory, the model is trained to be exactly the same as the empirical distribution of the finite samples, whereas in practice, the trained model can generate new samples or estimate the likelihood of unseen samples. Likewise, the overwhelming diversity of distribution learning models calls for a unified perspective on this subject. This paper provides a mathematical framework such that all the well-known models can be derived based on simple principles. To demonstrate its efficacy, we present a survey of our results on the approximation error, training error and generalization error of these models, which can all be established based on this framework. In particular, the aforementioned paradox is resolved by proving that these models enjoy implicit regularization during training, so that the generalization error at early-stopping avoids the curse of dimensionality. Furthermore, we provide some new results on landscape analysis and the mode collapse phenomenon.
translated by 谷歌翻译
给定$ n $数据点$ \ mathbb {r}^d $中的云,请考虑$ \ mathbb {r}^d $的$ m $ dimensional子空间预计点。当$ n,d $增长时,这一概率分布的集合如何?我们在零模型下考虑了这个问题。标准高斯矢量,重点是渐近方案,其中$ n,d \ to \ infty $,$ n/d \ to \ alpha \ in(0,\ infty)$,而$ m $是固定的。用$ \ mathscr {f} _ {m,\ alpha} $表示$ \ mathbb {r}^m $中的一组概率分布,在此限制中以低维度为单位,我们在此限制中建立了新的内部和外部界限$ \ mathscr {f} _ {m,\ alpha} $。特别是,我们将$ \ mathscr {f} _ {m,\ alpha} $的Wasserstein Radius表征为对数因素,并以$ M = 1 $确切确定它。我们还通过kullback-leibler差异和r \'{e} NYI信息维度证明了尖锐的界限。上一个问题已应用于无监督的学习方法,例如投影追求和独立的组件分析。我们介绍了与监督学习相关的相同问题的版本,并证明了尖锐的沃斯坦斯坦半径绑定。作为一个应用程序,我们在具有$ M $隐藏神经元的两层神经网络的插值阈值上建立了上限。
translated by 谷歌翻译
We develop and analyze M -estimation methods for divergence functionals and the likelihood ratios of two probability distributions. Our method is based on a non-asymptotic variational characterization of f -divergences, which allows the problem of estimating divergences to be tackled via convex empirical risk optimization. The resulting estimators are simple to implement, requiring only the solution of standard convex programs. We present an analysis of consistency and convergence for these estimators. Given conditions only on the ratios of densities, we show that our estimators can achieve optimal minimax rates for the likelihood ratio and the divergence functionals in certain regimes. We derive an efficient optimization algorithm for computing our estimates, and illustrate their convergence behavior and practical viability by simulations. 1
translated by 谷歌翻译
我们提出了一种统一的技术,用于顺序估计分布之间的凸面分歧,包括内核最大差异等积分概率度量,$ \ varphi $ - 像Kullback-Leibler发散,以及最佳运输成本,例如Wassersein距离的权力。这是通过观察到经验凸起分歧(部分有序)反向半角分离的实现来实现的,而可交换过滤耦合,其具有这些方法的最大不等式。这些技术似乎是对置信度序列和凸分流的现有文献的互补和强大的补充。我们构建一个离线到顺序设备,将各种现有的离线浓度不等式转换为可以连续监测的时间均匀置信序列,在任意停止时间提供有效的测试或置信区间。得到的顺序边界仅在相应的固定时间范围内支付迭代对数价格,保留对问题参数的相同依赖性(如适用的尺寸或字母大小)。这些结果也适用于更一般的凸起功能,如负差分熵,实证过程的高度和V型统计。
translated by 谷歌翻译
我们调查了一定类别的功能不等式,称为弱Poincar的不等式,以使Markov链的收敛性与均衡相结合。我们表明,这使得SubGoom测量收敛界的直接和透明的推导出用于独立的Metropolis - Hastings采样器和用于棘手似然性的伪边缘方法,后者在许多实际设置中是子表芯。这些结果依赖于马尔可夫链之间的新量化比较定理。相关证据比依赖于漂移/较小化条件的证据更简单,并且所开发的工具允许我们恢复并进一步延长特定情况的已知结果。我们能够为伪边缘算法的实际使用提供新的见解,分析平均近似贝叶斯计算(ABC)的效果以及独立平均值的产品,以及研究与之相关的逻辑重量的情况粒子边缘大都市 - 黑斯廷斯(PMMH)。
translated by 谷歌翻译
We consider the constrained sampling problem where the goal is to sample from a distribution $\pi(x)\propto e^{-f(x)}$ and $x$ is constrained on a convex body $\mathcal{C}\subset \mathbb{R}^d$. Motivated by penalty methods from optimization, we propose penalized Langevin Dynamics (PLD) and penalized Hamiltonian Monte Carlo (PHMC) that convert the constrained sampling problem into an unconstrained one by introducing a penalty function for constraint violations. When $f$ is smooth and the gradient is available, we show $\tilde{\mathcal{O}}(d/\varepsilon^{10})$ iteration complexity for PLD to sample the target up to an $\varepsilon$-error where the error is measured in terms of the total variation distance and $\tilde{\mathcal{O}}(\cdot)$ hides some logarithmic factors. For PHMC, we improve this result to $\tilde{\mathcal{O}}(\sqrt{d}/\varepsilon^{7})$ when the Hessian of $f$ is Lipschitz and the boundary of $\mathcal{C}$ is sufficiently smooth. To our knowledge, these are the first convergence rate results for Hamiltonian Monte Carlo methods in the constrained sampling setting that can handle non-convex $f$ and can provide guarantees with the best dimension dependency among existing methods with deterministic gradients. We then consider the setting where unbiased stochastic gradients are available. We propose PSGLD and PSGHMC that can handle stochastic gradients without Metropolis-Hasting correction steps. When $f$ is strongly convex and smooth, we obtain an iteration complexity of $\tilde{\mathcal{O}}(d/\varepsilon^{18})$ and $\tilde{\mathcal{O}}(d\sqrt{d}/\varepsilon^{39})$ respectively in the 2-Wasserstein distance. For the more general case, when $f$ is smooth and non-convex, we also provide finite-time performance bounds and iteration complexity results. Finally, we test our algorithms on Bayesian LASSO regression and Bayesian constrained deep learning problems.
translated by 谷歌翻译