我们考虑对二进制数据的独立分量分析。虽然实践中的基本情况，但这种情况比ICA持续不断开发，以便连续数据。我们首先假设连续值潜在空间中的线性混合模型，然后是二进制观察模型。重要的是，我们认为这些来源是非静止的;这是必要的，因为任何非高斯基本上都是由二值化摧毁的。有趣的是，该模型通过采用多元高斯分布的累积分布函数来允许闭合形式的似然。在与持续值为案例的鲜明对比中，我们证明了少数观察变量的模型的非可识别性;当观察变量的数量较高时，我们的经验结果意味着可识别性。我们为二进制ICA展示了仅使用成对边缘的二进制ICA的实用方法，这些方法比完全多变量可能性更快地计算。

The framework of variational autoencoders allows us to efficiently learn deep latent-variable models, such that the model's marginal distribution over observed variables fits the data. Often, we're interested in going a step further, and want to approximate the true joint distribution over observed and latent variables, including the true prior and posterior distributions over latent variables. This is known to be generally impossible due to unidentifiability of the model. We address this issue by showing that for a broad family of deep latentvariable models, identification of the true joint distribution over observed and latent variables is actually possible up to very simple transformations, thus achieving a principled and powerful form of disentanglement. Our result requires a factorized prior distribution over the latent variables that is conditioned on an additionally observed variable, such as a class label or almost any other observation. We build on recent developments in nonlinear ICA, which we extend to the case with noisy or undercomplete observations, integrated in a maximum likelihood framework. The result also trivially contains identifiable flow-based generative models as a special case.

A fundamental problem in neural network research, as well as in many other disciplines, is finding a suitable representation of multivariate data, i.e. random vectors. For reasons of computational and conceptual simplicity, the representation is often sought as a linear transformation of the original data. In other words, each component of the representation is a linear combination of the original variables. Well-known linear transformation methods include principal component analysis, factor analysis, and projection pursuit. Independent component analysis (ICA) is a recently developed method in which the goal is to find a linear representation of nongaussian data so that the components are statistically independent, or as independent as possible. Such a representation seems to capture the essential structure of the data in many applications, including feature extraction and signal separation. In this paper, we present the basic theory and applications of ICA, and our recent work on the subject.

这项正在进行的工作旨在为统计学习提供统一的介绍，从诸如GMM和HMM等经典模型到现代神经网络（如VAE和扩散模型）缓慢地构建。如今，有许多互联网资源可以孤立地解释这一点或新的机器学习算法，但是它们并没有（也不能在如此简短的空间中）将这些算法彼此连接起来，或者与统计模型的经典文献相连现代算法出现了。同样明显缺乏的是一个单一的符号系统，尽管对那些已经熟悉材料的人（如这些帖子的作者）不满意，但对新手的入境造成了重大障碍。同样，我的目的是将各种模型（尽可能）吸收到一个用于推理和学习的框架上，表明（以及为什么）如何以最小的变化将一个模型更改为另一个模型（其中一些是新颖的，另一些是文献中的）。某些背景当然是必要的。我以为读者熟悉基本的多变量计算，概率和统计以及线性代数。这本书的目标当然不是​​完整性，而是从基本知识到过去十年中极强大的新模型的直线路径或多或少。然后，目标是补充而不是替换，诸如Bishop的\ emph {模式识别和机器学习}之类的综合文本，该文本现在已经15岁了。

In recent years, several methods have been proposed for the discovery of causal structure from non-experimental data. Such methods make various assumptions on the data generating process to facilitate its identification from purely observational data. Continuing this line of research, we show how to discover the complete causal structure of continuous-valued data, under the assumptions that (a) the data generating process is linear, (b) there are no unobserved confounders, and (c) disturbance variables have non-Gaussian distributions of non-zero variances. The solution relies on the use of the statistical method known as independent component analysis, and does not require any pre-specified time-ordering of the variables. We provide a complete Matlab package for performing this LiNGAM analysis (short for Linear Non-Gaussian Acyclic Model), and demonstrate the effectiveness of the method using artificially generated data and real-world data.

这项工作介绍了一种新颖的原则，我们通过机制稀疏正规调用解剖学，基于高级概念的动态往往稀疏的想法。我们提出了一种表示学习方法，可以通过同时学习与它们相关的潜在因子和稀疏因果图形模型来引起解剖学。我们开发了一个严谨的可识别性理论，建立在最近的非线性独立分量分析（ICA）结果中，结果是模拟这一原理，并展示了如何恢复潜在变量，如果一个规则大致潜在机制为稀疏，如果某些图形连接标准通过数据生成过程满足。作为我们框架的特殊情况，我们展示了如何利用未知目标的干预措施来解除潜在因子，从而借鉴ICA和因果关系之间的进一步联系。我们还提出了一种基于VAE的方法，其中通过二进制掩码来学习和正规化潜在机制，并通过表明它学会在模拟中的解散表示来验证我们的理论。

One of the core problems of modern statistics is to approximate difficult-to-compute probability densities. This problem is especially important in Bayesian statistics, which frames all inference about unknown quantities as a calculation involving the posterior density. In this paper, we review variational inference (VI), a method from machine learning that approximates probability densities through optimization. VI has been used in many applications and tends to be faster than classical methods, such as Markov chain Monte Carlo sampling. The idea behind VI is to first posit a family of densities and then to find the member of that family which is close to the target. Closeness is measured by Kullback-Leibler divergence. We review the ideas behind mean-field variational inference, discuss the special case of VI applied to exponential family models, present a full example with a Bayesian mixture of Gaussians, and derive a variant that uses stochastic optimization to scale up to massive data. We discuss modern research in VI and highlight important open problems. VI is powerful, but it is not yet well understood. Our hope in writing this paper is to catalyze statistical research on this class of algorithms.

One often wants to estimate statistical models where the probability density function is known only up to a multiplicative normalization constant. Typically, one then has to resort to Markov Chain Monte Carlo methods, or approximations of the normalization constant. Here, we propose that such models can be estimated by minimizing the expected squared distance between the gradient of the log-density given by the model and the gradient of the log-density of the observed data. While the estimation of the gradient of log-density function is, in principle, a very difficult non-parametric problem, we prove a surprising result that gives a simple formula for this objective function. The density function of the observed data does not appear in this formula, which simplifies to a sample average of a sum of some derivatives of the log-density given by the model. The validity of the method is demonstrated on multivariate Gaussian and independent component analysis models, and by estimating an overcomplete filter set for natural image data.

Linear structural causal models (SCMs)-- in which each observed variable is generated by a subset of the other observed variables as well as a subset of the exogenous sources-- are pervasive in causal inference and casual discovery. However, for the task of causal discovery, existing work almost exclusively focus on the submodel where each observed variable is associated with a distinct source with non-zero variance. This results in the restriction that no observed variable can deterministically depend on other observed variables or latent confounders. In this paper, we extend the results on structure learning by focusing on a subclass of linear SCMs which do not have this property, i.e., models in which observed variables can be causally affected by any subset of the sources, and are allowed to be a deterministic function of other observed variables or latent confounders. This allows for a more realistic modeling of influence or information propagation in systems. We focus on the task of causal discovery form observational data generated from a member of this subclass. We derive a set of necessary and sufficient conditions for unique identifiability of the causal structure. To the best of our knowledge, this is the first work that gives identifiability results for causal discovery under both latent confounding and deterministic relationships. Further, we propose an algorithm for recovering the underlying causal structure when the aforementioned conditions are satisfied. We validate our theoretical results both on synthetic and real datasets.

For distributions $\mathbb{P}$ and $\mathbb{Q}$ with different supports or undefined densities, the divergence $\textrm{D}(\mathbb{P}||\mathbb{Q})$ may not exist. We define a Spread Divergence $\tilde{\textrm{D}}(\mathbb{P}||\mathbb{Q})$ on modified $\mathbb{P}$ and $\mathbb{Q}$ and describe sufficient conditions for the existence of such a divergence. We demonstrate how to maximize the discriminatory power of a given divergence by parameterizing and learning the spread. We also give examples of using a Spread Divergence to train implicit generative models, including linear models (Independent Components Analysis) and non-linear models (Deep Generative Networks).

这项调查旨在提供线性模型及其背后的理论的介绍。我们的目标是对读者进行严格的介绍，并事先接触普通最小二乘。在机器学习中，输出通常是输入的非线性函数。深度学习甚至旨在找到需要大量计算的许多层的非线性依赖性。但是，这些算法中的大多数都基于简单的线性模型。然后，我们从不同视图中描述线性模型，并找到模型背后的属性和理论。线性模型是回归问题中的主要技术，其主要工具是最小平方近似，可最大程度地减少平方误差之和。当我们有兴趣找到回归函数时，这是一个自然的选择，该回归函数可以最大程度地减少相应的预期平方误差。这项调查主要是目的的摘要，即线性模型背后的重要理论的重要性，例如分布理论，最小方差估计器。我们首先从三种不同的角度描述了普通的最小二乘，我们会以随机噪声和高斯噪声干扰模型。通过高斯噪声，该模型产生了可能性，因此我们引入了最大似然估计器。它还通过这种高斯干扰发展了一些分布理论。最小二乘的分布理论将帮助我们回答各种问题并引入相关应用。然后，我们证明最小二乘是均值误差的最佳无偏线性模型，最重要的是，它实际上接近了理论上的极限。我们最终以贝叶斯方法及以后的线性模型结束。

对比度学习是无监督表示学习的最新有前途的方法，其中通过从未标记的数据中求解伪分类问题来学习数据的特征表示。但是，了解哪些表示对比度学习产量并不直接。此外，对比度学习通常基于最大似然估计，这往往容易受到异常值污染的影响。为了促进对比度学习的理解，本文理论上首先显示了与共同信息（MI）最大化的联系。我们的结果表明，在某些条件下，密度比估计是必需的，足以使MI最大化。因此，在流行目标功能中完成的与密度比估计相关的对比学习可以解释为最大化MI。接下来，随着密度比，我们在非线性独立组件分析（ICA）中为潜在源组件建立了新的恢复条件。与现有工作相反，既定条件包括对数据维度的新见解，该洞察力显然得到了数值实验的支持。此外，受非线性ICA的启发，我们提出了一个新型框架，以估算低维度潜在源组件的非线性子空间，并以密度比建立了一些基本空间估计的理论条件。然后，我们通过异常抗体密度比估计提出了一种实用方法，可以看作是对MI，非线性ICA或非线性子空间估计的最大化。此外，还提出了样品有效的非线性ICA方法。我们从理论上研究了所提出的方法的异常体性。最后，在非线性ICA中并通过应用线性分类，在数值上证明了所提出方法的有用性。

这是机器学习中（主要是）笔和纸练习的集合。练习在以下主题上：线性代数，优化，定向图形模型，无向图形模型，图形模型的表达能力，因子图和消息传递，隐藏马尔可夫模型的推断，基于模型的学习（包括ICA和非正态模型），采样和蒙特卡洛整合以及变异推断。

结构方程模型（SEM）是一种有效的框架，其原因是通过定向非循环图（DAG）表示的因果关系。最近的进步使得能够从观察数据中实现了DAG的最大似然点估计。然而，在实际场景中，可以不能准确地捕获在推断下面的底层图中的不确定性，其中真正的DAG是不可识别的并且/或观察到的数据集是有限的。我们提出了贝叶斯因果发现网（BCD网），一个变分推理框架，用于估算表征线性高斯SEM的DAG的分布。由于图形的离散和组合性质，开发一个完整的贝叶斯后面是挑战。我们通过表达变分别家庭分析可扩展VI的可扩展VI的关键设计选择，例如1）表达性变分别家庭，2）连续弛豫，使低方差随机优化和3）在潜在变量上具有合适的前置。我们提供了一系列关于实际和合成数据的实验，显示BCD网在低数据制度中的标准因果发现度量上的最大似然方法，例如结构汉明距离。

我们开发了一个计算程序，以估计具有附加噪声的半摩托车高斯过程回归模型的协方差超参数。也就是说，提出的方法可用于有效估计相关误差的方差，以及基于最大化边际似然函数的噪声方差。我们的方法涉及适当地降低超参数空间的维度，以简化单变量的根发现问题的估计过程。此外，我们得出了边际似然函数及其衍生物的边界和渐近线，这对于缩小高参数搜索的初始范围很有用。使用数值示例，我们证明了与传统参数优化相比，提出方法的计算优势和鲁棒性。

We develop an optimization algorithm suitable for Bayesian learning in complex models. Our approach relies on natural gradient updates within a general black-box framework for efficient training with limited model-specific derivations. It applies within the class of exponential-family variational posterior distributions, for which we extensively discuss the Gaussian case for which the updates have a rather simple form. Our Quasi Black-box Variational Inference (QBVI) framework is readily applicable to a wide class of Bayesian inference problems and is of simple implementation as the updates of the variational posterior do not involve gradients with respect to the model parameters, nor the prescription of the Fisher information matrix. We develop QBVI under different hypotheses for the posterior covariance matrix, discuss details about its robust and feasible implementation, and provide a number of real-world applications to demonstrate its effectiveness.

我们证明了（a）具有通用近似功能的广泛的深层变量模型的可识别性，并且（b）是通常在实践中使用的变异自动编码器的解码器。与现有工作不同，我们的分析不需要弱监督，辅助信息或潜在空间中的条件。最近，研究了此类模型的可识别性。在这些作品中，主要的假设是，还可以观察到辅助变量$ u $（也称为侧面信息）。同时，几项作品从经验上观察到，这在实践中似乎并不是必需的。在这项工作中，我们通过证明具有通用近似功能的广泛生成（即无监督的）模型来解释这种行为，无需侧面信息$ u $：我们证明了整个生成模型的可识别性$ u $，仅观察数据$ x $。我们考虑的模型与实践中使用的自动编码器体系结构紧密连接，该体系结构利用了潜在空间中的混合先验和编码器中的Relu/Leaky-Relu激活。我们的主要结果是可识别性层次结构，该层次结构显着概括了先前的工作，并揭示了不同的假设如何导致可识别性的“优势”不同。例如，我们最薄弱的结果确定了（无监督的）可识别性，直到仿射转换已经改善了现有工作。众所周知，这些模型具有通用近似功能，而且它们已被广泛用于实践中来学习数据表示。

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact

贝叶斯神经网络具有潜在变量（BNN + LVS）通过明确建模模型不确定性（通过网络权重）和环境暂停（通过潜在输入噪声变量）来捕获预测的不确定性。在这项工作中，我们首先表明BNN + LV具有严重形式的非可识别性：可以在模型参数和潜在变量之间传输解释性，同时拟合数据。我们证明，在无限数据的极限中，网络权重和潜变量的后部模式从地面真理渐近地偏离。由于这种渐近偏差，传统的推理方法可以在实践中，产量参数概括不确定和不确定的不确定性。接下来，我们开发一种新推断过程，明确地减轻了训练期间不可识别性的影响，并产生高质量的预测以及不确定性估计。我们展示我们的推理方法在一系列合成和实际数据集中改善了基准方法。

我们介绍了一种从高维时间序列数据学习潜在随机微分方程（SDES）的方法。考虑到从较低维潜在未知IT \ ^ O过程产生的高维时间序列，所提出的方法通过自我监督的学习方法学习从环境到潜在空间的映射和潜在的SDE系数。使用变形AutiaceOders的框架，我们考虑基于SDE解决方案的Euler-Maruyama近似的数据的条件生成模型。此外，我们使用最近的结果对潜在变量模型的可识别性来表明，所提出的模型不仅可以恢复底层的SDE系数，还可以在无限数据的极限中恢复底层的SDE系数，也可以最大潜在潜在变量。我们通过多个模拟视频处理任务验证方法，其中底层SDE是已知的，并通过真实的世界数据集。