A fundamental problem in neural network research, as well as in many other disciplines, is finding a suitable representation of multivariate data, i.e. random vectors. For reasons of computational and conceptual simplicity, the representation is often sought as a linear transformation of the original data. In other words, each component of the representation is a linear combination of the original variables. Well-known linear transformation methods include principal component analysis, factor analysis, and projection pursuit. Independent component analysis (ICA) is a recently developed method in which the goal is to find a linear representation of nongaussian data so that the components are statistically independent, or as independent as possible. Such a representation seems to capture the essential structure of the data in many applications, including feature extraction and signal separation. In this paper, we present the basic theory and applications of ICA, and our recent work on the subject.
translated by 谷歌翻译
这项正在进行的工作旨在为统计学习提供统一的介绍,从诸如GMM和HMM等经典模型到现代神经网络(如VAE和扩散模型)缓慢地构建。如今,有许多互联网资源可以孤立地解释这一点或新的机器学习算法,但是它们并没有(也不能在如此简短的空间中)将这些算法彼此连接起来,或者与统计模型的经典文献相连现代算法出现了。同样明显缺乏的是一个单一的符号系统,尽管对那些已经熟悉材料的人(如这些帖子的作者)不满意,但对新手的入境造成了重大障碍。同样,我的目的是将各种模型(尽可能)吸收到一个用于推理和学习的框架上,表明(以及为什么)如何以最小的变化将一个模型更改为另一个模型(其中一些是新颖的,另一些是文献中的)。某些背景当然是必要的。我以为读者熟悉基本的多变量计算,概率和统计以及线性代数。这本书的目标当然不是​​完整性,而是从基本知识到过去十年中极强大的新模型的直线路径或多或少。然后,目标是补充而不是替换,诸如Bishop的\ emph {模式识别和机器学习}之类的综合文本,该文本现在已经15岁了。
translated by 谷歌翻译
One often wants to estimate statistical models where the probability density function is known only up to a multiplicative normalization constant. Typically, one then has to resort to Markov Chain Monte Carlo methods, or approximations of the normalization constant. Here, we propose that such models can be estimated by minimizing the expected squared distance between the gradient of the log-density given by the model and the gradient of the log-density of the observed data. While the estimation of the gradient of log-density function is, in principle, a very difficult non-parametric problem, we prove a surprising result that gives a simple formula for this objective function. The density function of the observed data does not appear in this formula, which simplifies to a sample average of a sum of some derivatives of the log-density given by the model. The validity of the method is demonstrated on multivariate Gaussian and independent component analysis models, and by estimating an overcomplete filter set for natural image data.
translated by 谷歌翻译
The framework of variational autoencoders allows us to efficiently learn deep latent-variable models, such that the model's marginal distribution over observed variables fits the data. Often, we're interested in going a step further, and want to approximate the true joint distribution over observed and latent variables, including the true prior and posterior distributions over latent variables. This is known to be generally impossible due to unidentifiability of the model. We address this issue by showing that for a broad family of deep latentvariable models, identification of the true joint distribution over observed and latent variables is actually possible up to very simple transformations, thus achieving a principled and powerful form of disentanglement. Our result requires a factorized prior distribution over the latent variables that is conditioned on an additionally observed variable, such as a class label or almost any other observation. We build on recent developments in nonlinear ICA, which we extend to the case with noisy or undercomplete observations, integrated in a maximum likelihood framework. The result also trivially contains identifiable flow-based generative models as a special case.
translated by 谷歌翻译
最近有一项激烈的活动在嵌入非常高维和非线性数据结构的嵌入中,其中大部分在数据科学和机器学习文献中。我们分四部分调查这项活动。在第一部分中,我们涵盖了非线性方法,例如主曲线,多维缩放,局部线性方法,ISOMAP,基于图形的方法和扩散映射,基于内核的方法和随机投影。第二部分与拓扑嵌入方法有关,特别是将拓扑特性映射到持久图和映射器算法中。具有巨大增长的另一种类型的数据集是非常高维网络数据。第三部分中考虑的任务是如何将此类数据嵌入中等维度的向量空间中,以使数据适合传统技术,例如群集和分类技术。可以说,这是算法机器学习方法与统计建模(所谓的随机块建模)之间的对比度。在论文中,我们讨论了两种方法的利弊。调查的最后一部分涉及嵌入$ \ mathbb {r}^ 2 $,即可视化中。提出了三种方法:基于第一部分,第二和第三部分中的方法,$ t $ -sne,UMAP和大节。在两个模拟数据集上进行了说明和比较。一个由嘈杂的ranunculoid曲线组成的三胞胎,另一个由随机块模型和两种类型的节点产生的复杂性的网络组成。
translated by 谷歌翻译
这项调查旨在提供线性模型及其背后的理论的介绍。我们的目标是对读者进行严格的介绍,并事先接触普通最小二乘。在机器学习中,输出通常是输入的非线性函数。深度学习甚至旨在找到需要大量计算的许多层的非线性依赖性。但是,这些算法中的大多数都基于简单的线性模型。然后,我们从不同视图中描述线性模型,并找到模型背后的属性和理论。线性模型是回归问题中的主要技术,其主要工具是最小平方近似,可最大程度地减少平方误差之和。当我们有兴趣找到回归函数时,这是一个自然的选择,该回归函数可以最大程度地减少相应的预期平方误差。这项调查主要是目的的摘要,即线性模型背后的重要理论的重要性,例如分布理论,最小方差估计器。我们首先从三种不同的角度描述了普通的最小二乘,我们会以随机噪声和高斯噪声干扰模型。通过高斯噪声,该模型产生了可能性,因此我们引入了最大似然估计器。它还通过这种高斯干扰发展了一些分布理论。最小二乘的分布理论将帮助我们回答各种问题并引入相关应用。然后,我们证明最小二乘是均值误差的最佳无偏线性模型,最重要的是,它实际上接近了理论上的极限。我们最终以贝叶斯方法及以后的线性模型结束。
translated by 谷歌翻译
The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, auto-encoders, manifold learning, and deep networks. This motivates longer-term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning.
translated by 谷歌翻译
我们考虑对二进制数据的独立分量分析。虽然实践中的基本情况,但这种情况比ICA持续不断开发,以便连续数据。我们首先假设连续值潜在空间中的线性混合模型,然后是二进制观察模型。重要的是,我们认为这些来源是非静止的;这是必要的,因为任何非高斯基本上都是由二值化摧毁的。有趣的是,该模型通过采用多元高斯分布的累积分布函数来允许闭合形式的似然。在与持续值为案例的鲜明对比中,我们证明了少数观察变量的模型的非可识别性;当观察变量的数量较高时,我们的经验结果意味着可识别性。我们为二进制ICA展示了仅使用成对边缘的二进制ICA的实用方法,这些方法比完全多变量可能性更快地计算。
translated by 谷歌翻译
这是一门专门针对STEM学生开发的介绍性机器学习课程。我们的目标是为有兴趣的读者提供基础知识,以在自己的项目中使用机器学习,并将自己熟悉术语作为进一步阅读相关文献的基础。在这些讲义中,我们讨论受监督,无监督和强化学习。注释从没有神经网络的机器学习方法的说明开始,例如原理分析,T-SNE,聚类以及线性回归和线性分类器。我们继续介绍基本和先进的神经网络结构,例如密集的进料和常规神经网络,经常性的神经网络,受限的玻尔兹曼机器,(变性)自动编码器,生成的对抗性网络。讨论了潜在空间表示的解释性问题,并使用梦和对抗性攻击的例子。最后一部分致力于加强学习,我们在其中介绍了价值功能和政策学习的基本概念。
translated by 谷歌翻译
This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models-including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation-which exploits a certain tensor structure in their low-order observable moments (typically, of second-and third-order). Specifically, parameter estimation is reduced to the problem of extracting a certain (orthogonal) decomposition of a symmetric tensor derived from the moments; this decomposition can be viewed as a natural generalization of the singular value decomposition for matrices. Although tensor decompositions are generally intractable to compute, the decomposition of these specially structured tensors can be efficiently obtained by a variety of approaches, including power iterations and maximization approaches (similar to the case of matrices). A detailed analysis of a robust tensor power method is provided, establishing an analogue of Wedin's perturbation theorem for the singular vectors of matrices. This implies a robust and computationally tractable estimation approach for several popular latent variable models.
translated by 谷歌翻译
这是机器学习中(主要是)笔和纸练习的集合。练习在以下主题上:线性代数,优化,定向图形模型,无向图形模型,图形模型的表达能力,因子图和消息传递,隐藏马尔可夫模型的推断,基于模型的学习(包括ICA和非正态模型),采样和蒙特卡洛整合以及变异推断。
translated by 谷歌翻译
我们介绍了一种确定全局特征解耦的方法,并显示其适用于提高数据分析性能的适用性,并开放了新的场所以进行功能传输。我们提出了一种新的形式主义,该形式主义是基于沿特征梯度遵循轨迹来定义对子曼群的转换的。通过这些转换,我们定义了一个归一化,我们证明,它允许解耦可区分的特征。通过将其应用于采样矩,我们获得了用于正骨的准分析溶液,正尾肌肉是峰度的归一化版本,不仅与平均值和方差相关,而且还与偏度相关。我们将此方法应用于原始数据域和过滤器库的输出中,以基于全局描述符的回归和分类问题,与使用经典(未删除)描述符相比,性能得到一致且显着的改进。
translated by 谷歌翻译
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact
translated by 谷歌翻译
大量的数据和创新算法使数据驱动的建模成为现代行业的流行技术。在各种数据驱动方法中,潜在变量模型(LVM)及其对应物占主要份额,并在许多工业建模领域中起着至关重要的作用。 LVM通常可以分为基于统计学习的经典LVM和基于神经网络的深层LVM(DLVM)。我们首先讨论经典LVM的定义,理论和应用,该定义和应用既是综合教程,又是对经典LVM的简短申请调查。然后,我们对当前主流DLVM进行了彻底的介绍,重点是其理论和模型体系结构,此后不久就提供了有关DLVM的工业应用的详细调查。上述两种类型的LVM具有明显的优势和缺点。具体而言,经典的LVM具有简洁的原理和良好的解释性,但是它们的模型能力无法解决复杂的任务。基于神经网络的DLVM具有足够的模型能力,可以在复杂的场景中实现令人满意的性能,但它以模型的解释性和效率为例。旨在结合美德并减轻这两种类型的LVM的缺点,并探索非神经网络的举止以建立深层模型,我们提出了一个新颖的概念,称为“轻量级Deep LVM(LDLVM)”。在提出了这个新想法之后,该文章首先阐述了LDLVM的动机和内涵,然后提供了两个新颖的LDLVM,并详尽地描述了其原理,建筑和优点。最后,讨论了前景和机会,包括重要的开放问题和可能的研究方向。
translated by 谷歌翻译
从人体中获取广泛的体征,称为生物医学体征或生物信号,它们可以处于细胞水平,器官水平或亚原子水平。脑电图来自大脑的电活动,心电图是心脏的电活动,来自肌肉声信号的电动作用,称为肌电图,眼睛的电视图等。研究这些信号对医生非常有帮助,它可以帮助他们检查和预测和治愈许多疾病。
translated by 谷歌翻译
Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows through the lens of probabilistic modeling and inference. We place special emphasis on the fundamental principles of flow design, and discuss foundational topics such as expressive power and computational trade-offs. We also broaden the conceptual framing of flows by relating them to more general probability transformations. Lastly, we summarize the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.
translated by 谷歌翻译
本论文主要涉及解决深层(时间)高斯过程(DGP)回归问题的状态空间方法。更具体地,我们代表DGP作为分层组合的随机微分方程(SDES),并且我们通过使用状态空间过滤和平滑方法来解决DGP回归问题。由此产生的状态空间DGP(SS-DGP)模型生成丰富的电视等级,与建模许多不规则信号/功能兼容。此外,由于他们的马尔可道结构,通过使用贝叶斯滤波和平滑方法可以有效地解决SS-DGPS回归问题。本论文的第二次贡献是我们通过使用泰勒力矩膨胀(TME)方法来解决连续离散高斯滤波和平滑问题。这诱导了一类滤波器和SmooThers,其可以渐近地精确地预测随机微分方程(SDES)解决方案的平均值和协方差。此外,TME方法和TME过滤器和SmoOthers兼容模拟SS-DGP并解决其回归问题。最后,本文具有多种状态 - 空间(深)GPS的应用。这些应用主要包括(i)来自部分观察到的轨迹的SDES的未知漂移功能和信号的光谱 - 时间特征估计。
translated by 谷歌翻译
In recent years, several methods have been proposed for the discovery of causal structure from non-experimental data. Such methods make various assumptions on the data generating process to facilitate its identification from purely observational data. Continuing this line of research, we show how to discover the complete causal structure of continuous-valued data, under the assumptions that (a) the data generating process is linear, (b) there are no unobserved confounders, and (c) disturbance variables have non-Gaussian distributions of non-zero variances. The solution relies on the use of the statistical method known as independent component analysis, and does not require any pre-specified time-ordering of the variables. We provide a complete Matlab package for performing this LiNGAM analysis (short for Linear Non-Gaussian Acyclic Model), and demonstrate the effectiveness of the method using artificially generated data and real-world data.
translated by 谷歌翻译
胎儿心电图(FECG)首先在20世纪初从母体腹表面记录。在过去的五十年中,最先进的电子技术和信号处理算法已被用于将非侵入性胎儿心电图转化为可靠的胎儿心脏监测技术。在本章中,已经对来自非侵入性母亲腹部录像进行了建模,提取和分析的主要信号处理技术,并详细介绍了来自非侵入性母亲腹部录像的型号的建模,提取和分析。本章的主要主题包括:1)FECG的电生理学从信号处理视点,2)母体体积传导介质的数学模型和从体表的FECG的波形模型,3)信号采集要求,4)基于模型的FECG噪声和干扰取消的技术,包括自适应滤波器和半盲源分离技术,以及5)胎儿运动跟踪和在线FECG提取的最近算法的进步。
translated by 谷歌翻译