In ordinary independent component analysis, the components are assumed to be completely independent, and they do not necessarily have any meaningful order relationships. In practice, however, the estimated "independent" components are often not at all independent. We propose that this residual dependence structure could be used to define a topographic order for the components. In particular, a distance between two components could be defined using their higher-order correlations, and this distance could be used to create a topographic representation. Thus we obtain a linear decomposition into approximately independent components, where the dependence of two components is approximated by the proximity of the components in the topographic representation.
translated by 谷歌翻译
Blind signal separation (BSS) and independent component analysis (ICA) are emerging techniques of array processing and data analysis that aim to recover unobserved signals or "sources" from observed mixtures (typically, the output of an array of sensors), exploiting only the assumption of mutual independence between the signals. The weakness of the assumptions makes it a powerful approach, but it requires us to venture beyond familiar second-order statistics. The objectives of this paper are to review some of the approaches that have been recently developed to address this exciting problem, to illustrate how they stem from basic principles, and to show how they relate to each other.
translated by 谷歌翻译
Independent component analysis (ICA) is a recently developed, useful extension of standard principal component analysis (PCA). The ICA model is utilized mainly in blind separation of unknown source signals from their linear mixtures. In this application only the source signals which correspond to the coefficients of the ICA expansion are of interest. In this paper, we propose neural structures related to multilayer feedforward networks for performing complete ICA. The basic ICA network consists of whitening, separation, and basis vector estimation layers. It can be used for both blind source separation and estimation of the basis vectors of ICA. We consider learning algorithms for each layer, and modify our previous nonlinear PCA type algorithms so that their separation capabilities are greatly improved. The proposed class of networks yields good results in test examples with both artificial and real-world data. Index Terms-Blind source separation, independent component analysis, neural networks, principal component analysis, signal processing, unsupervised learning.
translated by 谷歌翻译
Recently, different models of the statistical structure of natural images have been proposed. These models predict properties of biological visual systems and can be used as priors in Bayesian inference. The fundamental model is independent component analysis, which can be estimated by maximization of the sparsenesses of linear filter outputs. This leads to the emergence of principal simple cell properties. Alternatively, simple cell properties are obtained by maximizing the temporal coherence in natural image sequences. Taking account of the basic dependencies of linear filter outputs permit modeling of complex cells and topographic organization as well. We propose a unifying framework for these statistical properties, based on the concept of spatiotemporal activity ''bubbles.'' A bubble means here an activation of simple cells (linear filters) that is contiguous both in space (the cortical surface) and in time.
translated by 谷歌翻译
A unified view of the area of sparse signal processing is presented in tutorial form by bringing together various fields in which the property of sparsity has been successfully exploited. For each of these fields, various algorithms and techniques, which have been developed to leverage sparsity, are described succinctly. The common potential benefits of significant reduction in sampling rate and processing manipulations through sparse signal processing are revealed. The key application domains of sparse signal processing are sampling, coding, spectral estimation, array processing, component analysis, and multipath channel estimation. In terms of the sampling process and reconstruction algorithms, linkages are made with random sampling, compressed sensing, and rate of innovation. The redundancy introduced by channel coding in finite and real Galois fields is then related to over-sampling with similar reconstruction algorithms. The error locator polynomial (ELP) and iterative methods are shown to work quite effectively for both sampling and coding applications. The methods of Prony, Pisarenko, and MUltiple SIgnal Classification (MUSIC) are next shown to be targeted at analyzing signals with sparse frequency domain representations. Specifically, the relations of the approach of Prony to an annihilating filter in rate of innovation and ELP in coding are emphasized; the Pisarenko and MUSIC methods are further improvements of the Prony method under noisy environments. The iterative methods developed for sampling and coding applications are shown to be powerful tools in spectral estimation. Such narrowband spectral estimation is then related to multi-source location and direction of arrival estimation in array processing. Sparsity in unobservable source signals is also shown to facilitate source separation in sparse component analysis; the algorithms developed in this area such as linear programming and matching pursuit are also widely used in compressed sensing. Finally, the multipath channel estimation problem is shown to have a sparse formulation; algorithms similar to sampling and coding are used to estimate typical multicarrier communication channels.
translated by 谷歌翻译
This article considers high-order measures of independence for the independent component analysis problem and discusses the class of Jacobi algorithms for their optimization. Several implementations are discussed. We compare the proposed approaches with gradient-based techniques from the algorithmic point of view and also on a set of biomedical data.
translated by 谷歌翻译
This paper presents a new technique for achieving blind signal separation when given only a single channel recording. The main concept is based on exploiting a priori sets of time-domain basis functions learned by independent component analysis (ICA) to the separation of mixed source signals observed in a single channel. The inherent time structure of sound sources is reflected in the ICA basis functions, which encode the sources in a statistically efficient manner. We derive a learning algorithm using a maximum likelihood approach given the observed single channel data and sets of basis functions. For each time point we infer the source parameters and their contribution factors. This inference is possible due to prior knowledge of the basis functions and the associated coefficient densities. A flexible model for density estimation allows accurate modeling of the observation and our experimental results exhibit a high level of separation performance for simulated mixtures as well as real environment recordings employing mixtures of two different sources.
translated by 谷歌翻译
We present a class of algorithms for independent component analysis (ICA) which use contrast functions based on canonical correlations in a reproducing kernel Hilbert space. On the one hand, we show that our contrast functions are related to mutual information and have desirable mathematical properties as measures of statistical dependence. On the other hand, building on recent developments in kernel methods, we show that these criteria and their derivatives can be computed efficiently. Minimizing these criteria leads to flexible and robust algorithms for ICA. We illustrate with simulations involving a wide variety of source distributions, showing that our algorithms outperform many of the presently known algorithms.
translated by 谷歌翻译
We introduce the independent factor analysis (IFA) method for recovering independent hidden sources from their observed mixtures. IFA generalizes and uniies ordinary factor analysis (FA), principal component analysis (PCA), and independent component analysis (ICA), and can handle not only square noiseless mixing, but also the general case where the number of mixtures diiers from the number of sources and the data are noisy. IFA is a two-step procedure. In the rst step, the source densities, mixing matrix and noise covariance are estimated from the observed data by maximum likelihood. For this purpose we present an expectation-maximization (EM) algorithm, which performs unsupervised learning of an associated probabilistic model of the mixing situation. Each source in our model is described by a mixture of Gaussians, thus all the probabilistic calculations can be performed analytically. In the second step, the sources are reconstructed from the observed data by an optimal non-linear estimator. A variational approximation of this algorithm is derived for cases with a large number of sources, where the exact algorithm becomes intractable. Our IFA algorithm reduces to the one for ordinary FA when the sources become Gaussian, and to an EM algorithm for PCA in the zero-noise limit. We derive an additional EM algorithm speciically for noiseless IFA. This algorithm is shown to be superior to ICA since it can learn arbitrary source densities from the data. Beyond blind separation, IFA can be used for modeling multi-dimensional data by a highly constrained mixture of Gaussians, and as a tool for non-linear signal encoding.
translated by 谷歌翻译
Most source separation algorithms are based on a model of stationary sources. However, it is a simple matter to take advantage of possible non-stationarities of the sources to achieve separation. This paper develops novel approaches in this direction, based on the principles of maximum likelihood and minimum mutual information. These principles are exploited by efficient algorithms in both the off-line case (via a new joint diagonalization procedure) and in the on-line case (via a Newton-like procedure). Some experiments are presented showing the good performance of our algorithms and evidencing an interesting feature of our methods: their ability to achieve a kind of super-efficiency. The paper concludes with a discussion contrasting separating methods for non-Gaussian and non-stationary models and emphasizing that, as a matter of fact, 'what makes the algorithms work' is-strictly speaking-not the non stationarity itself but rather the property that each realization of the source signals has a time-varying envelope.
translated by 谷歌翻译
In this paper, we propose two methods for separating mixtures of independent sources without any precise knowledge of their probability distribution. They are obtained by considering a maximum likelihood (ML) solution corresponding to some given distributions of the sources and relaxing this assumption afterward. The first method is specially adapted to temporally independent non-Gaussian sources and is based on the use of nonlinear separating functions. The second method is specially adapted to correlated sources with distinct spectra and is based on the use of linear separating filters. A theoretical analysis of the performance of the methods has been made. A simple procedure for optimally choosing the separating functions from a given linear space of functions is proposed. Further, in the second method, a simple implementation based on the simultaneous diagonalization of two symmetric matrices is provided. Finally, some numerical and simulation results are given, illustrating the performance of the method and the good agreement between the experiments and the theory.
translated by 谷歌翻译
We introduce two new functionals, the constrained covariance and the kernel mutual information, to measure the degree of independence of random variables. These quantities are both based on the covariance between functions of the random variables in reproducing kernel Hilbert spaces (RKHSs). We prove that when the RKHSs are universal, both functionals are zero if and only if the random variables are pairwise independent. We also show that the kernel mutual information is an upper bound near independence on the Parzen window estimate of the mutual information. Analogous results apply for two correlation-based dependence functionals introduced earlier: we show the kernel canonical correlation and the kernel generalised variance to be independence measures for universal kernels, and prove the latter to be an upper bound on the mutual information near independence. The performance of the kernel dependence functionals in measuring independence is verified in the context of independent component analysis.
translated by 谷歌翻译
The independent component analysis (ICA) of a random vector consists of searching for a linear transformation that minimizes the statistical dependence between its components. In order to define suitable search criteria, the expansion of mutual information is utilized as a function of cumulants of increasing orders. An efficient algorithm is proposed, which allows the computation of the ICA of a data matrix within a polynomial time. The concept oflCA may actually be seen as an extension of the principal component analysis (PCA), which can only impose independence up to the second order and, consequently, defines directions that are orthogonal. Potential applications of ICA include data analysis and compression, Bayesian detection, localization of sources, and blind identification and deconvolution. Zusammenfassung Die Analyse unabhfingiger Komponenten (ICA) eines Vektors beruht auf der Suche nach einer linearen Transformation , die die statistische Abh~ingigkeit zwischen den Komponenten minimiert. Zur Definition geeigneter Such-Kriterien wird die Entwicklung gemeinsamer Information als Funktion von Kumulanten steigender Ordnung genutzt. Es wird ein effizienter Algorithmus vorgeschlagen, der die Berechnung der ICA ffir Datenmatrizen innerhalb einer polynomischen Zeit erlaubt. Das Konzept der ICA kann eigentlich als Erweiterung der 'Principal Component Analysis' (PCA) betrachtet werden, die nur die Unabh~ingigkeit bis zur zweiten Ordnung erzwingen kann und deshalb Richtungen definiert, die orthogonal sind. Potentielle Anwendungen der ICA beinhalten Daten-Analyse und Kompression, Bayes-Detektion, Quellenlokalisierung und blinde Identifikation und Entfaltung. R~sum~ L'Analyse en Composantes Ind6pendantes (ICA) d'un vecteur al6atoire consiste en la recherche d'une transformation lin6aire qui minimise la d6pendance statistique entre ses composantes. Afin de d6finir des crit6res d'optimisation appropribs, on utilise un d6veloppement en s6rie de l'information mutuelle en fonction de cumulants d'ordre croissant. On propose ensuite un algorithme pratique permettant le calcul de I'ICA d'une matrice de donn6es en un temps polynomial. Le concept d'ICA peut 6tre vu en r~alitb comme une extension de l'Analyse en Composantes Principales (PCA) qui, elle, ne peut imposer l'ind6pendance qu'au second ordre et d6finit par cons6quent des directions orthogonales. Les applications potentielles de I'ICA incluent l'analyse et la compression de donn6es, la d&ection bayesienne, la localisation de sources, et l'identification et la d6convolution aveugles. + This work was supported in part by the DRET. 0165-1684/94/$7.00 © 1994 Elsevier Science B.V. All rights reserved SSDI 0165-1684(93)E0093-Z 288 P. Comon / Signal Processing 36 (1994) 28~314
translated by 谷歌翻译
我们描述了一种以与基于独立成分分析(ICA)的方法类似的方式解混合自由独立随机变量的混合物的方法,该方法用于从其加性混合物中分离独立的随机变量。随机矩阵在此上下文中起到自由随机变量的作用,因此我们开发的方法(我们称之为自由成分分析(FCA))从矩阵的加性混合中解混合矩阵。我们描述了理论,各种算法,并将FCA与ICA进行了比较。我们表明,在每个应用程序中,例如图像和语音解混,ICCA都可以表现出与ICA相比,并且通常比ICA更好,因为ICA已经成功。我们的计算实验表明,不那么随机的矩阵,例如波形的图像和光谱图(在野外“更接近”比我们在理论上预期的更自由。
translated by 谷歌翻译
我们提出了两类改进的互信息估计量$ M(X,Y)$,来自根据一些联合概率密度$ \ mu(x,y)$分布的随机点样本。与基于分类的传统估计相比,它们基于$ k $最近邻距离的熵估计。这意味着它们具有数据效率($ k = 1 $我们将结构调整到最小可能的尺度),自适应(分辨率越高,数据越多),并且偏差最小。实际上,潜在熵估计的偏差主要是由于在最小分辨尺度上的密度的不均匀性,通常给出系统误差,其作为$ k / N $的函数按比例计算$ N $点。在数值上,我们发现两个家庭对于独立分布变得{\ it exact},即如果$ \ mu(x,y)= \ mu(x),则theestimator $ \ hat M(X,Y)$消失(达到统计波动) \亩(Y)$。这适用于所有测试的边际分布以及$ x $和$ y $的维度。此外,我们给出了超过2个随机变量之间冗余的估计量。我们详细比较了算法的算法。最后,我们证明了我们的估算器用于评估从独立成分分析(ICA)获得的组件的实际独立性,用于改进ICA以及用于估计盲源分离的可靠性的有用性。
translated by 谷歌翻译
A novel (differential) entropy estimator is introduced where the maximum entropy bound is used to approximate the en-tropy given the observations, and is computed using a numerical procedure thus resulting in accurate estimates for the entropy. We show that such an estimator exists for a wide class of measuring functions, and provide a number of design examples to demonstrate its flexible nature. We then derive a novel independent component analysis (ICA) algorithm that uses the entropy estimate thus obtained, ICA by entropy bound minimization (ICA-EBM). The algorithm adopts a line search procedure, and initially uses updates that constrain the demixing matrix to be orthogonal for robust performance. We demonstrate the superior performance of ICA-EBM and its ability to match sources that come from a wide range of distributions using simulated and real-world data. Index Terms-Blind source separation (BSS), differential entropy, independent component analysis (ICA), principle of maximum entropy.
translated by 谷歌翻译
A Bayesian ensemble learning method is introduced for unsupervised extraction of dynamic processes from noisy data. The data are assumed to be generated by an unknown nonlinear mapping from unknown factors. The dynamics of the factors are modeled using a nonlinear state-space model. The nonlinear mappings in the model are represented using multilayer perceptron networks. The proposed method is computationally demanding, but it allows the use of higher dimensional nonlinear latent variable models than other existing approaches. Experiments with chaotic data show that the new method is able to blindly estimate the factors and the dynamic process which have generated the data. It clearly outperforms currently available nonlinear prediction techniques in this very difficult test problem.
translated by 谷歌翻译
A number of current face recognition algorithms use face representations found by unsupervised statistical methods. Typically these methods find a set of basis images and represent faces as a linear combination of those images. Principal component analysis (PCA) is a popular example of such methods. The basis images found by PCA depend only on pairwise relationships between pixels in the image database. In a task such as face recognition, in which important information may be contained in the high-order relationships among pixels, it seems reasonable to expect that better basis images may be found by methods sensitive to these high-order statistics. Independent component analysis (ICA), a generalization of PCA, is one such method. We used a version of ICA derived from the principle of optimal information transfer through sigmoidal neurons. ICA was performed on face images in the FERET database under two different architectures, one which treated the images as random variables and the pixels as outcomes, and a second which treated the pixels as random variables and the images as outcomes. The first architecture found spatially local basis images for the faces. The second architecture produced a factorial face code. Both ICA representations were superior to representations based on PCA for recognizing faces across days and changes in expression. A classifier that combined the two ICA representations gave the best performance. Index Terms-Eigenfaces, face recognition, independent component analysis (ICA), principal component analysis (PCA), unsupervised learning.
translated by 谷歌翻译
Algorithms for data-driven learning of domain-specific overcomplete dictionaries are developed to obtain maximum likelihood and maximum a posteriori dictionary estimates based on the use of Bayesian models with concave/Schur-concave (CSC) negative log priors. Such priors are appropriate for obtaining sparse representations of environmental signals within an appropriately chosen (environmentally matched) dictionary. The elements of the dictionary can be interpreted as concepts, features, or words capable of succinct expression of events encountered in the environment (the source of the measured signals). This is a generalization of vector quantization in that one is interested in a description involving a few dictionary entries (the proverbial "25 words or less"), but not necessarily as succinct as one entry. To learn an environmentally adapted dictionary capable of concise expression of signals generated by the environment , we develop algorithms that iterate between a representative set of sparse representations found by variants of FOCUSS and an update of the dictionary using these sparse representations. Experiments were performed using synthetic data and natural images. For complete dictionaries, we demonstrate that our algorithms have im-Neural Computation 15, 349-396 (2003) proved performance over other independent component analysis (ICA) methods, measured in terms of signal-to-noise ratios of separated sources. In the overcomplete case, we show that the true underlying dictionary and sparse sources can be accurately recovered. In tests with natural images, learned overcomplete dictionaries are shown to have higher coding efficiency than complete dictionaries; that is, images encoded with an over-complete dictionary have both higher compression (fewer bits per pixel) and higher accuracy (lower mean square error).
translated by 谷歌翻译
Separation of complex valued signals is a frequently arising problem in signal processing. For example, separation of convolutively mixed source signals involves computations on complex valued signals. In this article, it is assumed that the original, complex valued source signals are mutually statistically independent, and the problem is solved by the independent component analysis (ICA) model. ICA is a statistical method for transforming an observed multidimensional random vector into components that are mutually as independent as possible. In this article, a fast fixed-point type algorithm that is capable of separating complex valued, linearly mixed source signals is presented and its computational efficiency is shown by simulations. Also, the local consistency of the estimator given by the algorithm is proved.
translated by 谷歌翻译