We consider the problem of learning deep generative models from data. We formulate a method that generates an independent sample via a single feedforward pass through a multilayer preceptron, as in the recently proposed generative adversarial networks (Goodfellow et al., 2014). Training a generative adversarial network, however, requires careful optimization of a difficult minimax program. Instead, we utilize a technique from statistical hypothesis testing known as maximum mean discrepancy (MMD), which leads to a simple objective that can be interpreted as matching all orders of statistics between a dataset and samples from the model, and can be trained by backpropagation. We further boost the performance of this approach by combining our generative network with an auto-encoder network, using MMD to learn to generate codes that can then be decoded to produce samples. We show that the combination of these techniques yields excellent generative models compared to baseline approaches as measured on MNIST and the Toronto Face Database.

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, auto-encoders, manifold learning, and deep networks. This motivates longer-term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning.

How can we perform efficient inference and learning in directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions, and large datasets? We introduce a stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case. Our contributions is two-fold. First, we show that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods. Second, we show that for i.i.d. datasets with continuous latent variables per datapoint, posterior inference can be made especially efficient by fitting an approximate inference model (also called a recognition model) to the intractable posterior using the proposed lower bound estimator. Theoretical advantages are reflected in experimental results.

变异推理（VI）的核心原理是将计算复杂后概率密度计算的统计推断问题转换为可拖动的优化问题。该属性使VI比几种基于采样的技术更快。但是，传统的VI算法无法扩展到大型数据集，并且无法轻易推断出越野数据点，而无需重新运行优化过程。该领域的最新发展，例如随机，黑框和摊销VI，已帮助解决了这些问题。如今，生成的建模任务广泛利用摊销VI来实现其效率和可扩展性，因为它利用参数化函数来学习近似的后验密度参数。在本文中，我们回顾了各种VI技术的数学基础，以构成理解摊销VI的基础。此外，我们还概述了最近解决摊销VI问题的趋势，例如摊销差距，泛化问题，不一致的表示学习和后验崩溃。最后，我们分析了改善VI优化的替代差异度量。

We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1 2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.

We propose a deep learning framework for modeling complex high-dimensional densities called Non-linear Independent Component Estimation (NICE). It is based on the idea that a good representation is one in which the data has a distribution that is easy to model. For this purpose, a non-linear deterministic transformation of the data is learned that maps it to a latent space so as to make the transformed data conform to a factorized distribution, i.e., resulting in independent latent variables. We parametrize this transformation so that computing the determinant of the Jacobian and inverse Jacobian is trivial, yet we maintain the ability to learn complex non-linear transformations, via a composition of simple building blocks, each based on a deep neural network. The training criterion is simply the exact log-likelihood, which is tractable. Unbiased ancestral sampling is also easy. We show that this approach yields good generative models on four image datasets and can be used for inpainting.

The ever-increasing size of modern data sets combined with the difficulty of obtaining label information has made semi-supervised learning one of the problems of significant practical importance in modern data analysis. We revisit the approach to semi-supervised learning with generative models and develop new models that allow for effective generalisation from small labelled data sets to large unlabelled ones. Generative approaches have thus far been either inflexible, inefficient or non-scalable. We show that deep generative models and approximate Bayesian inference exploiting recent advances in variational methods can be used to provide significant improvements, making generative approaches highly competitive for semi-supervised learning.

We explore an original strategy for building deep networks, based on stacking layers of denoising autoencoders which are trained locally to denoise corrupted versions of their inputs. The resulting algorithm is a straightforward variation on the stacking of ordinary autoencoders. It is however shown on a benchmark of classification problems to yield significantly lower classification error, thus bridging the performance gap with deep belief networks (DBN), and in several cases surpassing it. Higher level representations learnt in this purely unsupervised fashion also help boost the performance of subsequent SVM classifiers. Qualitative experiments show that, contrary to ordinary autoencoders, denoising autoencoders are able to learn Gabor-like edge detectors from natural image patches and larger stroke detectors from digit images. This work clearly establishes the value of using a denoising criterion as a tractable unsupervised objective to guide the learning of useful higher level representations.

We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference and learning. Our algorithm introduces a recognition model to represent an approximate posterior distribution and uses this for optimisation of a variational lower bound. We develop stochastic backpropagation -rules for gradient backpropagation through stochastic variables -and derive an algorithm that allows for joint optimisation of the parameters of both the generative and recognition models. We demonstrate on several real-world data sets that by using stochastic backpropagation and variational inference, we obtain models that are able to generate realistic samples of data, allow for accurate imputations of missing data, and provide a useful tool for high-dimensional data visualisation.

这是一门专门针对STEM学生开发的介绍性机器学习课程。我们的目标是为有兴趣的读者提供基础知识，以在自己的项目中使用机器学习，并将自己熟悉术语作为进一步阅读相关文献的基础。在这些讲义中，我们讨论受监督，无监督和强化学习。注释从没有神经网络的机器学习方法的说明开始，例如原理分析，T-SNE，聚类以及线性回归和线性分类器。我们继续介绍基本和先进的神经网络结构，例如密集的进料和常规神经网络，经常性的神经网络，受限的玻尔兹曼机器，（变性）自动编码器，生成的对抗性网络。讨论了潜在空间表示的解释性问题，并使用梦和对抗性攻击的例子。最后一部分致力于加强学习，我们在其中介绍了价值功能和政策学习的基本概念。

近似复杂的概率密度是现代统计中的核心问题。在本文中，我们介绍了变分推理（VI）的概念，这是一种机器学习中的流行方法，该方法使用优化技术来估计复杂的概率密度。此属性允许VI汇聚速度比经典方法更快，例如Markov Chain Monte Carlo采样。概念上，VI通过选择一个概率密度函数，然后找到最接近实际概率密度的家庭 - 通常使用Kullback-Leibler（KL）发散作为优化度量。我们介绍了缩窄的证据，以促进近似的概率密度，我们审查了平均场变分推理背后的想法。最后，我们讨论VI对变分式自动编码器（VAE）和VAE-生成的对抗网络（VAE-GAN）的应用。用本文，我们的目标是解释VI的概念，并通过这种方法协助协助。

标准化流是构建概率和生成模型的流行方法。但是，由于需要计算雅各布人的计算昂贵决定因素，因此对流量的最大似然训练是具有挑战性的。本文通过引入一种受到两样本测试启发的流动训练的方法来解决这一挑战。我们框架的核心是能源目标，这是适当评分规则的多维扩展，该规则基于随机预测，可以接受有效的估计器，并且超过了一系列可以在我们的框架中得出的替代两样本目标。至关重要的是，能量目标及其替代方案不需要计算决定因素，因此支持不适合最大似然训练的一般流量体系结构（例如，密度连接的网络）。我们从经验上证明，能量流达到竞争性生成建模性能，同时保持快速产生和后部推断。

Copulas是一种强大的工具，用于建模多变量分布，因为它们允许分别估计单变量边缘分布和联合依赖结构。然而，已知的参数Copulas提供有限的灵活性，特别是高尺寸，而常用的非参数方法遭受维度的诅咒。受欢迎的补救措施是构建一个基于树的条件双变量Copulas的层次结构。在本文中，我们提出了一种基于隐含生成神经网络的灵活，概念性的简单替代品。关键挑战是确保估计的拷贝分布的边际均匀性。我们通过学习具有未指定的边缘的多变量潜在分布而是所需的依赖结构来实现这一目标。通过应用概率积分变换，我们可以从高维拷贝分布中获得样本而不依赖参数假设或需要找到合适的树结构。来自金融，物理和图​​像生成的合成和实数据的实验证明了这种方法的性能。

生成神经网络能够模仿复杂的概率分布，例如手写文本，自然图像等。自从他们的初始化提出了几种模型。这些最成功的是基于对抗性（GaN），自动编码（VAE）和最大平均差异（MMD）相对复杂的架构和方案。令人惊讶的是，显然忽略了一个非常简单的架构（一个前馈神经网络）与明显的优化目标（Kullback_Leibler发散）结合。本文表明这种模型（表示其简单性的SGN）能够在视觉上和定量地竞争的样本与现有技术的前述方法相比。

In this paper, we tackle the problem of domain generalization: how to learn a generalized feature representation for an "unseen" target domain by taking the advantage of multiple seen source-domain data. We present a novel framework based on adversarial autoencoders to learn a generalized latent feature representation across domains for domain generalization. To be specific, we extend adversarial autoencoders by imposing the Maximum Mean Discrepancy (MMD) measure to align the distributions among different domains, and matching the aligned distribution to an arbitrary prior distribution via adversarial feature learning. In this way, the learned feature representation is supposed to be universal to the seen source domains because of the MMD regularization, and is expected to generalize well on the target domain because of the introduction of the prior distribution. We proposed an algorithm to jointly train different components of our proposed framework. Extensive experiments on various vision tasks demonstrate that our proposed framework can learn better generalized features for the unseen target domain compared with state-of-the-art domain generalization methods.

使用显式密度建模的生成模型（例如，变形式自动码码器，基于流动的生成模型）涉及从已知分布的映射，例如，从已知分布中找到映射。高斯，到未知的输入分布。这通常需要搜索一类非线性函数（例如，由深神经网络表示）。在实践中有效，相关的运行时/内存成本可以迅速增加，通常是应用程序中所需性能的函数。我们提出了一个更便宜的（更简单）的策略来估算基于内核传输运算符中的已知结果的此映射。我们表明我们的配方能够实现高效的分布近似和采样，并提供令人惊讶的良好的经验性能，与强大的基线有利，但有很大的运行时储蓄。我们表明该算法在小样本大小设置（脑成像）中也表现良好。

与CNN的分类，分割或对象检测相比，生成网络的目标和方法根本不同。最初，它们不是作为图像分析工具，而是生成自然看起来的图像。已经提出了对抗性训练范式来稳定生成方法，并已被证明是非常成功的 - 尽管绝不是第一次尝试。本章对生成对抗网络（GAN）的动机进行了基本介绍，并通​​过抽象基本任务和工作机制并得出了早期实用方法的困难来追溯其成功的道路。将显示进行更稳定的训练方法，也将显示出不良收敛及其原因的典型迹象。尽管本章侧重于用于图像生成和图像分析的gan，但对抗性训练范式本身并非特定于图像，并且在图像分析中也概括了任务。在将GAN与最近进入场景的进一步生成建模方法进行对比之前，将闻名图像语义分割和异常检测的架构示例。这将允许对限制的上下文化观点，但也可以对gans有好处。

近年来，由于其对复杂分布进行建模的能力，深层生成模型引起了越来越多的兴趣。在这些模型中，变异自动编码器已被证明是计算有效的，并且在多个领域中产生了令人印象深刻的结果。在这一突破之后，为了改善原始出版物而进行了广泛的研究，从而导致各种不同的VAE模型响应不同的任务。在本文中，我们介绍了Pythae，这是一个多功能的开源Python库，既可以提供统一的实现和专用框架，允许直接，可重现且可靠地使用生成自动编码器模型。然后，我们建议使用此库来执行案例研究基准测试标准，在其中我们介绍并比较了19个生成自动编码器模型，代表了下游任务的一些主要改进，例如图像重建，生成，分类，聚类，聚类和插值。可以在https://github.com/clementchadebec/benchmark_vae上找到开源库。

该报告解释，实施和扩展了“更紧密的变化界限不一定更好”所介绍的作品（T Rainforth等，2018）。我们提供了理论和经验证据，这些证据增加了重要性的重要性数量$ k $在重要性加权自动编码器（IWAE）中（Burda等，2016）降低了推理中梯度估计量的信噪比（SNR）网络，从而影响完整的学习过程。换句话说，即使增加$ k $减少了梯度的标准偏差，但它也会更快地降低真实梯度的幅度，从而增加梯度更新的相对差异。进行广泛的实验以了解$ k $的重要性。这些实验表明，更紧密的变化界限对生成网络有益，而宽松的边界对推理网络来说是可取的。通过这些见解，可以实施和研究三种方法：部分重要性加权自动编码器（PIWAE），倍增重要性加权自动编码器（MIWAE）和组合重要性加权自动编码器（CIWAE）。这三种方法中的每一种都需要IWAE作为一种特殊情况，但采用不同的重量权重，以确保较高的梯度估计器的SNR。在我们的研究和分析中，这些算法的疗效在多个数据集（如MNIST和Omniglot）上进行了测试。最后，我们证明了三种呈现的IWAE变化能够产生近似后验分布，这些分布与IWAE更接近真正的后验分布，同时匹配IWAE生成网络的性能，或者在PIWAE的情况下可能超过其表现。

在这项工作中，我们为生成自动编码器的变异培训提供了确切的可能性替代方法。我们表明，可以使用可逆层来构建VAE风格的自动编码器，该层提供了可拖动的精确可能性，而无需任何正则化项。这是在选择编码器，解码器和先前体系结构的全部自由的同时实现的，这使我们的方法成为培训现有VAE和VAE风格模型的替换。我们将结果模型称为流中的自动编码器（AEF），因为编码器，解码器和先验被定义为整体可逆体系结构的单个层。我们表明，在对数可能，样本质量和降低性能的方面，该方法的性能比结构上等效的VAE高得多。从广义上讲，这项工作的主要野心是在共同的可逆性和确切的最大可能性的共同框架下缩小正常化流量和自动编码器文献之间的差距。