生成对抗网络（GAN）是用于复杂数据生成建模的广泛使用的工具。尽管取得了经验成功，但由于发电机和鉴别器的最低最大优化，对GAN的训练尚未完全理解。本文分析了这些关节动力学时，当真实样品以及生成的样品是离散的，有限的集合，并且鉴别器基于内核。引入了一个简单而表达的框架，用于分析培训，称为$ \ textit {隔离点模型} $。在提出的模型中，真实样品之间的距离大大超过了内核宽度，因此每个生成的点都受到最多一个真实点的影响。我们的模型可以精确地表征好和不良最小值的收敛条件。特别是，分析解释了两种常见的故障模式：（i）近似模式崩溃和（ii）差异。提供了可预测复制这些行为的数值模拟。

Recent work has shown local convergence of GAN training for absolutely continuous data and generator distributions. In this paper, we show that the requirement of absolute continuity is necessary: we describe a simple yet prototypical counterexample showing that in the more realistic case of distributions that are not absolutely continuous, unregularized GAN training is not always convergent. Furthermore, we discuss regularization strategies that were recently proposed to stabilize GAN training. Our analysis shows that GAN training with instance noise or zerocentered gradient penalties converges. On the other hand, we show that Wasserstein-GANs and WGAN-GP with a finite number of discriminator updates per generator update do not always converge to the equilibrium point. We discuss these results, leading us to a new explanation for the stability problems of GAN training. Based on our analysis, we extend our convergence results to more general GANs and prove local convergence for simplified gradient penalties even if the generator and data distributions lie on lower dimensional manifolds. We find these penalties to work well in practice and use them to learn highresolution generative image models for a variety of datasets with little hyperparameter tuning.

我们为生成对抗网络（GAN）提出了一个新颖的理论框架。我们揭示了先前分析的基本缺陷，通过错误地对GANS的训练计划进行了错误的建模，该缺陷受到定义不定的鉴别梯度的约束。我们克服了这个问题，该问题阻碍了对GAN培训的原则研究，并考虑了歧视者的体系结构在我们的框架内解决它。为此，我们通过其神经切线核为歧视者提供了无限宽度神经网络的理论。我们表征了训练有素的判别器，以实现广泛的损失，并建立网络的一般可怜性属性。由此，我们获得了有关生成分布的融合的新见解，从而促进了我们对GANS训练动态的理解。我们通过基于我们的框架的分析工具包来证实这些结果，并揭示了与GAN实践一致的直觉。

We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs. As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters. We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramér GAN critic. Being an integral probability metric, the MMD benefits from training strategies recently developed for Wasserstein GANs. In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance. We also propose an improved measure of GAN convergence, the Kernel Inception Distance, and show how to use it to dynamically adapt learning rates during GAN training.

生成对抗网络（GAN）是强大的机器学习模型，能够生成具有高分辨率的所需现象的完全合成样本。尽管他们成功了，但GAN的训练过程非常不稳定，通常有必要对网络实施几种附属启发式方法，以达到模型的可接受收敛。在本文中，我们介绍了一种新颖的方法来分析生成对抗网络培训的收敛性和稳定性。为此，我们建议分解对手Min-Max游戏的目标功能，将定期gan定义为傅立叶系列。通过研究连续交替梯度下降算法的截短傅里叶序列的动力学，我们能够近似实际流量并确定GAN收敛的主要特征。通过研究$ 2 $ - 参数gan的旨在产生未知指数分布的训练流，从经验上证实了这种方法。作为副产品，我们表明gan中的融合轨道是周期性轨道的小扰动，因此纳什均值是螺旋吸引子。从理论上讲，这证明了在甘斯中观察到的缓慢和不稳定的训练。

Generative adversarial networks (GANs) are a class of machine-learning models that use adversarial training to generate new samples with the same (potentially very complex) statistics as the training samples. One major form of training failure, known as mode collapse, involves the generator failing to reproduce the full diversity of modes in the target probability distribution. Here, we present an effective model of GAN training, which captures the learning dynamics by replacing the generator neural network with a collection of particles in the output space; particles are coupled by a universal kernel valid for certain wide neural networks and high-dimensional inputs. The generality of our simplified model allows us to study the conditions under which mode collapse occurs. Indeed, experiments which vary the effective kernel of the generator reveal a mode collapse transition, the shape of which can be related to the type of discriminator through the frequency principle. Further, we find that gradient regularizers of intermediate strengths can optimally yield convergence through critical damping of the generator dynamics. Our effective GAN model thus provides an interpretable physical framework for understanding and improving adversarial training.

我们证明了由例如He等人提出的广泛使用的方法。（2015年）并使用梯度下降对最小二乘损失进行训练并不普遍。具体而言，我们描述了一大批一维数据生成分布，较高的概率下降只会发现优化景观的局部最小值不好，因为它无法将其偏离偏差远离其初始化，以零移动。。事实证明，在这些情况下，即使目标函数是非线性的，发现的网络也基本执行线性回归。我们进一步提供了数值证据，表明在实际情况下，对于某些多维分布而发生这种情况，并且随机梯度下降表现出相似的行为。我们还提供了有关初始化和优化器的选择如何影响这种行为的经验结果。

The modeling of probability distributions, specifically generative modeling and density estimation, has become an immensely popular subject in recent years by virtue of its outstanding performance on sophisticated data such as images and texts. Nevertheless, a theoretical understanding of its success is still incomplete. One mystery is the paradox between memorization and generalization: In theory, the model is trained to be exactly the same as the empirical distribution of the finite samples, whereas in practice, the trained model can generate new samples or estimate the likelihood of unseen samples. Likewise, the overwhelming diversity of distribution learning models calls for a unified perspective on this subject. This paper provides a mathematical framework such that all the well-known models can be derived based on simple principles. To demonstrate its efficacy, we present a survey of our results on the approximation error, training error and generalization error of these models, which can all be established based on this framework. In particular, the aforementioned paradox is resolved by proving that these models enjoy implicit regularization during training, so that the generalization error at early-stopping avoids the curse of dimensionality. Furthermore, we provide some new results on landscape analysis and the mode collapse phenomenon.

生成的对抗网络后面的数学力量提高了具有挑战性的理论问题。通过表征产生的分布的几何特性的重要问题，我们在有限的样本和渐近制度中对Wassersein Gans（WGAN）进行了彻底分析。我们研究了潜伏空间是单变量的特定情况，并且不管输出空间的尺寸如何有效。我们特别地显示出用于固定的样本大小，最佳WGAN与连接路径紧密相连，最小化采样点之间的平方欧几里德距离的总和。我们还强调了WGAN能够接近的事实（对于1-Wasserstein距离）目标分布，因为样本大小趋于无穷大，在给定的会聚速率下，并且提供了生成的Lipschitz函数的家族适当地增长。我们在半离散环境中获得了在最佳运输理论上传递新结果。

比较概率分布是许多机器学习算法的关键。最大平均差异（MMD）和最佳运输距离（OT）是在过去几年吸引丰富的关注的概率措施之间的两类距离。本文建立了一些条件，可以通过MMD规范控制Wassersein距离。我们的作品受到压缩统计学习（CSL）理论的推动，资源有效的大规模学习的一般框架，其中训练数据总结在单个向量（称为草图）中，该训练数据捕获与所考虑的学习任务相关的信息。在CSL中的现有结果启发，我们介绍了H \“较旧的较低限制的等距属性（H \”较旧的LRIP）并表明这家属性具有有趣的保证对压缩统计学习。基于MMD与Wassersein距离之间的关系，我们通过引入和研究学习任务的Wassersein可读性的概念来提供压缩统计学习的保证，即概率分布之间的某些特定于特定的特定度量，可以由Wassersein界定距离。

非线性自适应控制理论中的一个关键假设是系统的不确定性可以在一组已知基本函数的线性跨度中表示。虽然该假设导致有效的算法，但它将应用限制为非常特定的系统类别。我们介绍一种新的非参数自适应算法，其在参数上学习无限尺寸密度，以取消再现内核希尔伯特空间中的未知干扰。令人惊讶的是，所产生的控制输入承认，尽管其底层无限尺寸结构，但是尽管它的潜在无限尺寸结构实现了其实施的分析表达。虽然这种自适应输入具有丰富和富有敏感性的 - 例如，传统的线性参数化 - 其计算复杂性随时间线性增长，使其比其参数对应力相对较高。利用随机傅里叶特征的理论，我们提供了一种有效的随机实现，该实现恢复了经典参数方法的复杂性，同时可透明地保留非参数输入的表征性。特别地，我们的显式范围仅取决于系统的基础参数，允许我们所提出的算法有效地缩放到高维系统。作为该方法的说明，我们展示了随机近似算法学习由牛顿重力交互的十点批量组成的60维系统的预测模型的能力。

我们的工作侧重于额外的渐变学习算法，用于在双线性零和游戏中查找纳什均衡。该方法可以正式被认为是乐观镜下降\ Cite {DBLP：Cenf / ICLR / Mertikopouloslz19}的典型方法，用于中间梯度步骤，基本上导致计算（近似）最佳反应策略先前迭代的轮廓。虽然乍一看，由于不合理的大，但是对于迭代算法，中间学习步骤，我们证明该方法保证了持续收敛到均衡。特别是，我们表明该算法首先达到$ \ eta ^ {1 / rho} $ - 近似纳什均衡，以$ \ rho> 1 $，通过减少每次迭代的kullback-leibler分歧至少$ \ omega （\ eta ^ {1+ \ frac {1} {\ rho}）$，因为足够小的学习率，$ \ eta $直到该方法成为承包地图，并收敛到确切的均衡。此外，我们对乘法权重更新方法的乐观变体进行实验比较，\ Cite {Daskalakis2019LastITERATECZ}并显示我们的算法具有显着的实际潜力，因为它在加速收敛方面提供了大量的收益。

Generative Adversarial Networks (GANs) excel at creating realistic images with complex models for which maximum likelihood is infeasible. However, the convergence of GAN training has still not been proved. We propose a two time-scale update rule (TTUR) for training GANs with stochastic gradient descent on arbitrary GAN loss functions. TTUR has an individual learning rate for both the discriminator and the generator. Using the theory of stochastic approximation, we prove that the TTUR converges under mild assumptions to a stationary local Nash equilibrium. The convergence carries over to the popular Adam optimization, for which we prove that it follows the dynamics of a heavy ball with friction and thus prefers flat minima in the objective landscape. For the evaluation of the performance of GANs at image generation, we introduce the 'Fréchet Inception Distance" (FID) which captures the similarity of generated images to real ones better than the Inception Score. In experiments, TTUR improves learning for DCGANs and Improved Wasserstein GANs (WGAN-GP) outperforming conventional GAN training on CelebA, CIFAR-10, SVHN, LSUN Bedrooms, and the One Billion Word Benchmark.

计算科学和统计推断中的许多应用都需要计算有关具有未知归一化常数的复杂高维分布以及这些常数的估计。在这里，我们开发了一种基于从简单的基本分布生成样品，沿着速度场生成的流量运输的方法，并沿这些流程线执行平均值。这种非平衡重要性采样（NEIS）策略是直接实施的，可用于具有任意目标分布的计算。在理论方面，我们讨论了如何将速度场定制到目标，并建立所提出的估计器是一个完美的估计器，具有零变化。我们还通过将基本分布映射到目标上，通过传输图绘制了NEIS和方法之间的连接。在计算方面，我们展示了如何使用深度学习来代表神经网络，并将其训练为零方差最佳。这些结果在高维示例上进行了数值说明，我们表明训练速度场可以将NEIS估计量的方差降低至6个数量级，而不是Vanilla估计量。我们还表明，NEIS在这些示例上的表现要比NEAL的退火重要性采样（AIS）更好。

We introduce a new algorithm named WGAN, an alternative to traditional GAN training. In this new model, we show that we can improve the stability of learning, get rid of problems like mode collapse, and provide meaningful learning curves useful for debugging and hyperparameter searches. Furthermore, we show that the corresponding optimization problem is sound, and provide extensive theoretical work highlighting the deep connections to different distances between distributions.

We introduce a class of first-order methods for smooth constrained optimization that are based on an analogy to non-smooth dynamical systems. Two distinctive features of our approach are that (i) projections or optimizations over the entire feasible set are avoided, in stark contrast to projected gradient methods or the Frank-Wolfe method, and (ii) iterates are allowed to become infeasible, which differs from active set or feasible direction methods, where the descent motion stops as soon as a new constraint is encountered. The resulting algorithmic procedure is simple to implement even when constraints are nonlinear, and is suitable for large-scale constrained optimization problems in which the feasible set fails to have a simple structure. The key underlying idea is that constraints are expressed in terms of velocities instead of positions, which has the algorithmic consequence that optimizations over feasible sets at each iteration are replaced with optimizations over local, sparse convex approximations. In particular, this means that at each iteration only constraints that are violated are taken into account. The result is a simplified suite of algorithms and an expanded range of possible applications in machine learning.

我们研究了最近引入的最低最大优化框架的一种变体，其中最大玩具被限制以贪婪的方式更新其参数，直到达到一阶固定点为止。我们对此框架的平衡定义取决于最小玩家使用该方向来更新其参数的方向的提案分布。我们表明，鉴于一个平稳且有界的非Convex-Nonconcave目标函数，访问Min-player的更新的任何提案分布以及最大播放器的随机梯度甲骨文，我们的算法收敛于上述近似近似近似局部平衡，以众多的局部平衡。不取决于维度的迭代。我们的算法发现的平衡点取决于提议分布，在应用我们的算法来训练gans时，我们选择提案分布作为随机梯度的分布。我们从经验上评估了我们的算法，以挑战非凸孔测试功能和GAN培训中引起的损失功能。我们的算法在这些测试功能上收敛，并在用于训练gans时会在合成和现实世界中稳定训练，并避免模式崩溃

深度学习的概括分析通常假定训练会收敛到固定点。但是，最近的结果表明，实际上，用随机梯度下降优化的深神经网络的权重通常无限期振荡。为了减少理论和实践之间的这种差异，本文着重于神经网络的概括，其训练动力不一定会融合到固定点。我们的主要贡献是提出一个统计算法稳定性（SAS）的概念，该算法将经典算法稳定性扩展到非convergergent算法并研究其与泛化的联系。与传统的优化和学习理论观点相比，这种崇高的理论方法可导致新的见解。我们证明，学习算法的时间复杂行为的稳定性与其泛化有关，并在经验上证明了损失动力学如何为概括性能提供线索。我们的发现提供了证据表明，即使训练无限期继续并且权重也不会融合，即使训练持续进行训练，训练更好地概括”的网络也是如此。

最近在J. Math中引入的分配流程。成像和视觉58/2（2017）构成了一种高维动态系统，其在基本统计歧管上发展，并执行任何度量空间中给出的数据的上下文标记（分类）。给定图形的顶点索引数据点并定义邻域的系统。这些邻域与非负重量参数一起定义标签分配的演变的正则化，通过由信息几何的仿射电子连接引起的几何平均来定义对数据点的数量。关于进化游戏动态，分配流程可以被称为由几何平均耦合的复制器方程的大型系统。本文在重量参数上建立了保证连续时间分配流程的重量参数（标签）的融合，最多可忽略不计在实际数据的实际数据时不会遇到的情况。此外，我们对流动的吸引子分类并量化相应的吸引力盆地。这为分配流提供了会聚保证，该分配流程扩展到不同时间分配流程，这些流量是应用跑步-Kutta-munthe-KAAS方案的用于分配流的数值几何集成。若干反作用例说明违反条件可能需要关于上下文数据分类的分配流的不利行为。

Lipschitz regularized f-divergences are constructed by imposing a bound on the Lipschitz constant of the discriminator in the variational representation. They interpolate between the Wasserstein metric and f-divergences and provide a flexible family of loss functions for non-absolutely continuous (e.g. empirical) distributions, possibly with heavy tails. We construct Lipschitz regularized gradient flows on the space of probability measures based on these divergences. Examples of such gradient flows are Lipschitz regularized Fokker-Planck and porous medium partial differential equations (PDEs) for the Kullback-Leibler and alpha-divergences, respectively. The regularization corresponds to imposing a Courant-Friedrichs-Lewy numerical stability condition on the PDEs. For empirical measures, the Lipschitz regularization on gradient flows induces a numerically stable transporter/discriminator particle algorithm, where the generative particles are transported along the gradient of the discriminator. The gradient structure leads to a regularized Fisher information (particle kinetic energy) used to track the convergence of the algorithm. The Lipschitz regularized discriminator can be implemented via neural network spectral normalization and the particle algorithm generates approximate samples from possibly high-dimensional distributions known only from data. Notably, our particle algorithm can generate synthetic data even in small sample size regimes. A new data processing inequality for the regularized divergence allows us to combine our particle algorithm with representation learning, e.g. autoencoder architectures. The resulting algorithm yields markedly improved generative properties in terms of efficiency and quality of the synthetic samples. From a statistical mechanics perspective the encoding can be interpreted dynamically as learning a better mobility for the generative particles.