通常使用自回归生成模型,尤其是对于涉及顺序数据的那些任务。然而,由于链式有条件建模的内在特征(例如,暴露偏见或缺乏远距离连贯性),由于许多固有的缺陷而困扰着它们,严重限制了它们正确模型分布的能力。在本文中,我们提出了一种独特的方法,该方法称为训练自回旋生成模型,以利用精心设计的基于能量的学习目标。通过利用SoftMax操作的额外自由度,我们被允许使自回归模型本身成为基于能量的模型,用于衡量输入的可能性,而无需引入任何额外的参数。此外,我们表明可以有效地训练电子臂,并能够减轻暴露偏置问题并增加自回归生成模型的时间连贯性。广泛的经验结果涵盖了语言建模,神经机器翻译和图像产生等基准,证明了拟议方法的有效性。
translated by 谷歌翻译
在本文中,我们提出了一种新的生成模型,逐步逐步的去噪AutoEncoder(Sundae),不依赖于自回归模型。类似地与去噪扩散技术,在从随机输入开始并从随机输入开始并每次直到收敛改善它们时,日出施加Sundae。我们提出了一个简单的新改进运算符,它比扩散方法更少迭代,同时在定性地在自然语言数据集上产生更好的样本。Sundae在WMT'14英语到德语翻译任务上实现最先进的结果(非自回归方法),在巨大清洁的常见爬网数据集和Python代码的数据集上对无条件语言建模的良好定性结果来自GitHub。通过在模板中填充任意空白模式,Sundae的非自动增加性质开辟了超出左右提示的可能性。
translated by 谷歌翻译
Recently, contrastive learning attracts increasing interests in neural text generation as a new solution to alleviate the exposure bias problem. It introduces a sequence-level training signal which is crucial to generation tasks that always rely on auto-regressive decoding. However, previous methods using contrastive learning in neural text generation usually lead to inferior performance. In this paper, we analyse the underlying reasons and propose a new Contrastive Neural Text generation framework, CoNT. CoNT addresses bottlenecks that prevent contrastive learning from being widely adopted in generation tasks from three aspects -- the construction of contrastive examples, the choice of the contrastive loss, and the strategy in decoding. We validate CoNT on five generation tasks with ten benchmarks, including machine translation, summarization, code comment generation, data-to-text generation and commonsense generation. Experimental results show that CoNT clearly outperforms the conventional training framework on all the ten benchmarks with a convincing margin. Especially, CoNT surpasses previous the most competitive contrastive learning method for text generation, by 1.50 BLEU on machine translation and 1.77 ROUGE-1 on summarization, respectively. It achieves new state-of-the-art on summarization, code comment generation (without external data) and data-to-text generation.
translated by 谷歌翻译
Diffusion models have achieved state-of-the-art synthesis quality on visual and audio tasks, and recent works adapt them to textual data by diffusing on the embedding space. But the difference between the continuous data space and the embedding space raises challenges to the diffusion model, which have not been carefully explored. In this paper, we conduct systematic studies and analyze the challenges threefold. Firstly, the data distribution is learnable for embeddings, which may lead to the collapse of the loss function. Secondly, as the norm of embedding varies between popular and rare words, adding the same noise scale will lead to sub-optimal results. In addition, we find that noises sampled from a standard Gaussian distribution may distract the diffusion process. To solve the above challenges, we propose Difformer, a denoising diffusion probabilistic model based on Transformer, which consists of three techniques including utilizing an anchor loss function, a layer normalization module for embeddings, and a norm factor to the Gaussian noise. All techniques are complementary to each other and critical to boosting the model performance together. Experiments are conducted on benchmark datasets over two seminal text generation tasks including machine translation and text summarization. The results show that Difformer significantly outperforms the embedding diffusion baselines, while achieving competitive results with strong autoregressive baselines.
translated by 谷歌翻译
Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement. Their success hinges on the fact that the underlying physical phenomena are continuous. For inherently discrete and categorical data such as language, various diffusion-inspired alternatives have been proposed. However, the continuous nature of diffusion models conveys many benefits, and in this work we endeavour to preserve it. We propose CDCD, a framework for modelling categorical data with diffusion models that are continuous both in time and input space. We demonstrate its efficacy on several language modelling tasks.
translated by 谷歌翻译
神经自回归序列模型涂抹许多可能​​序列之间的概率,包括退化的序列,例如空或重复序列。在这项工作中,我们解决了一个特定的情况,其中模型为不合理的短序列分配高概率。我们定义了量化速率以量化此问题。在确认神经机翻译中高度过度的过天气后,我们建议明确地减少培训期间的过天平率。我们进行一组实验来研究建议的正规化对模型分布和解码性能的影响。我们使用神经电脑翻译任务作为测试用,并考虑三个不同大小的不同数据集。我们的实验显示了三个主要结果。首先,我们可以通过调整正规化的强度来控制模型的过天平率。其次,通过提高过度损失贡献,令牌的概率和等级在不应该是它的位置下降。第三,所提出的正则化影响光束搜索的结果,特别是当使用大梁时。用大梁的翻译质量(在BLEU中测量)的降解显着减少了较低的过天速速率,但与较小光束尺寸相比的劣化仍有剩余状态。从这些观察中,我们得出结论,高度过度的过度性是神经自回归模型中过于可能的短序列的退化情况背后的主要原因。
translated by 谷歌翻译
在几乎所有文本生成应用中,Word序列在左右(L2R)或左右(R2L)方式中构造,因为自然语言句子是写入L2R或R2L。但是,我们发现自然语言书面订单对文本生成至关重要。在本文中,我们提出了一种螺旋语言建模(SLM),这是一种普遍的方法,使人们能够构建超出L2R和R2L订单的自然语言句子。 SLM允许其中一个从结果文本内的任意令牌开始,并在所选的任意令牌中展开REST令牌。它使解码顺序除了语言模型困惑之外的新优化目标,这进一步提高了所生成文本的分集和质量。此外,SLM使得可以通过选择正确的开始令牌来操纵文本构建过程。 SLM还将生成排序引入了额外的正则化,以提高低资源方案中的模型稳健性。 8次广泛研究的神经机翻译(NMT)任务的实验表明,与传统的L2R解码方法相比,SLM高达4.7 BLEU增加。
translated by 谷歌翻译
非自动性变压器(NAT)是文本生成模型的家族,旨在通过并行预测整个句子来减少解码延迟。但是,这种延迟减少牺牲了捕获从左到右的依赖性的能力,从而使NAT学习非常具有挑战性。在本文中,我们介绍了理论和经验分析,以揭示NAT学习的挑战,并提出统一的观点来了解现有的成功。首先,我们表明,简单地通过最大化可能性来训练NAT可以导致边际分布的近似值,但在代币之间降低了所有依赖关系,在该数据集的条件总相关性可以测量删除的信息。其次,我们在统一的框架中正式化了许多以前的目标,并表明他们的成功可以得出结论,以最大程度地提高代理分布的可能性,从而减少了信息损失。实证研究表明,我们的观点可以解释NAT学习中的现象,并指导新培训方法的设计。
translated by 谷歌翻译
Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows through the lens of probabilistic modeling and inference. We place special emphasis on the fundamental principles of flow design, and discuss foundational topics such as expressive power and computational trade-offs. We also broaden the conceptual framing of flows by relating them to more general probability transformations. Lastly, we summarize the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.
translated by 谷歌翻译
DeNoising扩散模型代表了计算机视觉中最新的主题,在生成建模领域表现出了显着的结果。扩散模型是一个基于两个阶段的深层生成模型,一个正向扩散阶段和反向扩散阶段。在正向扩散阶段,通过添加高斯噪声,输入数据在几个步骤中逐渐受到干扰。在反向阶段,模型的任务是通过学习逐步逆转扩散过程来恢复原始输入数据。尽管已知的计算负担,即由于采样过程中涉及的步骤数量,扩散模型对生成样品的质量和多样性得到了广泛赞赏。在这项调查中,我们对视觉中应用的denoising扩散模型的文章进行了全面综述,包括该领域的理论和实际贡献。首先,我们识别并介绍了三个通用扩散建模框架,这些框架基于扩散概率模型,噪声调节得分网络和随机微分方程。我们进一步讨论了扩散模型与其他深层生成模型之间的关系,包括变异自动编码器,生成对抗网络,基于能量的模型,自回归模型和正常流量。然后,我们介绍了计算机视觉中应用的扩散模型的多角度分类。最后,我们说明了扩散模型的当前局限性,并设想了一些有趣的未来研究方向。
translated by 谷歌翻译
基于能量的模型(EBMS)允许极其灵活的概率分布规范。然而,它们不提供从这些分布中获得精确样本的机制。蒙特卡罗技术可以帮助我们获得样品,如果我们可以轻易采用可用的一些建议分布。例如,抑制采样可以提供精确的样本,但由于需要找到上限目标分布的提案分布,通常难以或不可能应用。大致马克洛夫链Monte Carlo采样技术通常更容易设计,利用在不断发展的样本上执行本地编辑的本地提案分布。然而,由于提案分布的本地性质,这些技术可能效率低下,并且不提供对样品质量的估计。在这项工作中,我们提出了一种新的近似采样技术,准拒绝采样(QRS),允许采样效率和采样质量之间进行权衡,同时提供显式收敛界限和诊断。 QRS大写从深度学习模型获得的高质量全球提案分布的可用性。我们展示了QRS采样对具有分布约束和解释生成的受控文本生成任务的分离EBMS对文本的有效性。我们表明,我们可以以采样效率的成本,从这些eBMS采样。
translated by 谷歌翻译
基于能量的模型(EBMS)最近成功地代表了少量图像的复杂分布。然而,对它们的抽样需要昂贵的马尔可夫链蒙特卡罗(MCMC)迭代在高维像素空间中缓慢混合。与EBMS不同,变形AutoEncoders(VAES)快速生成样本,并配备潜在的空间,使得数据歧管的快速遍历。然而,VAE倾向于将高概率密度分配到实际数据分布之外的数据空间中的区域,并且经常在产生清晰图像时失败。在本文中,我们提出了VAE的一个共生组成和ebm的vaebm,提供了两个世界的eBM。 VAEBM使用最先进的VAE捕获数据分布的整体模式结构,它依赖于其EBM组件,以明确地从模型中排除非数据样区域并优化图像样本。此外,VAEBM中的VAE组件允许我们通过在VAE的潜空间中重新处理它们来加速MCMC更新。我们的实验结果表明,VAEBM在几个基准图像数据集上以大量边距开辟了最先进的VAES和EBMS。它可以产生高于256 $ \倍的高质量图像,使用短MCMC链。我们还证明了VAEBM提供了完整的模式覆盖范围,并在分配外检测中表现良好。源代码可在https://github.com/nvlabs/vaebm上获得
translated by 谷歌翻译
The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, auto-encoders, manifold learning, and deep networks. This motivates longer-term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning.
translated by 谷歌翻译
Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that are more natural and better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the lower level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks which may require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize CTG techniques from the perspective of PLMs. We hope it can help researchers in related fields to quickly track the academic frontier, providing them with a landscape of the area and a roadmap for future research.
translated by 谷歌翻译
Transfer learning, where a model is first pre-trained on a data-rich task before being finetuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.
translated by 谷歌翻译
Attention-based autoregressive models have achieved state-of-the-art performance in various sequence-to-sequence tasks, including Text-To-Speech (TTS) and Neural Machine Translation (NMT), but can be difficult to train. The standard training approach, teacher forcing, guides a model with the reference back-history. During inference, the generated back-history must be used. This mismatch limits the evaluation performance. Attention forcing has been introduced to address the mismatch, guiding the model with the generated back-history and reference attention. While successful in tasks with continuous outputs like TTS, attention forcing faces additional challenges in tasks with discrete outputs like NMT. This paper introduces the two extensions of attention forcing to tackle these challenges. (1) Scheduled attention forcing automatically turns attention forcing on and off, which is essential for tasks with discrete outputs. (2) Parallel attention forcing makes training parallel, and is applicable to Transformer-based models. The experiments show that the proposed approaches improve the performance of models based on RNNs and Transformers.
translated by 谷歌翻译
For sequence generation, both autoregressive models and non-autoregressive models have been developed in recent years. Autoregressive models can achieve high generation quality, but the sequential decoding scheme causes slow decoding speed. Non-autoregressive models accelerate the inference speed with parallel decoding, while their generation quality still needs to be improved due to the difficulty of modeling multi-modalities in data. To address the multi-modality issue, we propose Diff-Glat, a non-autoregressive model featured with a modality diffusion process and residual glancing training. The modality diffusion process decomposes the modalities and reduces the modalities to learn for each transition. And the residual glancing sampling further smooths the modality learning procedures. Experiments demonstrate that, without using knowledge distillation data, Diff-Glat can achieve superior performance in both decoding efficiency and accuracy compared with the autoregressive Transformer.
translated by 谷歌翻译
Large pretrained language models generate fluent text but are notoriously hard to controllably sample from. In this work, we study constrained sampling from such language models: generating text that satisfies user-defined constraints, while maintaining fluency and the model's performance in a downstream task. We propose MuCoLa -- a sampling procedure that combines the log-likelihood of the language model with arbitrary (differentiable) constraints in a single energy function, and then generates samples in a non-autoregressive manner. Specifically, it initializes the entire output sequence with noise and follows a Markov chain defined by Langevin Dynamics using the gradients of the energy function. We evaluate MuCoLa on text generation with soft and hard constraints as well as their combinations obtaining significant improvements over competitive baselines for toxicity avoidance, sentiment control, and keyword-guided generation.
translated by 谷歌翻译
Generative AI has matured to a point where large-scale models can generate text that seems indistinguishable from human-written text and remarkably photorealistic images. Automatically measuring how close the distribution of generated data is to the target real data distribution is a key step in diagnosing existing models and developing better models. We present MAUVE, a family of comparison measures between pairs of distributions such as those encountered in the generative modeling of text or images. These scores are statistical summaries of divergence frontiers capturing two types of errors in generative modeling. We explore four approaches to statistically estimate these scores: vector quantization, non-parametric estimation, classifier-based estimation, and parametric Gaussian approximations. We provide statistical bounds for the vector quantization approach. Empirically, we find that the proposed scores paired with a range of $f$-divergences and statistical estimation methods can quantify the gaps between the distributions of human-written text and those of modern neural language models by correlating with human judgments and identifying known properties of the generated texts. We conclude the paper by demonstrating its applications to other AI domains and discussing practical recommendations.
translated by 谷歌翻译
神经文本生成模型可能遭受低多样性问题。已经提出了各种解码策略和基于培训的方法仅通过利用上下文特征来促进多样性,但很少他们考虑纳入句法结构线索。在这项工作中,我们建议使用语言注释,即演讲(POS),来指导文本生成。详细地,我们将POS引导SoftMax介绍以显式模拟两个后部概率:(i)下一页POS,(ii)来自目标POS的词汇的下一个令牌。进一步提出POS导游采样策略来解决POS的多样性来解决低多样性问题。广泛的实验和人类评估表明,与现有最先进的方法相比,我们的POS引导的Softmax和采样(POSG)可以在保持相当的质量的同时产生更多样化的文本。
translated by 谷歌翻译