For sequence generation, both autoregressive models and non-autoregressive models have been developed in recent years. Autoregressive models can achieve high generation quality, but the sequential decoding scheme causes slow decoding speed. Non-autoregressive models accelerate the inference speed with parallel decoding, while their generation quality still needs to be improved due to the difficulty of modeling multi-modalities in data. To address the multi-modality issue, we propose Diff-Glat, a non-autoregressive model featured with a modality diffusion process and residual glancing training. The modality diffusion process decomposes the modalities and reduces the modalities to learn for each transition. And the residual glancing sampling further smooths the modality learning procedures. Experiments demonstrate that, without using knowledge distillation data, Diff-Glat can achieve superior performance in both decoding efficiency and accuracy compared with the autoregressive Transformer.

Diffusion models have achieved state-of-the-art synthesis quality on visual and audio tasks, and recent works adapt them to textual data by diffusing on the embedding space. But the difference between the continuous data space and the embedding space raises challenges to the diffusion model, which have not been carefully explored. In this paper, we conduct systematic studies and analyze the challenges threefold. Firstly, the data distribution is learnable for embeddings, which may lead to the collapse of the loss function. Secondly, as the norm of embedding varies between popular and rare words, adding the same noise scale will lead to sub-optimal results. In addition, we find that noises sampled from a standard Gaussian distribution may distract the diffusion process. To solve the above challenges, we propose Difformer, a denoising diffusion probabilistic model based on Transformer, which consists of three techniques including utilizing an anchor loss function, a layer normalization module for embeddings, and a norm factor to the Gaussian noise. All techniques are complementary to each other and critical to boosting the model performance together. Experiments are conducted on benchmark datasets over two seminal text generation tasks including machine translation and text summarization. The results show that Difformer significantly outperforms the embedding diffusion baselines, while achieving competitive results with strong autoregressive baselines.

We present DiffusionBERT, a new generative masked language model based on discrete diffusion models. Diffusion models and many pre-trained language models have a shared training objective, i.e., denoising, making it possible to combine the two powerful models and enjoy the best of both worlds. On the one hand, diffusion models offer a promising training strategy that helps improve the generation quality. On the other hand, pre-trained denoising language models (e.g., BERT) can be used as a good initialization that accelerates convergence. We explore training BERT to learn the reverse process of a discrete diffusion process with an absorbing state and elucidate several designs to improve it. First, we propose a new noise schedule for the forward diffusion process that controls the degree of noise added at each step based on the information of each token. Second, we investigate several designs of incorporating the time step into BERT. Experiments on unconditional text generation demonstrate that DiffusionBERT achieves significant improvement over existing diffusion models for text (e.g., D3PM and Diffusion-LM) and previous generative masked language models in terms of perplexity and BLEU score.

在本文中，我们提出了一种新的生成模型，逐步逐步的去噪AutoEncoder（Sundae），不依赖于自回归模型。类似地与去噪扩散技术，在从随机输入开始并从随机输入开始并每次直到收敛改善它们时，日出施加Sundae。我们提出了一个简单的新改进运算符，它比扩散方法更少迭代，同时在定性地在自然语言数据集上产生更好的样本。Sundae在WMT'14英语到德语翻译任务上实现最先进的结果（非自回归方法），在巨大清洁的常见爬网数据集和Python代码的数据集上对无条件语言建模的良好定性结果来自GitHub。通过在模板中填充任意空白模式，Sundae的非自动增加性质开辟了超出左右提示的可能性。

Diffusion model, a new generative modelling paradigm, has achieved great success in image, audio, and video generation. However, considering the discrete categorical nature of text, it is not trivial to extend continuous diffusion models to natural language, and text diffusion models are less studied. Sequence-to-sequence text generation is one of the essential natural language processing topics. In this work, we apply diffusion models to approach sequence-to-sequence text generation, and explore whether the superiority generation performance of diffusion model can transfer to natural language domain. We propose SeqDiffuSeq, a text diffusion model for sequence-to-sequence generation. SeqDiffuSeq uses an encoder-decoder Transformers architecture to model denoising function. In order to improve generation quality, SeqDiffuSeq combines the self-conditioning technique and a newly proposed adaptive noise schedule technique. The adaptive noise schedule has the difficulty of denoising evenly distributed across time steps, and considers exclusive noise schedules for tokens at different positional order. Experiment results illustrate the good performance on sequence-to-sequence generation in terms of text quality and inference time.

非自动性变压器（NAT）是文本生成模型的家族，旨在通过并行预测整个句子来减少解码延迟。但是，这种延迟减少牺牲了捕获从左到右的依赖性的能力，从而使NAT学习非常具有挑战性。在本文中，我们介绍了理论和经验分析，以揭示NAT学习的挑战，并提出统一的观点来了解现有的成功。首先，我们表明，简单地通过最大化可能性来训练NAT可以导致边际分布的近似值，但在代币之间降低了所有依赖关系，在该数据集的条件总相关性可以测量删除的信息。其次，我们在统一的框架中正式化了许多以前的目标，并表明他们的成功可以得出结论，以最大程度地提高代理分布的可能性，从而减少了信息损失。实证研究表明，我们的观点可以解释NAT学习中的现象，并指导新培训方法的设计。

Recently, non-autoregressive (NAR) neural machine translation models have received increasing attention due to their efficient parallel decoding. However, the probabilistic framework of NAR models necessitates conditional independence assumption on target sequences, falling short of characterizing human language data. This drawback results in less informative learning signals for NAR models under conventional MLE training, thereby yielding unsatisfactory accuracy compared to their autoregressive (AR) counterparts. In this paper, we propose a simple and model-agnostic multi-task learning framework to provide more informative learning signals. During training stage, we introduce a set of sufficiently weak AR decoders that solely rely on the information provided by NAR decoder to make prediction, forcing the NAR decoder to become stronger or else it will be unable to support its weak AR partners. Experiments on WMT and IWSLT datasets show that our approach can consistently improve accuracy of multiple NAR baselines without adding any additional decoding overhead.

Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement. Their success hinges on the fact that the underlying physical phenomena are continuous. For inherently discrete and categorical data such as language, various diffusion-inspired alternatives have been proposed. However, the continuous nature of diffusion models conveys many benefits, and in this work we endeavour to preserve it. We propose CDCD, a framework for modelling categorical data with diffusion models that are continuous both in time and input space. We demonstrate its efficacy on several language modelling tasks.

产生人类想要的声音效果是一个重要的话题。但是，在这一领域，很少有研究声音发电。在这项研究中，我们调查了以文本提示为条件的声音，并提出了一个新型的文本对生成框架，该框架由文本编码器组成，矢量量化了变异自动编码器（VQ-VAE），解码器和歌手。该框架首先使用解码器将从文本编码器提取的文本特征传递到借助VQ-VAE的MEL光谱图中，然后使用Vocoder将生成的MEL光谱图转换为波形。我们发现，解码器显着影响发电性能。因此，我们专注于在这项研究中设计一个好的解码器。我们从传统的自动回解码器开始，该解码器已被证明是以前的Sound Generation Works中的最先进方法。但是，AR解码器始终按顺序预测MEL-SPECTROGIN图令牌，这引入了单向偏见和错误问题的积累。此外，使用AR解码器，声音生成时间随着声音持续时间线性增加。为了克服AR解码器引入的缺点，我们提出了一个基于离散扩散模型的非自动回形解码器，称为DiffSound。具体而言，DIFFSOUND可以在一个步骤中预测所有MEL光谱图令牌，然后在下一步中完善预测的令牌，因此可以在几个步骤后获得最优于预测的结果。我们的实验表明，与AR解码器相比，我们提出的差异不仅产生更好的文本到单一生成结果，而且还具有更快的生成速度，例如MOS：3.56 \ textit {v.s} 2.786，并且生成速度为五个比AR解码器快的时间。

在这项工作中，我们展示了一种新的神经机翻译方法（NMT），使用去噪扩散概率模型（DDPM），调整了文本数据，在该领域的最近进步之后。我们表明，可以使用在源句子上的扩散模型来无自动增加句子。我们还表明，我们的模型能够在培训期间无奈的语言成对（零拍摄学习）之间翻译。

Non-autoregressive neural machine translation (NAT) models suffer from the multi-modality problem that there may exist multiple possible translations of a source sentence, so the reference sentence may be inappropriate for the training when the NAT output is closer to other translations. In response to this problem, we introduce a rephraser to provide a better training target for NAT by rephrasing the reference sentence according to the NAT output. As we train NAT based on the rephraser output rather than the reference sentence, the rephraser output should fit well with the NAT output and not deviate too far from the reference, which can be quantified as reward functions and optimized by reinforcement learning. Experiments on major WMT benchmarks and NAT baselines show that our approach consistently improves the translation quality of NAT. Specifically, our best variant achieves comparable performance to the autoregressive Transformer, while being 14.7 times more efficient in inference.

The image captioning task is typically realized by an auto-regressive method that decodes the text tokens one by one. We present a diffusion-based captioning model, dubbed the name DDCap, to allow more decoding flexibility. Unlike image generation, where the output is continuous and redundant with a fixed length, texts in image captions are categorical and short with varied lengths. Therefore, naively applying the discrete diffusion model to text decoding does not work well, as shown in our experiments. To address the performance gap, we propose several key techniques including best-first inference, concentrated attention mask, text length prediction, and image-free training. On COCO without additional caption pre-training, it achieves a CIDEr score of 117.8, which is +5.0 higher than the auto-regressive baseline with the same architecture in the controlled setting. It also performs +26.8 higher CIDEr score than the auto-regressive baseline (230.3 v.s.203.5) on a caption infilling task. With 4M vision-language pre-training images and the base-sized model, we reach a CIDEr score of 125.1 on COCO, which is competitive to the best well-developed auto-regressive frameworks. The code is available at https://github.com/buxiangzhiren/DDCap.

Can continuous diffusion models bring the same performance breakthrough on natural language they did for image generation? To circumvent the discrete nature of text data, we can simply project tokens in a continuous space of embeddings, as is standard in language modeling. We propose Self-conditioned Embedding Diffusion, a continuous diffusion mechanism that operates on token embeddings and allows to learn flexible and scalable diffusion models for both conditional and unconditional text generation. Through qualitative and quantitative evaluation, we show that our text diffusion models generate samples comparable with those produced by standard autoregressive language models - while being in theory more efficient on accelerator hardware at inference time. Our work paves the way for scaling up diffusion models for text, similarly to autoregressive models, and for improving performance with recent refinements to continuous diffusion.

自回归（AR）和非自动增加（NAR）模型对性能和延迟具有自己的优势，将它们与一个模型相结合，可能会利用两者。目前的组合框架更多地关注多个解码范例的集成，具有统一的生成模型，例如，屏蔽语言模型。然而，由于训练目标和推理之间的差距，概括可能对性能有害。在本文中，我们的目标是通过在统一框架下保留AR和NAR的原始目标来缩小差距。具体地，我们通过将AR和NAR共同建模（左右，左右和直）与新引入的方向变量来提出定向变压器（Diformer），这通过控制每个的预测令牌在那方面有特定的依赖关系。通过方向实现的统一成功地保留了AR和NAR中使用的原始依赖性假设，保留了泛化和性能。 4 WMT基准测试的实验表明，Diformer优于当前的联合建模工作，适用于AR和NAR解码的1.5个以上的BLEU积分，也对最先进的独立AR和NAR模型具有竞争力。

The recently developed discrete diffusion models perform extraordinarily well in the text-to-image task, showing significant promise for handling the multi-modality signals. In this work, we harness these traits and present a unified multimodal generation model that can conduct both the "modality translation" and "multi-modality generation" tasks using a single model, performing text-based, image-based, and even vision-language simultaneous generation. Specifically, we unify the discrete diffusion process for multimodal signals by proposing a unified transition matrix. Moreover, we design a mutual attention module with fused embedding layer and a unified objective function to emphasise the inter-modal linkages, which are vital for multi-modality generation. Extensive experiments indicate that our proposed method can perform comparably to the state-of-the-art solutions in various generation tasks.

手语制作（SLP）旨在将口语语言自动转化为符号序列。 SLP的核心过程是将符号光泽序列转换为其相应的标志姿势序列（G2P）。大多数现有的G2P模型通常以自回归方式执行这种条件的远程生成，这不可避免地导致错误的积累。为了解决这个问题，我们提出了一种量化量子序列序列的生成的矢量量化扩散方法，称为poseVQ扩散，这是一种迭代性非自动入学方法。具体而言，我们首先引入量化量化变量自动编码器（姿势VQVAE）模型，以表示姿势序列作为一系列潜在代码。然后，我们通过最近开发的扩散体系结构的扩展来对潜在离散空间进行建模。为了更好地利用时空信息，我们介绍了一种新颖的体系结构，即CodeUnet，以在离散空间中生成更高质量的姿势序列。此外，利用学习的代码，我们开发了一种新型的顺序k-nearest-neighbours方法，以预测相应的光泽序列的姿势序列的可变长度。因此，与自回旋G2P模型相比，我们的模型具有更快的采样速度，并产生明显更好的结果。与以前的非自动入学G2P方法相比，PoseVQ扩散通过迭代改进改善了预测的结果，从而在SLP评估基准上获得了最新的结果。

我们介绍了文本到图像生成的矢量量化扩散（VQ-扩散）模型。该方法基于矢量量化变分性AutoEncoder（VQ-VAE），其潜像通过最近开发的去噪扩散概率（DDPM）的条件变体为基础。我们发现这种潜在空间方法非常适合于图像到图像生成任务，因为它不仅消除了具有现有方法的单向偏差，还允许我们结合掩模和更换的扩散策略，以避免积累错误，这是现有方法的严重问题。我们的实验表明，与具有类似数量的参数数量的传统自回归（AR）模型相比，VQ扩散产生明显更好的文本到图像生成结果。与以前的基于GAN的文本到图像方法相比，我们的VQ扩散可以通过大边缘处理更复杂的场景并提高合成的图像质量。最后，我们表明我们的方法中的图像生成计算可以通过Reparameter化进行高效。利用传统的AR方法，文本到图像生成时间随输出图像分辨率线性增加，因此即使对于正常尺寸图像也是相当耗时的。 VQ-扩散使我们能够在质量和速度之间实现更好的权衡。我们的实验表明，具有Reparameterization的VQ扩散模型比传统的AR方法快15倍，同时实现更好的图像质量。

Diffusion models have achieved great success in modeling continuous data modalities such as images, audio, and video, but have seen limited use in discrete domains such as language. Recent attempts to adapt diffusion to language have presented diffusion as an alternative to autoregressive language generation. We instead view diffusion as a complementary method that can augment the generative capabilities of existing pre-trained language models. We demonstrate that continuous diffusion models can be learned in the latent space of a pre-trained encoder-decoder model, enabling us to sample continuous latent representations that can be decoded into natural language with the pre-trained decoder. We show that our latent diffusion models are more effective at sampling novel text from data distributions than a strong autoregressive baseline and also enable controllable generation.

非自动进取的生成变压器最近表现出令人印象深刻的图像产生性能，并且比自动回归对应物更快。但是，从视觉令牌的真实关节分布中进行的最佳并行采样仍然是一个开放的挑战。在本文中，我们介绍了代币批评，这是一种辅助模型，用于指导非自动性生成变压器的采样。鉴于掩盖和重建的真实图像，对代币批判性模型进行了训练，以区分哪种视觉令牌属于原始图像，哪些是由生成变压器采样的。在非自动回归迭代采样过程中，令牌批评者用于选择要接受的代币以及拒绝和重新取样的代币。再加上最先进的生成变压器令牌 - 批判性可显着提高其性能，并且在挑战性的课堂条件化成像生成中，就产生的图像质量和多样性之间的权衡取舍了最近的扩散模型和gan 。

扩散概率模型采用前向马尔可夫扩散链逐渐将数据映射到噪声分布，学习如何通过推断一个反向马尔可夫扩散链来生成数据以颠倒正向扩散过程。为了实现竞争性数据生成性能，他们需要一条长长的扩散链，这使它们在培训中不仅在培训中而且发电。为了显着提高计算效率，我们建议通过废除将数据扩散到随机噪声的要求来截断正向扩散链。因此，我们从隐式生成分布而不是随机噪声启动逆扩散链，并通过将其与截断的正向扩散链损坏的数据的分布相匹配来学习其参数。实验结果表明，就发电性能和所需的逆扩散步骤的数量而言，我们的截短扩散概率模型对未截断的概率模型提供了一致的改进。