基于变压器的模型在多个领域和任务上显示了它们的有效性。自我注意力允许将所有序列元素的信息结合到上下文感知表示形式中。但是，全球和本地信息必须主要存储在相同的元素表示中。此外，输入序列的长度受到自我注意的二次计算复杂性的限制。在这项工作中，我们提出并研究了一个记忆启动的片段级循环变压器（复发记忆变压器）。内存允许借助复发的帮助存储和处理本地和全局信息，并可以在长序列的段之间传递信息。我们通过将特殊的内存令牌添加到输入或输出序列中，实现了一个内存机制，无需更改变压器模型。然后，对变压器进行了训练，以控制内存操作和序列表示处理。实验的结果表明，我们的模型与Transformer-XL在语言建模上的较小内存大小上的表现相同，并在需要更长序列处理的任务方面胜过它。我们证明，将内存令牌添加到TR-XL可以提高IT性能。这使得反复的内存变压器成为需要学习长期依赖性和内存处理中的通用性（例如算法任务和推理）的应用程序的有前途的体系结构。

我们介绍了块状变压器，该变压器以序列的反复方式应用变压器层，并且相对于序列长度具有线性复杂性。我们的复发单元在训练过程中在代币的块而不是单个令牌上运行，并利用块内并行计算，以便有效利用加速器硬件。单元本身非常简单。它仅仅是一个变压器层：它使用自我注意事项和交叉注意力来有效计算大量状态向量和令牌上的复发函数。我们的设计部分受到LSTM单元的启发，它使用LSTM风格的大门，但它可以将典型的LSTM单元缩放为几个数量级。我们的复发实现在计算时间和参数计数中都具有相同的成本作为传统的变压器层，但是在很长的序列中，语言建模任务中的语言建模任务的困惑极大地改善了。我们的模型比远程变压器XL基线的表现宽大，同时运行的速度是两倍。我们证明了它在PG19（书籍），Arxiv论文和GitHub源代码上的有效性。我们的代码已发布为开​​源。

现实世界中的数据是高维的：即使在压缩后，书籍，图像或音乐表演也很容易包含数十万个元素。但是，最常用的自回归模型，变压器非常昂贵，以缩放捕获这种远程结构所需的输入和层数。我们开发了感知者AR，这是一种自回归的模态 - 不合骨架构，它使用交叉注意力将远程输入映射到少数潜在的潜在，同时还可以维护端到端的因果关系掩盖。感知器AR可以直接进行十万个令牌，从而实现了实用的长篇小写密度估计，而无需手工制作的稀疏模式或记忆机制。当对图像或音乐进行培训时，感知器AR会生成具有清晰长期连贯性和结构的输出。我们的架构还获得了长期基准测试的最新可能性，包括64 x 64个Imagenet图像和PG-19书籍。

变形金刚在参加长语境时奋斗，因为计算量随着上下文长度而增长，因此它们不能有效地模拟长期存储器。已经提出了几种变体来缓解这个问题，但它们都有有限的内存容量，被迫降低旧信息。在本文中，我们提出了$ \ infty $ -former，它将Vanilla变压器与无限的长期记忆延伸。通过利用连续空间注意机制来参加长期内存，$ \ idty $ -former的注意力复杂性与上下文长度无关。因此，它能够在保持固定计算预算的同时进行任意长的上下文并维持“粘性存储器”。合成排序任务的实验展示了$ \ idty $ -former将信息从长序列中保留信息的能力。我们还通过培训从头开始培训模型以及微调预先培训的语言模型来执行语言建模实验，这表明了无限性的长期记忆的好处。

Recent work has improved language models (LMs) remarkably by equipping them with a non-parametric memory component. However, most existing approaches only introduce mem-ories at testing time or represent them using a separately trained encoder, resulting in suboptimal training of the language model. In this work, we present TRIME, a novel yet simple training approach designed for training LMs with memory augmentation. Our approach uses a training objective that directly takes in-batch examples as accessible memory. We also present new methods for memory construction and data batching, which are used for adapting to different sets of memories--local, long-term, and external memory--at testing time. We evaluate TRIME on multiple language modeling and machine translation benchmarks and show that it is able to achieve significant improvements across all the settings. Concretely, TRIME reduces the perplexity from 18.70 to 15.37 on WIKITEXT-103, by effectively leveraging a large memory set from the training corpus. Compared to standard LM training, TRIME adds negligible computational overhead and is compatible with different neural architectures, making it a versatile solution for training memory-augmented LMs.

状态空间模型已显示在建模远距离依赖性方面有效，特别是序列分类任务。在这项工作中，我们着重于对英语书籍，GitHub源代码和Arxiv数学文章的自回旋序列建模。基于围绕封闭激活功能的有效性的最新发展，我们提出了一个名为“封闭状态空间（GSS）”的新层，并表明它的训练速度明显快于TPU的S4（即DSS）的对角线版本，具有相当竞争力 - 基于变压器的基线，并表现出零击向更长的输入，同时直接实施。最后，我们表明，利用自我意见来建模局部依赖性，可以进一步提高GSS的性能。

Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been shown to achieve superior results on some sequence modeling tasks such as machine translation, with the added advantage that they concurrently process all inputs in the sequence, leading to easy parallelization and faster training times. Despite these successes, however, popular feed-forward sequence models like the Transformer fail to generalize in many simple tasks that recurrent models handle with ease, e.g. copying strings or even simple logical inference when the string or formula lengths exceed those observed at training time. We propose the Universal Transformer (UT), a parallel-in-time self-attentive recurrent sequence model which can be cast as a generalization of the Transformer model and which addresses these issues. UTs combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs. We also add a dynamic per-position halting mechanism and find that it improves accuracy on several tasks. In contrast to the standard Transformer, under certain assumptions UTs can be shown to be Turing-complete. Our experiments show that UTs outperform standard Transformers on a wide range of algorithmic and language understanding tasks, including the challenging LAMBADA language modeling task where UTs achieve a new state of the art, and machine translation where UTs achieve a 0.9 BLEU improvement over Transformers on the WMT14 En-De dataset. * Equal contribution, alphabetically by last name. † Work performed while at Google Brain.

变压器注意机制的二次计算和内存复杂性限制了对长序列建模的可扩展性。在本文中，我们提出了Luna，一种线性统一嵌套关注机制，使Softmax注意力具有两个嵌套线性关注功能，仅产生线性（与二次）的时间和空间复杂度相反。具体地，通过第一注意功能，LUNA将输入序列包装成固定长度的序列。然后，使用第二关注功能未包装包装序列。与更传统的关注机制相比，LUNA引入具有固定长度的附加序列作为输入和额外的相应输出，允许LUNA线性地进行关注操作，同时还存储足够的上下文信息。我们对三个序列建模任务的基准进行了广泛的评估：长上下文序列建模，神经机平移和大型预磨损的屏蔽语言建模。竞争甚至更好的实验结果表明了Luna的有效性和效率与各种各样相比

Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This paper proposes a systematic and unified benchmark, Long-Range Arena, specifically focused on evaluating model quality under long-context scenarios. Our benchmark is a suite of tasks consisting of sequences ranging from 1K to 16K tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic images, and mathematical expressions requiring similarity, structural, and visual-spatial reasoning. We systematically evaluate ten well-established long-range Transformer models (Reformers, Linformers, Linear Transformers, Sinkhorn Transformers, Performers, Synthesizers, Sparse Transformers, and Longformers) on our newly proposed benchmark suite. Long-Range Arena paves the way towards better understanding this class of efficient Transformer models, facilitates more research in this direction, and presents new challenging tasks to tackle. Our benchmark code will be released at https://github.com/google-research/long-range-arena.

变压器编码器模型在对话建模中显示出令人印象深刻的性能。但是，由于变压器在处理长序列方面效率低下，对话历史的长度通常需要被截断。为了解决此问题，我们提出了一种新的内存启动变压器，该变压器与现有的预训练编码器模型兼容，并可以有效地保存历史记录信息。它将单独的内存模块与预训练的变压器一起结合在一起，以在内存状态和当前输入上下文之间有效互换信息。我们在三个对话数据集和两个语言建模数据集上评估我们的模型。实验结果表明，与其他预训练的变压器基线相比，我们的方法已经达到了较高的效率和性能。

Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from O N 2 to O (N ), where N is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.

Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism. To remedy this, we propose, BIGBIRD, a sparse attention mechanism that reduces this quadratic dependency to linear. We show that BIGBIRD is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our theoretical analysis reveals some of the benefits of having O(1) global tokens (such as CLS), that attend to the entire sequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to 8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context, BIGBIRD drastically improves performance on various NLP tasks such as question answering and summarization. We also propose novel applications to genomics data.

Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. Second, an enhanced mask decoder is used to incorporate absolute positions in the decoding layer to predict the masked tokens in model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understand (NLU) and natural langauge generation (NLG) downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). Notably, we scale up DeBERTa by training a larger version that consists of 48 Transform layers with 1.5 billion parameters. The significant performance boost makes the single DeBERTa model surpass the human performance on the SuperGLUE benchmark (Wang et al., 2019a) for the first time in terms of macro-average score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of January 6, 2021, outperforming the human baseline by a decent margin (90.3 versus 89.8). The pre-trained DeBERTa models and the source code were released at: https://github.com/microsoft/DeBERTa 1 .

Dynamic evaluation of language models (LMs) adapts model parameters at test time using gradient information from previous tokens and substantially improves LM performance. However, it requires over 3x more compute than standard inference. We present Fast Weight Layers (FWLs), a neural component that provides the benefits of dynamic evaluation much more efficiently by expressing gradient updates as linear attention. A key improvement over dynamic evaluation is that FWLs can also be applied at training time so the model learns to make good use of gradient updates. FWLs can easily be added on top of existing transformer models, require relatively little extra compute or memory to run, and significantly improve language modeling perplexity.

Transformer models have achieved superior performance in various natural language processing tasks. However, the quadratic computational cost of the attention mechanism limits its practicality for long sequences. There are existing attention variants that improve the computational efficiency, but they have limited ability to effectively compute global information. In parallel to Transformer models, state space models (SSMs) are tailored for long sequences, but they are not flexible enough to capture complicated local information. We propose SPADE, short for $\underline{\textbf{S}}$tate s$\underline{\textbf{P}}$ace $\underline{\textbf{A}}$ugmente$\underline{\textbf{D}}$ Transform$\underline{\textbf{E}}$r. Specifically, we augment a SSM into the bottom layer of SPADE, and we employ efficient local attention methods for the other layers. The SSM augments global information, which complements the lack of long-range dependency issue in local attention methods. Experimental results on the Long Range Arena benchmark and language modeling tasks demonstrate the effectiveness of the proposed method. To further demonstrate the scalability of SPADE, we pre-train large encoder-decoder models and present fine-tuning results on natural language understanding and natural language generation tasks.

Transfer learning, where a model is first pre-trained on a data-rich task before being finetuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

变压器注意机制中的设计选择，包括弱电感偏置和二次计算复杂性，限制了其用于建模长序列的应用。在本文中，我们介绍了一个简单的，理论上的，单头的门控注意机制，配备了（指数）移动平均线，以将局部依赖性的电感偏置纳入位置 - 敏锐的注意机制中。我们进一步提出了一个具有线性时间和空间复杂性的大型变体，但通过将整个序列分为固定长度的多个块，仅产生最小的质量损失。对广泛的序列建模基准测试的广泛实验，包括远距离竞技场，神经机器翻译，自动回归语言建模以及图像和语音分类，表明，巨人比其他序列模型取得了重大改进，包括变种物的变体和最新的变体模型状态空间模型。

变形金刚在语言和视觉域中取得了成功。然而，将它们缩放到长期序列（例如长）或高分辨率图像，因为自我关注机构相对于输入序列长度具有二次时间和存储器复杂性。在本文中，我们提出了长短变压器（变压器-LS），是一种有效的自我关注机制，用于对语言和视觉任务进行线性复杂性建模的长序列。它用动态投影聚集了一种新的远程关注，以模拟远处相关性和短期注意，以捕获细粒度的局部相关性。我们提出了双重正径策略，以解释两个注意机制之间的规模不匹配。变压器-LS可以应用于自回归和双向模型，而无需额外复杂。我们的方法在语言和视觉域中的多个任务中优于最先进的模型，包括远程竞技场基准，自回归语言建模和想象成分类。例如，变换器-LS使用比以前的方法的一半在eNWIK8上实现0.97测试BPC，同时与其在同一硬件上的全部关注版本相比，可以更快地处理3倍。在Imagenet上，它可以获得最先进的结果（例如，适度大小的55.8M模型，仅在224x224 Imagenet-1K上培训，可以获得顶级1精度84.1％），同时在高分辨率上更加可扩展图片。源代码和模型在https://github.com/nvidia/transformer-ls上发布。

Causal transformer language models (LMs), such as GPT-3, typically require some form of positional encoding, such as positional embeddings. However, we show that LMs without any explicit positional encoding are still competitive with standard models, and that this phenomenon is robust across different datasets, model sizes, and sequence lengths. Probing experiments reveal that such models acquire an implicit notion of absolute positions throughout the network, effectively compensating for the missing information. We conjecture that causal attention enables the model to infer the number of predecessors that each token can attend to, thereby approximating its absolute position. Our findings indicate that causal LMs might derive positional awareness not only from the explicit positioning mechanism, but also from the effects of the causal mask.

我们在变压器中重新审视设计选择，并提出方法来解决它们在处理长序列中的弱点。首先，我们提出了一个名为“门控注意单元”的简单层，该层允许使用较弱的单头注意，而质量损失最小。然后，我们提出了一种与该新层的线性近似方法互补的，该方法对加速器友好且质量高度竞争。最终的型号（名为Flash）与短（512）和长（8K）上下文长度相匹配，在WIKI-40B上达到高达4.9 $ \ times $的训练速度和PG上的12.1 $ \ times $，在PG上达到了4.9 $ \ times $的困惑。-19用于自动回归语言建模，C4的4.8 $ \ times $用于掩盖语言建模。