Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O(L 2 ) to O(L log L), where L is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of N times, where N is the number of layers. The resulting model, the Reformer, performs on par with Transformer models while being much more memory-efficient and much faster on long sequences.

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 Englishto-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.0 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. * Equal contribution. Listing order is random. Jakob proposed replacing RNNs with self-attention and started the effort to evaluate this idea. Ashish, with Illia, designed and implemented the first Transformer models and has been crucially involved in every aspect of this work. Noam proposed scaled dot-product attention, multi-head attention and the parameter-free position representation and became the other person involved in nearly every detail. Niki designed, implemented, tuned and evaluated countless model variants in our original codebase and tensor2tensor. Llion also experimented with novel model variants, was responsible for our initial codebase, and efficient inference and visualizations. Lukasz and Aidan spent countless long days designing various parts of and implementing tensor2tensor, replacing our earlier codebase, greatly improving results and massively accelerating our research.† Work performed while at Google Brain.‡ Work performed while at Google Research.

Self-attention has recently been adopted for a wide range of sequence modeling problems. Despite its effectiveness, self-attention suffers from quadratic compute and memory requirements with respect to sequence length. Successful approaches to reduce this complexity focused on attending to local sliding windows or a small set of locations independent of content. Our work proposes to learn dynamic sparse attention patterns that avoid allocating computation and memory to attend to content unrelated to the query of interest. This work builds upon two lines of research: it combines the modeling flexibility of prior work on content-based sparse attention with the efficiency gains from approaches based on local, temporal sparse attention. Our model, the Routing Transformer, endows selfattention with a sparse routing module based on online k-means while reducing the overall complexity of attention to O(n 1.5 d) from O(n 2 d) for sequence length n and hidden dimension d. We show that our model outperforms comparable sparse attention models on language modeling on Wikitext-103 (15.8 vs 18.3 perplexity), as well as on image generation on ImageNet-64 (3.43 vs 3.44 bits/dim) while using fewer self-attention layers. Additionally, we set a new state-of-the-art on the newly released PG-19 data-set, obtaining a test perplexity of 33.2 with a 22 layer Routing Transformer model trained on sequences of length 8192. We open-source the code for Routing Transformer in Tensorflow. *

现实世界中的数据是高维的：即使在压缩后，书籍，图像或音乐表演也很容易包含数十万个元素。但是，最常用的自回归模型，变压器非常昂贵，以缩放捕获这种远程结构所需的输入和层数。我们开发了感知者AR，这是一种自回归的模态 - 不合骨架构，它使用交叉注意力将远程输入映射到少数潜在的潜在，同时还可以维护端到端的因果关系掩盖。感知器AR可以直接进行十万个令牌，从而实现了实用的长篇小写密度估计，而无需手工制作的稀疏模式或记忆机制。当对图像或音乐进行培训时，感知器AR会生成具有清晰长期连贯性和结构的输出。我们的架构还获得了长期基准测试的最新可能性，包括64 x 64个Imagenet图像和PG-19书籍。

基于变压器的模型广泛用于自然语言处理（NLP）。变压器模型的核心是自我关注机制，它捕获了输入序列中的令牌对的相互作用，并在序列长度上逐步取决于逐行。在更长的序列上培训此类模型是昂贵的。在本文中，我们表明，基于局部敏感散列（LSH）的伯努利采样注意机制降低了这种模型到线性的二次复杂性。我们通过考虑自我关注作为与Bernoulli随机变量相关的单独令牌的总和来绕过二次成本，原则上可以通过单个哈希进行一次（尽管在实践中，这个数字可能是一个小常数）。这导致了有效的采样方案来估算依赖于LSH的特定修改的自我关注（以便在GPU架构上进行部署）。我们在标准512序列长度上评估了胶水基准的算法，在那里我们看到了相对于标准预磨削变压器的良好性能。在远程竞技场（LRA）基准中，为了评估长序列的性能，我们的方法实现了与Softmax自我关注的结果一致，但具有相当大的加速和内存节省，并且通常优于其他有效的自我关注方法。我们的代码可以在https://github.com/mlpen/yoso获得

Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from O N 2 to O (N ), where N is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.

机器翻译历史上的重要突破之一是变压器模型的发展。不仅对于各种翻译任务，而且对于大多数其他NLP任务都是革命性的。在本文中，我们针对一个基于变压器的系统，该系统能够将德语用源句子转换为其英语的对应目标句子。我们对WMT'13数据集的新闻评论德语 - 英语并行句子进行实验。此外，我们研究了来自IWSLT'16数据集的培训中包含其他通用域数据以改善变压器模型性能的效果。我们发现，在培训中包括IWSLT'16数据集，有助于在WMT'13数据集的测试集中获得2个BLEU得分点。引入定性分析以分析通用域数据的使用如何有助于提高产生的翻译句子的质量。

变压器注意机制的二次计算和内存复杂性限制了对长序列建模的可扩展性。在本文中，我们提出了Luna，一种线性统一嵌套关注机制，使Softmax注意力具有两个嵌套线性关注功能，仅产生线性（与二次）的时间和空间复杂度相反。具体地，通过第一注意功能，LUNA将输入序列包装成固定长度的序列。然后，使用第二关注功能未包装包装序列。与更传统的关注机制相比，LUNA引入具有固定长度的附加序列作为输入和额外的相应输出，允许LUNA线性地进行关注操作，同时还存储足够的上下文信息。我们对三个序列建模任务的基准进行了广泛的评估：长上下文序列建模，神经机平移和大型预磨损的屏蔽语言建模。竞争甚至更好的实验结果表明了Luna的有效性和效率与各种各样相比

We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness. To approximate softmax attentionkernels, Performers use a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+), which may be of independent interest for scalable kernel methods. FAVOR+ can also be used to efficiently model kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with other kernels for the first time on large-scale tasks, beyond the reach of regular Transformers, and investigate optimal attention-kernels. Performers are linear architectures fully compatible with regular Transformers and with strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix, uniform convergence and low estimation variance. We tested Performers on a rich set of tasks stretching from pixel-prediction through text models to protein sequence modeling. We demonstrate competitive results with other examined efficient sparse and dense attention methods, showcasing effectiveness of the novel attention-learning paradigm leveraged by Performers.

多头注意力是最先进的变压器背后的推动力，它在各种自然语言处理（NLP）和计算机视觉任务中实现了出色的性能。已经观察到，对于许多应用，这些注意力头会学习冗余嵌入，并且大多数可以在不降低模型性能的情况下去除。受到这一观察的启发，我们提出了变压器的混合物（变压器-MGK）的混合物，这是一种新型的变压器架构，用每个头部的钥匙混合了变压器中的冗余头部。这些键的混合物遵循高斯混合模型，并使每个注意力头有效地集中在输入序列的不同部分上。与传统的变压器对应物相比，变压器-MGK会加速训练和推理，具有较少的参数，并且需要更少的拖船来计算，同时实现跨任务的可比性或更高的准确性。 Transformer-MGK也可以轻松扩展到线性注意力。我们从经验上证明了在一系列实用应用中变形金属MGK的优势，包括语言建模和涉及非常长序列的任务。在Wikitext-103和远程竞技场基准中，具有4个头部的变压器MGK具有与基线变压器具有8个头的可比性或更好的性能。

Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been shown to achieve superior results on some sequence modeling tasks such as machine translation, with the added advantage that they concurrently process all inputs in the sequence, leading to easy parallelization and faster training times. Despite these successes, however, popular feed-forward sequence models like the Transformer fail to generalize in many simple tasks that recurrent models handle with ease, e.g. copying strings or even simple logical inference when the string or formula lengths exceed those observed at training time. We propose the Universal Transformer (UT), a parallel-in-time self-attentive recurrent sequence model which can be cast as a generalization of the Transformer model and which addresses these issues. UTs combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs. We also add a dynamic per-position halting mechanism and find that it improves accuracy on several tasks. In contrast to the standard Transformer, under certain assumptions UTs can be shown to be Turing-complete. Our experiments show that UTs outperform standard Transformers on a wide range of algorithmic and language understanding tasks, including the challenging LAMBADA language modeling task where UTs achieve a new state of the art, and machine translation where UTs achieve a 0.9 BLEU improvement over Transformers on the WMT14 En-De dataset. * Equal contribution, alphabetically by last name. † Work performed while at Google Brain.

变压器注意机制中的设计选择，包括弱电感偏置和二次计算复杂性，限制了其用于建模长序列的应用。在本文中，我们介绍了一个简单的，理论上的，单头的门控注意机制，配备了（指数）移动平均线，以将局部依赖性的电感偏置纳入位置 - 敏锐的注意机制中。我们进一步提出了一个具有线性时间和空间复杂性的大型变体，但通过将整个序列分为固定长度的多个块，仅产生最小的质量损失。对广泛的序列建模基准测试的广泛实验，包括远距离竞技场，神经机器翻译，自动回归语言建模以及图像和语音分类，表明，巨人比其他序列模型取得了重大改进，包括变种物的变体和最新的变体模型状态空间模型。

Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism. To remedy this, we propose, BIGBIRD, a sparse attention mechanism that reduces this quadratic dependency to linear. We show that BIGBIRD is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our theoretical analysis reveals some of the benefits of having O(1) global tokens (such as CLS), that attend to the entire sequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to 8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context, BIGBIRD drastically improves performance on various NLP tasks such as question answering and summarization. We also propose novel applications to genomics data.

由于其二次复杂性，是变压器中的关注模块，其是变压器中的重要组件不能高效地扩展到长序列。许多工作侧重于近似于尺寸的圆点 - 指数的软MAX功能，导致分二次甚至线性复杂性变压器架构。但是，我们表明这些方法不能应用于超出点的指数样式的更强大的注意模块，例如，具有相对位置编码（RPE）的变压器。由于在许多最先进的模型中，相对位置编码被用作默认，设计可以包含RPE的高效变压器是吸引人的。在本文中，我们提出了一种新颖的方法来加速对RPE的转化仪的关注计算在核心化的关注之上。基于观察到相对位置编码形成Toeplitz矩阵，我们数在数学上表明，可以使用快速傅里叶变换（FFT）有效地计算具有RPE的核化注意。使用FFT，我们的方法实现$ \ mathcal {o}（n \ log n）$时间复杂性。有趣的是，我们进一步证明使用相对位置编码适当地可以减轻香草群关注的培训不稳定问题。在广泛的任务上，我们经验证明我们的模型可以从头开始培训，没有任何优化问题。学习模型比许多高效的变压器变体更好地执行，并且在长序列制度中比标准变压器更快。

我们介绍了块状变压器，该变压器以序列的反复方式应用变压器层，并且相对于序列长度具有线性复杂性。我们的复发单元在训练过程中在代币的块而不是单个令牌上运行，并利用块内并行计算，以便有效利用加速器硬件。单元本身非常简单。它仅仅是一个变压器层：它使用自我注意事项和交叉注意力来有效计算大量状态向量和令牌上的复发函数。我们的设计部分受到LSTM单元的启发，它使用LSTM风格的大门，但它可以将典型的LSTM单元缩放为几个数量级。我们的复发实现在计算时间和参数计数中都具有相同的成本作为传统的变压器层，但是在很长的序列中，语言建模任务中的语言建模任务的困惑极大地改善了。我们的模型比远程变压器XL基线的表现宽大，同时运行的速度是两倍。我们证明了它在PG19（书籍），Arxiv论文和GitHub源代码上的有效性。我们的代码已发布为开​​源。

稀疏的专家模型是一个三十年来的概念，作为深度学习中流行的建筑。这类体系结构包括专家的混合物，交换变压器，路由网络，基础层等，所有这些都以一个统一的想法，即每个示例都由参数的一个子集进行。通过这样做，稀疏度将参数计数与每个示例的计算分解，从而允许使用极大但有效的模型。最终的模型显示了各种领域的显着改善，例如自然语言处理，计算机视觉和语音识别。我们回顾了稀疏专家模型的概念，提供了对常见算法的基本描述，将深度学习时代的进步进行上下文化，并通过突出未来工作的领域来结束。

变形金刚在参加长语境时奋斗，因为计算量随着上下文长度而增长，因此它们不能有效地模拟长期存储器。已经提出了几种变体来缓解这个问题，但它们都有有限的内存容量，被迫降低旧信息。在本文中，我们提出了$ \ infty $ -former，它将Vanilla变压器与无限的长期记忆延伸。通过利用连续空间注意机制来参加长期内存，$ \ idty $ -former的注意力复杂性与上下文长度无关。因此，它能够在保持固定计算预算的同时进行任意长的上下文并维持“粘性存储器”。合成排序任务的实验展示了$ \ idty $ -former将信息从长序列中保留信息的能力。我们还通过培训从头开始培训模型以及微调预先培训的语言模型来执行语言建模实验，这表明了无限性的长期记忆的好处。

大型变压器模型在许多任务中产生令人印象深刻的结果，但培训昂贵，甚至微调，如此慢，在解码中，他们的使用和研究变得无法触及。我们通过利用稀疏性来解决这个问题。我们研究变压器中的所有层的稀疏变体，并提出缩放变压器，一个缩放变压器模型，使用稀疏层的型号有效地缩放，并在我们扩展模型大小时比标准变压器更快地执行不匹配的解码。令人惊讶的是，稀疏层足以获得与具有相同数量的参数的标准变压器相同的困惑。我们还与现有的稀疏性融合，即使存储器有限，也能够对长期序列进行快速推断。这导致在长期摘要上对最先进的表现竞争。

状态空间模型已显示在建模远距离依赖性方面有效，特别是序列分类任务。在这项工作中，我们着重于对英语书籍，GitHub源代码和Arxiv数学文章的自回旋序列建模。基于围绕封闭激活功能的有效性的最新发展，我们提出了一个名为“封闭状态空间（GSS）”的新层，并表明它的训练速度明显快于TPU的S4（即DSS）的对角线版本，具有相当竞争力 - 基于变压器的基线，并表现出零击向更长的输入，同时直接实施。最后，我们表明，利用自我意见来建模局部依赖性，可以进一步提高GSS的性能。

在这项工作中，我们介绍了内核化变压器，这是一个通用，可扩展的，数据驱动的框架，用于学习变压器中的内核功能。我们的框架将变压器内核作为光谱特征图之间的点产物近似，并通过学习光谱分布来学习内核。这不仅有助于学习通用的内核端到端，而且还可以减少变压器从二次到线性的时间和空间复杂性。我们表明，在准确性和计算效率方面，内核化的变压器实现了与现有的有效变压器体系结构相当的性能。我们的研究还表明，内核的选择对性能有重大影响，而内核学习变体是固定内核变压器的竞争替代方案，无论是长时间的序列任务。