Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism. To remedy this, we propose, BIGBIRD, a sparse attention mechanism that reduces this quadratic dependency to linear. We show that BIGBIRD is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our theoretical analysis reveals some of the benefits of having O(1) global tokens (such as CLS), that attend to the entire sequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to 8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context, BIGBIRD drastically improves performance on various NLP tasks such as question answering and summarization. We also propose novel applications to genomics data.

Transfer learning, where a model is first pre-trained on a data-rich task before being finetuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a;Radford et al., 2018), BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial taskspecific architecture modifications.BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

在本文中，我们试图通过引入深度学习模型的句法归纳偏见来建立两所学校之间的联系。我们提出了两个归纳偏见的家族，一个家庭用于选区结构，另一个用于依赖性结构。选区归纳偏见鼓励深度学习模型使用不同的单位（或神经元）分别处理长期和短期信息。这种分离为深度学习模型提供了一种方法，可以从顺序输入中构建潜在的层次表示形式，即更高级别的表示由高级表示形式组成，并且可以分解为一系列低级表示。例如，在不了解地面实际结构的情况下，我们提出的模型学会通过根据其句法结构组成变量和运算符的表示来处理逻辑表达。另一方面，依赖归纳偏置鼓励模型在输入序列中找到实体之间的潜在关系。对于自然语言，潜在关系通常被建模为一个定向依赖图，其中一个单词恰好具有一个父节点和零或几个孩子的节点。将此约束应用于类似变压器的模型之后，我们发现该模型能够诱导接近人类专家注释的有向图，并且在不同任务上也优于标准变压器模型。我们认为，这些实验结果为深度学习模型的未来发展展示了一个有趣的选择。

考虑了基于高维预测器的模式识别。定义了基于变压器编码器的分类器。分析了分类器朝向最佳错误分类概率的分类器的错误分类概率的收敛速率。结果表明，该分类器能够规避维度的诅咒，只要血管升性概率满足合适的分层组成模型。此外，通过考虑自然语言处理中的分类问题，理论上地在本文中地分析的变压器分类器之间的变压器分类器之间的差异，通过考虑自然语言处理中的分类问题来说明。

在这项工作中，我们介绍了内核化变压器，这是一个通用，可扩展的，数据驱动的框架，用于学习变压器中的内核功能。我们的框架将变压器内核作为光谱特征图之间的点产物近似，并通过学习光谱分布来学习内核。这不仅有助于学习通用的内核端到端，而且还可以减少变压器从二次到线性的时间和空间复杂性。我们表明，在准确性和计算效率方面，内核化的变压器实现了与现有的有效变压器体系结构相当的性能。我们的研究还表明，内核的选择对性能有重大影响，而内核学习变体是固定内核变压器的竞争替代方案，无论是长时间的序列任务。

变压器注意机制的二次计算和内存复杂性限制了对长序列建模的可扩展性。在本文中，我们提出了Luna，一种线性统一嵌套关注机制，使Softmax注意力具有两个嵌套线性关注功能，仅产生线性（与二次）的时间和空间复杂度相反。具体地，通过第一注意功能，LUNA将输入序列包装成固定长度的序列。然后，使用第二关注功能未包装包装序列。与更传统的关注机制相比，LUNA引入具有固定长度的附加序列作为输入和额外的相应输出，允许LUNA线性地进行关注操作，同时还存储足够的上下文信息。我们对三个序列建模任务的基准进行了广泛的评估：长上下文序列建模，神经机平移和大型预磨损的屏蔽语言建模。竞争甚至更好的实验结果表明了Luna的有效性和效率与各种各样相比

我们介绍了块状变压器，该变压器以序列的反复方式应用变压器层，并且相对于序列长度具有线性复杂性。我们的复发单元在训练过程中在代币的块而不是单个令牌上运行，并利用块内并行计算，以便有效利用加速器硬件。单元本身非常简单。它仅仅是一个变压器层：它使用自我注意事项和交叉注意力来有效计算大量状态向量和令牌上的复发函数。我们的设计部分受到LSTM单元的启发，它使用LSTM风格的大门，但它可以将典型的LSTM单元缩放为几个数量级。我们的复发实现在计算时间和参数计数中都具有相同的成本作为传统的变压器层，但是在很长的序列中，语言建模任务中的语言建模任务的困惑极大地改善了。我们的模型比远程变压器XL基线的表现宽大，同时运行的速度是两倍。我们证明了它在PG19（书籍），Arxiv论文和GitHub源代码上的有效性。我们的代码已发布为开​​源。

Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. Second, an enhanced mask decoder is used to incorporate absolute positions in the decoding layer to predict the masked tokens in model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understand (NLU) and natural langauge generation (NLG) downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). Notably, we scale up DeBERTa by training a larger version that consists of 48 Transform layers with 1.5 billion parameters. The significant performance boost makes the single DeBERTa model surpass the human performance on the SuperGLUE benchmark (Wang et al., 2019a) for the first time in terms of macro-average score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of January 6, 2021, outperforming the human baseline by a decent margin (90.3 versus 89.8). The pre-trained DeBERTa models and the source code were released at: https://github.com/microsoft/DeBERTa 1 .

Deep Learning and Machine Learning based models have become extremely popular in text processing and information retrieval. However, the non-linear structures present inside the networks make these models largely inscrutable. A significant body of research has focused on increasing the transparency of these models. This article provides a broad overview of research on the explainability and interpretability of natural language processing and information retrieval methods. More specifically, we survey approaches that have been applied to explain word embeddings, sequence modeling, attention modules, transformers, BERT, and document ranking. The concluding section suggests some possible directions for future research on this topic.

我们提出了一项实证研究，以适应现有的经过验证的文本对文本模型，以备长期输入。通过沿预训练管道的三个轴的全面研究 - 模型架构，优化目标和训练式语料库，我们提出了一种有效的食谱，以从现有的短篇小说模型中构建长篇小说模型。具体而言，我们用汇总仪的块关注替换了变压器中的全部注意力，并使用蒙版的跨度预测任务为模型预算，长度不同。就训练训练的语料库而言，我们发现，与使用通常在其域覆盖范围中通常受到限制的现有长文档语料库相比，使用大型开放域语料库的随机串联的短篇小说可以提高性能。通过这些发现，我们建立了一个长篇文本模型，该模型可以在长篇文本质量检查任务上实现竞争性能，并在五个长文本摘要数据集上建立新的最新技术，通常优于先前的方法，具有较大的模型大小。

基于变压器的预审前的语言模型（LMS）在自然语言的理解中无处不在，但由于其二次复杂性，无法应用于故事，科学文章和长文档等长序列。尽管已经提出了无数有效的变压器变体，但它们通常是基于需要从头开始的昂贵预处理的自定义实现。在这项工作中，我们提出了雪橇：滑动编码器和解码器，这是一种处理长序列的简单方法，可以重新使用和利用经过战斗测试的短文本预处理的LMS。具体而言，我们将输入分配到重叠的块中，用短文本LM编码器编码每个块，然后使用预审计的解码器将信息融合到跨块（Fusion-In-In-In-In-indecoder）之间。我们通过受控实验说明，雪橇提供了一种可行的策略，可以长期理解并评估我们在卷轴上的方法，这是一个基准，该基准在各种语言理解任务中具有七个数据集。我们发现，雪橇与高达50倍的专业型号具有竞争力，并且需要专用且昂贵的预处理步骤。

最近的工作表明，（1）增加输入长度或（2）增加模型大小可以提高基于变压器的神经模型的性能。在本文中，我们提出了一个名为Longt5的新模型，我们探讨了同时缩放输入长度和模型大小的效果。具体而言，我们综合了从长输入变压器（ETC）的关注思路，并采用了从摘要预训练（PEGASU）的预训练策略进入可扩展的T5架构。结果是我们称之为{\ EM瞬态全球}（TGLOBAL）的新关注机制，这些机制是模仿等本地/全球注意力机制，但不需要额外的侧面输入。我们能够实现最先进的结果，以若干摘要任务，优于问题应答任务的原始T5模型。

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.

本文对过去二十年来对自然语言生成（NLG）的研究提供了全面的审查，特别是与数据到文本生成和文本到文本生成深度学习方法有关，以及NLG的新应用技术。该调查旨在（a）给出关于NLG核心任务的最新综合，以及该领域采用的建筑;（b）详细介绍各种NLG任务和数据集，并提请注意NLG评估中的挑战，专注于不同的评估方法及其关系;（c）强调一些未来的强调和相对近期的研究问题，因为NLG和其他人工智能领域的协同作用而增加，例如计算机视觉，文本和计算创造力。

目前，用于训练语言模型的最广泛的神经网络架构是所谓的BERT，导致各种自然语言处理（NLP）任务的改进。通常，BERT模型中的参数的数量越大，这些NLP任务中获得的结果越好。不幸的是，内存消耗和训练持续时间随着这些模型的大小而大大增加。在本文中，我们调查了较小的BERT模型的各种训练技术：我们将不同的方法与Albert，Roberta和相对位置编码等其他BERT变体相结合。此外，我们提出了两个新的微调修改，导致更好的性能：类开始终端标记和修改形式的线性链条条件随机字段。此外，我们介绍了整个词的注意力，从而降低了伯特存储器的使用，并导致性能的小幅增加，与古典的多重关注相比。我们评估了这些技术的五个公共德国命名实体识别（NER）任务，其中两条由这篇文章引入了两项任务。

本次调查绘制了用于分析社交媒体数据的生成方法的研究状态的广泛的全景照片（Sota）。它填补了空白，因为现有的调查文章在其范围内或被约会。我们包括两个重要方面，目前正在挖掘和建模社交媒体的重要性：动态和网络。社会动态对于了解影响影响或疾病的传播，友谊的形成，友谊的形成等，另一方面，可以捕获各种复杂关系，提供额外的洞察力和识别否则将不会被注意的重要模式。

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. Span-BERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT large , our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE. 1 * Equal contribution. 1 Our code and pre-trained models are available at https://github.com/facebookresearch/ SpanBERT.

我们通过与与前面令牌的局部相似度，通过调节从大语料库检索的文档块来增强自动回归语言模型。尽管使用25美元\时分，我们的检索增强型变压器（RetroCro）的检索增强型变压器（RetroCr）对GPT-3和侏罗纪-1获得了可比性的性能。微调后，复古表演转换为下游知识密集型任务，如问题应答。复古结合了冷冻BERT猎犬，一种可微分的编码器和块状的横向机制，以预测基于数量级的令牌，而不是训练期间通常消耗的数量。我们通常从头开始训练复古，还可以快速改造预先接受的变压器，通过检索，仍然达到良好的性能。我们的工作通过以前所未有的规模开辟了通过显式内存改进语言模型的新途径。

We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness. To approximate softmax attentionkernels, Performers use a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+), which may be of independent interest for scalable kernel methods. FAVOR+ can also be used to efficiently model kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with other kernels for the first time on large-scale tasks, beyond the reach of regular Transformers, and investigate optimal attention-kernels. Performers are linear architectures fully compatible with regular Transformers and with strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix, uniform convergence and low estimation variance. We tested Performers on a rich set of tasks stretching from pixel-prediction through text models to protein sequence modeling. We demonstrate competitive results with other examined efficient sparse and dense attention methods, showcasing effectiveness of the novel attention-learning paradigm leveraged by Performers.